Forum

Multiple csv-files ...
 
Notifications
Clear all

Multiple csv-files and changing content

14 Posts
3 Users
0 Reactions
254 Views
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

Hi,

I like your examples with PowerQuery combining multiple csv-files and creating great pivots and graphs.

We started to rebuild our PerformancePortal with these techniques.

What is the best way to implement a change in file content? For instance: at some moment we want/need to have extra columns to show specific information. Is it possible to include newer columns into the 'merged' data model without having to change all previous csv-files?

 

kind regards, Maarten

 
Posted : 09/10/2020 3:51 am
(@catalinb)
Posts: 1937
Member Admin
 

Hi Maarten,

See this post for a solution, download that file attached: https://www.myonlinetraininghub.com/excel-forum/power-query/how-to-remove-duplicated-headers-when-importing-from-folder#p16593

 
Posted : 09/10/2020 11:12 am
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

Thanks for the response.

But: apparently it will not or cannot work for files from a SharePoint map. Do you have a likewise solution for that environment?

And I suspect that this solution deals with the adding of new files, but my real question was about file content.

For instance we have a lot of monthly files with project data, and starting from this month we want to register the responsabel account manager,

that will be registered in a new column.

 
Posted : 10/10/2020 9:39 am
(@catalinb)
Posts: 1937
Member Admin
 

What is a SharePoint Map? Is that a sharepoint site documents?

If yes, you can simply Sync that folder locally, and you will always have files syncronized with sharepoint site. Use the local synced folder as source for extracting data.

 
Posted : 10/10/2020 9:46 am
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

The power of using files from a SharePoint folder is that everyonbe with SharePoint rights can update the file; that's not possible when we use a synced folder.

But still no answer to my question: how to deal with new (necessary) columns.

 
Posted : 13/10/2020 2:59 am
(@catalinb)
Posts: 1937
Member Admin
 

Actually, the link I provided offers a solution where it does not matter how many columns are in the source files, so you do have an answer. My question in fact does not have an answer 🙂 ("What is a SharePoint Map?")

All you have to do is to change the query to get data from sharepoint folder instead of local folder. The function that reads the files will consider any new columns added.

 
Posted : 13/10/2020 3:17 am
(@catalinb)
Posts: 1937
Member Admin
 

The power of using files from a SharePoint folder is that everyonbe with SharePoint rights can update the file; that's not possible when we use a synced folder.

It's about how we read the data, not how we EDIT files.
Syncing the sharepoint folder with your desktop will only change how you READ the files: you read them from the local folder, instead of reading from web.
OneDrive Sync agent will take care of bringing the online changes to your computer, so I did not suggested editing the source files OFFLINE.

 
Posted : 13/10/2020 3:21 am
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

I'm sorry for the misunderstandings

  1. in Dutch a 'map' stands for a 'folder', so I mean the csv-files are in a SharePoint folder
  2. with "everyone with SharePoint rights can update the file" I mean 'update the data' within the Excel file with the Power Pivot and graphs (also located in SharePoint) , not the underlying cvs-files
  3. "The function that reads the files will consider any new columns added"; I just tried that, but the extra columns in the newest csv-files didn't show up, so what function do you mean?
 
Posted : 13/10/2020 7:15 am
(@catalinb)
Posts: 1937
Member Admin
 

Please upload a few sample csv files with different headers and just 1-2 rows of fake data.
Will prepare a sample.

 
Posted : 13/10/2020 10:27 am
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

I have some data csv files and a simple Excel sheet to read them.

The first 3 (folder Data) are the 'standard' ones, in the folder NewData I added one with extra columns.

In the Excel sheet I got the data of the first 3 by 'reading csv from folder'.

After making pivot table I placed the 4th file and updates all data, but no extra columns.

 
Posted : 15/10/2020 8:23 am
Philip Treacy
(@philipt)
Posts: 1629
Member Admin
 

Hi Maarten,

Did you mean to attach files? You need to click the Start Upload button after selecting the files.

Phil

 
Posted : 15/10/2020 8:31 am
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

Sorry, I didn't notice that!

 
Posted : 15/10/2020 10:42 am
(@catalinb)
Posts: 1937
Member Admin
 

Using the automatic file combiner is not dynamic. It ONLY reads the first file in that folder to get the headers and ignore all other headers from ALL other files.

Just use the file I already provided.

File attached here in case you don't want to download it from the other post i indicated.

Note that I had to change the csv delimiter, in your csv's the delimiter is semicolon instead of comma.

 
Posted : 15/10/2020 11:29 am
(@mastermd)
Posts: 35
Trusted Member
Topic starter
 

OK, thanks!
I will use it in our sheets.

And yes: in Dutch regional settings we always have to change things (. -> ,  and , -> 😉

 
Posted : 16/10/2020 6:56 am
Share: