In last week’s post we looked at how to combine multiple files together using Power Query. This week we’re going to stay within the same workbook, and combine multiple worksheets using Power Query.
Power Query Combine Multiple Rows
Let’s consider a case where the user has been creating a transactional history in an Excel file. It is all structured as per the image below, but resides across multiple worksheets; one for each month:
In Power Query, you can merge two or more columns in your query. You can merge columns to replace them with one merged column, or create a new merged column alongside the columns that are merged. You can only merge columns of a Text data type. The following data is used in the examples. Here are the steps to combine multiple worksheets with Excel Tables using Power Query: Go to the Data tab. In the Get & Transform Data group, click on the ‘Get Data’ option. Go the ‘From Other Sources’ option. Click the ‘Blank Query’ option. This will open the Power Query editor. If you have no other columns, you use the query editor to add a custom column with formula: = Text.Combine (List.Select (Record.FieldValues , each ' and null),' & ') Specializing in Power Query Formula Language (M) Message 2 of 18.
As you can see, they’ve carefully named each sheet with the month and year. But unfortunately, they haven’t formatted any of the data using Excel tables.
Now the file lands in our hands (you can download a copy here if you’d like to follow along,) and we’d like to turn this into one consolidated table so that we can do some analysis on it.
Naturally we’re going to reach to Power Query to do this, but how do we get started? We could just go and format the data on each worksheet as a table, but what if there were hundreds? That would take way too much work!
But so far we’ve only seen how to pull Tables, Named Ranges or files into Power Query. How do we get at the worksheets?
Basically, we’re going to start with two lines of code:
- Go to Power Query –> From Other Sources –> Blank Query
- View –> Advanced Editor
You’ll now see the following blank query:
Source = '
What we need to do is replace the second line (Source = “”) with the following two lines of code:
FullFilePath = 'D:TempCombine Worksheets.xlsx',
Source = Excel.Workbook(File.Contents(FullFilePath))
Of course, you’ll want to update the path to the full file path for where the file is saved on your system.
Once you click Done, you should see the following:
Cool! We’ve got a list of all the worksheets in the file!
The next step is to prep the fields we want to preserve as we combine the worksheets. Obviously the Name and Item columns are redundant, so let’s do a bit of cleanup here.
- Remove the Kind column
- Select the Name column –> Transform –> Data Type –> Date
- Select the Name column –> Transform –> Date –> Month –> End of Month
- Rename the Name column to “Date”
At this point, the query should look like so:
Next we’ll click the little double headed arrow to the top right of the data column to expand our records, and commit to expanding all the columns offered:
Hmm… well that’s a bit irritating. It looks like we’re going to need to promote the top row to headers, but that means we’re going to overwrite the Date column header in column 1. Oh well, nothing to be done about it now, so:
- Transform –> Use First Row As Headers –> Use First Row As Headers
- Rename Column1 (the header won’t accept 1/31/2008 as a column name) to “Date” again
- Rename the Jan 2008 column (far right) to “Original Worksheet”
We’re almost done, but let’s just do a bit of final cleanup here. As we set the data types correctly, let’s also make sure that we remove any errors that might come up from invalid data types.
- Select the Date column
- Home –> Remove Errors
- Set Account and Dept to Text
- Set Amount to Decimal Number
- Select the Amount column
- Home –> Remove Errors
- Set Original Worksheet to Text
Rename the query to “Consolidated”, and load it to a worksheet.
Before you do anything else, Save the File.
To be fair, our query has enough safe guards in it that we don’t actually have to do this, but I always like to play it safe. Let’s review the completed query…
Edit the Consolidated query, and step into the Source line step. Check out that preview pane:
Interesting… two more objects! This makes sense, as we created a new table and worksheet when we retrieved this into a worksheet. We need to filter those out.
Getting rid of the table is easy:
- Select the drop down arrow on the Kind column
- Uncheck “Table”, then confirm when asked if you’d like to insert a step
Select the next couple of steps as well, and take a look at the output as you do.
Aha! When you hit the “ChangedType” step, something useful happens… we generate an error:
Let’s remove that error from the Name column.
Power Query Combine Files With Different Columns
- Select the Name column –> Home –> Remove Errors
Power Query Combine Tabs
And we’re done. We’ve managed to successfully combine all the data worksheets in our file into one big table!
This method creates a bit of a loop in that I’m essentially having to reach outside Excel to open a copy of the workbook to pull the sheet listing in. And it causes issues for us, since Power Query only reads from the last save point of the external file we’re connecting to (in this case this very workbook.) I’d way rather have an Excel.CurrentWorkbook() style method to read from inside the file, but unfortunately that method won’t let you read your worksheets.
It would also be super handy to have an Excel.CurrentWorkbookPath() method. Hard coding the path here is a real challenge if you move the file. I’ve asked Microsoft for this, but if you think it is a good idea as well, please leave a comment on the post. (They’ll only count one vote from me, but they’ll count yours if you leave it here!)