Hello from France
I am brand new on this forum, and also new on power query. I have an issue, 2 ideas to solve it, but none of them works properly.
I have an Oracle database connected to a factory supervision, collecting data every 1 to 5 seconds. Due to the large amount of data, connecting directly to the tables without filtering is too long.
So I would like to get hourly data (pick one data every hour or average one hour data) between a start date and a end date, and of course do this directly in the query not to load the full amount of data.
My first method is :
when connecting with an ODBC connection, I directly enter my SQL query , using an aggregate table.
select
name, ts, max, avg
from
aggregates
where
(name='4.REAC1.K+.PPM.PV')
and
ts between '01-JAN-18 00:00:00.0' and '24-OCT-18 00:00:00.0'
and
period=1:00
I created 2 tables linked to Excel cells, to be able to replace the dates by parameters. But when I replace one of the dates by my parameter as below,
let
Source = Odbc.Query("dsn=IP21 64bits", "select name, ts, max, avg from aggregates where (name='4.REAC1.K+.PPM.PV') and ts between '" & datedebut &"' and '24-OCT-18 00:00:00.0' and period=1:00 and STEPPED = 1")
in
Source
I've got an error.
Formula.Firewall: Query 'Query1' (step 'Source') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination
Strangely, the SQL chain looks the same when using both methods, I do not understand why it doesn't work...
2nd method
directly use Language M to perform the query. But I honestly do not know how to do it, as my know ledge in SQL and in this language is very poor.
I know how to add start date and end date (not done in the code below), but I don't know how to get only hourly data, and not load all the data to transform them after. My basic code is
let
Source = Odbc.DataSource("dsn=IP21 64bits", [HierarchicalNavigation=true]),
IP_AIDef_1_Table = Source{[Name="IP_AIDef_1",Kind="Table"]}[Data]
in
IP_AIDef_1_Table
So any assistance for one of this method is welcome !!
Hope I was clear in my explanations.
regards
Michael
Hi Michael,
Great to see you're giving Power Query a try.
I recommend method 2 because this will be more efficient for Power Query. Once you have loaded your SQL table into Power Query (it only loads a preview) you can click on the time column filter buttons and filter for time between specific days/hours. Note: you may need to set the data type (Home tab) for the columns containing your dates and or time to date/time first.
Once you have your data filtered you can 'Group By' (Home tab) the days/hours to get an average before loading the final query into Excel or the Data Model.
Mynda
Hi Mynda
and thank you very much for your help and quick answer. I managed to get the preview as you explained, and then filter with the dates. At that point, here is my first question : this will automatically decrease the number of data retrieved by the query, right ? Query will not retrieve all the data and then filter them, but only pick the desired data ?
Then I tried to make the "group by" command. But I had an error, "error occurred but no ODBC error information was available". I also didn't see any criteria for the "group by" : how the system will know that I want to group data by hour, or minutes, or days ? Sorry for my silly questions, as I said I'm a newbie in SQL...
But I understand the principles of what you explained, which is already excellent ! Thanks !
regards
Michael
Hi Michael,
Correct, Power Query only imports the data that meets the filter criteria. i.e. you're reducing the data you bring into Excel so that only the data you need to work with is imported.
In this post I use the Group By tool, so that will show you what the 'Group by' dialog box should look like, but it sounds like it's time to learn Power Query properly. The forum is a place to help you if you get stuck, as opposed to teaching you step by step how to use Power Query.
It really is an amazing tool and quite quick to learn. You don't need to know or learn SQL to use Power Query. 99% of the features can be achieved using the GUI. If you're interested I have a Power Query course here.
Mynda