Import of data into PowerBI from Azure Data Explorer fails

[mschwit]

🐛 Problem

Update of Power BI reports fails with threshold errors

👣 Repro steps

1. Setup Finops data import with Azure Data Explorer
2. Configure Power BI Reports CostSummary and RateOptimization (KQL) to import Finops data
3. Import fails with different errors like:
   CostSummary report: Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions.. [0]Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException: Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions.. Timestamp=2024-12-15T00:41:18.7094667Z ClientRequestId=KPBI;d07a1f36-755d-4b15-9f17-6c393ae63469;492885ec-9d57-4741-a739-f9aa5eb04984;ae4ed709-4a5b-4bb3-9918-e1afc1e7c716 ActivityId=7a3a7cb7-93a4-4cfd-9b84-c1acfa1fcb76 ActivityType=GW.Http.CallContext ServiceAlias=FINOPSTESTMCB MachineName=KEngine000000 ProcessName=Kusto.WinSvc.Svc ProcessId=4348 ThreadId=8980 ActivityStack=(Activity stack: CRID=KPBI;d07a1f36-755d-4b15-9f17-6c393ae63469;492885ec-9d57-4741-a739-f9aa5eb04984;ae4ed709-4a5b-4bb3-9918-e1afc1e7c716 ARID=7a3a7cb7-93a4-4cfd-9b84-c1acfa1fcb76 > GW.Http.CallContext/7a3a7cb7-93a4-4cfd-9b84-c1acfa1fcb76) MonitoredActivityContext=(ActivityType=GW.Http.CallContext, Timestamp=2024-12-15T00:38:43.7394762Z, ParentActivityId=7a3a7cb7-93a4-4cfd-9b84-c1acfa1fcb76, TimeSinceStarted=154969.993 [ms])ErrorCode= ErrorReason= ErrorMessage= DataSource= DatabaseName= ClientRequestId= ActivityId=00000000-0000-0000-0000-000000000000 UnderlyingErrorCode=80DA0003 UnderlyingErrorMessage=The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions. . The exception was raised by the IDbCommand interface. Table: TagCosts.

RateOptimization: Expression.Error: We cannot apply field access to the type Null.. Microsoft.Data.Mashup.ErrorCode = 10335. Key = Tables. ;We cannot apply field access to the type Null.. The exception was raised by the IDbCommand interface. Table: HybridBenefitCosts.

RateOptimization: Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions.. [0]Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException: Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions.. Timestamp=2024-12-11T09:41:14.8226656Z ClientRequestId=KPBI;10e3fd75-3038-415f-8afa-33a764cbe1c9;3dee0e7e-4549-4001-bc84-98eca4126ea4;9f3f709a-dcc7-4282-984f-93b9e4422f02 ActivityId=e6b3b445-ef57-4e21-9273-9b40c847ba29 ActivityType=GW.Http.CallContext ServiceAlias=FINOPSTESTMCB MachineName=KEngine000000 ProcessName=Kusto.WinSvc.Svc ProcessId=9116 ThreadId=4268 ActivityStack=(Activity stack: CRID=KPBI;10e3fd75-3038-415f-8afa-33a764cbe1c9;3dee0e7e-4549-4001-bc84-98eca4126ea4;9f3f709a-dcc7-4282-984f-93b9e4422f02 ARID=e6b3b445-ef57-4e21-9273-9b40c847ba29 > GW.Http.CallContext/e6b3b445-ef57-4e21-9273-9b40c847ba29) MonitoredActivityContext=(ActivityType=GW.Http.CallContext, Timestamp=2024-12-11T09:40:23.2322805Z, ParentActivityId=e6b3b445-ef57-4e21-9273-9b40c847ba29, TimeSinceStarted=51590.3851 [ms])ErrorCode= ErrorReason= ErrorMessage= DataSource= DatabaseName= ClientRequestId= ActivityId=00000000-0000-0000-0000-000000000000 UnderlyingErrorCode=80DA0003 UnderlyingErrorMessage=The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions. . The exception was raised by the IDbCommand interface. Table: HybridBenefitCosts.

RateOptimization: Query execution lacks memory resources to complete (80DA0007): Partial query failure: Low memory condition (E_LOW_MEMORY_CONDITION). (message: 'hr: '2161770503' 'Engine under memory pressure, context: sequence datum materialize': ', details: 'StgError { kind: HResult(2161770503, "Engine under memory pressure, context: sequence datum materialize"), source: None }'). [0]Kusto.Data.Exceptions.KustoServicePartialQueryFailureLowMemoryConditionException: Query execution lacks memory resources to complete (80DA0007): Partial query failure: Low memory condition (E_LOW_MEMORY_CONDITION). (message: 'hr: '2161770503' 'Engine under memory pressure, context: sequence datum materialize': ', details: 'StgError { kind: HResult(2161770503, "Engine under memory pressure, context: sequence datum materialize"), source: None }'). Timestamp=2024-12-09T11:58:29.0251058Z ClientRequestId=KPBI;0ad649d9-b136-414d-90af-8b981b3b0b8a;ff38f981-440a-46d4-a465-76c8edeb8921;d5a0dfd8-d9ff-4853-94de-8d5d500b5813 ActivityId=ff26eec5-f70c-4565-adcd-1913f4127128 ActivityType=GW.Http.CallContext ServiceAlias=FINOPSTESTMCB MachineName=KSEngine000001 ProcessName=Kusto.WinSvc.Svc ProcessId=5228 ThreadId=5432 ActivityStack=(Activity stack: CRID=KPBI;0ad649d9-b136-414d-90af-8b981b3b0b8a;ff38f981-440a-46d4-a465-76c8edeb8921;d5a0dfd8-d9ff-4853-94de-8d5d500b5813 ARID=ff26eec5-f70c-4565-adcd-1913f4127128 > GW.Http.CallContext/ff26eec5-f70c-4565-adcd-1913f4127128) MonitoredActivityContext=(ActivityType=GW.Http.CallContext, Timestamp=2024-12-09T11:57:59.9530449Z, ParentActivityId=ff26eec5-f70c-4565-adcd-1913f4127128, TimeSinceStarted=29072.0609 [ms])ErrorCode= ErrorReason= ErrorMessage= DataSource= DatabaseName= ClientRequestId= ActivityId=00000000-0000-0000-0000-000000000000 UnderlyingErrorCode=80DA0007 UnderlyingErrorMessage=Partial query failure: Low memory condition (E_LOW_MEMORY_CONDITION). (message: 'hr: '2161770503' 'Engine under memory pressure, context: sequence datum materialize': ', details: 'StgError { kind: HResult(2161770503, "Engine under memory pressure, context: sequence datum materialize"), source: None }') . The exception was raised by the IDbCommand interface. Table: HybridBenefitCosts.

TagCosts: Query execution has exceeded the allowed limits (80DA0001): Partial query failure: Runaway query (E_RUNAWAY_QUERY). (message: hr: '2161770497' 'Aggregated string column size (8590603827 bytes) has exceeded the maximal allowed size (8589934592 bytes) during query evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)': ).

🤔 Expected

Import of data should work without errors

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.