[Hubs] Support for large datasets

[flanakin]

📝 Scenario

As a FinOps practitioner, I need to ingest data into a queryable data store in order to report on data at scale beyond $5M/mo

💎 Solution

Support large datasets (e.g., 500 GB/mo) with up to 7 years of historical data that refreshes when changed by adding an option to ingest data into Azure Data Explorer and update reporting to leverage that database.

📋 Tasks

Required tasks

Preview Give feedback

1. Decide on data store: SQL, ADX, Synapse
2. [Hubs] ADX ingestion – Ingest data into ADX #300
   Needs: Information Tool: FinOps hubs +2
   
3. [Hubs] ADX infra – Add ADX to the deployment #301
   Tool: FinOps hubs +1
   
4. [Hubs] ADX table – Create cost details table #376
   Tool: FinOps hubs +1
   
5. Update to ingest FOCUS 1.0
6. De-duplicate data that gets re-exported by Cost Management
7. Confirm the ADX SKU
8. Update Power BI reports
9. [Hubs] ADX – Documentation updates #670
   Skill: Documentation Tool: FinOps hubs +2
10. [Hubs] ADX – PowerShell updates #671
    Tool: FinOps hubs +1
11. Update CreateUiDefinition.json
12. Create pipeline to start/shutdown the ADX cluster based on settings.json config
13. Auto-start ADX before and shutdown after ingestion
14. Run CM exports on a custom schedule

Stretch goals

Preview Give feedback

1. Backfill all data in storage during setup
2. Implement retention policies for parquet data
3. Should we archive parquet data after ingestion?
4. [Hubs] ADX reports – Update Power BI to use Kusto #377
   Needs: Information Tool: FinOps hubs Tool: Power BI +3
5. [Hubs] ADX retention – Add retention for Kusto + set the values on the database #667
   Needs: Information Tool: FinOps hubs +2
6. [Hubs] ADX name check #668
   Needs: Information Tool: FinOps hubs +2

ℹ️ Additional context

There was an internal analysis of the optimal data store to use for the largest datasets and Azure Data Explorer was deemed to be the best option that balanced cost, performance, and scale.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.

[flanakin]

Closing this since we're tracking releases in a new way now and this is outdated.

[t-esslinger]

Hello @flanakin, is this feature still in your backlog? We would we highly interested in being able to handle also larger datasets more easily.

[flanakin]

@t-esslinger Sorry for missing the comment. Yes, this is still in the backlog. We're making progress slowly. I'm reopening this issue to track everything needed.