Hubs v0.1 – Power BI on CM exports

[flanakin]

🚀 Goal

Create an ARM template that configures a new Cost Management export to ADLS and generates a Power BI report that can work against that dataset. The template should also include a Data Factory pipeline to delete old files when data is exported each day.

📋 Tasks

v0.0.1 - finops-hub template (discuss)

Preview Give feedback

1. Power BI on raw CM exports spec #5
   Skill: Documentation +1
2. Generate aka.ms links for all report pages
3. Create finops-hub template with storage account #21
   Skill: Deployment +1
4. Split out 0.1 into smaller releases
5. Create basic Power BI reports #24
   Tool: Power BI +1
   
6. Document end-to-end setup for initial release #29
   Skill: Documentation +1
   
7. [Power BI] Move all dataset processing from PBI to the data layer #37
   Tool: Power BI +1
   
8. Document telemetry #51
   Skill: Deployment Skill: Documentation +2
   
9. finops-hub template with data factory instance #23
   Skill: Data factory Skill: Deployment +2
   
10. Update docs based on revised plans #58
    Skill: Documentation +1
    
11. Document branching strategy #65
    Skill: DevOps +1
    
12. Add containers and settings to storage #34
    Skill: Deployment +1
    
13. Pipeline to dedupe and convert to parquet #59
    Skill: Data factory +1
    
14. Automate Azure Quickstart Templates repo publishing #64
    Skill: Deployment Tool: PowerShell +2
    
15. Fix deployment warnings #124
    Skill: Deployment Type: Bug 🐛 +2
    
16. Auto-start the msexports tranform pipeline #126
    Skill: Data factory Skill: Deployment Tool: PowerShell +3
    
17. Enable usage by pipeline in Data Factory #125
    Skill: Data factory Skill: Deployment +2
    
18. Update Power BI data sources to use parquet #63
    1 of 1
    Status: 🔬 Code review Tool: FinOps hubs Tool: Power BI +3
    
19. Test deployment
20. Finalize documentation for Hubs v0.0.1 #144
    13 of 13
    Skill: Documentation Status: 🔄️ In progress Tool: FinOps hubs Type: Cleanup +4
    
21. Publish v0.0.1 template to Azure Quickstart Templates #26
    2 of 2
    Skill: Deployment Tool: FinOps hubs +2
    
22. Sync feature branches
23. Create new release
24. Release maintenance
25. Address all feedback from the release discussion

v0.0.2 - Self-managed exports (discuss)

Preview Give feedback

1. Minor AQT doc updates #168
   Good first issue 🏆 Micro PR 🔬 Skill: Documentation Status: 📦 Pending release Tool: FinOps hubs Type: Bug 🐛 +6
   
2. Power BI calculated tables #187
   Status: 📦 Pending release Tool: FinOps hubs Tool: Power BI +3
   
3. Set the default SKU in the hub CreateUiDefinition #186
   Status: 📦 Pending release Tool: FinOps hubs Type: Bug 🐛 +3
   
4. Ingest data across hub instances
5. Add actual cost data
6. Historical data refresh (rerating)
7. [Hubs] Managed exports – Create exports on behalf of customers #60
   Skill: Data factory Skill: Deployment Status: ✅ Released Tool: FinOps hubs +4
   
8. [Hubs] Historical data backfill – Preload historical data when a scope is added #67
   Skill: Data factory Tool: FinOps hubs +2
   
9. [PowerShell] New-FinOpsServicePrincipal – Manage billing account service principals #130
   Tool: PowerShell Type: Feature 💎 +2
   
10. Create pre/post-process scripts
11. Setup PR build
12. Power BI linked reports/datasets
13. Make sure all resources are tagged
14. Address all feedback from the release discussion

v0.0.3 - Additional datasets (discuss)

Preview Give feedback

1. [Hubs] Connector datasets – Add data parity with the Cost Management Power BI connector #83
   Skill: Data factory Tool: FinOps hubs Type: Feature 💎 +3
2. Static website to manage config
3. Address all feedback from the release discussion

v0.1 - FOCUS schema (discuss)

Preview Give feedback

1. Add normalization as a template parameter
2. Interim normalized schema spec #76
   Skill: Data factory Skill: Documentation Status: ✍️ Spec in progress +3
   
3. Transform CM exports to a normalized schema #81
   Skill: Data factory Status: ✍️ Spec in progress +2
4. Update Power BI reports to use new schema
5. Use FOCUS schema for cost details #43
   Skill: Data factory Type: Feature 💎 +2
6. Add ADX cluster/database
7. Define union schema in parquet
8. Add ADX union table
9. Add ADX FOCUS table and update policy
10. Add EventGrid trigger to ingest data into ADX
11. Change ADF to move parquet to archive/raw container
12. Address all feedback from the release discussion

Stretch

Preview Give feedback

1. Data issue: Add unique sub name, RG name, and resource name columns #38
   Needs: Information Tool: Power BI +2
2. Do we have a way to protect from people exporting data multiple times using parent scopes? At a minimum, cover this as a warning in docs.
3. Create feature proposal for deltas in Cost Management exports
4. Create feature proposal for how to deploy from Cost Management
5. Create feature proposal for Power BI code reuse across reports
6. Create feature proposal for Power BI versioning and targeted code reviews
7. Management groups with unsupported subscriptions (where exports don't work today)
8. Data freshness setting (refresh rate; currently 24h)
9. Commitment purchases (or actual+amortized as one)
10. Forecasting
11. Optimize storage with deltas
12. Post-ingestion validation
13. Historical data refresh
14. Cross-channel credit handling (e.g., EA MC, MACC)
15. Cost Management portal integration
16. Publish to Azure Marketplace
17. Hourly data via UsageAggregates x Pricesheet
18. Unsupported subscriptions (classic sponsorship/CSP) via UsageAggregates x Pricesheet
19. Consumption Insights connector migration support
20. [Hubs] Add support for the China cloud #259
    Tool: FinOps hubs Type: Feature 💎 +2
    
21. AWS support
22. GCP support

🙋‍♀️ Ask for the community

We'd love to get your feedback! Please leave any questions, comments, or change requests in the release discussions at the bottom of each task list above. We'll try to close each of those out before we finalize the release.

[flanakin]

Moving to projects