Skip to content

Commit

Permalink
Update CommitmentDiscountUsage transform to use parse_resourceid (#1219)
Browse files Browse the repository at this point in the history
  • Loading branch information
flanakin authored Dec 27, 2024
1 parent 5348d71 commit 4ad6bcc
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 8 deletions.
3 changes: 2 additions & 1 deletion docs/_resources/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,14 @@ Legend:
> ✏️ Changed:
>
> 1. Update required permissions on hubs page [Required permissions](../docs/_reporting/hubs/README.md).
> 1. ETL Pipeline - Fixed timezones for ADF triggers to resolve issue where triggers would not start due to unrecognized timezone.
> 1. Changed the **enablePublicAccess** parameter to exclude network components.
> - When disabled, a VNet will be created along with the required private endpoints and DNS zones to function in a fully private manner.
> 1. Updated `CommitmentDiscountUsage_transform_v1_0()` to use `parse_resourceid()`.
>
> 🛠️ Fixed:
>
> 1. Improved performance and memory consumption in the `parse_resourceid()` function to address out of memory errors during cost data ingestion ([#1188](https://github.com/microsoft/finops-toolkit/issues/1188))
> 1. Fixed timezones for Data Factory triggers to resolve issue where triggers would not start due to unrecognized timezone.
> 1. Fixed an issue where `x_ResourceType` is using the wrong value.
> - This fix resolves the issue for all newly ingested data.
> - To fix historical data, reingest data using the `ingestion_ExecuteETL` Data Factory pipeline.
Expand Down
12 changes: 5 additions & 7 deletions src/templates/finops-hub/modules/scripts/IngestionSetup.kql
Original file line number Diff line number Diff line change
Expand Up @@ -1246,12 +1246,10 @@ CommitmentDiscountUsage_transform_v1_0()
//
// Handle resource columns
| extend ResourceId = tolower(InstanceId)
// TODO: Switch to use parse_resourceid()
| parse ResourceId with "/subscriptions/" SubAccountId "/resourcegroups/" x_ResourceGroupName "/providers/" tmp_ResourceProvider "/" tmp_ResourceProviderPath
| extend segments = split(tmp_ResourceProviderPath, '/')
| mv-apply element = segments on (extend index = row_number() | where index % 2 == 0 | summarize ResourceName = strcat_array(make_list(element), '/'))
| mv-apply element = segments on (extend index = row_number() | where index % 2 == 1 | summarize x_ResourceType = strcat_array(make_list(element), '/'))
| extend x_ResourceType = strcat(tmp_ResourceProvider, '/', x_ResourceType)
| extend tmp_ResourceDetails = parse_resourceid(ResourceId)
| extend SubAccountId = tmp_ResourceDetails.SubAccountId
| extend x_ResourceGroupName = tmp_ResourceDetails.x_ResourceGroupName
| extend x_ResourceType = tmp_ResourceDetails.x_ResourceType
| lookup kind=leftouter (ResourceTypes | distinct x_ResourceType, ResourceType = SingularDisplayName) on x_ResourceType
| lookup kind=leftouter (Services | distinct x_ResourceType, ServiceName, ServiceCategory, x_ServiceModel) on x_ResourceType
//
Expand All @@ -1277,7 +1275,7 @@ CommitmentDiscountUsage_transform_v1_0()
x_CommitmentDiscountNormalizedRatio = InstanceFlexibilityRatio,
x_CommitmentDiscountQuantity = UsedHours * InstanceFlexibilityRatio,
x_IngestionTime = ingestion_time(),
x_ResourceGroupName = extract('/resourcegroups/([^/]+)/', 1, ResourceId),
x_ResourceGroupName,
x_ResourceType,
// x_RowId = hash_sha256(strcat(
// // DO NOT CHANGE COLUMNS OR COLUMN ORDER
Expand Down

0 comments on commit 4ad6bcc

Please sign in to comment.