Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CommitmentDiscountUsage transform to use parse_resourceid #1219

Merged
merged 2 commits into from
Dec 27, 2024
Merged

Update CommitmentDiscountUsage transform to use parse_resourceid #1219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a270787
Update CommitmentDiscountUsage_transform_v1_0() to use parse_resource…
flanakin Dec 15, 2024
96c123d
Merge branch 'dev' into flanakin/dev/hubs-cduparse
flanakin Dec 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/_resources/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,14 @@ Legend:
> ✏️ Changed:
>
> 1. Update required permissions on hubs page [Required permissions](../docs/_reporting/hubs/README.md).
> 1. ETL Pipeline - Fixed timezones for ADF triggers to resolve issue where triggers would not start due to unrecognized timezone.
> 1. Changed the **enablePublicAccess** parameter to exclude network components.
> - When disabled, a VNet will be created along with the required private endpoints and DNS zones to function in a fully private manner.
> 1. Updated `CommitmentDiscountUsage_transform_v1_0()` to use `parse_resourceid()`.
>
> 🛠️ Fixed:
>
> 1. Improved performance and memory consumption in the `parse_resourceid()` function to address out of memory errors during cost data ingestion ([#1188](https://github.com/microsoft/finops-toolkit/issues/1188))
> 1. Fixed timezones for Data Factory triggers to resolve issue where triggers would not start due to unrecognized timezone.
> 1. Fixed an issue where `x_ResourceType` is using the wrong value.
> - This fix resolves the issue for all newly ingested data.
> - To fix historical data, reingest data using the `ingestion_ExecuteETL` Data Factory pipeline.
Expand Down
12 changes: 5 additions & 7 deletions src/templates/finops-hub/modules/scripts/IngestionSetup.kql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1246,12 +1246,10 @@ CommitmentDiscountUsage_transform_v1_0()
//
// Handle resource columns
| extend ResourceId = tolower(InstanceId)
// TODO: Switch to use parse_resourceid()
| parse ResourceId with "/subscriptions/" SubAccountId "/resourcegroups/" x_ResourceGroupName "/providers/" tmp_ResourceProvider "/" tmp_ResourceProviderPath
| extend segments = split(tmp_ResourceProviderPath, '/')
| mv-apply element = segments on (extend index = row_number() | where index % 2 == 0 | summarize ResourceName = strcat_array(make_list(element), '/'))
| mv-apply element = segments on (extend index = row_number() | where index % 2 == 1 | summarize x_ResourceType = strcat_array(make_list(element), '/'))
| extend x_ResourceType = strcat(tmp_ResourceProvider, '/', x_ResourceType)
| extend tmp_ResourceDetails = parse_resourceid(ResourceId)
| extend SubAccountId = tmp_ResourceDetails.SubAccountId
| extend x_ResourceGroupName = tmp_ResourceDetails.x_ResourceGroupName
| extend x_ResourceType = tmp_ResourceDetails.x_ResourceType
| lookup kind=leftouter (ResourceTypes | distinct x_ResourceType, ResourceType = SingularDisplayName) on x_ResourceType
| lookup kind=leftouter (Services | distinct x_ResourceType, ServiceName, ServiceCategory, x_ServiceModel) on x_ResourceType
//
Expand All @@ -1277,7 +1275,7 @@ CommitmentDiscountUsage_transform_v1_0()
x_CommitmentDiscountNormalizedRatio = InstanceFlexibilityRatio,
x_CommitmentDiscountQuantity = UsedHours * InstanceFlexibilityRatio,
x_IngestionTime = ingestion_time(),
x_ResourceGroupName = extract('/resourcegroups/([^/]+)/', 1, ResourceId),
x_ResourceGroupName,
x_ResourceType,
// x_RowId = hash_sha256(strcat(
// // DO NOT CHANGE COLUMNS OR COLUMN ORDER
Expand Down