Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Microbatch] Leverage static partitioning in dbt-bigquery materialization strategy #538

Open
Tracked by #10624
MichelleArk opened this issue Sep 26, 2024 · 1 comment
Assignees
Labels
feature:microbatch Issues related to the microbatch incremental strategy pkg:dbt-bigquery Issue affects dbt-bigquery

Comments

@MichelleArk
Copy link
Contributor

In dbt-labs/dbt-bigquery#1334, we introduced the microbatch strategy which leverages the dynamic partition insert/overwrite mechanism.

However, static partitioning would be significantly more performant. Benchmarking here:
https://discourse.getdbt.com/t/benchmarking-incremental-strategies-on-bigquery/981

Let's:

  • Leverage static partitioning by setting partitions prior to calling the insert_overwrite_sql as part of the microbatch strategy
  • Build partitions using GENERATE_TIMESTAMP_ARRAY (for 'hour' batch_size) or GENERATE_DATE_ARRAY (for all others) + __dbt_internal_microbatch_event_time_start and __dbt_internal_microbatch_event_time_end jinja context vars.
@MichelleArk MichelleArk self-assigned this Sep 26, 2024
@MichelleArk MichelleArk added the feature:microbatch Issues related to the microbatch incremental strategy label Dec 17, 2024
@mikealfare mikealfare added the pkg:dbt-bigquery Issue affects dbt-bigquery label Jan 14, 2025
@mikealfare mikealfare transferred this issue from dbt-labs/dbt-bigquery Jan 14, 2025
@QMalcolm
Copy link
Contributor

I think this issue is still relevant. @MichelleArk is that correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature:microbatch Issues related to the microbatch incremental strategy pkg:dbt-bigquery Issue affects dbt-bigquery
Projects
None yet
Development

No branches or pull requests

3 participants