Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: support spilling large materialized CTEs to temporary tables #16858

Open
Tracked by #16867
maxjustus opened this issue Nov 16, 2024 · 0 comments
Open
Tracked by #16867
Labels
C-feature Category: feature

Comments

@maxjustus
Copy link
Contributor

Currently materialized CTEs reside fully in memory with no disk spilling capability. This means that if the CTE you’re materializing is larger than your configured query memory limit you will get a memory limit exceeded error.

The only current option to work around this while retaining the performance benefits of materialized CTEs is to pull the CTE into a create transient table … which is created before you run the query.

It would be ideal to have a setting that specifies a memory ratio threshold above which materialized CTEs are converted into temporary tables.

It could be called something like materialized_cte_spilling_memory_ratio to be consistent with the existing aggregate_spilling_memory_ratio and join_spilling_memory_ratio settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

1 participant