Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reading parquet with PyArrow ignores rechunk argument #16416

Closed
2 tasks done
itamarst opened this issue May 22, 2024 · 1 comment · Fixed by #16418
Closed
2 tasks done

Reading parquet with PyArrow ignores rechunk argument #16416

itamarst opened this issue May 22, 2024 · 1 comment · Fixed by #16418
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@itamarst
Copy link
Contributor

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

Just call pl.read_parquet(..., rechunk=False). The rechunk=False will be ignored.

Log output

No response

Issue description

Wen reading parquet using Pyarrow, the rechunk argument isn't passed down through all the functions, so it's hardcoded as always True, increasing memory usage.

Expected behavior

Respect rechunk.

Installed versions

Git version of Polars as of May 22, 2024.
@itamarst itamarst added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels May 22, 2024
@itamarst
Copy link
Contributor Author

I will submit a PR with a fix soon.

@itamarst itamarst mentioned this issue May 22, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(python): make read_parquet() respect rechunk flag when using pyarrow itamarst/polars
1 participant
@itamarst