Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gc-compaction: better job split heuristics #10213

Open
Tracked by #9114
skyzh opened this issue Dec 19, 2024 · 0 comments
Open
Tracked by #9114

gc-compaction: better job split heuristics #10213

skyzh opened this issue Dec 19, 2024 · 0 comments
Assignees
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug

Comments

@skyzh
Copy link
Member

skyzh commented Dec 19, 2024

The final stats for downloading is:

INFO compaction_loop{tenant_id=4d77bcd6107af3559d33545edb82b215 shard_id=0308}:run:scheduled_compact_timeline{timeline_id=d245f43c62cd81fc104e168b6786c110}: finish downloading layers, downloaded=14114111488, total=28083863552, ratio=0.50

While the estimation is:

INFO compaction_loop{tenant_id=4d77bcd6107af3559d33545edb82b215 shard_id=0308}:run: splitting compaction job: 000000000000000000000000000000000000..000000067F00004005000080000000D78000, estimated_size=4452556800

4GB vs 14GB

This caused each job is running super slow and spent a lot of time downloading files.

@skyzh skyzh added c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug labels Dec 19, 2024
@skyzh skyzh self-assigned this Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug
Projects
None yet
Development

No branches or pull requests

1 participant