Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GOT-31] Explore if we can make an iterator-based read-query execution pipeline #316

Open
jsign opened this issue Sep 30, 2022 · 0 comments
Labels
linear Sync issue with linear research

Comments

@jsign
Copy link
Contributor

jsign commented Sep 30, 2022

This is capturing an idea that I had a while ago. Currently, we store all the read-query results in memory and do the right formating after.

That might be comfortable for formatting but also means that we hold all the read-query results in memory which is very aggressive to the validator. Users could send queries that have very big results, and that will directly impact memory usage.

We should explore if we can leverage the iteration we do with rows.Next() in the lowest layer and pass that upstream to do the right formatting to have a bounded memory usage.

cc @brunocalza @sanderpick @asutula

GOT-31

@jsign jsign added the research label Sep 30, 2022
@jsign jsign moved this to 🆕 New in go-tableland backlog Sep 30, 2022
@brunocalza brunocalza added the linear Sync issue with linear label Mar 23, 2023
@brunocalza brunocalza changed the title Explore if we can make an iterator-based read-query execution pipeline [GOT-31] Explore if we can make an iterator-based read-query execution pipeline Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
linear Sync issue with linear research
Projects
No open projects
Status: 🆕 New
Development

No branches or pull requests

2 participants