You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
That might be comfortable for formatting but also means that we hold all the read-query results in memory which is very aggressive to the validator. Users could send queries that have very big results, and that will directly impact memory usage.
We should explore if we can leverage the iteration we do with rows.Next() in the lowest layer and pass that upstream to do the right formatting to have a bounded memory usage.
brunocalza
changed the title
Explore if we can make an iterator-based read-query execution pipeline
[GOT-31] Explore if we can make an iterator-based read-query execution pipeline
Mar 23, 2023
This is capturing an idea that I had a while ago. Currently, we store all the read-query results in memory and do the right formating after.
That might be comfortable for formatting but also means that we hold all the read-query results in memory which is very aggressive to the validator. Users could send queries that have very big results, and that will directly impact memory usage.
We should explore if we can leverage the iteration we do with
rows.Next()
in the lowest layer and pass that upstream to do the right formatting to have a bounded memory usage.cc @brunocalza @sanderpick @asutula
GOT-31
The text was updated successfully, but these errors were encountered: