New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOM when working with 1 GB PostgreSQL heap (table) file #409
Comments
Hello, at the moment fq has not been optimized to use less memory, focus has been to make it work at all :) The main reason it uses lots of memory at the moment is that it does very little "lazy" decoding, instead it will decode the full file and each field added will have to keep track of lots of metadata, it's name, parent, children, bit range, decode value, optional symbolic value, optional description string etc. I'm not familiar with the pghep format but for example the mp4 decoder in fq has a option to skip decoding of individual samples if that not interesting (you can still decode individual samples manually) which speeds up decoding a lot. But there are some options to improve the situation that i've thought about, most of them sadly quite complicated and not a easy task:
Let me know if you have other ideas |
What version are you using (
fq -v
)?How was fq installed?
My branch:
https://github.com/pnsafonov/fq/tree/postgres
Can you reproduce the problem using the latest release or master branch?
I can reproduce problem on my branch, where PostgreSQL parsers are implemented.
1 GB file:
https://github.com/pnsafonov/fq_testdata_postgres14
https://github.com/pnsafonov/fq_testdata_postgres14/raw/master/16397
What did you do?
I am trying to parse a heap (relation, table) file of maximum size (1 GB for PostgeSQL).
fq consumes 90-100 times memory than file size. For 80 mb file fq requires 7,5 GB RAM.
Kenel messages:
1 GB File:
Memory profiler results:
I check with disassembler.
TryFieldScalarFn.func1
is unnamed lambda inTryFieldScalarFn
.The text was updated successfully, but these errors were encountered: