-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Out of memory when loading a large JSON file. #18
Comments
You are correct. Let me think about that ... |
I think you could switch to a streaming parser for JSON, maybe something like this, if you don't mind the dependency: |
Yeah I tend to agree about the JSON parser Consider this one: https://golangrepo.com/repo/buger-jsonparser-go-json there are others also that achieve high throughput but support code gen as well they also compile to wasm which may be useful overall |
I think what would also work here is newline delimited JSON data like BigQuery is using ... |
I have a 941MB json file I would like to import into eliasdb, but I get an out of error crash. this is because the importer tries to load the data wholesale in stead of incrementally. A way to do incremental loading from a single json file would be great.
The text was updated successfully, but these errors were encountered: