Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: encoder / decoder of points, elements, should use bufio #423

Open
gbotrel opened this issue Jul 1, 2023 · 0 comments
Open

perf: encoder / decoder of points, elements, should use bufio #423

gbotrel opened this issue Jul 1, 2023 · 0 comments

Comments

@gbotrel
Copy link
Collaborator

gbotrel commented Jul 1, 2023

Reading 32bytes per 32bytes a 6GB list of points from a os.File is terrible for perf.

Without changing the code, just wrapping the os.File io.Reader in to a bufio.Reader (for example bufio.NewReaderSize(f, fr.Bytes*(1<<10))) improves performance by a 10x factor.

Similar logic for io.Writer and bufio.Writer .

However our APIs work with Writer and Reader, and doing a bufio.Reader inside the functions may result in the function reading too many bytes. (calls to read functions are sometime encapsulated or following one another with the same Reader) .

Maybe we could add a high level ReadFromFile to avoid putting the burden of wrapping the reader on the caller;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant