-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why is the first read slower? #277
Comments
Could it be the case from #218? |
Hi @tentacles-from-outer-space, thanks for your question! Modern storage devices have increasingly large RAM caches that can be used to service multiple identical requests. That is why a correct benchmark should always read a unique file from disk on every iteration to avoid reading from SSD cache instead of the actual stored file. Also, files can be cached in the OS and some storage devices have an initial startup time where they have to get out of a lower power mode and that can take some time too. For your benchmark, you might consider first writing a large number (e.g. 100) of unique fst files to get a benchmark for writing and then reading those unique files to get a correct benchmark of the reading performance. With such a setup it's safe to assume that there are no caching effects :-) |
I notice that in fresh session first use of
read_fst
is slower than next uses. The same file. The same number of cores. Just running the same command again.RStudio on server (80 cores), ~4GB file.
The first use is like twice slower.
Created on 2023-05-24 with reprex v2.0.2
Is it some kind of caching?
I run into this when I try to compare different setting of number of threads.
The text was updated successfully, but these errors were encountered: