Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use pread #100

Open
mxmlnkn opened this issue Nov 22, 2022 · 2 comments
Open

Use pread #100

mxmlnkn opened this issue Nov 22, 2022 · 2 comments
Labels
performance Something is slower than it could be

Comments

@mxmlnkn
Copy link
Owner

mxmlnkn commented Nov 22, 2022 •
edited
Loading

Currently, StenciledFile and other locations use locks to avoid race conditions between seek and read calls. I'm not sure why I never though of pread before but it might improve performance for multi-threaded access and even if not, it should simplify the code by removing the lock.
@mxmlnkn mxmlnkn added the performance Something is slower than it could be label Nov 22, 2022
@mxmlnkn
Copy link
Owner Author

mxmlnkn commented Nov 23, 2022

One reason why os.pread is not always possible is the recursion. In those cases the reading has to be done on pure Python file-like objects for which pread cannot be used. But, it should still be possible to implement as a fast-path for real files.

mxmlnkn added a commit that referenced this issue Aug 30, 2024
@mxmlnkn
[wip][feature] fusepy: Add FUSE 3 support
bc72c49 
fusepy/fusepy #66, #67, #101
fusepy/fusepy #100

First test with ratarmount worked!

 - [ ] I only monkey-patched readdir and getattr. The other changed
   methods should also be adjusted and tested and maybe we can do
   better, e.g., by letting the caller decide which interface they want
   to implement with a member variable as flag! Or do it via inspection
   like in fusepy/fusepy#101, but the overhead
   might be killer.
@mxmlnkn
Copy link
Owner Author

mxmlnkn commented Sep 8, 2024

This becomes even more important on my Lustre benchmarks because it advertises itself as having 4 MiB block size, which causes the Python buffered I/O reader to read 4 MiB chunks even if only 1 KiB are necessary. Using os.pread (when available, i.e., not on Windows), would also skip the buffering. The buffering probably should also be turned off independent of this change, but then for it to work reasonably fast and not cause 8 KiB accesses to Lustre, ratarmount also needs to forward the block size from the opened file!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Something is slower than it could be
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mxmlnkn