-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Binary format for data files: FASLs! #5
Comments
Yes, serializing to It's also not so clear where to store the fasl. I suppose in the usual ~/.cache/common-lisp folder? Is there a function to expand the cache path? ASDF can do this I think, need to find the right API point. So to implement this, why we need to do is simply to extend the
I didn't understand this. Can you give an example? |
Cool!
I mean that, when we |
Oh, I see. Same here, I'd like to know... |
Here's a progress and a complete-ish prototype (in comments) for loading data from FASLs: https://www.reddit.com/r/Common_Lisp/comments/12dxdic/dumping_objects_into_compiled_files/ |
I'm not sure I get the macro trick... What's special about it beside binding to a global variable? |
So the logic is:
|
OK, got the macro trick now, it's super smart! |
All that said, does this really belong to Nfiles? I believe Nfiles should leave the user with the option to choose their prefered serialization format. If we don't want to create a dedicated library just for this, I suggest to create a dedicated package at least, to decouple regular file management from specialized serialization. |
@aartaka Wanna work on it? |
We would also need some benchmarks to compare cl-prevalence with this approach. |
Yes, but not necessarily soon enough :) |
Our data files take time to load. And, even though we often load them asynchronously or optimize their reading in other ways, there's only so much we can speed up without changing the format. And, while SQLite or some other opaque format may be nice and performant, impelementation-native FASLs may be even faster and need no foreign interfaces. The only challenge being: how do we put data into FASL and read it back, portably?
Solution
One way I'm thinking about is
(defvar *data* ...s-expressions)
into some file,compile-file
on this newly written file,*data*
variable set to the right value. Which is probably faster and more overflow-resistant thanread
-ing the whole file.Alternative solutions
I'm pretty sure there are other ways, like using compiled function bodies or
eval
-ing the data fetched from FASL, and I'm conscious that my approach is not the most performant, but it skips at least some some roadbumps.The perfect approach would be to compile the Lisp data itself (and not its
cl-prevalence
-serialized representation) into FASLs.Additional context
I've done some of this compilation magic in Sade, and here is the relevant bit:
Those, however, are concerned with making executable files, and not FASLs, but they still set the tone for how we might approach the problem.
The text was updated successfully, but these errors were encountered: