Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After v3, I realized Storm was becoming hard to maintain. It was relying too much on reflection, felt bloated, and was performing poorly. There were a lot of ongoing discussions on the subject, like #211 or #222, that led me to realize what was the actual problem: Storm design was good enough for a simple toolkit but not for an actual database.
If fixing some of the performance issues can be done by tweaking a loop there and there, any other addition was a real pain and my motivation wasn't high enough to handle that level of plumbing.
Instead, I decided to write an actual database as a separate project, Genji, with the goal of, someday, building Storm on top of it.
However, Genji's API is still not yet stable and requires a little bit more work before being able to be relied upon. But as a first step, here is a draft of what it would look like.
Another aspect I wanted to fix was the Storm API. I believe the current API is error-prone and requires too much magic, as we can see in these issues. I think this is the opportunity to improve that design as well.
Storm now uses Genji. What does it mean?
Genji is a document-oriented, embedded SQL database written in Go.
It uses BoltDB as a storage engine by default: Nothing changes for Storm, it's still a toolkit for BoltDB. What changes though, is how that data is encoded, stored, and queried.
It supports other backends: This means that Storm can also be used to store data in-memory or in Badger. The Badger engine is still experimental though, but this is possible in the future.
It supports SQL: This means that if users need advanced queries, they can use SQL directly.
It's a document database: This means that we can now query nested fields as well
It supports schemas: This means that we can now apply constraints on some of the fields of the table/bucket
It is designed for performance: Even though Genji is still in an early phase and lacks optimizations, it was designed to scale and it's much easier to apply changes there.
What are the big changes in the API?
Take a look at the README file.
How can I contribute?
Inputs are welcome! Don't hesitate to comment on the PR or creating an issue.
Next steps