Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Storm v4 #290

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[WIP] Storm v4 #290

wants to merge 6 commits into from

Conversation

asdine
Copy link
Owner

@asdine asdine commented May 11, 2021

After v3, I realized Storm was becoming hard to maintain. It was relying too much on reflection, felt bloated, and was performing poorly. There were a lot of ongoing discussions on the subject, like #211 or #222, that led me to realize what was the actual problem: Storm design was good enough for a simple toolkit but not for an actual database.
If fixing some of the performance issues can be done by tweaking a loop there and there, any other addition was a real pain and my motivation wasn't high enough to handle that level of plumbing.
Instead, I decided to write an actual database as a separate project, Genji, with the goal of, someday, building Storm on top of it.
However, Genji's API is still not yet stable and requires a little bit more work before being able to be relied upon. But as a first step, here is a draft of what it would look like.

Another aspect I wanted to fix was the Storm API. I believe the current API is error-prone and requires too much magic, as we can see in these issues. I think this is the opportunity to improve that design as well.

Storm now uses Genji. What does it mean?

Genji is a document-oriented, embedded SQL database written in Go.

  • It uses BoltDB as a storage engine by default: Nothing changes for Storm, it's still a toolkit for BoltDB. What changes though, is how that data is encoded, stored, and queried.

  • It supports other backends: This means that Storm can also be used to store data in-memory or in Badger. The Badger engine is still experimental though, but this is possible in the future.

  • It supports SQL: This means that if users need advanced queries, they can use SQL directly.

  • It's a document database: This means that we can now query nested fields as well

  • It supports schemas: This means that we can now apply constraints on some of the fields of the table/bucket

  • It is designed for performance: Even though Genji is still in an early phase and lacks optimizations, it was designed to scale and it's much easier to apply changes there.

What are the big changes in the API?

Take a look at the README file.

How can I contribute?

Inputs are welcome! Don't hesitate to comment on the PR or creating an issue.

Next steps

  • Iterate on this PR until having a working version
  • Merge in a dedicated branch
  • Release one or several betas
  • Release version 4

store.go Outdated Show resolved Hide resolved
store.go Outdated Show resolved Hide resolved
store.go Outdated Show resolved Hide resolved
store.go Outdated Show resolved Hide resolved
@asdine asdine marked this pull request as ready for review May 11, 2021 11:00
@asdine asdine changed the title Storm v4 [WIP] Storm v4 May 11, 2021
Repository owner deleted a comment from codecov-commenter May 14, 2021
@codecov-commenter
Copy link

codecov-commenter commented May 14, 2021

Codecov Report

❗ No coverage uploaded for pull request base (master@23213e9). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #290   +/-   ##
=========================================
  Coverage          ?   33.33%           
=========================================
  Files             ?        2           
  Lines             ?      132           
  Branches          ?        0           
=========================================
  Hits              ?       44           
  Misses            ?       75           
  Partials          ?       13           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23213e9...8678f8c. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants