Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ADR for api cleanup and async commit #790

Merged
merged 8 commits into from
Aug 16, 2023
Merged

Conversation

cool-develope
Copy link
Collaborator

No description provided.

@cool-develope cool-develope requested a review from a team as a code owner June 12, 2023 21:00
The modified API will have the following structure:

```go
Tree interface {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo having read-only views as an interface makes sense

Copy link
Collaborator Author

@cool-develope cool-develope Jun 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just thinking iavl will be used as a SC layer and it may mislead if have a read-only interface.
Also, we have only one interface for iavl tree in store module.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read only would be good as we still need the iavl tree for commitments, store v2 adr talks close to nothing about this. I wouldn't use the current adr as final its very very high level with a lack of experimentation and research

@ValarDragon
Copy link
Contributor

I'd generally still suggest starting with separating out version management (#737 )

@cool-develope
Copy link
Collaborator Author

I'd generally still suggest starting with separating out version management (#737 )

@tac0turtle @yihuang , how about separating out version manager?
After Saveversion with batch, we will have only version-related functions as a modifier. IMO, separated interfaces looks meaningless in the store module.

@yihuang
Copy link
Collaborator

yihuang commented Jun 13, 2023

I'd generally still suggest starting with separating out version management (#737 )

@tac0turtle @yihuang , how about separating out version manager?

After Saveversion with batch, we will have only version-related functions as a modifier. IMO, separated interfaces looks meaningless in the store module.

I'd suggest figure out the minimal changes needed to support store v2 first (async commit, atomic commit), I don't see a big problem in current API design.
The MutableTree is a singleton, so I think it's not too bad for it to also do the version manager job, I'm ok either way :)

@tac0turtle
Copy link
Member

lets separate concerns, first what is needed for store v2 then a new api. I think conflating the two could be clouding what is all needed

@cool-develope
Copy link
Collaborator Author

Do we really require the Immutable Tree in Store V2, considering that iavl will solely be utilized for SC?
cc: @alexanderbez @tac0turtle

VersionExists(version int64) bool
DeleteVersionsTo(version int64) error
GetVersioned(key []byte, version int64) ([]byte, *cmtprotocrypto.ProofOps, error)
GetTree(version int64) (*iavl.Tree, error)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be unnecessary if we move all query functionalities to SS...

Copy link
Contributor

@alexanderbez alexanderbez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @cool-develope! I think this is a good jumping off point.

I can imagine, however, some of the finer points needing amendment as Store v2 work kicks off.

@tac0turtle
Copy link
Member

im still a bit lost with this adr, API cleanup has nothing to do with store v2. The commitment structure should be agnostic to what is going on at a higher level. I think we should remove any notion of store v2 and the plans there and work on creating a stable and sane API for this code base

@cool-develope cool-develope changed the title feat: draft ADR for api cleanup and store v2 feat: draft ADR for api cleanup and async commit Jul 18, 2023
@cool-develope
Copy link
Collaborator Author

im still a bit lost with this adr, API cleanup has nothing to do with store v2. The commitment structure should be agnostic to what is going on at a higher level. I think we should remove any notion of store v2 and the plans there and work on creating a stable and sane API for this code base

ok, I modified the ADR, and assumed ImmutableTree is still necessary.

@tac0turtle tac0turtle changed the title feat: draft ADR for api cleanup and async commit feat: ADR for api cleanup and async commit Jul 19, 2023
- The boundary between `ImmutableTree` and `MutableTree` is not clear.
- There are many public methods in `nodeDB`, making it less structured.

## Decision
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the external api to users? the below seems to focus on internal changes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think iavl needs the external API interface to clarify? I plan to keep the majority external API

@cool-develope
Copy link
Collaborator Author

just removed the parallel WorkingHash part, I think it is unnecessary since we can parallelize on a module basis.
Regarding the naming convention (Reader/Writer), we can organize it on the consumer side, I'd like to keep the current structure.

Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this will be iavl 2.0 we should remove fast node system.

### Async Commit

* Finalize the current version in memory during `Commit`.
* Perform async writes in the background.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the side effects to this?


```go
MutableTree interface {
SaveChangeSet(cs *ChangeSet) ([]byte, error) // this is for batch set
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is changeset here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to batch, just wanted to re-use the ChangeSet which is already defined in the iavl


### Negative

* Async Commit may result in increased memory usage and increased code complexity.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would add the added complexity in the code base. the asyncommit adds a layer of complexity

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you want to introduce some pseudo-code within this adr?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can be added to a con, no need for pseudo-code

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, a little confusing
do you want to add a more detailed description in increased code complexity?

Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this will be iavl 2.0 we should remove fast node system.

After the storage call last week its unclear how much time should be spent on this repo seeing that memiavl is likely the future in the sdk. Do you have any thoughts for this?

@cool-develope
Copy link
Collaborator Author

it is not iavl 2.0, just simple API cleanup
@kocubinski is currently working on iavl 2.0

Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaving an approval to move forward,

@tac0turtle tac0turtle merged commit b93a694 into master Aug 16, 2023
7 checks passed
@tac0turtle tac0turtle deleted the adr/cleanup_api branch August 16, 2023 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants