Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

archive: Refactor archive_storage method into subscription #6483

Merged
merged 81 commits into from
Nov 29, 2024
Merged

archive: Refactor archive_storage method into subscription #6483

merged 81 commits into from
Nov 29, 2024

Commits on Oct 9, 2024

  1. archive/api: Add stroage diff method 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    e17a555 View commit details
    Browse the repository at this point in the history

  2. storage/events: Add storage events and test serialization 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    4101ee6 View commit details
    Browse the repository at this point in the history

  3. archive: Add initial implementation for storage diff 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    d781d67 View commit details
    Browse the repository at this point in the history

  4. archive: Handle modified and delete cases 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    60e1a93 View commit details
    Browse the repository at this point in the history

  5. archive/events: Derive more impl for diff items 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    63da031 View commit details
    Browse the repository at this point in the history

  6. archive: Deduplicate input keys 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    25b9e5a View commit details
    Browse the repository at this point in the history

  7. archive: Move code to different module 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    52cfaf7 View commit details
    Browse the repository at this point in the history

  8. archive: Fetch value / hash or both 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    ee6bed7 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. events: Add subscription events for ArchiveStorageDiffEvent 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    8948e74 View commit details
    Browse the repository at this point in the history

  2. archive: Make archive_storageDiff a subscription 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    fbfb0dc View commit details
    Browse the repository at this point in the history

  3. archive: Modify storageDiff to leverage subscription backpressure 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    f7b33ed View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. archive: Deduplicate items to a separate function 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    f47db36 View commit details
    Browse the repository at this point in the history

  2. archive/tests: Check deduplication 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    97546a4 View commit details
    Browse the repository at this point in the history

  3. archive: Fix deduplication shortest key 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    baa3dd0 View commit details
    Browse the repository at this point in the history

  4. archive/tests: Check complex deduplication 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4683f6b View commit details
    Browse the repository at this point in the history

  5. archive: Simplify logic of trie iteration 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    36f59d2 View commit details
    Browse the repository at this point in the history

  6. archive: Improve documentation 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a084f47 View commit details
    Browse the repository at this point in the history

  7. archive/events: Add derive(Eq) to archive storage diff events 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    77cac50 View commit details
    Browse the repository at this point in the history

  8. archive/tests: Check query for main trie under prefix 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    e2324a4 View commit details
    Browse the repository at this point in the history

  9. archive: Add trace and debug logs for storage diff 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4b05913 View commit details
    Browse the repository at this point in the history

  10. archive: Send StorageDiffDone for partial trie queries 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    83cb9b4 View commit details
    Browse the repository at this point in the history

  11. archive/tests: Check no changes between blocks 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    85c810e View commit details
    Browse the repository at this point in the history

  12. archive/tests: Ensure added key 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    152d13b View commit details
    Browse the repository at this point in the history

  13. archive/tests: Extend test with interleaved values and hashes 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4f549bf View commit details
    Browse the repository at this point in the history

  14. rpc-v2: Use indexmap crate 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    d208068 View commit details
    Browse the repository at this point in the history

  15. archive: Move prevHash as last parameter to archive diff 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    369a4d2 View commit details
    Browse the repository at this point in the history

  16. archive: Preserve order of elements with indexMap 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a02cc84 View commit details
    Browse the repository at this point in the history

  17. archive/tests: Check deleted keys 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    637b446 View commit details
    Browse the repository at this point in the history

  18. events: Add common wrappers for construction 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    76efb18 View commit details
    Browse the repository at this point in the history

  19. archive: Propagate errors via events 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ca1fa20 View commit details
    Browse the repository at this point in the history

  20. archive: Check invalid parameters and events 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    03eefd7 View commit details
    Browse the repository at this point in the history

  21. archive: Fix clippy 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    9a1e792 View commit details
    Browse the repository at this point in the history

  22. Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    dbbfce1 View commit details
    Browse the repository at this point in the history

  23. Add prdoc 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    971026d View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Merge branch 'master' into lexnv/storage-diff

    @lexnv
    lexnv authored Nov 6, 2024
    Configuration menu
    Copy the full SHA
    2880058 View commit details
    Browse the repository at this point in the history

  2. archive/storage: Optimize space used for saved keys 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    b774855 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/lexnv/storage-diff' into lexnv/s… 

    …torage-diff
    @lexnv
    lexnv committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    838504b View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

    @lexnv
    lexnv committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    60abd9b View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2024

  1. archive: Propagate all queries to handle_trie_queries 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    dc5f001 View commit details
    Browse the repository at this point in the history

  2. archive: Simplify process_events 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    cdc4dc5 View commit details
    Browse the repository at this point in the history

  3. archive: Add missing line 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    de7cf79 View commit details
    Browse the repository at this point in the history

  4. archive: Refactor with FetchedStorage enum 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    199b46d View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

    @lexnv
    lexnv committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    3f4a4b8 View commit details
    Browse the repository at this point in the history

  6. archive: Refactor for optimal iteration cycles 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    f4ace06 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. archive: Adjust code comments 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    55489c7 View commit details
    Browse the repository at this point in the history

  2. archive: Rename starts_with to belongs_to_query 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    4f94890 View commit details
    Browse the repository at this point in the history

  3. Update prdoc 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    a7e54a5 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' into lexnv/storage-diff 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    5341227 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. common/events: Transform StorageResult into StorageEvent 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    25728a1 View commit details
    Browse the repository at this point in the history

  2. archive/api: Make archive_storage a subscription 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8860e7e View commit details
    Browse the repository at this point in the history

  3. rpc-v2: Rename ArchiveResult into ArchiveEvent 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    533ecb9 View commit details
    Browse the repository at this point in the history

  4. events: Add wrappers for ArchiveStorageEvent 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    0849e8e View commit details
    Browse the repository at this point in the history

  5. common/storage: Introduce a common storage subcription handler 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a2f92b6 View commit details
    Browse the repository at this point in the history

  6. archive/api: Remove pagination items from archive storage 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7fec81a View commit details
    Browse the repository at this point in the history

  7. common/events: Add constructor methods for clarity 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    b6523d2 View commit details
    Browse the repository at this point in the history

  8. archive: Remove ArchiveStorage in favour of common archive object 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    bb65583 View commit details
    Browse the repository at this point in the history

  9. archive: Remove config for storage parameters and use backpressure 

    instead

Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    1917672 View commit details
    Browse the repository at this point in the history

  10. archive/storage: Adjust testing to subscription based approach 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    702f521 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2024

  1. Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

    @lexnv
    lexnv committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    751d8ff View commit details
    Browse the repository at this point in the history

  2. archive: Remove variable 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    3c2bb7c View commit details
    Browse the repository at this point in the history

  3. Update cargo lock 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    8f8c0c0 View commit details
    Browse the repository at this point in the history

  4. archive: Mode deduplicate_storage_diff_items to inenr methods 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    07a459c View commit details
    Browse the repository at this point in the history

  5. archive/tests: Ensure other storage entries are not provided 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a151bf3 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2024

  1. archive/diff: Implement lexicographic_diff 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    a575b4a View commit details
    Browse the repository at this point in the history

  2. archive/tests: Check lexicographic diff 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    caa75cc View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    7ff0bdb View commit details
    Browse the repository at this point in the history

  4. archive: Remove commented code and add documentation 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    c6e40a0 View commit details
    Browse the repository at this point in the history

  5. Update substrate/client/rpc-spec-v2/src/archive/api.rs 

    Co-authored-by: James Wilson <[email protected]>
    @lexnv @jsdw
    lexnv and jsdw authored Nov 27, 2024
    Configuration menu
    Copy the full SHA
    a07b5c4 View commit details
    Browse the repository at this point in the history

  6. archive: Remove unneeded lifetime 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    ae00327 View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'origin/lexnv/storage-diff' into lexnv/s… 

    …torage-diff
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    fd48434 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/lexnv/storage-diff' into lexnv/s… 

    …torage-sub

Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    736a5ad View commit details
    Browse the repository at this point in the history

  9. archive: Check sink closed 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    791831f View commit details
    Browse the repository at this point in the history

  10. archive: Refactor process methods 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    3ddaf2d View commit details
    Browse the repository at this point in the history

  11. archive/tests: Adjust to new assumptions wrt StorageDone 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    774b525 View commit details
    Browse the repository at this point in the history

  12. archive: Rename StorageResult to Storage 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    4147442 View commit details
    Browse the repository at this point in the history

  13. storage/events: Include child trie key 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    c0f5ded View commit details
    Browse the repository at this point in the history

  14. archive/tests: Adjust testing 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    39aab7a View commit details
    Browse the repository at this point in the history

  15. archive/tests: Check childtrie results 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    b14a4bd View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2024

  1. Merge remote-tracking branch 'origin/master' into lexnv/storage-sub 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    9676499 View commit details
    Browse the repository at this point in the history

  2. archive/tests: Fix wrong merge 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    ff09c7e View commit details
    Browse the repository at this point in the history

  3. archive/events: Align with spec 

    Signed-off-by: Alexandru Vasile <[email protected]>
    @lexnv
    lexnv committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    087f0e7 View commit details
    Browse the repository at this point in the history