rpc-v2: Implement `archive_unstable_storageDiff` #5997

lexnv · 2024-10-09T14:02:06Z

This PR implements the archive_unstable_storageDiff.

The implementation follows the rpc-v2 spec from:

archive: Introduce archive_storageDiff for indexers json-rpc-interface-spec#159.
builds on top of archive: Use subscriptions for storageDiff json-rpc-interface-spec#161

cc @paritytech/subxt-team

Signed-off-by: Alexandru Vasile <[email protected]>

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs

niklasad1

LGTM, good stuff

substrate/client/rpc-spec-v2/src/archive/api.rs

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs

jsdw · 2024-11-21T12:25:09Z

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs

+				(None, None) => break,
+			};
+
+			let Some(fetch_type) = Self::belongs_to_query(&key, &items) else {


Aah ok I didn't notice until now: so the code iterates over all keys in storage, and then for each key it gets the added/removed/same diff compared to keys of other block and then removes any key that isn't actually something we care about, iiuc!

I was wondering about filtering the iters to contain only items which match belong_to_query before doing the diff, but then you'd have to do it twice, so it's a tradeoff! Prob how you've done it is better in case the query passed from the user is quite big!

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs

substrate/client/rpc-spec-v2/src/archive/tests.rs

jsdw

A few comments but overall looks great!

Signed-off-by: Alexandru Vasile <[email protected]>

Co-authored-by: James Wilson <[email protected]>

paritytech-workflow-stopper · 2024-11-27T15:38:16Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12053348230
Failed job name: cargo-clippy

Signed-off-by: Alexandru Vasile <[email protected]>

…torage-diff

This PR adapts the `archive_storage` implementation from a method to a subscription. This keeps the archive APIs uniform and consistent. Builds on: #5997 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]>

This PR implements the `archive_unstable_storageDiff`. The implementation follows the rpc-v2 spec from: - paritytech/json-rpc-interface-spec#159. - builds on top of paritytech/json-rpc-interface-spec#161 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]>

This PR adapts the `archive_storage` implementation from a method to a subscription. This keeps the archive APIs uniform and consistent. Builds on: #5997 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]>

lexnv added 8 commits October 9, 2024 10:54

archive/api: Add stroage diff method

e17a555

Signed-off-by: Alexandru Vasile <[email protected]>

storage/events: Add storage events and test serialization

4101ee6

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Add initial implementation for storage diff

d781d67

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Handle modified and delete cases

60e1a93

Signed-off-by: Alexandru Vasile <[email protected]>

archive/events: Derive more impl for diff items

63da031

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Deduplicate input keys

25b9e5a

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Move code to different module

52cfaf7

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Fetch value / hash or both

ee6bed7

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv added I5-enhancement An additional feature request. T3-RPC_API This PR/Issue is related to RPC APIs. D1-medium Can be fixed by a coder with good Rust knowledge but little knowledge of the codebase. labels Oct 9, 2024

lexnv self-assigned this Oct 9, 2024

lexnv added 18 commits November 4, 2024 18:40

events: Add subscription events for ArchiveStorageDiffEvent

8948e74

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Make archive_storageDiff a subscription

fbfb0dc

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Modify storageDiff to leverage subscription backpressure

f7b33ed

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Deduplicate items to a separate function

f47db36

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Check deduplication

97546a4

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Fix deduplication shortest key

baa3dd0

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Check complex deduplication

4683f6b

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Simplify logic of trie iteration

36f59d2

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Improve documentation

a084f47

Signed-off-by: Alexandru Vasile <[email protected]>

archive/events: Add derive(Eq) to archive storage diff events

77cac50

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Check query for main trie under prefix

e2324a4

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Add trace and debug logs for storage diff

4b05913

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Send StorageDiffDone for partial trie queries

83cb9b4

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Check no changes between blocks

85c810e

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Ensure added key

152d13b

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Extend test with interleaved values and hashes

4f549bf

Signed-off-by: Alexandru Vasile <[email protected]>

rpc-v2: Use indexmap crate

d208068

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Move prevHash as last parameter to archive diff

369a4d2

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv mentioned this pull request Nov 14, 2024

archive: Refactor archive_storage method into subscription #6483

Merged

niklasad1 reviewed Nov 14, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs Outdated Show resolved Hide resolved

niklasad1 approved these changes Nov 14, 2024

View reviewed changes

jsdw reviewed Nov 21, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/archive/api.rs Outdated Show resolved Hide resolved

jsdw reviewed Nov 21, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs Outdated Show resolved Hide resolved

jsdw reviewed Nov 21, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/archive/archive_storage.rs Outdated Show resolved Hide resolved

jsdw reviewed Nov 21, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/archive/tests.rs Show resolved Hide resolved

jsdw approved these changes Nov 21, 2024

View reviewed changes

lexnv and others added 10 commits November 26, 2024 13:23

Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

751d8ff

archive: Remove variable

3c2bb7c

Signed-off-by: Alexandru Vasile <[email protected]>

Update cargo lock

8f8c0c0

Signed-off-by: Alexandru Vasile <[email protected]>

archive: Mode deduplicate_storage_diff_items to inenr methods

07a459c

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Ensure other storage entries are not provided

a151bf3

Signed-off-by: Alexandru Vasile <[email protected]>

archive/diff: Implement lexicographic_diff

a575b4a

Signed-off-by: Alexandru Vasile <[email protected]>

archive/tests: Check lexicographic diff

caa75cc

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/master' into lexnv/storage-diff

7ff0bdb

archive: Remove commented code and add documentation

c6e40a0

Signed-off-by: Alexandru Vasile <[email protected]>

Update substrate/client/rpc-spec-v2/src/archive/api.rs

a07b5c4

Co-authored-by: James Wilson <[email protected]>

lexnv added 2 commits November 27, 2024 17:55

archive: Remove unneeded lifetime

ae00327

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/lexnv/storage-diff' into lexnv/s…

fd48434

…torage-diff

lexnv added this pull request to the merge queue Nov 27, 2024

Merged via the queue into master with commit afd065f Nov 27, 2024
194 of 198 checks passed

lexnv deleted the lexnv/storage-diff branch November 27, 2024 19:04

lexnv mentioned this pull request Nov 29, 2024

backport/rpc-v2: Implement archive_unstable_storageDiff (stable2412) #6709

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc-v2: Implement `archive_unstable_storageDiff` #5997

rpc-v2: Implement `archive_unstable_storageDiff` #5997

lexnv commented Oct 9, 2024 •

edited

Loading

niklasad1 left a comment •

edited

Loading

jsdw Nov 21, 2024 •

edited

Loading

jsdw left a comment

paritytech-workflow-stopper bot commented Nov 27, 2024

rpc-v2: Implement archive_unstable_storageDiff #5997

rpc-v2: Implement archive_unstable_storageDiff #5997

Conversation

lexnv commented Oct 9, 2024 • edited Loading

niklasad1 left a comment • edited Loading

Choose a reason for hiding this comment

jsdw Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

jsdw left a comment

Choose a reason for hiding this comment

paritytech-workflow-stopper bot commented Nov 27, 2024

rpc-v2: Implement `archive_unstable_storageDiff` #5997

rpc-v2: Implement `archive_unstable_storageDiff` #5997

lexnv commented Oct 9, 2024 •

edited

Loading

niklasad1 left a comment •

edited

Loading

jsdw Nov 21, 2024 •

edited

Loading