Crates.io reference SDK Crate

[pinkforest]

Why?

Does crates.io care about:

• Whether it gets 5 Million Request / Responses when it could get 1000 instead with same amount of data for the same pattern / volume scenario from 1000 users?
• Documenting about what crates.io guarantees and what it keeps as implementation detail?
• Validating the integrity of the provided data to consumer?
• Easily alllow consuming the data in the most optimal way?
• .. whilst performance testing various access / volume scenarios that can be continuously reported from CI as comparison between commits?
• How much data is transferred or other cloud billable resources consumed under different consumption scenarios?
• ... whilst ensuring the cache is optimally matched instead of missed between various scenarios?

How?

Create and maintain a consumer SDK crate that has an abstract interface which doesn't change where as the logistics in implementation detail will follow what ever evolves at crates.io.

The consumer crate would optimise between different access pattern / volume scenarios and feature gates based on what is required.

Example - Renovate
#4503

This needs to be properly quantified and calculated but I am providing perhaps an extreme example here as I don't have data for it -

There is a difference if we have 1000 people doing the same thing to crates.io where it can be either 5 Million Request / Responses pairs caused by one consumer software that could have been just 1000 requests to get 180MB archive that unpacks as 500MB for all those 1000 people separately when they were provided nice interface to consume to help choosing between those tradeoffs.

Would it be worse to have 5 Million Request / Responses - or - 100 that the clients could potentially cache continuously and poll updates on top of?

The fear of optimising things comes down to implementing things in optimised way vs plain blind 1:1 data resource mapping that can generate a lot of request/response pairs compared potential tradeoff with say 138MB metadata file and a local caching for it.

Example - Integrity Validation
Currently e.g. the index crate does not validate the data it spits out to it's consumers and responsibility / guarantees around it is hazy. This can be both used to test the integrity at the registry as well as at the consumer end.