Discussion of Scale-out Architecture for High-volume Repositories TAP (TAP 21)

[ergonlogic]

TAP-21 originated with theupdateframework/specification/issues/309. Following discussion at the TUF community meeting on 2024-11-01, we've drafted TAP-21 and submitted a PR (#189).

In addition to broader discussions of TAP-21, we are specifically seeking feedback to:

• Validate our motivation and rationale
• Validate the calculations we undertook (See: TAP-21 metadata overhead calculator)
• Better understand how to approach the following sections (which are sparse, atm):
  • Specification
  • Backwards Compatibility
  • Augmented Reference Implementation

We have thoughts on the sparse sections, but have not yet been able to articulate them clearly enough to include in the TAP.

[jku]

You mention TOFU but it's a little unclear how the client reacts when top level repository changes the "initial sub-repo metadata".

Assume client has cached earlier sub-repo metadata and the new sub-repo metadata is not compatible with the sub-repo metadata in client cache (as it can't be guaranteed to be). How does client react?

[ergonlogic]

You mention TOFU but it's a little unclear how the client reacts when top level repository changes the "initial sub-repo metadata".
Assume client has cached earlier sub-repo metadata and the new sub-repo metadata is not compatible with the sub-repo metadata in client cache (as it can't be guaranteed to be). How does client react?

By "initial sub-repo metadata", we mean only 1.root.json for each sub-repo. For each relevant subrepo, the client must then follow the usual TUF procedure of trying to download and validate 2.root.json for the sub-repo, etc.

Under normal circumstances, key rotation for a sub-repo would involve deploying a new n.root.json to the sub-repo. If any non-root keys were rotated, we'd also need to re-sign the corresponding metadata, etc. No changes to the top-level metadata would occur in this process.

So, the scenario you describe should not occur under normal operations. If it were to occur, presumably the top-level repo has to be considered canonical, since it is the root of trust (ie. the top-level repo's 1.root.json ships with the client).

The client behaviour should presumably be to:

1. Update top-level TUF repo metadata
2. For each sub-repo, verify that its cached sub-repo initial root metadata remains valid
   1. If valid, proceed as normal (update sub-repo metadata, etc.)
   2. If not valid, remove all of the sub-repo's TUF metadata, including initial root metadata, download the new initial root metadata for the sub-repo, validate it (against top-level TUF metadata), then proceed with normal sub-repo operations.

[jku]

For each sub-repo, verify that its cached sub-repo initial root metadata remains valid

This is currently not part of the spec but I believe it would be a good addition: verify-from-bootstrap feature in python-tuf

So, the scenario you describe should not occur under normal operations.

If this scenario is not expected to happen... then I'm thinking I maybe have not understood who the signers/keyowners of the sub-repo are. Can you talk more about that?

The reason I'm thinking I don't quite understand the setup is this:

• if the signers are project maintainers (or project-specific release automation), then signing keys will be lost and some maintainers will turn out to be malicious: In a high-volume repository the top-level will need to intervene somewhat regularly
• if the signers are repository automation (in other words the keys are KMS keys controlled by repository) I can see how content in top-level repo does not need to change ... but then I don't see the why sub-repos exist at all -- a single signing key (delivered as an artifact in top-level repo) would work just as well.

[ergonlogic]

• if the signers are project maintainers (or project-specific release automation), then signing keys will be lost and some maintainers will turn out to be malicious: In a high-volume repository the top-level will need to intervene somewhat regularly

I believe the client behaviour described above ought to handle this scenario reasonably well. But please correct me if you see a flaw in that process. We're working on incorporating it into the Specification section of the TAP.

• if the signers are repository automation (in other words the keys are KMS keys controlled by repository) I can see how content in top-level repo does not need to change ... but then I don't see the why sub-repos exist at all -- a single signing key (delivered as an artifact in top-level repo) would work just as well.

First off, we are primarily targeting this scenario ("signers are repository automation"). I'll update the TAP to reflect this.

That said, we considered re-using the same root metadata across all the sub-repos. However, when an online key (targets, snapshot or timestamp) is rotated, all of the metadata signed by that key needs to be re-signed. For a high-volume repository this can take a (relatively) long time. Separate root metadata per sub-repo allows for an incremental roll-out of new online keys.

Note that this still allows implementers to re-use keys across sub-repos. In fact, I think this would be the recommended approach.