Trusted Publishing: revoke manual API tokens once a publisher is used

[woodruffw]

This is related to #17241, and comes thinking about how we can/could have reduced the attacker's ability to pivot following the initial Ultralytics repository compromise.

The basic idea: we could revoke a subset of manually configured API tokens when a corresponding Trusted Publisher is configured and used.

Doing this in a general way will be tricky, since there are deployment/publishing setups where both an API token and a Trusted Publisher are used: for example, a project that builds some wheels on PyPI and others on their own build farms still needs both.

As such, here are some conditions (conjunctive) under which automatically revoking API tokens might make sense:

1. The project has a Trusted Publisher configured, and
2. The project has used the Trusted Publisher in the last N months (3? 6? 12?), and
3. The project has an API token configured, and
4. The API token has not been used in the last N months (3? 6? 12?)

If all (and other?) conditions are met, we could revoke the qualifying tokens and send the project's owners notification emails letting them know.

CC @di @sethmlarson @ewdurbin for thoughts, since this is me spitballing 🙂

[sethmlarson]

I think this is a great idea! We might be able to be more aggressive than 3 months. I suspect situations where an API key will continue to be used will happen on the order of minutes to hours (ie: we build packages on multiple platforms and only a subset supports Trusted Publishers).

[woodruffw]

I suspect situations where an API key will continue to be used will happen on the order of minutes to hours (ie: we build packages on multiple platforms and only a subset supports Trusted Publishers).

That's a good point! It'd be nice to have numbers/stats here, but I wouldn't be surprised if there was a bimodal distribution here: one (larger) cohort that has both TP and API tokens because they forgot to revoke the latter, and a second (smaller) cohort than has both and both are fully active.

[facutuesca]

If all (and other?) conditions are met, we could revoke the qualifying tokens and send the project's owners notification emails letting them know.

I think revoking unused tokens without user intervention might be a bit too aggressive. An alternative would be sending an email notifying the user that "the token will be revoked in 7 days unless this link is clicked" (or similar). That way we still proactively delete unused/risky tokens, but we give the project owners the ability to opt out of the deletion.

[miketheman]

I'll 👍 the "let's be careful about automated revocation". An additional qualifier I'd want is to only apply to tokens that are scoped to a project, and only that project, not a user's account.

And I don't think revoking is a great experience either - I'd prefer a notification post-usage (and prevent repeats within the default repeat_window of two weeks) that tells the user what this is, why it's a problem, and links them to the tokens management page to take action themselves.

[woodruffw]

I'll 👍 the "let's be careful about automated revocation". An additional qualifier I'd want is to only apply to tokens that are scoped to a project, and only that project, not a user's account.

Yeah, I think this would be a hard requirement -- anything that's user-scoped wouldn't be covered under this.

Agreed about revocation not having the best UX, although I think the recent Ultralytics incident shows that people aren't checking/heeding these notifications -- we already have a email notification in place for "you used an API token after configuring a trusted publisher," and it looks like that didn't move the needle here 😞

OTOH, maybe we could also improve that existing email notification a bit. I can look into that as well.

To iterate a bit, how does this policy sound to people?

• API tokens can only be considered for revocation if they're scoped to a single project, not user-wide
• API tokens must be unused for at least 1 month
• There must be a Trusted Publisher configured for that project, and it must have been active more recently than the API token
• The project's owner(s) are sent warning emails telling them that they have 7 days (then 3, then 1) to click a magic link that keeps the API token from being revoked
• If after 1 months of inactivity + 1 week of warnings (so 5 weeks total) nothing changes, then the API token is revoked and the owner(s) are sent a revocation email

I think this would avoid most of the UX tarpits in revocation (e.g. we wouldn't revoke for *completely inactive/dormant projects, only ones where the use pattern suggests that they've forgotten about their old API tokens), but it's likely I'm missing other edge cases 🙂

[di]

Regarding the heuristic we should use here to identify "unused" tokens: maybe we should warn on any API token for a project that wasn't used for the most recent release, after {some amount of time} has passed since that release was created?

[webknjaz]

@di would my supplementary notifications/warnings idea be useful in this context as well? #17241 (comment)

[di]

Possibly? But it would have to be added to the upload API, not the token exchange API.