Compact indexer versions API is not organizing gems added after a certain point.

[LukeIGS]

The versions api seems to be designed to group gems by their name, then versions, however at some point it seems to have stopped adding new versions of a given gem to the version list for that gem and recalculating the hash at the end of the line.

Steps to Reproduce

1. visit https://rubygems.org/versions
2. find 'zzzzzz', the last gem that's properly alphabetized
3. note that beyond this point each gem only has one version and there are multiple duplicates of gems that are listed further up in the file

Expected Behavior

Gems that exist further up in the file should have their versions appended to the list for that gem, and remain in alphabetical order.

Current Behavior

Duplicated entries exist in the versions file listing one version of the gem each.

Environment

Browser and its version: Vivaldi 6.1.3035.111 (Stable channel) (64-bit)

Additional Context

This is a problem for gem proxy servers like geminabox that need to query the version list from rubygems.org and merge it, but may have other implications as well that I'm unaware of with regards to bundler.

[simi]

Hello @LukeIGS! This is expected. versions are generated from static part (versions file, could be generated using bin/rake compact_index:update_versions_file), that's the one ordered by gem name and from dynamic part (check GemInfo#compact_index_versions method) which is appended to the static part and ordered by date.

The static file is at rubygems.org renegerated each month using https://github.com/rubygems/rubygems.org/blob/fcf030c9932df4fb6b0c5f99e878ef2790ca4fd9/config/deploy/versions-list-update-monthly.yaml.erb job.

Currently I have no idea how to merge those two lists for geminabox proxy functionality.

[LukeIGS]

Alright, if this is intentional than i can probably work around it when merging the file. Just wanted to make sure this wasn't a bug or something as the spec for how this works is pretty hard to track down. Would be nice if the rubygems.org documentation section included mention of the compact index api implementations or something i guess?

[simi]

@LukeIGS isn't https://github.com/rubygems/compact_index/#updating-the-versions-file enough?

[Castone22]

Yeah, I haven't had time to work on it, but from what I've gathered it should be good enough to just prepend our versions to the list, smack the updated date on it and strip out any gems that share a name with a local one. The brief simply states that it must be organized somehow. Rubygems.org does alphabetical followed by date added for any runtime updates between updates so we can just blindly take that. Our final order could be locals followed by remotes sans duplicated names

[segiddins]

The order really doesn't matter, so long as (the majority of the time) it is append-only. We do the monthly update of the "base" file only to keep file size minimal.

[LukeIGS]

@LukeIGS isn't https://github.com/rubygems/compact_index/#updating-the-versions-file enough?

Even if it is, it should probably be referenced somewhere in the rubygems.org documentation (https://guides.rubygems.org/rubygems-org-api/ might be a good spot) instead of having to go to a random git repo to find it, also the way that rubygems.org implemented it could be different than the gem.

I know i'm not the only person looking for this.

The order really doesn't matter, so long as (the majority of the time) it is append-only. We do the monthly update of the "base" file only to keep file size minimal.

That's good to know, and makes the job much easier for finalizing the versions list merge in geminabox.
Since this isn't a bug we can go ahead and close this.

[simi]

https://guides.rubygems.org/rubygems-org-api/ is good place, but it is not clear which endpoints are mandatory for gem/bundle commands (for gemserver functionality) and which are just RubyGems.org public API around additional functionality. It is on my list to document minimal gemserver API. There is some documentation effort already started by @segiddins at #3896.

[simi]

@LukeIGS can we close this issue?