base-files: hardcoded cache of base-files version per RELEASE/ARCH, as of 20240522

[rpardini]

base-files: hardcoded cache of base-files version per RELEASE/ARCH, as of 20240522

• base-files: hardcoded cache of base-files version per RELEASE/ARCH, as of 20240522
  • packages.debian.org and packages.ubuntu.com are having a meltdown and builds are stuck
  • this is a quick & dirty fix to get builds going
  • we should urgently replace this with the 'right thing'
  • trying to maintain this manually will cause insanity

[rpardini]

not proud of 1) having done the curl version 2) replacing it with a hardcoded list 3) admins of packages.ubuntu.com 4) myself

[igorpecovnik]

Little time for proper way - mitigating the ugliness:

• merging this only to v24.5 as this is urgent mainly for upcoming release? (and remove once done well)
• making as extension?

[rpardini]

this is too dirty to make into an extension.
the proper way will take a while, I'll be busy with smth else the next few days, so here's what we got.
merging to main should be safe; since versions are hardcoded & will be cached in OCI, it should work "forever" -- but we might miss updates published by Debian/Ubuntu.
Either way should buy us a few weeks to work proper way.

[amazingfate]

What about checking it from apt repo?

curl http://ports.ubuntu.com/ubuntu-ports/dists/noble/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'
curl http://deb.debian.org/debian/dists/bookworm/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'

[rpardini]

What about checking it from apt repo?

curl http://ports.ubuntu.com/ubuntu-ports/dists/noble/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'
curl http://deb.debian.org/debian/dists/bookworm/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'

This approach I considered before I wrote the current "packages.debian.org" approach.
It works, but:

• the Packages.xz download is significant (2mb for Ubuntu, 8mb for Debian)
• the decompression is also a bit heavy
• this needs to runs on every board build, which is fine for individuals, but CI runs 400+ of them

In comparison, back when it was working, the packages.xx.yy hit was small (few Kbs) and returned very fast.
Unfortunately ofc now packages.ubuntu.com is taking 10+ seconds to give a HTTP 500 error about 80% of the time, so that's down the drain.

The "proper way" I suggest is to use something similar to https://github.com/armbian/shallow -- a separate repo (with GHA action on a schedule) that gathers info for a given RELEASE + ARCH, massages it the way we need, and stores it in OCI/git/the matrix/wherever in a way that is easy for armbian/build to consume. Not only the base-files version, but also maybe simple info about every upstream package (so we can validate package lists / determine estimated size of rootfs's / generate a dependency tree etc).