v-2024-05-13-mbdb29-pg16

Breaking changes

This release upgrades both the PostgreSQL version to 16 and the MusicBrainz database schema version to 29. It requires to run commands taking down all services during the upgrade.

Docker images for the services db and musicbrainz are now prebuilt too. It shorten the build time by half an hour to just a few minutes. Running any command that requires Perl in the service musicbrainz should now be prefixed with carton exec -- .

Disk space requirement has been slightly readjusted: 250 GB (or 100 GB without indexed search), system included.

If your musicbrainz-docker is earlier than v-2023-05-16-mbdb28, you must first update to it (see its release notes) and follow the upgrade instructions below, or remove everything using sudo docker-compose down --rmi local --volumes and reinstall.

Upgrade instructions

Assuming your checked out version of musicbrainz-docker is v-2023-05-16-mbdb28 or later, then:

1. Ensure you’ve replicated up to the most recent replication packet available with the old schema 28. If you’re not sure, run:

   sudo docker-compose exec musicbrainz bash -c ./admin/replication/LoadReplicationChanges

   and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #29, but the database is currently at #28.”

2. Turn off cron jobs:

   admin/configure rm replication-cron
   sudo docker-compose up -d musicbrainz

3. Switch to the new code with:

   git fetch --tags origin
   git checkout v-2024-05-13-mbdb29-pg16

4. Upgrade the PostgreSQL data format:

   ./admin/upgrade-to-postgres16

   This step is expected to take some time, especially the last vacuumdb commands.
   (It took ~20 minutes with 4GB of shared buffers in test.)
   It should say “Upgrade complete!” at the end.

5. Upgrade the MusicBrainz database schema:

   mkdir -p local/compose/ && cp -a admin/lib/upgrade-db-schema/musicbrainz-stopped.yml local/compose/
   admin/configure add local-compose-musicbrainz-stopped
   sudo docker-compose up -d musicbrainz
   sudo docker-compose exec musicbrainz upgrade-db-schema.sh

   This step is expected to take a very long time, especially the last “Vacuuming DB”.
   (It took more than 2h with 4GB of shared buffers in test.)
   At the end it should say “Done” followed with a message that you can safely ignore as handled already: “UPDATE THE DB_SCHEMA_SEQUENCE IN DBDefs.pm TO 29 !”

   Troubleshooting

   The following lines are expected and can be safely ignored:

   NOTICE:  constraint "allowed_collection_entity_type" of relation "editor_collection_type" does not exist, skipping
   NOTICE:  constraint "label_code_length" of relation "label" does not exist, skipping
   

   If the last command quickly stops with the following error, it means that previous step 4 failed for some reason:

   ERROR:  relation "edit_note_change" already exists
   

   Please report this issue to us with the full log of the previous step(s) if possible.
   Then run the following commands instead:

   sudo docker-compose stop
   sudo docker-compose rm db
   sudo docker-compose build
   sudo docker-compose up -d musicbrainz
   sudo docker-compose exec musicbrainz upgrade-db-schema.sh
   

6. Load the first replication packet available with the new schema 29:

   sudo docker-compose exec musicbrainz bash -c 'carton exec -- ./admin/replication/LoadReplicationChanges --limit=1'

   and see what it tells you; if the upgrade, it should say “This packet was produced (or begins) at 2024-05-13 21:29:12.739887+00”

   Troubleshooting

   The following lines are expected and can be safely ignored (for the first packet only):

   The current row in musicbrainz.replication_control with key id='1' contains a different value in column current_schema_sequence (29) than the replication packet suggests it should have as the old value (28). at /musicbrainz-server/admin/replication/ProcessReplicationChanges line 497.
   WARNING:  amqp could not commit tx mode on broker 1, reply_type=2, library_errno=4
   

7. Turn cron jobs back on:

   admin/configure rm local-compose-musicbrainz-stopped
   admin/configure add replication-cron

8. Restart all services:

   sudo docker-compose up -d musicbrainz

Changes

• MBS-13358: Upgrade Perl version to 5.38.2 in musicbrainz service
• Perl module dependencies are now installed using Carton in musicbrainz service (see 98732f4):
  • Their versions now match production and will no longer fail on unexpected upstream breaking changes.
  • Running any command that requires Perl should now be prefixed with carton exec --
    See the above step 6. for example.
• MBS-13361: Upgrade PostgreSQL to version 16 in db service
• Upgrade MusicBrainz Server to v-2024-05-13-schema-change (release notes)
• MBVM-42: Prebuild all docker images (including for db and musicbrainz services)
  It shorten installation time by half an hour to just a few minutes.

v-2024-04-10-hotfix

Update instructions

This release can be safely skipped if you don’t use search indexes.

Just follow the new release notes for v-2024-04-09 instead if you didn’t already update to it.

Otherwise, that is assuming your checked out version of musicbrainz-docker is exactly v-2024-04-09, just run:

git fetch --tags origin && \
git checkout v-2024-04-10-hotfix && \
sudo docker-compose up --build -d

Changes

• Fix the regression MBVM-97 which made the indexer unable to reach the search server.
  No data loss is expected. In live indexing mode, the indexer will just catch up with updates.

v-2024-04-09

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-11-13 or later, then just run:

git fetch --tags origin && \
git checkout v-2024-04-10-hotfix && \
sudo docker-compose up --build -d

Additional update instructions for development setup only

Changes to the development setup require to remove dependencies that will be cleanly reinstalled on startup:

sudo docker-compose exec musicbrainz bash -c 'rm -fr node_modules perl_modules .cpanm' && \
sudo docker-compose restart musicbrainz

Additional update instructions for pre-PG 12 (2020) mirror only

In case your mirror is older than PostgreSQL 12 migration in 2020, if some update scripts didn’t work for some reason back in that time, you might have the following error in the file mirror.log or mirror.log.1 in the service musicbrainz since yesterday:

ERROR:  index row requires xxxxx bytes, maximum size is 8191

Then open a PostgreSQL invite, for example as follows:

sudo docker-compose exec musicbrainz bash -c admin/psql

Finally, run the following SQL instructions:

SET search_path = musicbrainz, public;
DROP INDEX IF EXISTS track_idx_musicbrainz_collate;
DROP INDEX IF EXISTS track_idx_name;
DROP INDEX IF EXISTS track_idx_txt;

See MBS-13536 for more info.

Changes

• Upgrade MusicBrainz Server to v-2024-04-09

• Make Postgres, RabbitMQ, and Redis hosts configurable via Docker environment variables

• Upgrade the version of Perl to 5.38 in development setup only

• Make bash history persistent for the musicbrainz service in development setup only

v-2024-03-11

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-11-13 or later, then just run:

git fetch --tags origin && \
git checkout v-2024-03-11 && \
sudo docker-compose up --build -d

Only Change

• Upgrade MusicBrainz Server to v-2024-03-11

v-2024-02-16

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-11-13 or later, then just run:

git fetch --tags origin && \
git checkout v-2024-02-16 && \
sudo docker-compose up --build -d

Changes

• Fix #270: Postgres db image build failing
• Upgrade MusicBrainz Server to v-2024-02-16

v-2024-01-08

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-11-13 or later, then just run:

git fetch --tags origin && \
git checkout v-2024-01-08 && \
sudo docker-compose up --build -d

Changes

• MBVM-93: Prune the entity cache after loading each replication packet
• Upgrade MusicBrainz Server to v-2024-01-08

v-2023-12-20

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-11-13 or later, then just run:

git fetch --tags origin && \
git checkout v-2023-12-20 && \
sudo docker-compose up --build -d

Only Change

• Upgrade MusicBrainz Server to v-2023-12-20

v-2023-11-27

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-11-13 or later, then just run:

git fetch --tags origin && \
git checkout v-2023-11-27 && \
sudo docker-compose up --build -d

Only Change

• Upgrade MusicBrainz Server to v-2023-11-27

v-2023-11-13

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-05-16-mbdb28 or later, then run:

git fetch --tags origin && \
git checkout v-2023-11-13 && \
sudo docker-compose up --build -d

Additional update instructions for development setup only

sudo docker-compose exec musicbrainz bash -c 'rm -fr node_modules perl_modules .cpanm' && \
sudo docker-compose rm --stop musicbrainz && \
sudo docker-compose up --build -d musicbrainz

Additional update instructions for all

In case you have any error related to the collation in the database, or preventively, run last:

sudo docker-compose exec musicbrainz bash -c './admin/RebuildIndexesUsingCollations.pl'

Changes

• Upgrade MusicBrainz Server to v-2023-11-13
  • Upgrade the version of Node to 20 as now required
    • Upgrade the version of Yarn to 4 now installed with Node
• Upgrade the base image for the service musicbrainz to Ubuntu Jammy
  • Upgrade the version of Perl to 5.34 as now supported
• Slightly simplify building the image for the service musicbrainz, to better follow the repository musicbrainz-server

v-2023-10-24-hotfix

Update instructions

Assuming your checked out version of musicbrainz-docker is v-2023-05-16-mbdb28 or later, then just run:

git fetch --tags origin && \
git checkout v-2023-10-24-hotfix && \
sudo docker-compose up --build -d

Changes

• Fix building Postgres 12 image for the service db (#259)
  (not affecting existing instances)
• Upgrade MusicBrainz Server to v-2023-10-24-hotfix
  (affecting existing mirror instances only).