Add SQL-based migration script, with migration dir

[spwoodcock]

Issue

• Currently we have no way of migrating new database edits on editing databases.
• We tested using alembic, but the setup is too complex for our needs and it's a bit of a black box.
• In the future we plan to remove SQLAlchemy anyway: Replace SQLAlchemy ORM with psycopg db driver plus optional query builder #254

Solution

I have made a migration bash script, that is bundled with the backend.

The way it works:

• SQL migration scripts are placed in the src/backend/migrations dir.
  • These should be idempotent, i.e. can run over and over without causing errors.
  • There should also be a commented out SQL script for how to revert the migration.
  • Scripts should be named sequentially, i.e. the first is 001-some-migration.sql, then they increment by one.
  • Example 000-remove-user-password.sql:

-- ## Migration to remove password field from public.users (replaced with OSM OAuth)


-- ## Apply Migration
-- Start a transaction
BEGIN;
-- Drop the 'password' column if it exists
ALTER TABLE IF EXISTS public.users
DROP COLUMN IF EXISTS password;
-- Commit the transaction
COMMIT;


-- ## Revert Migration (comment above, uncomment below)
-- -- Start a transaction
-- BEGIN;
-- -- Add the 'password' column back if it doesn't exist
-- ALTER TABLE public.users
-- ADD COLUMN IF NOT EXISTS password character varying;
-- -- Commit the transaction
-- COMMIT;

• When the docker compose stack starts, an additional container starts up and runs a bash script once.
• The script generates a table called migrations, which simply tracks the script name and execution date.
• The migrations directory is scanned for new files, and if there is no record in the database of being applied, the migration is applied.

Note

I also lazily bundled a quick fix for the CI backend docker image build in this PR.

tl;dr

Create a .sql script under src/backend/migrations and it will be applied when the backend next starts.

[nrjadkry]

@spwoodcock this looks good.
I have tested this in my setup too. One thing I notice is that, how should I run the migration script after I made a new migration file. Is there any script that runs migration files in the current db, without having to do docker restart like the one we have in alembic alembic upgrade head

By the way, have you added this in the documentation too. If not, shall I give it a go.

[spwoodcock]

Thanks for the reminder about docs 👍

I'll update before I merge & include info on running manually during development (during deploy to different environments this will be handled automatically).

Option 1 is like you say: docker compose restart migrations.

Option 2: docker compose exec api bash /migrate-entrypoint.sh.

Option 3:

Make sure you have the 4 env vars for the database connection set on your machine, then:

bash src/backend/migrate-entrypoint.sh