fails before generating views. ERR: table sqlite_master may not be modified

[pax]

generates checkins.db but seems to fail before generating views

note: it worked on an Ubuntu WSL but fails on macOS 12.5.1

later edit: I suspect this is a problem with my local set-up, dogsheep-beta index also throws the same error

full error:

Importing 2591 checkins  [###################################-]   98%  00:00:00
Traceback (most recent call last):
  File "/Users/pax/devbox/envAll/bin/swarm-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/cli.py", line 77, in cli
    ensure_foreign_keys(db)
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/swarm_to_sqlite/utils.py", line 145, in ensure_foreign_keys
    db[fk.table].add_foreign_key(fk.column, fk.other_table, fk.other_column)
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py", line 2123, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File "/Users/pax/devbox/envAll/lib/python3.8/site-packages/sqlite_utils/db.py", line 1086, in add_foreign_keys
    cursor.execute(
sqlite3.OperationalError: table sqlite_master may not be modified

[simonw]

I just had that error myself on macOS while running the tests:

ERROR tests/test_save_checkin.py::test_tables - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_venue - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_event - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_sticker - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_likes - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_with_ - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_users - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_photos - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_posts - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_view - sqlite3.OperationalError: table sqlite_master may not be modified

pytest --pdb shows it happening in the bit that adds foreign keys:

> /Users/simon/.local/share/virtualenvs/swarm-to-sqlite-daPW7yIJ/lib/python3.9/site-packages/sqlite_utils/db.py(1096)add_foreign_keys()
-> cursor.execute(
(Pdb) list
1096 >>	                cursor.execute(
1097 	                    "UPDATE sqlite_master SET sql = ? WHERE name = ?",
1098 	                    (new_sql, table_name),
1099 	                )
1100 	            cursor.execute("PRAGMA schema_version = %d" % (schema_version + 1))
1101 ->	            cursor.execute("PRAGMA writable_schema = 0")
1102 	        # Have to VACUUM outside the transaction to ensure .foreign_keys property
1103 	        # can see the newly created foreign key.
1104 	        self.vacuum()

[simonw]

Related:

• Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified simonw/sqlite-utils#235

[simonw]

OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above.

[simonw]

I think swarm-to-sqlite needs to avoid this error, maybe by setting up foreign keys in another way - or even by skipping foreign keys entirely on databases that don't support this kind of operation.