feat: Automatic sqlite-database-integration upgrade #136

dcalhoun · 2024-05-15T20:42:58Z

Resolves https://github.com/Automattic/dotcom-forge/issues/6939.
Fixes https://github.com/Automattic/dotcom-forge/issues/6922.
Relates to https://github.com/Automattic/dotcom-forge/issues/6598.
Relates to #132.
Relates to https://github.com/Automattic/local-environment/pull/245 and https://github.com/Automattic/local-environment/pull/228.

Proposed Changes

Update the sqlite-databse-integration installation to the latest release tag on app start.
Update the sqlite-databse-integration installation to the latest release tag on site server start.

Testing Instructions

Important

In order to observe the sqlite-integration-plugin upgrade, you must first install a legacy version of the plugin.

Check out the trunk branch.
Download a legacy version of sqlite-integration-plugin — e.g., v2.1.9.
"Install" the legacy version by replacing the existing copy found in the following locations.

Note: naming the directory sqlite-database-integration-main (legacy name) would simulate testing the rollout of these changes to existing installs/sites; naming the directory sqlite-database-integration (new name) would simulate testing an additional SQLite upgrade (second, third, etc) after a previous install (first SQLite upgrade, new site creation using these changes, etc).
- Bundled Installation: This is found in the cloned repository at wp-files/sqlite-database-integration-main.
- Downloaded Installation: On macOS, this is found in ~/Library/Application\ Support/Studio/server-files/sqlite-database-integration-main.
Start the app server. (Note, you should not run npm install at this point as it would update the legacy bundled installation.)
Create a new site ("Legacy Site").
Verify the Legacy Site's wp-content/mu-plugins directory contains the legacy version of the plugin.

1: Site actions remain stable without a network connection

Check out the proposed changes.
Run npm install to download/bundle the latest version of sqlite-database-integration.
Disable your internet connection.
Start the app server.
Create a new site.
Verify no errors are thrown during site creation or site server start.

2: New sites leverage the latest version

Check out the proposed changes.
Start the app server.
Create a new site.
Verify the sqlite-integration-plugin version found in the site's wp-content/mu-plugins directory is the latest release.

3: Existing sites upgrade when started

Check out the proposed changes.
Start the app server.
Start the Legacy Site's server.
Verify the sqlite-integration-plugin version found in the site's wp-content/mu-plugins directory is the latest release.

4: Woocommerce loads without database query errors

Check out the proposed changes.
Start the app server.
Install and activate the WooCommerce plugin (Steps outlined in https://github.com/Automattic/dotcom-forge/issues/6922).
Verify no database query errors are logged and the plugin page loads successfully. Note: If MailPoet is installed and activated, errors may display.

~~5: UpdraftPlus loads without database query errors~~

Test case removed as the proposed changes do not address all errors.

6: Fresh Studio installs successfully create and start sites

Delete all existing Studio sites.
Clear out the bundled and downloaded sqlite-database-integration installations.
Run npm install to download/bundle the latest version of sqlite-database-integration.
Start the app server.
Create a new site.
Verify the sqlite-integration-plugin version found in the site's wp-content/mu-plugins directory is the latest release.

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

Improve stability and enable version comparisons for future upgrades.

Ensure new sites receive the latest sqlite-database-integration fixes and improvements.

Ensure existing sites receive the latest fixes and features.

If version comparison fails, assume the installed version is the latest. For example, fetching the latest version while offline will result in a comparison failure.

Avoid repeatedly fetching the latest versions each time a site starts.

scripts/download-wp-server-files.ts

src/ipc-handlers.ts

src/lib/sqlite-versions.ts

vendor/wp-now/src/download.ts

dcalhoun · 2024-05-17T12:25:59Z

Expanded the testing instructions in the description as I realized this will also fix https://github.com/Automattic/dotcom-forge/issues/6922 and https://github.com/Automattic/dotcom-forge/issues/6598.

sejas

I tested all the scenarios and worked great in all of them.
I added a few comments. My only suggestion is to use the same source, wp.org or GitHub, for the bundled version and the start/updated version.

Updated existing SQLite version from 2.1.7 to 2.1.10

sqlite-existing-site.mp4

sqlite-new-site.mp4

sqlite-woo.mp4

sqlite-updraft-plus.mp4

scripts/download-wp-server-files.ts

src/ipc-handlers.ts

src/lib/sqlite-versions.ts

We only ever install the latest version, so we do not need the unnecessary complexity of fetching past versions. Also, this allows us to use the same download source the `sqlite-database-integration` plugin throughout the source.

src/setup-wp-server-files.ts

fluiddot · 2024-05-20T11:12:19Z

Testing Instructions
UpdraftPlus loads without database query errors

I tested the plugin and I'm encountering the same JS exceptions outlined in https://github.com/Automattic/dotcom-forge/issues/6598:

ReferenceError: updraftlion is not defined

I presume that using the latest SQLite version solved database errors, but as far as I've tested, I think this PR won't solve the issue. The plugin doesn't seem to function properly, clicking the Backup Now button results in no actions.

dcalhoun · 2024-05-20T12:16:08Z

I tested the [UpdraftPlus] plugin and I'm encountering the same JS exceptions outlined in Automattic/dotcom-forge#6598:
ReferenceError: updraftlion is not defined

Thank you for sharing this. From earlier testing, my perception was that this error was present for non-Playground sites as well, but it appears I was mistaken. I'll debug this further.

The `-main` suffix is no longer accurate now that we rely upon release tags rather than downloading the latest development branch.

dcalhoun

Thank you @sejas and @fluiddot for reviewing and helping me test this thoroughly. Apologies for the overlooked cases. I believe this is ready for review again.

I updated the test case 4 (WooCommerce) to note that errors remain if MailPoet is activated.
I removed test case 5 (UpdraftPlus) as the proposed changes do not address all the errors.
I added test case 6 to ensure that fresh installations of Studio — which lack sqlite-database-integration plugin installations entirely — work as expected.

src/setup-wp-server-files.ts

fluiddot

LGTM 🎊 ! I've tested the different test scenarios and worked as expected. Thanks @dcalhoun for adding this feature 🏅 !

I added a comment that shouldn't block the PR but might be interesting to consider.

Nitpick: In the Important section of Testing Instructions, there are references to the old SQLite filename sqlite-database-integration-main. I presume it should be sqlite-database-integration, without the suffix -main`.

fluiddot · 2024-05-20T17:45:17Z

src/lib/sqlite-versions.ts

+function getSqliteVersionFromInstallation( installationPath: string ): string {
+	let versionFileContent = '';
+	try {
+		versionFileContent = fs.readFileSync( path.join( installationPath, 'load.php' ), 'utf8' );


In general, we try to avoid using the sync version of FS functions to prevent blocking the main thread. I wonder if we could use the asynchronous one.

dcalhoun · 2024-05-20T18:35:10Z

Nitpick: In the Important section of Testing Instructions, there are references to the old SQLite filename sqlite-database-integration-main. I presume it should be sqlite-database-integration, without the suffix -main.

@fluiddot Thanks for the note!

My thought is that testing old-version SQLite installations with and without the -main suffix is relevant. The former applies for existing users receiving upgrades to existing installations with -main. The latter applies for users receiving upgrades to existing installations without -main (e.g., upgrading SQLite a second time or installing Studio for the first time after this change ships).

I updated the documentation account for both of these nuanced circumstances.

dcalhoun added the [Type] Enhancement Improvement upon an existing feature label May 15, 2024

dcalhoun self-assigned this May 15, 2024

dcalhoun added 2 commits May 16, 2024 19:57

feat: Download latest sqlite-databse-integration release tag

2e25c86

Improve stability and enable version comparisons for future upgrades.

feat: Starting the app updates sqlite-database-integration installation

3b20384

Ensure new sites receive the latest sqlite-database-integration fixes and improvements.

dcalhoun force-pushed the feat/automatic-sqlite-upgrade branch from fe6035d to 2db6c54 Compare May 16, 2024 23:59

dcalhoun added 3 commits May 16, 2024 20:04

feat: Starting a site updates an outdated sqlite-integration-plugin

847e40a

Ensure existing sites receive the latest fixes and features.

fix: Avoid unnecessary sqlite-database-integration upgrades

8afbeb0

If version comparison fails, assume the installed version is the latest. For example, fetching the latest version while offline will result in a comparison failure.

feat: Cache sqlite-database-integration versions for app session

f0aa6f2

Avoid repeatedly fetching the latest versions each time a site starts.

dcalhoun force-pushed the feat/automatic-sqlite-upgrade branch from 2db6c54 to f0aa6f2 Compare May 17, 2024 00:04

dcalhoun changed the title ~~feat: Automatic SQLite upgrade~~ feat: Automatic sqlite-database-integration upgrade May 17, 2024

dcalhoun commented May 17, 2024

View reviewed changes

scripts/download-wp-server-files.ts Show resolved Hide resolved

src/ipc-handlers.ts Show resolved Hide resolved

src/lib/sqlite-versions.ts Outdated Show resolved Hide resolved

vendor/wp-now/src/download.ts Outdated Show resolved Hide resolved

dcalhoun marked this pull request as ready for review May 17, 2024 00:41

dcalhoun requested review from a team May 17, 2024 00:41

dcalhoun mentioned this pull request May 17, 2024

WooCommerce errors: cannot create a manual order/ see the home dashboard #132

Closed

sejas approved these changes May 17, 2024

View reviewed changes

scripts/download-wp-server-files.ts Show resolved Hide resolved

src/ipc-handlers.ts Show resolved Hide resolved

src/lib/sqlite-versions.ts Show resolved Hide resolved

src/lib/sqlite-versions.ts Outdated Show resolved Hide resolved

dcalhoun added 2 commits May 18, 2024 07:35

refactor: Remove unused import

b0861b3

dcalhoun requested a review from sejas May 18, 2024 12:56

fluiddot reviewed May 20, 2024

View reviewed changes

src/setup-wp-server-files.ts Show resolved Hide resolved

refactor: Remove -main branch suffix from SQLite install

f1d2dae

The `-main` suffix is no longer accurate now that we rely upon release tags rather than downloading the latest development branch.

dcalhoun commented May 20, 2024

View reviewed changes

src/setup-wp-server-files.ts Show resolved Hide resolved

dcalhoun requested a review from fluiddot May 20, 2024 17:06

fluiddot approved these changes May 20, 2024

View reviewed changes

dcalhoun merged commit d296d88 into trunk May 21, 2024
14 checks passed

dcalhoun deleted the feat/automatic-sqlite-upgrade branch May 21, 2024 01:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Automatic sqlite-database-integration upgrade #136

feat: Automatic sqlite-database-integration upgrade #136

dcalhoun commented May 15, 2024 •

edited

Loading

dcalhoun commented May 17, 2024

sejas left a comment •

edited

Loading

fluiddot commented May 20, 2024

dcalhoun commented May 20, 2024

dcalhoun left a comment

fluiddot left a comment

fluiddot May 20, 2024

dcalhoun commented May 20, 2024

feat: Automatic sqlite-database-integration upgrade #136

feat: Automatic sqlite-database-integration upgrade #136

Conversation

dcalhoun commented May 15, 2024 • edited Loading

Proposed Changes

Testing Instructions

Pre-merge Checklist

dcalhoun commented May 17, 2024

sejas left a comment • edited Loading

Choose a reason for hiding this comment

fluiddot commented May 20, 2024

dcalhoun commented May 20, 2024

dcalhoun left a comment

Choose a reason for hiding this comment

fluiddot left a comment

Choose a reason for hiding this comment

fluiddot May 20, 2024

Choose a reason for hiding this comment

dcalhoun commented May 20, 2024

dcalhoun commented May 15, 2024 •

edited

Loading

sejas left a comment •

edited

Loading