chore(ci): add individual publishing of packages MONGOSH-1871 #2289

gagik · 2024-12-10T11:17:29Z

This adds the ability to individually publish packages in mongosh. Also, notably, all packages's version inside package.json will now match the latest released version and update accordingly upon release.

Structure

Package Bumping

With this change, there are 2 types of package bumping depending on what package it is:

mongosh release-related packages bump
mongosh and @mongosh/cli-repl are bumped only at mongosh release and have their versions synced.
This gets triggered only during a mongosh release.
npm packages bump
All other packages are bumped independently of mongosh using bump-packages from monorepo-tools. Each package can have divergent versions as a result.
This can be triggered anytime using GitHub actions which will create PR for bump package versions, merging of which will cause CI to publish the packages.

Individual Package Release

You can release an individual package by running a bump packages GitHub Action and merging the created PR.

mongosh Release

A mongosh release will cause both an npm package bump and a mongosh release bump before starting the release.

lerna.json

gagik · 2024-12-18T12:13:49Z

lerna.json

-  ],
-  "version": "0.0.0-dev.0"
+  "packages": ["configs/*", "packages/*", "scripts/docker"],
+  "version": "independent"


@addaleax I'm a bit unsure how to go about all this to the point where I can feel much confidence;
What I did here is

remove lerna force publish

use lerna independent mode

add the bump script
I feel like this should hypothetically do the trick. maybe a couple more changes might be all we need.

I tried i.e. running npm run bump-packages locally and I wonder if it is because of the fact that this PR itself technically changes all of the packages, we end up with a bunch of bumps; some reasonable others not so much.

I assume some of these packages we'd like to ignore right? And also some which we should keep at 0.0.0-dev.0 for our development?

I assume some of these packages we'd like to ignore right? And also some which we should keep at 0.0.0-dev.0 for our development?

Well, the mongosh and @mongosh/cli-repl packages specifically should still be bumped as part of the process we have now, that can be by using the 0.0.0-dev.0 placeholders but I don't think that's a strict requirement here

packages/snippet-manager/package.json

packages/build/src/npm-packages/publish.ts

packages/build/src/release.ts

addaleax · 2024-12-18T16:22:56Z

lerna.json

-  ],
-  "version": "0.0.0-dev.0"
+  "packages": ["configs/*", "packages/*", "scripts/docker"],
+  "version": "independent"


I assume some of these packages we'd like to ignore right? And also some which we should keep at 0.0.0-dev.0 for our development?

Well, the mongosh and @mongosh/cli-repl packages specifically should still be bumped as part of the process we have now, that can be by using the 0.0.0-dev.0 placeholders but I don't think that's a strict requirement here

gagik · 2024-12-20T17:10:25Z

@addaleax Didn't get to spend too much time on this before my holiday with some VSCode and coverage test back and forths. Would be happy to jump back into it in a week but if it'd be better to finish this earlier, feel free to pick it up 🙂 I have some WIP files that I may polish up and push at a later time also though not sure how useful they are; will see.

gagik · 2024-12-31T11:28:08Z

.evergreen/evergreen.yml.in

@@ -917,6 +917,9 @@ functions:
          {
          export NODE_JS_VERSION=${node_js_version}
          source .evergreen/setup-env.sh
+          npm run evergreen-release bump
+          git add .
+          git commit --no-allow-empty -m "chore(release): bump to prepare for mongosh release"


The idea here is that this will bump both "independent" and mongosh packages before the release draft and push it to main, so our branch is consistent with what we're releasing.
It might also make sense to only persist this after the release is published.

I am assuming this should work? Though not sure if the evergreen git setup has ability to push to main like this?

gagik · 2024-12-31T11:33:02Z

packages/build/src/npm-packages/constants.ts

+export const MONGOSH_RELEASE_PACKAGES = ['mongosh', '@mongosh/cli-repl'];
+
+/** Packages which always get ignored when doing a release or bump */
+export const IGNORE_BUMP_PACKAGES = [


ignoring these since these are private, unless it'd make sense to publish anyways?

gagik · 2024-12-31T11:34:48Z

packages/arg-parser/package.json

@@ -1,6 +1,6 @@
 {
  "name": "@mongosh/arg-parser",
-  "version": "0.0.0-dev.0",
+  "version": "2.3.7",


these were all set to 2.3.7 manually to get the current repository consistent with the latest release

gagik · 2024-12-31T11:37:38Z

.evergreen/install-npm-deps.sh

@@ -31,7 +31,5 @@ npm run mark-ci-required-optional-dependencies
 # along with its types, but npm wouldn't try and compile the addon
 (npm ci && test -e node_modules/mongodb-client-encryption) || npm ci --ignore-scripts

-npm run evergreen-release bump


unless I am misunderstanding something, I don't think this bump was necessary to get the dir-based packages installed for the sake of i.e. tests, etc..
It does seem like that this acted as a preliminary step for different CI runs but the idea going forward is that the repository will have the latest published version and only get bumped before releases and persist for future runs

I will double check this as I did see some weird error with browser-repl tests that may be caused by this

gagik · 2024-12-31T11:43:20Z

.github/workflows/bump-packages.yml

+        uses: mongodb-js/devtools-shared/actions/setup-bot-token@main
+        id: app-token
+        with:
+          app-id: ${{ vars.DEVTOOLS_BOT_APP_ID }}


I think this should be already set for us?

gagik · 2024-12-31T11:52:27Z

package.json

-    "preinstall": "node scripts/sort-workspaces.js"
+    "preinstall": "node scripts/sort-workspaces.js",
+    "bump-packages": "npm run bump-packages --workspace @mongosh/build",
+    "publish-packages": "lerna publish from-package --no-verify-access --no-push --no-git-tag-version --yes"


This is 1:1 same as compass, I assume should be fine here also with our lerna setup, unless there's some special option we should pass for mongosh packages.

nirinchev

I know this is modeled after what we already have and I'm not opposed to merging it as-is. That being said, I'd prefer if we move to a different publishing model, where we run the publish packages workflow on the bump packages PR, merge that PR, and only then tag the versions on main. That way we would eliminate a case where we have versions committed in main that haven't been published due to a glitch somewhere.

Additionally, we should move the npm token to an environment secret, which will require approvals for the workflow to run.

.github/workflows/bump-packages.yml

.github/workflows/release-packages.yml

Co-authored-by: Nikola Irinchev <[email protected]>

gagik · 2024-12-31T12:21:44Z

@nirinchev Agree, wanted to ask before expanding the scope but I think it'd be a lot nicer and cleaner with this setup. I can clean that up and the documentation

gagik force-pushed the gagik/individual-publishing branch from 879eb0f to 40d6e71 Compare December 10, 2024 11:17

WIP: individual publishing of packages

a2f246b

gagik force-pushed the gagik/individual-publishing branch from f48cc5c to a2f246b Compare December 18, 2024 12:05

gagik commented Dec 18, 2024

View reviewed changes

lerna.json Show resolved Hide resolved

use 2.3.7

417e1f4

gagik commented Dec 18, 2024

View reviewed changes

update package-lock.json and packages

447e8d1

addaleax reviewed Dec 18, 2024

View reviewed changes

gagik added 3 commits December 19, 2024 11:08

restore changes

4c97468

ensure compatibility with old bump

8c9c214

wip

03ba373

gagik force-pushed the gagik/individual-publishing branch from 8c9c214 to 41bc10a Compare December 20, 2024 14:55

switch to bump script

ad630dc

gagik force-pushed the gagik/individual-publishing branch from 41bc10a to ad630dc Compare December 30, 2024 11:48

gagik added 8 commits December 30, 2024 19:15

Merge branch 'main' into gagik/individual-publishing

28bfce7

fix package bumping

45469f9

check if bump is necessary for installs

be919e0

remove old tests

43c551d

add publish-packages

71aaa79

remove package.json changes

ceb13e7

add ignore list

42d19ea

set non-release package.json to 2.3.7

bad9d59

gagik changed the title ~~WIP: individual publishing of packages~~ chore(ci): add individual publishing of packages MONGOSH-1871 Dec 31, 2024

gagik added 2 commits December 31, 2024 12:24

add bumping commit to release draft

193aa09

set all to 2.3.7

a19f408

gagik commented Dec 31, 2024

View reviewed changes

gagik marked this pull request as ready for review December 31, 2024 11:38

gagik commented Dec 31, 2024

View reviewed changes

nirinchev reviewed Dec 31, 2024

View reviewed changes

.github/workflows/bump-packages.yml Outdated Show resolved Hide resolved

.github/workflows/release-packages.yml Outdated Show resolved Hide resolved

.github/workflows/release-packages.yml Outdated Show resolved Hide resolved

gagik and others added 2 commits December 31, 2024 13:18

Update .github/workflows/release-packages.yml

7ec4049

Co-authored-by: Nikola Irinchev <[email protected]>

fix removed compass skip

67c690d

gagik added 3 commits December 31, 2024 13:55

switch to use app token

9c4b387

use bump in install script

4f43369

remove bump step

c9fd356

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

gagik commented Dec 10, 2024 •

edited

Loading

gagik Dec 18, 2024 •

edited

Loading

addaleax Dec 18, 2024

addaleax Dec 18, 2024

gagik commented Dec 20, 2024

gagik Dec 31, 2024

gagik Dec 31, 2024

gagik Dec 31, 2024

gagik Dec 31, 2024

gagik Dec 31, 2024 •

edited

Loading

gagik Dec 31, 2024

gagik Dec 31, 2024

nirinchev left a comment

gagik commented Dec 31, 2024

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

Are you sure you want to change the base?

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

Conversation

gagik commented Dec 10, 2024 • edited Loading

Structure

Package Bumping

Individual Package Release

mongosh Release

gagik Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gagik commented Dec 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gagik Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirinchev left a comment

Choose a reason for hiding this comment

gagik commented Dec 31, 2024

gagik commented Dec 10, 2024 •

edited

Loading

gagik Dec 18, 2024 •

edited

Loading

gagik Dec 31, 2024 •

edited

Loading