Document rake build:checksum #325

pboling · 2022-10-27T22:51:01Z

Related to rubygems/rubygems#5942 and rubygems/rubygems#6022

Script uses a good regex, /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/, for pulling out a version match:
https://rubular.com/r/9QuDiGVjvlHOrt

deivid-rodriguez · 2023-06-16T13:59:46Z

security.md

+ rake build:checksum
+
+The checksums will be placed in the `checksums/` directory. It is recommended to commit checksums so others can verify
+the authenticity of a release.


I would migrate the previous instructions to use the builtin task and that's it. Officially recommending it should be a different discussion.

Makes sense. I'll update soon.

Removed the "recommendation" implication. Also improved the script to what I am currently using as I'm not sure when I'll get back around to the rake task, since in its current form it isn't useful to me at all, and I have a deep todo list. I'll remove the script portion if preferred.

security.md

pboling · 2023-10-10T18:05:08Z

Very strangely Gem::Version.new(version) compared to another of the same isn't working for pre-release versions. Outside pre-release versions, the script works. 😖

rubygems/rubygems#3086 (comment)

martinemde · 2023-10-10T22:35:49Z

security.md

+
+require "digest/sha2"
+
+VERSION_REGEX = /\d+\.\d+\.\d+([-.].+)*/.freeze


This is incorrect when applied to a filename since it will include .gem in the version number, making everything a pre-release.

Oooh, that's why it fails... interesting. I will see if I can find what rubygems uses for parsing gem names.

I knew it included the .gem, but it hadn't ever mattered until I had to deal with prereleases.

@martinemde Fixed!

pboling · 2023-10-11T06:14:18Z

# Final clause of Regex `(?=\.gem)` is a positive lookahead assertion
# See: https://learnbyexample.github.io/Ruby_Regexp/lookarounds.html#positive-lookarounds
# Used to pattern match against a gem package name, which always ends with .gem.
# The positive lookahead ensures it is present, and prevents it from being captured.
VERSION_REGEX = /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/.freeze

A good regex is hard to find. 🤣

pboling · 2023-10-12T17:22:42Z

security.md

@@ -72,61 +72,137 @@ Building Gems
 ### Sign with: `gem cert`

 1) Create self-signed gem cert
-
+ ```shell


The enumerated list was not proper markdown, and after a few syntax changes, which should also allow language-specific code highlighting, it is now proper markdown.

pboling · 2023-10-12T17:23:13Z

security.md

@@ -141,7 +217,7 @@ Reporting Security vulnerabilities

 ### Reporting a security vulnerability with someone else's gem

-If you spot a security vulnerability in someone else's gem, then you
+If you spot a security vulnerability in someone else's gem, then your


Just a typo you => your

pboling · 2023-10-12T17:25:13Z

security.md

+# See: https://learnbyexample.github.io/Ruby_Regexp/lookarounds.html#positive-lookarounds
+# Used to pattern match against a gem package name, which always ends with .gem.
+# The positive lookahead ensures it is present, and prevents it from being captured.
+VERSION_REGEX = /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/.freeze


This regex works perfectly.

https://rubular.com/r/9QuDiGVjvlHOrt

- rubygems/guides#325

pboling · 2023-10-20T03:45:22Z

Unfortunate discovery. The process of running a checksum script or task outside of the rake release results in checksums that do not match the actual released gem.

❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

de3b77b9434b36157efde7dc3617dc037708a83560bd54367e0dc1d0c9492e66ddaa624f217ed6759599d6b8208405db2f2f1671bbf9c4c725b8f33ec6886909
❯ bin/checksums
Looking for: "pkg/*.gem"
Found: 4 gems; latest is kettle-soup-cover-1.0.2.gem
[ GEM: kettle-soup-cover-1.0.2.gem ]
[ VERSION: 1.0.2 ]
[ GEM PKG LOCATION: pkg/kettle-soup-cover-1.0.2.gem ]
[ CHECKSUM SHA-256: 677d0f4778e065be83d9e77b0514b64731cb4313fa2b085056e9ec45f92991a5 ]
[ CHECKSUM SHA-512: de3b77b9434b36157efde7dc3617dc037708a83560bd54367e0dc1d0c9492e66ddaa624f217ed6759599d6b8208405db2f2f1671bbf9c4c725b8f33ec6886909 ]
[ CHECKSUM SHA-256 PATH: checksums/kettle-soup-cover-1.0.2.gem.sha256 ]
[ CHECKSUM SHA-512 PATH: checksums/kettle-soup-cover-1.0.2.gem.sha512 ]

... Running ...

git add checksums/* && git commit -m "🔒️ Checksums for v1.0.2"

[main e62712c] 🔒️ Checksums for v1.0.2
 2 files changed, 2 insertions(+)
 create mode 100644 checksums/kettle-soup-cover-1.0.2.gem.sha256
 create mode 100644 checksums/kettle-soup-cover-1.0.2.gem.sha512
❯ bundle exec rake release
Enter PEM pass phrase:
kettle-soup-cover 1.0.2 built to pkg/kettle-soup-cover-1.0.2.gem.
Tagged v1.0.2.
Pushed git commits and release tag.
Pushing gem to https://rubygems.org...
You have enabled multi-factor authentication. Please enter OTP code.
Code:   077236
Successfully registered gem: kettle-soup-cover (1.0.2)
Pushed kettle-soup-cover 1.0.2 to rubygems.org
❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

80971e7cd27a4d22b8a8d93e869e7a2543a21455e5ae0700b0321268f555cc81381bec88b024017e9e74fa4a0cbfa4bbfe2ad1db2f5fc0b77259906fce15198d
❯ bundle exec rake build
Enter PEM pass phrase:
kettle-soup-cover 1.0.2 built to pkg/kettle-soup-cover-1.0.2.gem.
❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

12354158c4a5c116aacb24cb16efd9b9a6240673c551042814b9f7e82d95de8d7cc87b5875a52e33bac660a5e182a581683af33cf7caef86930536d3006b2fe8

It seems that the only solution would be to run the checksums after release, but not running a new build of the gem package. The checksum commits would thus be after the release, and not in the git tag for the release. 😿

martinemde · 2023-10-20T04:38:02Z

Are you including the checksums files in the gem? Just a guess? (You would want to not do that.)

Is having the checksum in the tag even correct? How can a commit contain its own digest? Inception?

pboling · 2023-10-20T06:47:39Z

Are you including the checksums files in the gem?

Nope, I wondered the same thing, because if there were file changes between the gem builds, then they should have different checksums. Alas, no such luck.

Is having the checksum in the tag even correct?

I thought so, because the checksums are checksums of the built gem package. They have nothing to do with git.

How can a commit contain its own digest? Inception?

LOL, yeah it is a mind F8cker. It's really just that the gem package is wholly unrelated to the tracking of the project in git.

So the thing I don't understand is this - if the files are not changing, why does the checksum change everytime the gem is built? Either that's what is happening, or the packaged gem that results from rake build is fundamentally different than the packaged gem that results from rake release, and that lands somewhere between moderately and wildly disconcerting. 🤣 For a while I thought I was losing my mind, and then a pang of shame, realizing that every gem I have pushed, every checksum I have pushed does not match any of my releases, for the last year+. 🦺

I think I've found a bug. Because I assume that rake build and rake release should create identical packages, which would necessarily result in identical checksums.

pboling · 2023-10-20T06:51:10Z

To state as simply as possible:

Here's the repo I've been doing this evaluation in.

Somehow rake build and rake release create .gem packages that have different checksums. 😱

IMO, both tools are useful, but they must build identical packages...

segiddins · 2023-10-20T06:56:22Z

They do build identical packages, but with different timestamps unless you're setting the same SOURCE_DATE_EPOCH

pboling · 2023-10-20T06:58:43Z

Hero shit. ❤️ @segiddins. Thanks! I'll update to reflect that. 🔥

pboling · 2023-10-20T07:00:25Z

I think this should be prominently documented for both tasks, as for me, this is quite a violation of POLS. I'll work on PRs for that too. 🍔

Update

I haven't looked recently, so it if already is prominent, then I'm a victim of POLS assumptions. I may have never looked at the documentation for the tasks beyond the rake desc, because they seem extremely clearly named and obvious in function.

It seemed self-evident that both tasks should build an identical package, and I never considered timestamps, because that has never been on my radar... But this did cause me quite a problem, as I now have hundreds? of gems published with bad checksums, and bad release instructions. Since updated docs likely wouldn't have even helped me, I need to think about what a better solution might be.

pboling changed the title ~~Document rake dbuild:checksum~~ Document rake build:checksum Oct 27, 2022

deivid-rodriguez force-pushed the document-checksum branch from c1d04ac to dcc43c6 Compare June 16, 2023 13:55

deivid-rodriguez reviewed Jun 16, 2023

View reviewed changes

pboling force-pushed the document-checksum branch 2 times, most recently from acc0500 to f9be437 Compare October 10, 2023 03:55

segiddins approved these changes Oct 10, 2023

View reviewed changes

pboling commented Oct 10, 2023

View reviewed changes

security.md Outdated Show resolved Hide resolved

pboling force-pushed the document-checksum branch from f9be437 to 5bb4abd Compare October 10, 2023 17:54

martinemde reviewed Oct 10, 2023

View reviewed changes

🧑‍💻 Improved docs for checksums

2125037

pboling force-pushed the document-checksum branch from 5bb4abd to 2125037 Compare October 11, 2023 06:12

📝 Markdown, syntax, and grammar fixes

de6d7fe

pboling commented Oct 12, 2023

View reviewed changes

pboling added a commit to pboling/require_bench that referenced this pull request Oct 12, 2023

🔨 Update checksums script

26438f8

- rubygems/guides#325

📝 Explain usage of checksums

ae7a471

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document rake build:checksum #325

Document rake build:checksum #325

pboling commented Oct 27, 2022 •

edited

deivid-rodriguez Jun 16, 2023

pboling Jun 16, 2023

pboling Oct 10, 2023

pboling commented Oct 10, 2023

martinemde Oct 10, 2023

pboling Oct 10, 2023

pboling Oct 10, 2023

pboling Oct 11, 2023

pboling commented Oct 11, 2023

pboling Oct 12, 2023

pboling Oct 12, 2023

pboling Oct 12, 2023

pboling commented Oct 20, 2023

martinemde commented Oct 20, 2023 •

edited

pboling commented Oct 20, 2023 •

edited

pboling commented Oct 20, 2023 •

edited

segiddins commented Oct 20, 2023

pboling commented Oct 20, 2023

pboling commented Oct 20, 2023 •

edited


		require "digest/sha2"

		VERSION_REGEX = /\d+\.\d+\.\d+([-.].+)*/.freeze

Document rake build:checksum #325

Are you sure you want to change the base?

Document rake build:checksum #325

Conversation

pboling commented Oct 27, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pboling commented Oct 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pboling commented Oct 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pboling commented Oct 20, 2023

martinemde commented Oct 20, 2023 • edited

pboling commented Oct 20, 2023 • edited

pboling commented Oct 20, 2023 • edited

segiddins commented Oct 20, 2023

pboling commented Oct 20, 2023

pboling commented Oct 20, 2023 • edited

Update

pboling commented Oct 27, 2022 •

edited

martinemde commented Oct 20, 2023 •

edited

pboling commented Oct 20, 2023 •

edited

pboling commented Oct 20, 2023 •

edited

pboling commented Oct 20, 2023 •

edited