Generate SBOM during build/release process #3805

aabmass · 2024-03-27T14:55:52Z

I took a look at the actual API response

It says

Exact versions could not be resolved for some packages. For more information: https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#dependencies-included.

The relationships in the file look like
{ "relationshipType": "DEPENDS_ON", "spdxElementId": "SPDXRef-com.github.open-telemetry-opentelemetry-python", "relatedSpdxElement": "SPDXRef-pip-asgiref-3.7.2" }
My reading of this is "the opetnelemetry-python repo depends on X" rather than linking individual artifacts (e.g. opentelemetry-api) to their dependencies. I'm not sure how helpful that is for consumers of the SBOM.

I took a look at the actual API response

It says

Exact versions could not be resolved for some packages. For more information: docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#dependencies-included.

Right, I looked into the SBOM file, found several packages with an empty value for versionInfo, here they are:

{ "SPDXID": "SPDXRef-com.github.open-telemetry-opentelemetry-python", "name": "com.github.open-telemetry/opentelemetry-python", "versionInfo": "", "downloadLocation": "git+https://github.com/open-telemetry/opentelemetry-python", "licenseDeclared": "Apache-2.0", "filesAnalyzed": false, "supplier": "NOASSERTION", "externalRefs": [ { "referenceCategory": "PACKAGE-MANAGER", "referenceType": "purl", "referenceLocator": "pkg:github/open-telemetry/opentelemetry-python" } ] } { "SPDXID": "SPDXRef-pip-opencensus", "name": "pip:opencensus", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opencensus-ext-flask", "name": "pip:opencensus-ext-flask", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opentelemetry-api", "name": "pip:opentelemetry-api", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opentelemetry-exporter-jaeger", "name": "pip:opentelemetry-exporter-jaeger", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opentelemetry-instrumentation-sqlite3", "name": "pip:opentelemetry-instrumentation-sqlite3", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opentelemetry-opencensus-shim", "name": "pip:opentelemetry-opencensus-shim", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opentelemetry-sdk", "name": "pip:opentelemetry-sdk", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-opentelemetry-opentracing-shim", "name": "pip:opentelemetry-opentracing-shim", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-redis", "name": "pip:redis", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-redis-opentracing", "name": "pip:redis-opentracing", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-googleapis-common-protos", "name": "pip:googleapis-common-protos", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-requests", "name": "pip:requests", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" } { "SPDXID": "SPDXRef-pip-asgiref", "name": "pip:asgiref", "versionInfo": "", "downloadLocation": "NOASSERTION", "filesAnalyzed": false, "supplier": "NOASSERTION" }

aabmass · 2024-03-27T14:54:44Z

This API seems pretty barebones, I'm guessing this just generates an SBOM against the main branch and maybe cached at some random commit?

What if you are making a patch release, would the dep graph be correct?
If it is looking at main branch, I think our release process would show the .dev0 dependencies instead of actual released versions

This API seems pretty barebones, I'm guessing this just generates an SBOM against the main branch and maybe cached at some random commit?

No, apparently this functionality creates an SBOM file using the latest commit in main. There is a way to generate this file using the web interface (https://github.com/open-telemetry/opentelemetry-python/three dots button in the upper right corner/Insights/Dependency graph/Export SBOM). This creates a file that includes the latest commit from main in its name. I assume the same file gets generated when the REST API is being used.

What if you are making a patch release, would the dep graph be correct? If it is looking at main branch, I think our release process would show the .dev0 dependencies instead of actual released versions

The reason why the SBOM file includes released versions is because we have requirements.txt-like files that include released versions. For example, I made these changes, and pushed them to main:

diff --git a/docs/examples/metrics/reader/requirements.txt b/docs/examples/metrics/reader/requirements.txt index 2ccffaf3..23ff5bf6 100644 --- a/docs/examples/metrics/reader/requirements.txt +++ b/docs/examples/metrics/reader/requirements.txt @@ -1,5 +1,5 @@ Deprecated==1.2.13 -opentelemetry-api==1.15.0 +opentelemetry-api==1.20.0 opentelemetry-sdk==1.15.0 opentelemetry-semantic-conventions==0.36b0 typing_extensions==4.3.0 diff --git a/docs/examples/metrics/views/requirements.txt b/docs/examples/metrics/views/requirements.txt index be612711..d9590b3f 100644 --- a/docs/examples/metrics/views/requirements.txt +++ b/docs/examples/metrics/views/requirements.txt @@ -1,5 +1,5 @@ Deprecated==1.2.13 -opentelemetry-api==1.12.0 +opentelemetry-api==1.20.0 opentelemetry-sdk==1.12.0 opentelemetry-semantic-conventions==0.33b0 typing_extensions==4.3.0

Those changes produced the following changes in the SBOM file:

5c5 < "created": "2024-04-03T19:17:07Z", --- > "created": "2024-04-03T17:15:55Z", 16c16 < "documentNamespace": "https://github.com/SecuringCarter/opentelemetry-python/dependency_graph/sbom-7c05aa1b812baa89", --- > "documentNamespace": "https://github.com/SecuringCarter/opentelemetry-python/dependency_graph/sbom-75a76526c2dd97b4", 814a815,830 > "SPDXID": "SPDXRef-pip-opentelemetry-api-1.15.0", > "name": "pip:opentelemetry-api", > "versionInfo": "1.15.0", > "downloadLocation": "NOASSERTION", > "filesAnalyzed": false, > "licenseConcluded": "Apache-2.0", > "supplier": "NOASSERTION", > "externalRefs": [ > { > "referenceCategory": "PACKAGE-MANAGER", > "referenceLocator": "pkg:pypi/[email protected]", > "referenceType": "purl" > } > ] > }, > { 878a895,910 > "SPDXID": "SPDXRef-pip-opentelemetry-api-1.12.0", > "name": "pip:opentelemetry-api", > "versionInfo": "1.12.0", > "downloadLocation": "NOASSERTION", > "filesAnalyzed": false, > "licenseConcluded": "Apache-2.0", > "supplier": "NOASSERTION", > "externalRefs": [ > { > "referenceCategory": "PACKAGE-MANAGER", > "referenceLocator": "pkg:pypi/[email protected]", > "referenceType": "purl" > } > ] > }, > { 2317a2350,2354 > "relatedSpdxElement": "SPDXRef-pip-opentelemetry-api-1.15.0" > }, > { > "relationshipType": "DEPENDS_ON", > "spdxElementId": "SPDXRef-com.github.SecuringCarter-opentelemetry-python", 2333a2371,2375 > }, > { > "relationshipType": "DEPENDS_ON", > "spdxElementId": "SPDXRef-com.github.SecuringCarter-opentelemetry-python", > "relatedSpdxElement": "SPDXRef-pip-opentelemetry-api-1.12.0"

So, I now think that we probably should do something about these requirements-txt files that do not actually represent project dependencies.

After many more changes I managed to reduce the amount of packages in the SBOM file with an empty value of versionInfo to only one:

{ "SPDXID": "SPDXRef-com.github.SecuringCarter-opentelemetry-python", "name": "com.github.SecuringCarter/opentelemetry-python", "versionInfo": "", "downloadLocation": "git+https://github.com/SecuringCarter/opentelemetry-python", "licenseDeclared": "Apache-2.0", "filesAnalyzed": false, "supplier": "NOASSERTION", "externalRefs": [ { "referenceCategory": "PACKAGE-MANAGER", "referenceType": "purl", "referenceLocator": "pkg:github/SecuringCarter/opentelemetry-python" } ] }

I think the mechanism to generate an SBOM file that this PR introduces is valid. We probably need to change our repo in order to produce a "better" or more useful SBOM file.

Can we merge this and work on making the SBOM file better in subsequent PRs?

-Original file line number
+Diff line change
@@ -0,0 +1,20 @@
+from json import dumps
+from requests import get
+with open("opentelemetry-python.spdx.json", "w") as sbom_file:
+ sbom_file.write(
+ dumps(
+ get(
+ (
+ "https://api.github.com/repos/open-telemetry/"
+ "opentelemetry-python/dependency-graph/sbom"
+ ),
+ headers={
+ "Accept": "application/vnd.github+json",
+ "X-GitHub-Api-Version": "2022-11-28",
+ },
+ ).json(),
+ indent=4,
+ )
+ )

-Original file line number
+Diff line change
@@ -0,0 +1,55 @@
+name: SBOM
+on:
+ release:
+ types: [published]
+permissions: read-all
+jobs:
+ generate-sbom:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout core repo
+ uses: actions/checkout@v4
+ - name: Setup Python
+ uses: actions/setup-python@v5
+ with:
+ python-version: '3.10'
+ architecture: 'x64'
+ - name: Install requests
+ run: pip install requests
+ - name: Generate SBOM
+ run: python3 .github/workflows/generate_sbom.py
+ - name: Zip the SBOM file
+ run: zip sbom.zip opentelemetry-python.spdx.json
+ - name: Upload artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: SBOM.zip
+ path: ./sbom.zip
+ add-release-artifact:
+ needs: generate-sbom
+ runs-on: ubuntu-latest
+ permissions:
+ contents: write
+ steps:
+ - name: Download artifact from generate-sboms
+ uses: actions/download-artifact@v4
+ with:
+ name: SBOM.zip
+ - name: Upload release asset
+ uses: actions/upload-release-asset@v1
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ with:
+ upload_url: ${{ github.event.release.upload_url }}
+ asset_path: ./sbom.zip
+ asset_name: SBOM.zip
+ asset_content_type: application/zip

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate SBOM during build/release process #3805

Diff view

Diff view

There are no files selected for viewing

aabmass Mar 27, 2024

ocelotl Apr 2, 2024

aabmass Mar 27, 2024 •

edited by ocelotl

ocelotl Apr 3, 2024

ocelotl Apr 3, 2024

ocelotl Apr 3, 2024

Generate SBOM during build/release process #3805

Generate SBOM during build/release process #3805

Diff view

Diff view

There are no files selected for viewing

aabmass Mar 27, 2024

Choose a reason for hiding this comment

ocelotl Apr 2, 2024

Choose a reason for hiding this comment

aabmass Mar 27, 2024 • edited by ocelotl

Choose a reason for hiding this comment

ocelotl Apr 3, 2024

Choose a reason for hiding this comment

ocelotl Apr 3, 2024

Choose a reason for hiding this comment

ocelotl Apr 3, 2024

Choose a reason for hiding this comment

aabmass Mar 27, 2024 •

edited by ocelotl