Clearly defined certifier #2035

pxp928 · 2024-07-18T16:49:47Z

Description of the PR

Adds a new clearly defined certifier/guesser/processor/parser with unit tests.
Update to OSV and CD to use the new attestation headers with resource descriptor
adds ability to query CD on ingestion (default to false)

closes #1964

PR Checklist

All commits have a Developer Certificate of Origin (DCO) -- they are generated using -s flag to git commit.
All new changes are covered by tests
If GraphQL schema is changed, make generate has been run
If GraphQL schema is changed, GraphQL client updates/additions have been made
If OpenAPI spec is changed, make generate has been run
If ent schema is changed, make generate has been run
If collectsub protobuf has been changed, make proto has been run
All CI checks are passing (tests and formatting)
All dependent PRs have already been merged

Signed-off-by: pxp928 <[email protected]>

…ill need to be added Signed-off-by: pxp928 <[email protected]>

Signed-off-by: pxp928 <[email protected]>

…gestion error Signed-off-by: pxp928 <[email protected]>

Signed-off-by: pxp928 <[email protected]>

…alues ingestor is run as a service Signed-off-by: pxp928 <[email protected]>

Signed-off-by: pxp928 <[email protected]>

pkg/certifier/attestation/attestation_vuln.go

pkg/handler/processor/guesser/type_ite6.go

pkg/ingestor/parser/cyclonedx/parser_cyclonedx.go

pxp928 · 2024-07-18T20:47:49Z

FYI @nickvidal

Signed-off-by: pxp928 <[email protected]>

jeffmendoza · 2024-07-19T19:03:21Z

A few things (not blocking this pr)

We should post the purl -> coordinate code to the CD discord and ask for comments.
Test the parser for a not harvested component. This should give "NOASSERTION"
We should consider weather to save "NOASSERTION" or not, and/or to skip results that are not harvested, this should be discernible based on the "tools" section under "described" but there might be a better way. Again, something we can ask about in discord.

I'm excited to test this out with real data!

nickvidal · 2024-07-19T19:03:28Z

Thank you @pxp928 and @jeffmendoza!

nickvidal · 2024-07-19T19:07:56Z

@jeffmendoza wrt NOASSERTION, you might be interested in this recent development: https://opensource.org/blog/beyond-spdx-expanding-licenses-identified-by-clearlydefined

mihaimaruseac

Thank you!

SantiagoTorres · 2024-07-25T19:21:18Z

cmd/guacone/cmd/oci.go

@@ -88,7 +90,7 @@ var ociCmd = &cobra.Command{
 		// Set emit function to go through the entire pipeline
 		emit := func(d *processor.Document) error {
 			totalNum += 1
-			err := ingestor.Ingest(ctx, d, opts.graphqlEndpoint, transport, csubClient, opts.queryVulnOnIngestion)
+			err := ingestor.Ingest(ctx, d, opts.graphqlEndpoint, transport, csubClient, opts.queryVulnOnIngestion, opts.queryLicenseOnIngestion)


I'm seeing this pattern repeated throughout of adding booleans for some behavior changes. Would this be a good time to change the function signature to take opts as a single parameter instead of each bool separately?

yes great point. We can clean this up in a future pr

SantiagoTorres · 2024-07-25T19:22:28Z

cmd/guacone/cmd/s3.go

+	mp                      string // message provider name (sqs or kafka, will default to kafka)
+	mpEndpoint              string // endpoint for the message provider (only for polling behaviour)
+	poll                    bool   // polling or non-polling behaviour? (defaults to non-polling)
+	graphqlEndpoint         string // endpoint for the graphql server


I wonder if we could contain some of the fields below as a separate struct that's used across all different cmds to have a SSoT

SantiagoTorres · 2024-07-25T19:30:37Z

pkg/ingestor/parser/clearlydefined/clearlydefined_test.go

@@ -0,0 +1,156 @@
+//
+// Copyright 2022 The GUAC Authors.


do we care about copyright years? (I see some here are 2024, some are 2022)

ah, I thought I caught this...can be fixed in another PR.

pxp928 added 18 commits July 17, 2024 14:00

add new clearly defined certifier with attestation

f32f133

Signed-off-by: pxp928 <[email protected]>

add clearly definied example for tests and update definition

49f4dcb

Signed-off-by: pxp928 <[email protected]>

add new WIP clearly defined parser

cdc2dce

Signed-off-by: pxp928 <[email protected]>

updates clearlydefined parser

2f66ec5

Signed-off-by: pxp928 <[email protected]>

updates to the parser for parsing legal data

92d9cdc

Signed-off-by: pxp928 <[email protected]>

update cd certifier to add soruce query, wip cd parser, unit tests st…

a6ecc8f

…ill need to be added Signed-off-by: pxp928 <[email protected]>

updates to the certifier unit tests

70a8be8

Signed-off-by: pxp928 <[email protected]>

add helper function to convert purl to coordinate

08819f2

Signed-off-by: pxp928 <[email protected]>

update certifier to use new helper function

ae5b153

Signed-off-by: pxp928 <[email protected]>

update attestation header to use resource descriptor

fd447e9

Signed-off-by: pxp928 <[email protected]>

update cd certifier unit tests

6a99b58

Signed-off-by: pxp928 <[email protected]>

add srcId to source helper function with unit tests

85e5962

Signed-off-by: pxp928 <[email protected]>

update cd parser, unit tests needed

55eb79d

Signed-off-by: pxp928 <[email protected]>

update cd unit test parser

703178a

Signed-off-by: pxp928 <[email protected]>

add guacone license to run cd certifier, update processor and bulk in…

6a37462

…gestion error Signed-off-by: pxp928 <[email protected]>

update guaccollect with cd certifier

1931770

Signed-off-by: pxp928 <[email protected]>

add ability to ingest license information during ingestion

195bfc5

Signed-off-by: pxp928 <[email protected]>

add tests for scanner and add initalization for parser to clean out v…

e182ed5

…alues ingestor is run as a service Signed-off-by: pxp928 <[email protected]>

pxp928 requested review from mihaimaruseac and jeffmendoza as code owners July 18, 2024 16:49

pull-request-size bot added the size/XXL label Jul 18, 2024

pxp928 force-pushed the clearlyDefined-certifier branch 2 times, most recently from 397d932 to e182ed5 Compare July 18, 2024 20:05

fix failing unit tests

3127ab3

Signed-off-by: pxp928 <[email protected]>

pxp928 commented Jul 18, 2024

View reviewed changes

pkg/certifier/attestation/attestation_vuln.go Show resolved Hide resolved

pkg/handler/processor/guesser/type_ite6.go Show resolved Hide resolved

pkg/ingestor/parser/cyclonedx/parser_cyclonedx.go Show resolved Hide resolved

change cd logs to be debug

b63240d

Signed-off-by: pxp928 <[email protected]>

jeffmendoza approved these changes Jul 19, 2024

View reviewed changes

pxp928 requested a review from lumjjb July 24, 2024 16:33

mihaimaruseac approved these changes Jul 25, 2024

View reviewed changes

kodiakhq bot merged commit a0762a6 into guacsec:main Jul 25, 2024
8 checks passed

SantiagoTorres reviewed Jul 25, 2024

View reviewed changes

pxp928 deleted the clearlyDefined-certifier branch July 25, 2024 19:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clearly defined certifier #2035

Clearly defined certifier #2035

pxp928 commented Jul 18, 2024 •

edited

Loading

pxp928 commented Jul 18, 2024

jeffmendoza commented Jul 19, 2024 •

edited

Loading

nickvidal commented Jul 19, 2024

nickvidal commented Jul 19, 2024

mihaimaruseac left a comment

SantiagoTorres Jul 25, 2024

pxp928 Jul 25, 2024

SantiagoTorres Jul 25, 2024

SantiagoTorres Jul 25, 2024

pxp928 Jul 25, 2024

Clearly defined certifier #2035

Clearly defined certifier #2035

Conversation

pxp928 commented Jul 18, 2024 • edited Loading

Description of the PR

PR Checklist

pxp928 commented Jul 18, 2024

jeffmendoza commented Jul 19, 2024 • edited Loading

nickvidal commented Jul 19, 2024

nickvidal commented Jul 19, 2024

mihaimaruseac left a comment

Choose a reason for hiding this comment

SantiagoTorres Jul 25, 2024

Choose a reason for hiding this comment

pxp928 Jul 25, 2024

Choose a reason for hiding this comment

SantiagoTorres Jul 25, 2024

Choose a reason for hiding this comment

SantiagoTorres Jul 25, 2024

Choose a reason for hiding this comment

pxp928 Jul 25, 2024

Choose a reason for hiding this comment

pxp928 commented Jul 18, 2024 •

edited

Loading

jeffmendoza commented Jul 19, 2024 •

edited

Loading