-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat] - Optimize detector performance by reducing data passed to regex #2812
Conversation
While I think this is a good idea, it would be a mistake to set a default max length. I can think of several detectors that can easily exceed this — JWT, private key, GCP, and Docker (#2677) to name a few. Because there are hundreds of detectors, the safer approach would be to make this opt-in. This would also interfere with detectors that require multiple parts (e.g., client ID & secret, username & password, secret & URL). Edit: Incidentally, this would likely solve #2739. |
@@ -36,7 +36,7 @@ func TestAlchemy_Pattern(t *testing.T) { | |||
for _, test := range tests { | |||
t.Run(test.name, func(t *testing.T) { | |||
chunkSpecificDetectors := make(map[ahocorasick.DetectorKey]detectors.Detector, 2) | |||
ahoCorasickCore.PopulateMatchingDetectors(test.input, chunkSpecificDetectors) | |||
ahoCorasickCore.FindDetectorMatches(test.input, chunkSpecificDetectors) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this would compile anymore since the signature is now:
FindDetectorMatches(chunkData string) []DetectorMatch
It would need to be changed to:
-chunkSpecificDetectors := make(map[ahocorasick.DetectorKey]detectors.Detector, 2)
-ahoCorasickCore.FindDetectorMatches(test.input, chunkSpecificDetectors
+chunkSpecificDetectors := ahoCorasickCore.FindDetectorMatches(test.input)
Incidentally, I think the TestX_Pattern
should be put into a common test module instead of copied & pasted between tests.
pkg/detectors/detectors.go
Outdated
// MaxSecretSizeProvider is an optional interface that a detector can implement to | ||
// provide a custom max size for the secret it finds. | ||
type MaxSecretSizeProvider interface { | ||
ProvideMaxSecretSize() int64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about just MaxSecretSize
?
pkg/engine/engine.go
Outdated
matchedBytes := data.detector.Matches(data.chunk.Data) | ||
for _, match := range matchedBytes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does mergeMatches
aide in de-duplication at all? Some of my recent changes have been around de-duplicating results in chunks to prevent making the same network calls 2/10/20 times. Admittedly, a better solution would be #2262 rather than caching matches in a given chunk.
pkg/engine/engine_test.go
Outdated
for _, chunkSize := range chunkSizes { | ||
b.Run(fmt.Sprintf("ChunkSize_%d", chunkSize), func(b *testing.B) { | ||
b.ReportAllocs() | ||
b.SetBytes(int64(dataSize)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update this to chunkSize to update the throughput on the benchmarks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can increase the default to 1024 bytes for the multi-part credential case. Should be adequate in most cases. I'd like to see this optimization on by default, but we can provide an opt-out flag. |
1024 bytes would be safer, but could definitely still miss valid secrets. I think there'd be tremendous value in at least having a (hidden?) flag that runs both and checks for missed results (kind of like https://github.com/github/scientist), rather than binary on/off. Otherwise it can be tricky to identify affected detectors, which has been an issue with the verification overlap change. |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great improvement!
* Detectors beginning w/ a * Detectors beginning w/ b * Detectors beginning w/ c * Detectors beginning w/ d * Detectors beginning w/ e * Detectors beginning w/ f * Detectors beginning w/ f&g * fix * Detectors beginning w/ i-l * Detectors beginning w/ m-p * Detectors beginning w/ r-s * Detectors beginning w/ t * Detectors beginning w/ u-z * revert alconst * remaining fixes * lint
* Detector comparison mode * remove else * return error if results dont match * update default hidden flag to not scan entire chunks
… merging and extracting
…ex (trufflesecurity#2812) * optimize maching detetors * update method name * updates * update naming * updates * update comment * updates * remove testcase * update default match len to 512 * update * update test * add support for multpart cred provider * add ability to scan entire chunk * encapsulate matches logic within FindDetectorMatches * use []byte directly * nil chunk data * use []byte * set hidden flag to true * remove * [refactor] - multi part detectors (trufflesecurity#2906) * Detectors beginning w/ a * Detectors beginning w/ b * Detectors beginning w/ c * Detectors beginning w/ d * Detectors beginning w/ e * Detectors beginning w/ f * Detectors beginning w/ f&g * fix * Detectors beginning w/ i-l * Detectors beginning w/ m-p * Detectors beginning w/ r-s * Detectors beginning w/ t * Detectors beginning w/ u-z * revert alconst * remaining fixes * lint * [feat] - Add Support for `compareDetectionStrategies` Mode (trufflesecurity#2918) * Detector comparison mode * remove else * return error if results dont match * update default hidden flag to not scan entire chunks * fix tests * enhance encapsulation by including methods on DetectorMatch to handle merging and extracting * remove space * fix * update detector * updates * remove else * run comparison concurrently
…ex (trufflesecurity#2812) * optimize maching detetors * update method name * updates * update naming * updates * update comment * updates * remove testcase * update default match len to 512 * update * update test * add support for multpart cred provider * add ability to scan entire chunk * encapsulate matches logic within FindDetectorMatches * use []byte directly * nil chunk data * use []byte * set hidden flag to true * remove * [refactor] - multi part detectors (trufflesecurity#2906) * Detectors beginning w/ a * Detectors beginning w/ b * Detectors beginning w/ c * Detectors beginning w/ d * Detectors beginning w/ e * Detectors beginning w/ f * Detectors beginning w/ f&g * fix * Detectors beginning w/ i-l * Detectors beginning w/ m-p * Detectors beginning w/ r-s * Detectors beginning w/ t * Detectors beginning w/ u-z * revert alconst * remaining fixes * lint * [feat] - Add Support for `compareDetectionStrategies` Mode (trufflesecurity#2918) * Detector comparison mode * remove else * return error if results dont match * update default hidden flag to not scan entire chunks * fix tests * enhance encapsulation by including methods on DetectorMatch to handle merging and extracting * remove space * fix * update detector * updates * remove else * run comparison concurrently
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [trufflesecurity/trufflehog](https://togithub.com/trufflesecurity/trufflehog) | action | patch | `v3.78.0` -> `v3.78.1` | --- ### Release Notes <details> <summary>trufflesecurity/trufflehog (trufflesecurity/trufflehog)</summary> ### [`v3.78.1`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.78.1) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.78.0...v3.78.1) #### What's Changed - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.15 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2911](https://togithub.com/trufflesecurity/trufflehog/pull/2911) - fix(deps): update module github.com/microsoft/go-mssqldb to v1.7.2 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2912](https://togithub.com/trufflesecurity/trufflehog/pull/2912) - Add elasticsearch to tui by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/2915](https://togithub.com/trufflesecurity/trufflehog/pull/2915) - Improve GitHub wiki scan errs by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2917](https://togithub.com/trufflesecurity/trufflehog/pull/2917) - Update OpenAI detector by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2863](https://togithub.com/trufflesecurity/trufflehog/pull/2863) - Add flag to get information if trufflehog being ran from TUI by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/1644](https://togithub.com/trufflesecurity/trufflehog/pull/1644) - \[feat] - Introduce `channelmetrics` Package for Channel Metrics Collection by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2889](https://togithub.com/trufflesecurity/trufflehog/pull/2889) - \[feat] - Optimize detector performance by reducing data passed to regex by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2812](https://togithub.com/trufflesecurity/trufflehog/pull/2812) - Go should be installed before codeql initializes by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/2919](https://togithub.com/trufflesecurity/trufflehog/pull/2919) - \[fix] - continue on error by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2921](https://togithub.com/trufflesecurity/trufflehog/pull/2921) - Update Jenkins in tui by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/2925](https://togithub.com/trufflesecurity/trufflehog/pull/2925) - \[chore] - remove stutter in naming by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2926](https://togithub.com/trufflesecurity/trufflehog/pull/2926) - \[fix] - Correctly calculate EntireSpanChunkCalculator span by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2924](https://togithub.com/trufflesecurity/trufflehog/pull/2924) - Improve Git scaning logs by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2923](https://togithub.com/trufflesecurity/trufflehog/pull/2923) - \[chore] - address comments by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2920](https://togithub.com/trufflesecurity/trufflehog/pull/2920) - Make `cache.Cache` typed by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2930](https://togithub.com/trufflesecurity/trufflehog/pull/2930) - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.17 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2914](https://togithub.com/trufflesecurity/trufflehog/pull/2914) - \[chore] Polish channelmetrics package by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/2938](https://togithub.com/trufflesecurity/trufflehog/pull/2938) - Add `*.dia` to ignored extensions list by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2939](https://togithub.com/trufflesecurity/trufflehog/pull/2939) - Make the github action work with a path as input by [@​benbridts](https://togithub.com/benbridts) in [https://github.com/trufflesecurity/trufflehog/pull/2908](https://togithub.com/trufflesecurity/trufflehog/pull/2908) - fix(deps): update module github.com/snowflakedb/gosnowflake to v1.10.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2913](https://togithub.com/trufflesecurity/trufflehog/pull/2913) - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.19 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2944](https://togithub.com/trufflesecurity/trufflehog/pull/2944) - fix(deps): update module github.com/launchdarkly/go-server-sdk/v7 to v7.4.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2947](https://togithub.com/trufflesecurity/trufflehog/pull/2947) - \[bug] - Ensure BufferedFileWriter Flushes Buffer Contents to File Correctly by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2943](https://togithub.com/trufflesecurity/trufflehog/pull/2943) - Change filesystem symlink err handling by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2941](https://togithub.com/trufflesecurity/trufflehog/pull/2941) - Fix panic in MaxMind detector by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2948](https://togithub.com/trufflesecurity/trufflehog/pull/2948) - \[chore] - Update `discordwebhook` detector keyword by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2954](https://togithub.com/trufflesecurity/trufflehog/pull/2954) - \[fix] - Refactor Filtering Logic to Fix Known False Positive Handling in Overlapping Cases by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2946](https://togithub.com/trufflesecurity/trufflehog/pull/2946) - \[feat] - Update span calculation logic to use offset magnitude by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2957](https://togithub.com/trufflesecurity/trufflehog/pull/2957) - \[chore] - pin archiver dependency by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2958](https://togithub.com/trufflesecurity/trufflehog/pull/2958) - \[chore] - Remove replace by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2959](https://togithub.com/trufflesecurity/trufflehog/pull/2959) #### New Contributors - [@​benbridts](https://togithub.com/benbridts) made their first contribution in [https://github.com/trufflesecurity/trufflehog/pull/2908](https://togithub.com/trufflesecurity/trufflehog/pull/2908) **Full Changelog**: trufflesecurity/trufflehog@v3.78.0...v3.78.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/matter-labs/vault-auth-tee). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [amannn/action-semantic-pull-request](https://togithub.com/amannn/action-semantic-pull-request) | action | patch | `v5.5.2` -> `v5.5.3` | | [trufflesecurity/trufflehog](https://togithub.com/trufflesecurity/trufflehog) | action | minor | `v3.78.0` -> `v3.81.5` | --- ### Release Notes <details> <summary>amannn/action-semantic-pull-request (amannn/action-semantic-pull-request)</summary> ### [`v5.5.3`](https://togithub.com/amannn/action-semantic-pull-request/releases/tag/v5.5.3) [Compare Source](https://togithub.com/amannn/action-semantic-pull-request/compare/v5.5.2...v5.5.3) ##### Bug Fixes - Bump `braces` dependency ([#​269](https://togithub.com/amannn/action-semantic-pull-request/issues/269). by [@​EelcoLos](https://togithub.com/EelcoLos)) ([2d952a1](https://togithub.com/amannn/action-semantic-pull-request/commit/2d952a1bf90a6a7ab8f0293dc86f5fdf9acb1915)) </details> <details> <summary>trufflesecurity/trufflehog (trufflesecurity/trufflehog)</summary> ### [`v3.81.5`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.81.5) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.81.4...v3.81.5) #### What's Changed - Update README.md for github experimental by [@​joeleonjr](https://togithub.com/joeleonjr) in [https://github.com/trufflesecurity/trufflehog/pull/3160](https://togithub.com/trufflesecurity/trufflehog/pull/3160) - fix(deps): update module github.com/schollz/progressbar/v3 to v3.14.6 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3158](https://togithub.com/trufflesecurity/trufflehog/pull/3158) - \[analyze] Fix off-by-one error in generated data structures by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3162](https://togithub.com/trufflesecurity/trufflehog/pull/3162) - \[bug] - Create a new context with timeout per request by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3163](https://togithub.com/trufflesecurity/trufflehog/pull/3163) - \[analyze] Use permission enum values in openai analyzer by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3165](https://togithub.com/trufflesecurity/trufflehog/pull/3165) - update pattern by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3167](https://togithub.com/trufflesecurity/trufflehog/pull/3167) - Update Zulip detector by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2897](https://togithub.com/trufflesecurity/trufflehog/pull/2897) - fix(deps): update module golang.org/x/oauth2 to v0.22.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3168](https://togithub.com/trufflesecurity/trufflehog/pull/3168) - fix(deps): update module golang.org/x/sync to v0.8.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3169](https://togithub.com/trufflesecurity/trufflehog/pull/3169) - fix(deps): update github.com/tailscale/depaware digest to [`585336c`](https://togithub.com/trufflesecurity/trufflehog/commit/585336c) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3166](https://togithub.com/trufflesecurity/trufflehog/pull/3166) - Change log verbosity for detection errors by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3171](https://togithub.com/trufflesecurity/trufflehog/pull/3171) **Full Changelog**: trufflesecurity/trufflehog@v3.81.4...v3.81.5 ### [`v3.81.4`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.81.4) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.81.3...v3.81.4) #### What's Changed - \[bug] - add context timeout to ssh verification by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3161](https://togithub.com/trufflesecurity/trufflehog/pull/3161) **Full Changelog**: trufflesecurity/trufflehog@v3.81.3...v3.81.4 ### [`v3.81.3`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.81.3) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.81.2...v3.81.3) #### What's Changed - \[chore] - log detector type on error by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3159](https://togithub.com/trufflesecurity/trufflehog/pull/3159) **Full Changelog**: trufflesecurity/trufflehog@v3.81.2...v3.81.3 ### [`v3.81.2`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.81.2) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.81.1...v3.81.2) #### What's Changed - \[chore] - set custom transport for the Docker client by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3156](https://togithub.com/trufflesecurity/trufflehog/pull/3156) **Full Changelog**: trufflesecurity/trufflehog@v3.81.1...v3.81.2 ### [`v3.81.1`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.81.1) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.81.0...v3.81.1) #### What's Changed - \[chore] - enable block and mutex profiles by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3154](https://togithub.com/trufflesecurity/trufflehog/pull/3154) - Add Analyzers interface for Square by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3141](https://togithub.com/trufflesecurity/trufflehog/pull/3141) - Update module google.golang.org/api to v0.190.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3146](https://togithub.com/trufflesecurity/trufflehog/pull/3146) - quick patch for cfor enumeration by [@​joeleonjr](https://togithub.com/joeleonjr) in [https://github.com/trufflesecurity/trufflehog/pull/3155](https://togithub.com/trufflesecurity/trufflehog/pull/3155) - Add Analyzers interface for HuggingFace by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3140](https://togithub.com/trufflesecurity/trufflehog/pull/3140) **Full Changelog**: trufflesecurity/trufflehog@v3.81.0...v3.81.1 ### [`v3.81.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.81.0) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.80.6...v3.81.0) #### What's Changed - Add progress bar to CFOR by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3151](https://togithub.com/trufflesecurity/trufflehog/pull/3151) - \[fix] Always configure the engine with the default detectors by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3152](https://togithub.com/trufflesecurity/trufflehog/pull/3152) **Full Changelog**: trufflesecurity/trufflehog@v3.80.6...v3.81.0 ### [`v3.80.6`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.80.6) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.80.5...v3.80.6) #### What's Changed - Add Analyze interface to Stripe by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3132](https://togithub.com/trufflesecurity/trufflehog/pull/3132) - \[analyze] Combine access level into permission value by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3144](https://togithub.com/trufflesecurity/trufflehog/pull/3144) - \[chore] - move automaxprocs to init by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3143](https://togithub.com/trufflesecurity/trufflehog/pull/3143) - add twilio analyze relationships by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3148](https://togithub.com/trufflesecurity/trufflehog/pull/3148) - \[chore] Only set default detectors if none are provided by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3147](https://togithub.com/trufflesecurity/trufflehog/pull/3147) - CFOR Commit Scanner by [@​joeleonjr](https://togithub.com/joeleonjr) in [https://github.com/trufflesecurity/trufflehog/pull/3145](https://togithub.com/trufflesecurity/trufflehog/pull/3145) - \[perf] - Leverage pgzip for Parallel decompression by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3149](https://togithub.com/trufflesecurity/trufflehog/pull/3149) **Full Changelog**: trufflesecurity/trufflehog@v3.80.5...v3.80.6 ### [`v3.80.5`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.80.5) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/3.80.4...v3.80.5) #### What's Changed - Add permissions lookup tables by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3125](https://togithub.com/trufflesecurity/trufflehog/pull/3125) - Export maps from permission generation by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/3137](https://togithub.com/trufflesecurity/trufflehog/pull/3137) - \[chore] - Set GOMAXPROCS by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3136](https://togithub.com/trufflesecurity/trufflehog/pull/3136) - \[chore] - address linter by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3133](https://togithub.com/trufflesecurity/trufflehog/pull/3133) - \[refactor] - Improve Performance by Shifting Concurrency from Image to Layer Level by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3135](https://togithub.com/trufflesecurity/trufflehog/pull/3135) **Full Changelog**: trufflesecurity/trufflehog@3.80.4...v3.80.5 ### [`v3.80.4`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/3.80.4) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.80.3...3.80.4) #### What's Changed - Analyzer partial implementations by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3114](https://togithub.com/trufflesecurity/trufflehog/pull/3114) - fix(deps): update module github.com/aws/aws-sdk-go to v1.55.5 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3116](https://togithub.com/trufflesecurity/trufflehog/pull/3116) - Separate out printing statements with anlayzer logic for Shopify by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3123](https://togithub.com/trufflesecurity/trufflehog/pull/3123) - Separate out printing statements with anlayzer logic for Square by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3122](https://togithub.com/trufflesecurity/trufflehog/pull/3122) - Separate out printing statements with anlayzer logic for twilio by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3118](https://togithub.com/trufflesecurity/trufflehog/pull/3118) - Add new canary ID by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3117](https://togithub.com/trufflesecurity/trufflehog/pull/3117) - Update GitHub integration tests by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/3124](https://togithub.com/trufflesecurity/trufflehog/pull/3124) - Separate out printing statements with anlayzer logic for Slack by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3121](https://togithub.com/trufflesecurity/trufflehog/pull/3121) - Separate out printing statements with anlayzer logic for Stripe by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3120](https://togithub.com/trufflesecurity/trufflehog/pull/3120) - nitro detector was removed and needs to be deprecated by [@​0x1](https://togithub.com/0x1) in [https://github.com/trufflesecurity/trufflehog/pull/3102](https://togithub.com/trufflesecurity/trufflehog/pull/3102) - Separate out printing statements with anlayzer logic for SourceGraph by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3119](https://togithub.com/trufflesecurity/trufflehog/pull/3119) **Full Changelog**: trufflesecurity/trufflehog@v3.80.3...3.80.4 ### [`v3.80.3`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.80.3) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.80.2...v3.80.3) #### What's Changed - fix(deps): update module github.com/gabriel-vasile/mimetype to v1.4.5 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3108](https://togithub.com/trufflesecurity/trufflehog/pull/3108) - \[chore] Move openai log message to proper function by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3105](https://togithub.com/trufflesecurity/trufflehog/pull/3105) - fix(deps): update module github.com/aws/aws-sdk-go to v1.55.3 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3107](https://togithub.com/trufflesecurity/trufflehog/pull/3107) - \[analyze] Implement Analyzer interface for github by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3110](https://togithub.com/trufflesecurity/trufflehog/pull/3110) - Support openai project and fine grained tokens by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/3112](https://togithub.com/trufflesecurity/trufflehog/pull/3112) - \[analyze] Add description and user to openai metadata by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3111](https://togithub.com/trufflesecurity/trufflehog/pull/3111) - \[chore] - Manually update Depedencies by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3106](https://togithub.com/trufflesecurity/trufflehog/pull/3106) - Use non-canary credentials for AWS tests by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/3109](https://togithub.com/trufflesecurity/trufflehog/pull/3109) - Include default detectors when using a config that contains detectors by [@​harmonherring-pro](https://togithub.com/harmonherring-pro) in [https://github.com/trufflesecurity/trufflehog/pull/3115](https://togithub.com/trufflesecurity/trufflehog/pull/3115) **Full Changelog**: trufflesecurity/trufflehog@v3.80.2...v3.80.3 ### [`v3.80.2`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.80.2) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.80.1...v3.80.2) #### What's Changed - Added Twitter v2 Detector by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3016](https://togithub.com/trufflesecurity/trufflehog/pull/3016) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.20 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3077](https://togithub.com/trufflesecurity/trufflehog/pull/3077) - \[bug] - add verify check by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3079](https://togithub.com/trufflesecurity/trufflehog/pull/3079) - \[chore] - Reduce `VerificationOverlapWorker`s by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3082](https://togithub.com/trufflesecurity/trufflehog/pull/3082) - fix(deps): update module github.com/couchbase/gocb/v2 to v2.9.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3078](https://togithub.com/trufflesecurity/trufflehog/pull/3078) - fix(deps): update golang.org/x/exp digest to [`8a7402a`](https://togithub.com/trufflesecurity/trufflehog/commit/8a7402a) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3083](https://togithub.com/trufflesecurity/trufflehog/pull/3083) - fix(deps): update module github.com/googleapis/gax-go/v2 to v2.13.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3085](https://togithub.com/trufflesecurity/trufflehog/pull/3085) - fix(deps): update module google.golang.org/api to v0.189.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3086](https://togithub.com/trufflesecurity/trufflehog/pull/3086) - implemented a netsuite detector by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3068](https://togithub.com/trufflesecurity/trufflehog/pull/3068) - Remove onwater detector by [@​trufflesteeeve](https://togithub.com/trufflesteeeve) in [https://github.com/trufflesecurity/trufflehog/pull/3088](https://togithub.com/trufflesecurity/trufflehog/pull/3088) - Fixed Crash issue in atlassian V2 if data in response is empty array by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3091](https://togithub.com/trufflesecurity/trufflehog/pull/3091) - fix(deps): update module github.com/aws/aws-sdk-go to v1.55.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3087](https://togithub.com/trufflesecurity/trufflehog/pull/3087) - fix(deps): update module github.com/aws/aws-sdk-go to v1.55.2 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3094](https://togithub.com/trufflesecurity/trufflehog/pull/3094) - \[chore] - remove deps from docker image by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3097](https://togithub.com/trufflesecurity/trufflehog/pull/3097) - \[bug]- Invalid Seek for Non-Seekable Readers by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3095](https://togithub.com/trufflesecurity/trufflehog/pull/3095) - chore: fix some comments by [@​shangchenglumetro](https://togithub.com/shangchenglumetro) in [https://github.com/trufflesecurity/trufflehog/pull/3098](https://togithub.com/trufflesecurity/trufflehog/pull/3098) - Analyze by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3099](https://togithub.com/trufflesecurity/trufflehog/pull/3099) - fix(deps): update module cloud.google.com/go/secretmanager to v1.13.5 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3096](https://togithub.com/trufflesecurity/trufflehog/pull/3096) - \[chore] Fix Versioner interface for twitter by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3104](https://togithub.com/trufflesecurity/trufflehog/pull/3104) - Implement Analyzer interface for openai by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/3101](https://togithub.com/trufflesecurity/trufflehog/pull/3101) #### New Contributors - [@​shangchenglumetro](https://togithub.com/shangchenglumetro) made their first contribution in [https://github.com/trufflesecurity/trufflehog/pull/3098](https://togithub.com/trufflesecurity/trufflehog/pull/3098) **Full Changelog**: trufflesecurity/trufflehog@v3.80.1...v3.80.2 ### [`v3.80.1`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.80.1) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.80.0...v3.80.1) #### What's Changed - fix(deps): update golang.org/x/exp digest to [`e3f2596`](https://togithub.com/trufflesecurity/trufflehog/commit/e3f2596) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3071](https://togithub.com/trufflesecurity/trufflehog/pull/3071) - chore: fix goreleaser config and command line options for goreleaser v2 by [@​suzuki-shunsuke](https://togithub.com/suzuki-shunsuke) in [https://github.com/trufflesecurity/trufflehog/pull/3073](https://togithub.com/trufflesecurity/trufflehog/pull/3073) #### New Contributors - [@​suzuki-shunsuke](https://togithub.com/suzuki-shunsuke) made their first contribution in [https://github.com/trufflesecurity/trufflehog/pull/3073](https://togithub.com/trufflesecurity/trufflehog/pull/3073) **Full Changelog**: trufflesecurity/trufflehog@v3.80.0...v3.80.1 ### [`v3.80.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.80.0) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.79.0...v3.80.0) #### What's Changed - Add endorlabs detector by [@​shreyas-sriram](https://togithub.com/shreyas-sriram) in [https://github.com/trufflesecurity/trufflehog/pull/3015](https://togithub.com/trufflesecurity/trufflehog/pull/3015) - New Source: HuggingFace by [@​joeleonjr](https://togithub.com/joeleonjr) in [https://github.com/trufflesecurity/trufflehog/pull/3000](https://togithub.com/trufflesecurity/trufflehog/pull/3000) - Update README.md by [@​joeleonjr](https://togithub.com/joeleonjr) in [https://github.com/trufflesecurity/trufflehog/pull/3019](https://togithub.com/trufflesecurity/trufflehog/pull/3019) - fixing docs by [@​dylanTruffle](https://togithub.com/dylanTruffle) in [https://github.com/trufflesecurity/trufflehog/pull/3022](https://togithub.com/trufflesecurity/trufflehog/pull/3022) - fix(deps): update module github.com/charmbracelet/bubbletea to v0.26.6 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2998](https://togithub.com/trufflesecurity/trufflehog/pull/2998) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.11 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3025](https://togithub.com/trufflesecurity/trufflehog/pull/3025) - fix(deps): update module cloud.google.com/go/secretmanager to v1.13.2 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3024](https://togithub.com/trufflesecurity/trufflehog/pull/3024) - fix(deps): update module github.com/brianvoe/gofakeit/v7 to v7.0.4 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3026](https://togithub.com/trufflesecurity/trufflehog/pull/3026) - fix(deps): update module github.com/couchbase/gocb/v2 to v2.9.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3030](https://togithub.com/trufflesecurity/trufflehog/pull/3030) - update LaunchDarkly detector to use the caller-identity API by [@​pkaeding](https://togithub.com/pkaeding) in [https://github.com/trufflesecurity/trufflehog/pull/3018](https://togithub.com/trufflesecurity/trufflehog/pull/3018) - fix(deps): update module github.com/wasilibs/go-re2 to v1.6.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3033](https://togithub.com/trufflesecurity/trufflehog/pull/3033) - fix(deps): update module github.com/xanzy/go-gitlab to v0.106.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3035](https://togithub.com/trufflesecurity/trufflehog/pull/3035) - \[chore] - remove launchdarkly dep by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3034](https://togithub.com/trufflesecurity/trufflehog/pull/3034) - Fix race in `caflou` and `ldap` detectors by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/3028](https://togithub.com/trufflesecurity/trufflehog/pull/3028) - Elevenlabs detector by [@​dylanTruffle](https://togithub.com/dylanTruffle) in [https://github.com/trufflesecurity/trufflehog/pull/3023](https://togithub.com/trufflesecurity/trufflehog/pull/3023) - fix(deps): update module go.mongodb.org/mongo-driver to v1.16.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3036](https://togithub.com/trufflesecurity/trufflehog/pull/3036) - fix(deps): update module cloud.google.com/go/secretmanager to v1.13.3 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3037](https://togithub.com/trufflesecurity/trufflehog/pull/3037) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.14 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3042](https://togithub.com/trufflesecurity/trufflehog/pull/3042) - fix(deps): update module cloud.google.com/go/storage to v1.43.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3043](https://togithub.com/trufflesecurity/trufflehog/pull/3043) - fix(deps): update module golang.org/x/net to v0.27.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3046](https://togithub.com/trufflesecurity/trufflehog/pull/3046) - fix(deps): update module golang.org/x/crypto to v0.25.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3045](https://togithub.com/trufflesecurity/trufflehog/pull/3045) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.15 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3049](https://togithub.com/trufflesecurity/trufflehog/pull/3049) - fix(deps): update testcontainers-go monorepo to v0.32.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3050](https://togithub.com/trufflesecurity/trufflehog/pull/3050) - \[chore] - remove dead Chunker code by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3044](https://togithub.com/trufflesecurity/trufflehog/pull/3044) - chore(deps): update goreleaser/goreleaser-action action to v6 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3051](https://togithub.com/trufflesecurity/trufflehog/pull/3051) - fix(deps): update golang.org/x/exp digest to [`46b0784`](https://togithub.com/trufflesecurity/trufflehog/commit/46b0784) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3053](https://togithub.com/trufflesecurity/trufflehog/pull/3053) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.16 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3054](https://togithub.com/trufflesecurity/trufflehog/pull/3054) - fix(deps): update module github.com/google/go-containerregistry to v0.20.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3055](https://togithub.com/trufflesecurity/trufflehog/pull/3055) - Order GitLab repos consistently by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/3047](https://togithub.com/trufflesecurity/trufflehog/pull/3047) - Log more GitLab stuff by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/3040](https://togithub.com/trufflesecurity/trufflehog/pull/3040) - update package name by [@​0x1](https://togithub.com/0x1) in [https://github.com/trufflesecurity/trufflehog/pull/3020](https://togithub.com/trufflesecurity/trufflehog/pull/3020) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.17 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3057](https://togithub.com/trufflesecurity/trufflehog/pull/3057) - fix(deps): update module cloud.google.com/go/secretmanager to v1.13.4 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3059](https://togithub.com/trufflesecurity/trufflehog/pull/3059) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.18 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3062](https://togithub.com/trufflesecurity/trufflehog/pull/3062) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.19 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3064](https://togithub.com/trufflesecurity/trufflehog/pull/3064) - fix(deps): update module github.com/xanzy/go-gitlab to v0.107.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3069](https://togithub.com/trufflesecurity/trufflehog/pull/3069) - fix(deps): update golang.org/x/exp digest to [`1d5bc16`](https://togithub.com/trufflesecurity/trufflehog/commit/1d5bc16) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3070](https://togithub.com/trufflesecurity/trufflehog/pull/3070) - Atlassian Token Detector by [@​ankushgoel27](https://togithub.com/ankushgoel27) in [https://github.com/trufflesecurity/trufflehog/pull/3065](https://togithub.com/trufflesecurity/trufflehog/pull/3065) - fix(deps): update module github.com/google/go-containerregistry to v0.20.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/3072](https://togithub.com/trufflesecurity/trufflehog/pull/3072) - \[feat] - Streamlined File Handling with BufferedReaderSeeker by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3041](https://togithub.com/trufflesecurity/trufflehog/pull/3041) - \[perf] - Optimize MIME Type Detection to Reduce Allocations by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/3048](https://togithub.com/trufflesecurity/trufflehog/pull/3048) #### New Contributors - [@​pkaeding](https://togithub.com/pkaeding) made their first contribution in [https://github.com/trufflesecurity/trufflehog/pull/3018](https://togithub.com/trufflesecurity/trufflehog/pull/3018) **Full Changelog**: trufflesecurity/trufflehog@v3.79.0...v3.80.0 ### [`v3.79.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.79.0) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.78.2...v3.79.0) #### What's Changed - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.6 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2996](https://togithub.com/trufflesecurity/trufflehog/pull/2996) - Return targeted scan errors by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/2995](https://togithub.com/trufflesecurity/trufflehog/pull/2995) - Set GIT_DIR based on ScanOptions.Bare by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/3004](https://togithub.com/trufflesecurity/trufflehog/pull/3004) - Adding Larksuite Detectors + Tests by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/3008](https://togithub.com/trufflesecurity/trufflehog/pull/3008) - Pin STARRY-S/zip by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/2999](https://togithub.com/trufflesecurity/trufflehog/pull/2999) - Ensure that `detector-tests` workflow runs detector tests by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2994](https://togithub.com/trufflesecurity/trufflehog/pull/2994) - added "example" to detector badlist by [@​orionooooo](https://togithub.com/orionooooo) in [https://github.com/trufflesecurity/trufflehog/pull/3010](https://togithub.com/trufflesecurity/trufflehog/pull/3010) #### New Contributors - [@​orionooooo](https://togithub.com/orionooooo) made their first contribution in [https://github.com/trufflesecurity/trufflehog/pull/3010](https://togithub.com/trufflesecurity/trufflehog/pull/3010) **Full Changelog**: trufflesecurity/trufflehog@v3.78.2...v3.79.0 ### [`v3.78.2`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.78.2) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.78.1...v3.78.2) #### What's Changed - fix(deps): update golang.org/x/exp digest to [`fc45aab`](https://togithub.com/trufflesecurity/trufflehog/commit/fc45aab) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2931](https://togithub.com/trufflesecurity/trufflehog/pull/2931) - adding twitter + Consumer key detector by [@​abmussani](https://togithub.com/abmussani) in [https://github.com/trufflesecurity/trufflehog/pull/2963](https://togithub.com/trufflesecurity/trufflehog/pull/2963) - Fix test compilation errors by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2964](https://togithub.com/trufflesecurity/trufflehog/pull/2964) - Modularize scanning engine by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2887](https://togithub.com/trufflesecurity/trufflehog/pull/2887) - adding eraser ai detector by [@​0x1](https://togithub.com/0x1) in [https://github.com/trufflesecurity/trufflehog/pull/2961](https://togithub.com/trufflesecurity/trufflehog/pull/2961) - \[feat] - add metrics to the Engine by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2968](https://togithub.com/trufflesecurity/trufflehog/pull/2968) - Double archive `maxDepth` until bug is fixed by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2965](https://togithub.com/trufflesecurity/trufflehog/pull/2965) - Return match/reason from `detectors.IsKnownFalsePositive` by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2969](https://togithub.com/trufflesecurity/trufflehog/pull/2969) - Fix integration tests by [@​rosecodym](https://togithub.com/rosecodym) in [https://github.com/trufflesecurity/trufflehog/pull/2970](https://togithub.com/trufflesecurity/trufflehog/pull/2970) - \[fix] - implement MaxSecretSizeProvider for `auth0managementapitoken` detector by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2953](https://togithub.com/trufflesecurity/trufflehog/pull/2953) - Patch archived dependency by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/2971](https://togithub.com/trufflesecurity/trufflehog/pull/2971) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.2 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2962](https://togithub.com/trufflesecurity/trufflehog/pull/2962) - fix(deps): update golang.org/x/exp digest to [`7f521ea`](https://togithub.com/trufflesecurity/trufflehog/commit/7f521ea) by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2972](https://togithub.com/trufflesecurity/trufflehog/pull/2972) - fix(deps): update module github.com/google/go-containerregistry to v0.19.2 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2973](https://togithub.com/trufflesecurity/trufflehog/pull/2973) - fix(deps): update module go.mongodb.org/mongo-driver to v1.15.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2975](https://togithub.com/trufflesecurity/trufflehog/pull/2975) - fix(deps): update module cloud.google.com/go/storage to v1.42.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2977](https://togithub.com/trufflesecurity/trufflehog/pull/2977) - fix(deps): update module github.com/bradleyfalzon/ghinstallation/v2 to v2.11.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2980](https://togithub.com/trufflesecurity/trufflehog/pull/2980) - fix(deps): update module github.com/elastic/go-elasticsearch/v8 to v8.14.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2981](https://togithub.com/trufflesecurity/trufflehog/pull/2981) - fix(deps): update module github.com/getsentry/sentry-go to v0.28.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2986](https://togithub.com/trufflesecurity/trufflehog/pull/2986) - \[feat] - Add Option to Retain False Positives During Detection by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2967](https://togithub.com/trufflesecurity/trufflehog/pull/2967) - fix(deps): update module google.golang.org/api to v0.185.0 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2987](https://togithub.com/trufflesecurity/trufflehog/pull/2987) - clone more refs by [@​zricethezav](https://togithub.com/zricethezav) in [https://github.com/trufflesecurity/trufflehog/pull/2988](https://togithub.com/trufflesecurity/trufflehog/pull/2988) - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.5 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2992](https://togithub.com/trufflesecurity/trufflehog/pull/2992) - Disambiguate step names in `detector-tests` workflow by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2989](https://togithub.com/trufflesecurity/trufflehog/pull/2989) - fix(deps): update module github.com/googleapis/gax-go/v2 to v2.12.5 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2993](https://togithub.com/trufflesecurity/trufflehog/pull/2993) **Full Changelog**: trufflesecurity/trufflehog@v3.78.1...v3.78.2 ### [`v3.78.1`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.78.1) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.78.0...v3.78.1) #### What's Changed - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.15 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2911](https://togithub.com/trufflesecurity/trufflehog/pull/2911) - fix(deps): update module github.com/microsoft/go-mssqldb to v1.7.2 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2912](https://togithub.com/trufflesecurity/trufflehog/pull/2912) - Add elasticsearch to tui by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/2915](https://togithub.com/trufflesecurity/trufflehog/pull/2915) - Improve GitHub wiki scan errs by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2917](https://togithub.com/trufflesecurity/trufflehog/pull/2917) - Update OpenAI detector by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2863](https://togithub.com/trufflesecurity/trufflehog/pull/2863) - Add flag to get information if trufflehog being ran from TUI by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/1644](https://togithub.com/trufflesecurity/trufflehog/pull/1644) - \[feat] - Introduce `channelmetrics` Package for Channel Metrics Collection by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2889](https://togithub.com/trufflesecurity/trufflehog/pull/2889) - \[feat] - Optimize detector performance by reducing data passed to regex by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2812](https://togithub.com/trufflesecurity/trufflehog/pull/2812) - Go should be installed before codeql initializes by [@​dustin-decker](https://togithub.com/dustin-decker) in [https://github.com/trufflesecurity/trufflehog/pull/2919](https://togithub.com/trufflesecurity/trufflehog/pull/2919) - \[fix] - continue on error by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2921](https://togithub.com/trufflesecurity/trufflehog/pull/2921) - Update Jenkins in tui by [@​hxnyk](https://togithub.com/hxnyk) in [https://github.com/trufflesecurity/trufflehog/pull/2925](https://togithub.com/trufflesecurity/trufflehog/pull/2925) - \[chore] - remove stutter in naming by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2926](https://togithub.com/trufflesecurity/trufflehog/pull/2926) - \[fix] - Correctly calculate EntireSpanChunkCalculator span by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2924](https://togithub.com/trufflesecurity/trufflehog/pull/2924) - Improve Git scaning logs by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2923](https://togithub.com/trufflesecurity/trufflehog/pull/2923) - \[chore] - address comments by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2920](https://togithub.com/trufflesecurity/trufflehog/pull/2920) - Make `cache.Cache` typed by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2930](https://togithub.com/trufflesecurity/trufflehog/pull/2930) - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.17 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2914](https://togithub.com/trufflesecurity/trufflehog/pull/2914) - \[chore] Polish channelmetrics package by [@​mcastorina](https://togithub.com/mcastorina) in [https://github.com/trufflesecurity/trufflehog/pull/2938](https://togithub.com/trufflesecurity/trufflehog/pull/2938) - Add `*.dia` to ignored extensions list by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2939](https://togithub.com/trufflesecurity/trufflehog/pull/2939) - Make the github action work with a path as input by [@​benbridts](https://togithub.com/benbridts) in [https://github.com/trufflesecurity/trufflehog/pull/2908](https://togithub.com/trufflesecurity/trufflehog/pull/2908) - fix(deps): update module github.com/snowflakedb/gosnowflake to v1.10.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2913](https://togithub.com/trufflesecurity/trufflehog/pull/2913) - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.19 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2944](https://togithub.com/trufflesecurity/trufflehog/pull/2944) - fix(deps): update module github.com/launchdarkly/go-server-sdk/v7 to v7.4.1 by [@​renovate](https://togithub.com/renovate) in [https://github.com/trufflesecurity/trufflehog/pull/2947](https://togithub.com/trufflesecurity/trufflehog/pull/2947) - \[bug] - Ensure BufferedFileWriter Flushes Buffer Contents to File Correctly by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2943](https://togithub.com/trufflesecurity/trufflehog/pull/2943) - Change filesystem symlink err handling by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2941](https://togithub.com/trufflesecurity/trufflehog/pull/2941) - Fix panic in MaxMind detector by [@​rgmz](https://togithub.com/rgmz) in [https://github.com/trufflesecurity/trufflehog/pull/2948](https://togithub.com/trufflesecurity/trufflehog/pull/2948) - \[chore] - Update `discordwebhook` detector keyword by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2954](https://togithub.com/trufflesecurity/trufflehog/pull/2954) - \[fix] - Refactor Filtering Logic to Fix Known False Positive Handling in Overlapping Cases by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2946](https://togithub.com/trufflesecurity/trufflehog/pull/2946) - \[feat] - Update span calculation logic to use offset magnitude by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2957](https://togithub.com/trufflesecurity/trufflehog/pull/2957) - \[chore] - pin archiver dependency by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2958](https://togithub.com/trufflesecurity/trufflehog/pull/2958) - \[chore] - Remove replace by [@​ahrav](https://togithub.com/ahrav) in [https://github.com/trufflesecurity/trufflehog/pull/2959](https://togithub.com/trufflesecurity/trufflehog/pull/2959) #### New Contributors - [@​benbridts](https://togithub.com/benbridts) made their first contribution in [https://github.com/trufflesecurity/trufflehog/pull/2908](https://togithub.com/trufflesecurity/trufflehog/pull/2908) **Full Changelog**: trufflesecurity/trufflehog@v3.78.0...v3.78.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on Monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View the [repository job log](https://developer.mend.io/github/splunk/addonfactory-workflow-addon-release). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6W119--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Description:
This PR introduces optimizations to improve the performance of detectors by reducing the amount of data passed to the regex within the
FromData
method. The changes leverage the knowledge of keyword positions to extract relevant portions of the chunk data, where the secret is likely to reside.Key changes:
Introduced
DetectorMatch
struct to represent a detected pattern’s metadata, including the detector key, detector instance, and a slice ofmatchSpan
structs representing the start and end offsets of matched keywords within the chunk.Added
matchSpan
struct to represent a single occurrence of a matched keyword, containing the start and end byte offsets within the chunk.Implemented
Matches
method forDetectorMatch
to extract the relevant portions of the chunk data based on the start and end positions of each match. The end position is determined by taking the minimum of the keyword position +maxMatchLength
(set to 300) and the length of the chunk data.Introduced
FindDetectorMatches
function (previouslyPopulateMatchingDetectors
) to return a slice ofDetectorMatch
instances, each containing the detector key, detector, and a slice of matches. Adjacent or overlapping matches are merged using themergeMatches
function to avoid duplicating or overlapping the matched portions of the chunk data.Updated the detection logic to use the
Matches
method ofDetectorMatch
to extract the relevant portions of the chunk data before passing them to theFromData
method of the detector.Introduced
MaxSecretSizeProvider
interface that detectors can optionally implement to provide a custom maximum size for the secrets they detect. The interface includes a single methodProvideMaxSecretSize() int64
that returns the maximum size of the secret the detector expects to find.As part of the
FindDetectorMatches
function, it checks if a detector implements theMaxSecretSizeProvider
interface. If implemented, theProvideMaxSecretSize
method is called to obtain the detector-specific maximum secret size, which is used to determine the end position of the match span. If the interface is not implemented, the defaultmaxMatchLength
constant is used.Implemented the
MaxSecretSizeProvider
interface in the relevant detectors (PrivateKeyDetector
,GCPDetector
, andGCPApplicationDefaultCredentialsDetector
) and provided appropriate values for the maximum secret size based on the expected size of the secrets they detect. I might be missing some detectors that should implement this interface... i just can't think of them right now 😞The optimization is based on the assumption that most secrets shouldn’t exceed a certain length from the keyword’s position. By default, the
maxMatchLength
constant is set to 300 characters. However, detectors that require a larger or smaller max size can implement theMaxSecretSizeProvider
interface and provide their own value through theProvideMaxSecretSize
method.These changes significantly reduce the amount of data the regex within
FromData
has to process, leading to improved detector performance while still ensuring accurate secret detection. The introduction of theMaxSecretSizeProvider
interface allows for flexibility in handling different secret sizes based on the specific requirements of each detector.Sequence Diagram
Benchmarks
Benchmark assessing the performance of
FromData
with verification disabled across various chunk sizes.orange: old chunk size (10kB)
green: new chunk size (512B overestimate for most detectors)
Checklist:
make test-community
)?make lint
this requires golangci-lint)?