Releases: nexB/scancode.io
v34.6.1
Changelog
- Remove print statements from migration files.
- Display full traceback on error in the
execute
management command. - Log the Project message creation.
- Refactor the
get_env_from_config_file
to support empty config file.
What's Changed
Full Changelog: v34.6.0...v34.6.1
v34.6.0
Changelog
- Add a new
scan_for_virus
add-on pipeline based on ClamAV scan.
Found viruses are stored as "error" Project messages and on their related codebase
resource instance using theextra_data
field. #1182 - Add ability to filter by tag on the resource list view. #1217
- Use "unknown" as the Package URL default type when no values are provided for that
field. This allows to create a discovered package instance instead of raising a
Project error message. #1249 - Rename DiscoveredDependency
resolved_to
toresolved_to_package
, and
resolved_dependencies
toresolved_from_dependencies
for clarity and
consistency.
Addchildren_packages
andparent_packages
ManyToMany field on the
DiscoveredPackage model.
Add full dependency tree in the CycloneDX output. #1066 - Add a new
run
entry point for executing pipeline as a single command. #1256 - Generate a DiscoveredPackage.package_uid in create_from_data when not provided. #1256
What's Changed
- Add ability to filter by tag on the resource list view #1217 by @tdruez in #1247
- Increase size of CodebaseResource.status from 30 to 50 by @JonoYang in #1248
- Implement a ScanForVirus Pipeline #1182 by @tdruez in #1193
- Include virus report in the resource extra_data field by @keshav-space in #1250
- Use "unknown" as the Package URL default type for missing data #1249 by @tdruez in #1251
- Add children_packages m2m and rename resolved_to_package #1066 by @tdruez in #1252
- Add an entry point for executing pipeline as a single command by @tdruez in #1256
- Generate a package_uid in create_from_data when not provided #1256 by @tdruez in #1258
- Release 34.6.0 by @tdruez in #1259
Full Changelog: v34.5.0...v34.6.0
v34.5.0
Changelog
- Display the current path location in the "Codebase" panel as a navigation breadcrumbs. #1158
- Fix a rendering issue in the dependency details view when for_package or
datafile_resource fields do not have a value. #1177 - Add a new
CollectPygmentsSymbolsAndStrings
pipeline (addon) for collecting source
symbol, string and comments using Pygments. #1179 - Workaround an issue with the cyclonedx-python-lib that does not allow to load
SBOMs that contains properties with no values.
Also, a few fixes pre-validation are applied before deserializing thr SBOM for
maximum compatibility. #1185 #1230 - Add a new
CollectTreeSitterSymbolsAndStrings
pipeline (addon) for collecting source
symbol and string using tree-sitter. #1181 - Fix
inspect_packages
pipeline to properly link discovered packages and dependencies to
codebase resources of package manifests where they were found. Also correctly assign
the datasource_ids attribute for packages and dependencies. #1180 - Add "Product name" and "Product version" as new project settings. #1197
- Add "Product name" and "Product version" as new project settings. #1197
- Raise the minimum RAM required per CPU code in the docs.
A good rule of thumb is to allow 2 GB of memory per CPU.
For example, if Docker is configured for 8 CPUs, a minimum of 16 GB of memory is
required. #1191 - Add value validation for the search complex query syntax. #1183
- Bump matchcode-toolkit version to v5.0.0.
- Fix the content of the
package_url
field in CycloneDX outputs. #1224 - Enhance support for encoded
package_url
during the conversion to model fields. #1171 - Remove the
scancode_license_score
option from the Project configuration. #1231 - Remove the
extract_recursively
option from the Project configuration. #1236 - Add support for a
ignored_dependency_scopes
field on the Project configuration. #1197 - Add support for storing the scancode-config.yml file in codebase.
The scancode-config.yml file can be provided as a project input, or can be located
in the codebase/ immediate subdirectories. This allows to provide the configuration
file as part of an input archive or a git clone for example. #1236 - Provide a downloadable YAML scancode-config.yml template in the documentation. #1197
- Add support for CycloneDX SBOM component properties as generated by external tools.
For example, theResolvedUrl
generated by cdxgen is now imported as the package
download_url
.
What's Changed
- Display the current path location in the "Codebase" panel #1158 by @tdruez in #1173
- Add D2D for ELFs and Go binaries #1113 #1114 by @TG1999 in #1170
- Fix a rendering issue in the dependency details view #1177 by @tdruez in #1178
- Addon pipeline to collect pygments symbols by @keshav-space in #1179
- Workaround a loading issue with cyclonedx-python-lib #1185 by @tdruez in #1186
- Addon pipeline to collect tree-sitter symbols by @keshav-space in #1181
- Populate package and dependency attributes in inspect_packages by @AyanSinhaMahapatra in #1180
- Increase scancodeio version length by @TG1999 in #1202
- Add "Product name" and "Product version" as new project settings #1197 by @tdruez in #1204
- Skip source-inspector installation on darwin arm64 (not compatible) by @tdruez in #1205
- Raise the minimum RAM required per CPU in the docs #1191 by @tdruez in #1192
- Mock download get requests #1206 by @JonoYang in #1209
- Add value validation for the search complex query syntax #1183 by @tdruez in #1210
- Add tutorial for symbol and string collection by @keshav-space in #1198
- Bump matchcode-toolkit to v5.0.0 by @JonoYang in #1221
- Rename symbols pipelines by @keshav-space in #1222
- Add requires-review tag for resources not mapped by @TG1999 in #1218
- Fix the content of the
package_url
field in CycloneDX outputs #1224 by @tdruez in #1225 - Add support for the empty lists in delete_empty_properties #1185 by @tdruez in #1226
- Enhance support for encoded package_url in the conversion to fields by @tdruez in #1227
- Bump matchcode-toolkit version to v5.1.0 by @JonoYang in #1228
- Bump source-inspector to v0.5.1 by @keshav-space in #1233
- Improve the CycloneDX SBOM pre-validation fixes #1230 by @tdruez in #1232
- Enhance help text documentation for Project settings form #1197 by @tdruez in #1229
- Remove the license_score option from Project configuration #1231 by @tdruez in #1234
- Add new flag for approximate file matches in scanpipe.pipes.flag by @JonoYang in #1239
- Add support for CycloneDX SBOM component properties from external tools by @tdruez in #1241
- Add new resolved_to field on DiscoveredDependency #1066 by @tdruez in #1240
- Bump container-inspector and commoncode versions by @JonoYang in #1242
- Add support for scancode-config.yml in codebase #1236 by @tdruez in #1243
- Add support for ignored_dependency_scopes field for configuration by @tdruez in #1235
- Provide a downloadable scancode-config.yml template in docs #1197 by @tdruez in #1245
- Release 34.5.0 by @tdruez in #1246
Full Changelog: v34.4.0...v34.5.0
v34.4.0
Changelog
- Upgrade Gunicorn to v22.0.0 security release.
- Display the list of fields available for the advanced search syntax in the modal UI. #1164
- Add support for CycloneDX 1.6 outputs and inputs.
Also, the CycloneDX outputs can be downloaded as 1.6, 1.5, and 1.4 spec versions. #1165 - Update matchcode-toolkit to v4.1.0
- Add a new function
scanpipe.pipes.matchcode.fingerprint_codebase_resources()
, which computes
approximate file matching fingerprints for text files using the new
get_file_fingerprint_hashes
function from matchcode-toolkit. - Rename the
purldb-scan-queue-worker
management command topurldb-scan-worker
. - Add
docker-compose.purldb-scan-worker.yml
to run ScanCode.io as a PurlDB
scan worker service.
What's Changed
- Add support for CycloneDX 1.6 outputs and inputs by @tdruez in #1165
- Display the list of fields for the advanced search syntax #1164 by @tdruez in #1167
- Update docker-compose.yml by @JonoYang in #1133
- Fingerprint codebase resources by @JonoYang in #1163
- Workaround the unsupported new tools format in cyclonedx #1171 by @tdruez in #1172
Full Changelog: v34.3.0...v34.4.0
v34.3.0
Changelog
-
Associate resolved packages with their source codebase resource. #1140
-
Add a new
CollectSourceStrings
pipeline (addon) for collecting source string using xgettext. #1160
Full Changelog: v34.2.0...v34.3.0
v34.2.0
Changelog
- Add support for Python 3.12 and upgrade to Python 3.12 in the Dockerfile. #1138
- Add support for CycloneDX XML inputs. #1136
- Upgrade the SPDX schema to v2.3.1 #1130
Full Changelog: v34.1.0...v34.2.0
v34.1.0
Changelog:
- Add support for importing CycloneDX SBOM 1.2, 1.3, 1.4 and 1.5 spec formats. #1045
- The pipeline help modal is now available from all project views: form, list, details.
The docstring are converted from markdown to html for proper rendering. #1105 - Add a new
CollectSymbols
pipeline (addon) for collecting codebase symbols using
Universal Ctags. #1116 - Capture errors during the
inspect_elf_binaries
pipeline execution.
Errors on resource inspection are stored as project error message instead of global
pipeline failure.
The problematic resource path is stored in the message details and displayed in the
message list UI as a link to the resource details view. #1121 #1122 - Use the
package_only
option in scancodeget_package_data
API in
inspect_packages
pipeline, to skip license and copyright detection in
extracted license and copyright statements found in package metadata. nexB/scancode-toolkit#3689 - Rename the
match_to_purldb
pipeline tomatch_to_matchcode
, and add
MatchCode.io API settings to ScanCode.io settings. - In the DiscoveredPackage model, rename the "datasource_id" attribute to
"datasource_ids" and add a new attribute "datafile_paths". This is aligned
with the scancode-toolkit Package model, and package detection information
is now stored correctly. Also update the UI for discovered packages to
show the corresponding package datafiles and their datasource IDs.
A data migration is included to facilitate the migration of existing data. #1099 - Add PurlDB tab, displayed when the PURLDB_URL settings is configured.
When loading the package details view, a request is made on the PurlDB to fetch and
and display any available data. #1125 - Create a new management command
purldb-scan-queue-worker
, that runs
scancode.io as a Package scan queue worker for PurlDB.
purldb-scan-queue-worker
gets the next available Package to be scanned and
the list of pipeline names to be run on the Package from PurlDB, creates a
Project, fetches the Package, runs the specified pipelines, and returns the
results to PurlDB. #1078 nexB/purldb#236 - Update matchcode-toolkit to v4.0.0
Full Changelog: v34.1.0...v34.1.0
v34.0.0
Changelog:
-
Add ability to "group" pipeline steps to control their inclusion in a pipeline run.
The groups can be selected in the UI, or provided using the
"pipeline_name:group1,group2" syntax in CLI and REST API. #1045 -
Refine pipeline choices in the "Add pipeline" modal based on the project context.
- When there is at least one existing pipeline in the project, the modal now includes
all addon pipelines along with the existing pipeline for selection. - In cases where no pipelines are assigned to the project, the modal displays all
base (non-addon) pipelines for user selection. #1071
- When there is at least one existing pipeline in the project, the modal now includes
-
Rename pipeline for consistency and precision:
- scan_codebase_packages: inspect_packages
Restructure the inspect_manifest pipeline into:
- load_sbom: for loading SPDX/CycloneDX SBOMs and ABOUT files
- resolve_dependencies: for resolving package dependencies
- inspect_packages: gets package data from package manifests/lockfiles
A data migration is included to facilitate the migration of existing data.
Only the new names are available in the web UI but the REST API and CLI are backward
compatible with the old names. #1034 https://github.com/nexB/scancode.io/discussions/1035 -
Remove "packageFileName" entry from SPDX output. #1076
-
Add an add-on pipeline for collecting DWARF debug symbol compilation
unit paths when available from elfs. nexB/purldb#260 -
Extract all archives recursively in the
scan_single_package
pipeline. #1081 -
Add URL scheme validation with explicit error messages for input URLs. #1047
-
All supported
output_format
can now be downloaded using the results_download API
action providing a value for the newoutput_format
parameter. #1091 -
Add settings related to fetching private files. Those settings allow to
define credentials for various authentication types. #620 #203 -
Update matchcode-toolkit to v3.0.0
What's Changed
- Refine pipeline choices in the "Add pipeline" modal #1071 by @tdruez in #1072
- Add ability to "group" pipeline steps to control inclusion #1045 by @tdruez in #1055
- Improve the documentation about external service (integrations) by @tdruez in #1073
- Restructure pipelines for verbosity by @AyanSinhaMahapatra in #1074
- Add an add-on pipeline for collecting dwarfs from elfs by @TG1999 in #1068
- Run the extract_archives step in ScanSinglePackage #1081 by @tdruez in #1083
- Use new /collect/index_packages/ endpoint to populate PurlDB by @keshav-space in #1084
- Add URL scheme validation with explicit error messages #1047 by @tdruez in #1085
- Bump matchcode-toolkit version to v3.0.0 by @JonoYang in #1088
- Improve the results_download API action to accept output_format #1091 by @tdruez in #1092
- Add support for fetching authentications #620 by @tdruez in #1097
- Add settings for providing Skopeo credentials #203 by @tdruez in #1098
Full Changelog: v33.1.0...v34.0.0
v33.1.0
Changelog:
- Rename multiple pipelines for consistency and precision:
- docker: analyze_docker_image
- root_filesystems: analyze_root_filesystem_or_vm_image
- docker_windows: analyze_windows_docker_image
- inspect_manifest: inspect_packages
- deploy_to_develop: map_deploy_to_develop
- scan_package: scan_single_package
A data migration is included to facilitate the migration of existing data.
Only the new names are available in the web UI but the REST API and CLI are backward
compatible with the old names. #1044
- Generate CycloneDX SBOM in 1.5 spec format, migrated from 1.4 previously.
The Package vulnerabilities are now included in the CycloneDX SBOM when available. #807 - Improve the inspect_manifest pipeline to accept archives as inputs. #1034
- Add support for "tagging" download URL inputs using the "#" section of URLs.
This feature is particularly useful in the map_develop_to_deploy pipeline when
download URLs are utilized as inputs. Tags such as "from" and "to" can be specified
by adding "#from" or "#to" fragments at the end of the download URLs.
Using the CLI, the uploaded files can be tagged using the "filename:tag" syntax
while using the--input-file
arguments.
In the UI, tags can be edited from the Project details view "Inputs" panel.
On the REST API, a newupload_file_tag
field is available to use along the
upload_file
. #708
What's Changed
- Rename multiple pipelines #1044 by @tdruez in #1053
- Removed extra spaces from package copyright section. by @Divyansh044 in #1054
- Display the entire content of map files by @keshav-space in #1014
- Upgrade CycloneDX output to SPEC v1.5 #807 by @tdruez in #1057
- Update inspect_manifest to accept archives by @AyanSinhaMahapatra in #1037
- Add support for "tagging" URL inputs using # #708 by @tdruez in #1062
- Create "client" matching pipeline by @JonoYang in #1042
- scancode-action docs #599 by @tdruez in #1065
- Support patterns in ABOUT resource paths by @AyanSinhaMahapatra in #982
- Add support for tagging input files in CLI and UI #708 by @tdruez in #1069
New Contributors
- @Divyansh044 made their first contribution in #1054
Full Changelog: v33.0.0...v33.1.0
v33.0.0
Changelog:
- Upgrade Django to version 5.0 and drop support for Python 3.8 and 3.9 #1020
- Fetching "Download URL" inputs is now delegated to an initial pipeline step that is
always run as the start of a pipeline.
This allows to run pipelines on workers running from a remote location, external to
the main ScanCode.io app server. #410 - Migrate the Project.input_sources field into a InputSource model. #410
- Refactor run_scancode to not fail on scan errors happening at the resource level,
such as a timeout. Project error message are created instead. #1018 - Add support for the SCANCODEIO_SCAN_FILE_TIMEOUT setting in the scan_package pipeline. #1018
- Add support for non-archive single file in the scan_package pipeline. #1009
- Do not include "add-on" pipelines in the "New project" form choices. #1041
- Display a "Run pipelines" button in the "Pipelines" panel.
Remove the ability to run a single pipeline in favor of running all "not started"
project pipeline. #997 - Fix an issue where the pipeline details cannot be fetched when using URLs that
include credentials such as "user:pass@domain". #998
What's Changed
- Ignore whitespace files by @keshav-space in #976
- Upgrade multiple dependencies to their latest version by @tdruez in #984
- Provide context for project_resources_url in summary views by @keshav-space in #986
- Fix processing embedded archives by @AyanSinhaMahapatra in #1008
- Expand the scope of flag_whitespace_files by @keshav-space in #987
- Choose best package for PurlDB matched resources by @keshav-space in #975
- Enable local file packages in d2d pipeline by @AyanSinhaMahapatra in #992
- Use package_uid for cdx bom-ref by @keshav-space in #1016
- Refactor run_scancode to handle errors along success #1018 by @tdruez in #1019
- Upgrade Django to version 5.0.0 #1020 by @tdruez in #1021
- Fix minor doc typo in tutorial_api_analyze_package_archive.rst by @pombredanne in #1026
- feat: Resolve
make envfile
command by @jayanth-kumar-morem in #1029 - Add support for non-archive file in scan_package #1009 by @tdruez in #1031
- Bump matchcode-toolkit to 2.0.1 by @JonoYang in #1033
- Migrate the Project.input_sources field to its own model #410 by @tdruez in #1039
- Do not include add-on pipelines in the "New project" form choices #1041 by @tdruez in #1043
- Display a "Run pipelines" button in the "Pipelines" panel #997 by @tdruez in #1046
- Fix the pipeline details fetch for URLs with credentials #998 by @tdruez in #1048
- v33.0.0 by @tdruez in #1049
New Contributors
- @jayanth-kumar-morem made their first contribution in #1029
Full Changelog: v32.7.0...v33.0.0