Programmatic Scanning

[randygeyer]

Hey guys, great tool! I'd like to programmatically scan projects from a Java/Kotlin program without shelling out to the CLI (ProcessBuilder). What's the trick to get the plugins/commands on the classpath when called from a separate application within the same process? I've tried including the various ORT modules as implementation dependencies, but not successfully.

I see references to "pluginClasspath" dependencies in your build.gradle.kts files. Is something like that the key?

e.g. "pluginClasspath"(platform(projects.plugins.versionControlSystems))

Please point me to any docs/code on the topic. Thanks!

[mnonnenmacher]

If you want to add a specific scanner to the classpath, you can directly depend on it, for example, for ScanCode you would need this dependency:
https://central.sonatype.com/artifact/org.ossreviewtoolkit.plugins.scanners/scancode-scanner

implementation("org.ossreviewtoolkit.plugins.scanners:scancode-scanner:34.0.0")

If you want to add all scanner plugins to the classpath, you can depend on the BOM like this:
https://central.sonatype.com/artifact/org.ossreviewtoolkit.plugins/scanners

implementation(platform("org.ossreviewtoolkit.plugins:scanners:34.0.0"))

ORT finds plugins using Java's Service Loader. Each plugin extension point, like the ScannerWrapperFactory class for scanner implementations, has an ALL property that contains all plugins of that type that were found on the classpath. So to get all found scanners you can call ScannerWrapperFactory.ALL. Alternatively, you can get them with the more generic call Plugin.getAll<ScannerWrapperFactory<*>>() (the plugin class is in the org.ossreviewtoolkit.utils:common-utils module).

Be aware that many plugins rely on other tools being installed on the system, for example, to use the ScanCode plugin you would have to install ScanCode first.

Also, the next release will be the first one to be built with Java 21, so you would also have to at least use Java 21 to use ORT as a dependency.

You can also have a look at these two projects which also use ORT programmatically:
https://github.com/eclipse-apoapsis/ort-server
https://github.com/oss-review-toolkit/ort-workbench

[randygeyer]

Perfect! That's what I was missing. What's required for the Analyzer, besides the Analyzer module?

[sschuberth]

At minimum, only the package manager plugins, so use implementation(platform("org.ossreviewtoolkit.plugins:package-managers:34.0.0")). But you might want to take curation providers into use as well, to benefit from fixed-up metadata for packages.

[randygeyer]

Thank you very much, Martin and Sebastian. @sschuberth @mnonnenmacher

Are there any best practices or guidelines published anywhere? For example, should you always use curations providers, and which ones? Both the ort-config and Clearly Defined appear to be publicly available and FOSS. Why would you not use those?

Again, many thanks!

[sschuberth]

We deliberately do not use any of these by default, as it's a matter of trust whether you'd want to take e.g. ClearlyDefined into use. In general ORT tries to stay unopinionated WRT any data use that might have legal impact. The user needs to explicitly opt-in to these.

[randygeyer]

Makes sense. Thanks!

[randygeyer]

Hey guys, have you seen this jgit related issue on Windows yet? This happens at the end of a scan (scancode) run:

01:00:01.949 [DefaultDispatcher-worker-1 @coroutine#9] DEBUG org.ossreviewtoolkit.downloader.WorkingTree - Deleting cached working tree at 'C:\Users\redacted\AppData\Local\Temp\ort-DefaultWorkingTreeCache11600892395638152460' for https://github.com/oshai/kotlin-logging.git and revision e9c6ec570cd503c626fca5878efcf1291d4125b7 ...

java.nio.file.FileSystemException: Failed to delete one or more files. See suppressed exceptions for details.
	at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:313)
	at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively(Extensions.kt:127)
	at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively$default(Extensions.kt:124)
	at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.shutdown(WorkingTreeCache.kt:100)
	at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand$runScanners$5.invokeSuspend(ScannerCommand.kt:240)
       <snip>
	Suppressed: java.nio.file.AccessDeniedException: C:\Users\redacted\AppData\Local\Temp\ort-DefaultWorkingTreeCache11600892395638152460\.git\objects\pack\pack-022fea843be6ffa203eacc6df2164a57b327a9b6.idx

I ran into the same issue in my own code that clones a repo then tried to delete it recursively with kotlin.io.Path function. The only work-around I found was to switch to the blocking java.io.File operation.

java.nio.file.Path.deleteRecursively() - fails with same error above
java.nio.file.Path.toFile().deleteRecursively() - succeeds

I've only tested on Windows and not on linux yet. I'm testing on Windows at the moment because scancode isn't compatible with ARM-based Macs. :( I'm not switching/coercing the arch on my Mac to intel as the scancode installation requires. I'll be running in Docker when the code is further along.

EDIT: running ORT 36.0.0 on JDK 21.0.5, Kotlin 2.0.21. Also fails on JDK 21.0.4 / Kotlin 2.0.20.

[sschuberth]

This seems to be #9334. None of the core developers works regularly on Windows, nor do we currently run CI on Windows, so this went unnoticed so far.

I'm really confused what's causing this, because if some file would still be locked by JGit, I would also expect java.nio.file.Path.toFile().deleteRecursively() to fail.

[randygeyer]

I suspect it's related to non-blocking vs blocking file operations. The blocking delete is forcing the jgit code to close and release it's resources, while the non-blocking isn't. Why that's the case is a mystery. Like I said, I see the exact same behavior in my own unrelated code. I'm sticking with my workaround for now.

[randygeyer]

Hey fellas, I get this warning when running scans using the latest ScanCode version:
WARN o.o.plugins.scanners.scancode.ScanCodeResult - The output format version 4.0.0 exceeds the supported major version 3. Results may be incomplete or incorrect.

Here's my scancode version output:

D:\dev>scancode --version
ScanCode version: 32.3.0
ScanCode Output Format version: 4.0.0
SPDX License list version: 3.25

I see #8881 mentions this in this comment.

Which ScanCode version do you recommend using?

[sschuberth]

The one that we bundle with our Docker image, currently version 32.2.1:

ort/docker/versions.dockerfile

Line 30 in c13dda6

ARG SCANCODE_VERSION=32.2.1

[sschuberth]

BTW, we support ScanCode 32.3.0 / output format version 4.0.0 by now.