-
Notifications
You must be signed in to change notification settings - Fork 297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support the Bazel build system #264
Comments
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
There are plan to improve Bazel's legacy support for license-checking third party dependencies, which is maybe something we could make use of. |
Bazel team is improving license checking see this document and seem they also have some tools in https://github.com/bazelbuild/rules_license. There is some interest from the HERE side to develop support in ORT for Bazel so listing our standard requirements questions:
|
Each project using bazel contains a file called "WORKSPACE" at the toplevel. I think that's the easiest way to check.
Through
https://docs.bazel.build/versions/master/external.html These are listed in the WORKSPACE file; they should also be queriable via cquery. All of them are hashed for reproducible builds.
Yes.
My advice: Gerrit or or Bazel itself. There are others, but these should be relatively stable and well tested to begin with. (Most software at Google nowadays uses Bazel to build, e.g. Android stuff too). |
As since yesterday, on BazelCon, they had a session on upcoming bzlmod, which is the Bazel package Manager, intended to be integrates on Bazel 5.0. Few key points:
|
Thanks for the update. A super-important topic for ORT would be if that new system would allow to only query the transitive dependency tree incl. metadata, without actually building the project or even downloading the build artifacts (like Maven can do). Do you have any information on whether that's possible? |
https://github.com/vmware/rules_oss_audit uses Bazel inspect to analyze the dependency graph of a build and collect license information about each package it finds. |
Yep, all the process is done without building. |
@tsteenbe Yes, but in the case of vmware, they got the the dependencies explicitly already coming from RPM, so inspect works in some cases, but then, all the metadata comes from the RPM specfile itself. |
Note to myself: There's a Codelab for Bazel. |
This session from BazelCon 2021 might also be interesting: Solving the complexities of identifying and tracking open-source software (OSS) to comply with license requirements by using Bazel to create an accurate bill of materials containing OSS and third-party packages during a build. |
Also see the Bazel Central Registry. |
By now we're at Bazel 6.0 LTS which features Bzlmod. |
As per @alexeagle who I've met at PackageCon, something like |
We should be aiming for C++, Java / Android and Python support via Bazel to start with. |
A reference that could become useful: https://github.com/vmware/rules_oss_audit |
I've been looking into this for a few days now. To keep things simple I created two repositories with toy examples that use Bazel: Since Bazel is at the tail end of switching to bzlmod by now, it makes sense to focus on the second one first. Both repositories contain a Python program, a C++ program and an Android app. The latter consists of a library and an Activity that consumes the library. The Android app is from the Bazel Android tutorial. I just added OkHttp as an external dependency. The Python and C++ programs also have one external dependency each, which both have dependencies as well (at least in the bzlmod case). Unlike Bazel itself, bzlmod is an actual package manager that uses a well-known location in a well-known format to retrieve metadata, source and binary artifacts of dependencies. For projects that actually use bzlmod to manage dependencies, implementing "Bazel support" looks to be relatively straightforward. However, before one can do that, the first challenge is to determine what build targets a given code base contains. A promising approach seems to be looking for
The C++ program in the same repo uses bzlmod to pull in glog and its' dependencies. Running
Looking at https://bcr.bazel.build or rather the Github repo it points to, we find a handy-dandy That concludes the "good news" part. The bad news is that for the vast majority of projects that do not use C or C++, it seems highly unlikely that they will abandon pip/poetry, Maven, npm, etc. in favor bzlmod. So in practice there does not seem to be much difference between the workspace and the bzlmod world for these projects. Maybe the semi-good part about this is that existing functionality in ORT can be used to cover them, but how exactly that would work is still a bit blurry to me and most likely requires custom code for any type of programming language/ecosystem that needs to be supported. The first challenge here is to identify what exactly is being used to manage dependencies and what the relevant manifest/lock/etc. files are to process. I had a reasonably fruitful conversation with ChatGPT about this subject. Of course, it could be wrong and there could be a better approach. But so far |
Yes, we've always needed an abstraction layer so that tooling like this (SBOM generation, license compliance checks) can "read-through" the Bazel graph to the third-party dependencies for every language. Most languages (C++ being the notable exception) are implemented on top of a native package manager rather than throw it away. I can give you some pointers on this, I just connected with folks at https://www.endorlabs.com/blog/introducing-a-better-way-to-sca-for-monorepos-and-bazel who might share some of what they did. |
That would be useful, thanks @alexeagle! |
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
To double-check, probably a comparison to the results of https://github.com/snyk-labs/bazel2snyk makes sense, @haikoschol. |
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This tool examines Bazel build targets. That was also my initial intuition, but it turned out to be possible to gather the information the Analyzer is supposed to produce without doing so. If we wanted to change the approach in the Bazel package manager plugin, the main question would be how build targets fit into the ORT domain model. |
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This is based on oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This starts the work on issue oss-review-toolkit#264. Signed-off-by: Haiko Schol <[email protected]>
This starts the work on issue #264. Signed-off-by: Haiko Schol <[email protected]>
See https://bazel.build/ as used by newer versions of Gerrit .
Tasks
MODULE.bazel
files #8450The text was updated successfully, but these errors were encountered: