Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #5370, part of #59: Migrate to Bazel 6.5.0 #4886

Merged
merged 292 commits into from
Jun 13, 2024

Conversation

BenHenning
Copy link
Member

@BenHenning BenHenning commented Feb 28, 2023

Explanation

Fixes #5370
Fixes part of #59

This PR updates the project to use Bazel 6.5.0 instead of 4.0.0.

Note that most of the changes done so far in addressing #59 are centered around the concept of simplifying the Bazel maintenance as much as possible so that it's not too much more difficult than Gradle by the time we fully remove Gradle support from the project. While Bazel will always require more effort, there are many things that can be done to narrow the gap. This is a major step in that process since Bazel 4.x required using a custom Android toolchain (https://github.com/oppia/oppia-bazel-tools) which is not at all user friendly. Plus, there are many compatibility and performance improvements in later versions of Bazel that we want to be able to incorporate within the broader Oppia Android project.

Bazel 6.x was specifically chosen because:

Some other important details to note:

  • rules_kotlin 1.7.x is needed at a minimum for Bazel 5.x+ support. However, an additional fix was needed (Get rid of duplicates in the --processors and --processorpath arguments. bazelbuild/rules_kotlin#940) in order to fix a deviation in functionality that occurred starting in Bazel 5.x's java_plugin support which led to some file duplication in rules_kotlin (that was fortunately easy to fix). Unfortunately, this change wasn't backported to 1.7.x so this PR makes use of a custom patch to rules_kotlin 1.7.1 (https://github.com/oppia/rules_kotlin) that includes the needed change. We'll get this change properly once we can upgrade to 1.8.x, though that will also require updating Kotlin itself to 1.8.x due to Following instructions for a custom kotlinc distribution causes builds to fail bazelbuild/rules_kotlin#1019.
  • Bazel 6.x (maybe 5.x) requires at least build tools 30.0.0 since it completely removed support for the old D8 compat dexer. 32.0.0 was chosen in this PR as it's simply a newer, more up-to-date build tools (and removes D8 completely). With this upgrade to Bazel 6.x we'll be able to update the build tools version more often (so long as it doesn't introduce AGP incompatibilities since we can't upgrade Gradle).
  • As of Bazel 6.x, we're able to reenable Java header compilation and incremental dexing, both of which should have significant performance improvements for incremental builds of the app (and in fact we will have build errors if we disable incremental dexing).
  • In CI, we opted to not support build tools 29.0.2 or old builds of the app. Instead, we'll rely on build tools failing for certain PRs as an indicator that those PRs will require an update (once this PR is merged) in order to have CI run correctly. This is a lot easier than trying to figure out how to support before/after changes with some fairly complex environment differences.
  • There are a bunch of version updates that were needed to support the minimum version of Kotlin for rules 1.7.x (1.6 I think) as well as JDK 11 (which I think was needed for Bazel 5.x), and these have largely been taken care of in previous PRs to this one (though the JDK 11 update in CI was done in this PR, along with wiki documentation updates to address [Feature Request]: Update Bazel Linux instructions to use JDK 11 #5370). One such case of a necessary version upgrade: Dagger fails with Kotlin 1.5-M2 google/dagger#2511.
  • There was a change needed for the databinding java_plugin declaration to specify that it generates an API (in order for it to be used correctly in builds).
  • rules_java needed to be updated to support the newer version of Bazel.
  • The desugaring hack needed for kotlinx-coroutines-core-jvm was removed since it's no longer needed with the build tools & Bazel upgrade introduced in this PR.
  • This includes one small change in third-party to change all single-export wrappers that don't have additional plugins being enabled to aliases instead. This is more semantically correct as the wrappers may lose information (which caused problems when investigating adding Jetpack Compose support in Prototype Compose on Bazel  #5401). While this isn't directly required for the Bazel upgrade, this is the last PR needed for Jetpack Compose support so it's being added here for simplicity.
  • .bazelrc was updated to configure tools, tests, and builds to all use the remote JDK 11 available via Bazel rather than ever using the user's local JDK. This should improve build hermeticity and consistency across different user environments (see https://bazel.build/docs/bazel-and-java).
  • Setup docs were updated to remove setting up JDK 11 (or Java at all for Linux & Mac) now that the user no longer needs to install Java (see previous point) except for Windows. The Python instructions were also removed since Bazel 6.x includes fixes for Android tools that previously depended on Python 2.x.
  • CI was unchanged for Java setup since, as far as I can tell, it's still needed for sdkmanager.

There was also some small cleanup in unit_tests.yml that I noticed when updating CI versions.

Essential Checklist

  • The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
  • Any changes to scripts/assets files have their rationale included in the PR explanation.
  • The PR follows the style guide.
  • The PR does not contain any unnecessary code changes from Android Studio (reference).
  • The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
  • The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

N/A -- This is a build infrastructure change. It shouldn't impact the end user experience.

@oppiabot
Copy link

oppiabot bot commented Mar 8, 2023

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!

@oppiabot oppiabot bot added stale Corresponds to items that haven't seen a recent update and may be automatically closed. and removed stale Corresponds to items that haven't seen a recent update and may be automatically closed. labels Mar 8, 2023
@oppiabot
Copy link

oppiabot bot commented Mar 20, 2023

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!

@oppiabot oppiabot bot added stale Corresponds to items that haven't seen a recent update and may be automatically closed. and removed stale Corresponds to items that haven't seen a recent update and may be automatically closed. labels Mar 20, 2023
@BenHenning BenHenning changed the title Migrates to Bazel 5.0 & Kotlin 1.6 Migrates to Bazel 6.1.1 & Kotlin 1.6 Mar 24, 2023
Also, update TODO check script to have nicer output, and support
generating the exemption textproto file for easier updates in the
future.
This moves the codebase to using the recommended single top-level Dagger
library rather than replicating it in a bunch of different places.
This is needed for downstream work. It also includes ensuring that Guava
JRE can never be used (since only Android should ever be referenced by
the production app build).
There's some cleanup work needed beyond this, but this is the core
change to introduce Kotlin 1.6 support (at least for dev builds).

Moshi needed to be upgraded due to a metadata incompatibility when moving
over to Kotlin 1.6.
rules_kotlin moved its imports into more structured bzl files to load,
so this updates all references in the codebase to point to the correct
locations (which removes debug warnings that show up on the CLI).
Moshi 1.14 pulls in kotlin-reflect 1.7.0 which isn't compatible with the
rules_kotlin version we need for Bazel 4.x. Relatedly, this downgrades
rules_kotlin to 1.5.0, but it fortunately keeps all other changes needed
for 1.7.1 (which will be used in a later PR).

Some code fixes were needed, too, for unknown reasons (since the build
should've been using Kotlin 1.6 before). Either way, these fixes seem
reasonable.
Fixed all warnings that the compiler warned about.

Removed ViewModelProvider & fixed state leaking entirely by moving away
from Jetpack's ViewModel as a base class (since we aren't correctly
using that correctly).
@oppiabot
Copy link

oppiabot bot commented Mar 31, 2023

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!

@oppiabot oppiabot bot added the stale Corresponds to items that haven't seen a recent update and may be automatically closed. label Mar 31, 2023
Enables Java warnings-as-errors, though this doesn't yet apply to
kapt-generated code (such as the code from Dagger), but those warnings
were still manually fixed.

This also fixes a small import warning in a proto file, and warnings
when building oppia_dev_kitkat (by updating the main dex list, but it's
likely the build doesn't work anymore, anyway, and it's hard to test
locally).
There's a race condition when building large numbers of app tests
simultaneously that can lead to build failures. While the CI runs are
resilient now to this failure, it'd be better to try and fix it. This
removes the last non-AndroidX dependency from the codebase with hopes
that it helps reduce the likelihood of the error (though there are no
dependencies on it, so it's unlikely).
Conflicts:
	WORKSPACE
	app/build.gradle
	app/src/main/java/org/oppia/android/app/viewmodel/ViewModelBridgeFactory.kt
	config/proguard/kotlin-proguard-rules.pro
	domain/src/main/java/org/oppia/android/domain/oppialogger/analytics/AnalyticsController.kt
	scripts/assets/maven_dependencies.textproto
	scripts/src/java/org/oppia/android/scripts/common/BUILD.bazel
	scripts/src/java/org/oppia/android/scripts/common/CommandExecutorImpl.kt
	scripts/src/java/org/oppia/android/scripts/license/MavenDependenciesRetriever.kt
	scripts/src/java/org/oppia/android/scripts/maven/model/MavenListDependencies.kt
	scripts/src/java/org/oppia/android/scripts/maven/model/MavenListDependency.kt
	scripts/src/java/org/oppia/android/scripts/maven/model/MavenListDependencyTree.kt
	third_party/BUILD.bazel
	third_party/maven_install.json
	third_party/versions.bzl
	tools/kotlin/BUILD.bazel
	utility/build.gradle
	utility/src/main/java/org/oppia/android/util/parser/image/UrlImageParser.kt
@Rd4dev
Copy link
Collaborator

Rd4dev commented Jun 13, 2024

So I spent quite a while trying to find an alternative to zip as that's what @Rd4dev reported wasn't working on Windows (@adhiamboperes confirmed via chat that the build works on Mac). However, //:oppia is working for RD.

At this point, I realized that //:oppia_dev should already be failing on Windows, so fixing this ought not block this PR. Going ahead and merging it after build stats runs.

Got it, @BenHenning. Would that require making changes to Bazel Instructions with Windows wiki to use //:oppia and not //:oppia_dev?

Copy link
Collaborator

@Rd4dev Rd4dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting aside the existing //:oppia_dev issue on Windows, and the coverage issues with a few test targets, everything else LGTM. Thanks @BenHenning!

@oppiabot oppiabot bot unassigned Rd4dev Jun 13, 2024
Copy link

oppiabot bot commented Jun 13, 2024

Unassigning @Rd4dev since they have already approved the PR.

@oppiabot oppiabot bot added the PR: LGTM label Jun 13, 2024
Copy link

oppiabot bot commented Jun 13, 2024

Hi @BenHenning, this PR is ready to be merged. Please address any remaining comments prior to merging, and feel free to merge this PR once the CI checks pass and you're happy with it. Thanks!

@BenHenning
Copy link
Member Author

@Rd4dev that's probably a good idea (the suggested Windows doc change). Will make that and then enable auto-merge.

Separately, I realized that build stats won't run for this branch since it's using a newer version of Bazel. :)

Revert changes to Windows doc.
Copy link
Member Author

@BenHenning BenHenning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-reviewed Windows doc changes (mostly reversions).

@BenHenning BenHenning enabled auto-merge (squash) June 13, 2024 05:39
@BenHenning BenHenning merged commit add9f74 into develop Jun 13, 2024
43 checks passed
@BenHenning BenHenning deleted the migrate-to-newer-bazel-and-kotlin branch June 13, 2024 06:08
Vishwajith-Shettigar added a commit to Vishwajith-Shettigar/oppia-android that referenced this pull request Jun 15, 2024
@BenHenning BenHenning restored the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 02:43
@BenHenning BenHenning deleted the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 02:44
@BenHenning BenHenning restored the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 02:44
@BenHenning BenHenning deleted the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 02:45
@BenHenning BenHenning restored the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 18:53
@BenHenning BenHenning deleted the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 18:54
@BenHenning BenHenning restored the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 18:55
@BenHenning BenHenning deleted the migrate-to-newer-bazel-and-kotlin branch June 21, 2024 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Update Bazel Linux instructions to use JDK 11
5 participants