Verify Bazel CI test changes are working as expected

[BenHenning]

From #1904:

• Verify that the workflows behave correctly on forks where caching has to be disabled
• Verify that the automatic at-head builds on develop work correctly with the new workflow changes
• Verify that manual workflow runs work correctly with the changes
• (Once 1-3 are completed) Set the new 'Run test' workflow as required
• (After 4) Verify that the requirement doesn't break when creating a PR that doesn't run any affected tests (thereby not running the required workflow)
• Send out a communication explaining the new tests, what people can expect, how to investigate issues, and who to contact
• Fix whatever is going on with this: https://github.com/oppia/oppia-android/pull/2671/checks?check_run_id=1908497857 (affected targets issues)
• Fix hanging compute affected targets actions when Bazel files themselves are changed (Fix #2585: Adding BUILD.bazel in //app/viewmodel/ #2648)

[BenHenning]

Running on develop: https://github.com/oppia/oppia-android/actions/runs/559499045 (strange JVM crash failure, but tests otherwise look operational).

[BenHenning]

Manual run kicked off: https://github.com/oppia/oppia-android/actions/runs/560109387. Seems to be working, but I'll check in just to make sure before checking the above.

[BenHenning]

For the fork test, will need to wait for a new PR to be created to check.

[BenHenning]

Ugh, looks like the check might be failing when no targets are computed: https://github.com/oppia/oppia-android/actions/runs/560112155.

[BenHenning]

I think we can conclude that manually triggering the workflows does work, and that the no affected targets check is NOT working. I'm fairly confident once the underlying issue is fixed that the conclusion task should pass & thus work when its required, so I may preemptively check that once #2696 is verified as working & merged in.

[BenHenning]

/cc @fsharpasharp in case you're interested in following along on this.

[BenHenning]

Another issue: #2695.

[BenHenning]

Confirming forks fail at least due to the key management piece: #2690. There's probably a check missing here.

[BenHenning]

Per https://github.com/oppia/oppia-android/pull/2690/checks?check_run_id=1885145971 it looks like the fork check is failing unexpectedly. Those actions weren't supposed to run which is why they're failing.

[BenHenning]

Interestingly https://github.com/oppia/oppia-android/pull/2320/checks?check_run_id=1886178465 ran successfully so the computed affected tests bit is not yet reliable.

[BenHenning]

#2648 seems to show that Bazel file changes are causing the compute affected targets to hang: https://github.com/oppia/oppia-android/runs/1925361182. Adding this as another TODO.

[BenHenning]

#2775 has another example of the affected targets issue: https://github.com/oppia/oppia-android/pull/2775/checks?check_run_id=2036846344:

Affected tests (note that this might be all tests if on the develop branch): "//app:src/sharedTest/java/org/oppia/android/app/administratorcontrols/AdministratorControlsActivityTest","//app:src/sharedTest/java/org/oppia/android/app/administratorcontrols/AppVersionActivityTest","//app:src/sharedTest/java/org/oppia/android/app/completedstorylist/CompletedStoryListActivityTest","//app:src/sharedTest/java/org/oppia/android/app/faq/FAQListFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/faq/FAQListFragmentTest//app:src/sharedTest/java/org/oppia/android/app/player/state/StateFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/faq/FAQSingleActivityTest","//app:src/sharedTest/java/org/oppia/android/app/faq/FaqListActivityTest","//app:src/sharedTest/java/org/oppia/android/app/help/HelpFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/home/HomeActivityTest","//app:src/sharedTest/java/org/oppia/android/app/home/RecentlyPlayedFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/home/TopicSummaryViewModelTest","//app:src/sharedTest/java/org/oppia/android/app/home/WelcomeViewModelTest","//app:src/sharedTest/java/org/oppia/android/app/home/promotedlist/PromotedStoryListViewModelTest","//app:src/sharedTest/java/org/oppia/android/app/home/promotedlist/PromotedStoryViewModelTest","//app:src/sharedTest/java/org/oppia/android/app/mydownloads/MyDownloadsFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/onboarding/OnboardingFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/ongoingtopiclist/OngoingTopicListActivityTest","//app:src/sharedTest/java/org/oppia/android/app/options/OptionsFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/parser/CustomBulletSpanTest","//app:src/sharedTest/java/org/oppia/android/app/parser/HtmlParserTest","//app:src/sharedTest/java/org/oppia/android/app/player/audio/AudioFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/player/exploration/ExplorationActivityTest","//app:src/sharedTest/java/org/oppia/android/app/player/state/StateFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/profile/AddProfileActivityTest","//app:src/sharedTest/java/org/oppia/android/app/profile/AdminAuthActivityTest","//app:src/sharedTest/java/org/oppia/android/app/profile/AdminPinActivityTest","//app:src/sharedTest/java/org/oppia/android/app/profile/PinPasswordActivityTest","//app:src/sharedTest/java/org/oppia/android/app/profile/ProfileChooserFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/profileprogress/ProfilePictureActivityTest","//app:src/sharedTest/java/org/oppia/android/app/profileprogress/ProfileProgressFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/recyclerview/BindableAdapterTest","//app:src/sharedTest/java/org/oppia/android/app/settings/profile/ProfileEditActivityTest","//app:src/sharedTest/java/org/oppia/android/app/settings/profile/ProfileListFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/settings/profile/ProfileRenameActivityTest","//app:src/sharedTest/java/org/oppia/android/app/settings/profile/ProfileResetPinActivityTest","//app:src/sharedTest/java/org/oppia/android/app/splash/SplashActivityTest","//app:src/sharedTest/java/org/oppia/android/app/story/StoryActivityTest","//app:src/sharedTest/java/org/oppia/android/app/story/StoryFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/testing/DragDropTestActivityTest","//app:src/sharedTest/java/org/oppia/android/app/testing/ImageRegionSelectionInteractionViewTest","//app:src/sharedTest/java/org/oppia/android/app/testing/InputInteractionViewTestActivityTest","//app:src/sharedTest/java/org/oppia/android/app/testing/NavigationDrawerActivityTest","//app:src/sharedTest/java/org/oppia/android/app/testing/TestFontScaleConfigurationUtilActivityTest","//app:src/sharedTest/java/org/oppia/android/app/testing/TopicTestActivityForStoryTest","//app:src/sharedTest/java/org/oppia/android/app/topic/TopicActivityTest","//app:src/sharedTest/java/org/oppia/android/app/topic/TopicFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/topic/conceptcard/ConceptCardFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/topic/info/TopicInfoFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/topic/lessons/TopicLessonsFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/topic/practice/TopicPracticeFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/topic/questionplayer/QuestionPlayerActivityTest","//app:src/sharedTest/java/org/oppia/android/app/topic/revision/TopicRevisionFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/topic/revisioncard/RevisionCardFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/utility/RatioExtensionsTest","//app:src/sharedTest/java/org/oppia/android/app/walkthrough/WalkthroughActivityTest","//app:src/sharedTest/java/org/oppia/android/app/walkthrough/WalkthroughFinalFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/walkthrough/WalkthroughTopicListFragmentTest","//app:src/sharedTest/java/org/oppia/android/app/walkthrough/WalkthroughWelcomeFragmentTest","//app:src/test/java/org/oppia/android/app/home/HomeActivityLocalTest","//app:src/test/java/org/oppia/android/app/parser/StringToRatioParserTest","//app:src/test/java/org/oppia/android/app/player/exploration/ExplorationActivityLocalTest","//app:src/test/java/org/oppia/android/app/player/state/StateFragmentLocalTest","//app:src/test/java/org/oppia/android/app/profile/ProfileChooserFragmentLocalTest","//app:src/test/java/org/oppia/android/app/story/StoryActivityLocalTest","//app:src/test/java/org/oppia/android/app/testing/CompletedStoryListSpanTest","//app:src/test/java/org/oppia/android/app/testing/HomeSpanTest","//app:src/test/java/org/oppia/android/app/testing/OngoingTopicListSpanTest","//app:src/test/java/org/oppia/android/app/testing/ProfileChooserSpanTest","//app:src/test/java/org/oppia/android/app/testing/RecentlyPlayedSpanTest","//app:src/test/java/org/oppia/android/app/testing/TopicRevisionSpanTest","//app:src/test/java/org/oppia/android/app/testing/administratorcontrols/AdministratorControlsFragmentTest","//app:src/test/java/org/oppia/android/app/testing/options/AppLanguageFragmentTest","//app:src/test/java/org/oppia/android/app/testing/options/AudioLanguageFragmentTest","//app:src/test/java/org/oppia/android/app/testing/options/OptionsFragmentTest","//app:src/test/java/org/oppia/android/app/testing/options/ReadingTextSizeFragmentTest","//app:src/test/java/org/oppia/android/app/testing/player/state/StateFragmentAccessibilityTest","//app:src/test/java/org/oppia/android/app/topic/info/TopicInfoFragmentLocalTest","//app:src/test/java/org/oppia/android/app/topic/lessons/TopicLessonsFragmentLocalTest","//app:src/test/java/org/oppia/android/app/topic/questionplayer/QuestionPlayerActivityLocalTest","//app:src/test/java/org/oppia/android/app/topic/revisioncard/RevisionCardActivityLocalTest","//domain:src/test/java/org/oppia/android/domain/audio/AudioPlayerControllerTest","//domain:src/test/java/org/oppia/android/domain/audio/CellularAudioDialogControllerTest","//domain:src/test/java/org/oppia/android/domain/classify/AnswerClassificationControllerTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/dragAndDropSortInput/DragDropSortInputHasElementXAtPositionYRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/dragAndDropSortInput/DragDropSortInputHasElementXBeforeElementYRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/dragAndDropSortInput/DragDropSortInputIsEqualToOrderingClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/dragAndDropSortInput/DragDropSortInputIsEqualToOrderingWithOneItemAtIncorrectPositionClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputHasDenominatorEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputHasFractionalPartExactlyEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputHasIntegerPartEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputHasNoFractionalPartRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputHasNumeratorEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputIsEquivalentToAndInSimplestFormRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputIsEquivalentToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputIsExactlyEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputIsGreaterThanRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/fractioninput/FractionInputIsLessThanRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/imageClickInput/ImageClickInputIsInRegionRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/itemselectioninput/ItemSelectionInputContainsAtLeastOneOfRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/itemselectioninput/ItemSelectionInputDoesNotContainAtLeastOneOfRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/itemselectioninput/ItemSelectionInputIsProperSubsetOfRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/itemselectioninput/ItemsSelectionInputEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/multiplechoiceinput/MultipleChoiceInputEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numberwithunits/NumberWithUnitsIsEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numberwithunits/NumberWithUnitsIsEquivalentToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputIsGreaterThanOrEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputIsGreaterThanRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputIsInclusivelyBetweenRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputIsLessThanOrEqualToRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputIsLessThanRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/numericinput/NumericInputIsWithinToleranceRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/ratioinput/RatioInputEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/ratioinput/RatioInputHasNumberOfTermsEqualToClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/ratioinput/RatioInputIsEquivalentRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/textinput/TextInputCaseSensitiveEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/textinput/TextInputContainsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/textinput/TextInputEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/textinput/TextInputFuzzyEqualsRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/classify/rules/textinput/TextInputStartsWithRuleClassifierProviderTest","//domain:src/test/java/org/oppia/android/domain/exploration/ExplorationDataControllerTest","//domain:src/test/java/org/oppia/android/domain/exploration/ExplorationProgressControllerTest","//domain:src/test/java/org/oppia/android/domain/onboarding/AppStartupStateControllerTest","//domain:src/test/java/org/oppia/android/domain/oppialogger/OppiaLoggerTest","//domain:src/test/java/org/oppia/android/domain/oppialogger/analytics/AnalyticsControllerTest","//domain:src/test/java/org/oppia/android/domain/oppialogger/exceptions/ExceptionsControllerTest","//domain:src/test/java/org/oppia/android/domain/oppialogger/exceptions/UncaughtExceptionLoggerStartupListenerTest","//domain:src/test/java/org/oppia/android/domain/oppialogger/loguploader/LogUploadWorkManagerInitializerTest","//domain:src/test/java/org/oppia/android/domain/oppialogger/loguploader/LogUploadWorkerTest","//domain:src/test/java/org/oppia/android/domain/profile/ProfileManagementControllerTest","//domain:src/test/java/org/oppia/android/domain/question/QuestionAssessmentProgressControllerTest","//domain:src/test/java/org/oppia/android/domain/question/QuestionTrainingControllerTest","//domain:src/test/java/org/oppia/android/domain/topic/StoryProgressControllerTest","//domain:src/test/java/org/oppia/android/domain/topic/TopicControllerTest","//domain:src/test/java/org/oppia/android/domain/topic/TopicListControllerTest","//domain:src/test/java/org/oppia/android/domain/util/InteractionObjectExtensionsTest","//domain:src/test/java/org/oppia/android/domain/util/RatioExtensionsTest","//domain:src/test/java/org/oppia/android/domain/util/StateRetrieverTest","//domain:src/test/java/org/oppia/android/domain/util/StringExtensionsTest","//testing:src/test/java/org/oppia/android/testing/CoroutineExecutorServiceTest","//testing:src/test/java/org/oppia/android/testing/FakeEventLoggerTest","//testing:src/test/java/org/oppia/android/testing/FakeExceptionLoggerTest","//testing:src/test/java/org/oppia/android/testing/profile/ProfileTestHelperTest","//testing:src/test/java/org/oppia/android/testing/story/StoryProgressTestHelperTest","//testing:src/test/java/org/oppia/android/testing/time/FakeOppiaClockTest","//utility:src/test/java/org/oppia/android/util/data/AsyncResultTest","//utility:src/test/java/org/oppia/android/util/data/DataProvidersTest","//utility:src/test/java/org/oppia/android/util/data/InMemoryBlockingCacheTest","//utility:src/test/java/org/oppia/android/util/datetime/DateTimeUtilTest","//utility:src/test/java/org/oppia/android/util/extensions/BundleExtensionsTest","//utility:src/test/java/org/oppia/android/util/logging/EventBundleCreatorTest","//utility:src/test/java/org/oppia/android/util/networking/NetworkConnectionUtilTest","//utility:src/test/java/org/oppia/android/util/parser/CustomHtmlContentHandlerTest","//utility:src/test/java/org/oppia/android/util/profile/DirectoryManagementUtilTest","//utility:src/test/java/org/oppia/android/util/system/OppiaDateTimeFormatterTest"

Running the script locally did not yield issues, yet we see failures like the above happening on occasion. Seems to be a CI issue, so adding additional diagnostics to try and figure out why this is happening.

[BenHenning]

Forks are now verified as working. Some tweaks have been made, but both forks & non-forks have been working well for the last few weeks.

[BenHenning]

Also another update: CI should no longer hang on the compute affects tests issue since the empty matrix issue was fixed in #2696.

[BenHenning]

CI runtime was down substantially with #3035 making it closer to viable to enable the tests as required.

[BenHenning]

#2844 also needs to be resolved before we can enable Bazel tests, and it seems like recent changes might be working.

[BenHenning]

One of the last remaining major issues is sometimes the Bazel query output has formatting issues, resulting in invalid test targets like:

Test target: //app:src/test/java/org/oppia/android/app/topic/questionplayer/QuestionPlayerActivityLocalTest//utility:src/test/java/org/oppia/android/util/system/OppiaDateTimeFormatterTest

I've debugged this a bit and found out that it's due to output from Bazel query, but I've never been able to repro it locally. We might need to do a hacky workaround like detecting if there's a test target with this formatting and then manually split it up via Bash.

[BenHenning]

Some updates:

• Fix part of #2691: Migrate compute affected tests script to Kotlin #3374 is building resilience into the script to ensure the unexpected formatting issues don't trip up the output of the script
• Per Fix part of #2691: Migrate compute affected tests script to Kotlin #3374 (comment) we now have an isolated example of the query hang issue (seems to be tied to whenever WORKSPACE is changed, though I'm not 100% positive on that yet).

[BenHenning]

Now that batching is in, we're seeing #2844 and #3759 crop up.

Also, https://github.com/oppia/oppia-android/pull/3849/checks?check_run_id=3719505383 had basically all of its Bazel workflows fail due to a download flake from Google's CDN. We've seen this before, but never as badly as in that PR. bazelbuild/bazel#12010 is a Bazel-specific issue that had similar reports, and they introduced a flag to improve retries. I'm not sure that will work in cases like we saw above since every Bazel workflow failed (which essentially means there were a bunch of retries). Usually we see just one or two fail, so this flag might work for those cases. Hopefully the all-workflows-fail aren't as common.

[BenHenning]

So it looks like some recent PRs have revealed the compute_affected_tests can sometimes timeout. While I think that's believable (since CI runs much slower), I also noticed certain PRs (like #4886) can actually cause a permanent hang due to a bug I discovered in the CommandExecutorImpl. Fortunately, this is straightforward to fix since I had to fix it as part of #4092.

[BenHenning]

Now that #4092 is in and despite some of the unchecked items in the issue's main message body, I actually don't think there's much more to do in the context of this thread. We've now been using the compute affected tests flow for 3.5 years (with a bunch of updates to improve its performance and robustness). It's now operating quite well, I think--I have high confidence that it does the right thing in most places, and has very few false negatives. I think I can confidently say that this is verified as working now.