Skip to content

Latest commit

 

History

History
1471 lines (1283 loc) · 77.7 KB

README.md

File metadata and controls

1471 lines (1283 loc) · 77.7 KB
Raw

Table Of Contents

Introduction

I'm making this demo repo and writeup because it was surprisingly and frustratingly difficult to get Xamarin.UITest tests for Android to run on a Microsoft-hosted agent in an Azure DevOps pipeline. NO App Center. NO self-hosted agents. I just wanted to do everything in Azure DevOps.

(Once I got that to work, I did add in App Center UI testing...which was also surprisingly difficult, so hopefully this demo is helpful for that as well.)

This demo has grown into showing how to accomplish quite a few common goals for an Azure Devops continuous integration pipeline for the Android portion of a Xamarin app...

  • Each build gets its own versionCode and versionName.
  • Build the APK.
  • Sign the APK.
  • Publish the APK as a pipeline artifact.
  • Do unit tests (NUnit).
  • Do UI tests (Xamarin.UITest) in Azure DevOps, which involves several Android emulator steps.
  • Do UI tests in App Center.
  • Publish all test results (including device-labeled App Center test results in Azure DevOps test explorer).

This demo is not about getting started on unit testing or UI testing; the demo is about getting these things to work in an Azure DevOps pipeline.

You can see a successful run, a successful job overview, published artifacts, and unit+UI test results.

This repo is available as a visualstudio.com repo and a github repo. As of 2020-Dec-24, Azure DevOps offers a free tier with 30 build hours per month and 2 GiB of artifact storage. The free tier was more than enough for all the pipeline needs of this demo.

This writeup is available as a github readme, visualstudio.com readme, and blog post. The repo readmes will be kept up to date, but the blog post may not receive many updates after 2020-12-24. Readme section links are oriented for GitHub.

Notable Files

The XamarinPipelineDemo.Android/AzureDevOps/ folder has most of the notable files...

  • pipeline-android.yml: the pipeline definition and heart of this demo.
  • AndroidSetVersion.ps1: the script that manipulates the Android manifest file to update the versionName and versionCode attributes.
  • example.keystore: for signing the APK. Normally keystore files are sensitive and you wouldn't put them (and their passwords) in your repo, but this is a demo.

XamarinPipelineDemo.UITest/AppInitializer.cs: the autogenerated AppInitializer.cs has been modified so that you can specify which APK file to install for testing, or which keystore to match an already installed (signed) APK. I suggest the APK file methodology.

local_uitest_run.ps1: script to run UITest tests (on a local Android device or emulator) in way most similar to how the pipeline will do it.

appcenter_uitest_run.ps1: script to run UITest tests remotely via App Center. You'll need to set up your own App Center account (including app and device set) and modify the script to use that account.

Screenshots folder has some screenshots of the results of a working pipeline run, and some of the web interface you need to tangle with to get the pipeline working.

Getting Started On Local Machine

First, check that it works on your machine. Open the solution in Visual Studio 2019, and deploy the Release build to an Android emulator or connected Android device (just select Release build configuration and launch the debugger). The app should show you a page with a label that says "Some text.".

In Visual Studio's test explorer, run both the Nunit and UITest tests. Everything should pass.

Also, to run the UITest tests in the way most similar to how the pipeline will do it, install a recent stable nunit3-console release, go into the LocalScripts folder and run local_uitest_run.ps1. You'll get a test results file TestResult.xml and a detailed log uitest.log that is useful for troubleshooting. The script tries to use adb and msbuild from your PATH environment variable and a few other locations. You might have to add your adb or msbuild directories to your PATH. Also, you might have to set the ANDROID_HOME environment variable to something like C:\Program Files (x86)\Android\android-sdk and the JAVA_HOME environment variable to something like C:\Program Files\Android\jdk\microsoft_dist_openjdk_1.8.0.25

Getting Started On Azure DevOps

In the Pipelines => Library section of your Azure DevOps project, you need to do a few things.

You need to set up the keystore file and variables...

  • Upload example.keystore as a secure file.
  • Create a variable group named android_demo_var_group. In it, create the following variables...
    • androidKeystoreSecureFileName: example.keystore
    • androidKeyAlias: androiddebugkey
    • androidKeystorePassword: android
    • androidKeyPassword: android
  • Make the androidKeystorePassword and androidKeyPassword secret by clicking the padlock icon.

You need to create a pipeline from the yaml pipeline definition file...

  • Upload the repo to Azure DevOps.
  • Create a new pipeline.
  • When asked "where is your code?", choose "Azure Repos Git".
  • Select the XamarinPipelineDemo repo.
  • Select "Existing Azure Pipelines YAML file".
  • Select the XamarinPipelineDemo/XamarinPipelineDemo.Android/AzureDevOps/pipeline-android.yml as the existing yaml file.

Run the pipeline and you'll have to click a few things to give the pipeline access to these secure files and secret variables. To grant permission to the pipeline, you might have to go to the run summary page (see Screenshots folder).

Explanation Of The Journey: Deadends, Pitfalls, Solutions

Note that things may change after this demo was made (2020-12-19). Some limitations may go away, and some workaround no longer needed. I'd love to hear about them if you ever encounter a way this demo should be updated.

Must Use MacOS Agent

First of all, doing Xamarin.UITest tests on Microsoft-hosted agent in an Azure DevOps pipeline has some important constraints. Microsoft-hosted agents for Window and Linux run in a virtual machine that can not run the Android emulator, so only the MacOS agent can run the Android emulator (MS docs page).

With a self-hosted agent that is not a virtual machine, you can use any of the three OSes. With App Center, the tests are run on a real device, and there is no need to run the Android emulator, so again, you can use any of the three OSes.

MacOS Agent Pitfalls

The MacOS agent has a few pitfalls to watch out for.

  • MacOS must use Mono when dealing with .NET Framework stuff (originally made just for Windows). So, .NET Framework stuff that works on your Windows machine may not do so well in the pipeline.
    • Try to make your project target .NET Core or .NET 5 where possible, especially your unit test project.
    • You can't use DotNetCoreCLI task on a MacOS agent to run test projects that target .NET Framework. Mono's open issue 6984 says that you can do "dotnet build" on a .NET Framework project, but you can't "dotnet test".
  • Xamarin.UITest MUST be .NET Framework, so you can not use DotNetCoreCLI task to run Xamarin.UITest tests.
  • MacOS agent also doesn't support VSTest or VsBuild tasks.
  • The only thing left to do for Xamarin.UITest is a MSBuild task to build it, then directly run nunit3-console to run the Xamarin.UITest tests.
  • MacOS agents are case sensitive for path stuff while Windows is not, so make sure your pipeline stuff is case-appropriate.
  • On Windows, you might be used to using Unix-inspired PowerShell aliases like "ls" and "mv". Do not use those aliases. In MacOS, even inside a PowerShell script, commands like "ls" will invoke the Unix command instead of the PowerShell cmdlet.

MacOS Agent Directories

During pipeline execution, there are three major directories to think about:

  • Build.SourcesDirectory, which is often /Users/runner/work/1/s/.
  • Build.BinariesDirectory, which is often /Users/runner/work/1/b/.
  • Build.ArtifactStagingDirectory, which is often /Users/runner/work/1/a/.

The repo for your pipeline is automatically put in the Build.SourcesDirectory. The other two directories are just natural places for you to put stuff. For instance, build outputs to Build.BinariesDirectory and special files you want to download later (artifacts) to Build.ArtifactStagingDirectory. The PublishBuildArtifacts task even defaults to publishing everything in the Build.ArtifactStagingDirectory.

Fresh Autogenerated Pipeline

If you make a new pipeline for a Xamarin Android app, you get an autogenerated yaml file that...

  • Triggers on your main or master branch.
  • Selects desired agent type (the pool section vmImage value).
  • Sets buildConfiguration and outputDirectory variables.
  • Does usual nuget stuff so you download the nuget packages used by your solutions.
  • The XamarinAndroid task builds all "*droid*.csproj" projects (probably just one for you), generating an unsigned APK file.

That's it. You can't even access the unsigned APK file after the pipeline runs; you just get to know whether the agent was able to make the unsigned APK.

I'll explain how and why we add to the pipeline to accomplish the goals I mentioned in the introduction.

General Pipeline YAML Advice

You're going to have to learn nuances of yaml. If you don't already know yaml and the unique quirks of pipeline yaml, it's going to trip you up somewhere.

Pipeline Variables

One of the first learning hurdles for dealing with pipeline is learning enough to use variables effectively.

The variable section in a fresh autogenerated pipeline looks like this...

variables:
  name1: value1
  name2: value2

...which is nice and compact. But if you need to use a variable group, you have to go with the more verbose way...

variables:
  - group: nameOfVariableGroup
  - name: name1
    value: value1
  - name: name2
    value: value2

I still haven't read the entire MS Docs page on pipeline variables because it is so long. Unfortunately there are three different syntaxes for referencing variables. You can mostly use macro syntax, which looks like $(someVariable) and leads to the variable being processed just before a task executes. Macro syntax can not be used in trigger or resource sections, and can not be used as yaml keys.

If the pipeline encounters $(someVariable) and doesn't recognize someVariable as a variable, then the expression stays as is (because maybe it'll be usable by PowerShell or whatever you're executing).

So, if you get errors that directly talk about $(someVariable) rather than the value of someVariable, then someVariable isn't defined. You need to check your spelling, and if it's a variable from a variable group (defined in Library section of web interface), you need to explicitly reference the variable group in your variables section.

My pipeline yaml mostly uses macro syntax. One notable exception is runtime expression syntax ($[variables.someVariable]) in conditions and expressions, as is recommended. You can see the runtime expression syntax in my pipeline's step conditions, just search for "condition:" or "variables.". Another exception is Azure DevOps's surprising (but reasonable) way of setting/creating pipeline variables from scripts: outputting a line to standard output that conforms to logging command syntax; here's an example:

- pwsh: Write-Output "##vso[task.setvariable variable=someVariable]some string with spaces allowed"

Non-secret variables are mapped to environment variables for each task.

Pipeline Triggers

A freshly autogenerated pipeline might have a trigger section...

trigger:
- main

...which will make the pipeline trigger for every change to the main branch. But if you have multiple target platforms (android, iOS, uwp), each having their own pipeline, then you get a lot of unnecessary builds when you update something only relevant to one platform.

So, you probably want a path-based trigger. Note that wildcards are unsupported and all paths are relative to the root of the repo. Here's a trigger section for a hypothetical android pipeline...

trigger:
  branches:
    include:
    - main
  paths:
    include:
    # common
    - 'MyApp'
    - 'MyApp.NUnit'
    - 'MyApp.UITest'
    - 'Util'
    - 'XamarinUtil'
    - 'MyApp.sln'
    # platform
    - 'MyApp.Android'

Also, this path-based trigger stuff is why this demo's android pipeline yml file and android version script are under XamarinPipelineDemo.Android/AzureDevOps rather than under a root-level AzureDevOps folder. A change to these android-pipeline-specific file should only trigger an android pipeline build, and putting them under an android folder makes that easy trigger-wise.

Similarly, local_uitest_run.ps1 is in a LocalScripts folder instead of the XamarinPipelineDemo.UITest folder because changes to a local-use-only script should not trigger a pipeline build. There is also the option of having a XamarinPipelineDemo.UITest/LocalScripts folder and listing that folder in the yaml's trigger-paths-exclude list.

Pipeline Tasks

Some tasks support path wildcards in their inputs, some don't. Always check the task reference before using path wildcards. If you get an error message like "not found PathToPublish: /User/runner/work/1/a/*.apk", the fact that the path it couldn't find has a wildcard should make you double check whether wildcards are supported for that task input.

Sometimes the task is a wrapper around some tool, and the task's documentation doesn't go into much detail into the behavior of the tool. For instance, AndroidSigning is a wrapper around apksigner, and you have to get all the way down to the --out option section of the apksigner doc to learn that the absence of the option leads to the APK file being signed in place, overwriting the input APK.

Sometimes looking at the Azure pipeline tasks source code is useful.

Pipeline Scripts And Strings

In your pipeline, you might want to do something simple, like copy some files. Sometimes there is a task for what you want to do, like CopyFiles, but often there isn't. A good way to accomplish these small things is to use one of the script tasks...

  • Bash: runs on MacOS, Linux, and Windows.
  • BatchScript: runs on Windows.
  • CmdLine: uses bash on Linux and MacOS; uses cmd.exe on Windows.
  • PowerShell: runs on MacOS, Linux, and Windows.

I prefer PowerShell because...

  • It runs on all the agents in the same way.
  • It will run on people's local machines. It comes preinstalled in Windows and I think it's easy enough to install on Linux and MacOS.
  • I think it's the most capable of the languages. I think PowerShell helps keep simple tasks easy and can use anything in the .net ecosystem, like System.Collections.Generic.Dictionary<K,V>, which is especially nice for Xamarin developers.

In fact, I learned PowerShell because of dealing with Xamarin pipelines, and PowerShell is now my go-to language for quick Windows scripts.

There are a few ways to do scripts in pipelines, but first you should understand yaml multi-line strings. The > character causes the following indented block to be treated "folded style": as a single string with no line breaks (except one at the end). The | character causes the following indented block to be treated "literal style": as a single string with line breaks preserved. Good explanation of mult-line strings at this stackoverflow answer and yaml-multiline.info.

Here are some script examples...

- pwsh: SomeCommand | CommandReceivingPipedStuffFromPreviousCommand; SomeSeparateCommand
  displayName: 'some inline one-liner script'

- pwsh: |
    SomeCommand | CommandReceivingPipedStuffFromPreviousCommand
    SomeSeparateCommand
  displayName: 'some inline multi-liner script'

- task: PowerShell@2
  displayName: 'calling a PowerShell script file in the repo'
  inputs:
    filePath: '$(theScriptDir)/SomeScript.ps1'
    # '>' used so we can have multiple lines treated as one line
    arguments: >
      -SomeScriptArg "SomeValueInQuotes"
      -AnotherScriptArg AnotherValueShowingQuotesNotAlwaysNeeded

Note how | characters can appear in the scripts; that's totally fine.

Give Each Build An Increasing Android App Version

If the Azure DevOps pipeline is going to be making the APKs we'll be releasing, we need unique versionCode and versionName values for each build.

Reminder: versionCode is a positive integer that is used by Android to compare versions and is not shown to the user. versionName is text displayed to the user and that is its only use.

Short version: The 'Set build-based Android app version' task uses the YAML counter function on the pipeline name (Build.DefinitionName) to set the versionCode and the Build.BuildNumber to set the versionName. This task is executed right before the XamarinAndroid build task and calls a PowerShell script to modify the Android manifest file.

How To Set The Android App Version

James Montemagno's and Andrew Hoefling's "Mobile App Tasks for VSTS" (Azure DevOps plugin, Github repo) has an AndroidBumpVersion task that does half of the job: setting the versionCode and versionName.

Some people are not allowed to use Azure DevOps plugins (perhaps for security by their employer), so we will not use this as a plugin. Azure DevOps plugins are run via a Node server, so the plugin would use tasks/AndroidBumpVersion/task.ts, but thankfully James has also provided PowerShell and bash equivalents of his plugin tasks, so you can look at those files.

I went with his PowerShell script, fixed a bug, and cleaned it up (pull request 39, current code). The result is this demo's AndroidSetVersion.ps1.

(Note: recent versions of PowerShell are cross platform, so you can run PowerShell on MacOS and Linux. But again, be mindful of Unix commands overriding PowerShell aliases and you can't be case-insensitive.)

The essence of the script is that the Android manifest file is XML and inside the manifest element, set the android:versionCode and android:versionName attributes appropriately. Thankfully PowerShell has the XmlDocument class and the Select-XML cmdlet that gives you easy-to-manipulate SelectXmlInfo objects.

How To Choose The Version

The second half of the problem is how to have an increasing and meaningful versionCode and versionName. Azure DevOps pipelines will have pre-defined variables, including...

  • Build.BuildId: a positive integer that is build id that is unique across your organization and will appear in the build's URL (ex: dev.azure.com/SomeOrganization/SomeProject/_build/results?buildId=123456).
  • Build.BuildNumber: a string (not a number, especially if you set the name variable. The default format is "$(Date:yyyyMMdd)$(Rev:.r)", which looks like "20201231.7" and is unique only within the pipeline.
  • Build.DefinitionName: the name of the pipeline.

I think that the default Build.BuildNumber makes sense for versionName; it's unique, increasing, and easy for you to lookup the build/commit for the version name a user sees. I don't like Build.BuildId for versionCode because consecutive builds will probably not have consecutive versionCode values because of all the other builds in your Azure DevOps organization. Build.BuildId is probably just going to be a large, meaningless number for you.

Thankfully, Andrew Hoefling wrote “Azure Pipelines Custom Build Numbers in YAML Templates”, which shows how you can get a simple {1,2,3,...} progression for a build using the yaml counter function. MS docs on defining pipeline variables has a counter example too.

Here's a snippet that shows a simple pipelineBuildNumber that goes up {0,1,2,...} and a versionRevision that counts up but gets reset everytime you change the versionMajorMinor value.

variables:
  # for doing Major.Minor.Revision;
  # any time you change versionMajorMinor,
  # versionRevision uses a new counter
  - name: 'versionMajorMinor'
    value: '0.0'
  - name: 'versionRevision'
    value: $[counter(variables['versionMajorMinor'], 0)]
  # for doing simple pipeline build counter
  - name: 'pipelineBuildNumber'
    value: $[counter(variables['Build.DefinitionName'], 1)]

Build The APK File

Thankfully autogenerated android pipelines and internet examples give you a XamarinAndroid step that can build the apk for you. Here's the demo's step for that...

- task: XamarinAndroid@1
  inputs:
    projectFile: '**/*droid*.csproj'
    outputDir: '$(outputDir)'
    configuration: '$(buildConfiguration)'

One confusing thing though is some places will say to use the Gradle task instead of the deprecated Android build task. I am 90% sure Gradle is for native Android apps, not Xamarin. I do know that I've never had to use anything Gradle-related for my Xamarin stuff and XamarinAndroid seems fine.

Sign The APK File

Keystore Background

You'll want to sign the APK file so it can be installed on users' devices and distributed on Google Play. This repo already comes with a keystore file (remember: don't put your keystore in your repo; it should be more tightly controlled and uploaded as a secure file to Azure DevOps), but you can create your own keystore by following these MS Docs instructions (don't do the "Sign the APK" section).

You might get confused that if you make a keystore in Visual Studio, you have to choose a "keystore password", but not a "key password", and lots of other places talk about the "key password". The "key and certificate options" section of the apksigner doc might help you understand. A keystore can contain multiple keys, each identified by a key alias. The keystore itself password-protected, and each key might have its own password. This keytool example makes me think a common behavior is for a key password to default to the same as the keystore password.

One approach that has worked for me so far: when you are asked for a key password, and you don't recall there being a key password, you can probably put the keystore password.

Another confusion you may have is that the AndroidSigning task has an input named keystoreAlias (also called apksignerKeystoreAlias), but keystores do not have aliases; keys within keystores have aliases. You specify the keystore by the file name, then you specify the key by the key's alias. I have reported this misnaming as a problem on Developer Community.

AndroidSigning Task

This is the demo's AndroidSigning task (and required reference to appropriate variable group)...

variables:
  - group: android_demo_var_group

...

- task: AndroidSigning@3
  displayName: 'sign APK with example keystore'
  inputs:
    apkFiles: '$(outputDir)/*.apk'
    apksignerKeystoreFile: '$(androidKeystoreSecureFileName)'
    apksignerKeystoreAlias: '$(androidKeyAlias)'
    apksignerKeystorePassword: '$(androidKeystorePassword)'
    apksignerArguments: '--verbose --out $(finalApkPathSigned)'

Remember to follow the steps from Getting Started On Azure DevOps for uploading the keystore as secure file and creating the needed variable group with needed variables.

The task doc says it accepts wildcards for apkFiles. (Remember, don't assume tasks accept wildcards, check the task doc). Also, the doc states that the referenced keystore file must be a secure file, which should be fine for you. However, if you want to get around this restriction, you could use a PowerShell task to call apksigner directly.

Here is the error message if you try to use something other than a secure file for your keystore:

There was a resource authorization issue: "The pipeline is not valid. Job Job: Step AndroidSigning2 input keystoreFile references secure file /path/to/nonsecure/file which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

If you get errors like "can't find $(someVariable) secure file", that means the someVariable is not defined. Check that you are referencing the appropriate variable group in your yaml's variables section, and check that someVariable exactly matches what you have in your variable group.

By default, apksigner overwrites the APK file, and therefore the AndroidSigning task overwrites the APK file, which could be fine for you. But I wanted the signed APK to go into the artifact staging directory (path held in predefined variable Build.ArtifactStagingDirectory) with a particular file name (not the default com.demo.XamarinPipelineDemo.apk), so I used apksigner's --out argument.

Note that finalApkPathSigned puts the Build.BuildNumber and pipelineBuildNumber in the file name.

If you ever want to double check whether an APK has been signed, and by which keystore, use apksigner (possibly at C:\Program Files (x86)\Android\android-sdk\build-tools\SOME_VERSION\apksigner.bat). Do apksigner verify --print-certs THE_APK_PATH and the first line tells you about the key that signed the APK or DOES NOT VERIFY if not signed.

Likewise, for looking at keystore files, you can use keytool (possibly at C:\Program Files\Android\jdk\microsoft_dist_openjdk_1.8.0.25\bin\keytool.exe). keytool -v -list -keystore KEYSTORE_PATH will tell you about keys in the keystore, even if you provide no keystore password.

Publish The APK Files As Build Artifacts

I wanted to have both unsigned and signed APKs as build artifact.

Here is the MS Doc description of build artifacts and pipeline artifacts:

Build artifacts are the files that you want your build to produce. Build artifacts can be nearly anything that your team needs to test or deploy your app. For example, you've got .dll and .exe executable files and a .PDB symbols file of a .NET or C++ Windows app.

You can use pipeline artifacts to help store build outputs and move intermediate files between jobs in your pipeline. Pipeline artifacts are tied to the pipeline that they're created in. You can use them within the pipeline and download them from the build, as long as the build is retained. Pipeline artifacts are the new generation of build artifacts. They take advantage of existing services to dramatically reduce the time it takes to store outputs in your pipelines. Only available in Azure DevOps Services.

The "Pipeline artifacts are the new generation of build artifacts" makes me think maybe I should be producing pipeline artifacts instead of build artifacts, but build artifacts have been satisfactory so far. Publishing the APKs as a build artifact makes it easy for me to download the APKs generated by a build, and that's what I wanted. See [this screenshot](Screenshots/published artifacts.png) for how the web interface looks for displaying build artifacts, which can be downloaded by clicking on them.

The demo's PublishBuildArtifacts step for APKs...

- task: PublishBuildArtifacts@1
  displayName: 'publish APK artifacts'
  inputs:
    artifactName: 'apks'

Previously, an inline powershell script copied an unsigned APK file to Build.ArtifactStagingDirectory, and then AndroidSigning task created its APK and idsig outputs in Build.ArtifactStagingDirectory. PublishBuildArtifacts's pathToPublish input defaults to publishing the directory Build.ArtifactStagingDirectory, so the default works out. PublishBuildArtifact's source code suggests to me that published files are not removed, so keep that in mind when doing multiple publishes.

When you download the apks artifact, the download will be a zip file named apks.zip, which will contain an apks folder that will contain all the published files.

Note that pathToPublish does not support wildcards.

The demo does not specify the publishLocation input value, so the default of container is being used. I'm not sure what a container is, and I can't find anything that offers an explanation. There is this MS Doc about container jobs, but it talks about containers in the Docker sense. The publishLocation input reference says the container option will "store the artifact in Azure Pipelines" and that sounds good, and does make the artifact available for looking at and downloading when I view the build run. The alternate option for publishLocation is filePath, which copies the artifacts to "a file share", which I guess you'd have to set up

Build And Run Unit Tests

To build and run unit tests, DotNetCoreCLI will take care of...

  • Building the test project and its dependencies.
  • Discovering and running the tests in the test project.
  • Publishing the test results so you can see and explore them in Azure DevOps's web interface. This includes the build being marked with something like "90% of tests passing".

One requirement is that your test project is .NET Core or .NET 5. (Currently, "dotnet test" does not support Mono, but that may change.) Even if you get "dotnet test" to work on your Windows machine by making the project SDK-style ( format article, overview article), it won't work on the MacOS agent; you'll get errors about not having the references assemblies...

##[error]/Users/runner/.dotnet/sdk/3.1.404/Microsoft.Common.CurrentVersion.targets(1177,5): Error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks
/Users/runner/.dotnet/sdk/3.1.404/Microsoft.Common.CurrentVersion.targets(1177,5): error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [/Users/runner/work/1/s/XamarinPipelineDemo.NUnit/XamarinPipelineDemo.NUnit.csproj]

The step for unit tests is...

- task: DotNetCoreCLI@2
  displayName: 'unit tests'
  inputs:
    command: 'test'
    projects: '**/*NUnit*.csproj'
    configuration: '$(buildConfiguration)'

The projects input supports path wildcards. The code we are testing is already built with the Release build configuration, so if we build our test project with the same build configuration, we won't have to rebuild our dependencies.

Just so you know, if you dig in to the dotnet test doc, the configuration option defaults to Debug. The default build configuration is Debug for msbuild and other dotnet commands as well.

Remember that VSTest task is not available on MacOS agents. If you need an alternative to DotNetCoreCLI for testing, you'd have to do...

UI Tests In Azure DevOps

Set Up And Start Android Emulator

For setting up and starting the Android emulator, there are some good examples out there.

Eric Labelle's "Android UI Testing in Azure DevOps" article is for native Android apps, not Xamarin Android. The article covers more than just setting up the Android emulator. It talks about caching the AVD. I found that caching the AVD took the same or longer than just downloading the AVD fresh, but maybe I was doing something wrong.

Jan Piotrowski's azure-pipelines-android_emulator repo is good in that it gives you a pipeline yaml file with steps definitions for setting up and starting the Android emulator.

The MS Docs article "Build, Test, And Deploy Android Apps" has a section on starting the Android emulator.

You can see that the bash code in these articles are all pretty much the same. I think they're all derived from Andrey Mitsyk's comment on the azure-devops-docs issue thread about missing Android emulator documentation.

I made a few changes to Jan Piotrowski's pipeline steps for this demo...

variables:
  - name: adb
    value: '$ANDROID_HOME/platform-tools/adb'
  - name: emulator
    value: '$ANDROID_HOME/emulator/emulator'

# .. lots of stuff omitted here ...

- task: MSBuild@1
  displayName: 'build ui tests'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  inputs:
    solution: '**/*UITest*.csproj'
    configuration: '$(buildConfiguration)'

- bash: |
    set -o xtrace
    $ANDROID_HOME/tools/bin/sdkmanager --list
  displayName: 'list already installed Android packages'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-30;google_apis;x86'
  displayName: 'install Android image'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    $(emulator) -list-avds
    echo "no" | $ANDROID_HOME/tools/bin/avdmanager create avd -n uitest_android_emulator -k 'system-images;android-30;google_apis;x86' --force
    $(emulator) -list-avds
  displayName: 'create AVD'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    $(adb) devices
    nohup $(emulator) -avd uitest_android_emulator -no-snapshot -no-boot-anim -gpu auto -qemu > /dev/null 2>&1 &
  displayName: 'start Android emulator'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    $(adb) wait-for-device shell 'while [[ -z $(getprop sys.boot_completed | tr -d '\r') ]]; do sleep 1; done; input keyevent 82'
    $(adb) devices
  displayName: 'wait for Android emulator'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  timeoutInMinutes: 5

The set -o xtrace lines are so that script lines are printed before they are executed.

I wanted it to be easy to choose emulator/AppCenter/none for UI testing, so you can see the condition of steps depending on wantEmulatorUITests.

Some people like to put their UI test build step right after starting the Android emulator (and before the wait-for-Android-emulator step) to make better use of the long time that the Android emulator takes to get ready.

If there is an Android device that is especially beneficial to do emulator UI tests on, you can create an AVD for that device. The avdmanager command line reference seems incomplete, but googling will get you some avdmanager examples to learn from.

I've read that unsigned APKs can be installed on emulators, but I got the following error when trying to do a adb install unsigned.apk...

adb: failed to install /Users/runner/work/1/a/unsigned.apk: Failure [INSTALL_PARSE_FAILED_NO_CERTIFICATES: Failed collecting certificates for /data/app/vmdl1550487669.tmp/base.apk: Failed to collect certificates from /data/app/vmdl1550487669.tmp/base.apk: Attempt to get length of null array]

So, install a signed APK on your emulator, even if the APK is signed by the debug keystore.

Build UI Tests

Here's the step definition again...

- task: MSBuild@1
  displayName: 'build ui tests'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  inputs:
    solution: '**/*UITest*.csproj'
    configuration: '$(buildConfiguration)'

The situation is pretty simple once you learn that you can't use DotNetCoreCLI task for Xamarin.UITest project. MSBuild task is the only build task available on the MacOS agent (other than doing some script task that calls msbuild). You probably want to match the same build configuration that was used to compile the other projects, unless your UI test project has no dependencies that were built before.

Run Emulator UI Tests

Currently, dotnet test does not work with Mono, so the DotNetCoreCLI task does not work on the MacOS agent for running Xamarint.UITest tests. So, you have to run the tests via nunit3-console and publish the test results via the PublishTestResults task (which is nicely integrated into the Azure DevOps web interface for that build and for analysis across builds). For troubleshooting, you may want to publish the detailed UI test log (not the same thing as test results).

Here are the steps and relevant variable definitions...

variables:
  - name: uiTestDir
    value: '$(Build.SourcesDirectory)/XamarinPipelineDemo.UITest'
  - name: uiTestResultPath
    value: '$(Build.ArtifactStagingDirectory)/uitest_result.xml'
  - name: uiTestLogPath
    value: '$(Build.ArtifactStagingDirectory)/uitest.log'

# ... lots of stuff omitted here ...

- pwsh: |
    Set-PSDebug -Trace 1
    $env:UITEST_APK_PATH = "$(finalApkPathSigned)"
    $testAssemblies = Get-Item "$(uiTestDir)/bin/$(buildConfiguration)/XamarinPipelineDemo.UITest*.dll"
    nunit3-console $testAssemblies --output="$(uiTestLogPath)" --result="$(uiTestResultPath)"
  displayName: 'run ui tests'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  continueOnError: true
  timeoutInMinutes: 120

Note that nunit3-console defaults to putting the test results into ./TestResult.xml file, but the demo specifies a path for --result.

The demo app is installed as package com.demo.XamarinPipelineDemo, and the UI tests need to install package com.demo.XamarinPipelineDemo.test and the signatures of those two packages must match. Look at AppInitializer.cs for how it's done, but the basics is you either use ApkFile by itself or InstalledApp and KeyStore together. The ApkFile method is simpler. It even takes care of installing the APK onto the emulator.

If the app package and the test package don't have the same signature, you'll get an error like this:

System.Exception : Failed to execute: /Users/runner/Library/Android/sdk/platform-tools/adb -s emulator-5554 shell am instrument com.demo.XamarinPipelineFiddle.test/sh.calaba.instrumentationbackend.ClearAppData2 - exit code: 1
java.lang.SecurityException: Permission Denial: starting instrumentation ComponentInfo{com.demo.XamarinPipelineFiddle.test/sh.calaba.instrumentationbackend.ClearAppData2} from pid=5635, uid=5635 not allowed because package com.demo.XamarinPipelineFiddle.test does not have a signature matching the target com.demo.XamarinPipelineFiddle

If you get the error System.Exception : Timed out waiting for result of ClearAppData2 in your job log, and the detailed UI test log file contains...

AdbArguments: '-s emulator-5554 shell run-as com.demo.XamarinPipelineDemo.test ls "/data/data/com.demo.XamarinPipelineDemo.test/files/calabash_failure.out"'.
Finished with exit code 1 in 184 ms.
ls: /data/data/com.demo.XamarinPipelineDemo.test/files/calabash_failure.out: No such file or directory

...then the most likely explanation is that your app crashed. When you encounter this error, locally try out UI tests with the exact APK file that the pipeline was using on an Android emulator (not a real Android device). The crash might happen only in Release build configuration, or something else. Be aware that running the UI test on an Android emulator requires x86 to be one of the supported architectures (Android project properties => Android Options => Advanced => Supported Architectures).

The job log having the error System.Exception : Post to endpoint '/ping' failed after 100 retries. No http result received is most likely due to an app crash or not including x86 as a supported architecture. This error can also be due to problems with the agent pool (as in it's not your fault, you might have to re-run the build a few times and hopefully the problem passes).

Also, these "ClearAppData2" and "post to endpoint" errors might be followed by a TearDown: System.NullReferenceException error, and that is because the test runner still calls the test TearDown method, which might try to use the IApp object that was supposed to be set to the result of AppInitializer.StartApp. The real problem is the earlier errors, not the exception in TearDown.

The error Tcp transport error is something I've only seen due to agent pool problems. I just had to wait and retry a few times.

Publish Emulator UI Tests

Running nunit3-console will run the tests and generate a test result xml file and a test log file, but we still need to publish at least the test results...

- task: PublishBuildArtifacts@1
  displayName: 'publish ui test log artifact'
  inputs:
   artifactName: 'UI test log'
   pathToPublish: '$(uiTestLogPath)'
  continueOnError: true

- task: PublishTestResults@2
  condition: eq(variables.wantEmulatorUiTests, true)
  inputs:
    testRunTitle: 'Android UI Test Run'
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(uiTestResultPath)'
    # Android tests may randomly fail because of the System UI not responding (if you're using Prism);
    # see https://github.com/PrismLibrary/Prism/issues/2099 ;
    # tests may also fail due to pool agent problems;
    # using the following line still makes builds have warning status when UI tests fail
    # failTaskOnFailedTests: false

Publishing the UI test log as a general build artifact is for troubleshooting; the normal job log is pretty helpful, but you need to look at the UI test log in order to see any console printing your UI tests did.

Publishing the test results makes them nicely integrated with the Azure DevOps web interface and associated with the build.

Sometimes UI tests fail due to things like agent problems. Up to you whether you want to treat failing UI tests as warning or failure via the failTaskOnFailedTests input.

UI Tests In App Center

Set Up App Center

You'll need to have an App Center account, and you'll want to "Add new app". In the app's "Build" section, you'll select Azure DevOps for the service and select your Azure DevOps repo.

Then, the "Build" section will show you the repo branches, and you want to click on the wrench icon for the appropriate branch (most likely main or master). The wrench icon will be invisible until you hover over the branch info box.

Clicking on the wrench icon will bring you to build configuration settings for that branch; choose settings that make sense to you, but you will have to enable "Sign builds" and supply the appropriate keystore file and info.

Then go to the "Test" section, "Device sets" subsection, and create a new device set. You'll be using the device set name later.

Experiment With appcenter CLI

You should probably install the appcenter CLI and get some successful test runs with that before you try to use the AppCenterTest task in a pipeline. The appcenter CLI allows for much faster iteration, especially at the beginning. The CLI instantly tells you that you forgot a required argument, and the Azure DevOps pipeline might take minutes to make the same complaint. The CLI also uses your local files (APK, dlls, test-cloud.exe), so you don't have to wait for a pipeline build process either.

Check out LocalScripts/appcenter_uitest_run.ps1 for a working invokation of appcenter CLI. Note that appcenter CLI needs ANDROID_HOME and JAVA_HOME environment variables defined, which is taken care of by LocalScripts/common.ps1.

For convenience, here's a PowerShell snippet that calls appcenter CLI (the only optional argument is --test-output-dir):

appcenter test run uitest `
    --app "$orgName/$appName" `
    --app-path "$env:UITEST_APK_PATH" `
    --devices "$orgName/demo_device_set" `
    --test-series "master" `
    --locale "en_US" `
    --build-dir "..\$uiTestProjName\bin\$BuildConfiguration" `
    --uitest-tools-dir "..\$uiTestProjName\bin\$BuildConfiguration" `
    --test-output-dir $testOutputDir

Remember, you need to build the Xamarin.UITest project before you call appcenter; that's why the appcenter_uitest_run.ps1 script has a build step.

Run App Center UI Tests

The interplay between Azure DevOps and App Center is confusing, and I still don't fully understand it, but I will go over how I got my Azure DevOps pipeline to execute UI tests on real devices in App Center using the AppCenterTest task.

Here are the relevant pipeline steps for running the App Center UI tests (publishing test resuls in next subsection)...

################################################################################
# UI tests, preparatory steps common to AppCenter and emulator

- task: MSBuild@1
  displayName: 'build ui tests'
  condition: >
    and(
      succeeded(),
      or(
        eq(variables.wantAppCenterUiTests, true),
        eq(variables.wantEmulatorUiTests, true)
      )
    )
  inputs:
    solution: '**/*UITest*.csproj'
    configuration: '$(buildConfiguration)'

################################################################################
# AppCenter UI tests

# default nodejs version (v12) is not compatible with stuff used in AppCenterTest task
# https://github.com/microsoft/appcenter-cli/issues/696#issuecomment-553218361
- task: UseNode@1
  displayName: 'Use Node 10.15.1'
  condition: and(succeeded(), eq(variables.wantAppCenterUiTests, true))
  inputs:
    version: 10.15.1

- task: AppCenterTest@1
  condition: and(succeeded(), eq(variables.wantAppCenterUiTests, true))
  continueOnError: true
  inputs:
    appFile: '$(finalApkPathSigned)'
    appSlug: 'JacobEgnerDemos/XamarinPipelineDemo' # orgname or username, then '/', then app name
    devices: 'JacobEgnerDemos/demo_device_set' # uses same orgname or username, then '/', then device set name
    frameworkOption: 'uitest'
    runOptions: --test-output-dir "$(appCenterOutputDir)" # only needed if publishing test results in Azure DevOps
    serverEndpoint: 'AppCenterConnectionUserBasedFullAccess' # make a App Center user API token, then add service connection in Azure DevOps
    uiTestBuildDirectory: '$(uiTestAssemblyDir)' # directory that contains the uitest assemblies, not the build directory
    uiTestToolsDirectory: '$(uiTestAssemblyDir)' # build process puts test-cloud.exe in assembly dir

If you get errors like "Error: Command test prepare uitest ... is invalid", then you have a nodejs version problem. Unfortunately, (as of time of writing), Azure DevOps pipelines default to nodejs version 12, but AppCenterTest task requires nodejs version 10. Thus, the demo uses the UseNode task to set nodejs to version 10.15.1, just like this appcenter-cli issue 696 thread suggests.

Strangely enough, I can't find UseNode task doc via searching or looking through all the other task docs. There's NodeTool task doc, but looking at the source code of these tasks (usenode.ts, nodetool.ts) and comments, they don't seem to do the same thing.

The AppCenterTest task reference suggests that the prepareTests input defaults to true and therefore default behavior is to build the UI test project, but I never got that to work and had to use a separate MSBuild task. If the UI test project is not built, I think the first error you'll get is about not finding test-cloud.exe.

If you are successfully building the UI test project and still get a test-cloud.exe related error like this...

Preparing tests... failed.
Error: Cannot find test-cloud.exe, which is required to prepare UI tests.
We have searched for directory "packages\Xamarin.UITest.*\tools" inside "D:\" and all of its parent directories.
Please use option "--uitest-tools-dir" to manually specify location of this tool.
Minimum required version is "2.2.0".
##[error]Error: D:\a\_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.152.3\node_modules\.bin\appcenter.cmd failed with return code: 3

...then you need to be sure that your AppCenterTest task's uiTestToolsDirectory input is set to a folder that contains test-cloud.exe. There are a few existing discussions (like this azure-pipelines-tasks issue discussion) where people suggest pointing to Xamarin.UITest's nuget package folder, but you don't need to do that. When you build your Xamarin.UITest project, test-cloud.exe is put into the output folder alongside the generated dlls. So, I set uiTestToolsDirectory to that output folder.

Likewise, you might get an error complaining about a missing nunit.framework.dll...

Unable to find the nunit.framework.dll in the assembly directory. In Xamarin Studio you may have to right-click on the nunit.framework reference and choose Local Copy for it to be included in the output directory.

Preparing tests... failed.
Error: Cannot prepare UI Test artifacts using command: mono /Users/runner/work/1/s/packages/xamarin.uitest/3.0.12/tools/test-cloud.exe prepare "/Users/runner/work/1/a/XamarinPipelineDemo_20210108.2_1_Signed.apk" --assembly-dir "/Users/runner/work/1/s/XamarinPipelineDemo.UITest" --artifacts-dir "/Users/runner/work/1/a/AppCenterTest".

The NUnit library was not found, please try again. If you can't work out how to fix this issue, please contact support.

...because AppCenterTest task is looking for nunit.framework.dll in the assembly directory, and isn't finding it. It's possible you just haven't built the UITest project yet, or didn't assign the uiTestBuildDirectory input correctly. Even though the input name suggests a build directory, the reference page describes the input as "Path to directory with built test assemblies". So, I set both uiTestToolsDirectory and uiTestBuildDirectory to my UI test project's output directory where the dlls are generated.

You have to supply an appSlug input, or you'll get the error message Error: Input required: appSlug. The reference doc for that input says you need to specify it with format {username}/{app_identifier} and you can learn the values by looking at the URL of your app page in App Center: https://appcenter.ms/users/{username}/apps/{app_identifier} but username can also be your organization name and the URL might have format https://appcenter.ms/orgs/{orgname}/apps/{app_identifier}. My App Center app URL is https://appcenter.ms/orgs/JacobEgnerDemos/apps/XamarinPipelineDemo, so I used appSlug: 'JacobEgnerDemos/XamarinPipelineDemo'. I've personally tried this, and later I found this MS Docs page that agrees.

For the devices input, the AppCenterTest task reference is not helpful. The Starting A Test Run article says that for the appcenter cli, the devices argument can be the hexadecimal value or "the ID ... generated from the device set name". I had to experiment to figure out that the ID is not just the device set name. The device set ID is like the app slug: username or orgname, then '/', then device set name. My device set name is demove_device_set but the URL replaces the underscores with hyphens: https://appcenter.ms/orgs/JacobEgnerDemos/apps/XamarinPipelineDemo/test/device-sets/demo-device-set, and I once ran an App Center test run where I used hypens in the device set id, and it worked.

I recommend the named device set. In App Center, under Test and Devices sets, create a device set and name it.

If you don't want to use a named device set, you can determine the proper hexadecimal number by creating a test run in App Center, choosing a set of devices, and on the "submit" step, you'll be shown a command to "upload and schedule tests", and that command will contain something like --devices c2e61997. You don't actually have to submit the test run; you can just abandon the creation of the test run once you see the hexadecimal number.

The serverEndpoint input is required when using the default credentialsOption value of serviceEndpoint. The serverEndpoint input needs to specify the "service connection for AppCenter". Steps to create and use a service connection...

  • In App Center, create a full-access App Center user API token.
  • In Azure DevOps, go to your project settings, then pipelines section, then service connections entry.
    • Create a new service connection.
    • For service connection type, scroll down to the bottom and select "Visual Studio App Center".
    • Supply the API token and name the service connection (ex: AppCenterConnectionUserBasedFullAccess)
  • You'll either preemptively grant permission to all pipelines to use this service connection, or you'll have to click some stuff to approve the first time that a pipeline uses the service connection.
  • In pipeline definition, use the name of the service connection for your serverEndpoint input.

A read-only token will give you an Error: forbidden error. You need a full-access token.

It would be nice to make an app token (which only has access to one App Center app), but as of 2021-Jan, app tokens do not work. You'll get a Error: empty email address error if using an app token. You need to use a user token, which is associated with a user and has access to everything that user has access to. This stackoverflow discussion says that Microsoft's official advice as of 2020-Oct is:

For test you need to use the user level token only, app level token was not supported. Our test team was already working on this but currently there is no ETA on it.

Once they fix the app token issue, you can follow these instructions. to make an App Center app API token.

The artifactsDirectory input for AppCenterTest task defaults to $(Build.ArtifactStagingDirectory)/AppCenterTest. With normal inputs, the following files are output to that folder:

  • apps (folder)
    • XamarainPipelineDemo_20201231.1_9.apk (apk file put on devices to test)
  • AndroidTestServer.apk
  • manifest.json (mentions found dlls, found test methods, excluded tests, does not contain test results)
  • the rest of the files are all the dlls from the uiTestBuildDirectory
    • nunit*.dll (a bunch of NUnit dlls for running the tests)
    • Xamarin.UITest.dll
    • XamarinPipelineDemo.UITest.dll (the dll made from the ui test project)

These outputs are not useful. You already have these files elsewhere, except for manifest.json, which is still not useful.

Publish App Center UI Tests In Azure DevOps

By default, the results of your AppCenterTask tests are viewable in App Center. If you look at the log of the AppCenterTest task in Azure DevOps, you'll see a very brief "X passed, Y failed" summary and a link to a full test report at App Center.

In order to publish the full test results in Azure DevOps, you need to do a few extra steps. Here are the relevant pipeline steps (AppCenterTest is repeated for your convenience)...

- task: AppCenterTest@1
  condition: and(succeeded(), eq(variables.wantAppCenterUiTests, true))
  continueOnError: true
  inputs:
    appFile: '$(finalApkPathSigned)'
    appSlug: 'JacobEgnerDemos/XamarinPipelineDemo' # orgname or username, then '/', then app name
    devices: 'JacobEgnerDemos/demo_device_set' # uses same orgname or username, then '/', then device set name
    frameworkOption: 'uitest'
    runOptions: --test-output-dir "$(appCenterOutputDir)" # only needed if publishing test results in Azure DevOps
    serverEndpoint: 'AppCenterConnectionUserBasedFullAccess' # make a App Center user API token, then add service connection in Azure DevOps
    uiTestBuildDirectory: '$(uiTestAssemblyDir)' # directory that contains the uitest assemblies, not the build directory
    uiTestToolsDirectory: '$(uiTestAssemblyDir)' # build process puts test-cloud.exe in assembly dir

- pwsh: Expand-Archive "$(appCenterOutputDir)/nunit_xml_zip.zip" -DestinationPath "$(appCenterTestResultsDir)"
  displayName: 'unzip App Center test results zip'
  condition: and(succeededOrFailed(), eq(variables.wantAppCenterUiTests, true))
  continueOnError: true

- task: PublishTestResults@2
  displayName: 'simple-publish App Center UI test results'
  condition: and(succeededOrFailed(), eq(variables.wantAppCenterUiTests, true))
  inputs:
    testRunTitle: 'Android App Center UI Test Run (simple publish)'
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(appCenterTestResultsDir)/*.xml'

- pwsh: |
    Get-ChildItem "$(appCenterTestResultsDir)/*.xml" | ForEach-Object { Write-Output `
        ( "##vso[results.publish " `
        + "runTitle=Android App Center UI Test Run $($_.BaseName);" `
        + "resultFiles=$($_.FullName);" `
        + "type=NUnit;" `
        + "mergeResults=false;" `
        + "publishRunAttachments=true;" `
        + "failTaskOnFailedTests=false;" `
        + "testRunSystem=VSTS - PTR;" `
        + "]" `
        )}
  displayName: 'complicated-publish App Center UI test results with device name'
  condition: and(succeededOrFailed(), eq(variables.wantAppCenterUiTests, true))

First, we had to use AppCenterTest's runOptions input to specify a directory to output the test results; without the --test-output-dir option, test results won't be output to file at all.

To find out about --test-output-dir, I had to dig through the appcenter CLI source code. The README.md hints we are interested in the appcenter test run uitest command, but no documentation on options. I had to dig down into uitest.cs to find use of this.testOutputDir. Chasing testOutputDir around the code base made me think it was worth trying out, and it worked.

(Unfortunately, I never got the --merge-nunit-xml option to work. The option always caused me to get an error and I have filed an issue. Contact me if you ever get it to work.)

Once you specify a --test-output-dir to App Center test run, it'll make a nunit_xml_zip.zip file (other UI test frameworks will have different output file names, like junit_xml_zip.zip for junit).

That nunit_xml_zip.zip will contain xml files for each tested device, named like google_pixel_3_11_nunit_report.xml. The 11 is from the Android OS version, because you might test the Pixel 3 model with Android 10 and Android 11.

Azure DevOps supports a ExtractFiles task, but I went with an inline PowerShell script because I can test out the exact behavior on my system, rather than doing multiple pipeline runs to troubleshoot whatever I did wrong with the ExtractFiles task.

In Azure DevOps, when exploring test results published by the simple-publish step, the App Center UI test results will be named whatever you supplied as the testRunTitle input to the PublishTestResults task, and contain a "_1" style suffix if you tested more than one device. Currently, I don't know of a simple way to have the test results specify the used device.

But here is a complicated way to get your App Center test results labeled with the device info: do a PowerShell one-liner to publish each test result xml file with a test run title that uses the xml file name. The critical ingredient of the PublishTestResults task is that it will output something like the following to the log/stdout: ##[debug]Processed: ##vso[results.publish type=NUnit;mergeResults=false;runTitle=Android App Center UI Test Run;publishRunAttachments=true;resultFiles=/Users/runner/work/1/a/AppCenterTest/TestResults/google_pixel_10_nunit_report.xml,/Users/runner/work/1/a/AppCenterTest/TestResults/google_pixel_3_11_nunit_report.xml;failTaskOnFailedTests=false;testRunSystem=VSTS - PTR;]

This ##vso[results.publish ... magic spell is similar to how you can set pipeline variables by outputting ##vso[task.setvariable variable=someVariable]someValue,

So, my complicated-publish step just outputs that magic spell to stdout for each xml file. I don't know of anyone else who has done this.

Problems Accessing Stuff

You might have someone who can't access something, like build artifacts, regardless of permissions (and rememeber there are permissions under project settings, then permissions for pipelines, then permissions for EACH pipeline). The problem might be their “access level”. If their access level is “Stakeholder”, then it probably needs to be changed to "Basic" or better. “Basic”. You can check anyone’s organization-specific access level at URLs like this: dev.azure.com/TheAppropriateOrganization/_settings/users

Thanks To Those Who Helped Me