Testing UIs is challenging in any context, and mobile apps are no different.

This is how we started our previous article about UI testing in an Azure DevOps context. We learned out that we had to deep dive into it to answer all the questions raised by this concern. Fortunately, we ended up with a fully functional UI testing pipeline based on a custom Python script to upload UI artifacts.

Android UI Testing in Azure DevOps

We decided to take our solution a step further by wrapping up all the logic into a self-contained Azure DevOps extension.

It’s been in the making for a while now, but I finally did it! My first Azure DevOps custom build task is now available for anyone to try.

Keep in mind, this is still an early preview and have been tested mostly with Android UI Test in mind.

About the extension

You can install it or find more information on the task from the Azure DevOps Marketplace and the Github repo.

The task has been used in our Android UI Testing pipeline for a few weeks now. Essentially, the task is a typescript adaptation from my previous python script (see referenced article).

• It must run after the PublishTestResult task since it’s using the reports published by that task.
• It will search for screenshots to upload on each test failure

What it can do?

• You can configure the search folder for the screenshot.
• Upload your screenshots in the right test result within the attachment tab

What it cannot do… yet?

• Configure JSON attributes used to compose the screenshot search path
• Configure the screenshot’s file extensions

FAQ

I’ve added the task to my pipeline, but screenshots are never in the resulting test reports?

The following results are possible (Make sure to check the “Enable system diagnostics” option on the pipeline to get more insight on the task status):

• Succeeded: Everything went well, you should see a message at the end.
• SucceededWithIssues: There can be two reasons: The upload failed for some/all screenshots OR some/all screenshots files are missing. A message will be detailing the situation within the task result logs.

##vso[task.complete result=SucceededWithIssues;]All screenshots were missing. 
Task completed. Published 0/2 screenshots

• Skipped: There is no test failure associated with the current build.

##vso[task.complete result=Skipped;]No test failures found.

• Failed: Most of the time you’ll get this status if you forgot to specify the mandatory parameter organization.

##vso[task.complete result=Failed;]Organization is mandatory

Everything is setup but I don’t see my screens.

It could be from a few problems. The first step is to run your pipeline with Diagnostic, it should provide a better understanding of what happened.

• The search folder is wrong and the task can’t find your screenshots

2 tests failed. Will proceed with screenshot upload.
##vso[task.debug]screenshotFolder=./app/mywrongfolder
##vso[task.debug]Searching for image at path: ./app/mywrongfolder/com.sampleapp.ExampleUiTest/onNetworkError_errorDialogIsShown.png
##vso[task.debug]Failure - No screenshot found for com.sampleapp.ExampleUiTest/onNetworkError_errorDialogIsShown

What’s next?

• Unit test the task
• Setup a CI/CD pipeline to distribute this task automatically

How can you help?

• Test in your UI pipeline
• Test with NUnit
• Test on Azure Server

I’ve made it this far all by myself so I can’t wait to see what you guys can bring to this task and make it even better.

Fell free to add feature requests or bugs using the GitHub issues. Your help is, of course, welcomed.

Thanks and happy testing! :)

Testing UIs is challenging in any context, and mobile apps are no different. Furthermore in a CI context, it raises questions such as:

• How should we configure the test environment to emulate at best our physical devices ?
• What are the test result artifacts and how should we leverage them to make our pipeline relevant ?

At Genetec we massively rely on Azure DevOps for our CI/CD pipelines. While it is a satisfying solution most of the time, it is not necessarily mobile developer oriented and those questions will have to be answered by ourselves.

Even though I spend my time developing Android applications, I have quite a long background with CI/CD having passed through several platforms in the last few years still Azure DevOps remains a challenge for me. The best way to master something new is to deep dive into it.

We already had a great setup in place, but one critical piece was missing: UI Testing.

In this guide we’ll go over all the steps to integrate UI Testing on Android in your Continuous Integration pipeline using Azure DevOps starting with screenshot capture. Buckle up and let’s begin.

Capture screenshots when a test is failing

This part is essential in order to understand what’s going on during the test. We’ve all had to deal with tests failing on the CI but passing when executed locally. There’s plenty of articles already out there on how to do it so I won’t get into more details here. The best guide I’ve found and used is this article from Piotr Zawadzki

How to capture screenshots for failed UI tests

Using his guide you can get creative but the only essential part to respect for the following steps to work is the path where screenshots are pulled to and the naming convention.

Create the pipeline

Moving on the good stuff: The Pipeline! You can create a new one or add a stage or job to your existing one. I used a separate pipeline for a couple of reasons:

• Our current CI pipeline (compile → lint → unitTest → assemble) is still using the classic editor rather than YAML.
• It’s run as soon as the PR is open and the targetBranch is develop or release/* which is not what I want for the UI Tests. I want to be able to run UI Tests on any branch at any given time.
• I also want the pipeline to run only if the CI completes successfully (which could also be achieved with a new stage on the existing pipeline).

Take note that it is running on Azure DevOps with Hosted macOS 10.14 agents. These are Standard_DS2_v2 machines so definitively not the fastest, but it will get you there.

The pipeline consists of 6 steps:

1. Setup caching
2. Download the AVD
3. Create and launch emulator
4. Assemble APKs and launch instrumented tests
5. Publish test results
6. Upload screenshot for test failure

Setup Caching

Caching is optional and still experimental but will save you time. Make sure to read the Q&A and limitations in the official documentation. You can also look at the code on Github (CacheBetaV0, CacheBetaV1)

Pipeline caching can help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost to recreate or redownload the same files again. Caching is especially useful in scenarios where the same dependencies are downloaded over and over at the start of each run. This is often a time consuming process involving hundreds or thousands of network calls.

— DevOps documentation

I discovered 2 things that could reduce the pipeline’s duration:

• Caching the AVD: This is currently the only one I use (might be eventually discarded, more on this later). On average we save ~ 30 sec on the download time of approximately 1 min to 1 min and 15 sec
• Caching Gradle: I currently removed this part as it was unstable in V0. That’s where the most time would be saved, but the task is not yet ready for Gradle caches (see issue)

- task: CacheBeta@0
  displayName: 'Caching System Images for AVD'
  inputs:
    key: 'AVD_IMAGES_PIXEL_28'
    path: '$(ANDROID_HOME)/system-images'
    cacheHitVar: 'AVD_IMAGES_RESTORED'
  continueOnError: true
  condition: succeededOrFailed()

The cacheHitVar will be used in the next step to determine if we were able to successfully retrieve the AVD from the cache.

• You can take a look at Azure Pipeline CacheBeta known issues and feature request.
• Restore keys were added in V1, but I haven’t yet tried it and it is still undocumented.
• One feature I’m looking forward to is compression support which, in some cases, will save even more time.

Download the AVD

Next up is the AVD download and installation. We have to download our desired AVD system-image as it is not built-in with the Azure OSX images yet. Choose the API level you want for your emulator and let’s install it.

- bash: | 
    echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-28;google_apis;x86' 
    echo "AVD system-image successfully downloaded and installed." 
  displayName: 'Download and install emulator image' 
  condition: ne(variables.AVD_IMAGES_RESTORED, 'true')

We use the sdkmanager to download and install the system-image, but the key in this second task is the ne(variables.AVD_IMAGES_RESTORED, 'true') which lets us skip this step if the 1st step does not end in a cache missed.

Create and launch a Google Pixel emulator

Once the AVD image is downloaded and installed, we can create and launch the emulator.

- bash: | 
    echo "no" | $ANDROID_HOME/tools/bin/avdmanager create avd -n android_emulator -k 'system-images;android-28;google_apis;x86' -d 17 --force 
    echo "Emulator created successfully $(ANDROID_HOME/emulator/emulator -list-avds), launching it" 
    nohup $ANDROID_HOME/emulator/emulator -avd android_emulator -skin 1080x1920 -no-snapshot -no-audio -no-boot-anim -accel auto -gpu auto -qemu -lcd-density 420 > /dev/null 2>&1 & 
    $ANDROID_HOME/platform-tools/adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed | tr -d '\r') ]]; do sleep 1; done' 
    $ANDROID_HOME/platform-tools/adb devices echo "Emulator started" 
  displayName: 'Create and start emulator'

Using the avdmanager in the Android SDK, we can create the AVD for our emulator. To recall what this task does, it creates an AVD named android_emulator with the x86 system-image for API 28 (Pie).

An important thing not to forget is the -d 17 which is essentially the device definition for a Google Pixel. I learned the hard way that specifying -d 17 on AVD creation is not enough. You will need some additional flags when launching the emulator:

Required

• -qemu -lcd-density 420: Not sure why, but when creating a Pixel Device with Android Studio, the dpi is 420 but using the avdmanager it uses 480 which is not good.

Optional

• -no-boot-anim: Usually a bit quicker to boot.
• -no-audio: No need in our test, so I prefer to keep it disabled.
• -no-snapshot: Should not be necessary since Azure Hosted images are always fresh but I kept it.
• -gpu auto -accel auto: Just making sure the emulator uses gpu and acceleration
• -skin 1080x1920: The Android SDK bundled with Azure macOS images are missing the $ANDROID_HOME/skins folder, so you will have to specify one manually if you want to use skins.

If you want to change the target device or would like to run on multiple devices, you can see all the available ones using the following command:

> $ANDROID_HOME/tools/bin/avdmanager list device
id: 0 or "tv_1080p"
id: 1 or "tv_720p"
id: 2 or "wear_round"
id: 3 or "wear_round_chin_320_290"
id: 4 or "wear_square"
id: 5 or "Galaxy Nexus"
id: 6 or "Nexus 10"
id: 7 or "Nexus 4"
id: 8 or "Nexus 5"
id: 9 or "Nexus 5X"
id: 10 or "Nexus 6"
id: 11 or "Nexus 6P"
id: 12 or "Nexus 7 2013"
id: 13 or "Nexus 7"
id: 14 or "Nexus 9"
id: 15 or "Nexus One"
id: 16 or "Nexus S"
id: 17 or "pixel"
id: 18 or "pixel_c"
id: 19 or "pixel_xl"
id: 20 or "2.7in QVGA"
id: 21 or "2.7in QVGA slider"
id: 22 or "3.2in HVGA slider (ADP1)"
id: 23 or "3.2in QVGA (ADP2)"
id: 24 or "3.3in WQVGA"
id: 25 or "3.4in WQVGA"
id: 26 or "3.7 FWVGA slider"
id: 27 or "3.7in WVGA (Nexus One)"
id: 28 or "4in WVGA (Nexus S)"
id: 29 or "4.65in 720p (Galaxy Nexus)"
id: 30 or "4.7in WXGA"
id: 31 or "5.1in WVGA"
id: 32 or "5.4in FWVGA"
id: 33 or "7in WSVGA (Tablet)"
id: 34 or "10.1in WXGA (Tablet)"

Assemble APKs and launch instrumented tests

For the 4th step we will be using the standard gradle task from Android projects.

- bash: | 
    ./gradlew connectedDebugAndroidTest --stacktrace 
    ./gradlew --stop 
  displayName: 'Run Instrumented Tests' 
  continueOnError: true

One important thing is to make sure the continueOnError: true is present otherwise all the following steps will be ignored.

Publish test results

Next up, we publish test results using the standard PublishTestResults Azure task.

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFiles: '**/outputs/androidTest-results/**/TEST*.xml'
    failTaskOnFailedTests: true
    testRunTitle: 'Test results'
  condition: succeededOrFailed()

Notice two important details from this step:

1. failTaskOnFailedTests: true → this is needed since we allowed continueOnError on the previous step.
2. condition: succeededOrFailed() → without this flag, the step will be skipped as soon as there is a test failure.

Upload screenshots for failing tests

Here comes the whole reason behind this article.

Why?

Why do we have this additional step?

Well, it turns out the task PublishTestResults only handles attachments for Visual Studio Test (TRX), at least for now! I looked through and through for a solution to add our screenshots in the famous attachment tab in the test results.

Finally I found an explanation. It turns out we have two options for JUnit.

If you are not using the Visual Studio Test task, you must publish attachments in your test results in a different way.

• You can use the Copy and Publish Build Artifacts task to publish any additional files created in your tests. These will appear in the Artifacts page of your build summary.
• Use the REST APIs to publish the necessary attachments.

The NUnit and JUnit test result formats do not have a formal definition for attachments in the results schema, so the Publish Test Results task cannot publish attachments when these formats are used.

The first option is really not optimal nor convenient so I opted for the second approach and wrote a python script using their REST API.

It consists of three steps :

1 — Get the runId → Getting the runId is quite simple using the following api.

GET https://dev.azure.com/{organization}/{project}/_apis/test/runs?buildUri={buildUri}

You need to pass the current buildUri which is available through the pipeline variable $(Build.BuildUri).

2 — Get the test failures → Look for test failures in this testRun using the runId from previous step.

GET https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/results?outcomes=3&api-version=5.1

I added the additional parameter outcomes=3 in order to only retrieve failures (outcomes=Failed is also valid → got the enum values from here).

3 — Upload screenshot → For each test failure:

Remember that we saved our screenshots using the className and testName.

• Get the testCaseResultId from the JSON response located under this path: value.id
• Check if a file exists in the screenshot folder using two attributes from the JSON response value.automatedTestStorage/value.testCaseTitle. (i.e. with a UI Test testWhenEmpty_shouldFail located in a class named MyFirstUiTest, the screenshot would be under the following path: ./app/build/reports/androidTests/connected/screenshots/failures/com.testpackage.MyFirstUiTest/testWhenEmpty_shouldFail.png)
• Encode this image in base64 format
• Upload the screenshot if found in the previously mentioned directory

POST https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/Results/{testCaseResultId}/attachments?api-version=5.1-preview.1

Here’s the whole python script combining these three steps

Once we have our python script operational, we can add it as our final task in the pipeline.

- task: UsePythonVersion@0
  displayName: 'Use Python 3.x'
  condition: succeededOrFailed()
- bash: 'pip install requests'
  displayName: 'Install Python dependencies'
  condition: succeededOrFailed()
- task: PythonScript@0
  displayName: 'Upload screenshots'
  continueOnError: true
  condition: failed()
  inputs:
    scriptPath: './ci-cd-scripts/uiTestUploadScreenshot.sh'
    arguments: '$(System.TeamProject) $(azure.api.pat) $(Build.BuildUri) yourOrganizationName'

You need to adjust three things for this to work:

• Make sure to adjust the scriptPath to reflect the location of your python file in your project. Note that, Android projects can’t have file with .py extensions, that’s why my python script is in a .sh file.
• Change yourOrganizationName argument with yours.
• Also, you’ll have to configure a PAT (personal access token) with Test Management permissions and configure this PAT in a variable group to share with the pipeline (mine was namedPAT for API in the YAML)

Finally, we now have a complete pipeline to run our UI tests and consult our screenshots for test failures in the test result report. Here’s how it will look like:

And here’s the full YAML for the pipeline.

Next steps

• Bundle the python script in an Azure DevOps Pipeline Task (ongoing)
• Optimize execution time
• Improve caching setup
• Submit a PR to include AVD images in the osx image (documented issue here)
• Extract pipelines to another repository (patiently waiting for multi-repo support)

PublishTestResultScreenshot — Sneak peek

All in all, the python script has been working really well for the last few months but why stop there?

We can do better!

Other teams could benefit from this and improve it!

So I dug deeper in the Azure documentation and learned about custom Azure DevOps Pipeline Task. With this custom task we will be able to remove the last three tasks from our pipeline including the whole python script and replace it with one shiny new DevOps task.

Current status

Development is well underway and I have a working task currently under review and refinements whilst into the process of open-sourcing it.

I would also like to add translations, unitTests, and make it compatible with any kind of project since it’s not really an Android specific need.

Thanks for your time, hoping you learn something today.

Feel free to reach out and/or follow me for more tips and trick mostly related to Android Development and off course stay tuned, part2 is coming as soon as the PublishTestResultScreenshot task is made public.

Android UI Testing in Azure DevOps was originally published in Genetec Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.