🦾 Custom Test Lab for Android 🤖 ?

What does it take to build a Firebase test lab alternative?

This blog can help you understand what happens behind the scenes when you run Android Tests on a Firebase Test Lab and how to build a Device farm for a custom TestLab to run Android Tests.

Open Sourced Now!

GitHub - oianmol/OpenTestLabAndroid: A self hosted Android test lab to run android automation…

There will be multiple parts for this blog post, Here’s the breakdown!

1. Introduction
2. The world of ADB
3. Test Results Processing
4. gRPC Service and Client Setup

Part 1: Introduction

Recently, I have been helping my new team to refactor the legacy codebase and in the process I am helping them learn about writing tests for Android/JVM and the team has been writing a lot of JUnit and Android Instrumentation tests for the features and a lot of code is being migrated from typical MVC to MVI with view-models, coroutines, with well defined data sources.

Motivation

The Android apps for the business (POS and Kitchen Display) run on custom cheap hardware, so we wanted to use those devices to execute our test suite, Hence the need came for building our own Device Farm, which we were able to set up successfully in our office.

About JUnit and Android Test suite.

JUnit Tests: We execute our JUnit tests on CircleCI during the build process,

We have a mono-repo setup and the tooling only executes the build/test if there are projects which are affected because of changes which is great because it helps us reduce the build times on CI also 😬

Android Tests: These tests run on Android devices, and are also called Instrumentation tests.

These tests run with UI or without a user interface but execute under Android.os Process, hence they need instruments like an Android Device.

since we deploy our apps to custom Android devices we had to run the test suite in our device farm post gradle build on CircleCI,

YES! We wanted the checks to fail ❌ if our Android Tests 🧪 are failing!

Since we had already ditched the Firebase test lab for cost and other factors it was time we find a way to connect our CircleCI build process with our device farm situated in a remote location.

Part 2: The world of ADB!

How ADB (Android Debugging Bridge) adb-server 🦾 helped us!

Usually you run Android Tests via Gradle using the Android Gradle Plugin, but we can also run Android test via the command line, and you can read more about it here.

Test from the command line | Android Studio | Android Developers

ADB shell provides us a way to execute Instrument tests via the CLI, a basic command looks like the following

and of course, you can customise it for your needs,

For example: If you want to run Android tests for a certain package you can do something like, where com.android.test.package1 corresponds to the Android package containing instrumentation tests,

and also you can execute a specific test function from any class as well by specifying -e class ClassName#methodName

There are other use cases like test sharding and using the orchestrator for which this cli command can be modified to suit our needs.

Here’s the command that we are using to execute tests with the AndroidTestOrchestrator.

Part 3: Test Results Processing

How does CircleCI process the test results ?

Collect test data - CircleCI

Whenever Android Tests are executed, our custom InstrumentationRunListener writes the XML test reports to the Android Device’s external storage directory, these results are later pulled via adb pull command from the Android device.

Get Anmol Verma’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here are some of the methods that are overrided and used by our CustomInstrumentationRunListener.

Here’s an example of a JUnit XML Format, The test results are generated in this format so that CircleCI can read the reports and provide us with metrics.

you can read more about it here

JUnit XML format

We had already started uploading the test results for the JUnit test suites and we wanted to do the same for Android Tests.

If you execute the Android tests via ./gradlew connectedAndroidTest you end up with a proper test suite XML report generated with the AGP, but since we were running the tests via adb shell these XML reports were not generated for us.

You can check how the Android Gradle Plugin generates reports here.

We planned to generate similar test-suite reports for our Android Tests as well, by using the custom InstrumentationRunListener as described above.

Part 4: gRPC Service and Client Setup

Let’s go through the steps which are executed during our tests

We expose these methods via the TestLabGrpcService.

The Following dependencies are injected into the TestLabGrpcService

Let’s go through each grpc method and try to understand what they do at a high level.

uploadTestApk

This method is called once we have the application’s debug.apk and androidTest.apk generated during the CircleCI process and we upload these apks with the gRPC client via our custom GradleTask which runs after a successful ./gradlew assemble!

Our Device Farm Service then stores these apks in a working directory on the device farm server which is identified by the Circle CI build number and branch name.

executeTests

With this rpc method, We look for available devices to execute our android tests on and we ask the device farm to use the uploaded apk(s) from the previous step and install them to available devices, so both androidTest and androidDebug APK are installed to each and every device.

We are also using test sharding, which means we can divide our tests and run them on multiple Android devices for faster build times, Here’s an example of how we define the numShards and shardIndex for each adb device.

also, before we start tests we mark the device state as State.BUSY so that no other operations are done during the tests executing on these devices and State.FREE once tests are completed,

We also wait until we have a decent number of available devices to run the tests on else fail the build after N minutes of retrying/waiting.

pullReportFiles

🖥️ Once the adb shell command is done executing the tests, we have the test results available on the device, which are pulled using adb pull command and then everything related to the app and tests is deleted from the device, returning it to an original state.

📹 We also record videos for each test using the shell command while our InstrumentationRunListener is running, so it outputs separate videos for each test!

adb shell screenrecord /path/to/.mp4

📂 These test reports are already formatted as XML test suite files and can be fed to the Circle CI test reporting which also gives us metrics like how much time each test took and measure Metrics like P95.

Once we have test reports available from all the devices, since we have test sharding enabled, we will use this information to throw a GradleException 🤞🏼in case we have FAILING TESTS! ✌🏼

getDeviceFarmInfo

This gRPC method just gives us the state of the devices connected to the device farm :)

So to sum up we have the following different components connected to each other.

1. The Circle CI executes the Gradle task in the Custom Gradle plugin
2. The Gradle task connects to the device farm via gRPC and uploads the apk for test execution.
3. Once tests are complete we pull the test results and decide to throw GradleExecption if the tests are failing 😉

That’s everything that I could write for now, I might share more as I learn from my experience while working with this custom setup 🙌

I hope this helps and Thanks for reading 💚

I am available on X(Twitter), and reach out to me if you have any questions about this 🤓

#Android #UITesting #Infrastructure #Testing #Firebase