<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sean Wintermere on Medium]]></title>
        <description><![CDATA[Stories by Sean Wintermere on Medium]]></description>
        <link>https://medium.com/@seanwintermere170?source=rss-04e3ec93b532------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Sean Wintermere on Medium</title>
            <link>https://medium.com/@seanwintermere170?source=rss-04e3ec93b532------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:26:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@seanwintermere170/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evaluating Android Automation Tools With Deliberately Broken Builds]]></title>
            <link>https://medium.com/@seanwintermere170/evaluating-android-automation-tools-with-deliberately-broken-builds-7868f5d2ee09?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/7868f5d2ee09</guid>
            <category><![CDATA[androiddev]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[android-app-development]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 10:11:01 GMT</pubDate>
            <atom:updated>2026-11-17T10:11:01.563Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_WMer0981woeLKE3nSns6A.png" /></figure><p>How do you know your Android tests would actually catch a bug?</p><p>Most teams point at a green build. But a green build only shows that the tests passed. It says nothing about whether they’d fail when something breaks.</p><p>That gap makes it hard to pick the best Android app automation testing tools, because most Android UI automation tools can tap, type and check text just fine.</p><p>So I judge Android UI automation tools by breaking things on purpose. I take one working build, plant five realistic bugs in copies of it, and check which tools would turn red. Here’s the method, the five bugs, and how six tools score.</p><h3>My Evaluation Method: One Working Build and Five Faulty Variants</h3><p>To compare Android UI automation tools fairly, I make six versions of the same small screen, where a user adds and saves a delivery address.</p><ul><li><strong>One working build:</strong> This is the control. A good test suite must pass on it every time.</li><li><strong>Five faulty variants:</strong> Each one carries exactly one planted bug. A build flavor or feature flag switches each bug on, so the rest of the app stays identical.</li></ul><p>The same suite then runs against all six builds. A useful suite stays green on the working build, turns red on each faulty one and leaves enough evidence to find the bug without a rerun.</p><p>Each tool gets up to 2 points per defect, for 10 points in total.</p><ul><li><strong>2 points:</strong> The tool catches the defect with its own built-in APIs.</li><li><strong>1 point:</strong> The tool catches it only with extra code, another library or adb commands.</li><li><strong>0 points:</strong> The tool has no practical way to catch it.</li></ul><p>I rated these Android UI automation tools on the APIs each one offers, and I haven’t run these exact builds through every tool. When you run the five variants on your own app, your real results should replace my paper scores.</p><h3>Five Android Defects the Evaluation Should Detect</h3><p>I picked these five because each one breaks in a different layer of Android app automated testing. One is timing, one is Compose semantics, one is a system dialog, one is the activity lifecycle and one is storage.</p><h3>1. Delayed Validation Displays the Wrong Result</h3><p>The postcode field checks the value after a short pause, on a background thread. In the faulty build, an older check finishes after a newer one, so a valid postcode shows “Invalid postcode”.</p><p>A test only catches this if it waits for the final result. If it checks too early, it sees no message at all and passes.</p><h3>2. A Compose Control Exposes the Wrong State</h3><p>The “Set as default address” switch is a custom Compose control. In the faulty build, it looks on, but its semantics still say it’s off.</p><p>Screen reader users hear the wrong state. Android UI automation tools that read the semantics or accessibility tree should catch it, while a pixel-only check won’t.</p><h3>3. Permission Denial Leaves the Screen Unusable</h3><p>The “Use my location” button asks for location access. In the faulty build, tapping “Don’t allow” leaves a spinner on screen forever.</p><p>An app shouldn’t block the screen after a denial. Also, once a user taps Deny more than once, Android stops showing the system dialog for that permission. So the app needs a usable fallback, like typing the address by hand.</p><h3>4. Activity Recreation Deletes an Unsaved Address</h3><p>A user types half an address and rotates the phone. Android recreates the activity, and in the faulty build, the field comes back empty.</p><p>In Compose, this usually means the state used remember where it needed rememberSaveable. The test has to trigger a recreation and then check the field.</p><h3>5. Save Confirmation Appears Without Persisting the Change</h3><p>The user taps Save, and an “Address saved” message appears. In the faulty build, the address only lands in an in-memory cache and never reaches the database.</p><p>A test that checks the message passes. The only reliable check is to restart the app and look for the address again.</p><figure><img alt="macOS terminal window in foreground showing successful Android build tasks and files, with Android Studio IDE visible in the background." src="https://cdn-images-1.medium.com/max/1024/1*w_94ciJ-fbeUaZGV1zC8pA.png" /></figure><h3>Android App Automation Tools Evaluated Against These Defects</h3><p>Each of the six Android UI automation tools below gets a score out of 10, with the points it lost. I also show one short code example for the defect where the tool does best.</p><h3>1. BrowserStack App Automate</h3><p>BrowserStack App Automate runs your existing Appium, Espresso, Detox (for Android), Flutter and Maestro suites on real Android phones in the cloud. It doesn’t add a new way to find bugs, so its score follows the framework you run on it.</p><p><strong>Score: 9/10 when running Appium.</strong> It loses 1 point on delayed validation, because Appium has no automatic sync with the app.</p><p>Real devices help most with two of the five defects. The permission dialog’s layout and wording can change across Android versions and phone makers’ skins. Slower mid-range phones also make timing bugs like the postcode race easier to reproduce.</p><p>Every session records a video, device logs, Appium logs and network logs, and an AI agent helps analyze failures. The trade-offs are a paid plan and an extra network hop for each Appium command, which can make single tests slower than on a local emulator.</p><figure><img alt="BrowserStack App Automate dashboard showing a failed Android Espresso test named unsavedAddress_survivesActivityRecreation on a Google Pixel 8, highlighting an AssertionFailedWithCauseError where the address field cleared after activity recreation." src="https://cdn-images-1.medium.com/max/1024/1*1IGj45mhAMiyseYIftYBWA.png" /></figure><p><strong>My opinion:</strong> I’d use App Automate for the permission and timing variants on a handful of real phones before each release. It won’t fix a weak test. But when a deny path breaks only on one maker’s dialog, a real phone is where I’d expect to see it.</p><h3>2. Appium With UiAutomator2</h3><p>Appium drives the app from outside, through Google’s UI Automator, so it sees everything on screen. That includes system dialogs and the accessibility state of Compose controls.</p><p><strong>Score: 9/10.</strong> It loses 1 point on delayed validation, because it only has explicit waits and doesn’t know when background work is done.</p><p>It reads a Compose switch’s checked attribute, as long as the app sets testTagsAsResourceId so tags show up as IDs. It can tap “Don’t allow” in the system dialog or revoke access with mobile: changePermissions. It can rotate the device and fully restart the app to check the saved address:</p><p>driver.execute_script(“mobile: terminateApp”, {“appId”: “com.example.shop”})</p><p>driver.execute_script(“mobile: activateApp”, {“appId”: “com.example.shop”})</p><p>The weak spot is the postcode race. You must wait for the final message yourself, and a short wait can still pass on the stale result.</p><figure><img alt="VS Code running a TypeScript Appium test failing on a retry-permission element alongside an Android emulator showing a location permission dialog." src="https://cdn-images-1.medium.com/max/1024/1*_5y_SpIdcCaDrWGOeP1tww.png" /></figure><p><strong>My opinion:</strong> Among Android UI automation tools, Appium is the most complete black-box option here, and it works the same way on emulators and real phones. I’d accept slower tests for that reach. I’d just give the timing bug a wait for the exact final text.</p><h3>3. Espresso</h3><p>Espresso runs inside the app’s process and waits for the main thread to go idle before each step. That makes it fast and steady for View-based screens.</p><p><strong>Score: 6/10.</strong> It loses 1 point each on delayed validation, Compose state, permission denial and persistence.</p><p>Espresso only syncs with work posted to the main thread’s queue, so the postcode check needs an idling resource. It can’t match Compose nodes on its own, so you need the Compose test rule. It can’t tap system dialogs either, so the deny path needs UI Automator.</p><p>Recreation is its strong point. ActivityScenario.recreate() destroys and rebuilds the activity:</p><p>onView(withId(R.id.address)).perform(typeText(“12 Lake Road”))</p><p>scenario.recreate()</p><p>onView(withId(R.id.address)).check(matches(withText(“12 Lake Road”)))</p><p>For persistence, it can relaunch the activity, but the process stays alive. The in-memory cache survives, so the faulty build can still pass.</p><figure><img alt="Android Studio screen displaying a failed Espresso UI test “unsavedAddress_survivesActivityRecreation” in a Kotlin project, showing assertions, code editor, and Android emulator with the “Edit Address” screen." src="https://cdn-images-1.medium.com/max/1024/1*-QFjLUKkJtMVYFhStGEybQ.png" /></figure><h3>4. UI Automator</h3><p>UI Automator is Google’s framework for testing across apps and system screens. Its newer Kotlin API, stable since version 2.4, is much shorter to write.</p><p><strong>Score: 9/10.</strong> It loses 1 point on delayed validation, because waitForStable() only waits for the screen to stop changing. A stable screen doesn’t mean background tasks are done.</p><p>It reads Compose state through the accessibility tree and rotates the device with setOrientationLandscape(). Its tests can run outside the app’s process, so it can force-stop and restart the app to check persistence. For the deny path, it has a ready-made permission watcher:</p><p>uiAutomator {</p><p>startApp(“com.example.shop”)</p><p>watchFor(PermissionDialog) { clickDeny() }</p><p>onElement { textAsString() == “Use my location” }.click()</p><p>onElement { textAsString() == “Enter address manually” }.click()</p><p>}</p><figure><img alt="Android Studio showing a failed UI Automator permission denial test alongside a Pixel 8 emulator displaying a location access prompt." src="https://cdn-images-1.medium.com/max/1024/1*WGvFVVI0TfKrxwzhCdn-7w.png" /></figure><p><strong>My opinion:</strong> Of the Android UI automation tools here, UI Automator is the one I’d reach for first for the permission variant. It also sits well beside Espresso and Compose tests in one project. For a whole suite, though, its tests run slower and need more explicit waits than in-process tests.</p><h3>5. Jetpack Compose UI Testing</h3><p>Compose’s test APIs work on the semantics tree, the same data screen readers use. Every assertion waits for Compose to be idle, and a virtual clock moves time forward.</p><p><strong>Score: 7/10.</strong> It loses 1 point each on delayed validation, permission denial and persistence.</p><p>The Compose state defect is its home ground:</p><p>composeTestRule.onNodeWithTag(“defaultAddressSwitch”)</p><p>.performClick()</p><p>.assertIsOn()</p><p>For recreation, StateRestorationTester simulates saving and restoring state, which catches a missing rememberSaveable. It skips real activity recreation, though, so I’d still keep one real recreation test.</p><p>The virtual clock handles delays inside composables, but background work can still change the UI without the test knowing. So the postcode check needs waitUntil or an idling resource. It can’t touch the permission dialog, and it shares Espresso’s in-process limit on persistence.</p><figure><img alt="Android Studio code editor showing a failed Jetpack Compose UI test for a settings screen alongside the device emulator." src="https://cdn-images-1.medium.com/max/1024/1*kubkz4WI6XOXbqYmziBzxA.png" /></figure><p><strong>My opinion:</strong> If your screens are in Compose, I think these tests should be your base layer, because they’re quick and exact about state. I’d add UI Automator for anything outside the app.</p><h3>6. Detox</h3><p>Detox is a gray-box framework built for React Native apps. So it doesn’t fit a native Compose screen.</p><p><strong>Score: 6/10.</strong> It loses 2 points on Compose state and 1 point each on permission denial and recreation.</p><p>Its sync is the strongest here. It waits for network calls, timers and animations, so the postcode race shows up without manual waits. For persistence, device.launchApp({ newInstance: true }) kills and restarts the app:</p><p>await element(by.id(‘saveButton’)).tap();</p><p>await device.launchApp({ newInstance: true });</p><p>await expect(element(by.id(‘addressField’))).toHaveText(’12 Lake Road’);</p><p>Permissions are the weak spot. Detox starts Android apps with all permissions granted, and its permissions option only works on iOS, so the deny path needs adb commands. Rotation also often won’t recreate a React Native activity, because the default project handles orientation changes itself.</p><figure><img alt="VS Code editor showing a failing Detox end-to-end test for a React Native username validation screen running on an Android emulator." src="https://cdn-images-1.medium.com/max/1024/1*pIu95bqpPzliETY7pjHA8g.png" /></figure><p><strong>My opinion:</strong> For a React Native app, Detox’s sync saves a lot of flaky-test pain. For a native Android app, I wouldn’t pick it at all.</p><h3>Compare Detection Quality and Failure Evidence</h3><p>Here’s how the six Android UI automation tools land on the same 10-point scale. Appium, UI Automator and App Automate (with Appium) score 9, Compose testing scores 7, and Espresso and Detox score 6.</p><p>Detection is half the job. The other half is what you get when a test fails.</p><ul><li><strong>Espresso:</strong> A failed match prints the full view hierarchy in the error, which is often enough to spot the problem.</li><li><strong>Compose testing:</strong> printToLog() prints the semantics tree, so the wrong toggle state shows up directly.</li><li><strong>UI Automator:</strong> The newer API takes screenshots of a window or element and attaches them to test results in Android Studio.</li><li><strong>Appium:</strong> You get the page source XML, screenshots and logcat, and screen recording takes a separate command.</li><li><strong>Detox:</strong> It saves screenshots, videos and logs on Android, but its UI hierarchy capture is iOS-only.</li><li><strong>BrowserStack App Automate:</strong> Each session keeps video, device logs, Appium logs and network logs in one place. That helps most when a bug shows up only on one phone.</li></ul><figure><img alt="Chart comparing Android UI automation tools: BrowserStack App Automate, Appium, Espresso, UI Automator, Compose UI Testing, and Detox, scored out of 10 on catching five specific app bugs." src="https://cdn-images-1.medium.com/max/1024/1*G6tfQtdbCx28JkvOCTjB2g.png" /></figure><h3>How I’d Choose an Android Automation Stack From the Results</h3><p>None of these Android UI automation tools caught all five bugs with built-in APIs alone. For Android app automation testing, that means a stack beats a single tool.</p><p>I’d let Compose tests and Espresso handle fast in-app checks on every pull request. UI Automator would cover permissions and system screens in the same project. Appium would run a smaller end-to-end suite that restarts the app and checks real persistence.</p><p>Before each release, I’d run the permission and timing variants on a few real phones, through a cloud like App Automate or an in-house lab. I’d only bring in Detox if the app is React Native.</p><p>Most of all, I’d keep the five broken builds and rerun them after every tool change or upgrade. A tool only belongs among the top Android app automation testing tools for my app if it turns those builds red.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7868f5d2ee09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tools for Automated Mobile App Testing: What to Look For and What to Use]]></title>
            <link>https://medium.com/@seanwintermere170/tools-for-automated-mobile-app-testing-what-to-look-for-and-what-to-use-283152ed7bfc?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/283152ed7bfc</guid>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 12:16:40 GMT</pubDate>
            <atom:updated>2026-11-16T12:16:40.243Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcD_rV7z3Qp_rBkgvRrz3Q.png" /></figure><h4>Compare top open-source frameworks, real-device clouds, and AI platforms to find the right mobile testing tool.</h4><p>Automated mobile app testing tools offer different ways to create, manage, and execute tests across mobile applications. Some focus on code-based automation, while others simplify test creation through AI or provide cloud infrastructure for testing on real devices.</p><p>Each tool comes with different capabilities, supported platforms, and maintenance requirements. The right choice will always depend on the application you are building and other technical requirements.</p><p>In this article, I assessed some of the leading tools for automated mobile app testing. I compared their features, limitations, pricing, and use cases to help you understand what to look for and which tool best fits your needs..</p><h3>Why Does Choosing the Right Mobile Testing Tool Matter for Automation?</h3><p>It is very important to choose the right automation tool for mobile testing because it directly impacts how effectively your team can <strong>automate the test suite,</strong> <strong>maintain</strong> <strong>test coverage, </strong>and<strong> manage execution</strong>. If you end up with a tool that does not support your applications architecture or the required device capabilities, it can leave critical user flows untested.</p><p>Along with this, <strong>team expertise</strong> matters too. The code-based framework requires programming and maintenance skills. Meanwhile, the low-code platforms may have limitations when handling complex native interactions.</p><p>Therefore, If you choose a tool that fits in both brackets, that is the application workflow and the team’s capabilities, it will help you maintain a reliable test suite and reduce the maintenance overload.</p><h3>Do You Need a Framework or a Complete Testing Platform?</h3><p>To choose the right mobile testing solution, you first need to understand the difference between an open-source framework and a managed testing platform.</p><figure><img alt="Comparison table highlighting the differences between open-source testing frameworks and managed testing platforms based on cost, setup, device access, parallel testing, reporting and debugging, and ideal use cases." src="https://cdn-images-1.medium.com/max/1024/1*d72hN-ypPU5FMzZpNg4hWQ.png" /><figcaption>Open-source frameworks vs. managed testing platforms: Key differences at a glance</figcaption></figure><p>As the comparison shows, frameworks offer greater control but require more infrastructure management. Managed platforms reduce that workload by providing hosted devices and built-in testing capabilities.</p><ul><li><strong>Choose an open-source framework</strong> if your team has the engineering skills and infrastructure to manage test execution, reporting, and maintenance.</li><li><strong>Choose a managed platform</strong> if you need broad device coverage and parallel execution without building and maintaining your own device infrastructure.</li></ul><h3>How I Evaluated The 9 Mobile Testing Tools</h3><p>I compared nine mobile testing tools across five criteria that influence test reliability, maintenance effort, and release efficiency:</p><ul><li><strong>Test stability:</strong> How well each tool handles asynchronous UI updates, animations, network delays, and device-specific behavior.</li><li><strong>Maintenance effort:</strong> How much work is needed to update tests when selectors, layouts, or UI components change.</li><li><strong>Execution efficiency:</strong> Support for parallel testing and its impact on test execution time.</li><li><strong>Debugging capabilities:</strong> Availability of screenshots, video recordings, device logs, network logs, and UI hierarchy details.</li><li><strong>Team fit:</strong> How well each tool aligns with a team’s technical skills, existing workflows, and maintenance capacity.</li></ul><p>My aim is to give you a clear, unbiased assessment of each tool’s real strengths, limitations, costs, and trade-offs so you can choose based on what works for your team, not vendor claims.</p><h3>My Top Picks for Automated Mobile App Testing Tools</h3><p>Below are my top mobile app testing tools across 3 categories- <strong>real-device cloud labs, AI-driven and low-code platforms, </strong>and<strong> open-source frameworks</strong>.</p><p>Let’s compare their features, limitations, pricing, and best-fit use cases to find the right tool for your team.</p><h3>A. Real-Device Cloud Labs (Best for Scaling &amp; Device Coverage)</h3><p>These platforms provide access to physical devices across models and OS versions. Here are the top options for teams that need broad device coverage without in-house labs.</p><h4>1. BrowserStack App Automate</h4><figure><img alt="Screenshot of the BrowserStack App Automate dashboard showing test session details, including build status as “Completed”, session names, frameworks like Appium, OS versions (iOS and Android), device models (such as iPhone and Google Pixel), and execution durations." src="https://cdn-images-1.medium.com/max/1024/1*j_ZUifyotNZyqePHr4wang.png" /><figcaption>BrowserStack App Automate dashboard overview</figcaption></figure><p>BrowserStack App Automate is a cloud-based platform for running automated Android and iOS app tests on real devices. It supports multiple automation frameworks and provides parallel execution, device-level testing capabilities, and centralized test reporting. Teams can test across device configurations without managing their own physical device labs.</p><p><em>Platform information checked: November 2026.</em></p><p><strong>BrowserStack App Automate’s Key Features:</strong></p><ul><li><strong>Parallel test execution:</strong> Run automated tests simultaneously across multiple real devices to reduce execution time and expand device coverage.</li><li><strong>Native device features:</strong> Test biometric authentication, camera image injection, network conditions, and physical SIM-based OTP workflows. Feature availability varies by plan and supported device.</li><li><strong>Local environment testing:</strong> Use BrowserStack Local to establish a secure tunnel between internal staging environments or local servers and BrowserStack cloud devices.</li><li><strong>AI-driven failure analysis:</strong> The Test Failure Analysis agent categorizes failures, analyzes logs, identifies potential root causes, and suggests actionable next steps.</li><li><strong>Test reporting and debugging:</strong> Access video recordings, screenshots, device logs, network logs, and framework logs through a consolidated test timeline.</li></ul><p><strong>=&gt; BrowserStack App Automate Supported Automation Frameworks:</strong> Appium, Espresso, XCUITest, Flutter, Detox, and Maestro.</p><p><strong>=&gt; BrowserStack App Automate CI/CD Integration:</strong> Supports integrations with GitHub Actions, GitLab CI, Jenkins, CircleCI, Bitrise, Azure DevOps, and Bamboo.</p><p><strong>=&gt; BrowserStack App Automate Debugging Artifacts:</strong> Video recordings, screenshots, device logs, crash logs, network logs, and framework execution logs. Available artifacts depend on the framework and session configuration.</p><p><strong>=&gt; BrowserStack App Automate Pricing:</strong> Device Cloud starts at<strong> $199/month</strong>, while Device Cloud Pro starts at<strong> $249/month</strong>, both billed annually for one parallel test. <strong>Enterprise pricing</strong> is available on request. A <strong>free trial</strong> includes 100 minutes of automated mobile testing.</p><p><strong>BrowserStack App Automate Limitations:</strong></p><ul><li>Requires a stable internet connection for reliable cloud-based test execution.</li><li>Costs increase with additional parallel test capacity.</li><li>Some advanced device features are restricted to specific subscription plans.</li></ul><p><strong>=&gt; BrowserStack App Automate is Best For:</strong> QA and engineering teams that need to scale automated mobile testing across real devices without maintaining physical device infrastructure.</p><h4>2. TestGrid</h4><figure><img alt="Screenshot of the TestGrid real device and browser cloud dashboard showing a list of available Android, iOS, and desktop browsers (including Pixel, iPhone, Samsung Galaxy, Chrome, Edge, and Safari) with Firefox 201 (Version 114) highlighted in a red box." src="https://cdn-images-1.medium.com/max/1024/1*CgNy4QAWDHl68B1XmHosjQ.png" /><figcaption>TestGrid’s cloud testing dashboard</figcaption></figure><p>TestGrid is a mobile testing platform for manual and automated testing of native, hybrid, and mobile web apps on real Android and iOS devices. Teams can use its cloud infrastructure or deploy dedicated device labs on-premise or in a private cloud.</p><p><strong>TestGrid’s Key Features:</strong></p><ul><li><strong>Flexible deployment:</strong> Offers cloud-hosted testing and dedicated on-premise or private-cloud device labs.</li><li><strong>Real-time device monitoring:</strong> Tracks CPU, memory, battery usage, network performance, and UI responsiveness during test execution.</li><li><strong>Device and network simulation:</strong> Supports GPS, biometric authentication, gestures, camera input, and network condition testing.</li><li><strong>Codeless automation:</strong> Provides record-and-playback workflows and AI-assisted test creation from natural-language and BDD test cases.</li></ul><p><strong>=&gt; TestGrid Supported frameworks:</strong> Appium, Espresso, XCUITest, and Detox for mobile testing; Selenium, Cypress, Playwright, and WebdriverIO for web automation.</p><p><strong>=&gt; TestGrid CI/CD integrations:</strong> Jenkins, GitHub Actions, GitLab CI, Azure DevOps, and CircleCI, with API support for custom pipelines.</p><p><strong>=&gt; TestGrid Debugging artifacts:</strong> Video recordings, screenshots, ADB logs, Appium Inspector, Charles Proxy workflows, HAR files, crash reports, and network logs. Availability varies by framework and configuration.</p><p><strong>=&gt; TestGrid Pricing:</strong> Starter costs $249 per seat/month. Growth and Custom Device Lab plans have custom pricing. Dedicated device labs support on-premise and private-cloud deployments.</p><p><strong>TestGrid Limitations:</strong></p><ul><li>On-premise and private-cloud deployments require infrastructure setup and ongoing device maintenance.</li><li>Starter plans have device and AI token limits, which may require teams with higher usage to upgrade.</li><li>Advanced deployment and device-lab requirements may involve custom pricing.</li></ul><p><strong>=&gt; TestGrid is best for:</strong> Enterprises that need dedicated real-device infrastructure and teams combining code-based and codeless mobile test automation.</p><h3>B. AI-Driven &amp; Low-Code Platforms (Best for Mixed-Skill &amp; Agile Teams)</h3><p>These platforms reduce coding effort in test creation and maintenance. Here are my top options for teams seeking faster automation with less scripting:</p><h4>3. ACCELQ</h4><figure><img alt="Annotated screenshot of the ACCELQ test execution report dashboard displaying run status, summary perspectives, test cases list, and detailed step-by-step execution logs." src="https://cdn-images-1.medium.com/max/1024/1*fGDb3sJgaQ54tm1XD9dj7A.png" /><figcaption>ACCELQ test execution report dashboard</figcaption></figure><p>ACCELQ is a commercial test automation platform that supports codeless test authoring and AI-assisted automation. It brings mobile, web, and API testing into a shared environment, with reusable test components and centralized execution management. It suits QA teams that want to build and maintain automated tests with less conventional scripting.</p><p><em>Version checked: ACCELQ Continuous Automation Platform (September 2026).</em></p><p><strong>ACCELQ Key Features:</strong></p><ul><li><strong>Codeless test authoring:</strong> Visual and natural-language tools represent test scenarios without conventional scripting.</li><li><strong>Autonomous element identification:</strong> AI-powered identification and self-healing features address selected interface changes.</li><li><strong>Reusable test components:</strong> Shared business actions support consistent automation across related workflows.</li><li><strong>Change-impact analysis:</strong> Traceability features help identify affected test assets when requirements or applications change.</li><li><strong>Centralized execution management:</strong> Scheduling, parallel execution, and reporting organize test runs in one platform.</li></ul><p><strong>=&gt; ACCELQ AI Capabilities:</strong> Uses semantic UI analysis and AI-driven self-healing engines to dynamically identify element changes, maintain locators without human intervention, and automatically generate optimal test scenarios from application flows.</p><p><strong>=&gt; ACCELQ Supported Platforms:</strong> Android, iOS, web browsers, desktop applications, and backend APIs.</p><p><strong>=&gt; ACCELQ Human Intervention Required:</strong> Low for ongoing test creation and maintenance due to codeless NLP logic; minimal manual review needed for advanced custom exceptions.</p><p><strong>=&gt; ACCELQ Integrations:</strong> Native plugins for Jira, Azure DevOps, Jenkins, GitHub Actions, Bamboo, Bitbucket and BrowserStack.</p><p><strong>ACCELQ Limitations:</strong></p><ul><li>Commercial licensing and module selection affect the total cost of ownership.</li><li>Codeless abstractions offer less direct control than custom code for some scenarios.</li><li>Teams may depend on ACCELQ-specific assets and workflows when migrating platforms.</li></ul><p><strong>=&gt; ACCELQ Pricing:</strong> ACCELQ uses a commercial subscription model. Pricing depends on the selected modules, execution capacity, and enterprise requirements. A free trial is available.</p><p><strong>=&gt; ACCELQ is Best For:</strong> Enterprise QA teams seeking codeless test authoring, low-code automation, and cross-functional team collaboration.</p><h4>4. Panto AI</h4><figure><img alt="Screenshot of Panto AI uploading Wikipedia.apk" src="https://cdn-images-1.medium.com/max/1024/1*aEyPh2FKygNGw4Xq_U_46w.png" /><figcaption>Panto AI upload interface</figcaption></figure><p>Panto AI is an AI-driven testing platform designed to help teams create and execute mobile tests from natural-language instructions. It uses AI-assisted test generation and visual analysis to support application testing, including interfaces that may not expose conventional UI element hierarchies. Generated tests and results still require validation.</p><p><em>Version checked: Panto AI Mobile Testing Assistant (September 2026).</em></p><p><strong>Panto AI’s Key Features:</strong></p><ul><li><strong>Plain-English test authoring:</strong> Convert natural language prompt instructions directly into executable mobile test flows.</li><li><strong>Canvas and game engine support:</strong> Validate canvas-rendered apps (Unity, Unreal Engine, Roblox) where locator trees do not exist.</li><li><strong>Automated root-cause analysis:</strong> Diagnoses test failures instantly with detailed AI-generated failure insights.</li><li><strong>Multi-framework script export:</strong> Maintain and modify underlying tests seamlessly in open-source Appium or Maestro formats.</li></ul><p><strong>=&gt; Panto AI’s AI Capabilities: </strong>Employs visual and semantic AI models for computer-vision element tracking, zero-maintenance self-healing locators, autonomous app crawling swarms, and natural language prompt parsing.</p><p><strong>=&gt; Panto AI Supported Platforms:</strong> Native Android, iOS, React Native, Flutter, WebView, PWA, and canvas game engines (Unity, Unreal, Roblox).</p><p><strong>=&gt; Panto AI Human Intervention Required:</strong> Very low for routine test maintenance due to visual self-healing; low-to-moderate effort for initial prompt creation and edge-case verification.</p><p><strong>=&gt; Panto AI Integrations:</strong> Slack alerts, REST APIs, database connections, GitHub Actions, custom CI/CD pipelines, and IDE interfaces.</p><p><strong>Panto AI Limitations:</strong></p><ul><li>Independent benchmarks for generated-test accuracy were not established in the reviewed material.</li><li>Complex gestures and application-specific flows may require manual test design.</li><li>AI-generated tests and self-healing changes require review to prevent incorrect assertions.</li></ul><p><strong>=&gt; Panto AI Pricing:</strong> The free plan includes 15 test flows with a 5-minute execution limit. The Scale plan costs $999/month and includes 250 test flow generations and one dedicated parallel real device. Enterprise plans are also available.</p><p><strong>=&gt; Panto AI is Best For:</strong> Agile product teams wanting to leverage agentic generative AI to eliminate test maintenance and speed up authoring.</p><h3>C. Open-Source Frameworks (Best for Developer-Driven Teams)</h3><p>Open-source frameworks offer code-level control over test scripts, integrations, and execution. Here are the top options for teams with coding expertise and the resources to manage their testing infrastructure.</p><h4>5. Appium</h4><figure><img alt="A screenshot of an Appium Inspector session interface displaying automated mobile app testing elements." src="https://cdn-images-1.medium.com/max/1024/1*2s0ajxVdt6bkJUKUWJja5g.png" /><figcaption>Appium Inspector session interface</figcaption></figure><p>Appium is an open-source automation framework for native, hybrid, and mobile web applications. It uses the WebDriver protocol and a modular driver architecture to automate Android and iOS apps. Teams can write tests in multiple programming languages and reuse test logic across platforms.</p><p><em>Version checked: Appium 3.x ecosystem with UiAutomator2 and XCUITest drivers (September 2026).</em></p><p><strong>Appium Key Features</strong></p><ul><li><strong>Modular driver architecture:</strong> Install platform-specific drivers independently, including UiAutomator2 for Android and XCUITest for iOS.</li><li><strong>WebDriver protocol:</strong> Uses standard WebDriver commands through Appium client libraries.</li><li><strong>Cross-platform test reuse:</strong> Allows teams to share test logic across Android and iOS, with platform-specific adjustments where needed.</li><li><strong>Device interactions:</strong> Supports gestures, app management, and other device-level automation commands.</li><li><strong>UI inspection:</strong> Appium Inspector displays screenshots, element hierarchies, and accessibility attributes.</li></ul><p><strong>=&gt; Appium Supported Platforms:</strong> Native, hybrid, and mobile web applications on Android and iOS.</p><p><strong>=&gt; Appium Languages and Frameworks:</strong> Java, JavaScript, Python, Ruby, and C#, with compatible test runners such as JUnit and pytest.</p><p><strong>=&gt; Appium Setup and Maintenance:</strong> Moderate to high. Requires Node.js, the Appium server, platform drivers, and relevant SDKs. Driver compatibility and platform-specific selectors add to maintenance.</p><p><strong>=&gt; Appium CI/CD Integration:</strong> Supports command-line execution in CI pipelines and integration with remote device infrastructure.</p><p><strong>Appium Limitations:</strong></p><ul><li>Platform-specific selectors and behaviors can limit test script reuse.</li><li>Driver and SDK dependencies require compatibility management.</li><li>Its client-server architecture can make debugging failures across multiple layers more involved.</li></ul><p><strong>=&gt; Appium Pricing:</strong> Free under the Apache 2.0 license. Device infrastructure and hosted execution may involve additional costs.</p><p><strong>=&gt; Appium is Best For:</strong> Engineering teams that need flexible, code-based automation across Android and iOS.</p><h4>6. Espresso</h4><figure><img alt="A screenshot of the Android Studio IDE showing an automated UI test using the Espresso testing framework." src="https://cdn-images-1.medium.com/max/1024/1*ZfQ25erspT7KIFBdmaxicQ.png" /><figcaption>Espresso testing framework</figcaption></figure><p>Espresso is an open-source UI testing framework for native Android apps. It provides APIs to interact with app views, perform actions, and verify UI behavior. It integrates with AndroidX Test, JUnit, and Gradle.</p><p><em>Version checked: AndroidX Test Espresso 3.6.0 (September 2026).</em></p><p><strong>Espresso Key Features:</strong></p><ul><li><strong>UI synchronization:</strong> Coordinates test actions with supported UI operations.</li><li><strong>View matchers and assertions:</strong> Identify Android views and verify expected states.</li><li><strong>In-process instrumentation:</strong> Runs tests with application instrumentation.</li><li><strong>Intent testing:</strong> Espresso Intents validates and stubs application intents.</li><li><strong>Accessibility checks:</strong> Identifies selected accessibility issues during UI tests.</li></ul><p><strong>=&gt; Espresso Supported Platforms:</strong> Native Android apps on emulators and physical devices.</p><p><strong>=&gt; Espresso Languages and Frameworks:</strong> Kotlin and Java, with AndroidX Test, JUnit, and Gradle.</p><p><strong>=&gt; Espresso Setup and Maintenance:</strong> Low to moderate for Android developers. Requires AndroidX Test dependencies and instrumentation setup. Custom idling resources may be needed for asynchronous operations.</p><p><strong>=&gt; Espresso CI/CD Integration:</strong> Supports automated execution through Gradle instrumentation tasks in compatible CI environments.</p><p><strong>Espresso Limitations:</strong></p><ul><li>Supports Android UI automation, not native iOS testing.</li><li>System-level interactions may require additional testing APIs or frameworks.</li><li>Untracked asynchronous operations can cause flaky tests.</li></ul><p><strong>=&gt; Espresso Pricing:</strong> Free and open source. Device infrastructure and hosted execution may involve additional costs.</p><p><strong>=&gt; Espresso is Best For:</strong> Android development teams building native UI regression tests with Android Studio and Gradle.</p><h4>7. XCUITest</h4><figure><img alt="Screenshot of Xcode UI showing automated mobile app test results for an iOS app (Libi) running on an iPhone 11 simulator. The interface lists passed test steps such as launching the session, waiting for the app to idle, tapping the “Deep Insights” button, and capturing screenshots." src="https://cdn-images-1.medium.com/max/1024/1*LMFHVPVsI9MQxinXGpp38w.png" /><figcaption>Xcode UI</figcaption></figure><p>XCUITest is Apple’s UI testing technology, built on XCTest and XCUIAutomation. It lets developers interact with app elements, verify UI behavior, and inspect test failures through Xcode. It is designed for teams testing iOS and other supported Apple-platform applications within Apple’s development environment.</p><p><em>Version checked: Apple XCTest framework in Xcode (September 2026).</em></p><p><strong>XCUITest Key Features:</strong></p><ul><li><strong>Native UI automation:</strong> XCUIAutomation interacts with application controls through Apple’s testing APIs.</li><li><strong>Element queries:</strong> Queries identify interface elements using accessibility properties and supported attributes.</li><li><strong>UI test recording:</strong> Xcode records interaction sequences that developers can refine into reusable tests.</li><li><strong>Failure diagnostics:</strong> Test activities and attachments can include screenshots and execution details.</li><li><strong>Performance measurement:</strong> XCTest supports performance tests and metrics for detecting regressions.</li></ul><p><strong>=&gt; XCUITest Supported Platforms:</strong> Apple-platform UI testing through compatible Xcode tooling, including iOS and iPadOS applications.</p><p><strong>=&gt; XCUITest Languages and Frameworks:</strong> Swift and Objective-C with XCTest, XCUIAutomation, Xcode test targets, and Apple’s testing infrastructure.</p><p><strong>=&gt; XCUITest Setup and Maintenance Effort:</strong> Moderate. Configure Xcode test targets, simulators or physical devices, and signing settings where required. Maintain accessibility identifiers and manage timing-sensitive interactions.</p><p><strong>=&gt; XCUITest CI/CD Integration:</strong> The <em>xcodebuild</em> command supports automated UI test execution on compatible macOS runners and Apple device infrastructure.</p><p><strong>XCUITest Limitations:</strong></p><ul><li>Native UI automation targets Apple’s supported platforms and development toolchain.</li><li>Execution requires compatible Apple development infrastructure and Xcode.</li><li>Changes to accessibility identifiers can break existing element queries.</li></ul><p><strong>=&gt; XCUITest Pricing:</strong> XCUITest is included in Apple’s development and testing toolchain at no separate framework subscription cost. Mac hardware, hosted macOS runners, and third-party device-cloud services may incur additional charges.</p><p><strong>=&gt; XCUITest is Best For:</strong> iOS engineering teams that need native UI testing, Xcode integration, and detailed diagnostics for Apple applications.</p><h4>8. Maestro</h4><figure><img alt="Maestro mobile testing dashboard showing automated UI test flow metrics including pass rates, flake rates, and run statuses." src="https://cdn-images-1.medium.com/max/1024/1*JJijQWcyOSk4nO0t94QTEw.png" /><figcaption>Maestro mobile testing dashboard</figcaption></figure><p>Maestro is an open-source mobile UI automation framework that uses YAML files to define test flows. Its declarative syntax lets teams describe user interactions and assertions without writing extensive test code. Built-in waiting also helps handle common UI transitions without fixed sleep statements.</p><p><em>Version checked: Maestro CLI 2.12.0 (October 2026).</em></p><p><strong>Maestro Key Features:</strong></p><ul><li><strong>Declarative test flows:</strong> YAML files define readable, reusable test steps and assertions.</li><li><strong>Built-in waiting:</strong> Automatically waits for UI elements and transitions, reducing reliance on fixed delays.</li><li><strong>Cross-platform flows:</strong> Reuse test definitions across compatible Android and iOS applications.</li><li><strong>System interaction support:</strong> Automates selected permissions, app launches, and device-level interactions.</li><li><strong>CLI-based execution:</strong> Runs tests locally and integrates with CI pipelines and Maestro Cloud.</li></ul><p><strong>=&gt; Maestro Supported Platforms:</strong> Android emulators and physical devices, and iOS simulators. Maestro Cloud provides managed execution across supported Android and iOS configurations.</p><p><strong>=&gt; Maestro Languages and Frameworks:</strong> YAML-based flows support native Android and iOS apps, including applications built with React Native and Flutter.</p><p><strong>=&gt; Maestro Setup and Maintenance:</strong> Low to moderate. Requires CLI installation, a compatible device or simulator, and YAML flow definitions. Complex test logic and data-driven scenarios may require JavaScript or external tooling.</p><p><strong>=&gt; Maestro CI/CD Integration:</strong> Supports command-line execution in CI pipelines and hosted test execution through Maestro Cloud.</p><p><strong>Maestro Limitations:</strong></p><ul><li>Complex data-driven scenarios may require additional scripting or external services.</li><li>UI-layer automation offers less direct access to internal application state.</li><li>Platform-specific behaviors can limit the reuse of identical flows.</li></ul><p><strong>=&gt; Maestro Pricing:</strong> The CLI is free and open source. Maestro Cloud offers paid hosted execution plans.</p><p><strong>=&gt; Maestro is Best For:</strong> Development and QA teams that want readable, cross-platform UI automation with less scripting overhead.</p><h4>9. WebdriverIO</h4><figure><img alt="WebdriverIO automated end-to-end tests running in a VS Code terminal with parallel ChromeDriver workers." src="https://cdn-images-1.medium.com/max/1024/1*hfsIOoSG95Kxx0UBbbC-Sw.png" /><figcaption>WebdriverIO</figcaption></figure><p>WebdriverIO is an open-source automation framework for JavaScript and TypeScript. It supports web and mobile testing, using Appium for mobile app automation. Its built-in test runner, assertion utilities, and reusable commands help teams organize and execute automation suites.</p><p><em>Version checked: WebdriverIO v9 with Appium service backend (September 2026).</em></p><p><strong>WebdriverIO Key Features:</strong></p><ul><li><strong>Integrated test runner:</strong> Organizes test suites, configurations, and execution across multiple capabilities.</li><li><strong>Asynchronous command handling:</strong> Uses async/await to sequence browser and mobile commands in test scripts.</li><li><strong>Mobile interaction commands:</strong> Provides APIs for gestures, scrolling, tapping, and switching between native and webview contexts.</li><li><strong>Reusable utilities:</strong> Supports shared helpers, custom commands, and assertions across test suites.</li><li><strong>Service integrations:</strong> Connects with Appium, reporting tools, and remote execution providers.</li></ul><p><strong>=&gt; WebdriverIO Supported Platforms:</strong> Android and iOS through compatible Appium drivers, supporting native, hybrid, and mobile web applications.</p><p><strong>=&gt; WebdriverIO Languages and Frameworks:</strong> JavaScript and TypeScript, with WebdriverIO’s test runner, assertion libraries, and Appium drivers.</p><p><strong>=&gt; WebdriverIO Setup and Maintenance:</strong> Moderate. Requires Node.js, WebdriverIO configuration, Appium, and compatible platform drivers. Driver and package compatibility also require maintenance.</p><p><strong>=&gt; WebdriverIO CI/CD Integration:</strong> Supports execution through Node.js-based CI pipelines, containerized environments, and remote device-cloud infrastructure.</p><p><strong>WebdriverIO Limitations:</strong></p><ul><li>Mobile automation depends on Appium and its platform-specific drivers.</li><li>Complex test suites require JavaScript or TypeScript expertise.</li><li>Additional reporting and service integrations may require third-party packages.</li></ul><p><strong>=&gt; WebdriverIO Pricing:</strong> Free and open source under the MIT license. Device-cloud subscriptions and hosted infrastructure may involve additional costs.</p><p><strong>=&gt; WebdriverIO is Best For:</strong> JavaScript and TypeScript teams that want to manage web and mobile automation within a shared testing framework.</p><h3>What Should You Look for in a Mobile App Testing Tool?</h3><p>Before evaluating specific tools, map your requirements against the realities of mobile application architecture.</p><p>Look for these four capabilities:</p><ol><li><strong>Native and Cross-Platform Support:</strong> Check compatibility with native Android and iOS apps, as well as frameworks such as React Native and Flutter. Custom-rendered interfaces may require additional configuration or specialized selectors.</li><li><strong>Element Locator Reliability:</strong> Prioritize accessibility IDs, resource IDs, and other stable locators. Tests that depend heavily on absolute XPath expressions are more likely to break when the UI structure changes.</li><li><strong>Real Devices and Emulators:</strong> Emulators support routine UI tests, while real devices help validate hardware-dependent behavior, device-specific gestures, camera access, and biometric flows. Choose a tool that supports the testing environments your app requires.</li><li><strong>CI/CD Integration:</strong> Check support for your existing pipeline, whether it uses GitHub Actions, GitLab CI, or Jenkins. Test execution, build installation, result reporting, and failure diagnostics should fit into the pipeline without excessive custom scripting.</li></ol><h3>Summary Checklist: Which Automated Mobile App Testing Tool Should You Choose?</h3><p>Use this checklist to match your app’s testing needs, team skills, device coverage, and budget with the right tool category before making your final choice.</p><figure><img alt="Comparison chart titled Mobile App Testing Tools Comparison comparing Appium, Espresso, XCUITest, Maestro, WebdriverIO, ACCELQ, Panto AI, BrowserStack App Automate, and TestGrid across test stability, maintenance overhead, execution speed, and team alignment." src="https://cdn-images-1.medium.com/max/1024/1*2mFPDQJSlkmbJSeyzOIciQ.png" /><figcaption>Mobile App Testing Tools Comparison</figcaption></figure><h3>Final Thoughts</h3><p>A mobile automation setup should reduce the time teams spend maintaining tests and investigating failures. Before choosing a tool, evaluate how it handles UI changes, asynchronous operations, device variations, and parallel execution. Go for trials. Assess the tool against your requirements. Run the same critical user flows across representative Android and iOS devices, then assess how much effort goes into fixing failed tests over successive releases.</p><p>My recommendation is to give test stability and maintenance effort more weight than the number of features a platform advertises. Those two factors will have a much greater impact on how effectively a team can sustain mobile automation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=283152ed7bfc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Test Mobile Apps Under Poor Network Conditions?]]></title>
            <link>https://medium.com/@seanwintermere170/how-to-test-mobile-apps-under-poor-network-conditions-3a6fee364cf1?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a6fee364cf1</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[mobile-testing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 07:36:01 GMT</pubDate>
            <atom:updated>2026-11-08T07:36:01.466Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A minimalist 2D illustration of a hand holding a smartphone with a loading spinner and ‘No Connection’ alert on the screen, surrounded by floating icons representing dropped Wi-Fi, packet loss, and offline networks on a bright white background." src="https://cdn-images-1.medium.com/max/1024/1*FWuGKC9sSfKazzXdzc07hw.png" /></figure><p>Teams usually test apps on fast office Wi-Fi, yet users open them on trains, in lifts and in basements with one bar of signal. That’s where the spinner never stops, a photo upload quietly fails, and a frozen payment screen tempts someone to tap Pay twice.</p><p>None of those bugs show up on a fast connection, because a fast connection never makes the app wait, retry or resume. Poor network testing is how you find them before your users do.</p><p>This article covers the failures users actually hit, what breaks first, a simple journey test, and the tools that simulate each condition.</p><h3>What is Network Testing and Why is It Important?</h3><p>For a mobile app, network testing checks what happens when the connection is slow, lossy, missing or changing mid-task. Poor network testing focuses on those bad cases, because they run code that a fast connection never touches.</p><p>That code handles waiting, retrying and resuming. Its bugs tend to be the costly kind, such as double charges, lost uploads and wiped forms.</p><p>My main check after any network problem is simple. The user should end up in a correct state, with nothing lost and nothing done twice.</p><h3>What are the Network Failures that Users Experience?</h3><p>Real connections fail in four common ways. Each one hits a different part of your code, so poor network testing needs a check for each.</p><h3>High Latency and Limited Bandwidth</h3><p>This is the weak-signal case, where replies arrive late and large files move slowly. Screens with images, video or large API responses stall first, and any missing loading state becomes obvious.</p><p>Add delay and a low bandwidth cap to your heaviest screens. Then check that requests don’t give up too early or hang for minutes.</p><h3>Packet Loss and Unstable Connections</h3><p>Here some data never arrives and has to be sent again. A phone can show full bars and still stall at random.</p><p>The real risk is the silent repeat. A request that times out on the phone may still have reached the server, so a blind retry can repeat the action. Add a few percent of packet loss and repeat actions that change data, such as sending a message or placing an order.</p><h3>Going Offline Mid-Flow</h3><p>Users lose signal halfway through tasks in lifts and tunnels. For poor network testing, cut the network at each step of a flow and check two things:</p><ul><li><strong>Saved input:</strong> The app should keep everything the user typed or selected, so they can carry on when the signal returns.</li><li><strong>A clear message:</strong> The error should say the connection dropped and tell the user what to do next.</li></ul><h3>Switching Between Wi-Fi and Mobile Data</h3><p>When a phone leaves Wi-Fi range, it moves to mobile data and gets a new IP address. Open connections on the old network usually drop, so any request in flight fails.</p><p>Android apps watch for these changes through ConnectivityManager’s NetworkCallback, and iOS apps use NWPathMonitor. Trigger the handover during an upload, a payment and a video stream, and check that each one recovers. Apps that use QUIC can survive some switches, because the protocol supports connection migration.</p><h3>What Usually Breaks First?</h3><p>Three areas tend to break first in poor network testing, because they all depend on what the app does while it waits.</p><h3>Loading and Timeout States</h3><p>Look for screens with no loading state, spinners that never stop, and error messages that blame the user. Every request needs a timeout and a clear way to retry.</p><p>Platform defaults differ a lot. OkHttp uses 10-second connect, read and write timeouts by default, while URLSession waits 60 seconds. If nobody set them on purpose, a slow network will show you which default you got.</p><h3>Payments, Uploads, and In-Progress Actions</h3><p>In these flows, a half-finished request costs money or data, so they deserve the most poor network testing time. Check these three things on every release:</p><ul><li><strong>Double taps:</strong> The Pay button should disable after the first tap, since a frozen screen invites a second one.</li><li><strong>Retried payments:</strong> The backend should treat a retry as the same request. Payment APIs such as Stripe accept idempotency keys for exactly this case.</li><li><strong>Broken uploads:</strong> A large upload should resume or restart cleanly after a drop, and the app should never show success for a failed upload.</li></ul><h3>Offline Data, Re-Sync, and Duplicate Requests</h3><p>Apps that save work offline must sync it later without losing it or doubling it. Make a few changes offline, reconnect, and confirm each one reaches the server exactly once.</p><p>Then edit the same record on two devices while one is offline. Your conflict rule should work the same way every time, and the user should see which version won.</p><h3>Test One Critical User Journey From Good Network to No Network</h3><p>This journey test is the core of my poor network testing routine. I take one journey that earns money or holds user data, such as checkout or a photo upload, and run it through five network conditions while keeping everything else the same.</p><figure><img alt="Flowchart titled ‘One journey, five network conditions’ outlining a 5-step ux testing method: Step 1 Good Wi-Fi baseline, Step 2 High latency and low bandwidth, Step 3 Packet loss, Step 4 Offline mid-flow, and Step 5 Back online or a new network, with specific checks to perform for each condition." src="https://cdn-images-1.medium.com/max/1024/1*nExUv8inyiGfSp-AFCy_Hg.png" /></figure><p>I start with a baseline on good Wi-Fi and time each screen, so I can tell slow from broken later. After every run, I check the backend logs or database to confirm the action happened exactly once, because the app’s screen can’t tell you that on its own.</p><p>Run the whole ladder on one Android and one iOS device. Their timeout defaults and network APIs differ, so a pass on one platform says little about the other.</p><h3>Tools for Testing Poor Network Conditions</h3><p>These five tools cover poor network testing on real device clouds, a desktop proxy, and the free tools from Apple and Google. Each summary comes from the official docs.</p><h3>1. BrowserStack App Live</h3><p>App Live runs your app on real iOS and Android devices from the browser. Click Throttle Network on the session toolbar, and you can pick a preset profile, set your own bandwidth, latency and packet loss, or take the device offline. Custom bandwidth goes up to 50 Mbps.</p><p>Network throttling needs a Team, Team Pro or Enterprise plan. On Android it works through a proxy, so BrowserStack’s FAQ notes that apps that aren’t proxy-aware won’t feel the limits.</p><figure><img alt="Screenshot of BrowserStack App Live displaying network throttling options on the left and live Logcat logs on the right." src="https://cdn-images-1.medium.com/max/1024/1*sVtgSvGTts1Azej9TlxKaA.png" /></figure><blockquote><strong>My take:</strong> I like changing the network mid-session on a real device with no setup on my side, which suits the journey test above. For repeat runs in CI, you’d move the same profiles into automated tests.</blockquote><h3>2. Charles Proxy</h3><p>Charles is a desktop proxy that sits between your phone and the internet. You point the phone’s Wi-Fi proxy at your computer, install the Charles root certificate to read HTTPS, and then open Proxy &gt; Throttle Settings to cap bandwidth and add latency.</p><p>It only shapes traffic that goes through the proxy, and each phone needs its own setup. A license costs $50 per user after a 30-day trial.</p><figure><img alt="Screenshot of Charles Proxy displaying HTTP and HTTPS network traffic requests, server status, and connection details in the overview panel." src="https://cdn-images-1.medium.com/max/1024/1*vUaKdP2XQ-4Wn2C9WmCHcw.png" /></figure><blockquote><strong>My take:</strong> Charles is my pick when I need to watch the retries themselves, because every request and response sits in front of me. Its per-host throttling also lets you slow one third-party API while the rest of the app runs at full speed.</blockquote><h3>3. Xcode Network Link Conditioner</h3><p>Network Link Conditioner ships free with Apple’s developer tools. On an iPhone with developer mode on, open Settings &gt; Developer &gt; Network Link Conditioner and pick a profile such as 3G, Edge, 100% Loss or Very Bad Network. Since Xcode 11, you can also start a profile from Device Conditions in the Devices and Simulators window.</p><p>It works on Apple devices only, and the device route needs a Mac and a physical iPhone.</p><figure><img alt="Apple iOS Settings app showing the Network Link Conditioner screen with profiles like 3G, DSL, LTE, and Wi-Fi enabled." src="https://cdn-images-1.medium.com/max/1024/1*vhJ6AQJMEBfLBd6EtEqLtg.png" /></figure><blockquote><strong>My take:</strong> For quick iOS checks during development, this is the fastest option I know, and Very Bad Network makes a good stress profile. It shapes the whole device connection, so other apps on the phone slow down too.</blockquote><h3>4. Android Emulator Network Controls</h3><p>Every Android developer already has these controls. Open Extended controls &gt; Cellular to set the network type from GSM up to 5G, the signal strength, and a data status such as Roaming or Denied.</p><p>From the command line, emulator @Pixel8_API_34 -netspeed edge -netdelay gsm starts an emulator on a slow, high-latency link. The docs list no packet loss setting.</p><figure><img alt="Android Studio emulator displaying a mobile e-commerce checkout screen with an order summary, shipping address, and Place order button." src="https://cdn-images-1.medium.com/max/1024/1*SYKJJLKWyTZJUCyd2bK1Rw.png" /></figure><blockquote><strong>My take:</strong> It’s a sensible free first pass while you build a feature. I wouldn’t sign off a release on it, because it can’t drop packets and it still rides on your laptop’s network underneath.</blockquote><h3>5. HeadSpin</h3><p>HeadSpin offers real devices in several locations, including devices with live SIM cards. Its network configuration API sets download and upload limits, added round-trip time and a packet loss fraction, and you can also apply it through an Appium capability.</p><p>You need code to change conditions, its docs say browsers aren’t supported yet for this API, and pricing isn’t public.</p><figure><img alt="Headspin session performance dashboard showing London session impact curve, issue categories like connection reuse and slow page content, and performance issue cards with latency metrics." src="https://cdn-images-1.medium.com/max/1024/1*-Q_HPxf51MCpb0emLgkFUw.png" /></figure><blockquote><strong>My take:</strong> The live SIM devices help when carrier behavior matters to your app. The API-first setup suits automation teams more than a tester who wants to try a condition quickly.</blockquote><h3>Run Poor Network Tests on Real Devices With BrowserStack App Live</h3><p>Here’s how I’d run poor network testing for the journey on App Live. It takes one session per device and no local setup.</p><ol><li><strong>Pick real devices:</strong> Upload your build and choose the phone models most of your users own.</li><li><strong>Get a baseline:</strong> Run the journey once with throttling off.</li><li><strong>Apply a slow or lossy profile:</strong> Open Throttle Network, choose a preset, and run the journey again.</li><li><strong>Go offline during the last step:</strong> Choose Offline, then turn network simulation off and watch how the app recovers.</li><li><strong>Handle iOS offline separately:</strong> Use Offline Mode Simulation under iOS Settings, and note that the docs don’t support apps that check NWPath.Status for this mode.</li></ol><h3>Prioritize Network Failures Before Release</h3><p>Slowness usually gets the most attention in poor network testing, yet it does the least damage. People forgive a slow screen far more easily than a duplicate order or a lost upload.</p><p>So I rank network bugs by what they cost the user. Duplicates and lost data come first and should block a release. Stuck screens and lost input come second, because users can’t finish the task. Slow loading comes last, since the task still completes.</p><p>If you only have time for one test, drop the connection during the final step of your most valuable flow. Then check the server, because that’s where a double charge shows up first.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a6fee364cf1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Run XCUITest Tests Online on Real iPhones?]]></title>
            <link>https://medium.com/@seanwintermere170/how-to-run-xcuitest-tests-online-on-real-iphones-81b95b4f720c?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/81b95b4f720c</guid>
            <category><![CDATA[ios-development]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[xcuitest]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Sat, 07 Nov 2026 14:21:01 GMT</pubDate>
            <atom:updated>2026-11-07T14:21:01.506Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqtsWJERr0vgwjb3PCcXgQ.png" /></figure><p>Anyone who has travelled out of their home country, knows the plug problem.</p><p>Basically, you carry your phone charger, but each country wants a different socket, so you end up carrying a bag of adapters.</p><p>I think running XCUITest on real iPhones works the same way. Your tests pass on the iPhone on your desk. Then you move them to a device cloud, and each cloud wants the same build packed in a different shape. If you get the shape wrong and the run fails before a single test starts.</p><p>So to debunk this issue, I’ve explained how to run XCUITest tests on iPhone hardware locally first, then cover each cloud with the exact package and limits it expects.</p><h3>Prepare an XCUITest Build for Real Device Execution</h3><p>Everything you need to run XCUITest tests on iPhone hardware comes out of one build command. A real device run needs compiled artifacts, and the same artifacts feed the local run and all three clouds.</p><p>Build the app and the test runner for a generic device instead of a simulator:</p><pre>xcodebuild build-for-testing \<br>  -workspace YourProject.xcworkspace \<br>  -scheme YourScheme \<br>  -derivedDataPath ./build \<br>  -destination &quot;generic/platform=iOS&quot;</pre><p>Look inside ./build/Build/Products afterwards. The Debug-iphoneos folder holds YourApp.app and a separate YourAppUITests-Runner.app, with the .xctestrun manifest alongside it.</p><p>Signing is where first attempts usually fail. Set a valid provisioning profile and identity on the app target and the test runner target, then verify each bundle before uploading:</p><pre>codesign --verify --deep --verbose <br>./build/Build/Products/Debug-iphoneos/YourApp.app</pre><p>The Firebase and Google docs both say to expect valid on disk. Anything else shows up later as an upload error from the service, when the real problem is your certificates.</p><h3>Run XCUITest Tests on a Connected iPhone</h3><p>The cheapest way to run XCUITest tests on iPhone hardware is the handset already on your desk. One green run there confirms your signing, your scheme and your element queries. It also gives you a baseline to compare cloud results against.</p><h3>Connect and Identify Your iPhone</h3><p>Plug the iPhone in, open it past the lock screen, and tap Trust on the prompt. In current Xcode, physical devices are paired through Device Hub, which you open from the run destination menu by choosing Manage Devices.</p><p>Sign in under Settings, Apple Accounts, then turn on automatic signing. Apple’s docs say Xcode registers the device and creates a development provisioning profile for it on that first run.</p><p>Command line runs need the device identifier, which Apple’s devicectl tool prints:</p><pre>xcrun devicectl list devices</pre><h3>Run XCUITest From Xcode</h3><p>Pick the iPhone from the run destination menu, then press Cmd+U. Xcode installs the app and the runner, launches the runner, and shows results in the test navigator.</p><p>This is the quickest loop while you are still writing tests. Failures are easiest to read here, because you see the assertion, a screenshot at the point of failure and the element tree without downloading anything.</p><h3>Run XCUITest From the Command Line</h3><p>CI needs the terminal version, and it reuses the build you already have. Point test-without-building at the manifest and the device identifier:</p><pre>xcodebuild test-without-building \<br>  -xctestrun ./build/Build/Products/YourScheme_iphoneos26.0-arm64.xctestrun \<br>  -destination &quot;id=00008130-001A2B3C4D5E6F80&quot;</pre><p>Add -only-testing to narrow a run to one class, and -resultBundlePath to write an .xcresult bundle that xcresulttool can read.</p><h3>Execute the Same XCUITest Tests on Another iPhone</h3><p>Running the suite on a second handset separates real bugs from one device’s state. Nothing needs rebuilding. Register and trust the second iPhone, find its identifier, and pass it to the same test-without-building command.</p><p>This is where the local approach runs out. To run XCUITest tests on iPhone models from an older SE to a current iPhone 17, you have to own both, keep both charged, and hold their iOS versions still.</p><h3>Run XCUITest Tests Online on Real iPhones via Device Cloud</h3><p>A device cloud replaces that drawer of handsets with an API call. You upload the artifacts you already built, name the devices, and collect logs, video and results. Choosing a tool to run XCUITest tests online comes down to the package each one wants and how it charges for time.</p><h3>1. BrowserStack App Automate</h3><p>App Automate runs XCUI tests on physical iPhones in its data centres, and it takes the simplest package of the three. The app goes up as an .ipa, and the test suite is the runner bundle on its own.</p><pre>cd ./build/Build/Products/Debug-iphoneos<br>zip --symlinks -r MyAppUITests.zip YourAppUITests-Runner.app</pre><p>Upload both with the xcuitest/v2/app and xcuitest/v2/test-suite endpoints. Then start a build by posting app, testSuite and devices to xcuitest/v2/build. Test plans use a separate endpoint and a zip that bundles the runner with the .xctestrun file.</p><p>The build API is where this earns its place. You can shard a suite with only-testing or skip-testing strategies, switch on deviceLogs, networkLogs and debugscreenshots, request an .xcresult bundle with enableResultBundle, and reach a staging backend with local. The docs set idleTimeout between 60 and 900 seconds, with 900 as the default, so a hung test ends as a timed-out build.</p><figure><img alt="Screenshot of the BrowserStack App Automate dashboard showing a Ruby Appium Sample build with build status, user details, session metrics, and a list of executed test sessions on Amazon Fire TV Stick devices." src="https://cdn-images-1.medium.com/max/1024/1*7X0ejxVKoCE-Vd2fIOP4Dw.png" /></figure><p><strong><em>My take:</em></strong><em> This is the easiest of the three to put into a CI pipeline. I like that the test package is only the runner app, so there is very little to get wrong at the zip step. The sharding and log options also mean I can usually see why a test failed without running it again.</em></p><p><em>The one thing I would plan for is concurrency. Plans are sold by parallels, so I would decide how many devices I need at the same time before picking a plan.</em></p><h3>Firebase Test Lab</h3><p>Test Lab wants a different package. There is no separate .ipa. You zip the build products directly, with the folder and the manifest at the root of the archive.</p><pre>cd ./build/Build/Products<br>zip -r MyTests.zip Debug-iphoneos *.xctestrun</pre><p>Submit it with gcloud firebase test ios run and read results in the Firebase console. Before picking devices, check the Reduced Stability label and the High, Medium or Low capacity rating, which show how reliable and how plentiful each model is.</p><p>The free Spark plan allows five physical device runs per day. Blaze includes 30 physical device minutes daily, then charges $5 per hour per physical device, billed per minute. Two limits matter more than price: results carry no video on iOS 18 and above, and the service closes on 30 September 2027. Google’s replacement, Device Run, is in preview and accepts the same zip through gcloud alpha device-run sessions submit xctest, with an — xctest-timeout from one minute to one hour.</p><figure><img alt="Screenshot of Firebase Test Lab showing a passed Robo test on Pixel 2 running API Level 26, displaying crawl duration, statistics, and a crawl graph starting from the app’s main screen." src="https://cdn-images-1.medium.com/max/1024/1*JaGZePkn2uuXDhdlI5P_6Q.png" /></figure><p><strong><em>My take:</em></strong><em> I think Test Lab still makes sense if you only need a few runs a day and already use Firebase for other things. The free quota is enough for a quick smoke check on real hardware. Even so, I would not build a new pipeline on it today.</em></p><p><em>The shutdown date is fixed, Device Run is still in preview, and the missing video on newer iOS versions makes failures harder to understand.</em></p><h3>AWS Device Farm</h3><p>Device Farm accepts a third shape. The app is an .ipa, and the test package is either an .ipa made by placing the runner inside a Payload folder, or a zip holding the runner with your own .xctestrun files. The docs warn that a zip package must not contain a folder called Payload.</p><pre>mkdir Payload<br>cp -R ./build/Build/Products/Debug-iphoneos/YourAppUITests-Runner.app Payload/<br>zip -r YourAppUITest.ipa Payload</pre><p>The published limits shape how you plan runs. An automated run is capped at 150 minutes, and Device Farm tests at most five devices at once by default. Runs stay queued for up to 24 hours, and a run without a custom test environment is capped at 250 test cases.</p><p>Pricing is $0.17 per device minute after a one-time trial of 1,000 minutes, or $250 per device slot per month unmetered.</p><figure><img alt="AWS Device Farm test report showing 5 total tests with 3 passed and 2 failed on an Apple iPhone 13 Pro Max." src="https://cdn-images-1.medium.com/max/1024/1*mge78EF6aarBLur1Kwx9ng.png" /></figure><p><strong><em>My take: </em></strong><em>For me, Device Farm makes the most sense when your team already works in AWS, since billing and permissions sit in the same account. The unmetered slot price is also easy to budget for steady daily runs.</em></p><p><em>What I would watch is the 150 minute cap and the default of five devices at once. A large suite has to be split into smaller runs first, and that is extra planning work.</em></p><h3>Run XCUITest Across Multiple iPhone and iOS Versions</h3><p>Coverage is the main reason to pay for a cloud, and each one spreads a suite differently. Choose based on whether you care about exact models or about finishing quickly.</p><p>BrowserStack accepts regular expressions in the devices array, so {“device”: “iPhone.*”, “os_version”: “1[56]”} takes the first free iPhone matching that pattern. Its docs say this reduces queuing, and each entry uses one parallel. Device Run takes repeated — device flags, such as iphonese3–18–4 and iphonese3–26–3. Device Farm uses device pools.</p><p>Avoid pinning every run to one exact model and iOS version, because you then wait on that one device’s queue.</p><h3>Compare Local and Cloud XCUITest Results</h3><p>A local pass and a cloud pass are different kinds of evidence. Clouds re-sign your app with their own profile, wipe the device between sessions, and give you artifacts instead of a live debugger.</p><figure><img alt="A table comparing four mobile testing environments (Local Mac and iPhone, BrowserStack App Automate, Firebase Test Lab, and AWS Device Farm) based on the package format they expect, the number of devices allowed at once, and their main constraints." src="https://cdn-images-1.medium.com/max/1024/1*qWgA_onrbjtozu5f9O6eHQ.png" /></figure><p>Re-signing causes the most surprises. Push notifications and universal links depend on your own entitlements, which is why BrowserStack offers a resignApp flag you can set to false. When you run XCUITest tests on iPhone models in a cloud and a local pass turns into a failure, check signing first.</p><h3>Troubleshoot XCUITest Failures on Real Devices</h3><p>Most real device failures come from packaging, permissions or timeouts, and fewer come from the assertions. Work through them in that order before you change test code.</p><ul><li><strong>Upload rejected or the app will not install:</strong> Run codesign — verify — deep — verbose on the app bundle and the runner bundle, and confirm both return valid on disk. A simulator build is the other usual cause, since AWS requires an .ipa built for a device.</li><li><strong>The service cannot find your tests:</strong> Check the zip root. Device Run wants Debug-iphoneos and the .xctestrun file at the top level, and BrowserStack wants the runner bundle alone unless you are running a test plan.</li><li><strong>A permission alert stops the run:</strong> Location, notification and photo alerts are drawn by iOS, so the test needs an interruption monitor to handle them. They stay hidden locally once you have granted permission, then appear on every wiped cloud device.</li><li><strong>The build times out instead of failing:</strong> A hung element query uses up the clock. Lower BrowserStack’s idleTimeout, set — xctest-timeout on Device Run, and split suites that come close to Device Farm’s 150 minute cap.</li><li><strong>The failure will not reproduce:</strong> Pull the result bundle through enableResultBundle or -resultBundlePath and read it with xcresulttool.</li></ul><h3>XCUITest Local and Cloud Execution Checklist</h3><p>Use this before a cloud run, especially the first one after a signing or Xcode change. Each item maps to a failure covered above.</p><ul><li><strong>Build once for a generic destination:</strong> Run build-for-testing with -destination “generic/platform=iOS” and keep the derived data path fixed, so one set of artifacts feeds every run.</li><li><strong>Prove it on a connected iPhone first:</strong> Get one green local run before you run XCUITest tests on iPhone devices in the cloud and start paying for minutes.</li><li><strong>Keep a packaging script per service:</strong> The runner-only zip, the Debug-iphoneos plus .xctestrun zip and the Payload .ipa are three different shapes and cannot be swapped.</li><li><strong>Set timeouts and logs on purpose:</strong> Turn on device and network logs for the first runs, then keep the idle timeout low enough that a hang fails quickly.</li></ul><h3>Why I Script the Packaging First?</h3><p>For me, packaging is the real lock-in, more than dashboards or device counts. All three services take the same compiled output and want it arranged three ways. If that zip step is hard-coded into one CI job, moving later costs real time.</p><p>So I would write three small packaging scripts on day one, the same way I pack plug adapters before a trip, even if I only end up using one. Firebase Test Lab’s shutdown notice is the reason. The low-cost option today has an end date, and the team that treated packaging as a detail will spend a sprint on it in 2027.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81b95b4f720c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ultimate Guide to Android App Testing]]></title>
            <link>https://medium.com/@seanwintermere170/the-ultimate-guide-to-android-app-testing-4badfced22d6?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/4badfced22d6</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[android-app-development]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[android-development]]></category>
            <category><![CDATA[test-automation]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 15:02:24 GMT</pubDate>
            <atom:updated>2026-11-06T15:02:24.467Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJG3OU0zhQD6Yp86nEnItg.png" /></figure><p><strong>Key Takeaways:</strong></p><ul><li>Android app testing evaluates functionality, performance and compatibility across devices before release, preventing bugs and ensuring smooth user experience.</li><li>Tools like Appium, Espresso, and BrowserStack App Automate help automate testing across real devices, speeding up releases without sacrificing coverage.</li><li>Combine manual and automated tests, prioritize real device coverage, and test early to catch issues before they reach users.</li></ul><p>Android apps run in a messy real world. Users open them on different devices with varied screen sizes, OS and chipsets. An app that works well on one phone may crash on the other and it may be difficult to point out if it’s because of the device itself or something external like network settings.</p><p>That is why Android app testing needs more than a basic functional check. Teams need to test UI behavior, gestures and real-device conditions before release. This helps catch issues that emulators and limited local testing may miss.</p><p>In this guide, I’ll explain what Android app testing means, which components should be tested, what testing types matter most, and how to choose the right Android app testing tools.</p><h3>What is Android App Testing?</h3><p><strong>Android app testing verifies an app’s functionality, performance, security, and compatibility across Android devices and OS versions before release.</strong></p><p>It helps teams identify defects and ensure the app behaves as expected in real-world conditions.</p><p>Android fragmentation makes testing particularly important. Devices differ in screen sizes, hardware capabilities, OS versions, and manufacturer customizations. An app that works correctly on one phone may experience layout issues on another, consume excessive battery, or behave differently after a permission or network change.</p><p>Testing combines manual checks with automated test execution to uncover these issues before they affect users. The challenge is to identify the right combination of devices, environments, and test scenarios to achieve reliable coverage without unnecessarily increasing testing time.</p><h3>What Components are Tested Within Android?</h3><p>Android app testing looks at the full experience, not just whether a feature opens or a button works. You must assess how the app behaves across design, interaction, accessibility, device conditions, and security expectations.</p><h3>UI/UX</h3><p>UI/UX testing checks whether the app is easy to use and visually consistent across Android devices. This includes:</p><ul><li>Layouts</li><li>Fonts</li><li>Icons</li><li>Colors</li><li>Spacing</li><li>Navigation</li><li>Loading states</li><li>Error messages</li><li>Responsive behavior across screen sizes</li></ul><p>For Android apps, this matters because the same interface may look different on phones, tablets, foldables, or devices with custom manufacturer settings.</p><h3>Inputs and Gestures</h3><p>Android users interact with apps through taps, swipes, long presses, drag gestures, text input, voice input, biometrics, and hardware buttons. Testing checks whether these inputs work as expected across devices.</p><p>This also includes edge cases like:</p><ul><li>Accidental taps</li><li>Gesture conflicts</li><li>Screen rotation</li><li>Back button behavior</li><li>Keyboard opening</li><li>Touch target size</li></ul><h3>Accessibility</h3><p>Accessibility testing checks whether people with disabilities can use the app through tools like TalkBack, switch access, magnification, voice access, and external keyboards.</p><p>This includes:</p><ul><li>Labels</li><li>Focus order</li><li>Contrast</li><li>Touch target size</li><li>Screen reader output</li><li>Form errors</li><li>Captions</li><li>Support for dynamic text</li></ul><p>Accessibility testing helps make sure the app works for more users and aligns with standards like WCAG and platform accessibility guidelines.</p><h3>Compatibility</h3><p>Compatibility testing checks whether your app is working appropriately across different Android devices, varied by their model, screen size and more.</p><p>This is one of the biggest parts of Android testing because the ecosystem is fragmented. Teams need to confirm that the app behaves well on both newer and older devices, different OS versions, and real-world configurations.</p><h3>Security</h3><p>Security testing checks whether the Android app protects user data, authentication flows, APIs, permissions, storage, and communication with backend systems. Security testing is especially important for apps that handle payments, health data, identity, enterprise access, or personal information.</p><p>This includes testing for:</p><ul><li>Insecure data storage</li><li>Weak encryption</li><li>Exposed secrets</li><li>Unsafe permissions</li><li>Broken session handling</li><li>Insecure network calls.</li></ul><h3>What Types of Testing Are Implemented for Android?</h3><p>A single testing method cannot cover every aspect of an Android app. Different types of testing help QA teams in different aspects. This makes sure that the app delivers a consistent user experience.</p><p>The key types of Android testing include:</p><p><strong>1. Functional Testing:</strong> It verifies whether app features work as expected. It covers user actions such as logging in, searching for products, completing checkout, updating profiles, receiving notifications, and processing payments.</p><p><strong>2. UI Testing:</strong> It checks whether interface elements appear and behave correctly. It covers layouts, navigation, buttons, forms, menus, visual states, and screen transitions across Android devices.</p><p><strong>3. Compatibility Testing:</strong> It ensures that an app works correctly across different Android versions, device models, screen sizes, resolutions, and manufacturer-specific customizations.</p><p><strong>4. Performance Testing:</strong> It measures how efficiently an app operates under different conditions. It evaluates app launch times, response times, scrolling smoothness, CPU and memory usage, battery consumption, and behavior under load.</p><p><strong>5. Security Testing:</strong> It identifies vulnerabilities that could expose user data or compromise app functionality. It examines authentication, permissions, data storage, API communication, encryption, session management, and the handling of sensitive information.</p><p><strong>6. Accessibility Testing:</strong> It verifies whether users with different abilities can operate the app effectively. It checks compatibility with Android accessibility features such as TalkBack, Voice Access, Switch Access, screen magnification, captions, and larger text settings.</p><p><strong>7. Usability Testing:</strong> It evaluates how easily users can navigate the app and complete their tasks. It focuses on navigation clarity, user flows, error recovery, onboarding, and the overall user experience.</p><p><strong>8. Regression Testing:</strong> It ensures that code changes, new features, OS updates, or UI redesigns do not break existing functionality. Teams often automate regression tests to identify unintended side effects before releasing updates.</p><p><strong>9. Network Testing:</strong> It evaluates app behavior under different connectivity conditions. It covers slow networks, offline scenarios, interrupted connections, transitions between Wi-Fi and mobile data, and API timeouts.</p><p><strong>10. Localization Testing:</strong> It verifies that an app works correctly across its target languages and regional settings. It checks translated text, layout adjustments, date and currency formats, text direction, and other locale-specific elements.</p><h3>Manual Testing Vs Automated Testing for Android Apps</h3><p>Manual and automated testing both matter in Android app testing.</p><p><strong>Manual testing</strong> is better for judgment, exploration, and real user experience, whereas <strong>automated testing</strong> is better for speed, repeatability, and regression coverage.</p><figure><img alt="Comparison table of manual vs. automated testing highlighting differences in best use cases, speed, human judgment, repeatability, setup effort, maintenance, device coverage, long-term cost, ideal timing, and limitations." src="https://cdn-images-1.medium.com/max/1024/1*VDRUHKbKrtOfcaylUoJFVA.png" /></figure><h3>Selection Criteria I used for the top Android App Testing Tools</h3><p>To create this list, I looked at the criteria that matter most when testing Android apps at scale. The focus was on device coverage, automation support, debugging, integration options, and how well each tool fits real mobile QA workflows.</p><p>Here are the key aspects considered:</p><ul><li><strong>Real Device and OS Coverage:</strong> I looked at how well each tool supports testing across Android devices, OS versions, screen sizes, and manufacturer variations.</li><li><strong>Automation Capabilities:</strong> I evaluated support for automated test creation, execution, scripting, reusable test cases, and regression testing.</li><li><strong>Ease of Use:</strong> I considered how easy the tool is to set up and use for QA teams, developers, and mobile testers.</li><li><strong>Debugging and Reporting:</strong> I checked whether the tool provides screenshots, videos, logs, crash reports, performance data, and useful failure details.</li><li><strong>Integration Support:</strong> I reviewed compatibility with CI/CD tools, test frameworks, issue trackers, and development workflows.</li><li><strong>Manual Testing Support:</strong> I considered how well the tool supports exploratory testing, real-device sessions, gesture testing, and visual inspection.</li><li><strong>Performance and Stability Testing:</strong> I looked at whether the tool helps test app speed, responsiveness, resource usage, and behavior under real conditions.</li><li><strong>User Reviews and Market Feedback:</strong> I considered third-party reviews and user feedback to understand reliability, adoption, and real-world usability.</li></ul><h3>My Recommended Android App Testing Tools for 2026</h3><p>Android testing tools serve different needs. Some are built for automation frameworks, some focus on real-device testing, and others help with manual debugging, UI validation, or cross-platform test execution. The right choice depends on how your team builds, tests, and releases Android apps.</p><p><strong>Let’s take a look at the tools one by one:</strong></p><h3>Appium</h3><p>Appium is a code-first, open source testing tool suitable for Android applications. It works well for teams that want to automate mobile web apps without locking themselves into one platform or language.</p><p>Its biggest strength is cross-platform automation, especially when teams want to reuse mobile testing skills as they move on to iOS devices from Android.</p><figure><img alt="Screenshot of uiautomatorviewer inspecting an Android mobile app running on a Pixel 6 emulator, highlighting the resource ID, accessibility label, and UI hierarchy for the Transfer Money button." src="https://cdn-images-1.medium.com/max/1024/1*VcjpbQgNzWc2kxG6QMMCQw.png" /></figure><p><strong>Key features of Appium:</strong></p><ul><li><strong>Cross-platform mobile automation:</strong> Supports Android, iOS, mobile web, and hybrid apps through the WebDriver protocol.</li><li><strong>Language flexibility:</strong> Lets teams write tests in Java, JavaScript, Python, Ruby, C#, and other supported languages.</li><li><strong>Real device and emulator support:</strong> Can run tests on Android emulators, simulators, and real devices.</li><li><strong>Framework compatibility:</strong> Works with popular test frameworks and can integrate with CI/CD pipelines and device cloud platforms.</li></ul><p><strong>Pros and Cons of Appium:</strong></p><figure><img alt="Pros and cons of Appium mobile testing: a comparison of its open-source access, Android and iOS support, programming language compatibility, and Selenium integration against setup complexity, slower execution, coding requirements, and test flakiness." src="https://cdn-images-1.medium.com/max/1024/1*10w2dN3zmTZjGoDXaObUbQ.png" /></figure><p><strong>Pricing: </strong>Appium is open source and free to use.</p><p><strong>G2 Rating:</strong> <a href="https://www.g2.com/products/appium/reviews">4.4/5</a> (Based on 64 reviews)</p><h3>Espresso</h3><p>Espresso is the Android-native choice for UI testing. It is built for teams that want fast, reliable tests inside the Android ecosystem rather than a cross-platform automation layer.</p><p>One of its biggest strengths is synchronization. Espresso waits for the app’s main thread to become idle before running actions and assertions, which helps reduce timing issues in UI tests. This makes it a strong fit for Android teams that own the codebase and want stable tests for views, screens, and user interactions.</p><figure><img alt="Screenshot of Android Studio showing the Espresso UI Test Recorder interface with recorded user steps, element IDs, and an added UI assertion for a notes app." src="https://cdn-images-1.medium.com/max/1024/1*TpTtgqx9UOxLRKJl1qbf1A.png" /></figure><p><strong>Key features of Espresso:</strong></p><ul><li><strong>Native Android UI testing:</strong> Built for testing Android app UI behavior inside a single target app.</li><li><strong>Automatic synchronization:</strong> Waits for the app to become idle before running test commands.</li><li><strong>Clear test actions and assertions:</strong> Supports readable checks for clicks, text input, visibility, and UI state.</li><li><strong>Android Studio integration:</strong> Works naturally with Android Studio, Gradle, emulators, and real devices.</li></ul><p><strong>Pros and Cons of Espresso:</strong></p><figure><img alt="Pros and cons of Espresso Android UI testing framework, highlighting its fast and reliable native UI testing, strong synchronization, Android Studio and Gradle integration, and suitability for developer-owned testing, alongside its Android-only support, codebase access requirements, and limited cross-platform capabilities." src="https://cdn-images-1.medium.com/max/1024/1*CVYFvXdWUleeiWR2BsgibQ.png" /></figure><p><strong>Pricing:</strong></p><p>Espresso is part of Android’s testing ecosystem and is free to use.</p><p><strong>G2 Rating:</strong> Unavailable</p><h3>BrowserStack App Automate</h3><p>BrowserStack App Automate is a cloud-based mobile app testing platform that enables teams to run automated tests on real Android and iOS devices. It eliminates the need for an in-house device lab, allowing QA teams to validate apps across device models and OS versions.</p><p>App Automate is particularly useful for teams looking to expand device coverage, accelerate regression testing, and integrate automated tests into CI/CD pipelines without changing their existing frameworks.</p><figure><img alt="A screenshot of a Real-Device Android Testing interface showing a Samsung Galaxy S23 Ultra virtual session, highlighting features such as ‘Capture test evidence’, ‘Debug with device logs’, and developer testing tools." src="https://cdn-images-1.medium.com/max/1024/1*a6Y94ZWE7qeQ3DNIqLC2Dg.png" /></figure><p><strong>Key features of BrowserStack App Automate:</strong></p><ul><li><strong>Real Android devices:</strong> Tests apps across real Android devices and OS versions to identify device-specific issues.</li><li><strong>Framework support:</strong> Supports Appium, Espresso, Flutter, Detox, and Maestro for mobile app automation.</li><li><strong>Parallel test execution:</strong> Runs tests simultaneously across multiple devices to reduce execution time.</li><li><strong>Native feature testing:</strong> Supports biometrics, camera image injection, geolocation, and network simulation.</li><li><strong>Debugging tools:</strong> Provides screenshots, video recordings, device logs, and network logs to help investigate test failures.</li><li><strong>CI/CD integration:</strong> Integrates with CI/CD pipelines and supports secure local testing of apps hosted in development or staging environments.</li></ul><p><strong>Pros and Cons of BrowserStack App Automate:</strong></p><figure><img alt="Pros and cons of BrowserStack App Automate, highlighting real Android device access, support for popular mobile automation frameworks, parallel test execution, and detailed debugging logs and recordings, alongside subscription costs, test script maintenance, plan-dependent parallel testing capacity, and initial setup requirements." src="https://cdn-images-1.medium.com/max/1024/1*9bI3M9Nug4oiRpTyG2UE9w.png" /></figure><p><strong>Pricing:</strong> The Device Cloud plan starts at $199/month, billed annually. Pricing varies by plan and testing requirements.</p><p><strong>G2 Rating:</strong> <a href="https://www.g2.com/products/browserstack/reviews">4.4/5</a> (Based on 3344 reviews)</p><h3>Selendroid</h3><p>Selendroid is an older Android automation framework built around the Selenium style of testing. It brings WebDriver-like automation to Android native and hybrid apps, and mobile web testing.</p><p>It is not as commonly used today as Appium or Espresso, but it can still be useful for teams with older Android test suites or Selenium-heavy workflows. Selendroid supports real devices and emulators, and it can also be added as a node in Selenium Grid for scaling test execution.</p><figure><img alt="A screenshot of the Eclipse IDE showing the step-by-step process to run a Selendroid test using TestNG, including context menus and the final test run results in the console." src="https://cdn-images-1.medium.com/max/1024/1*_fXH1fx1RflW-9sQxL9R8Q.png" /></figure><p><strong>Key features of Selendroid:</strong></p><ul><li><strong>Selenium-based Android testing:</strong> Lets teams write Android tests using the Selenium client API.</li><li><strong>Native, hybrid, and mobile web support:</strong> Can automate different Android app types from the UI layer.</li><li><strong>Real device and emulator support:</strong> Runs tests on Android hardware devices and emulators.</li><li><strong>Selenium Grid support:</strong> Can be integrated into Selenium Grid for parallel and distributed test execution.</li></ul><p><strong>Pros and Cons of Selendroid:</strong></p><figure><img alt="Pros and cons of Selendroid, an open-source Android test automation framework, highlighting its Selenium compatibility, support for native, hybrid, and mobile web apps, and real device and emulator testing, alongside limited adoption, newer Android version support, a smaller community than Appium, and outdated setup and maintenance." src="https://cdn-images-1.medium.com/max/1024/1*zkj8ECQRzDL5JCNmzM_7ZQ.png" /></figure><p><strong>Pricing: </strong>Selendroid is open source and free to use.</p><p><strong>G2 Rating:</strong> <a href="https://www.g2.com/products/selendroid/reviews">4.3/5</a> (Based on 4 reviews)</p><h3>Robot Framework</h3><p>Robot Framework is a keyword-driven automation framework with a great focus on readability. Teams can write tests in a structured format that is easier to understand for less code-heavy manual testers and QA professionals.</p><p>For mobile automation, it usually works with Appium through AppiumLibrary. This makes it useful when teams want Android automation with reusable keywords, readable test cases, and strong reporting.</p><figure><img alt="Dashboard interface showing a Login Tests Report with summary, statistics, test details, and execution logs." src="https://cdn-images-1.medium.com/max/1024/1*7kPicHNBZZoaozfBihwJPA.png" /></figure><p><strong>Key features of Robot Framework:</strong></p><ul><li><strong>Keyword-driven test design:</strong> Lets teams write readable test cases using reusable keywords.</li><li><strong>AppiumLibrary support:</strong> Enables Android and iOS testing through Appium.</li><li><strong>Extensible framework:</strong> Supports custom libraries in Python, Java, and other languages.</li><li><strong>Built-in reports and logs:</strong> Generates detailed execution logs and reports after test runs.</li></ul><p><strong>Pros and Cons of Robot Framework:</strong></p><figure><img alt="Pros and cons of Robot Framework for test automation, highlighting its easy-to-read keyword-driven approach, detailed reporting and execution logs, support for mobile, web, API, and system testing, and open-source community, alongside Appium setup requirements for Android testing, challenges managing keyword-heavy tests at scale, slower performance compared to Espresso for Android-only testing, and the need for proper framework structure." src="https://cdn-images-1.medium.com/max/1024/1*IjuLunt_nnNzYN-Mww7cvg.png" /></figure><p><strong>Pricing: </strong>Robot Framework is open source and free to use.</p><p><strong>G2 Rating:</strong> Unavailable</p><h3>Android Studio</h3><p>Android Studio is not just an IDE, but also a native testing workspace. It gives developers and QA teams the tools to write, run, debug, profile, and inspect Android apps from one environment. Since it is closely tied to the Android platform, teams can run from local tests to device-based debugging without leaving the Android development workflow.</p><figure><img alt="Screenshot of Android Studio showing a successful instrumented test suite run with 13 out of 14 tests passing and execution logs displayed in the bottom window." src="https://cdn-images-1.medium.com/max/1024/1*bqw87-uTB_Hr9NyQM5T2hg.png" /></figure><p><strong>Key features of Android Studio:</strong></p><ul><li><strong>Built-in test support:</strong> Lets teams create, run, and view test results from the IDE.</li><li><strong>Android Emulator:</strong> Supports testing across Android devices, API levels, screen sizes, network speeds, rotation, location, and hardware sensors.</li><li><strong>Debugging and profiling tools:</strong> Helps inspect crashes, memory usage, CPU usage, layout behavior, and performance issues.</li><li><strong>Gradle and command-line support:</strong> Allows teams to run tests locally or through CI/CD pipelines.</li></ul><p><strong>Pros and Cons of Android Studio:</strong></p><figure><img alt="A comparison table on a bright white background with green and red header rows labeled ‘Pros’ and ‘Cons’." src="https://cdn-images-1.medium.com/max/1024/1*hihQdyg28106XhIY2rpzfw.png" /></figure><p><strong>Pricing: </strong>Android Studio is free to download and use.</p><p><strong>G2 Rating:</strong> <a href="https://www.g2.com/products/android-studio/reviews">4.5/5</a> (Based on 648 reviews)</p><h3>Calabash</h3><p>Calabash is a legacy mobile testing framework that got popular for behavior-driven Android and iOS testing. It focuses on BDD-first, with teams writing tests in Cucumber-style scenarios, then using Ruby step definitions to automate app interactions.</p><p>For modern Android teams, Calabash is usually not the first choice anymore. Active development has declined, and many teams have moved to Appium, Espresso, or other newer tools.</p><figure><img alt="Screenshot of a legacy Xamarin Test Cloud dashboard showing test run results from May 11, 2017. It displays test bundle preparation steps on the left terminal window, a 4/13 failed test summary pie chart in the center, and device/OS coverage details on the right." src="https://cdn-images-1.medium.com/max/1024/1*IRk5qRELG0ltokfV2I1Cpg.png" /></figure><p><strong>Key features of Calabash:</strong></p><ul><li><strong>BDD-style test automation:</strong> Uses Cucumber syntax so tests can be written in a more readable scenario format.</li><li><strong>Android and iOS support:</strong> Was built for cross-platform mobile app automation.</li><li><strong>Ruby-based test execution:</strong> Uses Ruby step definitions to connect readable scenarios with app actions.</li><li><strong>Native app interaction:</strong> Can automate user actions and verify UI behavior in mobile apps.</li></ul><p><strong>Pros and Cons of Calabash:</strong></p><figure><img alt="A comparison table on a bright white background with green and red header rows labeled ‘Pros’ and ‘Cons’" src="https://cdn-images-1.medium.com/max/1024/1*0YaULBIf1bthYdJiyQrU0A.png" /></figure><p><strong>Pricing: </strong>Calabash is open source and free to use.</p><p><strong>G2 Rating:</strong> Unavailable</p><h3>Perfecto</h3><p>Perfecto is an enterprise-focused mobile testing platform for teams that need secure real-device testing at scale. Perfecto works well when teams need broad device coverage and it is especially useful for large organizations that need stability and repeatable testing across distributed teams.</p><figure><img alt="Screenshot of the Perfecto Android Testing cloud workspace showing automated test steps, a live device session running a stock market app, run guidance prompts, and assertion settings." src="https://cdn-images-1.medium.com/max/1024/1*28VX-6qms6upHuid0-wRSA.png" /></figure><p><strong>Key features of Perfecto:</strong></p><ul><li><strong>Real Android device cloud:</strong> Provides access to real Android and iOS devices for interactive and automated testing.</li><li><strong>Enterprise-grade infrastructure:</strong> Supports secure cloud environments, private cloud options, compliance needs, and high availability.</li><li><strong>Automation framework support:</strong> Works with tools like Appium, Espresso, Selenium, and other DevOps integrations.</li><li><strong>Debugging and analytics:</strong> Offers reports, CI dashboards, heatmaps, root cause analysis, and test failure insights.</li></ul><p><strong>Pros and Cons of Perfecto:</strong></p><figure><img alt="A comparison table on a bright white background with green and red header rows labeled ‘Pros’ and ‘Cons’" src="https://cdn-images-1.medium.com/max/1024/1*HHDQvxdRZ7bK7khUSgXIzA.png" /></figure><p><strong>Pricing: </strong>Perfecto uses subscription-based pricing. Enterprise pricing is custom and depends on device access, users, parallel executions, cloud type, and advanced testing needs.</p><p><strong>G2 Rating:</strong> <a href="https://www.g2.com/products/perfecto/reviews">4.4/5</a> (Based on 95 reviews)</p><h3>Best Practices for Testing Android Applications</h3><p>Here are some best practices to keep in mind when engaging in testing Android applications:</p><ul><li><strong>Test on real devices, not only emulators:</strong> Emulators are useful during development, but real devices reveal hardware, performance, camera, battery, and manufacturer-specific issues that simulated environments may miss.</li><li><strong>Prioritize device coverage based on user data:</strong> Focus on the Android versions, device models, screen sizes, and manufacturers your users rely on most. This helps teams identify compatibility issues without testing every available device.</li><li><strong>Automate repeatable test flows:</strong> Automate high-priority user journeys, such as login, checkout, and form submission, to speed up regression testing and catch recurring defects across builds.</li><li><strong>Test network and permission changes:</strong> Verify how the app behaves when connectivity drops, requests time out, or users grant, deny, or revoke permissions. These scenarios can expose failures that routine functional checks may overlook.</li><li><strong>Include accessibility testing early:</strong> Check TalkBack announcements, focus order, touch target sizes, contrast, and text scaling during development. Early checks help identify barriers before they affect critical user journeys.</li><li><strong>Run regression tests before every release:</strong> Rerun relevant tests after code changes, dependency updates, and Android OS updates to catch defects that affect existing functionality.</li></ul><h3>Conclusion</h3><p>The most recurring debate around Android app testing, in 2026, is about <em>how much real device testing is actually necessary and can emulators provide enough confidence to release an Android app?</em></p><p>I don’t think Android testing teams need to test on every device to release a reliable application. However, I also believe that the bigger mistake is to assume that passing tests on emulators or any high-end phone means that the app is ready to ship to everyone.</p><p>You should use emulators for fast repeatable checks during development. Validate critical user journeys on real devices before you release the application. Include devices with different performance capabilities and manufacturer customizations, especially those that your users particularly rely on. When production issues reveal gaps in your coverage, use them to refine your device testing strategy. The basic idea is to catch the failures your users are most likely to experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4badfced22d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Root Causes of Flaky Mobile Tests]]></title>
            <link>https://medium.com/@seanwintermere170/root-causes-of-flaky-mobile-tests-b6661b162ee6?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6661b162ee6</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 14:54:59 GMT</pubDate>
            <atom:updated>2026-11-03T14:54:59.953Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Illustration of a smartphone with a crossed-out retry icon and the text ‘Fix Flaky Mobile Tests’, representing the elimination of unstable test runs in CI/CD pipelines." src="https://cdn-images-1.medium.com/max/1024/1*LZWQ19f9NrqT6VsINCfuDQ.png" /></figure><p>I used to treat the “Re-run failed jobs” button in CI as a routine capability rather than a silent admission of defeat.</p><p>When your mobile test execution suite turns red, clicking retry until it turns green doesn’t mean your build is stable. It means your test harness is compromised. In our mobile release pipelines, retrying failed suites eroded developer trust, delayed critical hotfixes, and built an environment where legitimate regressions were regularly missed. This article goes deeper into why mobile suites become unstable. How to diagnose the underlying breakage, and the exact steps to eliminate non-deterministic execution once and for all.</p><h3>The High-Cost of the “Re-run Build” Anti-Pattern</h3><p>Treating flakiness as a tolerable inconvenience has a hidden, steep engineering cost. When our core native app grew beyond twenty engineers, the automatic reply cycle silently degraded pipeline efficiency.</p><ul><li><strong>CI Resource Burn:</strong> A retry rate of roughly 18% translated to an extra 42 compute hours per week on our build farm, inflating cloud pipeline costs.</li><li><strong>Release Velocity Bottlenecks: </strong>Pull requests that should have taken twenty minutes to clear took an average of 75 minutes due to second and third retry iterations.</li><li><strong>Alert Fatigue and Ignored Bugs:</strong> Real UI regressions got dismissed as transient flakiness, leading to production defects slipping into staged release rollouts.</li><li><strong>Developer Context Switching</strong>: Engineers constantly context-switched back to PRs they thought were complete, destroying focus and sprint velocity.</li></ul><h3>Root Causes of Flaky Mobile Tests</h3><p>Disposing of flaky mobile tests needs understanding the distinct execution constraints of mobile environments compared to desktop browsers.</p><h3>Root Cause 1: Asynchronous Driver Synchronization &amp; Dynamic Layout Shifts</h3><p>Most mobile test frameworks send commands over an intermediary layer (like WebDriverAgent on iOS or UiAutomator2 on Android). If an animation, network request, or layout calculation is still executing and then the driver runs a tap command. Often the driver either overlooks the hit area or targets an invalid coordinate. Hardcoded sleeps mask this temporarily, but during CI CPU throttle spikes, fixed delays inevitably breaks.</p><h3>Root Cause 2: Brittle Locator Strategies &amp; DOM/View-Hierarchy Shifts</h3><p>Depending on absolute accessibility hierarchies, auto-generated test IDs, or deep XPath queries causes rapid degradation. A minor UI change like wrapping a button in a new design token container or changing dynamic text strings, breaks the locator resolution. This takes place even though visually the component works as expected.</p><h3>Root Cause 3: OS Energy Management and Doze Mode Collision</h3><p>Android’s Doze Mode, background app refresh policies, and OEM aggressive battery-optimization profile often stop background app threads during long test runs. In one test suite, our background push-token handoff failed in 1 out of 6 runs. This was just because the OS dropped the process into an idle state during asset loading.</p><h3>Root Cause 4: Shared State, Dirty Test Artifacts, and Network Variance</h3><p>Mobile operating systems persist tokens, SQLite databases, shared references, and Keychain/Keystore values across test runs unless specifically purged. If test B assumes an empty state but test A ran before it without a complete sandbox reset, unexpected authorization states or cached state data will cause test B to fail intermittently.</p><h3>Tooling Ecosystem</h3><p>Diagnosing non-deterministic failures needs purpose-built observability tools across different layers of your test stack.</p><h3>Flaky Test Analyzers (CI/CD)</h3><p>Specialized analytics tools (like Datadog CI Test Visibility or Buildkite Test Engine) track test execution health across branches, calculating failure variance, quarantine status, and flake scores over rolling periods.</p><p><strong>My take: </strong>These dashboards are invaluable for identifying system test drift across large teams. But they are diagnostic and not remediative. They tell you which test is failing 12% of the time but don’t enlighten you why the Android view hierarchy stalled on that specific render frame.</p><h3>Tracer and Profiler Utilities</h3><p>Mobile-native tracing tools like Android Studio Profiler, Perfetto, and Xcode Instruments record low level CPU spikes, thread contention, and memory leaks while automated test runs.</p><p><strong>My take: </strong>When you hit an obscure timing bug tied to background garbage collection, running an instrumented build with Perfetto is irreplaceable. The learning curve is steep, and configuring these utilities inside headless CI containers is extremely difficult. This makes it more suited for local debugging sessions.</p><h3>Interactive Real-Device Triage Platforms</h3><p>When a test fails exclusively in CI on specific OEM hardware, reproducing the issue interactively on an identical hardware and OS matrix is necessary.</p><h4>Appium Inspector</h4><p>Appium Inspector is a designated open-source desktop GUI that connects directly to active Appium sessions. It allows engineers to inspect the live application view hierarchy, query accessibility attributes, and manually trigger gestures to diagnose locator failures.</p><figure><img alt="Appium Inspector user interface displaying a mirrored Android app screen on the left, an App Source XML element tree in the center, and a Selected Element details panel on the right." src="https://cdn-images-1.medium.com/max/1024/1*y0zAm8oC01Hzgn8LPgBeug.png" /></figure><p><strong>Key Capabilities</strong>:</p><ul><li><strong>View Hierarchy Diagnostics</strong>: Inspects complete native XML trees, compose nodes, and XCUI element hierarchies on running sessions.</li><li><strong>Live Locator Validation</strong>: Swiftly author and test XPath, accessibility ID, and class-chain selectors against live driver state.</li><li><strong>Code Snippet Generation</strong>: Documents basic manual communications and auto-generates driver commands across Java, Python, and JavaScript.</li><li><strong>Multi-Framework Compatibility</strong>: Interacts natively with iOS (XCUITest), Android (UiAutomator2), and Windows desktop automation drivers.</li></ul><p>Appium Inspector is free and open-source software (Apache 2.0 license) and has a rating of 4/5 on G2.</p><blockquote><strong>Personal opinion: </strong>Appium Inspector is a reliable open-source standard for authoring initial element locators, but its utility falls during deep triage. Latency over remote sessions can be disruptive, and it offers restricted debugging context when trying to trace device logs, video playback, and sensor state side-by-side.</blockquote><h4>Scrcpy</h4><p>Scrcpy is an ultra-lightweight, open-source utility built by Genymobile. It mirrors physical Android screens to a computer via USB or TCP/IP. It concentrates completely on low-latency display performance and direct hardware interaction without requiring any installed device agents.</p><figure><img alt="Screenshot of the Scrcpy GUI application dashboard showing the “Tools” section with ADB and Scrcpy status indicators (ADB Installed: Yes, Scrcpy Installed: Yes, Device Connected: Yes via USB), wireless connection configuration options, a resources section with official documentation and GitHub links, and useful command-line snippets at the bottom." src="https://cdn-images-1.medium.com/max/1024/1*tPydoh0z2VbB16Fvzlu-XA.png" /></figure><p><strong>Key capabilities:</strong></p><ul><li><strong>High-FPS Local Display Mirroring</strong>: Streams screen output at 30–60 fps with low operational latency (usually sub-70ms).</li><li><strong>Agentless USB/TCP Execution</strong>: Works fully over standard <em>adb</em> without needing root permissions, APK installations, or cloud accounts.</li><li><strong>Hardware-Level Gesture Testing</strong>: Good for debugging subtle multi-touch events, physical key presses, and fast biometric gesture sequence.</li><li><strong>Lightweight Resource Footprint</strong>: Minimal host CPU consumption, making it effortless to execute alongside heavy IDEs and local emulators.</li></ul><p>Scrcpy is free and open source. The tool has a rating of 4/5 on SourceForge.</p><blockquote><strong>Personal opinion: </strong>For local hardware desks, Scrcpy is good. It offers unmatched responsive control for manual replication. But, it lacks test analytics, session recordings, and centralized cloud orchestration. This makes it useless for distributed teams needing on-demand access to dozens of fragmented device models.</blockquote><h4>BrowserStack App Live</h4><p>It is an enterprise-grade, cloud-hosted real-device triage platform that offers immediate, browser-based access to thousands of physical Android and iOS devices. It brings together interactive manual testing with complete diagnostic telemetry, network emulation, and hardware-level mocking.</p><figure><img alt="Screenshot of the BrowserStack App Live dashboard displaying an uploaded application list on the left and an Android real device selection menu on the right with Google Pixel models highlighted." src="https://cdn-images-1.medium.com/max/1024/1*4vhUodDpC-YJTzVBhdM1Mw.png" /></figure><p><strong>Key Capabilities:</strong></p><ul><li><strong>Synchronized Tri-Pane Debugging</strong>: Correlates real-time device video, streaming systems console logs, and network traffic side-by-side.</li><li><strong>Advanced Environmental Simulation:</strong> Toggles dynamic network throttling profiles, GPS geolocation spoofing, and biometrics mocking on demand.</li><li><strong>Seamless CI/CD &amp; Jira Integration:</strong> One-click bug logging that attaches full session video replays, device metadata, and network traces straight into developer tickets.</li></ul><p>The pricing starts at $39 for individual users. App Live is ranked 4.4/5 on Gartner.</p><blockquote><strong>Personal opinion:</strong> In terms of solving the triage lifecycle end-to-end, BrowserStack App Live edges ahead as the more better platform. Its real-time device log aggregation, integrated network emulation, and ability to replicate edge-case OEM behaviors on physical hardware make it substantially faster for isolating device-specific flakiness than juggling local hardware rigs or standalone utilities.</blockquote><h3>Best Practices to Remove Mobile Test Flakiness</h3><p>Converting an unstable suite into an actionable gate needs rigorous test engineering:</p><ul><li><strong>Adopt Explicit Dynamic Polling(AwaitForCondition):</strong> Replace static <em>sleep()</em> calls with condition-based polling that checks for component visibility, clickable state, and interaction readiness before issuing gestures.</li><li><strong>Enforce Hermetic Isolation(Full state &amp; Keychain cleanse): </strong>Each test must start from a clean state. Automate the clearing of local storage, app caches, and authentication keys between individual test cases rather than depending on suite-level teardowns.</li><li><strong>User Resilient Semantic Locators(Unique native TestIDs/Tags): </strong>Standardize on dedicated accessibility attributes (such as <em>testID</em> on React Native, <em>accessibilityIdentifier</em> on iOS, or <em>testTag</em> on Jetpack Compose). Never use text contents or structural XPaths that shift with design revamps.</li><li><strong>Mock External Network Boundaries(WireMock/Local Proxy):</strong> Decouple end-to-end UI verification from live backend staging environments. Use local mocking proxies (like WireMock or MockWebServer) to remove flakiness caused by staging server hiccups, latency spikes, or third-party API rate limits.</li></ul><h3>The Shift to Deterministic Build Artifacts</h3><p>Tackling flakiness at the test code level only addresses half the problem. If your build outputs vary depending on transient environment states, tests will mirror that instability.</p><p>The next evolution is treating your mobile test targets as deterministic build artifacts. This means pre-compiling hermetic test bundles where all dynamic dependency updates, third-party SDK telemetries, and A/B feature flag variants are strictly pinned at compile time.</p><p>In future, your CI pipeline should inject feature-flag states and configuration flags directly into your build configuration rather than toggling them via the UI during setup. Decoupling UI logic verification from dynamic external configuration converts non-deterministic test suites into dependable, repeatable regressions checks that teams can trust.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6661b162ee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[App Parallel Testing Software for Android and iOS]]></title>
            <link>https://medium.com/@seanwintermere170/app-parallel-testing-software-for-android-and-ios-da69edf6e7b4?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/da69edf6e7b4</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[mobile-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 10:31:01 GMT</pubDate>
            <atom:updated>2026-10-31T10:31:01.556Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A simplified diagram showing the workflow of parallel mobile testing for Android and iOS. It starts with CI/CD build execution, moves to 100% concurrent execution on multiple devices, and ends with a budget split showing parallels allocated across PR builds, Nightly runs, and Retries, illustrating the concept of sharing one concurrency budget." src="https://cdn-images-1.medium.com/max/1024/1*loQadyrPSDjSKWboucDMqA.png" /></figure><p>Did you know a mobile test can fail without running a single step? If it waits too long for a free device in a cloud queue, it gets cancelled, and on the device clouds I checked, that cutoff is 15 minutes by default.</p><p>In most cases, the test itself is fine and the device plan is the problem. Two or three pipelines are drawing from the same pool without knowing about each other. But setting up app parallel testing software takes only a few minutes, so I’ve spent most of this article on sharing that pool well.</p><h3>What Is Parallel Mobile Test Execution?</h3><p>Parallel mobile test execution means running several test sessions at once, each on its own device. A suite of 300 two-minute tests needs about 10 hours on one device and roughly 30 minutes on 20.</p><p>In practice, the speed-up is smaller than that division suggests, and for two reasons. Every session spends time on setup before the first test step, such as installing the app and starting the driver, and that cost doesn’t shrink when you add devices. The run also finishes only when its slowest shard does, so a shard full of long checkout tests sets your build time.</p><p>That’s why I split suites by recorded test duration instead of by test count. It’s also why the whole approach depends on every session getting a device the moment it asks for one.</p><h3>App Parallel Testing Software for Mobile Test Automation</h3><p>All four options below handle cloud mobile testing, and each one runs Appium, Espresso, and XCUITest suites on real devices.</p><p>Device counts and framework lists look similar across vendors. So when I compare app parallel testing software, I look at what happens when demand goes over the limit, going by what each vendor documents today.</p><h3>Sauce Labs</h3><p>On Sauce Labs’, you get public real devices, extra sessions wait 15 minutes by default, and you can stretch that to 30 minutes. The docs also warn that a wait below two minutes can stop tests from starting at all, which is good to know if you’re going to shorten it to fail faster.</p><figure><img alt="Sauce Labs test results dashboard showing an automated browser test navigating to DuckDuckGo." src="https://cdn-images-1.medium.com/max/1024/1*g6Ct4FSUD6O9VtPCAL0zlw.png" /></figure><p>Team-level limits come with Enterprise plans only, and you can’t set a limit for each user.</p><blockquote><strong>My take:</strong> The documented queue rules make it easier to plan, but team limits need an Enterprise plan. But without the enterprise plan, a single large run can still take a whole team’s devices.</blockquote><h3>BrowserStack App Automate</h3><p>App Automate queues extra sessions by default, and above 5 parallels the queue is the same size as your parallel count. In effect, you can ask for twice your plan before anything is rejected. Also, the session that waits more than 15 minutes is auto cancelled.</p><p>Queues are kept per user, so its easier to track down the shortcoming. If each pipeline runs under its own service account, each one gets its own queue, so a nightly backlog can’t fill the queue your PR builds is using.</p><p>The Plan API reports running and queued sessions against your limits, so a pipeline can check for room before it starts.</p><figure><img alt="BrowserStack App Automate dashboard showing App Profiling metrics for an iPhone 12 Pro Max, including an app size of 15.43 MB and 96.83% slow frames." src="https://cdn-images-1.medium.com/max/1024/1*IKIQ3UwHp2q14IPOMuJ_nA.png" /></figure><blockquote><strong>My take:</strong> This is the app parallel testing software I’d pick when several pipelines share one plan, since team reservations, per-user queues, and the Plan API cover each part of a shared budget. The catch is that none of it happens on its own, so you still have to wire those controls into CI.</blockquote><h3>Kobiton</h3><p>Kobiton focuses on the detail it keeps for each session. It can turn recorded manual sessions into Appium scripts, and its Session Explorer logs commands, screenshots, and device metrics for each run.</p><figure><img alt="Screenshot of the Kobiton dashboard showing the Private &amp; Local Devices tab with a red box highlighting the Device List icon in the left sidebar menu." src="https://cdn-images-1.medium.com/max/1024/1*vPlrV87R6RYskdkLR9ZhLQ.png" /></figure><blockquote><strong>My take:</strong> This helps when you’re chasing a single failure. Its published material doesn’t share concurrency limits or explain how sessions queue, though, so for capacity planning you’d need to measure that yourself first.</blockquote><h3>Katalon TestCloud</h3><p>Katalon’s app parallel testing software runs Katalon Studio, Appium, and Playwright tests on real devices and sends results to Katalon TestOps. Katalon’s product page promises parallel sessions in place of a queue, but it doesn’t state any session limits.</p><figure><img alt="Execution dashboard showing process execution logs, run statuses, execution IDs, and completion durations." src="https://cdn-images-1.medium.com/max/1024/1*xWEy0xPDxh9NPAsmjWcfeA.png" /></figure><blockquote><strong>My take: </strong>It makes the most sense if your team already works in Katalon Studio. A new budget will need a number to divide, so ask yourself Katalon for the concurrency limit before planning around it.</blockquote><h3>How to Run Mobile App Test Suites in Parallel?</h3><p>Before you split mobile app parallel test suites across more threads, two things need to be right. The config has to request the devices you expect, and the tests can’t share data.</p><h3>Configure Parallel Tests for Android and iOS</h3><p>For cross platform parallel app testing, I keep one codebase and list both platforms in one file. With the BrowserStack SDK, the session count is your platforms multiplied by parallelsPerPlatform, so this config asks for 8.</p><pre># browserstack.yml<br>userName: &lt;your-browserstack-username&gt;<br>accessKey: &lt;your-browserstack-access-key&gt;<br>app: bs://&lt;your-uploaded-app-id&gt;<br>parallelsPerPlatform: 4<br>platforms:<br>  - deviceName: Samsung Galaxy S24<br>    platformVersion: &quot;14.0&quot;<br>    platformName: android<br>  - deviceName: iPhone 15<br>    platformVersion: &quot;17&quot;<br>    platformName: ios</pre><p>I like having that number written in one place, because it’s the figure your concurrency budget has to account for.</p><p>If you run Appium parallel testing on your own machines, every session needs its own ports. The UiAutomator2 driver needs a unique udid and systemPort for each Android session, and the XCUITest driver needs a unique udid and wdaLocalPort for each iOS session.</p><p>Port clashes are the first thing I check when a local parallel run fails for no clear reason. On a device cloud, each session gets its own host, so this problem goes away.</p><h3>Isolate Test Sessions and Test Data</h3><p>Running tests in parallel brings out shared state that sequential runs were hiding. A test that passes alone and fails next to others is usually sharing something, so before I add threads, I check three things.</p><ul><li><strong>Accounts:</strong> Each thread logs in as its own test user, so two sessions never share a cart or a login.</li><li><strong>Data:</strong> Each test creates its own data with a unique name, because a check like “the first order in the list” breaks once another session adds an order.</li><li><strong>Order:</strong> No test relies on another one running first, because the order isn’t fixed once tests run in parallel.</li></ul><h3>Why Parallel Mobile Tests Queue in the Cloud?</h3><p>Each pipeline that uses your app parallel testing software usually looks fine on its own. For example, say your PR build asks for 8 sessions and your nightly run asks for 20. On a 25-parallel plan, that’s 28 requests for 25 slots, so 3 sessions wait.</p><p>Queue wait doesn’t grow evenly as you add load. At half your limit, a new session starts almost straight away, but near the limit, every session that finishes already has another one waiting for its device. That’s why a plan that’s busy 95% of the time feels far slower than one that’s busy 80% of the time.</p><p>If a few more PR builds start, the queue fills up too, and new sessions are rejected before they run. On BrowserStack, the error reads “All parallel tests are currently in use, including the queued tests.”</p><p>Retries make it worse because failures tend to arrive together. When a backend outage fails 30 tests at once, an automatic retry asks for 30 new sessions right when the pool is already full.</p><h3>How to Set a Shared Concurrency Budget?</h3><p>A concurrency budget is a short table that splits your app parallel testing software plan across the workloads that use it. The only hard rule is that the total matches your plan, because every parallel you promise twice turns into a queued session later.</p><p>For a 25-parallel plan, mine would look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6jB3HzbymQmrSd16qHs8jw.png" /></figure><p>I give PR builds a large share because a developer is waiting on each one, while nobody watches the nightly run finish. The small reserve keeps an urgent hotfix from sitting behind routine work.</p><h3>Allocate Capacity Across PR Builds and Nightly Tests</h3><p>Enforce the split in two places, because each one catches a different mistake.</p><p>In BrowserStack, I reserve parallels per team and run the PR and nightly pipelines under separate service accounts in separate teams. The reservation caps each workload, and the separate accounts give each one its own queue.</p><p>In CI, I cap how many jobs each workflow can start, so a single workflow can’t ask for more than its share.</p><pre># .github/workflows/pr-mobile-tests.yml<br>concurrency:<br>  group: pr-mobile-${{ github.head_ref || github.run_id }}<br>  cancel-in-progress: true<br><br>jobs:<br>  mobile-tests:<br>    runs-on: ubuntu-latest<br>    strategy:<br>      max-parallel: 2<br>      matrix:<br>        shard: [1, 2, 3, 4]<br>    steps:<br>      - uses: actions/checkout@v4<br>      - run: ./run-mobile-tests.sh --shard ${{ matrix.shard }}</pre><p>The cancel-in-progress setting stops the older run on a branch when a new commit lands, so three quick pushes don’t hold three sets of devices. Setting max-parallel to 2, with 4 sessions in each job, keeps this workflow at 8 sessions, which fits inside its budget of 10.</p><p>For the nightly run, I also check the Plan API before starting and wait until enough parallels are free. BrowserStack’s troubleshooting docs suggest the same approach, with one condition: the check should use the same credentials as the tests, so it reads the right team’s limit.</p><h3>Limit Retries and Release Unused Sessions</h3><p>I keep retries to one per test and run them as a small batch after the main run ends, so they never compete with the first pass for devices. A test that fails twice in a row is usually telling you something real.</p><p>I also skip retries when a large share of the suite fails at once. That pattern points to an outage or a broken environment, and retrying it just moves the queue around.</p><p>Idle sessions are the other way devices go to waste. BrowserStack ends a session after 90 seconds without an Appium command, and until then the device still counts against your plan. Calling driver.quit() in a finally block or a teardown hook returns the device as soon as a test ends, even when it fails.</p><h3>How to Measure Parallel Mobile Testing Performance</h3><p>After any change to the budget, I watch four numbers in my app parallel testing software and CI.</p><ul><li><strong>Build time:</strong> This tells you whether adding parallels made builds faster, or whether the slowest shard is still setting the pace.</li><li><strong>Queue wait:</strong> This is the time between asking for a session and the first command, and it shows whether workloads are colliding.</li><li><strong>Slot use:</strong> Running sessions divided by your team’s limit shows how full the plan is, and readings near 100% at peak usually come with queue wait.</li><li><strong>Retry rate:</strong> A rising retry rate usually means flaky tests are eating into the budget.</li></ul><p>If PR builds queue while the nightly share sits unused, I’d change the split before buying more parallels.</p><h3>Final Thoughts</h3><p>I think most teams buy more parallels the first time their app parallel testing software starts queueing. It helps for a while, until someone adds the next pipeline and the same queue comes back.</p><p>A three-row budget, team reservations, and capped retries cost nothing to set up. Once they’re in place, your queue wait and slot use numbers will show you whether you need more devices or a better split.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da69edf6e7b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best iOS App Testing Tools in 2026: Compared on Cost, Coverage, and Limits]]></title>
            <link>https://medium.com/@seanwintermere170/best-ios-app-testing-tools-in-2026-compared-on-cost-coverage-and-limits-8c9852bf77a5?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c9852bf77a5</guid>
            <category><![CDATA[ios-development]]></category>
            <category><![CDATA[mobile-development]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[test-automation]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 17:06:01 GMT</pubDate>
            <atom:updated>2026-10-30T17:06:01.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Illustration titled “iOS APP TESTING TOOLS Best in 2026: Cost, Coverage &amp; Limits” showing a laptop with Xcode code and an iPhone displaying test results, surrounded by icons for a checklist, stopwatch, piggy bank, cloud sync, and approval checkmark on a white background." src="https://cdn-images-1.medium.com/max/1024/1*Y2JI-SFQzGYowJI8oqo6sA.png" /></figure><p>When I compare iOS application testing tools, I look beyond feature checkboxes. A framework that only runs on simulators does not give me the same coverage as a real-device cloud, even if both support UI automation.</p><p>Two limitations show why that distinction matters. Firebase’s available-devices documentation states that iOS 18 and later devices do not support videos in test results. Detox does not drive physical iPhones at all. I would want to know both before choosing either tool.</p><p>For this comparison, I focus on three questions: what hardware can I test on, how will I be charged, and which documented limitations could leave gaps in my coverage?</p><h3>How These Tools Were Chosen</h3><p>EI used vendor documentation, changelogs, pricing pages and public issue trackers to compare the tools below, with the first week of September 2026 as the research window and Xcode 26.6 as the reference toolchain.</p><p>Where I reference a reported defect, I name the issue so you can check the affected configuration and follow the discussion. I treat those reports as specific risks to investigate, rather than proof that every team will encounter the same problem.</p><p>I give the four selection criteria the following weightings. These show what matters most in my recommendations; they are not numerical scores assigned to individual tools.</p><ul><li><strong>Capability and cross-platform reuse, 30%:</strong> I look at what the tool actually does: drive the app through the accessibility layer, instrument its process, or measure performance. I also consider how much test code I could reuse on Android.</li><li><strong>Simulator and real-device access, 30%:</strong> I distinguish simulator support from physical-device support, then check whether real-device access is available locally, through a paid cloud, or both. That tells me what coverage I can get and what I would need to pay for.</li><li><strong>CI support and maintenance load, 25%:</strong> I check whether the tool runs without manual interaction, how frequently its iOS driver changes, and which dependencies I would need to pin to keep CI stable across Xcode releases.</li><li><strong>Pricing model, 15%:</strong> I compare seat subscriptions, device minutes and device hours against the work I need to do. A short automated run and a four-hour exploratory session can have very different costs on the same platform.</li></ul><h3>10 Best iOS App Testing Tools in 2026</h3><h3>1. XCUITest</h3><p>XCUITest is Apple’s UI automation layer inside XCTest, driven from Xcode 26.6. It reaches simulators and physical devices, it is Swift or Objective-C only.</p><figure><img alt="Xcode IDE displaying Swift XCTest code for Instagram UI tests, showing testLaunchPerformance and testLogin functions." src="https://cdn-images-1.medium.com/max/1024/1*op63ai0Fx5JJVhcR2rr5uQ.png" /></figure><p>The cost I would watch with this free option is parallel execution. Apple Developer Forums thread 817089, filed on 26 February 2026 against Xcode 26.2, reports parallel suites passing on the first run and failing on the second with “No matching device” for cloned simulator UDIDs.</p><p>CoreSimulator leaves those clones stuck in a Creating state, and the reporter cleared simulators, deleted ~/Library/Developer/XCTestDevices and restarted Xcode without shifting it.</p><blockquote><strong>Verdict:</strong> I would start with XCUITest for a native iOS app, especially when developers own the tests. Before scaling the suite, I would budget for macOS runners and check that parallel execution stays reliable on the team’s Xcode version.</blockquote><h3>2. Appium</h3><p>Appium 3.7.0 shipped on 24 August 2026, and the iOS side runs through appium-xcuitest-driver 11.17.7 from 15 July 2026. The reason I would accept its setup cost is the ability to share an automation framework and reusable test logic across iOS and Android.</p><p>Driver maintenance is the recurring cost I would plan for. Versions 11.17.3 through 11.17.7 all shipped inside eight days in July 2026, including a WebDriverAgent scaling factor fix, so I would pin the driver in my CI image rather than resolve it at build time.</p><blockquote><strong>Verdict:</strong> I would choose Appium when one team owns both platforms and wants to share test infrastructure. I would still allow for platform-specific locators, behaviours and regular driver upgrades.</blockquote><h3>3. Maestro</h3><p>Maestro CLI 2.10.0 landed on 31 August 2026. Flows are YAML, the approach is black box against the iOS accessibility layer, and setup needs only xcode-select — install.</p><figure><img alt="Kotlin code editor running a Maestro UI automation test alongside an iPhone simulator displaying the Uber app ride selection screen in London." src="https://cdn-images-1.medium.com/max/1024/1*KE39vHb7M7PSRm-3_UZRrw.png" /></figure><p>I would check the platform support page before budgeting for device coverage. Maestro documents that it runs on any iOS Simulator managed by Xcode, and that scaling locally is difficult because of macOS hardware, which routes parallel and real-device execution to Maestro Cloud.</p><blockquote><strong>Verdict:</strong> I would shortlist Maestro for readable smoke tests and a quick first flow. I would account for its cloud costs separately if the team needs physical iPhones or more parallel execution than its Macs can support.</blockquote><h3>4. Detox</h3><p>Detox 20.51.3 is the current release for React Native projects. Its grey-box approach synchronises with the app’s idle state instead of relying on fixed sleeps. That makes it worth considering when timing-related failures are making a React Native suite difficult to maintain.</p><p>The changelog shows how much iOS 26 disturbed React Native tooling. Version 20.48.0 added simulator launch architecture selection for iOS 26 and later, 20.48.1 fixed the architecture flag for simulators below 26, and 20.50.0 handled the iOS 26.1 liquid glass overlay.</p><figure><img alt="JavaScript code editor showing a Detox E2E test script for a React Native login journey." src="https://cdn-images-1.medium.com/max/1024/1*vait7-yZeWv4L_uvnjo5vA.png" /></figure><p>The boundary I would keep explicit is physical-device coverage. It drives iOS simulators only and cannot run on a physical iPhone, which BrowserStack’s Detox FAQ states plainly and which issue 4694 left unresolved when it closed in May 2025.</p><blockquote><strong>Verdict:</strong> I would consider Detox for React Native regression tests, with a separate real-device check before release. I would not choose it for a native Swift app or treat a passing Detox suite as proof that the app works on physical iPhones.</blockquote><h3>5. Xcode Simulator</h3><p>The Simulator ships with Xcode 26.6, boots in seconds and scripts cleanly through xcrun simctl. I would use it for fast feedback during development and routine checks before moving to physical hardware.</p><figure><img alt="Xcode interface displaying SwiftUI code for LandmarksSplitView alongside an iPhone simulator preview of the Landmarks app." src="https://cdn-images-1.medium.com/max/1024/1*7wDrnCrrMYryYBYYErbSsw.png" /></figure><p>It cannot reproduce biometrics, camera input, cellular handoff, thermal throttling, GPU driver behaviour, production StoreKit, or push through production APNs. Your app also compiles for the Mac’s architecture against simulated frameworks, so a crash that depends on the device runtime never appears.</p><p><strong>Key Highlight:</strong> I would check both simulator directories when troubleshooting parallel runs. Test clones live in ~/Library/Developer/XCTestDevices, not in ~/Library/Developer/CoreSimulator/Devices, so erasing devices from Simulator.app leaves those test clones untouched.</p><blockquote><strong>Verdict: </strong>I would make the Simulator part of the daily development loop, then use physical iPhones for checks that depend on device hardware or runtime behaviour. It gives me fast feedback, but it would not be my final release check.</blockquote><h3>6. BrowserStack App Live</h3><p>I would use App Live for interactive testing and bug reproduction on real iPhones. Individual runs $39 per month billed annually with unlimited minutes, and Team Pro at $249 adds four devices in one session.</p><figure><img alt="Dashboard interface of BrowserStack Live running the Uber web app on a real iPhone, showing side control menus and device metadata." src="https://cdn-images-1.medium.com/max/1024/1*O-5r1BpV7ckLfkNTtpZeKw.png" /></figure><ul><li><strong>Image injection replaces a camera that is taped shut:</strong> Every lens in the fleet is physically covered, so you upload a JPG or PNG up to 288 by 288 pixels and it becomes the camera input for scan flows on iOS 13 and later.</li><li><strong>Biometrics and network conditions run from the toolbar:</strong> Face ID paths are exercised by choosing Pass, Fail or Cancel, and network simulation takes explicit bandwidth, latency and packet loss numbers.</li></ul><p>The strongest use case for me is reproducing a customer-reported failure. I would look for the reported iPhone model and iOS version in the available inventory, then check the same flow on that combination.</p><blockquote><strong>Verdict:</strong> I would choose App Live for manual release checks, exploratory testing and reproducing device-specific bugs without maintaining an iPhone collection. Its unlimited interactive minutes make the seat model worth considering for longer testing sessions, provided the required devices and OS versions are available.</blockquote><h3>7. Firebase Test Lab</h3><p>Firebase Test Lab runs XCTest on Google-hosted iPhones and iPads.</p><p>On Blaze, physical devices are free for 30 minutes a day then $5 per device hour, virtual devices are free for 60 minutes a day then $1 per device hour, and the no-cost Spark plan caps you at five physical tests a day.</p><figure><img alt="VS Code editor displaying a Flutter iOS project.pbxproj file with the PRODUCT_BUNDLE_IDENTIFIER line highlighted." src="https://cdn-images-1.medium.com/max/1024/1*_DhKRrQwAO6GwxyffUkW-Q.png" /></figure><p><strong>Key Highlight:</strong> I would check the available failure artifacts before relying on Test Lab for debugging. Firebase’s available-devices documentation states that iOS 18 and later devices do not support videos in test results, so those runs will not provide a recording of the failure.</p><blockquote><strong>Verdict: </strong>I would consider Firebase Test Lab for short XCTest runs where the free allowance or hourly pricing fits the workload. If a developer needs a video to understand an intermittent failure, I would plan another way to capture it.</blockquote><h3>8. Kobiton</h3><p>Kobiton bills device minutes rather than seats. Startup begins at $83 per month for 500 minutes, Accelerate at $399 for 3,000, and Scale is annual-only at 7,500 minutes, with Appium, XCUITest, Espresso and Selenium supported against real devices.</p><figure><img alt="Code block showing Appium setDriver method with Kobiton capabilities for Android calculator app testing." src="https://cdn-images-1.medium.com/max/1024/1*tWJveHwM-s4Q_vk2U9abGA.png" /></figure><p>I would estimate monthly usage before choosing a plan. Five hundred minutes is roughly eight hours, so one full day of exploratory testing could consume almost the entire Startup allowance.</p><blockquote><strong>Verdict:</strong> I would shortlist Kobiton for real-device testing when the team’s usage is predictable. For frequent exploratory sessions, I would compare the expected monthly minutes with an unlimited interactive plan before committing.</blockquote><h3>9. TestingBot</h3><p>TestingBot prices Live at $20 per month billed annually, Automated at $50 with 1,000 to 20,000 minutes by parallel count, and Automated Pro at $90 with unlimited minutes and up to 24 parallel tests.</p><blockquote><strong>Verdict:</strong> I would shortlist TestingBot when entry cost matters, then confirm that the chosen plan includes the required iPhones, iOS versions and testing mode. A low subscription price only helps if it covers the devices the team needs to test.</blockquote><h3>10. Apptim</h3><p>I would use Apptim to answer a different question: did this build become slower or more resource-intensive? It captures CPU, memory, frames per second, energy use and launch timings during a session, then produces a comparison report between two builds of the same app.</p><p>Apptim Desktop runs free forever on a personal licence, with unlimited local tests, reports stored on your own machine, and a one-user cap.</p><p>The CLI, cloud devices through AWS Device Farm and parallel sessions need the paid Custom plan, which Apptim quotes on a call rather than publishing a price.</p><p>On iOS it needs a connected device and the Xcode toolchain on macOS, so it will not slot into a Linux CI runner.</p><blockquote><strong>Verdict:</strong> I would add Apptim when functional tests pass but the team still needs to compare performance between builds. I would use it alongside UI automation, with a connected iPhone and a Mac available for iOS measurements.</blockquote><h3>Which iOS App Testing Tool Should be in Your Stack</h3><p>I would build the stack around who owns testing, which platforms the team ships, and how much maintenance it can support. These are the three starting points I would recommend.</p><ul><li><strong>Solo developer or small team:</strong> I would start with XCUITest on the Simulator and add physical-device checks before release. App Live’s Individual plan at $39 per month is one option if I need access to more iPhones than I own. I would add an automation cloud when execution time or device coverage justifies the cost.</li><li><strong>Product team with one QA:</strong> I would keep developer-owned smoke tests in XCUITest and give the QA real-device access for exploratory work and bug reproduction. I would add Apptim when the team needs repeatable performance comparisons between builds.</li><li><strong>Team shipping weekly on both platforms:</strong> I would consider Appium 3.7.0 with a pinned driver and enough parallel slots to finish within the build window. I would also reserve maintenance time for driver upgrades and platform-specific test changes.</li></ul><h3>Where iOS Testing Gets Expensive</h3><p>Each pricing page reads clearly on its own, and the comparison between them does not, because these products meter three different things. The table normalises them to what roughly $39 per month buys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oI5qgNaxhcKyyY1bO2NMlw.png" /></figure><p>Per-minute pricing stops looking cheap the moment a human starts testing. One four-hour exploratory pass is 240 device minutes, which already exceeds what $39 buys on a minute meter, before your automated suite has run once.</p><p>The cost most teams miss is macOS CI capacity. XCUITest and Detox both need macOS runners, hosted macOS minutes bill at a multiple of Linux minutes on every major provider, and the parallel-clone defect above means a failed retry burns them twice.</p><h3>A Green Test Suite Still Leaves Gaps</h3><p>A passing suite tells you the paths you wrote assertions for still work on the configuration you ran them on. Three classes of iOS failure sit outside that statement, and all three reproduce on demand.</p><ul><li><strong>Process death and state restoration:</strong> Terminate the backgrounded app with xcrun simctl terminate &lt;udid&gt; &lt;bundle.id&gt;, reopen from the app switcher, then confirm that a submitted request is not replayed and that form state restores.</li><li><strong>Permission revocation mid-session:</strong> Revoke camera or photo library access from Settings while the screen using it is foregrounded, because iOS terminates the app when a granted permission is withdrawn and most suites never exercise that path.</li><li><strong>Release build configuration:</strong> Run the same checks against a distribution build, because debug builds surface exceptions as warnings where release builds abort the process.</li></ul><p>React-native-screens issue 3940, filed 24 April 2026, shows the shape of it. An Expo SDK 54 app crashed 0.15 seconds after launch on an iPhone 17 Pro running iOS 26.4.1, with an uncaught NSException leaving the app in -[RNSTabBarController updateTabBarAppearance].</p><p>The reporter states it does not reproduce on simulators, and does not reproduce in a dev-client build connected to Metro. It appeared only in TestFlight store builds on physical A18 Pro hardware.</p><p>Both conditions sit outside an ordinary suite. The simulator misses it because the silicon is wrong and the debug build misses it because the configuration is wrong, so the pipeline stays green while every TestFlight user crashes on launch.</p><h3>Conclusion</h3><p>The advice circulating in QA communities has the cost argument backwards. XCUITest and the Simulator are free right up to the point where you need them to run wide and fast.</p><p>Past that point you are buying macOS runners and debugging CoreSimulator clone state instead of writing assertions.</p><p>The more damaging habit is treating simulator-capable tools as device-capable tools. Firebase Test Lab will not hand you a video from an iOS 18 device, Maestro needs its cloud for physical hardware, and Detox never touches it.</p><p>A table that marks those three as real-device tools is saying something false about your coverage.</p><p>So here is the question worth arguing about. When your iOS suite last went green, did it run on a physical device with the OS build your largest user segment is on, or on whichever simulator your CI image had cached?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c9852bf77a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top Reasons for App Store Rejections and How to Fix Them]]></title>
            <link>https://medium.com/@seanwintermere170/top-reasons-for-app-store-rejections-and-how-to-fix-them-dc497dd7bd4a?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc497dd7bd4a</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <category><![CDATA[ios-development]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 16:12:29 GMT</pubDate>
            <atom:updated>2026-10-26T16:12:29.790Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjQqgKiexuHLy8Wps8Tpfw.png" /></figure><p>Why did Apple reject my app? A simple question for almost every rejection email we’ve received. But in my experience, the answer is always almost the same, simple.</p><p>Apple’s own numbers point to a plain answer, which is that the reviewer usually found a problem nobody tested before submission.</p><p>Apple’s 2025 App Store Transparency Report counts 2,093,244 rejected submissions out of 9,100,620 reviewed. The Performance section of the guidelines, which covers crashes, bugs and incomplete apps, was cited more often than every other section put together.</p><p>That changes how I look at app store rejections. About half of what Apple cites is a quality problem a QA team could have found first. I’ll separate those from the policy calls testing can’t fix, and end with the test run I do on real devices before every submission.</p><h3>What Apple’s Numbers Say About App Store Rejections</h3><p>Apple publishes an App Store Transparency Report every year, and the 2025 edition shows where app store rejections pile up. The figures below come straight from that report.</p><ul><li><strong>The rejection rate:</strong> Apple reviewed 9,100,620 submissions and rejected 2,093,244 of them, which is about 23%. One app can be submitted many times, so these figures count review rounds and not unique apps.</li><li><strong>Performance leads the table:</strong> The Performance section drew 1,354,418 rejection citations. Safety, Business, Design, Legal and Other drew 1,350,329 between them, so Performance alone came out slightly ahead of all five combined.</li><li><strong>Second attempts work:</strong> Apple counted 387,087 submissions approved after an earlier rejection. Apple also notes that one submission can break several guidelines, which is why the citation totals add up to more than the rejections.</li></ul><p>On average, Apple’s App Review page says more than 40% of the issues that stay unresolved relate to guideline 2.1 App Completeness, which covers crashes, placeholder content and incomplete information.</p><p>The page also says App Review typically finishes 50% of submissions in under 24 hours and 90% in under 48 hours. So every avoidable rejection costs a day or two of waiting, on top of the time it takes to fix the problem.</p><h3>Rejections QA Could Have Caught</h3><p>Apple’s App Review Guidelines ask for final builds that were tested on a device, with working URLs, demo account details and a live backend. Every one of these app store rejections fails that bar in a way a tester could have spotted first.</p><p>The fixes are simple. The hard part is remembering to test the reviewer’s path through the app, along with the user’s path.</p><h3>Crashes and Bugs on the Reviewer’s Device</h3><p>Crashes are the classic cause of app store rejections under guideline 2.1. Apple’s help pages say reviewers run apps on real devices, connected to networks with real-world conditions and running the latest software.</p><p>The fix starts with testing the exact build you upload. I install it through TestFlight on a clean device, because a release build is compiled with different settings from the debug build on my desk.</p><p>There is also a small escape hatch for apps that are already live. If a bug fix update turns up other issues in review, Apple can offer to approve the fix and let you handle the rest in your next submission. This doesn’t apply to legal or safety issues.</p><h3>Broken Links, Placeholder Content and Dead Demo Accounts</h3><p>Apple files all three of these under guideline 2.1, even though they feel like paperwork. Each one takes minutes to check and a full review round to fix afterwards.</p><ul><li><strong>Links:</strong> Every link in the app has to work, and Apple requires a support link with current contact details plus a privacy policy link. I tap every external link in the release build, including the ones tucked inside settings and onboarding.</li><li><strong>Placeholder content:</strong> Apple says items that still contain placeholder text or images cannot be approved. A search for “lorem”, “TODO” and “test” across the string files catches most of it.</li><li><strong>Demo accounts:</strong> If any feature sits behind a login, Apple wants a valid username and password in the App Review Information section, with the backend switched on during review. For features that need special hardware or a hard-to-copy setup, Apple asks for a demo video or the hardware itself.</li></ul><p>I sign in with the exact demo credentials on submission day. An expired password, or a one-time code sent to a phone the reviewer can’t reach, stops the review on its first screen.</p><figure><img alt="Screenshot of the Apple Developer portal under Certificates, Identifiers &amp; Profiles during the process of registering a new App ID. The form shows the platform set to iOS, macOS, tvOS, watchOS, an App ID Prefix team ID, the description “Raywenderlich Emitron”, and an explicit Bundle ID set to “com.raywenderlich.raywenderlich”. Below the details section is a Capabilities list featuring options like Access WiFi Information, App Attest, and App Groups." src="https://cdn-images-1.medium.com/max/1024/1*gPs_CEWQymAoEHzeztPrEQ.png" /></figure><h3>Missing Permission Strings and Account Deletion</h3><p>Both of these sit under guideline 5.1.1, Apple’s data collection rule. Both tend to break during updates, when a new SDK or sign-up flow arrives.</p><p>Any API that reads sensitive data needs an Info.plist key with a purpose string explaining clearly why the app needs it, ideally with an example.</p><p>App Store Connect flags a missing key at upload with error ITMS-90683, and that message warns that third-party SDKs can trigger it even when your own code never calls the API.</p><p>Vague wording that passes upload can still be rejected by the reviewer. So I list every purpose string from the exported .ipa and read them the way a reviewer would:</p><pre>plutil -p Payload/YourApp.app/Info.plist | grep UsageDescription</pre><p>Account deletion is the other common miss. Under guideline 5.1.1(v), an app that supports account creation must let people start deleting the account inside the app.</p><p>Apple’s help page adds that deactivation alone is not enough, and that guest accounts count too. Apps outside regulated industries should not force a phone call or email, and Sign in with Apple tokens should be revoked through Apple’s REST API.</p><p>In my test plan, deletion is a real test case. I create an account, delete it from settings, and confirm the old login no longer works.</p><h3>In-App Purchases the Reviewer Can’t Complete</h3><p>Guideline 2.1(b) says in-app purchases must be complete, up to date, visible to the reviewer and working. Reviewers buy in the same sandbox you test in, so a purchase that fails for you there will fail for them.</p><ul><li><strong>Agreements and products:</strong> Paid purchases don’t work in review until the Account Holder has accepted the Paid Apps Agreement and it is in effect. The app should also load product details through StoreKit, so the screen matches what App Store Connect holds.</li><li><strong>Restore and disclosure:</strong> Guideline 3.1.1 asks for a restore mechanism, and guideline 3.1.2(c) asks you to describe what the user gets before they subscribe. I test restore on a fresh install with the same sandbox account, because developers rarely walk that path.</li></ul><p>BrowserStack App Automate runs this flow on real iOS devices with no extra capability. The test enters the App Store credentials when the purchase sheet appears, and the device is wiped after the session.</p><p>On Android, the appStoreConfiguration capability signs in a Google account at session start, and the docs recommend a fresh account used nowhere else.</p><h3>iPad Layouts and IPv6 Networks</h3><p>These two catch teams who only test on the phone in their pocket and the office Wi-Fi. Both are written into the guidelines and are easy to reproduce.</p><p>Guideline 2.4.1 says iPhone apps should run on iPad whenever possible. Apple’s help pages explain that an iPhone app with no iPhone-only hardware dependency installs on iPad in compatibility mode, even if you remove iPad as a destination in Xcode.</p><p>So if the app can be installed on an iPad, I test it on an iPad, including rotation and the on-screen keyboard.</p><p>Guideline 2.5.5 says apps must be fully functional on IPv6-only networks. Code with hard-coded IPv4 addresses can fail there while working perfectly on normal Wi-Fi.</p><p>A Mac can share its connection as an IPv6-only network through the Create NAT64 Network option in Internet Sharing, which appears when you hold the Option key. Apple engineers on the developer forums explain that software without IPv6 support simply fails on this network.</p><figure><img alt="Screenshot of the macOS System Preferences Sharing window. In the left sidebar, “Internet Sharing” is selected. On the right panel, “Share your connection from” is set to “Thunderbolt Ethernet,” and “Wi-Fi” is checked under “To computers using:”. A red box highlights the “Wi-Fi Options…” button at the bottom right." src="https://cdn-images-1.medium.com/max/1024/1*zJ4fdE6WMJsFdGOSCXhzcQ.png" /></figure><h3>Rejections That Aren’t QA’s Job</h3><p>Some app store rejection reasons are decisions, and a tester can only point them out. They trace back to the planning stage, so the fix belongs to product, legal or leadership.</p><ul><li><strong>Minimum functionality:</strong> Guideline 4.2 rejects apps that feel like a repackaged website, such as web clippings, content aggregators and collections of links. More testing won’t change that outcome, because the app needs native features worth installing.</li><li><strong>Crowded categories and copycats:</strong> Guideline 4.3 names dating, flashlight, sound effects, wallpaper, simple timers and fortune telling as categories where Apple expects a meaningfully different experience. Guideline 4.1 blocks apps that copy another app or use its name or icon.</li><li><strong>Payment model:</strong> Guideline 3.1.1 requires Apple’s in-app purchase for digital content, and the rules for linking to outside payment now differ by storefront. That is a business and legal call, and QA can only check the build against it.</li><li><strong>Who submits the app:</strong> Apps in regulated fields such as banking, healthcare, gambling and air travel should come from the legal entity that provides the service. An individual account hits guideline 5.1.1(ix) however clean the build is.</li></ul><p>These app store rejections are not rare either. In 2025, Design drew 415,532 citations and Legal drew 495,673, although Legal also includes privacy items that testing can catch.</p><p>If you believe the reviewer got it wrong, you can reply on the App Review page in App Store Connect, book a 30-minute App Review appointment, or appeal to the App Review Board. Apple asks for one appeal per rejected submission, with specific reasons the app complies.</p><h3>Google Play Rejections Follow Different Rules</h3><p>Google puts more of its checking before the first public release. A Google Play app rejected at the production access step has often failed a testing requirement, which has no direct match on Apple’s side.</p><figure><img alt="A comparison table showing app release requirements for the Apple App Store and Google Play. The table compares four specific gates: testing before release, review timing, platform floor, and login-gated features." src="https://cdn-images-1.medium.com/max/1024/1*dUbPa2sqPOAlcmwGEM2DLQ.png" /></figure><p>The 14-day rule trips people up, because testers who opt out early don’t count and a tester who rejoins starts a fresh run of days.</p><p>The production access form also asks how testers used the app and what you changed after their feedback. Google says plainly that review is not meant to be your troubleshooting step, and points developers to the pre-launch report in Play Console for warnings, errors and performance issues.</p><figure><img alt="Screenshot of the Google Play Console dashboard showing a list of draft applications in internal testing, highlighting app status, zero installed audience, and last updated dates." src="https://cdn-images-1.medium.com/max/1024/1*GUlBCYWBKIS5NBN8eIml6Q.png" /></figure><h3>A Pre-Submission Test Run on Real Devices</h3><p>My pre-submission testing follows the reviewer’s path, because that path decides most app store rejections. It runs on the release candidate, the same build that goes to App Store Connect or Play Console.</p><ol><li><strong>Install the release build cleanly:</strong> I install the TestFlight or internal testing build on a device with no previous data. Then I open it with no account signed in, the way a reviewer meets it.</li><li><strong>Run the regression suite on real devices:</strong> App Automate runs Appium, XCUITest and Espresso suites on real iPhones, iPads and Android phones, so iPad and older OS coverage happens in one pass. Its network simulation adds slower profiles such as 2G and 3G.</li><li><strong>Walk the reviewer’s path by hand:</strong> I sign in with the exact demo credentials, tap every link, and complete a sandbox purchase. Then I restore that purchase on a fresh install and delete the test account.</li><li><strong>Check the network edge cases:</strong> I repeat the core flow on the Mac’s NAT64 network for IPv6, and once with airplane mode on to see how the app handles no connection.</li><li><strong>Read the build and the listing:</strong> I list the purpose strings, confirm the privacy policy and support URLs load, and compare the screenshots with the current UI. On Android, I read the pre-launch report before promoting the build.</li></ol><p>This run is the cheapest fix for app store rejections that I know of, because every step costs less than one extra review round.</p><h3>Final Thoughts</h3><p>A rejection email is a bug report from the one tester you can’t schedule. It arrives a day or two after submission, names a guideline number, and costs a full review round to close.</p><p>Apple’s 2025 numbers say about half of those reports point at performance problems. Google now states in its own help pages that reviewers are not there to troubleshoot your app.</p><p>My view is that the reviewer should never be the first person to open the release build on an iPad, over an IPv6 network, with the demo account. Once that run is part of the test plan, the app store rejections that remain are the ones worth arguing about.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc497dd7bd4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Run Espresso Tests on Local and Cloud Android Devices?]]></title>
            <link>https://medium.com/@seanwintermere170/how-to-run-espresso-tests-on-local-and-cloud-android-devices-6ed5669b0bde?source=rss-04e3ec93b532------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ed5669b0bde</guid>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[mobile-development]]></category>
            <category><![CDATA[androiddev]]></category>
            <category><![CDATA[android]]></category>
            <category><![CDATA[test-automation]]></category>
            <dc:creator><![CDATA[Sean Wintermere]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 22:01:01 GMT</pubDate>
            <atom:updated>2026-10-17T22:01:01.590Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Illustration showing the process of running Espresso tests on local Android devices and cloud devices using a single APK build." src="https://cdn-images-1.medium.com/max/1024/1*Qxw2fELXc777JI14dWb-hA.png" /></figure><p>I have lost a lot of hours to a test that passes on my Pixel and fails on a Samsung ten minutes later. My first instinct used to be that Samsung had done something strange, and I was usually wrong.</p><p>So now I hold the app APK and the Espresso test APK still and change only the device underneath them. One note before the commands, because Espresso runs inside your app process as Android instrumentation and cannot run on an iPhone. You would reach for XCUITest there.</p><h3>Prepare One Espresso Build for Multiple Devices</h3><p>If every device gets its own build, I cannot tell whether the device or the build caused a difference. So I build once and reuse the same two files everywhere.</p><pre>./gradlew :app:assembleDebug :app:assembleDebugAndroidTest</pre><p>On a standard app module, Gradle drops the outputs in these two paths. Product flavors move them.</p><pre>app/build/outputs/apk/debug/app-debug.apk<br>app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk</pre><p>None of this works until you declare the runner, so I check that first.</p><pre>android {<br>    defaultConfig {<br>        testInstrumentationRunner &quot;androidx.test.runner.AndroidJUnitRunner&quot;<br>    }<br>}</pre><p>I write down the versions Gradle resolved rather than the ones the docs print. Android’s Espresso setup page still shows 3.6.1, while the current stable release is 3.7.0 from 30 July 2025.</p><pre>./gradlew :app:dependencies \<br>  --configuration debugAndroidTestRuntimeClasspath</pre><ul><li><strong>I turn on Test Orchestrator and accept the slower run:</strong> Setting execution ‘ANDROIDX_TEST_ORCHESTRATOR’ with clearPackageData: ‘true’ gives every test its own Instrumentation invocation and clears shared state, which kills the most common reason a suite passes on my desk phone and fails on a freshly provisioned one.</li><li><strong>Check the Orchestrator version too:</strong> It has sat at 1.6.1 since 31 June 2024 while the rest of the line shipped in July 2025, so plenty of projects run a mismatched pair.</li></ul><figure><img alt="macOS Terminal window showing the directory listing of Android debug and androidTest APK build outputs." src="https://cdn-images-1.medium.com/max/1024/1*mj7nlSX-Q3F30d6E62kVoA.png" /></figure><h3>Run Espresso Tests on a Connected Android Device</h3><p>I always start with a phone on my desk, because adb gives me a fast debugging loop. Cloud devices win on breadth and lose on the first hour of diagnosis.</p><h3>Connect and Identify Your Pixel or Samsung Device</h3><p>Before I run anything I record what each device actually is, because I quote that record when a failure turns out to be environmental.</p><pre>adb devices -l<br>adb -s PIXEL_SERIAL shell getprop ro.product.model<br>adb -s PIXEL_SERIAL shell getprop ro.build.version.sdk</pre><p>Then I collect the four values that explain most visibility failures, on both phones.</p><pre>adb -s PIXEL_SERIAL shell wm size<br>adb -s PIXEL_SERIAL shell wm density<br>adb -s PIXEL_SERIAL shell settings get system font_scale<br>adb -s PIXEL_SERIAL shell getprop ro.product.locale</pre><p>Density and font scale decide whether a view sits fully on screen, and that decides whether Espresso’s visibility check passes before your assertion runs.</p><figure><img alt="Terminal window on a MacBook showing the output of the adb devices -l command, displaying attached Android devices including a Google Pixel 7 and a Samsung Galaxy S23." src="https://cdn-images-1.medium.com/max/1024/1*RheccMMytPEjGKh0sXYA6w.png" /></figure><h3>Install Both APKs and Execute the Test Suite</h3><p>Installation takes two commands with different flags, and the flags trip people up. Install the app APK, then the test APK, then confirm the runner.</p><pre>adb -s PIXEL_SERIAL install -r -g app-debug.apk<br>adb -s PIXEL_SERIAL install -r -t app-debug-androidTest.apk<br>adb -s PIXEL_SERIAL shell pm list instrumentation</pre><ul><li><strong>The test APK needs the -t flag:</strong> Gradle marks test APKs as testOnly, and skipping the flag fails the install in a way that later looks like a missing runner, which sends you hunting in the wrong place.</li><li><strong>I use -g to grant manifest permissions at install time:</strong> It clears out a category of OEM permission dialog, though I drop it when I am testing the permission flow itself.</li><li><strong>An empty pm list instrumentation means the test APK never installed:</strong> Fix that before you read your test code for faults.</li></ul><p>The run needs -w, because that flag makes the shell wait for instrumentation to finish. When I compare two devices I pin a single method rather than running the whole suite.</p><pre>adb -s PIXEL_SERIAL shell am instrument -w \<br>  -e class com.example.tests.LoginTest#validLogin \<br>  com.example.app.test/androidx.test.runner.AndroidJUnitRunner</pre><p>The option I almost never see anyone use is -e log true. It reports which tests your filters would select and runs none of them, so you can check a filter before it burns device time.</p><p><strong>Fun fact:</strong> If you pass both package and class, am instrument gives the package filter precedence. A CI script that builds both from config runs the entire package and hands you a green pass for a class it never executed.</p><figure><img alt="macOS Terminal executing adb shell am instrument for com.example.tests.LoginTest#validLogin on a Pixel 7 emulator, finishing in 2.341 seconds with status “OK (1 test)”." src="https://cdn-images-1.medium.com/max/1024/1*r9X1Y3y05dIj98EfVoxtGw.png" /></figure><h3>Execute the Same Tests on a Second Device</h3><p>You have already run these commands, so point them at a different serial. I normalise the animation settings first, because two phones with different animation scales are not running the same test at all.</p><pre>adb -s SAMSUNG_SERIAL shell settings put global window_animation_scale 0<br>adb -s SAMSUNG_SERIAL shell settings put global transition_animation_scale 0<br>adb -s SAMSUNG_SERIAL shell settings put global animator_duration_scale 0<br><br>adb -s SAMSUNG_SERIAL install -r -g app-debug.apk<br>adb -s SAMSUNG_SERIAL install -r -t app-debug-androidTest.apk<br><br>adb -s SAMSUNG_SERIAL shell am instrument -w \<br>  -e class com.example.tests.LoginTest#validLogin \<br>  com.example.app.test/androidx.test.runner.AndroidJUnitRunner</pre><p>Android’s Espresso documentation names those same three settings under Developer options, so I apply them to the Pixel too. The APKs and the method now match, and only the hardware changed.</p><h3>Run Espresso Tests Online When Local Devices Are Unavailable</h3><p>Two phones on a desk works fine. Once I need six models across three Android versions on every pull request, hosted devices start paying for themselves.</p><h3>Tools You Can Use to Run Espresso Tests Online</h3><p>Four options cover most teams, and the right one usually follows wherever your CI runs. Here is what each does well and where I hit its limits.</p><p><strong>1. Build-managed Gradle devices</strong></p><ul><li><strong>Best for:</strong> Repeatable virtual devices you define in your build script and launch with a task such as ./gradlew pixel2api30DebugAndroidTest.</li><li><strong>Limitation:</strong> It manages emulators only, and ATD images run headless, so screenshot tests that need hardware rendering fail.</li></ul><p><strong>2. Firebase Test Lab</strong></p><ul><li><strong>Best for:</strong> Teams already on Google infrastructure, calling gcloud firebase test android run with — device model=Pixel7,version=33 and — test-targets.</li><li><strong>Limitation:</strong> The documentation tells you to run Orchestrator locally first because a misconfiguration eats quota, and the Gradle plugin still sits at 0.0.1-alpha05.</li></ul><p><strong>3. AWS Device Farm</strong></p><ul><li><strong>Best for:</strong> Teams whose CI already lives in AWS, where Dex Test Parser enumerates your tests so you can preview what the service will find.</li><li><strong>Limitation:</strong> That parser and your device pools drive test selection rather than the -e filters you use locally, so the two invocations stop resembling each other.</li></ul><p><strong>4. BrowserStack App Automate</strong></p><ul><li><strong>Best for:</strong> Moving a local reproduction onto hosted physical hardware unchanged, because the build API takes the same class, package, annotation and size filters you used over adb.</li><li><strong>Limitation:</strong> Sharding eats parallel slots multiplicatively, so four shards across three devices block twelve parallels, and the platform queues anything beyond your plan.</li></ul><figure><img alt="A comparison infographic illustrating four online Espresso testing options: Build-managed Gradle devices, Firebase Test Lab, AWS Device Farm, and BrowserStack App Automate, highlighting best use cases and limitations for each." src="https://cdn-images-1.medium.com/max/1024/1*i1BHzyKf2-IJ3mtSFQWQkQ.png" /></figure><p>I use App Automate for the rest of this because that filter parity keeps the comparison honest.</p><h3>Run the Espresso Test with BrowserStack App Automate</h3><p>Three curl calls cover the flow, and the payload uses the same pair of APKs from the start.</p><pre>curl -u &quot;USERNAME:ACCESS_KEY&quot; \<br>  -X POST &quot;https://api-cloud.browserstack.com/app-automate/espresso/v2/app&quot; \<br>  -F &quot;file=@app-debug.apk&quot;<br><br>curl -u &quot;USERNAME:ACCESS_KEY&quot; \<br>  -X POST &quot;https://api-cloud.browserstack.com/app-automate/espresso/v2/test-suite&quot; \<br>  -F &quot;file=@app-debug-androidTest.apk&quot;</pre><p>Both return a bs:// reference, which I pass into the build call with the same method I ran locally.</p><pre>curl -u &quot;USERNAME:ACCESS_KEY&quot; \<br>  -X POST &quot;https://api-cloud.browserstack.com/app-automate/espresso/v2/build&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;app&quot;: &quot;bs://APP_HASH&quot;,<br>    &quot;testSuite&quot;: &quot;bs://TEST_HASH&quot;,<br>    &quot;devices&quot;: [&quot;Google Pixel 7-13.0&quot;, &quot;Samsung Galaxy S23-13.0&quot;],<br>    &quot;class&quot;: [&quot;com.example.tests.LoginTest&quot;],<br>    &quot;deviceLogs&quot;: true<br>  }&#39;</pre><ul><li><strong>Device strings use a name and OS version format:</strong> A regular expression such as “Samsung Galaxy .*” grabs the first available match, which shortens queue time and costs reproducibility, so I pin the exact model when I chase a failure.</li><li><strong>Enable the logs you need first:</strong> Instrumentation logs and video run by default, while deviceLogs and networkLogs stay off until you ask for them.</li></ul><figure><img alt="BrowserStack App Automate dashboard showing Espresso regression build results with one passed session on Google Pixel 7 and one failed session on Samsung Galaxy S23." src="https://cdn-images-1.medium.com/max/1024/1*MISowzd1mDUrHmEf1zcZsw.png" /></figure><h3>Compare Local and Cloud Espresso Results</h3><p>A comparison only tells me something when both sides ran the same tests and produced comparable artifacts. Gradle writes local results to two fixed paths, and CI consumes the XML.</p><pre>app/build/reports/androidTests/connected/<br>app/build/outputs/androidTest-results/connected/</pre><p>For adb runs I add -r and take the raw instrumentation stream, which carries the class, method, status and stack trace in a format that parses the same way wherever it came from.</p><pre>adb shell am instrument -w -r \<br>  -e class com.example.tests.LoginTest#validLogin \<br>  com.example.app.test/androidx.test.runner.AndroidJUnitRunner</pre><p>Diffing the INSTRUMENTATION_STATUS lines lets one script check a local run against a cloud session, and teams skip this step more than any other. Keep the selection identical on both sides, because if one environment ran the full suite and the other ran a single method, your own setup created the difference you found.</p><h3>Troubleshoot Tests That Fail on Only One Device</h3><p>When a test fails on exactly one phone, I chase the mechanism rather than the manufacturer. Four causes cover nearly everything I have seen.</p><ul><li><strong>Unregistered asynchronous work fails on the slower phone first:</strong> Espresso synchronises only with operations that post to the MessageQueue, so background work needs an IdlingResource that you register through IdlingRegistry.getInstance().register() in @Before. The phone that failed is usually the one where the call took longer.</li><li><strong>Leaked application state shows up as an order-dependent failure:</strong> My development phone already holds a session and cached data that a fresh cloud device has never seen, so one run with clearPackageData settles it.</li><li><strong>The display environment changes what Espresso can see:</strong> Screen size, density, font scale and API level all feed the visibility check, so a view that sits fully visible on one handset may need a scrollTo() on the other.</li><li><strong>Permission dialogs and autofill overlays intercept clicks:</strong> Installing with -g clears them when they have nothing to do with the scenario.</li></ul><p>One gap caught me out. The Espresso Device API gives you onDevice().setScreenOrientation() and @RequiresDeviceMode, but Android’s documentation says it works only on virtual devices with emulator control enabled. On a physical Pixel or Samsung, including one hosted in a device cloud, you fall back to ActivityScenario.recreate() or UiAutomator.</p><h3>Espresso Cross-Device Execution Checklist</h3><p>I work through this before I treat two runs as comparable. Every item names a command to run or a value to write down.</p><ol><li>Build both APKs once and reuse them everywhere.</li><li>Record the Espresso and runner versions Gradle resolved.</li><li>Confirm both devices with adb devices -l.</li><li>Record the API level, wm size, wm density, font_scale and locale.</li><li>Zero all three animation scales on every device you compare.</li><li>Install with -r -g for the app and -r -t for the test APK.</li><li>Validate the filter with -e log true, then run one method on each phone.</li><li>Upload the same APKs and pin the same filter in the cloud payload.</li><li>Pin an exact device and OS version when you reproduce a failure.</li><li>Diff the local XML against the cloud session, then check synchronisation and application state before you blame the hardware.</li></ol><h3>Final Thoughts</h3><p>Adding devices multiplies whatever your suite already is. A test with an unregistered IdlingResource across ten hosted handsets gives you ten copies of the same flake and a dashboard that looks like real coverage. One Orchestrator flag fixes that, and nine more phones do not.</p><p>Starting narrow has held up for me. I run one test on one known device, write down every environment value, then add a second device with the same build and filter. The cloud matrix comes last, because by then a difference in results means something.</p><p>So here is my question for your suite. When it fails on the second device, do you know whether the device differs, or whether the first device covered for an unsynchronised call?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ed5669b0bde" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>