I Gave an AI Agent an Android Emulator: Can It Actually Test a React Native App?

An experiment with React Native, Maestro, Android emulators, and agent-driven exploratory testing

AI coding agent can understand a React Native component without ever seeing the application running. A tester cannot.

It needs to launch the app, understand what is on screen, interact with it, wait for state transitions, recover from unexpected states, and distinguish between an application bug, a flaky test, and a broken execution environment.

So I wanted to test a narrower question:

What happens if I give an AI agent an Android emulator and ask it to behave like a mobile QA engineer?

I wanted the agent to explore the running application first, decide what is worth testing, turn those observations into automation, and later detect regressions without being told what changed.

To make the experiment controlled, I built a small React Native application called AgentBank. Not generate tests from source code.

Not repair an existing Maestro flow.

Actually explore the running application first, decide what is worth testing, turn those observations into automation, and later detect regressions without being told what changed.

To make the experiment controlled, I built a small React Native application called AgentBank and deliberately kept the application simple.

The Test Application:

AgentBank has a login screen and a small authenticated dashboard. The starting human-written Maestro suite contained only three flows:

01-required-fields
02-invalid-credentials
03-login-dashboard-logout

Before involving an agent, all three passed.

Experiment A — Explore an App You Know Nothing About

For the first experiment, the agent could interact with the emulator but could not inspect: application source code, my existing Maestro tests, or valid credentials.

Its job was simply to explore.

The agent found the emulator, identified AgentBank, launched it, and began inspecting the screen through screenshots and the Android UI hierarchy.

It discovered that:

• submitting both fields empty produced required-field errors;
• whitespace-only email was treated as empty;
• whitespace-only password behaved differently;
• malformed email plus a password proceeded to authentication and eventually produced the generic invalid-credentials error;
• entered values survived a Home/resume cycle;
• Back/reopen affected state differently.

It also noticed the React Native development warning UI without automatically treating it as an application defect. Instead of inventing credentials, it asked for them. That was the first interesting result.

The agent wasn’t simply executing instructions. It was building a model of the UI, probing that model, and recognizing when it lacked enough information to continue.

Experiment B — Give It Credentials, but Still No Tests

The agent logged in and independently discovered the authenticated dashboard:

It tapped the account area, balance, greeting, and other UI regions. It tried scrolling. It searched for additional navigation. More interestingly, it started testing lifecycle behaviour without being instructed how the app should behave. It discovered

Dashboard
│
├── Home → Resume ──────► Dashboard
│
└── Back → Reopen ─────► Login

It tested logout and verified that reopening afterward did not restore the authenticated dashboard, tried rapid login taps, tried repeated logout actions.

Experiment C — Now Turn Those Observations Into Tests

Now I ask the agent to create Maestro automation with one important restriction:

It could not inspect my existing human-written Maestro tests.

It created its own isolated suite and chose seven scenarios:

Required-field validation
Invalid-login recovery
Login/logout boundary
Home/resume
Back/reopen
Whitespace handling
Repeated login/logout cycles

Then I ran it. And the tests failed. But not because of AgentBank.

Restarting the development application exposed an environment dependency: Metro wasn’t running, the app displayed:

Unable to load script

The agent classified the failures as environmental rather than changing its selectors or assertions.

I restored Metro and ran the suite again.

This time:

6 of 7 flows passed without modification.

The remaining test was Back/reopen.

This was the moment I expected the usual automation response:

add a wait and make the test green.

The agent didn’t do that. It reproduced the sequence manually. Manual reopening reached login. It reran the unchanged Maestro test.It passed. It tried another direct Android launch variation. That passed too.

Experiment D — Human Tests vs Agent Tests

Now I removed the final restriction. The agent could inspect my original Maestro suite. I asked it to compare the two.

Get Sirsha Banerjee’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It covered things the agent had missed, including:

• valid email + wrong password;
• loading becoming visible;
• loading disappearing;
• explicit post-logout error state;
• a compact login → dashboard → logout contract.

The agent suite found different gaps:

• failed-login → successful-login recovery;
• stale validation error clearing;
• authenticated Home/resume;
• post-logout restoration;
• whitespace behaviour;
• lifecycle behaviour.

There was another important difference. The agent-generated suite did substantially more work. It performed seven successful login sequences and fourteen app restarts. Its repeated-session test alone took roughly 100 seconds. So more automation was not automatically better automation.

The agent didn’t replace the human test suite.

It challenged it.

Experiment E — Can It Find Regressions Without Knowing What Changed?

I used one isolated agent session to introduce three controlled regressions.

Then I started a completely fresh testing session.

The testing agent was forbidden from inspecting: source code, Git changes, modification timestamps or the private record describing the injected bugs. It could use the emulator, ADB, UI hierarchy, screenshots, existing tests, and historical observations.

Determine whether AgentBank still behaves like the previously observed baseline.

The three injected changes were:

1. Logout no longer cleared authenticated state

2. Dashboard balance changed:
   2480.55 EUR → 2480.6 EUR

3. Login fields remained editable while
   "Signing in..." was visible

Regression 1 — Balance Formatting

The existing happy-path test failed. It didn’t stop at the failed assertion, it logged in manually multiple times and reproduced the same value.

Regression 2 — Logout

The agent tapped Log out. Nothing happened. Rather than immediately reporting a bug, it repeated the interaction at different positions and waited.

The dashboard remained visible. Then it backgrounded and resumed the application. The authenticated dashboard was still there.

Regression 3 — Editing During Authentication

While Signing in... was visible, the agent discovered that the email field could still be edited. That was indeed the third regression I had injected.

But the agent did not classify it as a confirmed regression. Why?

Because the historical evidence available to it did not explicitly prove that the input had previously been locked during authentication.

Its conclusion was effectively:

I can prove this behaviour exists now.
I cannot prove from my baseline that it changed.

The Result:

Where This Worked

The agent was particularly useful for:

Exploratory breadth.
It tried lifecycle transitions, repeated interactions, whitespace, recovery paths, and state changes that weren’t in my original regression suite.

Turning observations into candidate automation.
Six of seven generated flows worked unchanged once the environment was healthy.

Failure investigation.
It frequently distinguished between product behaviour, test behaviour, and infrastructure problems before modifying anything.

Regression comparison.
Historical screenshots, assertions, execution records, and known UI behaviour gave it something concrete to compare against.

Where It Didn’t

There were equally clear limitations.

Tool access matters

Several attempts failed before the first tap because the agent sandbox couldn’t access ADB. A model can reason about Android testing perfectly well and still test absolutely nothing if its execution environment cannot reach the emulator.

More generated tests did not mean a better suite

The generated suite contained useful new coverage, but also repeated authentication and cleanup extensively. It was a good exploration artifact.It was not yet the regression architecture I would maintain.

Baseline quality became the ceiling

The third injected regression exposed the most important limitation.

The agent could observe that the field was editable.

But without historical evidence establishing that it used to be locked, it couldn’t confidently prove regression.

So, Would I Let an AI Agent Test a React Native App?

Yes.

But I would not hand it the regression suite and walk away.

The architecture I would use today looks more like:

Humans still decide which behaviours are important enough to become contracts.

Try the experiment yourself

The complete AgentBank demo application and Maestro test suites are available on GitLab:

AgentBank — AI Agent Testing Experiment