When Are Two Mobile AI Runs Comparable?
A meaningful comparison starts with a shared question, a defined baseline, and a clear record of what changed.
Two Mobile AI runs can both be marked “successful” and still answer different questions. Their app states, devices, network conditions, agent versions, or completion checks may differ. The point is not to make every run identical. It is to make the relationship between runs explicit enough to interpret.
Begin with the decision
Are you checking repeatability under a defined baseline, testing a specific condition, or deciding whether a new agent version improves a task set? Each question calls for a different design. Repeatability holds relevant conditions steady; robustness deliberately changes a condition; release comparison changes the agent version while preserving the conditions needed for a fair test.
ARMARRAY has previously discussed repeatability and physical execution and the role of standardized device cohorts. A comparison contract makes the next step explicit: what does this particular test allow the team to conclude?
Write down the comparison contract
Before execution, record four elements:
- The question and task: task revision, input, starting fixture, and completion criterion.
- The intended baseline: app and relevant software versions, device cohort or configuration, account and data state, and configured operating conditions that matter.
- The planned variation: the factor or factors allowed to differ, and why. If several factors change together, interpret the result as a combined scenario.
- The evidence, outcome, and uncertainty rules: observations that support completion, handling for missing or conflicting evidence, and the link between each attempt and its result.
Benchmark design offers a practical reference point. AndroidWorld gives tasks specific initialization, success-checking, and teardown logic, and reports that task variations can materially affect agent performance. Starting state and outcome checks are part of what makes a comparison interpretable; a result label alone cannot carry that context.
One task, different comparison questions
Consider an agent asked to save a selected item to a reading list and verify that it appears there.
If the question is whether a new agent version behaves differently, keep the task revision, app build, account fixture, device cohort, configured network condition, and completion check stable. The intended change is the agent version. If another condition also changes, record it rather than attributing the outcome to the agent alone.
Get ARMARRAY’s stories in your inbox
Join Medium for free to get updates from this writer.
If the question is whether the workflow tolerates a network interruption, keep the task, agent, app, device cohort, and completion rule stable, then introduce a defined interruption. Record what was configured and what the device actually observed. A run in which the interruption never occurred is not evidence about recovery from it.
If the question is how the workflow behaves across device configurations, choose variants that represent the conditions of interest. Keep the task and outcome rule consistent, report by configuration, and avoid combining distinct groups into one unexplained score.
The task is recognizable across all three cases, yet the claim changes. Sharing the same task does not, by itself, make two runs comparable. The task, controlled conditions, and evidence need to fit the question.
Separate configured conditions from observed conditions
A test plan may specify an app build, device cohort, locale, or network profile. A particular run can still encounter a different state: an update may not have applied, a device may be recovering, or connectivity may change during execution. The plan describes what the team intended. The run record describes what was present and observed.
Keep those descriptions separate. When they differ, preserve the deviation with the attempt. Then decide whether it belongs in the planned comparison, needs separate analysis, or calls for a rerun. An earlier attempt should remain available when it helps explain the result.
Treat incomplete evidence as a result
A missing screenshot, a state check that never ran, or a device that stopped responding before judgment can leave the outcome unknown. That is different from evidence that shows the task failed.
Connect task and attempt identifiers, agent and app versions, device context, configured and observed conditions, action records, resulting observations, timestamps, and error or recovery events to the result. Mark an unresolved check inconclusive or incomplete according to the evaluation rule; do not silently count it as a pass or failure.
Keep the conclusion within the evidence
Two observations do not establish stable behavior across repeated attempts, devices, or operating conditions. A result from one defined cohort supports a claim about that cohort and setup. A deliberate network variation supports a claim about the condition tested. Broader claims need planned repetitions or coverage, and should report attempts, represented conditions, unresolved outcomes, and gaps.
Before running, a team should be able to complete this sentence: “We are comparing ___, holding ___ stable, varying ___, and will use ___ as evidence.” If the blanks cannot be filled in, the comparison design is not ready.
Real Android infrastructure helps when it provides the execution context a question requires and keeps that context connected to observations and outcomes. Teams can retain their existing benchmark and evaluation logic while making the evidence behind a conclusion easier to inspect. A comparison is only as useful as the conditions and evidence behind it.
References
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Read the full article on ARMARRAY.
