An ESP32 and a Local Vision Model Drive My iPhone

iPhone Automation Without Xcode, a Jailbreak, or the Cloud

Photo by Dennis Brendel on Unsplash

Apple spent 17 years convincing us that controlling an iPhone requires Xcode, a $99 developer account, and a Mac that costs more than your couch.

Henry Yuan spent $7 to prove them wrong.

His repo — esp32-iphone-agent — showed up on my timeline on October 11th as a 12-star MIT-licensed shitpost: “A $7 ESP32 lets an AI agent control a stock iPhone.” Hackaday picked it up on October 14th. By the time I cloned it, it had almost 3,000 stars. You are still early.

The trick isn’t AI. It’s hardware heresy.

Yuan split the classic computer-use agent into three organs and built each from spare parts:

• Eyes: AirPlay mirroring

• Hands: An ESP32-S3 pretending to be a USB mouse and keyboard

• Brain: Any OpenAI-compatible vision endpoint

No WebDriverAgent. No CoreDevice hack. No app on the phone that Apple has to sign. Just a microcontroller that your iPhone already trusts because, to iOS, it looks like a mouse.

I rebuilt it the way a solo maker would actually run it: not on two DGX Sparks running Qwen3 Flash like Henry did for the demo, but on one consumer GPU with Ollama and a 7B vision model. And because the README hand-waves past the scariest part, I’m going to talk about the security model. What we just built is literally a Wi-Fi-controlled USB HID implant.

This is the teardown.

1. The Eyes: UxPlay plus a GStreamer JPEG pipeline

The original v0.1.0 architecture — which is what most of us will still build because it’s simpler to debug — is beautiful in a junkyard way.

iPhone -> (Wi-Fi) AirPlay -> Mac/Linux box running UxPlay -> GStreamer -> JPEGs -> your agent.

On my Linux box:

brew install uxplay
uxplay -n ESP32-Agent -p 7000 -fps 15 -vr 480p

UxPlay is the unsung hero here. It’s a real AirPlay receiver built on GStreamer plugins, emulating an Apple TV. Your iPhone thinks it’s just projecting to a TV. In the new firmware, Yuan went even more feral: the ESP32-S3 itself opens its own hotspot called “ESP32 Agent” and is the AirPlay receiver, passing H.264 through to an iPhone app over BLE. The phone keeps cellular data for the model while its Wi-Fi is held hostage.

Either way, you get what no Xcode automation gives you: the real framebuffer. Not a screenshot API, the actual rendered screen at about 5 fps, 480x1040. Low enough to keep BLE happy, high enough for a VLM to read.

My GStreamer pipeline for the legacy rig:

appsrc -> videoconvert -> jpegenc quality=85 -> appsink -> Python

Feed that into /v1/chat/completions as base64 images. That’s your eyes.

The new version is cleaner: iPhone -> Screen Mirroring (Wi-Fi) -> ESP32-S3 hotspot (AirPlay receiver, H.264 passthrough) -> BLE -> ESP32 Agent app (same iPhone) -> vision model. The phone powers the board.

2. The Hands: ESP32-S3 as USB HID, and why AssistiveTouch is the trick

Here’s the bit that made me grin out loud.

Apple will never let a random USB device tap the screen. But it will let an accessibility mouse move a cursor. Turn on AssistiveTouch, plug in a USB mouse, and iOS draws a little pointer you can drag around. Tap equals click. Swipe equals drag.

Yuan’s firmware does exactly that: the board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an on-screen pointer.

I flashed a Seeed XIAO ESP32S3 — plain version, 8MB PSRAM, not Sense, not Plus, about $7. Any ESP32-S3 with native USB and PSRAM works. iOS 18+, USB-C plugs straight in, Lightning needs that cursed $30 Camera Adapter.

Toolchain:

arduino-cli lib install "NimBLE-Arduino@2.5.1"
hardware/flash.sh

The board exposes two HID interfaces via TinyUSB: relative mouse + ASCII keyboard. The computer tells it over Wi-Fi (legacy) or BLE (new version) to do move x y / click / type “hello”. It does.

The genius: pointer drift. Because the mouse is relative, every move accumulates error. Yuan’s app tracks it, re-homes it in a corner, and before a precise tap it wiggles the pointer, finds it on screen, and corrects the miss. That wiggle is why my automation stopped missing the tiny x in Safari.

Closing the loop is everything: after each action it waits for the screen to settle and tells the model whether anything changed. A lost board, hotspot, or mirroring link pauses the run and resumes from the same step.

3. Swapping the DGX Sparks for Ollama on one consumer GPU

The README is honest: the project was developed with nvidia/Qwen3.8-Flash served on two DGX Sparks. Cool if you have $20k lying around.

My maker stack:

•Host: M2 MacBook Air + RTX 4060 eGPU, or just a 4070 laptop

• Server: ollama serve or llama.cpp server with vision

• Model: qwen2.5vl:7b or minicpm-v 2.6. Qwen is more spatially accurate, MiniCPM is faster.

• API: Any OpenAI-compatible chat API that accepts images

ollama pull qwen2.5vl:7b
ollama serve
# endpoint: http://localhost:11434/v1/chat/completions

In the iPhone app you can set a separate planner and actor model — and you should, on one GPU. I run Qwen2.5VL 7B as actor (fast taps) and a larger 32B quantized model as planner when I’m on Wi-Fi, or the same 7B with a longer system prompt locally.

Per-action latency on local 7B, measured from JPEG in to HID command out:

•Eyes (AirPlay to JPEG): 180–250ms

• Vision inference (Ollama 7B Q4_K_M on RTX 4060): 1.8–2.4s

• Hands (Wi-Fi to USB HID to pointer settle): 350–600ms

Get Aeon Flex, Elriel Assoc. 2133 [NEON MAXIMA]’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

• Screen settle wait: 800ms

Total: about 3.5 seconds per step. Cloud GPT-4o does it in 1.2s but leaks your banking screen to OpenAI. Local is slower, but it’s yours.

BLE carries about 300–500 kbps, so the screen is mirrored small and low fps; each step takes a few seconds, most of it the model. That’s not a bug, it’s physics.

4. Planner vs Actor: One model or two

Yuan ships with two prompts in prompts/ — plain text, bundled with the app. Edit them, rebuild, done.

This matters more than you think.

One-model setup: One VLM gets “Goal: set alarm 8am Mon/Tue. Current screen:. What action?” It outputs {“action”:”tap”, “x”:512, “y”:1340}. Simple, but it forgets why it tapped after 6 steps and loops.

Two-model setup: A planner (bigger, slower) says “Step 1: open Clock app. Step 2: tap Alarm. Step 3:…” The actor (smaller, faster) only executes step 2 given the current screen. This is how Yuan’s app runs: decode the stream, send screenshots to any vision model, and run a planner and an actor — tap, swipe, draw, type, press keys, go home, finish.

On one GPU, run the planner every 3 steps, not every step. Cache the plan. The actor fails less when it has a short horizon.

My prompt diff that fixed stuck loops:

“If the screen didn’t change after your last action, you MISSED. Wiggle pointer, increase tap radius by 20px, or swipe instead. Never repeat identical coordinates.”

Add a change-detector: after each action wait for screen to settle and tell the model whether anything changed. If image diff is less than 2 percent, force a re-plan.

5. Latency and Failure Loops: Calibration, Re-plans, Stuck States

Real talk: this rig fails hilariously.

Failure modes I logged over 50 tasks:

1. Calibration drift: After 10 taps, pointer is 30px off. Fix: re-home in corner every 5 actions and wiggle before precise taps. Don’t trust absolute coordinates — the device screen width and height are stored with a calibration that is relative.

2. iOS black holes: iOS does not mirror some screens (passcode entry, Apple Pay, DRM video). Mirroring stops when the phone locks. Your agent goes blind. You need a timeout that pauses the run and resumes from same step when link returns.

3. Keyboard hell: Keyboard input is ASCII only. Want to type Chinese through pinyin? Not yet. It types via USB keyboard events, so you get what a US keyboard sends.

4. Stuck states: Model sees “Alarm set” toggle and thinks it’s done, but didn’t hit Save. Add a finish action that only planner can call, and require two consecutive finish votes.

My fix loop:

while not done:
  frame = get_jpeg()
  if no_change: calibration_wiggle()
  plan = planner(goal, history) if step % 3 == 0 else cached_plan
  action = actor(plan, frame)
  execute_hid(action)
  sleep(0.8)

Every run is kept on the phone: replay step by step with taps drawn on top, export trace for analysis. Use it. Export runs via Runs -> open a run -> Export trace.

6. The Threat Model: What this proves about Wi-Fi-controlled HID implants

The README says “Only automate devices and accounts you own.” It should say in bigger font: this is a BadUSB with a radio.

You plugged a $7 board into your iPhone that:

• Is powered by the phone

• Emulates mouse plus keyboard (so it can type anywhere, even in password fields)

• Runs a Wi-Fi hotspot that your phone auto-joins, with its own AirPlay receiver stack

• Accepts BLE commands to drive that HID, with firmware you flashed via flash.sh

In the legacy version, commands came over Wi-Fi from a Python server. In the new version, it’s BLE from an app, but the board still controls HID. If that BLE characteristic isn’t authenticated with a rolling secret — and NimBLE-Arduino defaults aren’t — anyone within 10 meters can tell your phone to start typing.

This links directly to hardware implant work. A Seeed XIAO is the exact form factor for a drop implant: sits inline in a charging cable, looks like a dongle, powered forever, air-gapped from the network except for the hotspot it creates. Apple trusts it because it’s HID. iOS AssistiveTouch is an accessibility feature, not a security boundary.

Defenses:

• Flash with secrets.h changed, enable NimBLE secure connections

• Never leave board plugged in unattended

• iOS: Settings -> Accessibility -> Touch -> AssistiveTouch -> Devices -> forget when done

• If you’re building this for red-team: log that AirPlay mirroring doesn’t capture passcode / Apple Pay — you won’t get Secure Enclave screens, but you will get everything else

Yuan did responsible disclosure: repo has SECURITY.md and responsible use note. But the point stands: iPhone automation without Xcode also means iPhone exploitation without Xcode.

BOM for the solo build (what I actually used):

• Seeed XIAO ESP32S3 (∼$7 per the readme)

• Lightning-to-USB Camera Adapter if you’re on older iPhone (∼$30)

• Mac or Linux host, arduino-cli plus ESP32 core 3.x, UxPlay, python venv

• Ollama or llama.cpp with a vision model — I benchmarked qwen2.5vl:7b

Total cost if you own a GPU: $7. Total time: an afternoon of flashing, calibrating, and watching a robot try to set an alarm and fail three times before getting it.

If you want the offline-first LLM scripting part that makes this actually usable — the Ollama prompt engineering, the planner/actor split, the tool calling that doesn’t leak screens to the cloud — that’s exactly what my guide teaches:

Aider + OpenClaw: Offline-First AI Scripting:

How to Use Aider + OpenClaw: The Offline-First AI Scripting Beast

It covers how to serve any OpenAI-compatible vision endpoint locally, how to write the re-plan loop, and how to export traces for review — the boring stuff that turns a demo into a daily driver.

Apple built a walled garden. We just found the mouse hole.