ESP32 iPhone Agent

Example 1 — draw on a canvas. "Open Whiteboard and draw a smiley face": the agent searches for the app, opens it and draws the face with continuous strokes. Every step's screenshot, with its tap or stroke drawn on top, stays on the step wall.

Example 2 — set an alarm. "Set an alarm for 7:00 AM that repeats every day, then save it": the agent opens Clock, types 0700 into the time wheel, ticks the days under Repeat several taps at a time, and saves. Left: the real iPhone with the ESP32 plugged in; right: the web UI.

The agent sees the phone through AirPlay screen mirroring and touches it through a tiny ESP32-S3 board that plugs into the phone's USB port as a mouse and keyboard. Any OpenAI-compatible vision model can be the brain — a cloud API, or a model you run yourself.

Why

Most iPhone automation drives the phone through XCTest / WebDriverAgent. That needs a Mac, Xcode, a signed helper app and an Apple developer account (a free account's signature expires every 7 days). This project uses the same paths a person uses — the screen and a pointer — so the phone needs nothing installed.

How it works

• Eyes — AirPlay mirroring. The phone mirrors its screen to UxPlay running on your computer; a GStreamer pipeline turns the stream into JPEG frames for the agent and the web page.
• Hands — a USB HID bridge. An ESP32-S3 plugged into the phone shows up as a standard USB mouse and keyboard. With iOS AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe. The board connects to the computer over Wi-Fi and executes simple line commands (move, click, type, …); all logic stays on the computer.
• Brain — a planner and an actor. A planner model writes a short plan from the goal and the current screen. An actor model then looks at each new screenshot and picks the next action: tap, swipe, draw a path, type, press a key, go home, or finish. Both roles can be the same model.
• Closing the loop. The pointer is relative, so the server tracks where it is. Before each tap it wiggles the pointer, finds it on the mirrored screen and corrects any miss, refining the calibration as it goes. After each action it checks whether the screen actually changed and tells the model, so a missed tap is noticed and retried instead of built upon. Repeated actions trigger a re-plan; a dropped link or a bad reply is recovered from.

Hardware

Use a USB-C hub with power pass-through if the phone should keep charging while the board is plugged in.

Quick start

Set up with an AI coding agent. Paste this into Claude Code, Codex or a similar agent on the computer the board will connect to:

Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine:
flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start
the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key.

1. Flash the board

Install arduino-cli and the ESP32 core (3.x), then:

cp hardware/hid_bridge/secrets.example.h hardware/hid_bridge/secrets.h
# edit secrets.h: 2.4 GHz Wi-Fi name/password and the IP address of the computer running the server
hardware/flash.sh            # optional arguments: [serial port] [--debug]

2. Install and start the server

# UxPlay — macOS:
brew install uxplay
# UxPlay — Debian / Ubuntu:
sudo apt install uxplay gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-libav

cd server
python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env         # set LLM_BASE_URL / LLM_MODEL / LLM_API_KEY
set -a && . ./.env && set +a && .venv/bin/python -m hidagent

Open http://<computer-ip>:8890 in a browser.

3. Set up the phone (once)

1. Settings → Accessibility → Touch → AssistiveTouch: turn it on.
2. Plug the board into the phone. A round pointer appears; the web page shows controller connected.
3. Control Center → Screen Mirroring → iPhone Agent. The live screen appears in the web page.
4. In the web page, open Manual controls → Calibrate. The server moves the pointer, finds it on the live screen and calibrates itself in a few seconds; if it cannot see the pointer, it asks you to click exactly on it.

4. Give it a task

Type a goal (for example "Open Settings and go to Wi-Fi") and press Start, or click one of the examples.

Web UI

• Live phone — click to tap, drag to swipe, scroll to scroll; manual Home, Back, typing and keys.
• Live log — every plan, thought and action as it happens, with timings.
• Step wall — the screenshot the model decided on for every step, with its tap or swipe drawn on top, plus the model's reasoning and how long thinking and acting took. Click a card to zoom in.
• Taps are marked with a red ring and a spreading ripple, swipes and strokes with a yellow path; the Motion switch turns the animation off. Taps the model marked precise (pointer checked before tapping) are labelled in the log.
• Emergency stop (or Esc) stops the agent and releases the pointer. Clear history empties the log.
• English and Simplified Chinese interface.

Configuration

Set in server/.env (see .env.example):

Local model (optional)

A cloud API is enough. If you want everything on your own hardware, this is the setup the project was developed and timed with:

• Model: nvidia/Qwen3.8-Flash-Next-NVFP4
• Serving: Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold — two NVIDIA DGX Sparks with tensor parallelism (TP=2), OpenAI-compatible API
• Speed: about 1.3 s per action and 2 s per plan, with thinking mode off:

LLM_BASE_URL=http://<dgx-spark-ip>:8888/v1
LLM_MODEL=Qwen3.8-Flash-Next
LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'

Any other OpenAI-compatible server (vLLM, SGLang, llama.cpp, Ollama, …) with a vision model works the same way.

Debug mode

Set HIDAGENT_DEBUG=1 to log every board command with its latency and to save one folder per agent run:

~/.hidagent/traces/20261012-005207/
├── steps.jsonl   # every model call (prompt, raw reply, latency) and every action
├── 002.jpg       # the screenshot each model call saw
└── ...

hardware/flash.sh --debug builds firmware that echoes each command and its duration over USB serial.

Board protocol

The board holds no logic. On connecting it sends hello <last reset reason> <uptime ms>; then the server sends one command per line over TCP and the board answers ok or err <reason>.

If the Wi-Fi link drops, the board releases all buttons and reconnects on its own. While the phone is locked, its USB port sleeps: the board then tries to wake it and otherwise answers err phone usb asleep instead of waiting. A watchdog restarts the board if it ever stops responding.

Project layout

hardware/   ESP32-S3 firmware (Arduino) and flash script
server/     agent, web UI, screen capture and board link (Python, FastAPI)
            serve/: optional minimal server to host a Hugging Face vision model yourself

Roadmap

• Test a fully local setup with DiffusionGemma (server/serve/ has an experimental OpenAI-compatible server)
• Landscape apps (screen coordinates and calibration assume portrait)
• Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)
• More example tasks across popular apps

Limitations

• iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them.
• Mirroring stops when the phone locks; keep auto-lock off while the agent works.
• Keyboard input is ASCII only.
• Re-calibrate after changing AssistiveTouch tracking speed.
• Landscape apps are not supported yet: drawings and taps there land in the wrong place.
• On Linux machines with several network interfaces, limit Avahi (mDNS) to the interface on the phone's network if the phone does not list the receiver.

Responsible use

Only automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved. The agent acts with your hands: watch it, and keep the emergency stop within reach.

Contributing

Issues and pull requests are welcome. Run the checks before sending a change:

cd server && pip install -e ".[dev]" && ruff check . && pytest

Related projects

Other ways people drive an iPhone with code or AI, and how this project differs. Corrections welcome.

Other routes: WebDriverAgent/XCUITest-based tools such as mobile-mcp and Midscene need a Mac and code signing; tools built on macOS iPhone Mirroring need a Mac.

Acknowledgements

UxPlay (AirPlay receiver) · Arduino ESP32 core and TinyUSB (USB HID) · FastAPI

License

MIT