onehumanbeing/esp32-iphone-agent

选用指南

AI 根据 README 整理 · 2026/10/19 17:46:55

适用场景

适合希望用AI模型通过自然语言指令自动化操作iPhone的场景，如设置闹钟、打开应用等，无需电脑介入。

查看 README 依据

<div align="center"> <img src="docs/images/logo.png" alt="ESP32 Agent logo" width="160"> # ESP32 iPhone Agent **Let an AI model use a real iPhone with a $7 ESP32. Just the phone and the board: no computer, no jailbreak.** English · [简体中文](README.zh-CN.md) Demo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTube demo](https://www.youtube.com/watch?v=Ly4im-P4jYg) [](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml)

opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.) <p align="center"><img src="docs/images/app-screens.png" alt="ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration" width="70%"></p> *Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.* ## How it works ``` iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──BLE──▶ ESP32 Agent app (same iPhone)

目标用户

面向具备一定硬件和开发经验的用户，需要自行购买ESP32-S3开发板并完成固件烧录和iOS应用构建。

查看 README 依据

by another agent. See [Export runs for agent review](docs/run-data-export.md) for batch export, USB copying and file details. ## Hardware | Part | Notes | |---|---| | **ESP32-S3 board** | Tested with a [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (plain version, 8 MB PSRAM; not Sense, not Plus). About $7. Any ESP32-S3 with native USB on its USB-C port and PSRAM should work. |

| **iPhone** | iOS 18 or later, with cellular data (the Wi-Fi is taken by the board's hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. | | **Vision model** | Any OpenAI-compatible chat API that accepts images: a cloud API or your own server. | | **Mac with Xcode** | Only to flash the board and build the app once. Not needed at run time. | ## Quick start ### 1. Flash the board (plugged into the Mac) Install [arduino-cli](https://arduino.github.io/arduino-cli/), the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:

使用方式

使用方式为：将ESP32-S3板通过USB-C连接iPhone，在配套应用中完成蓝牙配对、屏幕镜像、指针校准和模型配置后，输入任务指令即可运行。

查看 README 依据

(`cd ios && xcodegen`). A free Apple ID can sign it (re-sign every 7 days). The one-tap hotspot join needs the Hotspot Configuration capability (a paid account); without it the app shows the hotspot password to join by hand. ### 3. Set up (once, in the app) Plug the board into the iPhone and open **ESP32 Agent**. Setup walks through four steps and checks each one itself: 1. **Plug Me In** — pair the board over Bluetooth. 2. **Let Me See the Screen** — join the board's Wi-Fi, then Control Center → Screen Mirroring → *ESP32 Agent*.

3. **Let Me Use the Pointer** — turn on AssistiveTouch; the app calibrates the pointer by itself. 4. **Give Me a Brain** — enter your model endpoint and key (stored in the Keychain), then choose a vision model from the endpoint's model list (or enter its model ID). ### 4. Give it a task Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press **Start**. Stop it from the Dynamic Island, or come back to the app. Siri, Shortcuts, widgets and Control Center can run saved tasks. ## Model and prompts

部署要求

需要ESP32-S3开发板（带原生USB和PSRAM）、iOS 18或更高版本的iPhone（需蜂窝数据）、任意兼容OpenAI的视觉模型API，以及一台用于初始烧录和构建的Mac（带Xcode）。

查看 README 依据

by another agent. See [Export runs for agent review](docs/run-data-export.md) for batch export, USB copying and file details. ## Hardware | Part | Notes | |---|---| | **ESP32-S3 board** | Tested with a [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (plain version, 8 MB PSRAM; not Sense, not Plus). About $7. Any ESP32-S3 with native USB on its USB-C port and PSRAM should work. |

| **iPhone** | iOS 18 or later, with cellular data (the Wi-Fi is taken by the board's hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. | | **Vision model** | Any OpenAI-compatible chat API that accepts images: a cloud API or your own server. | | **Mac with Xcode** | Only to flash the board and build the app once. Not needed at run time. | ## Quick start ### 1. Flash the board (plugged into the Mac) Install [arduino-cli](https://arduino.github.io/arduino-cli/), the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:

已知限制

作者明确说明：BLE带宽限制导致屏幕镜像分辨率低（480×1040，≤5fps）；iOS不镜像密码输入、Apple Pay等屏幕；运行时Wi-Fi被热点占用，模型需通过蜂窝数据访问；键盘输入仅支持ASCII，暂不支持横屏应用。

查看 README 依据

- [ ] A curated list of models that work best with this setup, shown inside the iOS app ([suggested by @asharthinks](https://x.com/asharthinks/status/2110740775030125019)) ## Limitations - BLE carries about 300-500 kbps, so the screen is mirrored small (480×1040, ≤5 fps); each step takes a few seconds, most of it the model. - iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them. Mirroring stops when the phone locks. - While the agent runs, the phone's Wi-Fi is the board's hotspot (no internet over it); the model is reached over

cellular data. - The app leaves itself with a private iOS API (`suspend`); fine for a self-built app, to be replaced before the App Store. - Keyboard input is ASCII only. Landscape apps are not supported yet. ## Responsible use Only automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved. The agent acts with your hands: watch it, and keep the stop button within reach. ## Contributing Issues and pull requests are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md). ## Related projects