agent-desktop: Vercel Labs' Rust CLI for Accessibility-Tree Computer Use

Desktop computer use for any agent: snapshot the UI as JSON, act on a ref, snapshot again. No screenshot guessing.

What it does

agent-desktop is a "computer use CLI for AI agents" — a CLI and C library for macOS, Windows, and Linux that turns OS accessibility APIs into a structured control surface:

1. Snapshot. agent-desktop snapshot --app Finder -i returns JSON with a ref for every interactive element, plus available actions.
2. Act. The agent calls click, type, set-value, and similar commands with a snapshot-scoped ref such as @s8f3k2p9:e8.
3. Verify. The agent snapshots again to confirm the result.

The control loop stays inside your agent. agent-desktop doesn't plan or reason. Design choices from the README:

• Headless by default. Ref actions go through the accessibility API without moving the pointer or stealing focus; pass --headed for physical input.
• Stale refs fail loudly. Refs belong to one snapshot. When the UI changed, you get structured errors (STALE_REF, AMBIGUOUS_TARGET) with a disposition that says whether a retry is safe.
• JSON envelope everywhere. Every command returns ok plus either data or a structured error.
• MCP and skills. agent-desktop mcp exposes commands as MCP tools; version-matched skills install via skills tooling or AGENTS.md.
• Privacy warning. Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content — keep them out of public logs.

Homepage: agent-browser.dev/desktop. Desktop sibling to Vercel Labs' browser-agent tooling.

Who it's for

• Builders of coding agents, MCP tools, and internal automation that need to operate native apps, not just a browser tab.
• Teams that want deterministic handles and retry semantics instead of vision-model click coordinates.
• Developers who can grant OS permissions: Accessibility on macOS, an AT-SPI session on Linux. On Windows, apps at the terminal's integrity level need no extra grant per the README.

Less suited to apps with poor accessibility metadata (some games, canvas-heavy UIs). Structure-first automation only sees what the app exposes.

Stars / language / license

15,159 stars · Rust · Apache-2.0

Data verified: 2026-11-05 09:31 Asia/Shanghai (GitHub API get_repository + README get_file_contents)

Trending note: +2,470 stars today (Trending selection captured 09:26 Asia/Shanghai) — largest daily growth on today's list.

Category tags

Agent-Skill · computer-use · desktop-automation · accessibility · Rust · CLI · MCP

How it differs from similar projects

Compared with Anthropic / OpenAI computer-use agents (CUA). Hosted CUA models generally work from screenshots: the model looks at pixels and emits coordinates. That approach is universal but costs vision tokens, latency, and occasional misclicks. agent-desktop reads the accessibility tree, so targets are semantic elements with stable refs, and staleness is detected rather than guessed. It's also model-agnostic — a tool your agent calls, not a model you rent. The two approaches can be combined.

Compared with browser automation (Playwright, browser agents). Those stop at the browser. agent-desktop targets native windows across three OSes.

Compared with classic RPA / AutoHotkey-style scripting. Coordinate scripts break silently. Snapshot-scoped refs and typed error dispositions are built for an LLM loop that must know when to re-observe.

Compared with OS-specific libraries (pywinauto, AppleScript/JXA, AT-SPI bindings). One JSON contract and CLI across platforms, so the agent prompt doesn't change per OS.

TidyHow hasn't benchmarked success rate or latency against CUA stacks, and the README doesn't publish such a comparison. Treat the trade-offs above as architectural, not measured.

Minimal loop (from README)

agent-desktop snapshot --app Finder -i
agent-desktop click '@s8f3k2p9:e8'
agent-desktop snapshot --app Finder -i

Practical notes

• Grant permissions deliberately, and only on machines where an agent is meant to act.
• Treat snapshot output as sensitive data.
• On dense UIs, use skeleton or partial snapshots to limit token use (see docs).

Repo link

https://github.com/vercel-labs/agent-desktop