# StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

Page: https://stenobird.com/podcast/daily-paper-cast-7079649/stateact-program-state-before-pixels-for-long-horizon-computer-use-agents
Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/stateact-program-state-before-pixels-for-long-horizon-computer-use-agents.md
Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649)
Published: 2026-07-29T03:53:44+00:00
Episode link: https://share.transistor.fm/s/736ebc34
Audio file: https://media.transistor.fm/736ebc34/2fbf3716.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/stateact-program-state-before-pixels-for-long-horizon-computer-use-agents
Duration seconds: 1289

## Resource

🤗 Upvotes: 53 | cs.SE, cs.CV Authors: Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao, Yutong Dai, Ziji Shi, Hanshu Yan, Jun Hao Liew, Silvio Savarese, Junnan Li Title: StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents Arxiv: http://arxiv.org/abs/2607.22798v1 Abstract: Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoni…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/stateact-program-state-before-pixels-for-long-horizon-computer-use-agents/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/stateact-program-state-before-pixels-for-long-horizon-computer-use-agents.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.