gui-agents

Agent S: the state-of-the-art open source computer use agents

STEP-GUI: The top GUI agent solution in the galaxy. Developed by the StepFun-GELab team and powered by StepFun’s cutting-edge research capabilities.

💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.

[ICLR 2026 Oral] ScaleCUA is the open-sourced computer use agents that can operate on cross-platform environments (Windows, macOS, Ubuntu, Android).

An understudy watches. Then performs.

The AI agent that physically operates a phone — the way you do.

[ICLR'25 Oral] UGround: Universal GUI Visual Grounding for GUI Agents

Open, source-backed catalog of computer-use agents: reviewed papers, GUI models, datasets, benchmarks, tools, and reusable JSONL/SQLite data.

[ECCV 2026] Official Repository for "DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents"

Official code for "Focused Omni-Modal GUI Agents“

Official code for "Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents".

Official implementation of UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning

[AAAI 2026] Test-Time Reinforcement Learning for GUI Grounding via Region Consistency https://arxiv.org/abs/2508.05615

DataClaw: Agentic Tailoring Multimodal Data from Raw Streams — coming soon (code, weights, dataset & DataClaw-val upon acceptance).

[CVPR 2025] Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

Code repo for "Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding"

[ACL'25 (Findings)] Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

Curated resources about automated GUI computer-use via LLMs. Highly opinionated, focus is on quality vs quantity.

PreAct: Computer-Using Agents that Get Faster on Repeated Tasks (arXiv:2606.17929). Compile successful runs into verified state-machine programs and replay them 8.5–13× faster with no per-step LLM calls.

Papers, benchmarks, demos, and tools for multimodal coding agents: visual feedback for UI, CAD, 3D, games, charts, and robotics.