When AI Can “Understand” the Screen: Qwen UI Agent’s Ambition to Operate Across Devices
On August 20, Alibaba officially introduced Qwen UI Agent, a GUI agent foundation model designed around real-world environments.
Its target environments span mobile devices, desktop computers, web browsers, and DeepSearch workflows.
The core idea can be explained in one sentence:
AI can understand what is displayed on a screen and interact with apps and software in ways similar to a human user.
No application rewrite is required.
No dedicated API is necessary.
Instead, the AI can see the interface, understand what is happening, and perform the operation directly.
That may sound simple.
In reality, it addresses one of the hardest problems facing computer-use agents today.
The “Sim-to-Real Gap” of GUI Agents
The GUI-agent race has attracted plenty of attention over the past two years.
But many systems encounter the same problem:
They perform impressively inside simulators—and become much less reliable on real devices.
Alibaba describes this as the “sim-to-real gap.”
A simulated environment is usually clean and predictable.
The network is stable.
There are no unexpected advertisements.
Pop-ups are limited.
Authentication rarely expires.
The interface behaves exactly as expected.
The real world is completely different.
Modern mobile applications can contain hundreds of functions and layers of navigation. Pop-ups appear unexpectedly. Notifications interrupt workflows. Network conditions change. Login sessions expire. Buttons move. Content loads asynchronously.
That complexity is difficult to reproduce inside a laboratory simulator.
And there is another problem.
Real users rarely ask an AI agent to perform only three or four clicks inside a single application.
A genuine business workflow may involve:
Phone → desktop → browser → document → web application → phone again.
Most traditional assistants are still primarily reactive: users issue commands, and the agent responds.
Cross-device coordination, long-horizon planning, error recovery, and autonomous execution remain much harder.
Qwen UI Agent is therefore attempting to address three fundamental gaps at once:
Simulation vs. reality
Single-point capability vs. complete workflows
Human-driven operation vs. scalable automation
“Boot Camp” on Real Devices: 100+ Phones and 150+ Apps
How do you teach an AI system to operate in the real world?
Alibaba chose a surprisingly direct approach:
Put the model into the real world.
The team built an environment containing more than 100 real smartphones and 150 applications for task construction, trajectory collection, model training, and evaluation.
It also developed the MobileWorld-Real benchmark, covering more than 400 tasks across more than 100 applications.
Think of it as a real-device “boot camp” for AI.
Not simulated phones.
Not virtual applications.
But:
Real phones + real apps + real networks + real interruptions.
The main version of Qwen UI Agent contains only 27 billion parameters and is built on the Qwen 3.5 family.
Alibaba also provides a 35B-A3B MoE version, which activates approximately 3B parameters per token, as well as a lightweight 4B version.
By 2026 standards, 27B parameters is not particularly large.
But the important variable here is not simply parameter count.
It is the quality of the environment in which the model learns to operate.
English Version Part 2
Five First-Place Finishes Across Six Benchmarks
Ultimately, real-world performance needs to be measured by results.
According to the project's published benchmark results, Qwen UI Agent achieved first place in five of six major GUI benchmarks, outperforming several competing systems, including GPT-5.6 Sol and Claude Opus 4.8.
Its mobile performance is particularly notable.
MobileWorld: 82.1% — 12.0 percentage points ahead of GPT-5.6 Sol and 14.6 points ahead of Claude Opus 4.8.
MobileWorld-Real: 92.2% — demonstrating strong performance on real-device tasks.
AndroidDaily: 97.5% — approaching a near-perfect score.
Desktop performance is also competitive.
OSWorld-Verified reaches 79.5%, while the system reportedly reduces execution steps by approximately 58% compared with the baseline.
On the browser side:
WebArena: 73.6%
And for screen-grounding capability:
ScreenSpot-Pro: 81.5%.
The significance of these figures is not simply that a 27B model beats a larger model on individual benchmarks.
The more interesting point is that real-device training appears to be changing the efficiency equation.
Instead of relying exclusively on scale, the system is attempting to gain capability from better data, better environments, and better training trajectories.
Hybrid Operation: GUI + CLI
Qwen UI Agent is not limited to clicking buttons.
It supports a combination of:
GUI interaction + CLI commands.
For desktop tasks, command-line actions account for a significant share of execution, with roughly 40% of actions reportedly handled through batch-style output.
That matters because clicking is not always the most efficient way to operate a computer.
For example:
GUI can locate a webpage, identify a button, or visually understand a document.
CLI can download files, process large datasets, rename hundreds of files, or execute repetitive operations much faster.
In DeepSearch scenarios, the two modes can work together:
GUI → locate information → CLI → download and analyze data → GUI → inspect the result.
The model also supports online reinforcement learning on trajectories exceeding 100 steps, with training reportedly running across approximately 10,000 concurrent environments.
Long-horizon stability is critical.
A GUI agent that can perform five clicks is a demo.
A GUI agent that can reliably execute a 100-step business workflow is much closer to a production system.
Safety: Sensitive Actions Still Require Human Approval
Greater autonomy also creates greater risk.
Qwen UI Agent incorporates safety controls throughout task execution.
For illegal or high-risk requests, the system can refuse and terminate the task.
For sensitive operations—including payments, data deletion, and privacy authorization—the agent can stop at critical points and ask the user for confirmation.
The underlying principle is simple:
AI can operate the interface for you.
But high-risk decisions remain yours.
That distinction could become increasingly important as computer-use agents move from experimentation into real businesses.
Cross-Device Workflows + DeepSearch
Built around a Harness-style framework, Qwen UI Agent is designed to connect tasks across mobile devices and computers.
It can also work with DeepSearch, combining web information retrieval with GUI interaction.
This creates an important possibility.
A large number of traditional applications do not expose modern APIs.
Normally, integrating AI with those applications would require developers to build connectors or modify the software.
A GUI agent takes another route:
If a human can operate the software through the screen, the AI may be able to operate it through the screen as well.
That could dramatically expand the number of applications accessible to AI agents.
The Hidden Infrastructure Variable: Compute
There is, however, another side to the story.
Every time Qwen UI Agent:
sees the screen → interprets the interface → plans an action → executes it → checks the result → decides what to do next
it consumes compute.
And the training infrastructure is even more demanding.
Hundreds of real devices.
Thousands of concurrent environments.
Long-horizon trajectories.
Continuous reinforcement learning.
All of these have real infrastructure costs.
This is where Unexhub enters the picture.
Unexhub positions itself as a Token aggregation and distribution compute trading center, connecting AI model providers and compute suppliers through a more flexible marketplace.
Its model includes:
Instant settlement — suppliers can receive usage-based revenue more quickly instead of waiting through long settlement cycles.
API bidding — prices can dynamically respond to supply and demand.
Early-stage zero commission — early suppliers can access the platform with zero commission during the initial stage.
Aggregated AI supply — developers can access multiple AI providers through one ecosystem.
Millisecond-level routing — requests can be routed according to model availability, price, and performance requirements.
For developers, the potential advantage is not simply a lower headline price.
The more important benefit is choice.
Instead of being locked into a single provider's pricing structure, developers can compare and access aggregated supply.
Unexhub states that developers can pay according to actual token consumption, with eligible usage potentially reaching as low as 30% of prevailing market prices, while suppliers can connect their APIs and receive real-time usage-based revenue.
In other words:
Developers gain more opportunities to optimize compute costs, while suppliers gain another channel to monetize their capacity.
Two Layers of the Same Transformation
Qwen UI Agent is lowering the barrier at the application layer.
Unexhub is attempting to lower the barrier at the infrastructure layer.
At the application level:
“I need to operate this software”
is gradually becoming:
“AI can operate it for me.”
At the infrastructure level:
“I have to buy compute from one provider”
is gradually becoming:
“I can access aggregated supply and let market mechanisms determine the price.”
Put together, the direction becomes clear:
The cost of deploying AI agents is being reduced from both sides.
The application layer reduces the skill required to operate software.
The infrastructure layer attempts to reduce the cost of running models.
Let's Talk About What Actually Matters
The most important achievement of Qwen UI Agent is not necessarily that it scores a few percentage points higher than GPT-5.6 Sol on a particular benchmark.
The bigger achievement is that it demonstrates a different engineering path:
A GUI agent trained on real devices can potentially operate in the real world without requiring dedicated APIs, software modifications, or perfectly controlled simulators.
Three numbers tell an interesting story:
27B parameters.
100+ real devices.
92.2% on a real-device benchmark.
Together, they outline a possible path from:
“AI that performs well in the laboratory”
to:
“AI that can actually use computers in the real world.”
It may not be the largest model.
It may not be the model with the most parameters.
But it could become one of the models that is best at one deceptively difficult task:
Using a screen like a human.
And that may ultimately be the real breakthrough.
中文版 Part 1
当 AI“看懂”屏幕:Qwen UI Agent 的跨端野心
8 月 20 日,阿里正式发布 Qwen UI Agent。
这是一个以真实世界操作环境为核心设计的 GUI 智能体基座模型,覆盖移动端、电脑端、网页端以及 DeepSearch 深度搜索环境。
它最大的亮点,其实一句话就能说清楚:
AI 可以直接理解屏幕上的界面,并像人一样操作 App 和电脑软件。
不需要修改原有程序。
不一定需要专门开发 API。
AI 只需要“看懂”屏幕,就可以完成相应操作。
听起来简单,但这实际上正在挑战计算机使用型 Agent 最难的问题之一。
GUI 智能体最大的难题:模拟世界与真实世界之间的鸿沟
过去两年,GUI 智能体赛道并不缺玩家。
但很多模型都遇到了一个类似的问题:
在模拟器里表现很好,一到真实设备上就容易失灵。
阿里团队把这种问题称为:
“Sim-to-Real Gap”——模拟与真实之间的鸿沟。
实验室里的界面通常非常干净。
网络稳定。
没有突然弹出的广告。
很少出现登录过期。
按钮位置也不会随意变化。
但真实世界完全不同。
如今的超级 App 往往拥有大量功能和复杂入口。
弹窗会突然出现。
通知会随时打断任务。
网络状态会变化。
登录状态可能失效。
页面内容可能异步加载。
这些真实世界中的复杂性,很难通过普通模拟器完全复制。
更麻烦的是,真实用户需要完成的任务,往往也不是在一个 App 里点几下就结束。
一个完整的办公流程可能是:
手机 → 电脑 → 浏览器 → 文档 → Web 应用 → 手机。
而传统智能体大多仍然是被动式的:
用户下命令 → AI 执行。
跨设备协同、长程任务规划、错误恢复以及自主执行能力,依然是难点。
因此,Qwen UI Agent 项目的目标其实同时指向三个矛盾:
模拟与真实的矛盾;
单点能力与完整工作流的矛盾;
人工驱动与规模化自动执行的矛盾。
真机“军训”:100 多台手机、150 多款应用
怎么解决?
阿里选择了一种看起来最“笨”,但可能也是最直接的方法:
把 AI 直接扔进真实世界训练。
团队搭建了一个包含 100 多台真实手机、150 多款应用的移动环境,用于任务构建、轨迹采集、模型训练和评测。
同时,他们还建立了 MobileWorld-Real 真机基准,覆盖 400 多个任务和 100 多款应用。
可以把它理解成一座专门为 AI 建立的“真机军训营”。
不是虚拟手机。
不是模拟 App。
而是:
真手机 + 真应用 + 真网络 + 真干扰。
Qwen UI Agent 的主版本只有 27B 参数,基于 Qwen 3.5 系列基础模型构建。
此外还提供 35B-A3B MoE 版本,每个 Token 仅激活约 3B 参数,以及轻量级 4B 版本。
以 2026 年的模型生态来看,27B 并不算特别大的参数规模。
但这里真正值得关注的变量,并不是参数量本身。
而是:
模型究竟是在什么样的环境里学会“使用电脑”的。
中文版 Part 2
六大基准五项第一:真机表现才是硬指标
最终,GUI Agent 是否真正有价值,还是要回到实际任务完成率。
根据项目公布的核心测试结果,Qwen UI Agent 在 6 项主要 GUI 基准测试中拿下 5 项第一,表现超过 GPT-5.6 Sol、Claude Opus 4.8 等多个对比模型。
移动端表现尤其突出:
MobileWorld:82.1% ——领先 GPT-5.6 Sol 12.0 个百分点,领先 Claude Opus 4.8 14.6 个百分点。
MobileWorld-Real:92.2% ——在真实手机环境任务中展现出较高完成率。
AndroidDaily:97.5% ——已经接近满分。
电脑端同样表现不弱。
OSWorld-Verified 达到 79.5%,同时相比基线减少约 58% 的执行步骤。
浏览器端:
WebArena:73.6%。
界面定位能力:
ScreenSpot-Pro:81.5%。
这些数据真正值得关注的,并不只是“27B 模型超过了某个更大模型”。
更重要的是:
真机训练正在改变 GUI Agent 的能力获得方式。
过去很多人首先想到的是:
更大参数 → 更强能力。
而 Qwen UI Agent 展示的是另一条路径:
更真实的数据 + 更真实的环境 + 更长的任务轨迹 → 更强的实际操作能力。
混合操作:GUI + CLI,才是真正的电脑使用
Qwen UI Agent 并不只是“点屏幕”。
它支持:
GUI 图形界面 + CLI 命令行混合执行。
这是一个非常重要的设计。
因为很多时候,点击并不是最高效的操作方式。
例如:
GUI 更适合寻找网页、识别按钮、理解文档和观察视觉界面。
而:
CLI 更适合下载文件、批量处理数据、批量重命名文件,以及执行重复性任务。
在电脑任务中,CLI 动作占据了相当比例,其中约 40% 的动作以批处理方式完成。
DeepSearch 场景同样可以形成协同:
GUI → 找到网页信息 → CLI → 下载并分析数据 → GUI → 查看结果。
这比纯粹依赖鼠标点击更加高效。
模型还支持 100 多步超长轨迹的在线强化学习,并可以在约 1 万个并发环境中同时训练。
为什么长程任务如此重要?
因为:
能够点五下的 Agent 是 Demo。
能够稳定完成 100 步业务流程的 Agent,才更接近生产工具。
安全:高风险操作,AI 不能替你做决定
能力越强,安全就越重要。
Qwen UI Agent 将安全机制贯穿任务执行过程。
面对违法或者高风险请求,可以直接拒绝并终止任务。
面对支付、数据删除、隐私授权等敏感操作,则会在关键节点主动停下来,向用户说明情况。
高风险操作必须经过用户确认才能继续。
这背后的逻辑非常清楚:
AI 可以替你操作。
但涉及重大风险的决定,必须由你自己确认。
随着 Computer Use Agent 越来越多地进入企业环境,这种“执行权”和“决策权”分离的机制会越来越重要。
跨设备 + DeepSearch:AI 开始真正“使用电脑”
基于 Harness 框架,Qwen UI Agent 支持手机与电脑之间的跨设备任务衔接。
同时,它还可以与 DeepSearch 联动,把:
网页搜索 + 信息理解 + GUI 操作
连接起来。
这意味着一个很现实的问题可能开始出现新的解决方案。
现实世界里有大量传统软件没有开放 API。
过去,要让 AI 操作这些软件,通常需要开发专门的接口或者修改原有系统。
GUI Agent 提供了另一条路线:
只要人能够通过屏幕操作这个软件,AI 理论上也可以通过屏幕操作它。
这会显著扩大 AI Agent 可以触达的软件范围。
算力层的暗线:AI 越会“用电脑”,算力消耗越大
但还有一个容易被忽略的问题。
Qwen UI Agent 每完成一次:
看屏幕 → 理解界面 → 规划动作 → 执行动作 → 检查结果 → 决定下一步
都需要消耗算力。
训练阶段更是如此。
100 多台真实设备。
上万并发环境。
超长任务轨迹。
持续强化学习。
这些背后都是实实在在的基础设施成本。
这正是 Unexhub 试图解决的问题。
Unexhub 将自身定位为 Token 聚合分发的算力交易中心,希望连接不同规模的 AI 模型和 Token 供应商,通过市场化机制提供更加灵活的算力供给。
其中包括:
即时结算:供应商产生 Token 调用后,可以更快获得对应收益,降低传统账期造成的资金压力。
API 竞价模式:价格可以随着市场供需动态变化,而不是长期锁定在固定价目表。
早期入驻零佣金:早期供应商可以获得零佣金权益,降低进入平台的成本。
多模型聚合:汇聚不同模型与供应商,为开发者提供更多选择。
毫秒级路由:根据模型、价格、供给以及性能需求进行请求路由。
对于开发者来说,真正的价值不只是“价格低”。
更重要的是:
选择变多了。
开发者不必长期绑定单一供应商,而是可以进入更大的供给池,根据实际 Token 使用需求寻找更具竞争力的方案。
Unexhub 表示,开发者可以按照实际 Token 使用量付费,在符合条件的场景下,最低可享受约为市场价格 3 折的使用成本;供应商则可以通过 API 一键接入,并获得实时调用分成。
因此,它试图建立的并不是简单的“低价 Token 平台”。
而是一套:
开发者降低算力采购成本,供应商提升算力变现效率的双向市场机制。
应用层与基础设施层,正在发生同一场变革
Qwen UI Agent 降低的是应用层门槛。
Unexhub 尝试降低的是基础设施层门槛。
过去:
“我需要操作这个软件。”
现在逐渐变成:
“AI 可以替我操作。”
过去:
“我只能向一家供应商购买算力。”
未来则可能逐渐变成:
“我可以进入聚合供给池,让市场帮助我发现价格。”
两种变化放在一起看,方向非常清晰:
AI Agent 的落地成本正在两端同时下降。
应用层削平的是“操作软件”的技能门槛。
基础设施层削平的,则是“运行模型”的成本门槛。
说点实在的:真正重要的不是“参数有多大”
Qwen UI Agent 真正值得关注的地方,并不只是某一项基准测试领先 GPT-5.6 Sol 多少个百分点。
更重要的是,它展示了一条不同的工程路径:
一个在真实设备环境中训练的 GUI Agent,可以在不依赖专用 API、不修改原有程序、也不完全依赖理想化模拟器的情况下,更稳定地进入真实世界执行任务。
把三个数字放在一起看,会更有意思:
27B 参数。
100+ 台真实设备。
92.2% 真机任务完成率。
它们共同勾勒出了一条从:
“实验室里刷分的 AI”
走向:
“真正会使用电脑的 AI”
的路线。
它可能不是参数最大的模型。
也不一定是最“聪明”的模型。
但它可能正在成为一种非常特殊的 AI:
最会“使用屏幕”的模型之一。
而这,或许才是 GUI Agent 真正走向现实世界之后,最值得关注的变化。