MobileWorld Benchmark Evaluates Autonomous Mobile VLM Agents via MCP and Interactive Tasks

A literature review highlights MobileWorld, a new benchmark designed to evaluate autonomous mobile VLM agents across 201 real-world tasks in around 20 mobile applications. The benchmark introduces two key evaluation axes: user interaction tasks requiring agents to query users for missing information and tasks augmented with Model Context Protocol (MCP) tools for direct API data retrieval. Evaluating AI agents on interactive dialogue and direct tool integration goes beyond basic screen-tapping benchmarks, reflecting real-world smartphone usage. As on-device AI advances toward functional automation, MobileWorld establishes a stricter standard to reveal current limitations in multimodal reasoning and hybrid tool usage. MobileWorld employs a planner-executor architecture where a VLM planner relies solely on screenshots (without accessibility trees) to output text actions, which a grounding model like UI-Inst-7B converts into exact (x,y) screen coordinates. The top-performing combination of Gemini-Pro and UI-Inst-7B reached an average success rate of ~52%, with accuracy dropping sharply on tasks involving user interactions and MCP tool calls.