AI Agents Crack Under Week-Long Android Builds: Pass Rate 28%

Google's Android Bench 2.0 drops top AI agents from 91% to roughly 28% pass rates by replacing small patches with multi-day app engineering tasks.

@sharedot

AI Agents Crack Under Week-Long Android Builds: Pass Rate 28%

By @sharedot · · 6 pages

  • AI Frontier
  • AI Benchmarks
  • Coding Agents
  • Android

Google's Android Bench 2.0 drops top AI agents from 91% to roughly 28% pass rates by replacing small patches with multi-day app engineering tasks.

Google swaps bug patches for week-long app builds

Google published Android Bench 2.0 on 16 September, adding 30 long-horizon tasks that would take a human engineer several days to a week.

The tasks include building a multi-screen food-delivery app from design mocks, migrating libraries and architecture, adding features like widgets and Picture-in-Picture, and converting Flutter or React Native apps to native Android.

The new benchmark aligns its testing structure with the Harbor framework, replacing the earlier focus on small bug fixes and local code changes.

The score collapse is the story

The reversal is stark: on the original task set, top models reached pass rates near 91%, but on the new long-horizon tasks the best result is about 28%, according to both Tbreak UAE and Developer Tech News.

OpenAI's GPT-6 Astra leads the long-horizon leaderboard at 28%, while Gemini 3.8 Flash records 8% per Tbreak UAE. Developer Tech News reports the leaderboard also includes Gemini 3.7 Flash, Anthropic's Fable 5.1, Kimi K3, and Qwen 3.8-Max.

Matthew McCullough, VP of Product Management for Android Developer, told Developer Tech News that "on multi-day engineering tasks, binary pass or fail grading doesn't capture the full picture."

A completion rate that credits partial progress

Android Bench 2.0 reports both pass rate and completion rate, so an agent that converts 40 screens to Jetpack Compose and satisfies 90% of requirements is no longer flattened to zero over one failed edge case, Developer Tech News explains.

The completion score combines functionality, regression checks, requirements and visual fidelity, with runtime checks, database inspection, accessibility-tree analysis and a multimodal visual judge per Tbreak UAE.

Model cards also display average compute expenses per task, and the evaluation applies objective deductions when a model disobeys instructions or breaks structural project constraints.

Where agents stall, and what it means for builders

According to Developer Tech News, models handle fresh code far better than refactoring complex existing codebases, where success depends on architectural hierarchy rather than code volume.

Deterministic conversions scale impressively — Java-to-Kotlin translations and ViewModel migrations across more than 125 files and 8,000 lines of code — but dynamic runtime validation, breaking framework changes, and unreleased libraries cause trouble.

Cross-platform conversions remain hardest: no model achieved a perfect pass rate, with frontier systems peaking at an 80% completion score.

Tbreak UAE notes the benchmark pairs models with provider agents — GPT 5.6 Sol with Codex, Gemini 3.8 Flash with Antigravity — and that harness design measurably affects token use, meaning a model can deliver a convincing first 80% of an app while leaving architecture cleanup, edge cases and polish to a human.

Sources

  1. tbreak.com › Android Bench 2.0 tests AI agents on week-long app builds
  2. developer-tech.com › Google's Android Bench 2.0 tests AI models on complex tasks

More on AI Frontier