EyesAI

android accessibility agent

EyesAI — Autonomous Android Accessibility Agent

An Android agent for blind users where only the Controller, never the model, declares success.

role
developer
status
academic
scale
87% task completion (eval)
stack
Kotlin · FastAPI · Gemini · TFLite
type
academic · no public repo

FIG.01 // EYESAI · SUB-GOAL LOOP

Only the Controller can end a task

The Planner plans sub-goals and the Executor picks each action, but neither declares success. Every exit runs through the Controller.

Only the Controller can end a task01 · VOICEuser taskPLANPlannersub-goals onlyON FAILEDre-planmax 1SUB-GOALSsuccess_criteriamax_steps 5 each02 · PERCEIVEscreen stateXML + screenshot03 · ~19 ACTIONSExecutorgets screenshot04 · DEVICEUI actionAccessibilityServiceSETTLE250msno half-drawn frameCHECK · FLASH-LITEVerifierXML only05 · CONTROLLERowns terminationONLY EXIT06 · ANSWERTTS answerre-read from screeneach sub-goal carries its ownsuccess_criteria and a 5-step budgetIN_PROGRESSDONE · FAILED · IN_PROGRESSFAILEDlast DONE → extractAnswerDONE → next sub-goalstops after 3 perception failures;the answer is never the plan's done_text

An academic major project: a Kotlin and Jetpack Compose Android app with a FastAPI backend that drives the phone end to end through an AccessibilityService, for blind and low-vision users. In the project's evaluation the agent reached 87% task completion, the dual-pipeline on-device detection 88.2% accuracy at 151ms, and hybrid voice commands 94% accuracy across English, Hindi and Hinglish.

01

Only the Controller ends a task

The Planner turns a spoken task into sub-goals, the Executor picks each action, and the Verifier reads the screen after it. None of them ends the run. A Controller owns the loop and is the only component that can declare a task done or failed.

Every sub-goal carries success_criteria and max_steps = 5. After each action the loop waits a 250ms settle delay, so perception does not sample a half-drawn frame. A failed sub-goal earns at most 1 re-plan, and the run stops after 3 perception failures.

The part that decides, the part that acts and the part that checks are never the same code, so a confident wrong plan still has to survive an independent verification step.

max_steps 5250ms settle≤ 1 re-planstop after 3 perception failures

02

Each role gets the model it needs

Models are sized per role. The Verifier gets the accessibility XML only, on flash-lite. The Executor, which chooses among ~19 actions, gets a screenshot as well. Each role runs on a Gemini → Groq fallback chain.

Actions are expressed against the accessibility tree, not screen coordinates, so the agent targets the element a screen reader would name rather than a pixel that moves with the device.

~19 actionsGemini → Groq fallback

03

The answer is re-read from the screen

When a task asks for information, the final answer is extracted from the screen after the last step rather than taken from the plan's done_text, so what the user hears is what the phone actually shows.

Everything reaches the user as speech and haptics. The Look screen draws no bounding boxes, because boxes are worthless to a blind user; the detection overlay in the screenshots is a debug view of the pipeline.

screens