EyesAI:
Eyes for the phone
A Kotlin and Jetpack Compose Android app with a FastAPI backend that controls the device end-to-end through an AccessibilityService — a Planner → Executor → Verifier agent backed by on-device vision and TTS-first feedback, for blind and low-vision users.
No public repository or live deployment.
CORE CAPABILITIES
Autonomous Agent
Planner → Executor → Verifier owned by a Controller, reaching 87% task completion.
Accessibility Tree Control
A perception pass over the accessibility tree drives open_app, tap_element, tap_element_input_text_and_enter and done.
Dual-Pipeline Vision
On-device TFLite with COCO labels alongside an ML Kit labeler — 88.2% accuracy at 151ms latency.
Live Narration
Gemini Live scene narration over a throttled 1-fps frame feed from CameraX.
Multilingual Voice
English, Hindi and Hinglish commands at 94% accuracy — Levenshtein static matching with a Gemini function-calling fallback.
Non-Visual Feedback
TTS and haptics first, with displayLarge 36/44 and bodyLarge 17/24 accessibility typography.
On-Device Interface
Accessibility-first screens; the detection overlay is a debug view of the vision pipeline.
Home
Vision Pipeline
Autonomous Agent
AGENT LOOP
A Controller owns the loop. The four stages are separate components on purpose — the part that decides, the part that acts and the part that checks are never the same code, so a confident wrong plan still has to survive an independent verification step before the run continues.
Perception
The AccessibilityService dumps the current node tree — what is on screen, what is clickable, what is labelled. This is the agent's only view of the device; there is no screenshot-and-guess step.
Planner
Given the goal and the current tree, the Planner chooses exactly one next action. Planning one step at a time rather than a full script means a surprise dialog or a slow network does not invalidate a plan already committed to.
Executor
Performs the chosen action against the real device, then waits a 250ms settle delay before anything reads the screen again — enough for a transition or a keyboard to finish animating, so perception does not sample a half-drawn frame.
Verifier
Reads the screen after the action and returns Done, Failed or InProgress. The Controller ends the run on Done or Failed and otherwise loops, so progress is asserted by a separate check rather than assumed by the component that acted.
ACTION SPACE
open_app
Launch a target application.
tap_element
Tap a node resolved from the accessibility tree.
tap_element_input_text_and_enter
Focus a field, type, and submit.
done
Signal task completion back to the Controller.
Four actions is the whole vocabulary. A small action space is a deliberate constraint: every step the agent can take is one the Verifier knows how to check, and the model cannot invent an operation nobody wrote a check for. Text entry is a single action rather than three — focus, type and submit are fused, because the intermediate states are where an agent driving a real keyboard tends to lose the thread.
Everything is expressed against the accessibility tree, not screen coordinates, so the agent targets the element a screen reader would name rather than a pixel that moves with the device.
The core decision
Nothing is shown, everything is said
Most computer-vision demos put boxes on a camera feed. For the users this app is built for, that output does not exist. The interface had to be designed around the channels a blind user actually receives.
No bounding boxes on Look
The Look screen draws a full-bleed camera preview and nothing else. Boxes and label overlays are worthless to a blind user and noise to a sighted one, so the detection results never reach the screen.
Results arrive as speech and touch
Everything the vision pipeline finds is delivered through text-to-speech and haptics. The output channel is chosen for the user the app is for, not for the developer debugging it.
Typography sized for low vision
The type scale runs deliberately above the Material 3 baseline — displayLarge at 36/44 and bodyLarge at 17/24 — and is defined in sp, so Android's own font-size preference still scales it further.
The overlay image is a debug view
The detection screenshot in the gallery above is the debug visualisation of the pipeline, kept for development. It is not what a user of the app experiences.
The app ships four screens — Home, Look, Shop and Notes. Look is the full-bleed camera; Shop reads barcodes; Notes holds captured text. The navigation between them is reachable by voice, so the screen layout never becomes the thing standing between a user and a feature.
VISION — LOCAL FIRST
EfficientDet-Lite2 · TFLite
object_detection.tflite running on-device across 80 COCO classes. No round trip, so the common case never depends on the network.
ML Kit image labeler
A second pass that labels the scene rather than boxing objects, covering things the detector has no class for.
Gemini fallback
When both on-device passes come back empty or low-confidence, the question escalates to a hosted model — the expensive route is the exception, not the default.
Gemini Live narration
A throttled 1-fps JPEG feed to a live session for continuous scene narration. Gemini Live accepts at most one frame per second, so the feed is capped at the ceiling rather than flooding it.
Barcode scanning
The Shop screen runs ML Kit barcode scanning over the same CameraX analysis stream for product identification.
Two on-device passes run continuously over the CameraX analysis stream so an answer is already available when the user asks, rather than starting a request when they do. The measured pipeline sits at 88.2% accuracy and 151ms latency.
VOICE — THREE LAYERS
Porcupine wake word
"Hey Eyes AI" is detected on-device, so the app can be reached without finding a button on a screen the user cannot see.
StaticCommandMatcher
Known phrasings are matched by Levenshtein distance against a multilingual pattern table at a 70% similarity threshold — covering English, Hindi and Hinglish forms of the same command.
Gemini function calling
Anything the static matcher cannot resolve above threshold falls through to a hosted model that maps the utterance onto the same function surface.
The hybrid exists because the two approaches fail differently. String matching is instant, free and offline, but brittle against phrasing it has never seen. A model handles arbitrary phrasing but costs a round trip on every utterance. Routing the common commands through the matcher and the long tail through the model gets 94% accuracy across English, Hindi and Hinglish without paying network latency on the phrases users repeat all day.
TECHNICAL ARCHITECTURE
android
Kotlin
Application language
Jetpack Compose + Material 3
UI layer
CameraX
PreviewView, ImageCapture, ImageAnalysis
intelligence
TFLite + ML Kit
Dual-pipeline object detection
Gemini Live
Live scene narration
Porcupine
"Hey Eyes AI" wake word
backend
FastAPI
Service layer with WebSocket streaming
FAISS
Vector search
Vertex AI
Hosted model access