Hackathon submission · 2026 · Judge briefing

The small stuff sighted people
never think about.

Glance is a native iPhone visual assistant for blind and low-vision people. Point the camera, ask one precise question, hear one clear answer — processed entirely on the device, by an on-device Gemma 3n model, with no cloud in the loop.

See the full brief

Native iOS · SwiftUI On-device Gemma 3n Zero cloud in the demo path 6 core tasks + 2 stretch

●The experience

It looks like this in your hand.

Black-and-white liquid glass over a live camera view. One glowing control. Nothing else competing for attention while you're mid-errand.

Ready · on device

●The idea

Not a describe-everything camera. A precise-answer camera.

Most camera-based assistants for blind and low-vision users are built around one idea: describe the whole scene, continuously, and let the user parse it. That's generically impressive in a demo and exhausting in daily life — it turns every errand into a wall of description for one useful sentence buried inside it.

Glance inverts that. The user decides when the camera matters. They tap or speak, point at one thing, ask one precise question — "what bill is this?", "is this expired?", "which one's the black jacket?" — and get one short spoken answer back. No passive camera, no background recording, no ambient surveillance of the user's life to make the demo look smart.

Why this matters for dignity, not just UX

A tool that only watches when explicitly asked treats its user as someone directing an assistant, not someone being monitored by one. That distinction is the actual product decision, and it's why the six tasks below are scoped tightly instead of chasing a general "vision AI" pitch.

2.2B

people worldwide live with a vision impairment.

WHO, World Report on Vision, 2019

1B+

of those cases involve impairment that could have been prevented or is still unaddressed.

WHO, World Report on Vision, 2019

2–3GB

is all the on-device memory Gemma 3n needs to run — small enough for a phone, not a server rack.

Google DeepMind, Gemma 3n, 2025

●Why we believe this works

This isn't a hunch. It's backed by prior research and existing tools.

Three separate threads of evidence say the same thing: blind and low-vision users already want exactly this interaction pattern, academic work has validated it since 2010, and the on-device model needed to run it without a server now actually exists.

Prior research · 2010, UIST

VizWiz: Nearly Real-time Answers to Visual Questions

Bigham, Jayant, Ji, Little, Miller, et al. built and field-tested the exact interaction Glance uses: a blind user takes a photo, asks a question about it, and gets a short answer back. Their 11-participant field deployment — and the 100,000+ questions VizWiz went on to answer — is the earliest strong evidence that "photo + question → short answer" is the interaction pattern blind users actually want, not just one a sighted team assumes they'd want.

Read the paper (ACM DL) ↗

Model research · 2025, Google DeepMind

Gemma 3n: built from first principles for phones

Gemma 3n natively handles image, audio, video, and text input, and — unlike prior generations — was explicitly architected (MatFormer, Per-Layer Embeddings, a MobileNet-v5-based vision encoder) to run on phones and tablets, developed in collaboration with Qualcomm, MediaTek, and Samsung. That's what makes "one on-device model answers all six tasks" a real engineering option in 2026, not a hackathon fantasy.

Read the announcement ↗

How Glance compares to existing camera-based assistants
Tool	How it answers	Connectivity	Interaction model
Be My Eyes	Live human volunteer over video call	Requires a live internet connection	Continuous conversation with a person
Microsoft Seeing AI	Mix of on-device and cloud AI channels	Several channels need connectivity	Multiple specialized modes, chosen by the user
Google Lookout	On-device and cloud-assisted detection modes	Some modes need connectivity	Continuous scanning modes plus quick-look scan
Glance	Single on-device Gemma 3n model, task-prompted	Zero network calls in the core path	One tap or phrase, one frame, one short answer

●Architecture

One model, six tasks, zero servers.

Instead of hand-writing separate OCR, currency-classification, and object-detection pipelines, Glance sends the captured frame and a task-specific prompt to a single on-device Gemma 3n model and speaks back whatever short answer it returns.

Tap or speak

User initiates every request. Nothing runs passively.

Capture one frame

A single camera frame, not a live feed.

Route the intent

Read text? Identify object? Check status? Guide me?

Ask Gemma 3n

Frame + task prompt → on-device inference, fully local.

Speak the answer

One short sentence, spoken aloud and shown in large text.

No always-on camera, no face recognition, no remote storage. The one deliberate exception is the Phase 2 real-place navigation feature, which needs live MapKit/CoreLocation routing to work at all — that path never touches the camera or the model, and is scoped, disclosed, and off by default from the core experience.

●Scope, on purpose

Six everyday questions. Two stretch goals.

The core scope is deliberately narrow — precision beats breadth for trust. Two navigation-adjacent features were added afterward as lower-priority stretch goals, and are labeled as such below.

01
Read a page
Books, letters, and printed documents.
02
Read a menu or sign
The useful words, without the visual clutter.
03
Identify currency
Tell a one-dollar bill from a twenty.
04
Find expiration dates
A short answer from a crowded package.
05
Read a prescription label
Medication name and dosage, clearly.
06
Identify an object or color
Like finding the black jacket.
07
Ad-hoc scene description
"What's in front of me?" — one on-demand snapshot, not passive narration.Phase 2
08
Navigation
Vision-guided nudges to a nearby object, or real MapKit turn-by-turn to a place.Phase 2

●Team

Three builders, three tightly-scoped halves.

Backend · On-device model

Sanjay Vellore

Owns the Gemma 3n integration — runtime, per-task prompt design, and turning raw model output into short spoken answers.

Backend · Capture & orchestration

Karthik Tummala

Owns capture, voice input/output, intent routing, the orchestration engine, and the Phase 2 navigation module.

UI/UX & PWA

Naman Jain

Owns the SwiftUI experience — built for full VoiceOver and Dynamic Type support — plus the public marketing landing page.

●Roadmap

What's shipped, what's next.

Core (hackathon MVP)

Live camera capture + one-tap or voice invocation
On-device Gemma 3n answering the 6 core tasks via task-specific prompting
Spoken answers plus large-text display, full VoiceOver support
Follow-up questions on the same captured frame

Phase 2 / stretch

Ad-hoc scene description ("what's in front of me?")
Vision-guided local navigation to a visible or nearby object
Real MapKit turn-by-turn navigation to an actual place
Live GPS-triggered step advancement for navigation (beyond Phase 2)

●Judged on

Mapped directly against the judging rubric.

Rather than leave this to chance, here's exactly how Glance lines up against each criterion — with the specific code, tests, and decisions behind each claim, not just the pitch.

Exceptional-tier bar from the rubric, and how Glance meets it
Criterion	"Exceptional" requires	How Glance meets it
Creativity	"The project has an interesting design."	One on-device Gemma 3n model answers all 6 core tasks (plus 2 Phase 2 stretch tasks) through task-specific prompting — `PromptBuilder.swift` routes every intent (read text, identify object, check status, follow-up, describe scene, guide-to) through the same model with a different prompt, instead of six separate hand-built pipelines. That's the opposite of the field's default "describe everything" approach.
Implementation	"Works smoothly. Everything is well designed and there are little to no bugs."	`GlanceGemmaModelClient` wraps Google's LiteRT-LM engine running Gemma 3n E2B on-device, GPU-backed. `ResponseParser.swift` is built honesty-first: a machine-checkable `UNCLEAR` token (backed by a short-response-only uncertainty check) triggers a fixed fallback answer instead of ever forwarding a bad guess. 18 unit tests cover every intent and edge case (arrival-phrase detection, guidance-loop termination, uncertain-frame handling), plus real on-device integration tests against the actual model file.
Presentation	"Drafts an eloquent introduction and methodically explains the process of coding."	This page is that introduction — the Architecture section walks the 5-step pipeline in order, this table explains the specific engineering decisions behind each claim, and the team has a rehearsed demo script (menu → currency → prescription/expiration → object ID) staged with real physical props.
Q&A session	"Always respond to all questions with paragraph explanations, examples, and references to the code."	See "Anticipated questions" directly below — five real questions judges are likely to ask, answered in full with the exact files and logic involved, rehearsed ahead of time rather than improvised.
Relevance	"Addresses an important problem or need and demonstrates clear, meaningful real-world impact."	2.2 billion people live with vision impairment (WHO, 2019); the "photo + question → short answer" pattern was validated by real blind users as far back as VizWiz (UIST 2010). See the Evidence section for the full case, including how Glance compares to Be My Eyes, Seeing AI, and Lookout.

Anticipated questions

"How does this actually work with zero internet?"

Gemma 3n (the E2B variant) is bundled on-device and run through Google's LiteRT-LM engine, configured for GPU (Metal) on both the language and vision backends. GlanceGemmaModelClient lazily initializes one engine per session, converts the captured frame to JPEG, and sends it with a task-specific text prompt through engine.createConversation() — there is no network call anywhere in that path.

"What happens when the model gets something wrong, or isn't sure?"

Every camera-dependent prompt asks the model to reply with a machine-checkable UNCLEAR token if it isn't confident, rather than trusting free-text hedging. If that token — or one of a small backup list of uncertainty phrases, only trusted on short responses so a legitimately hedged-but-useful answer on a noisy real photo isn't discarded — is detected, Glance returns the same honest fallback line every time: "I couldn't read that clearly, try moving closer," instead of ever forwarding a guess.

"Why one model instead of six separate hand-built pipelines?"

PromptBuilder.swift routes every QueryIntent case through the same Gemma 3n model with a different task-specific prompt — the currency prompt asks for a denomination like "$1, $5, $10, $20," while the dosage prompt is distinguished from the expiration-date prompt purely by keying on hint text like "dosage"/"prescription"/"medication." One model, six behaviors, instead of one bespoke system per task.

"How does the vision-guided navigation know when to stop?"

GuidanceSession recursively captures a frame, calls the model with a .guideTo intent, speaks the instruction, and repeats every 2 seconds — it only ends when ResponseParser's arrival-detection logic finds the literal phrase "arrived at" in the model's response, or the user explicitly stops it. A session token invalidates stale in-flight callbacks so a delayed response from an old frame can't incorrectly end — or continue — a session that's already moved on.

"What's real here versus just planned?"

Every module named on this page is working Swift code, not a stub — including 18 unit tests asserting exact prompt content and parsing behavior per task, and integration tests that run real on-device inference against the actual model file with specific expected outputs (e.g. "amoxicillin," "2027," "teriyaki"). We're equally upfront about what's a deliberate hackathon cut: IntentRouter is documented in its own comments as "no ML classifier, just substring matching," and MapNavigator speaks a full turn-by-turn route up front rather than advancing live off GPS — both flagged as scoped-for-time decisions, not oversights.

●Sources

What we're citing

World Health Organization, World Report on Vision, 2019 — who.int
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., et al., VizWiz: Nearly Real-time Answers to Visual Questions, UIST 2010 — dl.acm.org
Google DeepMind / Google Developers Blog, Announcing Gemma 3n preview: powerful, efficient, mobile-first AI, 2025 — developers.googleblog.com
Google AI for Developers, Gemma 3n model overview — ai.google.dev