Saffron & Smoke:
Order by talking
Voice-to-voice restaurant ordering on a Pipecat pipeline with Gemini Live native audio — 18 agent-callable tools drive both the order state and the client canvas, over a hybrid pgvector + keyword menu search and a complete restaurant back-office.
CORE CAPABILITIES
Voice-to-Voice
Gemini Live native audio over a full-duplex WebSocket, 24kHz mono PCM16 out, with Silero VAD turn-taking.
18 Agent Tools
search_menu, add_to_order, check_delivery_area, confirm_order, request_bill, set_table and more drive both order state and the UI canvas.
Hybrid Menu Search
Keyword ranking fused with pgvector cosine similarity via Reciprocal Rank Fusion (k=60), on 1024-dim mistral-embed vectors.
Delivery Gate
Pincode-based service-area check backed by a dedicated service and repository.
Session Guardrails
Single-use Redis WebSocket tickets with a 30s TTL, a global 3-session concurrency cap, and a 900s max session length.
Full Back-Office
11 admin routes with role-based access: kitchen display, floor plan, tables & QR, menu CMS, analytics and more.
Ordering Experience
A conversation on the left, an agent-driven canvas on the right.
Landing
Voice Ordering
Agent-Driven Canvas
Order Confirmed
The hard part
A voice agent that cannot be wrong about money
Taking an order out loud is not a chat problem. The guest hears one number and holds the restaurant to it, there is no message to scroll back to, and anything the agent says that the screen contradicts is a dispute at the door. Most of the system prompt exists to remove the agent's ability to improvise.
The agent may not do arithmetic
It is told it is bad at arithmetic and forbidden from computing or remembering a price in its head. Every cart-changing tool returns the new authoritative grand total, and the only number the agent is allowed to say aloud is one from its most recent tool response.
Speech and screen cannot disagree
Every tool call draws a card on the canvas — menu results, the cart, the confirmation. Because every order change has to go through a tool, the agent physically cannot claim it changed something the screen does not show.
Confirmation is never volunteered
confirm_order fires only on an explicit instruction from the guest. The agent recaps items, address and grand total, asks once, and waits — a conversation feeling finished is not permission to place an order.
A loop guard at MAX_REPEATS = 2
If the agent calls the same tool with the same arguments three times in a row, the pipeline injects an instruction telling it to stop repeating and use the data it already has. Voice models that get stuck do so audibly, so the guard is a user-facing fix, not a log-level one.
The charge policy is computed server-side and handed to the agent as a finished figure: 10% service charge and 5% GST on every bill, plus a 5% AC charge on dine-in in an air-conditioned zone. GST is invoiced the way Indian tax invoices require it — split into CGST and SGST at half the rate each, with the pair forced to sum back to the exact GST amount so rounding cannot open a one-rupee gap on the printed bill.
HYBRID MENU SEARCH
Keyword ranking
A lexical pass over names, descriptions and tags — exact for an item ID like M01 or a dish name said clearly.
pgvector cosine similarity
mistral-embed vectors at 1024 dimensions, indexed with HNSW under vector_cosine_ops, for the requests that describe a dish instead of naming it.
Reciprocal rank fusion, k = 60
The two ranked lists are fused rather than one being preferred, so a result strong in either channel surfaces without tuning a weight between incomparable scores.
Graceful keyword-only fallback
If the embedding API key or model is unavailable the semantic helpers return nothing and callers drop to keyword search. Menu search degrades; it does not break.
People do not order the way a menu is written. "Something not too spicy with paneer" has no keyword overlap with Paneer Lababdar; "M01" has nothing but keyword overlap. The 27 items across six categories carry spice levels, dietary tags and popularity, so structured requests route to filter_menu while open ones go through the fused search.
SESSION GUARDRAILS
Single-use WebSocket tickets
The socket is opened with a ticket issued over HTTP and held in Redis for 30 seconds. Redemption is an atomic get-and-delete, so a ticket works exactly once and a replayed one is simply absent.
Global concurrency cap
MAX_CONCURRENT_CALLS = 3 is enforced in Redis, not in process memory — the limit holds across every Cloud Run instance rather than per container, which is the only way a cap means anything under autoscaling.
Hard session ceiling
MAX_SESSION_SECONDS = 900. The caller is warned before the cap and the session is closed at it, so an abandoned tab cannot hold a live model connection open indefinitely.
Pincode delivery gate
Delivery addresses are checked against configured service areas at registration, so an order is refused before it is built rather than after the kitchen sees it.
A live voice session is the most expensive thing in the system and the easiest to leave running. Every limit is therefore enforced in shared state rather than per process, because a per-instance cap on an autoscaling platform is not a cap.
Two contexts
Delivery and dine-in are not the same agent
Delivery
Identity comes first — nothing can be added to a cart until the caller is looked up or registered. A new customer gives name, phone and full address one field at a time, the pincode is checked against the service areas, and a 4-digit recall code is assigned and read back digit by digit so it survives being heard rather than seen. Returning customers give the code and skip the whole exchange. Preferences and allergies mentioned in passing are stored on the customer record for next time.
Dine-in
A QR code on the table binds the session to the table before the guest speaks, so the agent already knows where they are sitting and never asks. The two identity tools are swapped out for set_table and request_bill. confirm_order means something different here — it is the kitchen-fire step, and the guest can keep ordering rounds after it. The bill is requested at the end, which signals staff rather than closing a transaction.
11 SHARED TOOLS
Delivery only
Dine-in only
The tool set is assembled per connection, not filtered at call time: a delivery session is handed 16 tools and a dine-in session 13, out of 18 defined in total. The agent is never shown a tool that is wrong for the room it is in, so "ask for the table number" cannot happen on a delivery call. Special requests like "no onions" ride along as special_instructions on add_to_order rather than as free text the kitchen has to interpret.
AGENT TOOLS (SELECTED)
search_menu
Hybrid keyword + vector menu lookup.
add_to_order
Push an item into the live order state.
check_delivery_area
Validate a pincode against service areas.
set_table
Bind the session to a dine-in table.
confirm_order
Finalise the order and update the canvas.
request_bill
Apply GST and service-charge bill policy.
6 of 18 tools shown.
BACK-OFFICE
11 admin routes gated by role-based access control. Dine-in runs through QR table binding, customers are recalled with 4-digit codes, and billing applies a configurable GST and service-charge policy.
TECHNICAL ARCHITECTURE
voice
Pipecat
Full-duplex realtime pipeline
Gemini Live
gemini-3.1-flash-live-preview, voice "Charon"
Silero VAD
Via LLMUserAggregatorParams
backend
FastAPI + SQLAlchemy
API and data layer
PostgreSQL + pgvector
Menu embeddings, Alembic migrations
Redis
WebSocket tickets and concurrency limiter
client
Next.js 16 / React 19
Ordering client and back-office
Tailwind v4
Styling layer
ProtobufFrameSerializer
24kHz mono PCM16 audio over WebSocket