voice-first ai ordering
Saffron & Smoke — Voice-First AI Ordering
Voice ordering where the agent can only speak prices its tools returned.
- role
- full-stack AI engineer
- status
- voice backend paused
- scale
- 17 agent tools
- stack
- Pipecat · Gemini Live · FastAPI · pgvector
- type
- personal project
FIG.01 // SAFFRON-SMOKE · ONE SPOKEN ORDER
The agent can only say a total a tool returned
One tool result drives both the spoken total and the card on screen, so speech and screen cannot disagree.
Prices come only from tool results
The agent is told it is bad at arithmetic and forbidden from computing or remembering a price in its head. Every cart-changing tool returns the new authoritative grand total, and the only number the agent may say aloud is one from its most recent tool response.
Money is Decimal throughout, and the charge policy is computed server-side: GST is split into CGST and SGST at half the rate each, with the pair forced to sum back to the exact GST amount so rounding cannot open a one-rupee gap on the printed bill.
The same tool result that feeds the spoken total also draws the card on the canvas. Because every order change has to go through a tool, the agent cannot claim it changed something the screen does not show.
10% service charge5% GST → CGST + SGST5% AC charge on dine-in
A tool set built per room, per session
The tool set is assembled per connection, not filtered at call time. Of 17 tools defined, a delivery session is handed 16 and a dine-in session 12, with 11 shared. The agent is never shown a tool that is wrong for the room it is in, so "ask for the table number" cannot happen on a delivery call.
There is no table-select tool, and a test asserts its absence. A dine-in table is bound only by scanning its QR token, before the guest speaks, so the agent already knows where they are sitting and never asks.
Delivery puts identity first: nothing enters a cart until the caller is looked up or registered, the pincode is checked against the service areas, and a 4-digit recall code is read back digit by digit.
17 tools defined16 delivery · 12 dine-in11 shared
A concurrency cap that survives crashes
A live voice session is the most expensive thing in the system and the easiest to leave running. MAX_CONCURRENT_CALLS = 3 is enforced in Redis rather than in process memory, so the limit holds across every Cloud Run instance instead of per container.
The slots live in a Redis sorted set with stale entries pruned, so a session that dies without cleaning up does not hold its slot for good, and a regression test covers the missing-EndFrame leak. On a Redis error the limiter fails open.
Around it, the socket opens only with a single-use ticket held in Redis for 30 seconds and redeemed by an atomic get-and-delete, and every session is closed at MAX_SESSION_SECONDS = 900 after a warning.
3 concurrent sessions30s ticket TTL900s session ceiling
FIG.02 // SAFFRON-SMOKE · HYBRID MENU SEARCH
Two ranked lists, fused rather than weighed
"M01" is all keyword; "something not too spicy with paneer" has no keyword overlap at all. Both lists are kept and their ranks fused.