A hands-on voice AI project built around Nimbus — a fictitious all-in-one cloud business suite. It pairs a live, customer-facing voice assistant embedded in a real web product with an interactive playground that exposes the full cascaded pipeline behind it: speech-to-text, a retrieval-grounded language model, and text-to-speech. Explore the finished experience, then open the hood and tune every stage yourself.
One link takes you to the finished voice agent living inside the Nimbus product. The other opens the engineering playground that reveals — and lets you reconfigure — the pipeline that powers it.
A talk-to-it voice assistant embedded directly in the Nimbus website — the kind of AI concierge you'd expect on a modern SaaS product. Speak naturally and it answers questions about Nimbus's plans, pricing and policies, then goes a step further and acts: adding items to your cart, totalling it up, surfacing the most expensive products, working out annual savings, and explaining the refund policy. Every answer is grounded in the site's own knowledge base, so the assistant stays on-topic and accurate instead of guessing.
The same assistant, with the roof taken off. This playground illustrates the full cascaded pipeline — ASR → LLM → TTS — and turns every stage into a control you can adjust. Swap the LLM model, choose how knowledge is fetched (RAG top-k retrieval, RAGless full-context, or none), pick a reranker, set response length and how many turns of history are kept verbatim, toggle tools on or off, and switch between streaming and batch. On the voice side you control the ASR and TTS engines, voice, barge-in (interrupting mid-sentence), and endpoint silence. A live latency panel then shows exactly what each choice costs — end-to-end time, time-to-first-token, ASR/TTS timings, the retrieved chunks, and per-tool-call latency.
A voice agent isn't one model — it's a relay of three specialised stages. Here's what happens between you speaking and hearing a reply.
Automatic Speech Recognition transcribes your spoken words into text the moment you finish talking, using silence detection to know when your turn ends.
The language model reads the transcript, pulls the most relevant facts from Nimbus's knowledge base (RAG), calls tools when an action is needed, and drafts a grounded reply.
Text-to-Speech turns the reply back into natural spoken audio — streamed as it's generated so the answer starts playing with minimal delay.