What makes voice agents different from chatbots

To someone who has never worked in Voice AI, it might seem like simply attaching some voice API over the existing chat agents but its not quite true.

Voice Agents require a fundamental shift from asynchronous text processing to real-time, low-latency streaming. While chat agents operate on a "request-response" cycle that can tolerate seconds of processing time, voice agents exist in a synchronous, human-time environment where a 500ms delay feels like an interruption.

Here is the breakdown of the technical components that differentiate a Voice Agent from a standard Chat Agent:

1. STT (Speech-to-Text): Unlike text input where the content is already structured, STT must process streaming audio in real-time. It requires low-latency transcription (often using models like Whisper-Turbo or Deepgram or Sarvam(an Indian Providers)) that provides partial, fluctuating transcripts that update as the user continues to speak. The system must handle ambient noise, regional accents, and the "disfluency" of human speech (ums, ahs, and self-corrections) before the LLM can interpret intent.

2. ASD/VAD (Active Speech Detection / Voice Activity Detection): This is the "turn-taking" engine. In chat, the user hits "send" to indicate they are done. In voice, the agent must autonomously decide when the user has stopped speaking based on silence thresholds, energy levels, and acoustic patterns. If the VAD is too aggressive, it cuts the user off mid-sentence; if it is too slow, it creates awkward dead air that mimics a dropped call.

3. RAG (Retrieval-Augmented Generation): In voice, retrieval must be extremely optimized to avoid blocking the LLM generation loop. Because the latency budget is so tight (ideally <1.5s total round trip), RAG systems for voice often use high-speed vector caching or pre-fetched context. Unlike chat, where you might perform complex multi-step retrieval, voice RAG must return data in milliseconds, or the agent will appear to "stutter" while waiting for the database.

4. LLM (Large Language Model): Voice LLMs face a "determinism" constraint. While chat models can be creative and verbose, a Voice Agent needs to be concise and conversational. Crucially, the LLM must support "streaming token output," where the system begins sending audio fragments to the user before the full sentence is generated. It also requires specific tuning to avoid generating complex markdown or long lists that sound unnatural when spoken aloud.

5. TTS (Text-to-Speech): This is the final hurdle. The TTS engine must be capable of generating "phonetic streams" rather than waiting for an entire paragraph to finish. It needs to handle prosody—the natural rise and fall of human intonation—and integrate seamlessly with the LLM to handle interruptions (barge-in). If a user interrupts the agent while it is speaking, the TTS buffer must be cleared instantly to allow the agent to react to the new input.

The Tech Stack that we were using:

• Telephony: Vobiz (PSTN integration, Indian DID routing, call recording).

• Orchestration: LiveKit running on Azure Container Instances.

• Speech-to-Text (STT): Sarvam AI. We deliberately chose Sarvam over global engines because its servers are in India, slashing transit latency, and its Indic language/accent transcription accuracy on noisy phone calls is noticeably superior.

• LLM: Iterated across multiple models, settling on GPT-4o-mini.

• Text-to-Speech (TTS): ElevenLabs for natural voice cadence, with Deepgram wired as an automated operational fallback whenever API credits ran dry.

• Telemetry & Tracing: Langfuse.