If you need to know how to build an ai voice agent that doesn't fall apart on a live call, you start by ignoring the AI. The neural text-to-speech and the prompt engineering? That takes a day. The three months of engineering you are about to sign up for is dedicated entirely to moving audio packets around the globe in under 200 milliseconds without dropping the call when a human interrupts the machine. We know this because we built one. The full stack—from the telephony trunk to the compliance dashboard—is battle-tested in our outbound AI voice product, and the scar tissue is fresh.

This guide skips the toy demos. We will walk through the real-time telephony layer, the precise mechanics of turn-taking, and the operational back-office you need to prevent your agent from turning into a hallucinating loose cannon on a customer line.

How to Build an AI Voice Agent With Real-Time Telephony

The core of learning how to build an ai voice agent is accepting that a voice bot is not a text bot with a headset. It is a websocket management nightmare. You are building a low-latency relay race where audio frames, text transcripts, and function calls all need to cross the finish line in sync. Our stack grounds this in three components: Twilio for the public switched telephone network (PSTN) leg, LiveKit for real-time audio transport, and OpenAI's Realtime API for speech-to-speech inference.

1. Dialing In: Twilio as the PSTN Brain

Your agent has to exist on the legacy copper wire. We use Twilio as the Session Border Controller (SBC). When an outbound campaign triggers, the system places a call via the Twilio Programmable Voice API. The critical configuration is not the from number; it's twiml or the <Stream> verb. You must instruct Twilio to open a bidirectional media stream pointing directly at your LiveKit server URL.

Do not route the audio through your own media server in a single, fragile monolith before hitting the AI. Twilio's media stream format is raw 8-bit PCM mono at 8kHz. The first integration bug is usually a mismatch in this media format. Use a lightweight adapter that converts Twilio's base64 audio frames into the format expected by your speech service without adding latency. For us, LiveKit acts as the multicasting layer that takes this single input and distributes it to the AI, the transcription service, and the recording bus simultaneously.

2. LiveKit: The Bus for Audio Frames

You might think you can pipe Twilio directly into OpenAI. You cannot, not if you ever want a human in the loop or a second agent transfer. LiveKit is the extensible real-time transport. It manages rooms for each call. One participant is the Twilio phone trunk; the other participant is the AI agent process. LiveKit handles the WebRTC negotiation, audio redelivery, and data channel messaging.

The concrete takeaway: run LiveKit as your audio bus. The livekit-agents framework provides the AgentSession abstraction that handles the audio track subscription. Every time the remote caller speaks, LiveKit pushes that track to your worker. This architecture isolates the problem of global latency into a single component: the network complexity hits here, inside the room, not inside your model logic.

3. The Model Is the Easy Part

For our AI Calling Agent, the speech-to-speech layer uses OpenAI's Realtime API. This isn't a cascade of separate STT → LLM → TTS services glued together. The Realtime API accepts .raw audio chunks directly and returns audio chunks, maintaining the semantic cadence of the conversation. It handles inflection and pacing natively. You provide the system prompt—the role, the strict business logic—and the tool functions (like check_inventory or book_appointment), and the model doesn't just generate text; it generates instructions for when to call those tools.

The configuration is an event loop:

• session.update to set the voice and turn detection.
• conversation.item.create for user audio or text.
• response.create to trigger generation.

But here is the warning: the model works perfectly on a local loopback. Turn on the Twilio stream, and suddenly the model is talking over the customer. That brings us to the real challenge.

Handling Interruptions (Barge-in) Without Cutting Off the Brain

The hardest engineering challenge in an AI voice agent isn't the voice quality; it is interrupting the model mid-sentence and making it listen again without sounding like a broken robot. When a human coughs or says "Wait, no," the Vancouver pipeline you built instantly goes out of sync.

The Problem: The Race Condition

Most naive implementations treat the call pipeline as a walkie-talkie: one channel open at a time. But commercial telephony is full-duplex with bleed. You cannot use a simple Voice Activity Detector (VAD) that just toggles a "speaking" state, because background noise and the caller's echo will trigger it.

Our expert note from the field: "Interruption/barge-in handling on a live call, keeping the Realtime speech session in lockstep with the Twilio media stream, and giving ops an auditable transcription + review surface are what separate a demo from something you can put on real outbound calls."

The Solution: Cancellation Tokens and Audio Flushing

To handle barge-in correctly, you must treat the AI's audio output buffer as volatile.

1. Detect the Intent: LiveKit agents provide a TurnDetector class. When enabled, the agent monitors the audio level of the incoming user track. This isn't just dB threshold; it's semantic endpointing.
2. Truncate the Buffer: The moment an interruption is detected, your handler must fire a response.cancel event to OpenAI's Realtime API. This is the critical piece. It stops the token generation instantly.
3. Flush the Queue: You cannot just stop generating; you must flush the audio buffer already sent to Twilio. If you don't, the caller hears the dregs of the old sentence while the agent is trying to say a new one. Send a digital silence frame or terminate the MediaStreamTrack.
4. Reset the Context: With the audio flushed, you push the human's new audio into the model. The trick is design—your prompts must be stateless enough to withstand a sudden jump from "Let me tell you about our premium—" to "That's too expensive." Always ensure the interruption context is pushed as the most recent user event.

The Latency Budget

You have roughly 200ms of glass-to-glass latency before the conversation feels "off." That window includes Twilio's PSTN encoding, transport to your LiveKit instance, inference on the Realtime API, and audio packet travel back. You cannot fix physics, but you can co-locate your inference and LiveKit nodes in the same cloud region as Twilio's media edge.

The Operational Dashboard: Why You Build It Before Prompts

You cannot see a voice agent. It's invisible software. A user saying "your bot cut me off four times" is anecdotal and often wrong. You need ground truth. The only thing that makes an AI voice agent maintainable is an ops dashboard that logs concrete call data. Our case study led us to build exactly this: a full back-office surface.

The Auditable Component Stack

Using Next.js over Postgres and a CRM, we track these entities:

The Tuning Loop

Without this dashboard, you are tuning prompts in the dark. With it, a QA operator listens to a recording where the agent stalled mid-call. They don't guess. They look at the transcription, verify the crash happened after function_call, and adjust the tool execution timeout. "Design for turn-taking and a graceful fallback when the model stalls mid-call before you tune prompts." The dashboard is the fallback's visual front-end.

Case Study: The LiveKit + Twilio Architecture in Practice

This isn't a theoretical diagram. Our AI Calling Agent project uses exactly this architecture.

• Agents: An outbound AI voice ops product that manages campaigns.
• Flow: The ops dashboard initiates the outbound call. Twilio opens a stream. LiveKit routes the audio. The AI talks.
• Result: Instead of a brittle Python script running on a laptop, the client gets a Next.js surface where live calls are visible, transcription is auditable, and the agent's voice and behavior can be reconfigured without a code deploy.

If you want to scope a similar build or explore our AI services, you can see the patterns live in that case study.

FAQ

What LiveKit framework is best for voice agents?

The livekit-agents Python or Node.js SDK provides the AgentSession and TurnDetector classes required for full-duplex interrupt handling. This framework abstracts WebRTC transport, allowing you to focus on audio chunk routing rather than network state management.

How do I debug a voice agent that talks over the user?

Check the "Turn Detection" settings in your LiveKit agent. The default Voice Activity Detector (VAD) may be too sensitive to echo, or too slow to release. Adjust the threshold and silence_duration_ms in the LiveKit agent session, and ensure you are calling response.cancel on the OpenAI Realtime API the moment an interruption is detected.

Does a production voice agent need a dashboard?

Yes. A dashboard that displays live calls, searchable transcriptions, and campaign analytics turns an opaque AI into a supportable product. Without it, you cannot audit customer complaints or improve the agent's accuracy. For a look at how this surfaces in practice, see our work on AI calling operations.

Ready to move beyond a script and into a containerized, stateful agent? This is the initial ramp. The differences in milliseconds are what define a finished product. We can help you get started.