An ai receptionist setup that never misses a call hinges on three things: a reliable telephony connection, a real-time speech AI that can hold a natural conversation, and a dead‑simple handoff path to a human when the call demands it. We built a full voice‑operations platform for outbound campaigns (see the case study), and the same stack — LiveKit, OpenAI’s Realtime API, Twilio — works flawlessly for inbound reception. This guide shows you how to assemble it, where the AI earns its keep, and where you must hand off fast.

What you’ll need before you start

You’ll provision a phone number, a real‑time voice pipeline, an LLM with speech‑to‑speech, and a scheduling/CRM connector. The good news: you can set up an AI receptionist without IT expertise — Zoom, for example, offers templates in its admin portal, and ElevenLabs provides out‑of‑the‑box calendar and CRM integrations. But if you want full control over latency, custom logic, and data ownership, you’ll build it yourself. Here’s the stack we use:

You can see the full dashboard we built for managing calls, agents, and transcriptions at /work/ai-calling-agent. It’s the same surface you’d use to monitor your AI receptionist.

Step 1: Connect a phone number to your voice pipeline

Takeaway: Twilio Media Streams give you a raw audio WebSocket that you can pipe directly into your AI agent, avoiding the jitter and delay of SIP‑based approaches.

1. Buy a Twilio number and point its voice webhook to your server.
2. On the webhook, respond with TwiML that opens a <Connect><Stream> to your WebSocket endpoint.
3. Your server receives 8 kHz µ‑law audio and sends it to the AI agent, which returns audio in the same format.

Here’s the minimal TwiML that starts a bidirectional stream:

<Response>
  <Connect>
    <Stream url="wss://your-server.com/stream">
      <Parameter name="callerId" value="{{From}}" />
    </Stream>
  </Connect>
</Response>

On your server, you’ll forward that stream to LiveKit’s room (or directly to the Realtime API if you skip the media server). The LiveKit room can mix multiple tracks — useful if you later add whisper messages for human handoff. If you’d rather use a turnkey service, platforms like GoTo Connect and RingCentral offer no‑code setups; businesses reported that their AI receptionist “has solved the problem of employees having to give out personal numbers and getting flooded with service calls.” But with a custom WebSocket stream you control every millisecond of latency.

Step 2: Build the AI agent that answers, triages, and books

Takeaway: A good AI receptionist prompt is a crisp business rule book, not a novella. It tells the agent who it is, what it knows, and exactly when to escalate.

Use the system prompt to define the persona, the business context, and the triage flows. For example, an AI receptionist for a dental clinic might look like this:

You are the front‑desk assistant for Bright Smile Dental. Answer warmly but concisely.

Triage rules:
- If the caller wants to book an appointment, ask for preferred date/time and check availability via the `check_availability` function.
- If the caller asks about insurance or billing, say "I'll connect you with our billing specialist" and return `transfer`.
- If the caller is a sales call, politely decline and return `hangup`.
- If the caller has an emergency (pain, broken tooth), immediately return `transfer` with priority high.

Do not make up pricing or medical advice. Never confirm an appointment before the function succeeds.

Behind the scenes, the Realtime API calls server‑side functions that you define. For booking, you’d implement a check_availability function that queries your calendar and returns open slots. For transfer, you’d have the function return a signal that your Twilio handler uses to invoke <Dial> or <Enqueue>. (We’ll cover handoff in the next section.)

This same pattern is what makes an AI receptionist earn its keep: it automatically filters sales calls, books routine appointments, and protects your team from after‑hours interruptions. Platforms like ElevenLabs and Synthflow offer plug‑and‑play versions of this, but the modular approach lets you hook into any CRM or custom logic you already have.

Step 3: Set up handoff rules — where the AI stops and the human starts

Takeaway: The moment a caller says “I need to speak to a person” or the AI cannot resolve the query, the system must hand off without friction. Doing otherwise destroys trust.

Define explicit handoff triggers in your AI’s function‑calling schema:

{
  "name": "transfer_call",
  "description": "Transfer the call to a live agent. Use when the caller asks for a manager, has a billing issue, or the AI cannot help.",
  "parameters": {
    "type": "object",
    "properties": {
      "reason": { "type": "string" },
      "priority": { "enum": ["normal","high"] }
    }
  }
}

When the AI returns transfer_call, your server modifies the Twilio call. The cleanest path is to <Dial> a phone number or <Enqueue> the caller into a TaskRouter workflow. For a whisper message (so the human agent knows what’s happening before they pick up), you can use <Conference> with a startConferenceOnEnter that plays a summary from the AI.

This hybrid model is what firms like Smith.ai offer with a human‑in‑the‑loop. But if you build it yourself, you get the exact same experience — and you never pay per‑minute for a human to answer a sales call again.

Step 4: Monitor, transcribe, and improve

Takeaway: A dashboard that shows call transcripts, sentiment, and handoff reasons turns your AI receptionist from a black box into a continuously improving assistant.

We built a back‑office surface for our outbound voice product that does exactly this (see the dashboard). For an inbound receptionist, you need the same:

• Call log with time, caller ID, duration, and outcome (booked, transferred, hung up).
• Transcripts stored in Postgres and searchable. You can use OpenAI’s Whisper for post‑call transcription if you don’t capture it from the stream.
• Agent configuration so you can tweak prompts and triage rules without a deploy.

These monitoring tools are the difference between an AI receptionist that annoys callers and one that gets better every week. Retell AI’s platform shows a 600ms end‑to‑end latency metric, and you can achieve the same by keeping the audio path short and handling barge‑in in the same process. When you spot a call where the AI stumbled, you adjust the prompt or add a new function — and the next caller gets the fix.

Should you build or buy?

If you need a quick setup with zero code, Zoom’s AI Concierge or GoTo Connect’s virtual receptionist will get you running in an afternoon. For budget‑conscious teams, Goodcall offers basic answering at a low price. But when you need custom workflows, sub‑second latency, and full control over your data, build on the stack we’ve outlined. Our AI receptionist solution packages exactly this — phone‑to‑AI pipeline, scheduling integrations, and a real‑time dashboard — so you can launch faster than stitching it all together yourself. Check our pricing and get started.

FAQ

How much does an AI receptionist setup cost?

DIY costs start at a few cents per minute for telephony and AI tokens, plus a modest server. For a turnkey setup with dashboard, latency tuning, and handoff logic, see our AI voice agent pricing.

Can an AI receptionist handle multiple calls at once?

Yes. The architecture we use (WebSocket streams + stateless workers) lets you spin up an agent per call. Platforms like Retell AI claim every call is answered at once, and your own implementation can scale to dozens of concurrent calls on a single server if you manage audio streams efficiently.

What's the latency like on an AI receptionist?

With a well‑tuned stack you can hit sub‑second response times. Retell AI reports end‑to‑end latency of about 600 ms, and our own pipeline using LiveKit and OpenAI’s Realtime API achieves similar results by keeping all processing in‑stream and detecting interruptions natively.