Deploying Voice Agents in Production

Everyone’s seen the demos. A voice agent that sounds human, responds naturally, and handles complex conversations. Then you try to deploy one on a real website and discover why most companies are still using chatbots.

I’ve deployed ElevenLabs voice agents on client production sites. Here’s what I learned.

The Demo vs Reality Gap

The demo:

Works perfectly on localhost
Has unlimited testing time
Uses your developer account with generous limits

The reality:

Needs to handle real users who don’t follow scripts
Has to work on mobile, with background noise
Costs money per minute of conversation

Cost Warning

Voice AI costs significantly more than text. A 5-minute conversation might cost $0.50-1.00. Plan your pricing accordingly.

The Technical Stack

Here’s what a production voice agent deployment looks like:

User Browser
    ↓
Widget JS (embedded on client site)
    ↓
WebSocket connection
    ↓
ElevenLabs Conversational AI
    ↓
Your backend API (for custom tools)

Key Challenges

1. Latency Matters

Users expect instant responses. Even a 500ms delay feels wrong. You need to:

Use the closest ElevenLabs region
Optimise your tool responses
Consider voice activity detection settings

2. Error Handling

What happens when:

The user’s microphone fails?
The WebSocket disconnects?
Your backend API times out?

Every failure mode needs a graceful fallback.

3. Context Persistence

Users expect the agent to remember what they said. But WebSocket reconnections can lose context. You need to:

Store conversation state
Pass context on reconnection
Handle graceful degradation

What Actually Works

After several deployments, here’s my checklist:

Area	Solution
Hosting	Cloudflare Workers for low latency
State	KV for conversation context
Fallback	Text chat if voice fails
Analytics	Track conversation completion rates

The Business Case

Voice agents work best for:

Lead qualification - Quick questions before booking a call
FAQ handling - Common questions with consistent answers
After-hours support - When humans aren’t available

They’re not great for:

Complex technical support
Emotional conversations
Tasks requiring visual confirmation

Start Small

Don’t try to replace your entire support team. Start with one use case, measure results, then expand.

Next Steps

If you’re considering voice agents:

Define a specific, bounded use case
Calculate the cost per conversation
Build a fallback path to human support
Test with real users, not just demos

Voice AI is real and it’s useful. But it’s not magic - it’s engineering.