Real-time voice AI agent integration

A single conversational turn is a serial chain, and every link spends from the same budget. Audio has to arrive over the telephony carrier. Speech-to-text has to transcribe it. The language model has to decide what to say. Text-to-speech has to synthesise the reply. The audio has to travel back. Add those up honestly and there is no room anywhere for a step that takes its time.

Roughly half a second is where a caller stops experiencing a pause as a pause. Past that, people start talking over the agent, repeating themselves, or asking whether anyone is there — and once that happens the turn is lost regardless of how good the answer was going to be. Holding end-to-end latency under 500ms is not a performance nice-to-have here; it is the difference between a conversation and a broken one.

That constraint decides things a text system never has to think about: streaming rather than waiting for complete responses, keeping the model's work small enough per turn to finish inside the budget, and treating every added hop as something that must justify its own milliseconds.

The platform runs on Telnyx and Genesys for carrier-grade inbound calling, which is what puts this in production rather than in a demo. Real calls arrive on real numbers, concurrently, from people who did not agree to be part of a test.

Agents are provisioned dynamically rather than being defined ahead of time as a fixed set. A call can be matched to the right configuration as it arrives, and queries routed live to the right handling path during the conversation — the routing decision belongs in the call, not in a deployment.

Concurrency is the part that separates this from a single-session prototype. Multiple inbound calls run at once, each holding its own conversational state, each independently inside its own latency budget.

GPT-4 handles dialogue management — tracking what the caller wants across turns, deciding what to say next, and knowing when the conversation needs to move somewhere else. Retell and Vapi handle the speech layer on either side of it.

Splitting the responsibility this way keeps each piece replaceable. Speech vendors and model providers both move quickly, and the parts of a voice stack that are commodities today were differentiators eighteen months ago. Keeping the orchestration separate from the speech and the reasoning means neither one is a rewrite when a better option shows up.