what i learned trying to make a phone agent sound less robotic on turn-taking
A developer details tuning when a phone agent should start speaking after a caller stops, combining VAD silence detection with STT partial-transcript signals — waiting when the transcript ends on an incomplete clause, responding sooner when it reads as complete. They also report TTS start latency must be tuned jointly with end-of-turn logic, since a slow voice start reads as an error rather than a natural beat.
- A Reddit practitioner reports combining VAD with STT partial-transcript checks—waiting on trailing conjunctions, responding sooner on complete thoughts—beats silence timers alone.
- The same writeup says TTS latency must be tuned alongside end-of-turn logic, since a slow voice start reads as a mistake even with solid turn-taking.
Related
how are you handling barge-in on voice agents without cutting off real speech?
A developer building phone agents describes the persistent difficulty of barge-in: sensitive VAD stops playback on coughs and background noise, while lenient VAD misses genuine interruptions. They run STT in parallel with TTS output and kill playback when caller speech overlaps, but short backchannels like 'yeah' still trigger false interrupts. They ask whether a two-stage pipeline (fast VAD gate followed by an STT confidence check) or SIP-layer endpointing could work without adding noticeable latency.
A developer building phone agents describes the persistent difficulty of barge-in: sensitive VAD stops playback on coughs and background noise, while lenient VAD misses genuine interruptions. They run STT in parallel with TTS output and kill playback when caller speech overlaps, but short backchannels like 'yeah' still trigger false interrupts. They ask whether a two-stage pipeline (fast VAD gate followed by an STT confidence check) or SIP-layer endpointing could work without adding noticeable latency.
Fusion-runtime – self-hosted voice agents running STT, LLM and TTS in one process
Fusion-runtime is an open-source runtime for building self-hosted voice agents that combine speech-to-text, an LLM, and text-to-speech in a single process. The LLM serves as the conversational reasoning layer between transcription and speech synthesis. It targets developers who want voice agents without depending on external cloud APIs or multiple separate services.
Fusion-runtime is an open-source runtime for building self-hosted voice agents that combine speech-to-text, an LLM, and text-to-speech in a single process. The LLM serves as the conversational reasoning layer between transcription and speech synthesis. It targets developers who want voice agents without depending on external cloud APIs or multiple separate services.
I measured the pause before an AI voice agent replies, and on xAI's realtime engine it has a floor around 1.3 seconds
A developer building a phone agent on xAI's realtime voice engine measured the time from end of speech to the agent's reply using a small script, xAI-ttfa.mjs. With xAI's server-side end-of-turn detection, the floor sits around 1.3 seconds; driving end-of-turn with the developer's own VAD lowers it to about 1.2 seconds. The finding matters to voice-agent builders because turn-taking latency directly shapes how natural phone conversations feel.
A developer building a phone agent on xAI's realtime voice engine measured the time from end of speech to the agent's reply using a small script, xAI-ttfa.mjs. With xAI's server-side end-of-turn detection, the floor sits around 1.3 seconds; driving end-of-turn with the developer's own VAD lowers it to about 1.2 seconds. The finding matters to voice-agent builders because turn-taking latency directly shapes how natural phone conversations feel.
Harvey: Claude-powered AI operator for a one-person solar contractor, running CRM, mail, docs, books, and voice in the field
Harvey is an AI operator inside a custom business platform built by a solo solar contractor in New Hampshire, with tools into his CRM, mailbox, documents, books, and equipment monitoring. Claude handles chat (bring-your-own-key), while voice uses Deepgram's voice agent with Grok for fast, hands-free reasoning. It assists real field jobs — recalling cases, taking dictation, reading spec sheets — for one-person trades businesses.
Harvey is an AI operator inside a custom business platform built by a solo solar contractor in New Hampshire, with tools into his CRM, mailbox, documents, books, and equipment monitoring. Claude handles chat (bring-your-own-key), while voice uses Deepgram's voice agent with Grok for fast, hands-free reasoning. It assists real field jobs — recalling cases, taking dictation, reading spec sheets — for one-person trades businesses.
Developers discuss where voice-agent latency hides after streaming is fixed
A developer running a production voice-agent stack (VAD, STT, LLM, TTS) reports turns still feel slow despite streaming, short prompts, and limited tool calls. Traces point to the model component, and the thread asks which latency metrics practitioners actually track in real-time agents.
A developer running a production voice-agent stack (VAD, STT, LLM, TTS) reports turns still feel slow despite streaming, short prompts, and limited tool calls. Traces point to the model component, and the thread asks which latency metrics practitioners actually track in real-time agents.