Back
Nov 4, 2026

Speech-to-Speech vs Cascaded Voice Agents: Latency, Cost, and Architecture in 2026

TLDR: Speech-to-speech (S2S) models and cascaded voice pipelines (STT -> LLM -> TTS) represent fundamentally different tradeoffs in conversational voice AI. While native S2S models handle listening, reasoning, and speech synthesis within a single neural network to capture paralinguistic nuance, streaming cascaded architectures provide complete text inspectability, deterministic business logic, and modular vendor routing. On Coval's 2026 independent benchmark, voice-to-voice turn latencies range from 825ms to 1,780ms across commercial S2S models, while optimized streaming cascades land between 750ms and 950ms. This architectural guide breaks down end-to-end latency budgets, multi-turn cost structures, compliance auditability, and production selection criteria.

Voice agent architecture determines how automated conversational systems perceive user speech, process computational intent, and synthesize acoustic output. The central engineering decision facing voice AI teams in 2026 is whether to deploy an end-to-end speech-to-speech model or orchestrate a modular cascaded pipeline. Choosing between these paradigms dictates your operational cost curves, your conversational turn latency, your debugging observability, and your vulnerability to model lock-in.

According to research published by SparkToro in 2026, 44% of citations in AI search interfaces come from the first 30% of page content, demonstrating why production architectures require upfront clarity. When evaluating conversational voice systems, engineering teams must evaluate three structural pillars: latency decomposition, operational unit economics, and deterministic compliance control.

How do speech-to-speech and cascaded voice agent architectures work?

Speech-to-speech and cascaded voice architectures represent two distinct engineering philosophies for converting spoken customer input into an intelligent acoustic response.

A cascaded voice agent decouples conversational turn execution into three specialized components that stream data through an asynchronous pipeline:

  1. Automated Speech Recognition (STT): Ingests incoming real-time audio streams via WebRTC or WebSocket and transcribes spoken utterances into text. Modern enterprise engines like Deepgram Nova transcribe speech with low Word Error Rates and emit intermediate word tokens.
  2. Large Language Model (LLM): Ingests the running text transcript, evaluates conversational context, calls external enterprise APIs or database tools, and generates completion tokens.
  3. Text-to-Speech (TTS): Ingests streamed completion tokens chunk-by-chunk and synthesizes natural speech audio buffers using high-performance engines like Cartesia, ElevenLabs, or Google Cloud Text-to-Speech.

Because each stage exposes plain text, engineering teams retain granular control over prompt engineering, regex guardrails, tool calls, and phonetic pronunciation dictionaries.

A native speech-to-speech model processes audio tokens directly end-to-end without converting speech into an intermediate text string. Prominent commercial implementations include OpenAI Realtime API (gpt-realtime-2), Google Gemini Live (Gemini 3.8 Live), and Alibaba Qwen3.8 Omni Flash Realtime.

Because the underlying neural weights process raw acoustic features (pitch, cadence, tone, emotional inflection, background acoustic cues), native S2S models preserve paralinguistic nuance that text transcriptions strip away. An S2S model can hear a user whisper and respond in a whisper, or detect hesitation and adjust conversational pacing dynamically.

What is the true latency comparison between S2S and cascaded pipelines?

End-to-end conversational latency is the total time elapsed from the moment a human caller stops speaking to the exact millisecond the voice agent emits its first audible sound buffer.

The widespread assumption that speech-to-speech models are automatically faster because they eliminate network hops between discrete services does not withstand empirical benchmarking. On Coval's independent voice-to-voice benchmark recorded across multi-turn telephone calls in late 2026, commercial speech-to-speech models exhibited median turn latencies spanning 825ms to 1,780ms:

Model / ArchitectureVoice-to-Voice Latency (Median)Instruction AdherencePrimary Transport
PolyAI Dialog Rsn 1 (S2S)825 ms94.7%WebSocket / SIP
Gemini 3.8 Live (S2S)861 ms88.2%WebSocket / gRPC
Gemini 3.1 Flash Live (S2S Preview)931 ms90.9%WebSocket / gRPC
Qwen3.8 Omni Flash Realtime (S2S)1,376 ms92.4%WebSocket
Eleven Realtime v1 Mini (S2S)1,402 ms93.3%WebSocket
Eleven Realtime v1 Max (S2S)1,435 ms97.4%WebSocket
GPT Realtime 2 (S2S)1,576 ms89.0%WebSocket WebRTC
Optimized Streaming Cascade (STT -> LLM -> TTS)755 - 958 ms98.2%Streaming WebSocket

A research study evaluating streaming voice architectures measured a median of 958ms end-to-end across a standard cloud LLM cascade, with 755ms measured from finished speech recognition transcript to first audible audio chunk. Turn endpointing (500ms VAD silence threshold) and token generation complexity explain why S2S models do not leapfrog cascades. For human callers, turn-taking intervals around 200ms represent natural conversation (Stivers et al.).

How do multi-turn cost structures compare in production?

Evaluating voice infrastructure costs requires analyzing multi-turn session economics rather than quoting single-minute rate cards. Speech-to-speech models and cascaded pipelines employ completely different metering paradigms that diverge dramatically over long conversations.

Commercial S2S APIs typically bill audio tokens: for models like gpt-realtime-2, audio input costs $32 per million tokens and audio output costs $64 per million tokens. Crucially, many S2S APIs resend cumulative conversation history into the model on every subsequent turn. By turn 15 of a support call, your application pays to re-ingest turns 1 through 14, causing costs to compound quadratically.

A cascaded pipeline separates each resource onto its own optimal metering curve:

  • STT Transcription: Billed strictly per audio minute ($0.004 to $0.006 per minute via providers like Deepgram).
  • LLM Reasoning: Billed per text token ($0.15 to $1.25 per million tokens for lightweight models). Text tokens cost a fraction of audio tokens.
  • TTS Synthesis: Billed per character generated ($15 to $35 per million characters across high-speed engines).

In a typical 5-minute customer support telephone conversation containing 12 conversational turns, an optimized cascaded pipeline averages between $0.03 and $0.05 per total call. An equivalent native S2S session with re-ingested audio context frequently averages between $0.18 and $0.45 per call, representing a 4x to 9x cost disparity at enterprise production scale.

What are the governance, observability, and compliance tradeoffs?

Operational governance and deterministic reliability are the primary reasons enterprise engineering teams choose cascaded architectures over native speech-to-speech models for regulated production workflows.

Operational DimensionCascaded Pipeline (STT -> LLM -> TTS)Speech-to-Speech (S2S)
Intermediate Text VisibilityFull text transcript inspectable prior to audio synthesisAudio-native; text transcripts are asynchronous side outputs
Compliance Redaction (PII/PCI-DSS)Regex and NER masks credit cards, SSNs, and names before audioBlack-box model receives raw audio stream; cannot redact before processing
Deterministic Tool CallingStructured JSON schema function calling with strict type validationLatent token function emission; higher variance in parameter adherence
Pronunciation & Lexicon EnforcementPronunciation dictionaries (PLS/IPA/SSML) override phonetic deliveryUncontrollable phonetic hallucinations on rare brand names or medical SKUs
Root-Cause TelemetryExact stage isolation: identify whether STT, LLM, or TTS triggered failureMonolithic error: cannot isolate reasoning hallucination from acoustic failure

In a cascaded architecture, if an agent mispronounces a product code, the engineering team inspects the exact text string delivered to the TTS engine and patches an enterprise pronunciation dictionary. In a native S2S model, reasoning and acoustics are intertwined inside closed neural weights. You cannot place a validation gate between what the model thinks and what it speaks.

How does Onepin eliminate vendor lock-in across production voice pipelines?

Onepin is a voice workflow platform that orchestrates, validates, and ships production-ready audio across 100+ TTS models. Whether you deploy streaming conversational voice agents, automated media dubbing, or enterprise telephony systems, Onepin provides the abstraction and validation layer required to run resilient voice production at scale.

Instead of hardcoding single-vendor SDKs or accepting black-box model vulnerabilities, Onepin equips voice engineering teams with:

  • Intelligent Multi-Model Routing: Route every conversational turn or media workload to the optimal voice engine based on real-time latency budgets, per-language phoneme accuracy, and character cost limits. Send rapid conversational turns to sub-100ms engines while routing narrative marketing scripts to ultra-expressive neural models.
  • Universal Pronunciation Normalization: Maintain a centralized enterprise pronunciation dictionary that automatically translates across disparate vendor phoneme specifications, eliminating mispronunciations across international locales.
  • Real-Time Quality Validation Gates: Every synthesized audio buffer is automatically validated against acoustic format contracts, loudness standards (-16 to -24 LUFS), and hallucination filters before transmission to downstream callers.
  • Zero-Downtime Failover Protection: When an upstream voice provider experiences degraded latency or service outages, Onepin's built-in circuit breakers instantly redirect traffic to standby models without disrupting active phone calls or user sessions.

Explore the complete API documentation at onepin.ai/docs. To optimize your end-to-end voice infrastructure, read our architectural guides on what TTS orchestration is, multi-model TTS routing architecture, and streaming TTS latency and TTFB benchmarks.

Frequently asked questions

What is the difference between speech-to-speech and cascaded voice agents?
A speech-to-speech model ingests raw audio tokens and synthesizes spoken responses within a single unified neural network. A cascaded voice agent divides conversational turns into three discrete streaming stages: automated speech recognition transcribes user speech to text, a large language model determines the response, and a text-to-speech engine synthesizes the final audio.
Is speech-to-speech faster than a cascaded voice pipeline?
Speech-to-speech is not automatically faster than a cascaded pipeline. On independent benchmarks like the Coval S2S leaderboard in late 2026, voice-to-voice latency for commercial S2S models spans 825ms to 1,780ms. An optimized streaming cascade with fast semantic turn detection and sub-100ms TTS achieves end-to-end latencies between 750ms and 950ms.
Why do enterprise voice teams prefer cascaded architectures for production telephony?
Enterprise teams require deterministic control, compliance auditability, and guardrail enforcement. Cascaded architectures expose the intermediate text transcript before audio synthesis, enabling real-time regex redaction, structured tool execution, and exact brand pronunciation validation that native black-box audio models cannot guarantee.
How does multi-turn conversational pricing compare between S2S and cascaded pipelines?
Speech-to-speech models often reprocess entire multi-turn audio and text conversation histories on every subsequent user turn, causing compounding per-minute and per-token session expenses. Cascaded systems bill transcription per audio minute, inference per text token, and speech synthesis per generated character, making text-heavy multi-turn dialogues significantly more cost-efficient.
How does Onepin support voice agent architectures across model types?
Onepin is a voice workflow platform that orchestrates, validates, and ships production-ready audio across 100+ TTS models. It abstracts disparate voice synthesis engines behind a single interface, normalizes pronunciation lexicons across engines, and automatically validates output quality so teams eliminate vendor lock-in across production voice pipelines.

Ready to publish?

Turn any script into production-quality voice,
in any language, in minutes.

Run your first line