Back
Nov 12, 2026

TTS Audio Quality Metrics: Why PESQ Fails on AI Voice and What Replaces It

TLDR: Traditional objective speech metrics like PESQ (ITU-T P.862) assume time-aligned reference audio and fail on generative text-to-speech. Production AI voice pipelines use non-intrusive neural estimators like Microsoft DNSMOS (ITU-T P.835) and UTMOS to score naturalness, acoustic artifacts, and background distortion without human listening panels. Scoring every output against baseline thresholds stops degraded clips before they ship.

Objective speech quality evaluation in AI voice systems measures perceptual clarity, naturalness, and acoustic distortion using automated algorithms rather than manual listening tests. The core challenge in modern voice infrastructure is that legacy telecom metrics like PESQ require an exact reference recording, which generative models do not have. Production voice teams that rely on manual ear checks or outdated reference metrics ship silent degradation when models drift or fail under load.

Engineering teams transitioning from traditional VoIP pipelines to generative voice AI often carry over metrics built for codecs like G.711. Those metrics break immediately on neural synthesizers. Understanding the operational distinction between intrusive reference metrics and non-intrusive neural predictors is the foundation of reliable automated audio quality assurance.

Why does PESQ fail on generative text-to-speech?

PESQ (Perceptual Evaluation of Speech Quality, standardized as ITU-T P.862 in 2001) is an intrusive speech quality metric that compares a degraded audio signal against an original, time-aligned reference signal. In telecommunications, an engineer sends a clean human voice recording through a network link, collects the received packet stream, and runs PESQ to calculate perceptual degradation caused by packet loss, jitter, and codec compression.

Generative text-to-speech models do not have a reference recording. When a model like ElevenLabs, Cartesia, or Google Cloud Text-to-Speech receives text input, it synthesizes brand new acoustic waveforms with variable prosody, pitch movement, and timing.

Comparing generative TTS output to a human speaker's recording using PESQ produces false failures:

  1. Acoustic misalignment: Generative models change phoneme durations dynamically. Even minor micro-pauses or breath marks cause sample-level misalignment, triggering catastrophic PESQ score drops.
  2. Prosodic divergence: A voice model can deliver an expressive, pleasant performance that sounds entirely natural to human ears while diverging completely from the reference recording's fundamental frequency ($F_0$) track.
  3. Bandwidth mismatches: Standard PESQ was designed for narrowband (8 kHz) and wideband (16 kHz) speech. Modern generative models output 24 kHz or 48 kHz full-band audio, exceeding the mathematical assumptions of ITU-T P.862.

According to perceptual speech testing research across ITU-T benchmarks, intrusive metrics show near-zero correlation with human subjective ratings when applied to modern neural speech synthesizers. Treating PESQ as a quality gate for synthetic voice pipelines leads either to rejecting flawless audio or ignoring genuine hallucinations.

What is the difference between intrusive and non-intrusive speech metrics?

The fundamental architectural distinction in speech evaluation is whether the scoring algorithm requires clean reference audio. Intrusive metrics require paired ground-truth audio, whereas non-intrusive (no-reference) metrics evaluate speech directly from the output waveform.

MetricStandardization / FrameworkReference RequiredTarget MeasurementPrimary Use Case
PESQITU-T P.862Yes (Intrusive)Codec & channel degradationTelephony & VoIP testing
POLQAITU-T P.863Yes (Intrusive)HD voice & network delayCellular network benching
DNSMOSITU-T P.835 / MicrosoftNo (Non-intrusive)Speech (SIG), Noise (BAK), Overall (OVRL)Real-time audio filtering & TTS QA
UTMOSUTokyo-SaruLab / InterspeechNo (Non-intrusive)Synthetic speech naturalness MOSTTS model evaluation & ranking
Subjective MOSITU-T P.808No (Listening Test)Human perceptual Mean Opinion ScoreAcademic validation & ground truth

For automated text-to-speech pipelines, non-intrusive neural models represent the only viable path for scoring 100% of production volume in real time.

How do DNSMOS and UTMOS score synthetic audio?

Non-intrusive evaluation models use neural networks trained on vast datasets of human listening test scores collected under standardized protocols like ITU-T P.808. Instead of comparing samples to an original track, they extract learned spectral and acoustic features to predict what human listeners would rate the clip on a 1.0 to 5.0 Mean Opinion Score (MOS) scale.

Microsoft DNSMOS: Multi-dimensional perceptual evaluation

DNSMOS (Deep Noise Suppression Mean Opinion Score) was developed by Microsoft to evaluate speech enhancement and audio processing without subjective listening panels. Standardized in alignment with ITU-T P.835, DNSMOS outputs three distinct scores between 1.0 and 5.0:

  • SIG (Speech Quality): Measures the clarity, naturalness, and distortion of the speech signal itself.
  • BAK (Background Noise): Measures the intrusiveness and presence of background hiss, hum, or environmental noise.
  • OVRL (Overall Quality): A composite score reflecting overall audio acceptability.

In automated TTS quality pipelines, DNSMOS detects vocoder hiss, robotic buzzes, and clipping artifacts. If a generated clip returns an OVRL score above 4.2 but its BAK score drops below 3.5, the pipeline detects background acoustic noise introduced by a misconfigured post-processing filter.

UTMOS: Dedicated neural scoring for synthetic speech

UTMOS, developed by researchers at the University of Tokyo for the VoiceMOS Challenge, is explicitly trained to score synthetic speech generated by neural TTS systems. Unlike general speech codecs, neural TTS models introduce unique synthesis errors: phoneme stretching, unnatural intonation flatlining, and metallic phase artifacts.

UTMOS processes raw audio through self-supervised speech representations (such as wav2vec 2.0) to output a single predicted MOS score. In production benchmarks, UTMOS achieves a Pearson correlation exceeding 0.85 with human crowd-sourced evaluations conducted under ITU-T P.808 protocols. Running UTMOS on every generated output allows developers to set quantitative thresholds, such as flagging any clip scoring below 3.6 for automated regeneration.

How to build an automated TTS audio quality gate in production

Listening to 50 audio clips during development does not protect a production system that generates 10,000 clips per day. In large-scale speech generation, a 1% defect rate means 100 defective files reach users daily. An automated quality gate replaces manual spot-checks with deterministic scoring filters.

A robust production evaluation loop executes three sequential verification stages:

1. Structural and format compliance

Before neural scoring runs, the pipeline validates baseline audio container properties:

  • Sample rate consistency: Verifies whether the output matches target specifications (e.g., 8 kHz G.711 for telephony, 24 kHz PCM for web playback).
  • Loudness normalization: Confirms audio meets target integrated loudness (such as -16 LUFS for podcasts or -24 LUFS for broadcast) and checks that true peaks do not exceed -1.0 dBTP.
  • Leading and trailing silence: Detects missing silence padding that causes downstream media players to cut off opening consonants.

2. Pronunciation and phoneme validation

Neural MOS models measure acoustic smoothness, not factual pronunciation correctness. A model can synthesize the word "Onepin" with pristine clarity while mispronouncing it as three separate disconnected syllables. The automated quality gate must run forced alignment or phoneme recognition against a dedicated pronunciation glossary to confirm proper nouns and brand names are articulated accurately.

3. Perceptual thresholding and automated retry

Every clip passes through non-intrusive neural scoring:

  • Pass condition: UTMOS $\ge 3.8$ and DNSMOS OVRL $\ge 4.0$. The clip ships directly to the delivery CDN or voice agent runtime.
  • Fail condition: Quality scores fall below threshold. The pipeline triggers an automated retry using modified temperature settings, a secondary voice ID, or a fallback TTS provider.
  • Drift alerting: Batch-level averages are monitored over time. If the moving average score for a specific voice drops by more than 0.25 points following a provider model update, engineers receive an automated regression alert.

How Onepin automates audio quality validation across 100+ TTS models

Building, calibrating, and maintaining non-intrusive neural evaluation models alongside dynamic pronunciation dictionaries requires significant engineering overhead.

Onepin is a voice workflow platform that orchestrates, validates, and ships production-ready audio across 100+ TTS models. Instead of forcing teams to build internal scoring pipelines or risk shipping unreviewed speech, Onepin validates every generated file automatically:

  • Unified quality gating: Every audio clip is scored for naturalness, loudness compliance, and acoustic cleanliness before delivery.
  • Cross-model pronunciation verification: Enforces custom lexicons across providers, ensuring proper nouns and technical terms sound identical whether synthesized on ElevenLabs, Cartesia, or OpenAI.
  • Zero single-vendor lock-in: When a provider suffers latency spikes or model degradation, Onepin seamlessly reroutes traffic to a verified backup without downtime.

If your team currently relies on human ears or outdated audio metrics to catch synthetic speech defects, explore onepin.ai/docs to deploy automated voice validation at scale.

Frequently asked questions

What is the difference between PESQ and UTMOS for TTS quality evaluation?
PESQ (ITU-T P.862) is an intrusive metric that requires time-aligned reference speech and was designed for telecommunication codecs, making it fail on generative TTS where no reference audio exists. UTMOS is a non-intrusive neural Mean Opinion Score predictor that evaluates synthetic speech naturalness directly from the raw audio waveform without needing reference audio.
Why does PESQ fail on generative text-to-speech models?
Generative TTS synthesizes novel speech variations, differing pitch contours, and unique pacing that do not match a reference waveform sample by sample. Intrusive algorithms penalize natural acoustic variability as distortion, producing low PESQ scores even when human listeners rate the generated audio as highly natural.
What metrics does DNSMOS measure in synthetic speech?
DNSMOS is a deep learning model standardized under ITU-T P.835 that predicts three subjective quality dimensions: speech quality (SIG), background noise intrusiveness (BAK), and overall perceptual quality (OVRL). It allows automated audio pipelines to separate voice clarity from acoustic noise without human listening tests.
How does automated TTS quality scoring prevent production failures?
Automated quality scoring evaluates every generated clip against quantitative baseline thresholds for naturalness and clarity before delivery. Clips falling below threshold are automatically flagged, retried with alternative voice configurations, or routed to fallback models before reaching end users.
How does Onepin handle audio quality validation across multiple TTS models?
Onepin is a voice workflow platform that orchestrates, validates, and ships production-ready audio across 100+ TTS models. It automatically scores every generated clip against delivery baselines, detects acoustic regressions across model updates, and routes audio to verified providers so production systems never ship degraded audio.

Ready to publish?

Turn any script into production-quality voice,
in any language, in minutes.

Run your first line