CMU · Language Technologies Institute

Zhuoyan Terry Tao

M.S. student at CMU's Language Technologies Institute and researcher at WavLab, building models that perceive and repair speech: target-speaker restoration, speech quality from partial audio, and multilingual speech. Drawn to problems where signal meets meaning. ✌︎

  • Interspeech 2026 oral · first author
  • Amazon AGI collaboration
  • ESPnet & VERSA contributor
  • Apple SWE intern ’26
Open to Summer 2027 internships
Curriculum Vitae ↗︎ Publications
Terry at Kualoa Ranch, Oahu
Kualoa Ranch · Oahu, Hawaii
01

About

Speech Processing · Audio ML · NLP

I'm Zhuoyan Tao, Terry to friends and colleagues. I'm a master's student in CMU's Intelligent Information Systems program at the Language Technologies Institute, advised by Prof. Shinji Watanabe at WavLab.

My work sits at the boundary of signal processing, machine learning, and linguistics, and keeps circling one question: what listeners take from speech beyond the words, and how to measure and recover it when the signal is partial, degraded, or switching between languages. Right now I'm the lead student researcher on target-speaker speech restoration with Amazon AGI.

Before CMU I finished dual B.S. degrees at USC, where I worked on multilingual speech with Prof. Shrikanth Narayanan at SAIL, and spent summer 2026 at Apple building simulation tooling for satellite connectivity. Outside of research, I enjoy exploring new places ☀︎

Education Carnegie Mellon University M.S. Intelligent Information Systems Language Technologies Institute Expected Dec 2027 University of Southern California B.S. Computer Science B.S. Applied and Computational Mathematics GPA 3.84 · Magna Cum Laude · May 2026
Advisors Prof. Shinji Watanabe CMU WavLab · 2025–Present Prof. Shrikanth Narayanan USC SAIL · 2023–2026
Honors Phi Beta Kappa Elected 2025 Trustee Scholar Full-tuition merit scholarship, USC Grand Challenges Scholar NAE · USC Viterbi, 2026
Publishes as Zhuoyan Tao · Google Scholar
02

News

  • New preprint with Aditya Kommineni and Prof. Shrikanth Narayanan: Off-target posterior mass, a training-free diagnostic for fine-tuning spoken language diarization probes. Submitted to JASA Express Letters.

  • The Sidon speech-restoration recipe was merged into ESPnet (#6636; I wrote 41 of its 52 commits).

  • ANCHOR was presented as an oral at Interspeech 2026 in Sydney.

  • TurnBench, a multi-domain benchmark for turn-taking in spoken dialogue, was accepted to IEEE SLT 2026 (Palermo, December 2026).

  • Started the M.S. in Intelligent Information Systems at CMU LTI, after a summer as a software engineering intern at Apple.

Earlier news

  • First-author paper ANCHOR accepted to Interspeech 2026.

  • Graduated from USC with B.S. degrees in Computer Science and Applied & Computational Mathematics, Magna Cum Laude, and was named a Grand Challenges Scholar.

03

Research

Current focus · CMU WavLab × Amazon AGI

Target-Speaker Speech Restoration

Two people on a damaged phone call: recover just one of them, cleanly, given a few seconds of their voice. It is a generative problem with an unforgiving failure mode, restoring the wrong speaker.

  • Lead student researcher; models trained on 7.8K two-party calls.
  • Built the evaluation suite: word error rate (WER, and permutation-invariant cpWER for multi-speaker audio), speaker similarity, wrong-speaker rate, VERSA quality metrics, and paired-bootstrap significance tests. Benchmarked 9 external systems.
  • Made encoder training 1.20× faster with 26% less memory by moving to fused scaled-dot-product attention (SDPA) with a custom backward pass for the position bias.
  • Generative restoration
  • Speaker conditioning
  • Evaluation
  • PyTorch · ESPnet
CMU WavLab

Perceptual Speech Quality

Predicting how listeners would rate speech from partial audio, with no clean reference, for streaming and generative systems. ANCHOR cut PLCMOS error by 21–35% on 2-second prefixes and located a 4–6 s perceptual context horizon. Since then: a ~20× faster one-pass variant at equal accuracy, a zero-shot A/B judge within 0.01 Spearman correlation of the top VoiceMOS 2026 entry on URGENT 2026 pairs, and best-of-N TTS sample selection from the first 2 seconds.

ANCHOR · Interspeech 2026 oral → Live demo ↗︎
USC SAIL Lab

Multilingual & Code-Switched Speech

Spoken language diarization labels which language is spoken when, which code-switching conversation needs. I study what pre-trained encoders (Whisper, MMS) already know about language identity, and built off-target posterior mass, a training-free diagnostic that forecasts when fine-tuning a lightweight probe will pay off.

Preprint · submitted to JASA Express Letters →
CMU WavLab · Challenge system

Prosody, Stance & Pragmatic Intent

An English–Spanish speech-to-speech translation system for the Interspeech 2026 S2ST Challenge, built on HuBERT units and prosody features so the translation carries stance, emotion, and dialog function, not only the words.

Open Source · ESPnet & VERSA

Open-Source Speech Tooling

In ESPnet (10k+ stars): the Sidon speech-restoration recipe (41 of 52 commits: GPU-side self-supervised feature extraction, vocoder stages, XEUS backbone) and parallel speaker-embedding extraction for TTS, a step that took 40+ hours, now >20× faster. In VERSA, WavLab's evaluation toolkit: a permutation-invariant multi-source metric fix and Slurm launcher safeguards.

Sidon #6636 ↗︎ Speedup #6227 ↗︎ ESPnet PRs ↗︎ VERSA PRs ↗︎
04

Publications

Author names in order; my name in bold. Also on Google Scholar.

Peer-reviewed

ANCHOR: Autoregressive Non-intrusive Chunk-Ordered Refinement for Joint Multi-Resolution Speech Quality Modeling

Zhuoyan Tao, Jiatong Shi, Hye-jin Shim, Shinji Watanabe
Proc. Interspeech 2026, pp. 5672–5677

Predicts how listeners will rate speech before the utterance finishes, with 21–35% lower PLCMOS error than ARECHO on 2-second prefixes.

Streaming and generative systems need quality estimates from partial audio, but existing predictors assume the full utterance. Extending ARECHO (NeurIPS 2025 Spotlight), ANCHOR models chunk- and utterance-level quality in one autoregressive decoder with dual-resolution query tokens, and shows that judgments converge after a 4–6 s perceptual context horizon.

Correction The published version reports 48%. We later found a resampling bug in the toolkit that labelled the baseline's PLCMOS training data; with corrected labels the gain is 21–35%. UTMOS, DNSMOS, and NISQA results and the 4–6 s horizon are unaffected. Erratum in preparation.

Interspeech 2026 · Oral First author PDF ↗︎ DOI ↗︎ arXiv ↗︎ Demo ↗︎

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Freeman Jiang, Ramon Sanabria, Soham Deshmukh, …, Zhuoyan Tao, …, Hung-yi Lee, Shinji Watanabe (19 authors)
Proc. IEEE Spoken Language Technology Workshop (SLT) 2026

No current turn-taking system matches humans, who start speaking a median 151 ms before a turn ends, without excessive false positives.

Thirty hours of hand-labeled, triple-annotated dyadic conversation across six interaction styles, a standardized protocol for end-of-turn and interruption detection, 14 benchmarked systems, and a released 104-hour training set and public leaderboard.

My contribution Helped design the evaluation protocol and submitted an ANCHOR-based model as an initial baseline.

IEEE SLT 2026 PDF ↗︎ arXiv ↗︎ Leaderboard ↗︎

Under review

Off-target posterior mass: A training-free diagnostic for fine-tuning spoken language diarization probes

Zhuoyan Tao, Aditya Kommineni, Shrikanth Narayanan
Submitted to JASA Express Letters · arXiv:2611.13430

A single training-free number, computed from language-ID posteriors, that forecasts whether fine-tuning a diarization probe will pay off.

Off-target posterior mass (OTPM) correlates with probe gains at 0.56 on unseen synthetic bilingual data and 0.23 on real-world corpora (DISPLACE; MUCS Hindi–English code-switching). The evaluation was pre-registered on unseen language pairs and run across Whisper, MMS, and classical language-ID encoders.

JASA-EL · Under review First author PDF ↗︎ arXiv ↗︎
05

Experience

Industry

May – Aug 2026

Apple

Software Engineering Intern · Satellite Connectivity · Austin, TX
  • Sole author of a sweep-orchestration tool for the team's satellite network simulator, scaling throughput ~1000× to ~2000 concurrent experiments on an internal compute cluster; presented org-wide.
  • Used it to run analysis the team couldn't before: swept projected demand over years of growth to model how capacity and performance evolve, with Gaussian-process sensitivity analysis ranking the most influential parameters.
  • ~12× speedup on the simulator's interference-computation hot path via batched vectorization; unified random-number generation and the simulation clock for reproducibility.
  • Root-caused a Python/Go clock-scheduler race that silently skipped ~half of scheduling cycles and under-reported capacity by up to ~2× under load.
  • Python
  • Go
  • Gaussian processes
  • Vectorization
  • Cluster computing

Research

Jun 2025 – Present

CMU WavLab

Researcher · Speech & Audio ML · Advisor: Prof. Shinji Watanabe · Pittsburgh, PA
  • Lead student researcher on target-speaker speech restoration with Amazon AGI (details).
  • First-author Interspeech 2026 oral on speech-quality prediction from partial audio; English–Spanish system for the Interspeech 2026 S2ST Challenge.
  • Merged ESPnet and VERSA contributions, including the Sidon restoration recipe and a >20× speaker-embedding speedup.
  • PyTorch
  • ESPnet
  • VERSA
  • HuBERT · WavLM
  • SLURM
Aug 2023 – May 2026

USC SAIL Lab

Student Researcher · Multilingual Speech · Advisor: Prof. Shrikanth Narayanan · Los Angeles, CA
  • Built off-target posterior mass, a training-free diagnostic for when fine-tuning spoken language diarization probes pays off; first-author paper submitted to JASA Express Letters.
  • Pre-registered the evaluation on unseen language pairs and tested it on two held-out real-world corpora; isolated which utterances gain most in low-resource and code-switching settings.

Teaching & projects

Aug 2023 – May 2026

USC MATH 229 · Multivariable Calculus

Supplemental Instruction Leader · 3–4 weekly sessions for 98 students; review sessions reached ~300 attendees, the highest in program history

Toolkit

Speech & ML
PyTorch · torchaudio · Hugging Face Transformers · ESPnet · VERSA · LoRA
Models
Whisper · MMS · WavLM · HuBERT · w2v-BERT
Systems
Python · C++ · Go · Bash · Docker · Linux · SLURM · Git
Compute
Distributed training (DDP) · profiling (py-spy, cProfile) · vectorization · NCSA Delta & PSC Bridges-2 GPU clusters
Data
NumPy · pandas · Matplotlib · Gaussian-process regression
06

Contact

Open to Summer 2027 internships in speech & audio ML, as a research scientist or ML engineering intern. Email is the fastest way to reach me, and my CV has the details. Research collaborators are always welcome too.

Email me
Email
terrytao@andrew.cmu.edu
Google Scholar
scholar.google.com
GitHub
github.com/ZhuoyanTao
LinkedIn
linkedin.com/in/zhuoyan-tao
Curriculum Vitae
PDF · one page