CMU · Language Technologies Institute

Zhuoyan Tao (Terry)

I'm Terry, a master's student at CMU working with Prof. Shinji Watanabe at WavLab. I study speech restoration and speech quality: how to recover a speaker's voice from damaged audio, and how to tell whether it sounds right.

Seeking Summer 2027 research or ML engineering internships in speech and audio
Explore research Résumé ↗︎
Portrait of Zhuoyan (Terry) Tao

Current research

CMU WavLab × Amazon AGI · Lead student researcher

Target-Speaker Speech Restoration

Recovering one speaker from a damaged two-person phone call using a short voice sample. Trained models on 7.8K calls and built an evaluation suite benchmarking nine external systems.

CMU WavLab · Interspeech 2026 oral

Perceptual Speech Quality

I developed ANCHOR to predict speech quality before an utterance finishes. It reduces PLCMOS error by 21–35% on two-second audio prefixes.

Paper → Try the demo ↗︎

Open-source contributions

Publications

Author names in order; my name in bold. Also on Google Scholar.

Peer-reviewed

ANCHOR: Autoregressive Non-intrusive Chunk-Ordered Refinement for Joint Multi-Resolution Speech Quality Modeling

Zhuoyan Tao, Jiatong Shi, Hye-jin Shim, Shinji Watanabe
Proc. Interspeech 2026, pp. 5672–5677

Predicts how listeners will rate speech before the utterance finishes, with 21–35% lower PLCMOS error than ARECHO on 2-second prefixes.

Details

Streaming and generative systems need quality estimates from partial audio, but existing predictors assume the full utterance. Extending ARECHO (NeurIPS 2025 Spotlight), ANCHOR models chunk- and utterance-level quality in one autoregressive decoder with dual-resolution query tokens, and shows that judgments converge after a 4–6 s perceptual context horizon.

Interspeech 2026 · Oral First author PDF ↗︎ DOI ↗︎ arXiv ↗︎ Try the demo ↗︎

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Freeman Jiang, Ramon Sanabria, Soham Deshmukh, …, Zhuoyan Tao, …, Hung-yi Lee, Shinji Watanabe (19 authors)
Proc. IEEE Spoken Language Technology Workshop (SLT) 2026

No current turn-taking system matches humans, who start speaking a median 151 ms before a turn ends, without excessive false positives.

Details

Thirty hours of hand-labeled, triple-annotated dyadic conversation across six interaction styles, a standardized protocol for end-of-turn and interruption detection, 14 benchmarked systems, and a released 104-hour training set and public leaderboard.

My contribution Helped design the evaluation protocol and submitted an ANCHOR-based model as an initial baseline.

IEEE SLT 2026 PDF ↗︎ arXiv ↗︎ Leaderboard ↗︎

Under review

Off-target posterior mass: A training-free diagnostic for fine-tuning spoken language diarization probes

Zhuoyan Tao, Aditya Kommineni, Shrikanth Narayanan
Submitted to JASA Express Letters · arXiv:2611.13430

A single training-free number, computed from language-ID posteriors, that forecasts whether fine-tuning a diarization probe will pay off.

Details

Off-target posterior mass (OTPM) correlates with probe gains at 0.56 on unseen synthetic bilingual data and 0.23 on real-world corpora (DISPLACE; MUCS Hindi–English code-switching). The evaluation was pre-registered on unseen language pairs and run across Whisper, MMS, and classical language-ID encoders.

JASA-EL · Under review First author PDF ↗︎ arXiv ↗︎

Experience

Research positions

Jun 2025 – Present

CMU WavLab

Researcher · Speech & Audio ML · Advisor: Prof. Shinji Watanabe · Pittsburgh, PA
  • Lead student researcher on target-speaker speech restoration with Amazon AGI (project overview).
  • First-author Interspeech 2026 oral on speech-quality prediction from partial audio; English–Spanish system for the Interspeech 2026 S2ST Challenge.
  • Merged ESPnet and VERSA contributions, including the Sidon restoration recipe and a >20× speaker-embedding speedup.
Aug 2023 – May 2026

USC SAIL Lab

Student Researcher · Multilingual Speech · Advisor: Prof. Shrikanth Narayanan · Los Angeles, CA
  • Built off-target posterior mass, a training-free diagnostic for when fine-tuning spoken language diarization probes pays off; first-author paper submitted to JASA Express Letters.
  • Pre-registered the evaluation on unseen language pairs and tested it on two held-out real-world corpora; isolated which utterances gain most in low-resource and code-switching settings.

Industry

May – Aug 2026

Apple

Software Engineering Intern · Satellite Connectivity · Austin, TX
  • Built a tool to run satellite-network simulations at scale, increasing experiment throughput ~1000× and supporting ~2000 concurrent experiments.
  • Sped up interference calculations ~12× and fixed timing issues that caused the simulator to under-report network capacity.

Teaching & projects

Aug 2023 – May 2026

USC MATH 229 · Multivariable Calculus

Supplemental Instruction Leader · 3–4 weekly sessions for 98 students; review sessions reached ~300 attendees, the highest in program history

About

Before CMU, I studied computer science and applied mathematics at USC, where I worked on multilingual speech with Prof. Shrikanth Narayanan at SAIL. I spent summer 2026 at Apple building simulation tools for satellite connectivity.

I also enjoy making games and working with audio. At USC, I led the audio team for Bone Boy, a student-built game released on Steam, and taught multivariable calculus review sessions.

Education Carnegie Mellon University M.S. Intelligent Information Systems Language Technologies Institute Expected Dec 2027 University of Southern California B.S. Computer Science B.S. Applied and Computational Mathematics GPA 3.84 · Magna Cum Laude · May 2026
Advisors Prof. Shinji Watanabe CMU WavLab · 2025–Present Prof. Shrikanth Narayanan USC SAIL · 2023–2026
Publishes as Zhuoyan Tao · Google Scholar

Honors

News

  • New preprint with Aditya Kommineni and Prof. Shrikanth Narayanan: Off-target posterior mass, a training-free diagnostic for fine-tuning spoken language diarization probes. Submitted to JASA Express Letters.

  • The Sidon speech-restoration recipe was merged into ESPnet (#6636; I wrote 41 of its 52 commits).

  • ANCHOR was presented as an oral at Interspeech 2026 in Sydney.

  • TurnBench, a multi-domain benchmark for turn-taking in spoken dialogue, was accepted to IEEE SLT 2026 (Palermo, December 2026).

  • Started the M.S. in Intelligent Information Systems at CMU LTI, after a summer as a software engineering intern at Apple.

Earlier news

  • First-author paper ANCHOR accepted to Interspeech 2026.

  • Graduated from USC with B.S. degrees in Computer Science and Applied & Computational Mathematics, Magna Cum Laude, and was named a Grand Challenges Scholar.

Contact

I'm seeking Summer 2027 research or ML engineering internships in speech and audio. Reach me at terrytao@andrew.cmu.edu.

Interested in collaborating on speech restoration, perceptual evaluation, or multilingual speech? I'd be glad to hear from you.