arXiv 2607.28175 · 2026 AGENTIC SPEECH RECOGNITION

AgenticASRRefining Speech Recognition in Real-World Scenarios via an Agentic Approach

Turn spoken-form ASR output into clean, intent-preserving text, and revise earlier output when later speech changes its meaning.

Zixuan Jiang*, Binghao Qiang*, Jiaying Chi*, Yanqiao Zhu, Kai Yu, Xie Chen†
* Equal contribution · † Corresponding author

Paper Code AASR-Bench App

Audio

BilingualEnglish + Chinese

ASR-agnosticWorks with any ASR that returns text

RevisableOnline or offline refinement

01 / ABSTRACT

Abstract

Automatic speech recognition (ASR) has achieved substantial gains in transcription accuracy, yet verbatim transcription does not necessarily produce readily usable text. It retains fillers, repetitions, false starts, and self-corrections that increase reading effort, obscure the speaker’s final intent, and propagate unresolved or abandoned content to downstream tasks. Existing spoken-to-written methods process completed audio or transcripts but cannot revise emitted text when later speech changes how preceding content should be interpreted. We therefore formulate Agentic Speech Recognition (AgenticSR), an audio-to-clean-text task that removes disfluencies, resolves self-corrections, and normalizes written form while preserving the speaker’s final intent. AgenticASR implements this task through an ASR–Refiner architecture that repeatedly transforms a bounded active context and replaces its corresponding output span as audio arrives. This enables continual emission and revision over streams of arbitrary duration. We also introduce AASR-Bench, a bilingual benchmark with fine-grained atomic rubrics. Across multiple ASR front ends, AgenticASR attains the highest AASR-Bench scores among evaluated systems. A human–AI agreement study shows that rubric-based judgments align with independent expert assessments. Ablations characterize Refiner capacity, context length, and the quality–latency trade-off between online and offline inference. Together, these results establish AgenticASR as a practical framework for intent-preserving clean transcription during ongoing speech.

02 / TASK

From verbatim speech to usable text.

Speech contains false starts, fillers, repetitions, and self-corrections. AgenticSR resolves them while keeping the speaker’s final intent, so the output can be read and used downstream.

AgenticASR transforms speech into clean written text — AgenticASR produces clean, intent-preserving text rather than a verbatim transcript.

Emit

As the user speaks, the ASR front end produces a provisional transcript.

Refine

A compact language-model Refiner converts the active context from oral to written form.

Revise

When new audio arrives, only the affected part of the output is replaced, so an earlier guess can be corrected in place.

03 / DEMO

One system, two languages.

Both videos show the same behavior: spoken input becomes readable text, and earlier output is revised when later speech changes its meaning.

EN / English 00:34

English streaming example: disfluencies and incomplete phrasing are rewritten into clean text.

ZH / 中文 00:50

中文演示：系统在保留最终意图的同时，持续清理口语表达并更新局部结果。

04 / METHOD

How AgenticASR handles streaming speech.

The Refiner is kept separate from acoustic recognition, so the same text-to-text correction model can work with different ASR backbones.

AgenticASR data pipeline and online inference method — Method overview: the five-stage data pipeline creates Oral/Clean training pairs; online inference uses VAD and a sliding window with default K=3.

Train the transformation

The data pipeline generates Seed, Oral, and Clean text, then runs ASR simulation, semantic quality control, and global deduplication.

Bound the latency

For streaming input, the Refiner processes a local window of K chunks and replaces the aligned output span instead of waiting for the whole utterance.

Measure what WER misses

AASR-Bench scores Content, Format, Filter, and Rephrase separately, so transcription quality is not reduced to token error.

05 / RESULTS

AgenticASR sets the highest Overall score on AASR-Bench.

AgenticASR has the highest Overall score among the Qwen3-ASR systems and improves every Whisper configuration over its API baseline.

79.95Overall · Qwen3-ASR-1.7B

+27.45vs. FormalASR-1.7B

6,637atomic bilingual rubrics

0.82human–AI Spearman agreement

Scene-level Overall scores for Qwen3-ASR systems — Scene-level Overall scores across ten usage scenes and a pass-through control. AgenticASR with the 1.7B frontend leads both baselines in every scene.

What the table shows

With Qwen3-ASR-1.7B, AgenticASR reaches 79.95 Overall and ranks first on all four rubric dimensions. Across matched ASR backbones, it improves on the API transformation baseline by 1.73–10.02 points with substantially lower latency.

With Whisper, the gain over the API baseline grows from 1.73 points at Base to 7.39 points at Large. The strongest improvements come from filtering and final-intent rephrasing.

Traditional token metrics remain useful diagnostics, but AASR-Bench reveals formatting, filtering, and correction-resolution failures that WER, CER, and MER cannot capture.

Table 3 · Main results on AASR-Bench. Higher is better except WER/CER/MER and latency.
ASR model	LM	Content	Format	Filter	Rephrase	WER/CER/MER ↓	Latency ↓	Overall
Qwen3-ASR-0.6B	Qwen3.5-Flash	87.50	28.97	73.13	49.13	26.82/17.01/21.91	60.08	66.47
FormalASR-0.6B	–	86.63	14.35	36.51	13.29	38.67/28.34/34.38	3.42	48.76
Qwen3-ASR-0.6B	AgenticASR	87.30	54.94	78.80	69.16	14.64/7.79/10.23	6.60	76.15
Qwen3-ASR-1.7B	Qwen3.5-Flash	90.21	35.48	75.82	52.10	24.60/15.72/20.29	60.89	69.93
FormalASR-1.7B	–	90.11	19.69	40.59	15.70	34.07/24.47/30.48	3.46	52.50
Qwen3-ASR-1.7B	AgenticASR	90.24	65.19	78.89	72.83	12.70/6.86/9.01	9.59	79.95
Whisper Base	Gemini-2.5-Flash	47.04	6.09	62.63	16.67	53.14/39.83/46.41	12.00	37.09
Whisper Base	AgenticASR	38.69	6.95	71.96	32.47	55.62/41.33/46.70	5.86	38.82
Whisper Small	Gemini-2.5-Flash	58.79	29.04	65.08	27.94	56.98/43.74/45.78	11.99	48.78
Whisper Small	AgenticASR	52.58	29.57	72.79	47.40	58.09/44.09/44.28	6.89	51.72
Whisper Large	Gemini-2.5-Flash	80.23	51.58	63.10	36.13	31.50/21.45/25.33	8.04	62.90
Whisper Large	AgenticASR	76.16	55.87	77.75	63.01	27.51/18.19/19.63	4.42	70.29

The best value in each ASR family is shown in bold in the paper. The LM column lists the downstream transformation system; FormalASR performs direct speech-to-clean-text recognition.

06 / ABLATIONS

What the ablations show.

Two effects stand out: larger Refiners rewrite with more context, and a three-chunk online window recovers most of the useful context on the right.

Effect of active window size on online revision — Window size K=3 keeps enough local context to revise a destination across VAD boundaries.

Online window

Moving from K=1 to K=3 raises Rephrase from 36.17 to 70.47 and Explanation from 19.43 to 74.00, while latency grows by only 0.87 s. K=3 closes the gap to offline inference to 2.36 Rephrase points and 1.20 Explanation points.

This lets AgenticASR revise an earlier destination when a later chunk contains the speaker’s correction.

Table 4 · Human agreement
Measure	0.6B	1.7B
Spearman ρ	0.8222	0.8064
Quadratic-weighted κ	0.8313	0.7918

ABLATION A

Human agreement

Double-blind experts and the Gemma-4-31B-IT judge show strong agreement across 100 sampled utterances.

The mean Spearman correlation is 0.8222 for Qwen3-ASR-0.6B and 0.8064 for Qwen3-ASR-1.7B; quadratic-weighted agreement is 0.8313 and 0.7918.

ABLATION B

Refiner capacity

With Qwen3-ASR-1.7B fixed, scaling the Refiner from 0.5B to 4B raises Overall by 4.66 points; the largest gains are in Format and Rephrase.

Overall rises from 78.76 to 83.42, while latency increases from 9.21 to 10.77 s. Larger Refiners are a better fit for offline use where latency is less critical.

Table 5 · Refiner capacity
Refiner	Overall	Cont.	Fmt.	Filt.	Reph.	Lat. (s)
Qwen2.5-0.5B-Instruct	78.76	88.00	63.40	78.36	69.85	9.21
MiniCPM-5-1B	79.95	90.24	65.19	78.89	72.83	9.59
Qwen2.5-4B-Instruct	83.42	91.00	74.43	83.31	75.68	10.77

Table 6 · Offline and online AgenticASR with Qwen3-ASR-1.7B
Setting	Rephrase ↑	Latency (s) ↓	Explanation ↑
Offline	72.83	9.59	75.20
Window = 1	36.17	11.28	19.43
Window = 2	65.08	11.70	55.06
Window = 3	70.47	12.15	74.00

ABLATION C

Online window

Moving from K=1 to K=3 raises Rephrase from 36.17 to 70.47 and Explanation from 19.43 to 74.00, while latency grows by only 0.87 s.

At K=3, the gaps to offline shrink to 2.36 Rephrase points and 1.20 Explanation points, recovering nearly all useful right context for online revision.

07 / BENCHMARK

What counts as clean text?

AASR-Bench is a bilingual benchmark built on fine-grained rubrics. Each sample is scored on the transformations it actually requires, not on a single overall text metric.

917 samples

6,637 total atomic rubrics

Each sample has at least one Content question. Format, Filter, and Rephrase rubrics are added when those phenomena are present. The benchmark covers ten usage scenes plus a pass-through control.

Explore AASR-Bench

Table 1 · Distribution of atomic rubrics
Category	Questions	Share (%)	Coverage
Content	3,448	51.95	917
Format	1,498	22.57	741
Filter	882	13.29	882
Rephrase	809	12.19	623
Total	6,637	100.00	–

08 / CITE

Cite AgenticASR.

If this project is useful, please cite the paper.

@misc{jiang2026agenticasrrefiningspeechrecognition,
      title={AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach},
      author={Zixuan Jiang and Binghao Qiang and Jiaying Chi and Yanqiao Zhu and Kai Yu and Xie Chen},
      year={2026},
      eprint={2607.28175},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.28175},
}

arXiv abstract Repository

Page structure inspired by the Academic Project Page Template and Nerfies; visual language adapted for AgenticASR.