💡 ASR tools (Whisper, YouTube) produce a usable first draft - especially for clean, single-speaker English audio - but fail in predictable, measurable ways: proper nouns guessed phonetically, overlapping speakers dropped, sound cues omitted, speaker IDs absent. Word Error Rate (WER) = (S + D + I) / N × 100%. Two name errors in a 12-word segment = WER 16.7%, which does not pass the FCC accuracy benchmark without a human correction pass.
Key takeaways
- WER formula: WER = (Substitutions + Deletions + Insertions) / Reference word count × 100%. A substitution on a proper noun scores the same as any word error but carries a higher meaning penalty.
- ASR fails most on: proper nouns (phonetic guesses), overlapping speech (one speaker dropped), non-speech audio (sound cues omitted), speaker IDs (absent by default), and low-resource languages.
- The FCC accuracy benchmark (47 CFR § 79.1) requires captions to match spoken words verbatim, in order, without paraphrasing, including speaker identification - none of which Whisper provides out of the box.
- Auto-captions are a time-saver, not a finished deliverable. The break-even is a human correction pass in Subtitle Edit: 1-2 minutes of correction per minute of clean-audio video.
- Broadcast delivery (SCC for US TV, SMPTE-TT for streaming) requires a human QC pass regardless - FCC accuracy includes speaker IDs and sound cues, neither of which ASR provides.
What does "auto-caption" actually produce, technically?
Auto-captioning is Automatic Speech Recognition (ASR): the tool converts the acoustic signal to a word lattice, picks the most probable word sequence, and writes timed text segments. YouTube uses Google's proprietary ASR. OpenAI Whisper is the leading open-source alternative, trained on 680,000 hours of multilingual audio across 99 languages, including Vietnamese. Neither tool adds speaker identification, sound effect notation, or music cues - those are human decisions.
The output is a raw SRT or VTT file: dialogue only, no [MUSIC], no [LAUGHTER], no NARRATOR: labels. For a documentary or a panel discussion, that is already a significant gap from what broadcast captioning requires. The primary keyword for every frame of this workflow is auto captions vs human captioning, and the distinction starts right here: ASR gives you words, human captioning gives you a complete, standards-compliant record.
Where does ASR fail? The error-type table
I keep this table in my workflow notes because it tells me how much correction time to budget before I open the file:
| Error type | ASR behavior | Example | FCC benchmark hit |
|---|---|---|---|
| Proper nouns | Phonetic guess from the vocabulary | "Dr. Sarah Chen" → "Dr. Sara Chin" | Accuracy |
| Overlapping speech | Dominant speaker wins; second speaker dropped or garbled | Interviewer's interjection missing | Completeness |
| Non-speech audio | Omitted entirely by default | [EXPLOSION] absent from transcript | Accuracy (IP standard requires sound IDs) |
| Speaker identification | No labels; all dialogue presented as one speaker | Panel discussion reads as monologue | Accuracy (IP: "must identify speakers") |
| Numbers and dates | Inconsistent numeral vs. word form | "Q4 2026" → "queue for 2026" | Accuracy |
| Punctuation | Comma and period placement varies; question marks often missing | Run-on sentences across cue boundaries | Accuracy (required verbatim by FCC) |
| Accented or low-resource audio | Higher substitution rate; words skipped on difficult segments | Non-native speaker in English interview | Accuracy + Completeness |
| Technical terms / acronyms | Spelled phonetically or dropped | "CEA-708" → "seye 708" | Accuracy |
How do I calculate WER on a sample?
Word Error Rate is the standard metric. The formula:
WER = (S + D + I) / N × 100%
S = Substitutions (wrong word)
D = Deletions (reference word missing from hypothesis)
I = Insertions (extra word added by ASR)
N = total words in the reference transcript
Here is a worked example I run on the first 30 words of any new project to gauge correction effort:
Reference (what was actually said, 12 words):
"Dr. Sarah Chen confirmed the Q4 results will drop Thursday."
ASR hypothesis (Whisper output, 12 words):
"Dr. Sara Chin confirmed the Q4 results will drop Thursday."
Alignment:
Dr. Sarah Chen confirmed the Q4 results will drop Thursday.
Dr. Sara Chin confirmed the Q4 results will drop Thursday.
[S] [S]
S = 2, D = 0, I = 0, N = 12
WER = (2 + 0 + 0) / 12 x 100% = 16.7%
A 16.7% WER on a 12-word segment means two wrong words, both of them names. The FCC accuracy criterion specifically requires "correct proper names" - that 16.7% would not pass a QC check for broadcast. I run this calculation on a representative 30-50 word sample, covering a name-dense segment, to decide how deep a correction pass the file needs.
Whisper's research paper (Radford et al., 2022) reports approximately 2.7% WER on LibriSpeech test-clean using the Large V2 model - that is clean, studio-quality read speech. Real conversational audio, panel discussions, or accented speakers perform notably worse. The paper explicitly notes performance scales with training data volume, so lower-resource languages have higher error rates. For broadcast QC, Whisper is my starting point, never my endpoint.
How does ASR accuracy map to the FCC's four quality benchmarks?
The FCC adopted four closed caption quality benchmarks in 2014 under 47 CFR § 79.1(j)(2). No specific numeric threshold is mandated - the standard uses "to the fullest extent possible". But the IP captioning standard adds precision: captions "must match the spoken words verbatim, in order, without paraphrasing" and "must identify the speakers."
Here is how raw ASR output maps to each benchmark:
| FCC benchmark | Definition (§ 79.1) | Raw ASR status | What a human pass fixes |
|---|---|---|---|
| Accuracy | Matches spoken words verbatim, in order, including proper names and punctuation; identifies speakers | Partial: dialogue mostly correct on clean audio; proper nouns, punctuation and speaker IDs routinely wrong | Name corrections, punctuation, speaker labels, sound cues |
| Synchronicity | Captions coincide with speech; displayed at a readable speed | Usually acceptable for Whisper/YouTube; minor drift on rapid speech | Adjust cue boundaries on rapid-fire speech; check CPS |
| Completeness | Runs beginning to end; no drops | Usually complete except during overlapping speech or very low-SNR audio | Fill dropped segments; add sound cue events |
| Placement | Does not cover important visual content | Not controlled by ASR (depends on player/renderer) | Human sets placement in SCC PAC row or SRT positioning |
When is auto-captioning enough, and when do I call in a human?
This is the question I get most from first-time clients. Here is how I think through it:
- Auto-captions alone for: informal internal communications, social-media content where you explicitly disclose "auto-generated captions", or any video where you review and correct the output yourself before publishing.
- Auto-captions as a first draft (with a human correction pass) for: most online video where the audio is clean and single-speaker. The correction pass in Subtitle Edit takes 1-2 minutes of editor time per minute of video for good-quality audio. That is the break-even where ASR saves meaningful time.
- Human captioning from scratch for: multi-speaker panel recordings, heavily accented audio, any content with significant technical terminology or proper-noun density, and any broadcast or streaming deliverable where FCC quality benchmarks apply. A correction pass on bad ASR can take longer than writing from scratch.
- Broadcast delivery (SCC for US TV) requires a human QC pass regardless: FCC's accuracy benchmark includes speaker identification and sound effects, neither of which ASR provides. For SCC files, I always run a caption-inspector decode to verify the byte-level encoding and PAC row assignments.
My caption QC service includes an accuracy review against the four FCC benchmarks, speaker ID and sound-cue additions, and a caption-inspector decode for SCC files. If you have a Whisper draft and need it broadcast-ready, that is the step that gets it there.
FAQ
Can I use Whisper output directly for a broadcast SCC delivery?
No. Whisper produces SRT or VTT - neither is an SCC file. Beyond the format conversion (which SeConv handles: seconv input.srt ScenaristClosedCaptionsDropFrame --fps:29.97 --output-folder out --overwrite), the content needs a human accuracy pass: speaker IDs, sound cues, proper-noun corrections, and punctuation review must all be done before an SCC file is broadcast-compliant. The FCC accuracy benchmark requires verbatim dialogue including correct proper names and speaker identification.
What is a realistic WER for Whisper on a typical talking-head interview?
For a clean, single-speaker interview in English in a quiet environment, Whisper Large typically reaches a WER in the range of 3-8% on straightforward segments. Segments with technical jargon, proper nouns, or crosstalk will be worse. I always do a quick manual WER spot-check on the first dense-name segment of a new project: if it is above 10-15%, I budget time for a thorough correction pass rather than a quick skim.
Does Whisper produce speaker identification?
No. Standard Whisper produces a single undifferentiated transcript. Speaker diarization is a separate step; open-source tools like pyannote.audio can be combined with Whisper, but results are imperfect. For broadcast captioning where the FCC IP standard requires speaker identification, a human editor must add speaker labels after the ASR pass.
How does WER differ from caption accuracy in an FCC compliance sense?
WER is a numerical metric: it counts word-level substitutions, deletions, and insertions relative to a reference transcript. FCC accuracy is a legal standard that uses qualitative language and adds requirements WER does not capture: speaker identification, correct punctuation, non-speech audio, and no paraphrasing. A file can have a low WER and still fail the FCC benchmark if it omits speaker labels and sound effects.
Is YouTube's automatic caption generation good enough for accessibility compliance?
Not without review. WCAG 1.2.2 (the standard incorporated by Section 508 for federal agencies) requires captions for prerecorded synchronized media. The W3C guidance notes that captions must include dialogue and important background sounds - which auto-captions do not reliably provide. Section508.gov guidance explicitly states that auto-captioning alone is insufficient. After a correction pass, export the cleaned SRT to WebVTT with SeConv (seconv input.srt WebVTT --output-folder out --overwrite) and attach it via an HTML5 <track kind="captions"> element.
Official Sources
- 47 CFR Part 79 - eCFR - primary text of FCC closed captioning quality benchmarks (§ 79.1(j)(2)). Verified November 2026.
- FCC Closed Captioning Quality Standards Order (2014) - FCC.gov - report and order adopting the four quality benchmarks. Verified November 2026.
- OpenAI Whisper - GitHub - model documentation, language list (99 languages including Vietnamese), and performance notes. Verified November 2026.
- Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022) - arXiv - Whisper paper with WER results (2.7% on LibriSpeech test-clean, Large V2). Verified November 2026.
- Video and Synchronized Media - Section508.gov - guidance that auto-captioning alone does not satisfy WCAG 1.2.2. Verified November 2026.
作者:Đào Huy(Lucas),越南语译员与本地化专家(英、中、法译越)。 See translation services →

