TAHQEEQ ASR model vs annotation

Headline

Model comparison — best of each variant

Best variant of each architecture (CTC, RNN-T, and the cross-corpus TAWSEEM baseline) scored on the same held-out val split (sha1-hashed 13.7% of Batch 34+35) and the same train slice the TAHQEEQ-trained models actually saw. TAWSEEM v0 is a cross-corpus baseline (trained on a different annotation convention, never on this TAHQEEQ data), so its train WER ≈ val WER — the bar to beat is generalization, not memorisation.

WER by split × project × model

Edit operations + categories

Bars grouped by edit-op / category, one color per model. Bigger gaps between models indicate where each model's failure modes diverge — e.g. CTC-best drops fewer words (lower delete count) than the overfit ckpts, while TAWSEEM substitutes more (different annotation convention treats certain words as different).

Category legend

What each bar in the category chart actually means, and how to act on it.

Confidence vs error — diagnostic scatter (per model)

Each dot is one clip. X-axis is the percentile rank of hypothesis_score within each model × split, NOT the raw score — because raw scores aren't comparable across architectures (see note below). Top-right of each panel = high WER + model is confident (suspect annotations). Bottom-right = high WER + model uncertain (hard audio). Suspect filter (norm WER ≥ 0.15 AND ≥ 3 diffs) is applied at the suspect-cards level — not drawn here since the y-axis is raw WER.

WER by surah × model

Which Quran passages does each model struggle with? Y-axis is corpus WER (Σ edit operations / Σ reference words per surah × model), not clip-mean WER — see the note below for why. Surahs sorted by corpus WER averaged across the 3 models, top-25 shown. Stability filter: ≥ 100 reference words per (surah × model). When all 3 models agree a surah is hard, the bars cluster high together — that's a data signal (acoustic difficulty or domain shift) rather than a model-quality signal.

Word-pair confusion (per model)

Top 20 most-frequent (annotation word, model word) substitutions for the best variant of each architecture. Comparing the tables surfaces model-specific failure modes — e.g. CTC may favour different confusions than RNN-T, and TAWSEEM's mismatches are dominated by annotation-convention differences.

TAHQEEQ CTC (best val 0.1450)

TAHQEEQ RNN-T (best val 0.1877)

TAWSEEM v0 (cross-corpus)

Mistake-flag validation (per model)

Does the annotator-flagged has_mistakes correlate with model WER? If clips marked True have higher WER, the model is meaningfully picking up on annotated mistakes. All 5 models grouped per box — a stronger True/False WER gap indicates a more discriminating model. Train and val on independent y-axes for legibility.

Position of error within the clip (per model)

Normalized position of each edit op (ref_pos / (ref_len − 1)) by op type, one panel per model. A right-leaning distribution = errors cluster at clip ends (context-decay or VAD trimming signal). A uniform distribution = the model fails evenly throughout the clip.

Cross-model — content drops & inserts

Best variant per architecture (CTC v0, RNN-T v0, TAWSEEM v0) scored on the same Batch 34+35 audio with the same att_context_size=[20, 9]. The delete-op count is "model dropped a word that was in the annotation" — high drop counts on the TAHQEEQ-trained models indicate over-pruning; high counts on TAWSEEM partly reflect different annotation conventions, but extreme differences still surface real behavioural gaps.

Top suspect annotation candidates

Ranking signal: TAHQEEQ CTC (best val 0.1450). Selection and sort are both driven by the CTC v0 model — not the RNN-T baseline. Specifically: a clip qualifies as a "suspect" if CTC v0's norm WER ≥ 0.15 AND CTC v0 has ≥ 3 word-level diffs against the annotation. Ranking is by CTC v0's hypothesis_score descending — when CTC is most confident in a hypothesis that disagrees with the annotation, the annotation is the likelier culprit. The other models' hypotheses appear as additional panes for cross-model agreement; if RNN-T and TAWSEEM converge on the same hypothesis as CTC, the case for a noisy annotation gets stronger.

Diff legend (word-aligned; character-level highlights inside replacements): red = characters only on the annotation side (model dropped them) · green = characters only in the hypothesis (model added them) · yellow = character substitution (true letter swap). A small spelling diff (one silent ا, a diacritic flip) shows up as a thin red/green sliver inside the word; full word replacements appear yellow across the whole word.

Random samples (fair-comparison reference)

Unfiltered random clips — no suspect mask, no confidence sort — so the cards reflect typical performance, not cherry-picked failures. 20 val + 20 train clips, sampled deterministically (seed 42) so the same clips appear across builds. Use this as the sanity-check baseline: if the model "looks good" here, the suspect/hard-audio sections are showing the long tail rather than the typical case.