Two industry-standard measures, plus latency, across every model in the benchmark. Numbers come from the full evaluation set (3025 isiZulu clips from African Next Voices).
How does model size relate to WER?
Percentage of words the model got wrong. Lower is better.
| Model | WER | N | Errors |
|---|---|---|---|
| facebook/omniASR-LLM-7BOpen | 39.1% | 3025 | 0 |
| facebook/omniASR-LLM-3BOpen | 43.8% | 3025 | 0 |
| facebook/omniASR-LLM-1BOpen | 43.9% | 3025 | 0 |
| dsfsi/anv-whisper-large-v3-turbo-anv-zulOpen | 46.8% | 3025 | 0 |
| facebook/omniASR-LLM-300MOpen | 48.0% | 3025 | 0 |
| facebook/omniASR-CTC-1BOpen | 48.1% | 3025 | 0 |
| facebook/omniASR-CTC-3BOpen | 50.4% | 3025 | 0 |
| sitwala/whisper-large-v3-turbo-anv-zul-250hOpen | 50.4% | 3025 | 0 |
| sitwala/whisper-large-v3-turbo-anv-zul-150hOpen | 51.2% | 3025 | 0 |
| facebook/omniASR-CTC-300MOpen | 55.3% | 3025 | 0 |
| sitwala/whisper-large-v3-turbo-anv-zul-50hOpen | 57.3% | 3025 | 0 |
| dsfsi/anv-whisper-small-anv-zulu-first-batchOpen | 60.2% | 3025 | 0 |
| openai/whisper-large-v3Open | 126.1% | 3025 | 0 |
| sitwala/whisper-large-anv-sotOpen | 172.9% | 3025 | 0 |
| openai/whisper-smallOpen | 186.5% | 3025 | 0 |
| sitwala/whisper-large-v3-anv-sotOpen | 197.1% | 3025 | 0 |
| Average across models | 79.8% |
Percentage of individual characters wrong. Lower is better.
| Model | CER | N | Errors |
|---|---|---|---|
| facebook/omniASR-LLM-7BOpen | 13.7% | 3025 | 0 |
| facebook/omniASR-LLM-1BOpen | 14.7% | 3025 | 0 |
| facebook/omniASR-LLM-3BOpen | 14.7% | 3025 | 0 |
| facebook/omniASR-CTC-1BOpen | 15.0% | 3025 | 0 |
| facebook/omniASR-LLM-300MOpen | 16.4% | 3025 | 0 |
| facebook/omniASR-CTC-300MOpen | 16.9% | 3025 | 0 |
| facebook/omniASR-CTC-3BOpen | 18.5% | 3025 | 0 |
| dsfsi/anv-whisper-large-v3-turbo-anv-zulOpen | 19.8% | 3025 | 0 |
| sitwala/whisper-large-v3-turbo-anv-zul-250hOpen | 22.2% | 3025 | 0 |
| sitwala/whisper-large-v3-turbo-anv-zul-150hOpen | 22.6% | 3025 | 0 |
| sitwala/whisper-large-v3-turbo-anv-zul-50hOpen | 25.3% | 3025 | 0 |
| dsfsi/anv-whisper-small-anv-zulu-first-batchOpen | 26.2% | 3025 | 0 |
| openai/whisper-large-v3Open | 49.2% | 3025 | 0 |
| sitwala/whisper-large-anv-sotOpen | 54.9% | 3025 | 0 |
| sitwala/whisper-large-v3-anv-sotOpen | 57.9% | 3025 | 0 |
| openai/whisper-smallOpen | 96.8% | 3025 | 0 |
| Average across models | 30.3% |
Average time it takes per sample.
| Model | Avg latency (per sample) |
|---|---|
| facebook/omniASR-CTC-300MOpen | 0.05s |
| facebook/omniASR-CTC-1BOpen | 0.07s |
| sitwala/whisper-large-v3-turbo-anv-zul-250hOpen | 0.09s |
| sitwala/whisper-large-v3-turbo-anv-zul-150hOpen | 0.09s |
| sitwala/whisper-large-v3-turbo-anv-zul-50hOpen | 0.11s |
| dsfsi/anv-whisper-large-v3-turbo-anv-zulOpen | 0.11s |
| facebook/omniASR-CTC-3BOpen | 0.12s |
| dsfsi/anv-whisper-small-anv-zulu-first-batchOpen | 0.16s |
| openai/whisper-smallOpen | 0.35s |
| sitwala/whisper-large-anv-sotOpen | 0.39s |
| sitwala/whisper-large-v3-anv-sotOpen | 0.49s |
| openai/whisper-large-v3Open | 0.61s |
| facebook/omniASR-LLM-1BOpen | 1.47s |
| facebook/omniASR-LLM-3BOpen | 1.50s |
| facebook/omniASR-LLM-7BOpen | 1.60s |
| facebook/omniASR-LLM-300MOpen | 2.39s |
WER (Word Error Rate): out of every 100 spoken words, how many did the model get wrong?
CER (Character Error Rate): same idea but at the character level.
Lower is always better. 0% would mean a perfect transcription.