The State of Wolof Voice AI — Benchmarks, Failure Modes, and the Gap to Production.

1. Executive summary

Read time: 5 minutes. The rest of the report is the evidence.

The state in one sentence

Voice AI support for Wolof in 2026 is measurably behind comparable-volume languages across every major commercial provider, and the gap shows up in the specific places that matter most for production — numerals, code-switching, negation, and name extraction.

Why you should read this now

Roughly twelve million people in Senegal, Gambia, and Mauritania speak Wolof as their primary language, most of them code-switching with French in urban contexts. Three of the largest consumer apps in Senegal — Wave, Orange's Max It, Sonatel's emerging ecosystem — have shipped or are actively shipping voice features in 2026. They are making provider-selection decisions without public benchmarks to guide them. This report is the first public benchmark of commercial Wolof voice AI, measured on 104 Senegalese voice samples against six provider configurations, and it exists to replace marketing claims with ground truth.

What we measured

One hundred and four Wolof / French / code-switched voice samples from Senegalese merchant and field-pilot recordings, each labelled with a reference transcript, expected numeric amount, and expected merchant-transaction intent. We ran each sample against:

• OpenAI Whisper — gpt-4o-transcribe and whisper-1, direct API calls with no custom prompts.
• Google Gemini 2.5 Flash — native audio transcription via Vertex AI.
• Google Cloud Speech-to-Text v2 — Chirp 2 with French (fr-FR) as the primary probe; Chirp 2 with Wolof (wo-SN) as a deliberate second probe to measure what the API actually does when asked for Wolof.
• The Kuma Labs end-to-end pipeline — Whisper primary + Wolof-aware downstream stages (number parser, intent classifier, validator), as a reference for what a vertically-integrated product layer can add on top of a commercial ASR.

Four metrics per (sample, provider) pair: Word Error Rate, numeral extraction accuracy (using a shared open-source Wolof number parser so every provider is scored on a level playing field), intent classification accuracy (top-1 exact match), and code-switch preservation via fastText language ID. Latency and cost per 1000 requests reported alongside.

Full methodology in §3. Full matrix in §4. The failure-mode catalogue in §5 is the section worth your time if you are choosing a provider.

The three findings that matter

The three findings that matter

Finding 1 — Every provider shows high WER on Wolof. Mean WER ranges from 0.73 (Google STT Chirp 2 wo-SN probe) to 1.05 (Whisper-1) across the 104-sample corpus. Kuma's end-to-end pipeline achieves 0.77 mean WER despite running the full 7-stage pipeline on top. The best raw-ASR performer (Chirp 2 wo-SN probe at 0.73) still gets more than seven out of every ten reference tokens wrong on average. WER > 1.0 on Whisper-1 means it hallucinated more tokens than the reference contained — typical when a model encounters a language it was not trained for and produces phonetically-nearest English.

Finding 2 — Numeral extraction is the critical gap. Every provider's ASR numeral capture rate is under 33%, and CFA-strict accuracy is 0% across the board. The Kuma end-to-end pipeline captures numerals at 32.8% (ASR-captured, lenient) — best in class — but still misses two-thirds of amounts. The dërëm ×5 gap is the core issue: ASR systems hear the spoken number but no system (including Kuma) yet applies the CFA conversion reliably across all Wolof commerce forms. This is not a transcription failure; it is a downstream semantic failure that will persist until Wolof-specific numeral post-processing is standardized.

Finding 3 — Intent classification on in-vocabulary labels is strong; schema gaps drive most failures. Kuma's end-to-end pipeline achieves 43% overall intent top-1 accuracy, but this hides a sharp split: sale classification reaches 90.6%, credit reaches 55.6%, and customer_lookup reaches 50%. The 0% rates on inventory, complaint, and other are schema gaps — these labels are outside the production classifier's four-label vocabulary, so no transcript in those categories can score a match. Code-switched utterances (n=73 of 104) show the highest WER variance, confirming that mixing-language boundaries remain the hardest single ASR challenge.

Chirp 2 wo-SN probe row = deliberate experiment: what does Chirp 2 return when asked for Wolof? It accepts the request silently and returns non-Wolof output — but scores lower WER because the degenerate output overlaps the reference tokens by chance more often than hallucinated English. See §3.2.1.

The failure modes that matter most

Five recurring, production-critical failure patterns, expanded in §5 with transcripts, audio, and downstream consequences:

1. Dropped negation morphology — the Wolof -ul suffix is routinely dropped, silently flipping a repayment status from unpaid to paid. This is the most dangerous single failure in the report.
2. The dërëm ambiguity, both directions — the same bare Wolof numeral can mean direct CFA or implicit-dërëm (×5). Dropping the unit produces 5× under-reports; over-applying it produces 5× over-reports. Most production failures in this class are the product layer silently picking one reading. See Failure 3·2 for the full design tension and our resolution.
3. Wolof names misread as French phrases — "Awa" transcribed as "à voir"; creates ghost customer records in production.
4. Prompt-echo artifacts — when custom prompts are used to improve accuracy, they can be echoed verbatim into the transcript on silent or near-silent audio, producing syntactically-valid but wildly-wrong amounts. We reproduced a two-hundred-billion-franc hallucination from this class deterministically during benchmark development.
5. Illusory Wolof support at the API level — one commercial provider accepts wo-SN as a language code and returns output that is not Wolof, with no error or warning. Teams doing vendor evaluation by the interface contract will reach the wrong conclusion.

What this report is and is not

This is a specific snapshot: six provider configurations, 104 samples, measured on 2026-04-28, off-the-shelf models with no fine-tuning. It is the first public benchmark of its kind for Wolof, not the definitive one. Our hope is that it ages quickly — that successors replace it with better numbers on larger corpora — and we have released the full corpus, eval harness, and rolling-leaderboard infrastructure (§8) to make that replacement easy.

If you are a product team building voice UX for francophone West Africa, skip to §5 and §7. If you are a researcher or open-source contributor, skip to §8. If you are evaluating providers for a procurement decision, read §3 then §4.

The rest of the document is the evidence for the sentence at the top of this page.

2. Why this benchmark matters now

Four things changed in the last eighteen months that make a public benchmark of Wolof voice AI urgent rather than academic.

2.1 Francophone African fintech is shipping voice

In 2025 and early 2026, the largest Senegalese and pan-African consumer financial apps began integrating voice as a primary input modality — not just as a novelty accessibility feature. Wave launched voice-guided transfers for users with limited literacy. Orange's Max It super-app rolled out an in-app voice assistant for balance queries and money-transfer initiation. Cofina and smaller regional BNPL platforms are running voice pilots for loan-officer field work. These are not research projects — they are production surfaces serving millions of daily users.

Every one of them makes provider-selection decisions based on benchmarks. None of those benchmarks, as of the publication of this report, are public for Wolof. Decisions are being made blind, or on marketing claims, or on a single engineer's ad-hoc spot-check against three audio files. That is not a sustainable foundation for a system that millions of Senegalese will rely on for their cash management.

2.2 The super-apps are consolidating the interface — and the voice layer inside it

Max It, Wave, and the emerging Sonatel ecosystem are converging on the same product pattern: one app that is simultaneously the user's bank, their mobile-operator dashboard, their insurance provider, and their ride-hailing interface. The voice layer, when it arrives, is the single interface across all of that. The stakes of getting Wolof transcription wrong are not "the weather app is slightly annoying" — they are "the user just tried to pay their electricity bill and the wrong amount was debited".

Super-apps typically build their voice layer in one of two ways: an in-house ASR team (expensive, slow, reliable), or a thin wrapper around a commercial provider (fast, cheap, uncertain). The second pattern is overwhelmingly the one we see in the market. It works beautifully in English and French. It degrades in ways that are difficult to measure, and impossible to price, in Wolof. This report measures it.

2.3 Public-sector voice UX is a serious design choice, not a future

Senegal's Digital Senegal 2025 program includes explicit commitments to voice-based access for public services. Electoral registration, social protection enrolment, agricultural subsidy claim workflows — all three have active pilots that depend on users being able to dictate their information in the language they actually speak at home. For a meaningful share of the population that language is Wolof, not French.

A voice UX that works only in French is not accessibility — it is a French-language filter applied to a supposedly-universal service. The technical gap between "we accept French" and "we accept the language the citizen speaks" is exactly the gap this benchmark quantifies. Public-sector procurement teams evaluating vendors need these numbers; there is no neutral source for them today.

2.4 The Wolof benchmark vacuum is structural, not accidental

A reasonable question is: why did this not exist already? The short answer is that African languages sit at the intersection of three forces that each independently suppress benchmark work:

• Commercial: the providers (OpenAI, Google, Meta) do publish WER numbers for Wolof and other African languages in their model cards, but these are self-reported on proprietary test sets. No external party has reproduced them on a shared corpus. Self-reported numbers are not verifiable numbers.
• Academic: the NLP research community has done excellent linguistic work on Wolof (Guérin 2021 in particular is foundational), but applied-benchmark work on production ASR providers is rarely published — it is slow, unglamorous, and the providers update their models faster than peer-reviewed papers ship.
• Ecosystem: African AI startups that are doing the work internally (Lelapa AI, kaatyb.ai, Orange Labs Dakar) have the results, but publishing them involves revealing either a commercial vendor's weakness (reputationally costly) or the startup's own pilot data (competitively costly). The incentive is to keep the numbers private.

The result is an information vacuum that this report exists to partially fill. We publish because we think the ecosystem is net-better when the numbers are public than when they live in a dozen internal Notion docs scattered across a dozen startups. Kuma Labs has a commercial interest in the space, which this report will be honest about; we have tried to write the methodology in a way that would let any competent team reproduce our results and publish their own.

2.5 What this report is, specifically

A snapshot. A set of numbers and failure modes, measured on a specific day, against specific providers, using specific models. Voice providers update their models on timelines of weeks to months; the numbers in this report will age. What will age more slowly is the failure-mode catalogue in §5 — the structural problems (dropped negations, silent unit conversions, language-code-accepted-but-not-supported) are likely to persist through several model updates because they are not what the providers are currently optimising for.

If the numbers in this report become inaccurate through a future model release, that is a good outcome, and Kuma Labs commits to publishing a refresh within 30 days of any major ASR model release from OpenAI, Google, or Meta that meaningfully changes the picture. The GitHub repository in §8 will include the rolling results.

What we ask of readers: take the numbers seriously as ground truth for the specific day they were measured, and take the methodology seriously as a template you are welcome to adapt. The goal is not to be the definitive benchmark — the goal is to be the first credible public one, and to be obviously-reproducible so that successors can replace it.

3. Methodology

This benchmark measures what production systems actually get wrong when Wolof-speaking users talk to them. Every design choice below serves that goal — and the choices we deliberately did not make are as important as the ones we did.

3.1 Corpus

Target composition — 104 samples. A single manifest drives the benchmark:

Each sample carries reference transcript, expected numeric amount, expected intent label, speaker demographic (age band, gender, L1), and acoustic conditions (market / studio / phone-call). All ground-truth labels are verified by at least one native Wolof speaker other than the recorder.

Provenance. Samples are drawn from Kuma Labs' labelled evaluation set assembled during the Phase 4 merchant pilot work, plus additional recordings captured specifically for this report where existing coverage is thin (notably numerals 10⁶–10⁹ and Mauritanian/Gambian variant boundary cases, which we flag but do not publish until a v2 benchmark).

Public release. The 104-sample corpus is published under CC-BY with speaker consent obtained under Kuma's dual-consent framework (verbal at capture, written at release). Any sample whose consent is not affirmatively re-confirmed before publication is swapped out, even if it means a smaller final corpus; the harness and results numbers are re-run on whatever set lands.

What this corpus is and isn't. It is a credible cross-section of the phonetics, numeral forms, and code-switch patterns that production Senegalese voice applications will encounter. It is not a full diagnostic dataset for Wolof ASR at the linguistic-phenomenon level — that is a multi-year research effort. We do what we can in 104 samples and are explicit about what we skip.

3.2 Providers benchmarked

We benchmark each provider in two modes wherever applicable: raw ASR only (what the provider delivers off-the-shelf) and, where the provider is part of Kuma's own production pipeline, end-to-end with Kuma's downstream stages (language router, number parser, intent classifier, validator). Holding these apart is the central methodological choice of the report — raw-provider columns show where the ASR market is today; the Kuma-end-to-end column shows what a vertically-integrated product layer can compensate for.

Provider inventory

3.2.1 Chirp 3 and the "illusory Wolof support" probe

Two notes on Google STT v2 worth surfacing before the results:

1. Chirp 3 was unavailable on our Google Cloud project across us-central1, global, europe-west4, and us-east5 as of 2026-04-22. This benchmark therefore reports Chirp 2 only. We will re-run the matrix when Chirp 3 becomes generally available in our region and publish updated numbers on the rolling leaderboard (§8.3).
2. Chirp 2 silently accepts wo-SN as a language code. It does not reject the request; it returns output. The output is not Wolof. We include this as a deliberate probe row in the matrix — it quantifies the gap between API-level "supports Wolof" claims and actual transcription behaviour.

3.2.2 Raw mode is genuinely raw

For each raw-provider adapter we disabled everything that would flatter the numbers:

• No merchant vocabulary prompt. Kuma's production Whisper adapter supplies a Wolof/French merchant-vocabulary prompt. That prompt is Kuma's engineering; it is not what an OpenAI customer gets by default. Raw-Whisper rows are prompt-less.
• No decode retries. All SDKs configured with max_retries=0. Latency is honest round-trip time, not retry-masked.
• Gemini thinking disabled. thinking_budget=0 by default, matching how an ASR-only caller would configure the model. Thinking-on is measured separately where we want to quantify the delta; it is not conflated.
• Single-shot inference. No n-best decoding, no ensemble, no voting. Each provider gets one attempt per sample.

3.3 Metrics

All four core metrics are computed for every (sample, provider) pair and written to a CSV row. The matrix runner and metrics functions are in the public harness (§8).

3.3.1 Word Error Rate (WER)

Computed with jiwer on the lower-cased, punctuation-normalized reference and hypothesis. Empty-hypothesis against non-empty reference yields 1.0 (every word deleted). We report WER at the individual-sample level in the failure catalogue and aggregated per provider × language segment in the results table.

3.3.2 Numeral accuracy

Every provider transcript — raw or end-to-end — is passed through the same open-source Wolof number parser (wolof-numbers, PyPI, also used in Kuma's production pipeline). The first extracted integer is compared exact-match against the sample's expected_amount. This matters: every provider is scored on a level playing field for numerals. The comparison is not "Kuma's number parser versus Whisper alone" — it is "can each provider's transcript make it through a shared, openly-auditable Wolof numeral extractor".

One known limitation we surface rather than hide: wolof-numbers handles Wolof tokens and some French anchors (mille) but does not currently parse full French multi-word amounts ("mille cinq cents") or Arabic digits. This is a parser limitation, not a provider limitation, and affects all providers identically.

3.3.3 Intent accuracy

Measured only where a provider returns structured intent output. In this benchmark that is the Kuma end-to-end pipeline only — raw ASR providers return transcripts, not classified merchant intents. The intent column for raw providers is "not measured" rather than zero; treating it as zero would misrepresent the raw-ASR comparison.

Scoring is top-1 exact-match on the intent label (credit / sale / customer_lookup / reminder), case-insensitive, whitespace-stripped.

3.3.4 Code-switch handling

Each sample's reference transcript and each provider's hypothesis are run through the fastText lid.176 language-identification model (k=3, minimum confidence 0.10). A hypothesis is considered to "preserve code-switching" when every language detected in the reference also appears in the hypothesis. Monolingual samples reduce to "did the hypothesis retain the expected language".

When fastText is unavailable in the run environment the metric returns available=false and the matrix records "not measured" for that row rather than defaulting to a pass or fail. fastText lid.176 was available for the 2026-04-28 benchmark run.

3.3.5 Latency

Wall-clock milliseconds from first byte sent to last byte received, measured with time.monotonic() around the provider call. We report P50 and P95 per provider. We report this honestly: we did not shard, did not batch, did not warm the endpoint ahead of each call. First-call cold starts are kept in the data — Vertex AI Gemini calls in particular show a noticeable cold-start penalty that an SLO-sensitive caller must budget for.

3.3.6 Cost per 1000 requests

Derived from the providers' publicly documented prices as of the benchmark run date, multiplied by the measured audio-seconds consumed. Negotiated or volume-discounted rates are not used; the goal is a number a reader can reproduce with a published rate card.

3.4 What we explicitly do not measure

Being specific about scope boundaries is itself a finding — readers who need these numbers should know to budget a separate benchmark for them.

• Fine-tuned model accuracy. Every provider is tested off-the-shelf. Custom-trained Whisper, Gemini tuning, or Google STT custom classes would move the numbers; that is a different report.
• On-device / self-hosted inference. Meta's MMS, Wav2Vec-BERT, and similar open-weight Wolof models are excluded from v1 — they require GPU infrastructure we do not currently host. A v2 benchmark with open-weight comparators is on the Kuma roadmap.
• Multi-country Wolof variants. Senegalese Wolof only. Mauritanian and Gambian variants differ phonetically; we flag sample-level outliers where they surfaced during labeling but do not publish variant-split numbers.
• Speaker-specific adaptation. No fine-tuning on speaker embeddings, no per-user vocabulary priors. Each sample is treated as a cold-start utterance from an unknown speaker.
• Streaming ASR latency. All providers are called in file-upload mode. Streaming recognize endpoints (Chirp 3 streaming, Whisper real-time) have different latency profiles and are not part of this report.
• Downstream task accuracy beyond MerchantIntent. The intent labels used here cover merchant-transaction semantics. KYC name capture, sentiment analysis, and open-ended question answering are out of scope.

4. Results

All numbers below are produced by the open-source eval harness (§8) from results/state-of-wolof-2026/2026-04-28T18-50/matrix_rescored.csv. Total: 624 calls, 622 successful, 2 errors (one audio-too-long, one ASR timeout on Kuma end-to-end).

4.1 WER by language segment

Kuma end-to-end achieves the best WER on wo-SN (0.655) — the Wolof-only segment — because Kuma's Whisper adapter is prompted with Wolof/French merchant vocabulary. Gemini and Whisper-1 both show WER > 1.0 on fr-SN samples, meaning they hallucinate more tokens than the reference contains on French-accented Senegalese speech. Chirp 2 fr-FR is best on French-only samples (0.803) but degrades badly on pure Wolof (0.961).

4.2 WER by difficulty

4.3 Numeral extraction accuracy

CFA-strict rate is 0 for every provider — a finding, not a bug. The open-source wolof-numbers parser correctly identifies spoken Wolof numerals (ASR-captured column) but the dërëm ×5 CFA multiplier is never applied because every ASR provider produces phonetically plausible but semantically inconsistent transcripts. The gap between ASR rate and CFA rate is the dërëm gap — the core production problem this benchmark quantifies.

4.4 Intent classification (Kuma end-to-end only)

4.5 Latency

Gemini 2.5 Flash P95 cold-start: the maximum latency of 10,608ms on a single call reflects a cold-start penalty on Vertex AI. Kuma end-to-end P95 of 22,530ms includes the full pipeline (ASR + 6 downstream stages). The raw Whisper gpt-4o-transcribe P50 of 1,093ms is the fastest single-provider option for latency-sensitive deployments.

4.6 Errors

2 of 624 calls errored (0.3%). Both errors are on the Kuma end-to-end provider. All raw-ASR providers returned transcripts for all 104 samples.

5. Failure-mode catalogue

This section is the reason the report exists. Aggregate WER numbers are useful, but they do not tell a product team what will go wrong on Monday morning with real users. Below are twenty-two concrete failures (twenty original entries plus two new silence-handling entries), organized into four product-engineering categories operators face when shipping voice AI: silence handling, language misclassification, numeral parsing, and prompt leakage. Two addenda follow for completeness: intent classification errors (specific to systems that do downstream classification) and pipeline fragility (cold starts, downstream timeouts).

Taxonomy at a glance

1. Silence handling

2. Language misclassification

3. Numeral parsing

4. Prompt leakage

5. Intent classification — Kuma addendum

The five entries below apply only to systems that emit structured intent JSON. Of the providers benchmarked, only Kuma's end-to-end pipeline does this — the raw ASR providers (Whisper, Gemini, Chirp 2) are scored "—" for intent. These failures matter for product teams building anything resembling Kuma's transaction-extraction layer; they do not matter if you are evaluating ASR alone.

6. Other pipeline fragility

Summary of the catalogue

If we had to compress this section into a single paragraph for a product manager's weekly digest, it would be this:

The ASR is the easy part. Whisper and Gemini both produce credible Wolof transcripts most of the time. What goes wrong in production is almost always downstream — dropped negations, silent unit conversions, confidence thresholds that do not match observed calibration, prompt artifacts that pass validation because the schema is too permissive. Fixing the ASR is Google's or OpenAI's job. Fixing the pipeline around it is the product team's job. This report is mostly about the second category, because that is where the actionable work sits.

6. Kuma end-to-end results

The Kuma end-to-end pipeline runs Whisper gpt-4o-transcribe as its primary ASR, followed by six downstream stages: language router, Wolof number parser, named-entity recognition, intent classifier, HITL gate (disabled for benchmarking), and validator. This section reports what the full pipeline delivers on the same 104 samples as the raw-ASR providers above.

6.1 What the pipeline adds over raw Whisper

Comparing Kuma end-to-end to its underlying raw-Whisper (gpt-4o-transcribe) ASR row in the matrix:

• WER: 0.770 vs 0.814 — a modest improvement. The downstream stages do not re-transcribe; they add structured output on top of what Whisper produced.
• Numeral ASR rate: 0.328 vs 0.134 — Kuma's wolof-numbers parser extracts numerals from Wolof morphology that the generic Whisper output expresses differently, giving 2.4× better numeral capture.
• CFA-strict rate: 0 for both — the dërëm gap has not been closed yet. Both systems hear the number; neither applies the CFA conversion reliably.
• Intent classification: Kuma end-to-end adds intent (raw Whisper cannot). Top-1 accuracy: 43.1% overall, 90.6% on sale, 55.6% on credit, 50% on customer_lookup, 0% on out-of-schema classes.

6.2 Pipeline errors on this corpus

Two of 104 calls errored: one audio file was 46.5s (above the 30s synchronous limit), and one ASR stage timed out. Both errors are recoverable — the first by routing to the async /v1/jobs endpoint, the second by retry. Raw ASR providers returned 0 errors on the same 104 files.

6.3 What the pipeline does not yet solve

• CFA-strict numeral accuracy. The dërëm ×5 conversion fires correctly on explicit dërëm tokens but not on bare commerce numerals where the unit is implied. This is the main open item for wolof-numbers v0.3.
• Schema completeness. The production intent classifier does not cover inventory, complaint, or other. Forty-two of the 104 corpus samples fall into these out-of-vocabulary classes; extending the schema is a Phase 09 item.
• ASR latency. End-to-end P50 of 9,242ms and P95 of 22,530ms are too slow for real-time voice UX. The pipeline is built for asynchronous merchant-input use cases where latency is measured in seconds, not sub-second streaming.

7. Implications for builders

This section is for the people who will build with these numbers. We group recommendations by product category because the failure modes that matter differ sharply by use case — a BNPL team has a very different set of concerns than a public-sector registration team, and recommendations written for one audience do not transfer cleanly to the other.

7.1 If you are building a BNPL, lending, or repayment-tracking product

The risk you should care about most: dropped negations and direction flips.

A BNPL workflow built on voice input is a negation-detection problem with a repayment system wrapped around it. The single most dangerous failure in §5 is Failure 2·5 — the dropped -ul suffix that turns not paid into paid.

1. Treat negative intents as a separate pipeline. Do not trust "Fatou has not paid" and "Fatou has paid" to be distinguished by a single ASR pass. Build an explicit negation-detection stage after transcription.
2. Require dual confirmation for any transaction that would zero out a debt. The cost of an extra confirmation step is tiny; the cost of a missed negation is a bad debt that compounds.
3. Use the currency default, not explicit currency. Your customers will not say "XOF" or "francs CFA" aloud. Default explicitly to XOF in your schema.
4. Plan for the dërëm unit in small-value transactions. If you are serving merchants with daily transactions under 1000 XOF, the dërëm × 5 CFA convention will show up. Encode it or your ledger will under-report five-fold.
5. Set an amount sanity cap per tenant. Informed by the tenant's own historical transaction ceiling. The 10⁸ XOF default Kuma uses is a reasonable starting point for merchant pilots.

Minimum bar for deploying voice BNPL in Wolof production: human-in-the-loop confirmation on any record where direction, amount, or customer-name confidence is below 0.90. HITL is not optional at the current state of the art.

7.2 If you are building a super-app voice assistant

The risk you should care about most: cold starts and prompt-echo artifacts.

1. Budget for cold starts; do not pretend they are not there. Gemini on Vertex took 28 seconds on the first call in our benchmark. Either pay for a warm ping, or show the user an explicit "first request may take longer" state.
2. Audit your prompt-echo exposure. If your team has added a custom prompt, your downstream parser must treat prompt vocabulary as untrusted input. See Failure 4·1.
3. Confidence thresholds should be calibrated on your own traffic. Every provider's confidence field means something different. Collect a month of production scores against observed outcomes and derive empirically.
4. Code-switched utterances will dominate your traffic; design for them from day one. Providers that force a single language code will systematically lose one of the two language segments.
5. Log the raw transcript alongside the parsed intent, always. When something goes wrong in production, your debugging path is (raw audio) → (raw transcript) → (parsed intent).

Minimum bar for deploying voice in a super-app: streaming transcript display to the user during capture, explicit confirmation for any transaction-initiating intent, and an escalation path to human support that includes the raw audio.

7.3 If you are building a public-sector registration or enrolment workflow

The risk you should care about most: name extraction and KYC identity matching.

1. Normalize Senegalese phonemes explicitly before matching. The ñ / n / ny variation is not an error — it is orthographic variation. Your matching layer must treat Ñouguirane, Nguirane, and Nyouguirane as the same name.
2. Do not require users to spell their names. "Spell it for me" is a French-literacy requirement smuggled into what should be an accessibility feature. Use phonetic matching against a gazetteer (Kuma's wolof-ner gazetteer of 539 Senegalese given names is open-source).
3. Budget for elder-speaker edge cases. Voice UX rolled out to the general population encounters demographics that rarely show up in merchant-pilot data. Your QA plan must include representative testing on populations the benchmark does not cover.
4. Publish your vendor evaluation. The procurement-transparency norm in francophone African public-sector AI is still being established; early contributors set the standard.

Minimum bar for deploying voice in public-sector registration: a documented accessibility exception path, in-person staff support at every registration point, and a published vendor evaluation.

7.4 If you are a researcher or open-source contributor

The risk you should care about most: dataset representativeness.

1. Contribute samples to the open corpus (§8). The most valuable contribution is not another research paper; it is more labelled audio under open license. We specifically need multi-country Wolof variants, multi-L1 speakers, elder speakers, dense-acoustic-environment recordings, and low-literacy speakers.
2. Benchmark the open-weight comparators. Meta's MMS, Wav2Vec-BERT variants, Whisper fine-tunes from Lelapa AI. Anyone with GPU infrastructure can slot them into the harness.
3. Extend the failure taxonomy. The six-category taxonomy in §5 is descriptive, not claimed-exhaustive. Propose new categories or refine the existing six.
4. Replicate under different acoustic conditions. Our corpus skews toward clean-studio and known-market recordings. Phone-call-quality audio (8 kHz, narrowband codec) would produce different numbers.
5. Publish negative results. Negative results keep the ecosystem honest and save every subsequent team the cost of re-discovery.

7.5 A note across all reader categories

Every subsection above ends with a "minimum bar" recommendation that would strike a reader from a high-income-country product context as conservative. That is intentional. The current state of the art for Wolof voice UX does not support "ship it and iterate on telemetry" as a deployment strategy; the failures that will surface in your first month of production data include categories (direction flips, dropped negations, silent language-code degradation) that a telemetry-based iteration loop struggles to detect, because the errors are not logged — they are written to the ledger as correct.

The conservative defaults buy you time to build the observability layer that lets you eventually relax them. Starting from the permissive defaults because "the numbers in English are fine" will produce an unhappy product and an unhappy set of users. The numbers in Wolof are not yet the numbers in English. This report exists, in part, to make that measurable.

8. Open dataset and reproducibility

This report is written to be reproduced. The three components below are released simultaneously with publication; the release is permissive enough that a competent reader can re-run our matrix, disagree with our numbers, and publish their own findings without asking for permission.

8.1 The benchmark corpus

The 104-sample corpus used for the numbers in §4 is published at github.com/kuma-labs/state-of-wolof-voice-2026/corpus/ under CC-BY-4.0. Each sample ships with:

• The audio file (FLAC where source quality permits, OGG Vorbis otherwise).
• The reference transcript in Wolof / French / both as appropriate.
• Speaker demographic metadata (age band, gender, L1, region).
• Acoustic-condition metadata (studio / market / phone-call / etc.).
• Expected numeric amount and expected intent label where applicable.

Speakers consented to open release under Kuma's dual-consent framework (verbal at capture, written at release). Samples whose release consent could not be re-confirmed before publication are excluded from the open corpus; the harness and results numbers are re-run on whatever subset lands. If you are extending this corpus, please continue the consent norm.

8.2 The eval harness

The matrix runner, metrics module, and all six provider adapters are published at github.com/kuma-labs/state-of-wolof-voice-2026/harness/ under Apache-2.0. A single command reproduces the matrix:

uv run python -m scripts.eval.state_of_wolof_2026.run_matrix \
  --corpus corpus.yaml \
  --audio-root ./audio \
  --providers raw-whisper-gpt-4o raw-whisper-1 raw-gemini \
              raw-google-stt-chirp2 raw-google-stt-chirp2-wo kuma-e2e \
  --output-dir results/$(date +%Y-%m-%dT%H-%M)

The harness is written to be extended. Adding a new provider means implementing a single async def transcribe(audio_path) -> EvalResult function and registering it in the matrix runner. We actively welcome contributions, in particular:

• Open-weight comparators (Meta MMS, Lelapa AI fine-tunes, Wav2Vec-BERT variants).
• Other commercial providers we did not include (Deepgram, AssemblyAI, Azure Speech).
• Extended metrics (character error rate, per-phoneme diagnostic WER, confidence calibration curves).

8.3 The rolling leaderboard

Benchmark numbers age. Provider models update on timelines of weeks, not years. To prevent this report from becoming a stale reference, Kuma Labs maintains a rolling leaderboard at kuma-labs.com/research that publishes:

• Re-runs of the matrix against any significant new model release from OpenAI, Google, or Meta, within 30 days of the release.
• Community submissions — if you run the harness against a model or stack we have not included and the numbers are independently verifiable, we will add the results with clear attribution.
• Corpus additions — when samples are added, the expanded corpus version is tagged and previous numbers are re-run for comparability.

The goal is not a static reference; it is an evolving picture of what production Wolof voice actually looks like, maintained publicly.

9. What's next — Kuma's roadmap

This report is one milestone in a longer research cadence. The concrete near-term work, in roughly the order we expect to ship it:

• wolof-numbers v0.3 to PyPI with expanded French-numeral parsing, closing the limitation surfaced in Failures 9 and 19 above.
• wolof-ner v0.2 to PyPI, with the 539-name Senegalese gazetteer expanded to cover broader francophone African naming conventions.
• v2 benchmark, extending to ~200 samples with multi-country Wolof (Mauritanian, Gambian), elder-speaker inclusion, and narrowband-codec replication. The full corpus delta released alongside.
• Open-weight comparator runs once our GPU hosting is in place — targeting Meta MMS, Whisper fine-tunes from Lelapa AI, and one Senegalese academic-partnership model.
• A follow-up report in this series, likely either "The State of Wolof Voice UX in Production" (post-pilot observability findings from the actual deployments driving the metrics above) or "The State of French-African Voice AI" broadened to cover Wolof, Bambara, Pulaar, Hausa.

Between major reports, the rolling leaderboard in §8.3 carries the incremental updates.

Contact for any of the above: Abdul Kane, kaneaziz@gmail.com, kuma-labs.com.

10. Acknowledgments

Proboutik — design partnership for Phase 4 merchant pilot that produced the labelled corpus this benchmark builds on. Specific thanks to the Proboutik product and data teams for consent-framework review, and to the partner merchants in Dakar who contributed recordings under consent.

Linguistic foundations — Maximilien Guérin's 2021 reference grammar of Wolof (A Reference Grammar of Wolof, Cambridge) informed the numeral-morphology handling in wolof-numbers and the code-switch taxonomy in §5. The specific decision to naturalize milyon and alfunni (and to reject alfu / alfii) as loanwords comes from Guérin's loanword analysis.

Native Wolof speaker reviewers — each sample in the corpus was reviewed by at least one native Wolof speaker other than the recorder for transcript accuracy. Reviewers are credited individually in the public corpus repository with their consent.

Peer feedback on the draft — colleagues at kaatyb.ai (Ismaila Seck), Lelapa AI, Orange Labs Dakar, and the wider francophone African AI community who reviewed pre-publication drafts and whose feedback shaped the failure-mode taxonomy and the implications-for-builders recommendations. Named contributors are listed in the repository; any error that remains is ours alone.

Providers evaluated — OpenAI, Google Cloud, and Google AI (Gemini) offer the models benchmarked here. Being named in this report is neither an endorsement nor a criticism; the numbers are the numbers. We thank the provider relations teams that answered documentation questions during benchmarking. We are particularly interested in provider responses to the findings in §5 and §7 — the rolling leaderboard in §8.3 is the right place for those responses.

Funding, where relevant — this benchmark was produced by Kuma Labs on internal time and resources. No external funding, grants, or provider-subsidized compute credits were used in the measurement phase. The eval-harness infrastructure is reusable internal work from Kuma's Phase 4 pilot and is released simultaneously with this report.