SpeechRecognition.md

# Gecko Speech Recognition implementation architecture

This document summarizes the architecture, call flows, threading model, object

lifetimes, and known gaps for the new SpeechRecognition implementation, that is

able to perform recognition locally, without relying on an external service.

For now, it is intended to help reviewing the implementation, and will be turned

into an architecture overview document prior to landing.

## Architecture

Here's an high level outline of the components of this system, :

Content process:

- `SpeechRecognition` (main thread): implementation of the "recognition" side of

  the [Web Speech

  API](https://webaudio.github.io/web-speech-api/#speechreco-section)

- `SpeechRecognitionBackend` (main thread, real-time audio thread, IPC thread):

  audio ingestion and processing, audibility detection, IPC communication with

  the utility process (sending audio, receiving transcript).

- `SpeechRecognitionChild`: content-side endpoint of each top-level

  `PSpeechRecognition` actor, bound on the dedicated `SpeechIPC` thread.

Utility Process (HWInference)

- `SpeechRecognitionParent`: utility-side endpoint of the top-level

  `PSpeechRecognition` actor. `HWInferenceChild` constructs it with the trusted

  `ContentParentId` supplied by `ContentParent`, so install requests are

  attributed to their true requester.

  `IsModelAvailable`/`IsModelInstalled` run on whichever thread dispatches the

  actor's IPC messages (no dedicated thread for those). `InstallModels` creates

  a progress token and relays the request to the main process over

  `PHWInference` (see Security, below), attaching the trusted `ContentParentId`;

  it does not decide consent itself. Only an actual recognition session (after `Init()`

  succeeds) spawns a dedicated `Parakeet` thread that receives audio, runs

  speech recognition, and sends transcription results back to the content

  process.

- `parakeet.cpp` (mudler/parakeet.cpp) streaming C-API library (via

  `LlamaRuntimeLinker`, loading it dynamically from `libmozinference`), backed

  by `libggml`, optionally using GPU acceleration, for now only macOS.

  `SpeechRecognitionParent` implements the actual speech recognition using the

  library's cache-aware streaming API.

- `HWInferenceChild`: utility→main process bridge for speech recognition model

  availability/install requests, and to get a file descriptor passed down from

  the main process.

Main Process:

- `ContentParent::RecvAcquireHWInferenceProcess` and

  `RecvReleaseHWInferenceConnection`: hold the HWInference utility process

  alive while speech recognition needs it.

- `ContentParent::RecvCreateSpeechRecognition`: receives a

  `PSpeechRecognitionParent` endpoint created in content and asks

  `UtilityProcessManager` to forward it, with the trusted `ContentParentId`,

  to the utility process.

- `UtilityProcessManager`: launches/binds the HWInference utility process and

  sends each speech-recognition parent endpoint over `PHWInference`.

- `HWInferenceParent` (main-process side of `PHWInference`): receives calls for

  model availability checks, calls to get a file descriptor down to the

  HWInference process directly via IPC, and `InstallModel` requests relayed

  from the Utility process. Every one of these carries only a task + model id;

  before doing anything else, `HWInferenceParent` resolves that id to concrete

  `model`/`revision`/`filename` coordinates via the `nsIMLModelResolver`

  registered for the task (contract id

  `@mozilla.org/ml/model-resolver;1?task=<task>`) — for speech recognition,

  `SpeechModelResolver` (`SpeechModelResolver.{h,cpp}`), which just expands the

  id against the same compiled-in model table `SpeechModelFor` reads

  (see Security, below). Before downloading, it asks that same component to

  authorize the download (`nsIMLModelResolver::authorizeDownload`, passing the

  resolved coordinates); `SpeechModelResolver` owns the consent decision and

  shows the doorhanger through

  `SpeechModelDownloadPermissionRequest.{h,cpp}` (see Security, below). Uses

  `nsIMLModelHub` to call into `ModelHub` for both the download and the

  local-cache check.

- `nsIMLModelHub`: thin XPCOM component that wraps `ModelHub`, allowing its use

  from native code.

## Security

The model-provisioning threat model — why a compromised content process cannot

name an arbitrary model artifact, and why it cannot fake consent to a download

— is generic to every `HWInference` consumer and is documented in

[Security](/toolkit/components/ml/HWInference), using speech

recognition as its worked example. Speech recognition contributes

`dom::SpeechModelFor` (a language to an opaque id, in the Utility

process, from a table generated from `models.yaml` at build time),

`SpeechModelResolver` (the id back to `ModelHub` coordinates, in the main

process, from that same table, and the consent decision for a download).

The speech-specific access control layered on top is described below.

Speech recognition shares the single `HWInference` process with every other

consumer of the facility; giving chrome-driven and content-driven inference an OS

process each is future work, see [One process, many

users](/toolkit/components/ml/HWInference).

## Access control and gating

Three independent mechanisms gate access to on-device speech recognition.

They are easy to conflate, so this is the one place that lists them all.

| Mechanism | What it controls | Who sets it | Checked in |

|---|---|---|---|

| Permissions Policy `on-device-speech-recognition` | Per-frame: cross-origin iframes are blocked unless delegated | The embedding page (`allow="on-device-speech-recognition"` / `Permissions-Policy` header); `self` by default | `available()`, `install()` |

| AI Controls | Per-profile: the user disabling on-device AI globally or per-feature | The user, in `Settings > Firefox AI`; backed by `browser.ai.control.speechRecognition` (falls back to `browser.ai.control.default` when `"default"`) | `available()`, `install()`, `start()` |

| Model-download permission doorhanger | Per install transaction (site + requested languages): whether this download may proceed | The user, once per transaction; not persisted | `install()` only |

All three are checked before any IPC to the Utility process. A page that

fails one of the first two never touches the model hub or inference backend.

### Permissions Policy: `on-device-speech-recognition`

Registered in `PermissionsPolicyUtils.cpp` with a default value of `self`, like

`camera`/`microphone`. Checked via

`PermissionsPolicyUtils::IsFeatureAllowed(doc, u"on-device-speech-recognition"_ns)`.

`available()` resolves `"unavailable"` when disallowed, matching the spec's

availability algorithm, which never rejects. `install()` rejects with

`NotAllowedError`. `start()` does not check this policy directly.

### AI Controls

`browser.ai.control.speechRecognition` is a string pref with three states:

`"default"`, `"available"`, `"blocked"`. When it is `"default"`, the global

`browser.ai.control.default` applies instead, so blocking on-device AI

generally also blocks speech recognition without a dedicated setting ever

being touched. `IsBlockedByAIControls()` (`SpeechRecognition.cpp`) resolves

this fallback and is called from all three entry points: `available()`

(resolves `"unavailable"`), `install()` and `start()` (both reject/throw

`NotAllowedError`). Unlike the download doorhanger, this is durable and

profile-wide: once blocked, every call is rejected until the user changes it

in `about:preferences`. Surfaced in the AI Controls settings UI via

`OnDeviceModelManager.mjs`/`aiFeatures.mjs`

(`OnDeviceModelFeatures.SpeechRecognition`).

### Model-download permission doorhanger

Only relevant to `install()`. Before downloading model bytes, Gecko shows a

permission doorhanger (prompt type `"speech-recognition-model-download"` in

`ContentPermissionPrompt.sys.mjs` / `PermissionUI.sys.mjs`) naming the

requesting site and the download size. There is no "remember this decision"

checkbox: it is shown again the next time a download is actually needed, since

it gates the download, not the feature.

The prompt, the consent decision, and the download all run in the **parent

(main) process** (`HWInferenceParent::RecvInstallModel`, authorized by

`SpeechModelResolver`, then a `SpeechModelDownloadPermissionRequest` shown

against the requesting tab's `<browser>` element); content only asks, and the

request passes through the Utility process on the way (see "Consent to a

model download cannot be faked by content" above for why that placement is

what makes consent unspoofable).

Concurrent `install()` calls for the same window and language set are coalesced

in the content process into one `SpeechRecognitionInstallTransaction`, so only

one `InstallModels` reaches the Utility process; all the coalesced promises

settle together.

The prompt is skipped, and installation resolves `true` directly, when the

requested model is already cached (there is nothing to download, so nothing to

consent to). Tests bypass the UI with

`media.webspeech.recognition.model-download.prompt.testing` (then

`media.navigator.permission.disabled` decides allow/deny); see

`test_install_overlap.html`,

`browser_speech_recognition_model_download_prompt.js`.

`start()` never downloads: `SpeechRecognitionParent::RetrieveModel` requires the

model to already be installed and fails the session otherwise. Only `install()`

can trigger a download.

The sequence below shows the consent leg; the download leg continues from the

"Allow" branch (see "Downloading and installing a model").

```{mermaid}

sequenceDiagram

  autonumber

  box Content Process

    participant JS as Script

    participant SR as SpeechRecognition

    participant BE as SpeechRecognitionBackend

    participant SRC as SpeechRecognitionChild

end

  box Utility Process (HWInference)

    participant SRP as SpeechRecognitionParent

    participant HWC as HWInferenceChild

end

  box Main Process

    participant HWP as HWInferenceParent

    participant Resolver as SpeechModelResolver

    participant PR as SpeechModelDownloadPermissionRequest

    participant CPP as ContentPermissionPrompt.sys.mjs

    participant User as User

end

  JS->>SR: SpeechRecognition.install({langs: ["en-US"]})

  Note over SR: Permissions Policy, AI Controls,<br/>transient activation checks

  SR->>BE: ::Install(langs, browsingContext)

  BE->>SRC: SendInstallModels(langs, browsingContextId)

  SRC->>SRP: PSpeechRecognition::InstallModels

  Note over SRP: map langs->id,<br/>create progress token

  SRP->>HWC: InstallModel(task, id, innerWindowId, contentId, token)

  HWC->>HWP: PHWInference::InstallModel

  HWP->>Resolver: Resolve(id) -> model, revision, filename

  Note over HWP: resolve innerWindowId to a WindowGlobalParent,<br/>check it is owned by contentId

  HWP->>Resolver: AuthorizeDownload(model, revision, filename, window, token)

  Note over Resolver: skip prompt if already installed

  Resolver->>PR: new SpeechModelDownloadPermissionRequest(principal, <browser>, sizeMB, token)

  PR->>CPP: nsContentPermissionUtils::AskPermission()

  CPP->>User: Show doorhanger (site, sizeMB)

  User->>CPP: Allow / Not now

  CPP-->>PR: Allow() / Cancel()

  PR-->>Resolver: true / false

  alt Allow

    Resolver-->>HWP: true

    Note over HWP: Download via nsIMLModelHub<br/>(see "Downloading and installing a model")

  else Not now

    Resolver-->>HWP: false

    HWP-->>HWC: false

    HWC-->>SRP: false

    SRP-->>SRC: false

    SRC-->>BE: false

    BE-->>SR: Promise resolves(false)

end

  SR-->>JS: Promise resolves(bool)

```

## Design choices

### The `HWInference` process

Speech recognition runs its inference in `HWInference`, a utility process that

doesn't run JavaScript, has a GPU-process-like sandbox with no display server

access, and delegates model management to `ModelHub`. It is described in

[HWInference](/toolkit/components/ml/HWInference); what matters here is

that speech recognition receives audio, runs the model, and produces timed text

fragments, and that the model file arrives as a `FileDescriptor` so nothing is

copied and large models can be `mmap`ed.

### `parakeet.cpp`

`parakeet.cpp` (mudler/parakeet.cpp) is a third-party C++ library that performs

cache-aware streaming speech recognition using a Parakeet-family (RNN-T/joint)

model, via a streaming C API (`parakeet_capi.h`). It uses `libggml` underneath

for the actual computations (accelerated or not). It is a good choice because

we already vendor `libggml`, as it is the backend of `llama.cpp`, that we use

for e.g. text summarization.

In this patch set, the Metal backend (macOS) has been vendored. The Vulkan

backend (Windows, Linux, Android) will be worked on in a second stage. The CPU

backend works on all platforms.

#### The speech recognition itself

This is best explained in comments in the code, see

`SpeechRecognitionParent::ProcessAudioStreaming` in `SpeechRecognitionParent.cpp`.

The model loads from a file descriptor (`parakeet_capi_load_fd`), opens a

streaming session for the recognition language (`parakeet_capi_stream_begin_lang`,

given a locale that model knows, see Language→model mapping below, and falling

back to language auto-detection if it rejects it anyway), then is fed audio as

it arrives (`parakeet_capi_stream_feed`). The

model keeps its own encoder/decoder caches across feeds, and finalized words are

drained (`parakeet_capi_stream_drain_words`) and emitted as final results at

streaming latency.

This will have to be tuned and I have made most parameters tweakable using prefs

for this purpose.

#### The models

I have uploaded a few models to our bucket, an english-only model, and a

multilingual model. I expect that more models will be added in the future, both

with different performance characteristics, but also containing different

languages, and with different capabilities, such as token timestamping,

diarisation, punctuation correctness, etc. Consequently, the routing is

currently minimal: english goes to the english model, and the other model takes

whatever of the rest it recognizes (see Language→model mapping below).

Our bucket also contains a Voice Activity Detection (VAD) model (Silero VAD),

that can be used to detect speech activity in audio data, but I haven't wired it

yet.

### Lifetimes, thread model

#### Content process

The audio is produced by a real-time thread. It is best to do as little as

possible on it. Consequently, only downmixing to mono (that is almost free) is

done there, and the audio is immediately enqueued to a wait-free ring buffer.

A dedicated thread (called `SpeechResampler`) polls every 20ms, resamples the

audio to the model's sampling rate (constant at 16kHz), and dispatches a block

of audio to the `HWInference` process once more than 40ms is buffered. It is

started on recognition start, stopped on recognition stop or abort.

A **single** thread per content process handles the IPC from the content process

to the HWInference process. Because the `SpeechRecognition` object has both

static and instance methods, all the IPC calls run on this thread. This thread

uses a stable serial event target for the lifetime of the content process once

created; its `LazyIdleThread` releases the backing OS thread while idle.

There is no shared speech-recognition manager actor. Each static call or active

recognition session gets its own top-level `PSpeechRecognition` endpoint pair.

Content creates the pair, sends the parent endpoint to `ContentParent` over

`PContent`, and binds the child endpoint on the `SpeechIPC` thread.

What decides whether speech recognition needs that connection is

`sIPCActorUsers` (main thread only), counting `IPCActorUserGuard`s. One is

held:

- for the lifetime of every `SpeechRecognition` object, taken in its

  constructor via `AcquireProcessKeepAlive()` and dropped in

  `DisconnectFromOwner()` as well as the destructor, so a torn-down window

  releases it without waiting for GC;

- for the duration of each in-flight "transaction" -- the static

  `available()`/`install()` calls, via `RunWithTransientSession()`;

- for an active recognition session (`Start()` to `Stop()`/`Abort()`), from

  `EnsureIPC()`.

On the zero-to-one transition, content sends `AcquireHWInferenceProcess` over

`PContent`; `ContentParent` retains a `UtilityProcessKeepAlive` for the

HWInference process. This process hold is independent of the per-call

`PSpeechRecognition` actors. Endpoint creation itself is synchronous and does

not wait for the utility process: IPC queues messages until both endpoints are

bound, or rejects them if launch or binding fails.

Reaching zero does not release the process immediately. It arms a

`media.webspeech.recognition.idle_shutdown_grace_ms` (default 5s) timer,

cancelled by the next acquisition, so a `stop()`/`start()` cycle or a burst

of static calls reuses the warm process rather than paying for a relaunch. 0

releases immediately, which is what tests use. Once the timer fires, content

sends `ReleaseHWInferenceConnection` and `ContentParent` drops its keep-alive.

This grace period is a speech recognition policy and lives here, not in

`UtilityProcessManager`: other HWInference consumers may want a different one.

`PSpeechRecognition` actors can be created for two reasons:

- transient instances are created and shortly after closed for

  available/install calls. A number of those instances can be active at once,

  e.g. when available/install calls are spammed.

- long-running instances are created for speech recognition. They are kept alive

  until the user stops the recognition. A single instance can be active at once

  (to be relaxed when we allow concurrent speech recognition, after performance

  testing).

#### HWInference process

`SpeechRecognitionParent` handles most of the recognition process. It has a

dedicated thread, started during speech recognition session init, closed during

speech recognition session shutdown. It essentially loops, dequeues audio,

massages it a little bit and feeds it to `parakeet.cpp`'s streaming API.

Its lifetime is dictated by the content process, and only a single session can

be active at once in Firefox (for now, prior to performance testing, this

matches Chrome).

Parakeet objects have the same lifetime as a recognition session. The

initialization requires IPC and is highly asynchronous, to acquire the model

file, but after the init phrase, everything happens on the dedicated thread,

except appending to the SPSC ring buffer, since the audio comes from IPC.

`ActorDestroy()` must not join the dedicated thread synchronously

(`nsIThread::Shutdown()`): it runs on the main thread from inside an IPC

message dispatch, and `Shutdown()` spins a nested native event loop to wait

for the thread, which can reenter and crash. It uses `AsyncShutdown()`

instead, which only requests shutdown; the dedicated thread's own loop

observes `mShouldContinueProcessing`/`mActorDestroyed` and exits on its own,

freeing the Parakeet objects itself as the last thing it does (freeing them

from `ActorDestroy()` after only *requesting* shutdown would race with the

thread still using them). `InitializeParakeetContext()` also checks

`mActorDestroyed` before doing any work, since it can still be mid-flight

(e.g. delayed behind a model fetch) when the actor is torn down concurrently.

#### Main process

The main process is only used to create the `HWInference` process, to decide

when to tear it back down (see "Process lifetime" in the [HWInference

docs](/toolkit/components/ml/HWInference)), and to interact with

ModelHub.

`browser_speech_recognition_process_lifetime.js` covers the speech side of

that: run a session, close the tab, and the process is gone.

### Threads used

**Content process**

- `Main thread` for the implementation of the DOM api

- `MediaTrackGraph` real-time audio thread produces audio data

- Dedicated `SpeechIPC` thread to use the `PSpeechRecognition` actor

  from a stable thread, both for static calls and instance calls

- Dedicated `SpeechResampler` thread to consume audio data, resample the audio,

  and dispatch the resampled block to the `SpeechIPC` thread, which sends it

  over IPC

**HWInference process**

- Main thread receives commands and audio via IPC, produces audio into a ring buffer

- Dedicated `Parakeet` thread receives command, initialize recognition, consumes

  audio from the ring buffer, performs inference

**Parent process**

- No new threads

## Sequence diagrams

This section shows the flow of events and interactions between the different

components involved in the SpeechRecognition process. It covers a simple

scenario: calling `available()` with a language identifier, calling `install()`

with the same language identifier, then starting recognition from a

`MediaStreamTrack`.

### Checking model availability

This diagram covers shows the sequence of events that happens when calling:

```js

SpeechRecognition.available({langs: ["en-US"], processLocally: true});

```

```{mermaid}

sequenceDiagram

  autonumber

  box Content Process

    participant JS as Script

    participant SR as SpeechRecognition

    participant BE as SpeechRecognitionBackend

    participant CC as ContentChild

    participant SRC as SpeechRecognitionChild

end

  box Main Process

    participant CP as ContentParent

    participant UPM as UtilityProcessManager

    participant HWP as HWInferenceParent

    participant NSIMLMH as nsIMLModelHub

    participant MH as ModelHub

end

  box Utility Process (HWInference)

    participant SRP as SpeechRecognitionParent

    participant HWC as HWInferenceChild

end

  JS->>SR: SpeechRecognition.available({langs: ["en-US"], processLocally: true})

  SR->>BE: SpeechRecognitionBackend::Available(langs)

  BE->>CC: AcquireHWInferenceProcess() (first user only)

  CC->>CP: PContent::AcquireHWInferenceProcess

  CP->>UPM: AcquireContentHWInferenceProcess()

  BE->>BE: Create PSpeechRecognition endpoints

  BE->>CC: CreateSpeechRecognition(parent endpoint)

  CC->>CP: PContent::CreateSpeechRecognition

  CP->>HWP: StartContentSpeechRecognition(endpoint, contentId)

  HWP->>HWC: PHWInference::NewContentSpeechRecognition(endpoint, contentId)

  HWC->>SRP: Bind parent endpoint

  BE->>SRC: Bind child endpoint on SpeechIPC

  BE->>SRC: SendIsModelInstalled(langs)

  SRC->>SRP: SendIsModelInstalled

  Note over SRP: map langs->model

  SRP->>HWC: PHWInferenceChild::IsModelInstalled(model,rev,file)

  HWC->>HWP: PHWInferenceChild::SendIsModelInstalled

  HWP->>NSIMLMH: "nsIMLModelHub.isModelInstalled(...)"

  NSIMLMH->>MH: "ModelHub.isModelInstalled(...)" (local cache only)

  MH-->>NSIMLMH: bool

  NSIMLMH-->>HWP: bool

  HWP-->>HWC: bool

  SRP-->>SRC: bool

  SRC-->>BE: bool

  alt installed

    BE->>SRC: `Close()`

    BE-->>SR: Promise resolves: available

  else not installed

    BE->>SRC: SendIsModelAvailable(langs)

    SRC->>SRP: SendIsModelAvailable

    SRP->>HWC: PHWInferenceChild::IsModelAvailable(model,rev,file)

    HWC->>HWP: PHWInferenceChild::SendIsModelAvailable

    HWP->>NSIMLMH: "nsIMLModelHub.isModelAvailable(...)"

    NSIMLMH->>MH: "ModelHub.isModelAvailable(...)" (cache, else network HEAD)

    MH-->>NSIMLMH: bool

    NSIMLMH-->>HWP: bool

    HWP-->>HWC: bool

    SRP-->>SRC: bool

    SRC-->>BE: bool

    BE->>SRC: `Close()`

    alt available

      BE-->>SR: Promise resolves: downloadable

    else not available

      BE-->>SR: Promise resolves: unavailable

end

end

  SR-->>JS: Promise resolves

```

### Downloading and installing a model

This is the full, end-to-end flow of what happens when running:

```js

SpeechRecognition.install({langs: ["en-US"]});

```

from the content-process call all the way down to `ModelHub` and back. It

folds in the consent leg from the previous diagram (compressed to the

doorhanger's "Allow" outcome; see that diagram for the "Not now" branch and

the already-installed short-circuit) so this one is self-contained,

rather than picking up mid-flight in the main process.

```{mermaid}

sequenceDiagram

  autonumber

  box Content Process

    participant JS as Script

    participant SR as SpeechRecognition

    participant BE as SpeechRecognitionBackend

    participant SRC as SpeechRecognitionChild

end

  box Utility Process (HWInference)

    participant SRP as SpeechRecognitionParent

    participant HWC as HWInferenceChild

end

  box Main Process

    participant HWP as HWInferenceParent

    participant Resolver as SpeechModelResolver

    participant PR as SpeechModelDownloadPermissionRequest

    participant CPP as ContentPermissionPrompt.sys.mjs

    participant User as User

    participant NSIMLMH as nsIMLModelHub

    participant MH as ModelHub

end

  JS->>SR: SpeechRecognition.install({langs: ["en-US"]})

  Note over SR: Permissions Policy, AI Controls,<br/>transient activation checks

  SR->>BE: ::Install(langs, browsingContext)

  BE->>SRC: SendInstallModels(langs, browsingContextId)

  SRC->>SRP: PSpeechRecognition::InstallModels

  Note over SRP: map langs->id,<br/>create progress token

  SRP->>HWC: InstallModel(task, id, innerWindowId, contentId, token)

  HWC->>HWP: PHWInference::InstallModel

  HWP->>Resolver: Resolve(id) -> model, revision, filename

  Resolver-->>HWP: model, revision, filename

  Note over HWP: resolve innerWindowId to a WindowGlobalParent,<br/>check it is owned by contentId

  HWP->>Resolver: AuthorizeDownload(model, revision, filename, window, token)

  Note over Resolver: skip prompt if already installed<br/>(see doorhanger diagram)

  Resolver->>PR: new SpeechModelDownloadPermissionRequest(principal, <browser>, sizeMB, token)

  PR->>CPP: nsContentPermissionUtils::AskPermission()

  CPP->>User: Show doorhanger (site, sizeMB)

  User->>CPP: Allow

  CPP-->>PR: Allow()

  PR-->>Resolver: true

  Resolver-->>HWP: true

  HWP->>NSIMLMH: downloadModel(...)

  NSIMLMH->>MH: getModelDataAsFile(...)

  activate MH

  MH--)NSIMLMH: progress callback

  NSIMLMH--)HWP: progress callback

  Note over HWP: progress notification drives the prompt's progress UI

  MH--)NSIMLMH: Download complete

  deactivate MH

  NSIMLMH-->>HWP: download success/fail

  HWP-->>HWC: bool

  HWC-->>SRP: bool

  SRP-->>SRC: bool

  SRC-->>BE: bool

  BE-->>SR: bool

  SR-->>JS: Promise resolves(bool)

```

### Starting recognition and processing audio

This is what happens after running `start(...)` on a `SpeechRecognition`

instance that is processing locally, passing it a `MediaStreamTrack`. Again, the

initial process creation isn't repeted and is similar to the first diagram.

There are three loops running in parallel at with different interval, in

different process and with different thread priorities in this diagram:

```{mermaid}

sequenceDiagram

  autonumber

  box Content Process

    participant JS as Script

    participant SR as SpeechRecognition

    participant MTG as MediaTrackGraph

    participant BE as SpeechRecognitionBackend

    participant CC as ContentChild

    participant SRC as SpeechRecognitionChild

end

  box Utility Process (HWInference)

    participant SRP as SpeechRecognitionParent

    participant HWC as HWInferenceChild

    participant WLIB as parakeet.cpp

end

  box Main Process

    participant CP as ContentParent

    participant UPM as UtilityProcessManager

    participant HWP as HWInferenceParent

    participant MH as ModelHub

end

  JS->>SR: "start([track])"

  SR->>SR: "Validate track or getUserMedia"

  SR->>BE: "new Backend, Start()"

  BE->>BE: "Create PSpeechRecognition endpoints"

  BE->>CC: "CreateSpeechRecognition(parent endpoint)"

  CC->>CP: "PContent::CreateSpeechRecognition"

  CP->>HWP: "StartContentSpeechRecognition(endpoint, contentId)"

  HWP->>HWC: "PHWInference::NewContentSpeechRecognition"

  HWC->>SRP: "Bind parent endpoint with trusted contentId"

  BE->>SRC: "Bind child endpoint on SpeechIPC"

  BE->>SRC: "SendInit(lang, phrases)"

  SRC->>SRP: SendInit

  SRP->>HWC: PHWInference::GetModelFile

  HWC->>HWP: RecvGetModelFile

  HWP->>MH: getModelFileAsBlob(...)

  MH-->>HWP: Blob

  HWP-->>HWC: FileDescriptor

  HWC-->>SRP: FileDescriptor

  SRP->>SRP: `FileDescriptor` to `FILE*`

  SRP->>WLIB: `parakeet_capi_load_fd(fileno(FILE*))`

  activate WLIB

  WLIB->>WLIB: `fread`, compile shaders, etc.

  WLIB-->>SRP: ctx

  deactivate WLIB

  SRP->>WLIB: `parakeet_capi_stream_begin_lang(ctx, lang)`

  WLIB-->>SRP: stream

  SRP-->>SRC: Init resolved true

  SRC-->>BE: Init resolved true

  BE->>BE: Start resampling thread

  SRP->>SRP: Start Parakeet thread

  loop Audio capture loop, real-time thread, every ~3 to 20ms

    MTG->>MTG: SpeechTrackListener::NotifyQueuedChanges

    MTG->>BE: SpeechRecognitionBackend::DataCallback

    BE->>BE: Downmix, enqueue frames on real-time thread

end

  loop Speech resampling loop, SpeechResampler thread, polls every 20ms

    BE->>BE: Resample to 16kHz once >40ms buffered

    BE->>SRC: SendAudioDataViaIPC(16kHz f32)

    SRC->>SRP: SendProcessAudioData(16Khz f32)

    SRP->>SRP: Enqueue

end

  loop Parakeet streaming loop, dequeues as audio arrives

    SRP->>SRP: Dequeue

    SRP->>WLIB: `parakeet_capi_stream_feed(stream, chunk)`

    WLIB-->>SRP: committed text delta, EOU/EOB bitmask

    SRP->>WLIB: `parakeet_capi_stream_drain_words(stream)`

    WLIB-->>SRP: finalized words + timing/confidence

    SRP-->>SRC: OnRecognitionResult(text, final)

    SRC-->>BE: Result callback

    BE-->>SR: Dispatch result event

    Note over JS: recognized text fragments received by script

    SR-->>JS: SpeechRecognitionResult

end

```

## Open Issues / Not Quite Done / Limitations

### Spec

The Web Speech specification is still ambiguous in areas that matter for this

implementation, especially availability/install semantics, lifecycle ordering

around `abort()`/`stop()`/restart, and event timing. The implementation follows

the current interoperable behavior where practical and keeps remaining

mismatches localized.

Lifecycle behavior has automated coverage for async end ordering,

start-after-error, abort, and session cleanup. DOM event timestamps (`start`,

`audiostart`, `result`, etc.) are now surfaced: `PSpeechRecognition` sends a

`TimeStamp` alongside results, and `SpeechRecognitionParent` sets it from

`TimeStamp::Now()` at emission time. Per-word/per-token timing remains

engine-internal and is not exposed on the DOM event, since the Web Speech result

has no per-word timing field.

### Testing

Testing is automated through mochitests, gtests, and WPT expectation updates.

Current coverage includes availability/install flows, the download permission

prompt, lifecycle and fuzz/interleaving tests, aborted-session cleanup, result

confidence and event timing, real Parakeet e2e coverage, multilingual

recognition, and follow-up phrase-boost coverage.

Remaining gaps are additional upstream WPT automation and broader real-model

scenario coverage.

Running the mochitests locally (headless, Linux):

```

./mach mochitest --headless dom/media/webspeech/recognition/test/

```

- Headless `AudioContext`s stay suspended without a running audio server:

  PipeWire + pipewire-pulse + wireplumber must be running with

  `XDG_RUNTIME_DIR` set (already the case in a normal desktop session; only

  needs starting manually in a bare CI-like environment).

- Tests tagged `parakeet-asr` in `mochitest.toml` need the real Parakeet model.

  `testing/mochitest/runtests.py` auto-starts `testing/tools/serve_model.py`

  (a local stand-in model hub on port 8766) for the run whenever such a test

  is active; nothing needs to be started by hand for a normal

  `./mach mochitest` invocation on this directory.

- `browser.ml.modelHub.testing` mocks `IsModelAvailable` in

  `HWInferenceParent` and the parent-side download + already-installed check

  (`SpeechModelDownloadPermissionRequest.cpp`), plus model retrieval in

  `SpeechRecognitionParent::RecvInit`, so start()-heavy tests (fuzzing, session

  lifecycle) never need a real model file. It does **not** mock `GetModelFile`

  itself: there's no lightweight stand-in for an actual

  parseable model, so a test wanting to exercise real recognition still needs

  the local model server above.

- `MOZ_LOG=SpeechRecognitionParent:5,SpeechRecognitionBackend:5,SpeechRecognition:5`

  is the fastest way to see the IPC/session lifecycle across all three

  layers when a test misbehaves.

### `"speechstart"`/`"speechend"` events

Content-side callbacks are wired (`SpeechRecognitionChild::RecvOnSpeechChange` →

backend → DOM), but `SpeechRecognitionParent` never calls

`SendOnSpeechChange(...)`. One remaining task is to add some code to use a

minuscule VAD (Voice Activity Detection) model and get timing of speech

start/end.

### Global concurrency limit scope

The system currently only supports a single active session via static

`sActiveSession` across the entire `HWInference` process. Chrome does the same.

We will be able to relax this when we understand better the performance story

when there is no hardware acceleration. Having a bunch of recognition sessions

running concurrently is fine when there is hardware acceleration, granted the

same model is used for all session (or there is otherwise enough memory

available).

### Error handline / propagation

`HandleRecognitionErrorFromBackend` maps only `"concurrent-session"` to

`service-not-allowed`, defaulting others to `network`. This will be expanded

and clarified.

### Language→model mapping

`dom/media/webspeech/recognition/models.yaml` is an ordered list of models, each

declaring the locales it recognizes, spelled the way that model expects them

(for the multilingual model, its prompt dictionary keys). A request picks the

first entry that recognizes it, negotiated with

`LocaleService::NegotiateLanguages`, so `es-MX` is recognized as the model's

generic `es`; there is no catch-all, and a tag no entry recognizes is simply not

supported. `media.webspeech.recognition.model.<language subtag>` restricts the

choice to the model it names. A request with no language at all takes the first

entry and passes no locale, leaving the engine to its model file's own default.

I plan to add more models (much smaller, more specialized with different

variants, etc.) prior to landing.

`SpeechModelFor` in `SpeechRecognitionModelMapping.{h,cpp}` answers both, over

a table generated from the yaml: the model id and the locale, returned together

as a match, or no match when the language is unsupported. The Utility process

calls it for `IsModelAvailable`/`IsModelInstalled`/`InstallModels`, so a language

becomes an id before it crosses to the main process; the main process works only

from the coordinates it receives, using `SpeechModelSizeMB` for the prompt's

download size. A session is started with the negotiated locale rather than the

tag content asked for.

`available()` reports `unavailable` and `install()` resolves false for an

unrecognized language, without launching the HWInference process or downloading

anything, and `start()` fires `service-not-allowed`.

### Phrase boost

Phrase boost is implemented in a follow-up patch. The base architecture passes

`SpeechRecognitionPhraseIPC` entries over `PSpeechRecognition::Init`; the

follow-up wires those hints into `parakeet.cpp` and adds dedicated coverage.

### Hardware acceleration on non-macOS

Still needs to be done, CPU-based inference works well though.

Source code

Revision control

Copy as Markdown

Other Tools