Whisper in your browser: how the transcription runs

The model: Whisper, base or small

Whisper is a speech recognition model that OpenAI published in 2022 with its code and weights under the MIT license. It listens to 30 seconds of sound at a time and writes out the words, with a time for each segment. This site offers two sizes of it:

• The smaller model (the default) is Whisper base in whisper.cpp's 5-bit quantization format (q5_1). It is the lighter download.
• The larger model is Whisper small in the 8-bit format (q8_0): a bigger download that needs more memory.

Quantization rounds the model's weights to fewer bits, which makes the files smaller. How the two sizes compare on this engine, per language, is not published yet: we measure before we describe. The language menu lists the languages Whisper can be set to; with automatic detection, the engine picks the language in the first window that has speech and keeps it for the rest of the file.

The engine: whisper.cpp compiled to WebAssembly

The model runs in whisper.cpp, an open-source C/C++ version of Whisper, which we compile to WebAssembly (a program format every current browser can run) with a small interface of our own. There are three builds, and the page tries them in this order:

• WebGPU, when your browser offers a graphics adapter with the features the engine needs. We have not measured this build on a real graphics card yet, so we make no claim about it.
• WebAssembly on several threads, one per logical core your browser reports. Threads need the tab to be cross-origin isolated, which the site's headers provide.
• WebAssembly on one thread, when threads are not available or your browser reports one logical core. It gives the same kind of result.

If a build fails to start, or the WebGPU build fails during a run, the page restarts the engine on the next build and carries on. The line above the transcript names the build that did the work.

What the page downloads, and when

Opening the page downloads only the page and its own files. The engine, the media reader, a model, and the voice detector come from this site when you add your first file. Only one engine build is kept: the WebGPU build takes the place of a CPU build when it starts with a GPU device. Sizes are in MB, where 1 MB is 1,000,000 bytes:

The models come in parts of at most 20 MiB (a MiB is 1,048,576 bytes, so 20 MiB is about 21 MB), because the host serves no single file over 25 MiB. Before the engine uses a part, your browser checks it against its SHA-256 fingerprint, so a damaged download is refused instead of producing a wrong transcript. The verified parts stay in your browser's Cache Storage when it has room for them, so your next file, and your next visit, skip the download.

Why your recording is never uploaded

Every step runs inside your tab, the site has no endpoint that could accept a file, and the page is allowed to connect only to this site. The check that your recording is not uploaded is on the privacy and security page, with the steps to watch the requests yourself.

How a long file is read and transcribed

A one-hour recording is about 115 MB of sound once it is decoded to 16 kHz mono (3,600 seconds x 16,000 samples x 2 bytes, or twice that as the 32-bit numbers the engine works with). So, where the file can be read in slices, the page does not decode the whole file at once. A second worker opens the file with Mediabunny and reads its sound track 30 seconds at a time: Mediabunny decodes compressed sound with your browser's own decoder, and uncompressed sound, as in a WAV file, itself. It hands over the next slice only when the engine is ready for it.

The slices are joined into windows of about three minutes. Each window ends in the middle of the quietest half second in the 15 seconds before the three-minute mark, to put the cut in a pause rather than inside a word, and the windows follow each other with no gap and no overlap. A small voice detector marks the stretches with speech in each window, so the model skips silence instead of writing words over it. Each window is transcribed with the last words before it, at most 200 characters, as a prompt, meant to give the model the sentence the cut may have split, and its times are placed back on the file's own clock.

When a window is done, its paragraphs appear on the page, so you can start reading and searching while the rest runs. When the file cannot be read in slices, because Mediabunny does not open its container or your browser has no decoder that reads its sound in slices, the page decodes the whole file at once with your browser's own decoder instead, without Mediabunny, and then refuses a file over 500 MB or longer than 1 hour.

How the paragraphs and timestamps are made

Whisper returns segments of text with a start and an end time, and a time for each word. The page joins them into paragraphs: a new one starts at a pause of 2 seconds or more, measured from the times of the words on each side, and a paragraph that grows past about 600 characters ends at the next sentence end. There are no speaker labels, so a paragraph break marks a pause or a long stretch of speech, never a change of voice.

The timestamp of a paragraph is the time its first word starts, written as [hh:mm:ss] and rounded down to the second. It is approximate: good for finding a passage in the recording, not for subtitles. Copy text and Download TXT write the same paragraphs, with a blank line between them and the timestamps when they are shown.

Why the time varies

Your device does the work, on whichever build the page could start, so the same file takes a different time on a new laptop and on an old one, and the model and the length of the recording change it too. That is why the page promises no speed. Once the engine reports its first progress, the page measures how many seconds of work each second of sound takes on your device, smooths that over the windows so one slow moment does not swing it, and shows the time left.

What your device needs

• A current browser with WebAssembly. Chrome, Edge, Firefox, and Safari all have it; our automated test runs in Chromium.
• Free memory: the engine holds the model in memory while it runs, and the larger model needs more. We give no figure until we have measured it.
• A connection on every visit: the page needs one to open, and your first file also downloads the engine and the model. There is no offline mode. When your browser has room, it keeps the model, so later files and visits skip that download.

The open-source components of the engine, the models, and the media reader are listed with their licenses on the credits page.

Sources

• OpenAI, Whisper repository: The model's code, its MIT license, and the table of model sizes (base and small).
• Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (2022): The paper that introduces Whisper.
• whisper.cpp: The C/C++ port of Whisper that the engine compiles to WebAssembly (version 1.9.4, with one line added in the multi-threaded build, listed on the credits page), and its quantized model files.
• Silero VAD: The voice activity detector that finds the stretches with speech.
• Mediabunny: The media library that reads your file in slices where it can, decoding compressed sound with the browser's decoder and uncompressed sound itself.
• MDN, AudioDecoder (WebCodecs): The browser's own decoder that turns the compressed sound into samples.
• MDN, CacheStorage: Where the verified model parts are kept between visits.
• web.dev, Making your website cross-origin isolated using COOP and COEP: Why the site sends these headers: WebAssembly threads need SharedArrayBuffer.
• Cloudflare Workers, Limits: The 25 MiB limit per static file, the reason the models come in parts.