Distil-Whisper Models: 6 times speed up and 49% smaller size?

[lukestanley]

Tomorrow, HuggingFace's team are set to release their distilled Whisper models, which claim to be "6 times faster, 49% smaller, and perform within 1% WER on out-of-distribution evaluation sets." You should be able to find them here tomorrow: HuggingFace Collection and GitHub.
If they're in PT format, I assume that this script should do the trick, perhaps better minds can confirm tomorrow: Convert PT to GGML Script. This should bring about even faster speed improvements. For those of us (like me) with limited hardware, we might get high-quality ASR on our own devices that's faster than real-time, and doesn't rely on network connectivity. So, tomorrow should be a good day!

[ggerganov]

Exciting!

Edit: initial support has been added in this PR - #1424

Please note that the current implementation is not optimal since it does not support the proposed chunking strategy (see #1414 (reply in thread) for more info). Therefore, the transcription quality using whisper.cpp compared to the original distil-whisper transformers implementation can be lower in some cases.

[bobqianic]

It looks like the initial release is limited to English since the training set consisted solely of English audio. Check this out for further information: huggingface/distil-whisper#6 (comment)

[bobqianic]

In README the author mentioned:

Distil-Whisper is a distilled version of Whisper for English speech recognition that is 6 times faster, 49% smaller, and performs within 1% word error rate (WER) on out-of-distribution evaluation sets. Multilingual support will be provided soon through distillation training code.

[jordibruin]

They should be released now. Curious to see performance metrics 🙂

[AdithyanI]

They are released : https://www.linkedin.com/posts/sanchit-gandhi_distil-whisper-is-now-available-in-transformers-activity-7125902709794189313-aumY?utm_source=share&utm_medium=member_desktop

Anyone already ran the model? Would love to see some results :)
I am not deep into this yet, otherwise would do it myself.

[fblissjr]

Tested it with speculative decoding per HF's example on the model page (https://huggingface.co/distil-whisper/distil-large-v2). Pretty happy with results.

[bobqianic]

I've tested with whisper.cpp, and it seems that the automatic language detection isn't working—the distilled model can't correctly identify the language used in the audio.

[patrickvonplaten]

The models are all English-only - we should probably clarify this better in the model card.
~co-author

[AdithyanI]

How do I convert this to something whisper.cpp can use?
I tried following this : https://github.com/ggerganov/whisper.cpp/tree/master/models

But I cannot find any .pt files in here : https://huggingface.co/distil-whisper/distil-medium.en/tree/main

[lukestanley]

Yeah I get the same error. To be clear, the standard "make tiny.en" worked fine when ran in the same way as the converted model.

Steps to reproduce with logs:

mkdir fast_asr
cd fast_asr/
git clone https://github.com/openai/whisper
git clone https://github.com/ggerganov/whisper.cpp
git clone https://huggingface.co/distil-whisper/distil-medium.en
python3 ./whisper.cpp/models/convert-h5-to-ggml.py ./distil-medium.en/ ./whisper .
cd whisper.cpp
make
make tiny.en
user@computer:~/fast_asr/whisper.cpp$ ./main -m /home/user/fast_asr/whisper.cpp/models/ggml-tiny.en.bin -f samples/jfk.wav -ml 16
whisper_init_from_file_no_state: loading model from '/home/user/fast_asr/whisper.cpp/models/ggml-tiny.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab = 51864
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 384
whisper_model_load: n_text_head = 6
whisper_model_load: n_text_layer = 4
whisper_model_load: n_mels = 80
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 1
whisper_model_load: adding 1607 extra tokens
whisper_model_load: model ctx = 73.62 MB
whisper_model_load: model size = 73.54 MB
whisper_init_state: kv self size = 2.62 MB
whisper_init_state: kv cross size = 8.79 MB
whisper_init_state: compute buffer (conv) = 11.17 MB
whisper_init_state: compute buffer (encode) = 61.76 MB
whisper_init_state: compute buffer (cross) = 3.67 MB
whisper_init_state: compute buffer (decode) = 18.82 MB

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | COREML = 0 | OPENVINO = 0 |

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...

[00:00:00.000 --> 00:00:00.940] And so my
[00:00:00.940 --> 00:00:01.750] fellow
[00:00:01.750 --> 00:00:04.000] Americans ask
[00:00:04.000 --> 00:00:05.970] not what your
[00:00:05.970 --> 00:00:06.880] country can do
[00:00:06.880 --> 00:00:08.000] for you
[00:00:08.000 --> 00:00:09.030] ask what you
[00:00:09.030 --> 00:00:10.260] can do for your
[00:00:10.260 --> 00:00:11.000] country.

whisper_print_timings: load time = 100.34 ms
whisper_print_timings: fallbacks = 0 p / 0 h
whisper_print_timings: mel time = 21.30 ms
whisper_print_timings: sample time = 10.52 ms / 27 runs ( 0.39 ms per run)
whisper_print_timings: encode time = 683.69 ms / 1 runs ( 683.69 ms per run)
whisper_print_timings: decode time = 87.84 ms / 27 runs ( 3.25 ms per run)
whisper_print_timings: prompt time = 0.00 ms / 1 runs ( 0.00 ms per run)
whisper_print_timings: total time = 947.55 ms
user@computer:~/fast_asr/whisper.cpp$ ./main -m /home/user/fast_asr/ggml-model.bin -f samples/jfk.wav -ml 16
whisper_init_from_file_no_state: loading model from '/home/user/fast_asr/ggml-model.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab = 51864
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 1024
whisper_model_load: n_audio_head = 16
whisper_model_load: n_audio_layer = 24
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 1024
whisper_model_load: n_text_head = 16
whisper_model_load: n_text_layer = 2
whisper_model_load: n_mels = 80
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 4
whisper_model_load: adding 1607 extra tokens
whisper_model_load: model ctx = 756.86 MB
whisper_model_load: model size = 756.66 MB
whisper_init_state: kv self size = 3.50 MB
whisper_init_state: kv cross size = 11.72 MB
whisper_init_state: compute buffer (conv) = 25.82 MB
whisper_init_state: compute buffer (encode) = 162.24 MB
whisper_init_state: compute buffer (cross) = 7.33 MB
whisper_init_state: compute buffer (decode) = 47.74 MB

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | COREML = 0 | OPENVINO = 0 |

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...

whisper_print_timings: load time = 572.91 ms
whisper_print_timings: fallbacks = 0 p / 0 h
whisper_print_timings: mel time = 18.31 ms
whisper_print_timings: sample time = 0.87 ms / 2 runs ( 0.44 ms per run)
whisper_print_timings: encode time = 14216.68 ms / 1 runs (14216.68 ms per run)
whisper_print_timings: decode time = 15.56 ms / 2 runs ( 7.78 ms per run)
whisper_print_timings: prompt time = 0.00 ms / 1 runs ( 0.00 ms per run)
whisper_print_timings: total time = 14897.49 ms

[lukestanley]

@bobqianic what did you do differently to make your working ggml model? I wonder if there is a hidden dependency issue, e.g with Transformers?

[bobqianic]

what did you do differently to make your working ggml model?

I'm using the distilled large-v2, not the medium.en. I just did a quick test and it seems I'm not able to get any output from the medium.en either.

[patrickvonplaten]

If it's still relevant we also added the checkpoints in original OAI format here:

• https://huggingface.co/distil-whisper/distil-large-v2/blob/main/original-large-32-2-en.bin
• https://huggingface.co/distil-whisper/distil-medium.en/blob/main/original-medium-24-2-en.bin

[patrickvonplaten]

.cpp checkpoints have also been added here:
[FP32]

• a) https://huggingface.co/distil-whisper/distil-large-v2/blob/main/ggml-large-32-2.fp32.en.bin
• b) https://huggingface.co/distil-whisper/distil-medium.en/blob/main/ggml-medium-32-2.en.fp32.bin
  [FP16]
• a) https://huggingface.co/distil-whisper/distil-large-v2/blob/main/ggml-large-32-2.en.bin
• b) https://huggingface.co/distil-whisper/distil-medium.en/blob/main/ggml-medium-32-2.en.bin

[bobqianic]

I ran a few initial tests using whisper.cpp and it seems that even when using the distil-large-v2 model, there's an issue with correctly transcribing audio that's not in English. The output from the model is garbage. I'm not entirely sure whether this is an issue with the whisper.cpp code or with the model itself. @ggerganov

Japanese:

Chinese:

English:

[ggerganov]

So I am reading the paper and I realize 2 things:

• these models do not have timestamp prediction capability
• they require new custom chunk-based transcription strategy

Adding the no_timestamp to the initial sequence fixes both medium and large distill, though the quality at the chunk boundaries is obviously degraded:

diff --git a/whisper.cpp b/whisper.cpp
index ccac6aa..9e00513 100644
--- a/whisper.cpp
+++ b/whisper.cpp
@@ -3942,6 +3942,7 @@ static void whisper_process_logits(
         // suppress task tokens
         logits[vocab.token_translate]  = -INFINITY;
         logits[vocab.token_transcribe] = -INFINITY;
+        logits[vocab.token_prev]       = -INFINITY;
 
         if (params.logits_filter_callback) {
             params.logits_filter_callback(&ctx, &state, tokens_cur.data(), tokens_cur.size(), logits.data(), params.logits_filter_callback_user_data);
@@ -4560,7 +4561,8 @@ int whisper_full_with_state(
 
     // these tokens determine the task that will be performed
     std::vector<whisper_token> prompt_init = { whisper_token_sot(ctx) };
-    if (whisper_is_multilingual(ctx)) {
+    //if (whisper_is_multilingual(ctx)) { // large-distill
+    if (true) {                           // medium.en-distill
         const int lang_id = whisper_lang_id(params.language);
         state->lang_id = lang_id;
         prompt_init.push_back(whisper_token_lang(ctx, lang_id));
@@ -4569,6 +4571,7 @@ int whisper_full_with_state(
         } else {
             prompt_init.push_back(whisper_token_transcribe(ctx));
         }
+        prompt_init.push_back(whisper_token_not(ctx));
     }
 
     int seek = seek_start;

make -j && ./main -m models/ggml-medium.en-distill.bin -f samples/gb0.wav -nt -pc -ps

@ficapy Since you already have the python stuff setup, could you please run the medium distill version on the samples/gb0.wav sample audio in this repo (you might have to run make samples to download it, if you don't have it) and show me the transcription output that is produced?

[ggerganov]

@sanchit-gandhi Thanks, I just reverse-engineered that the missing no_timestamps token is necessary. However, whisper.cpp is currently lacking the chunking strategy with stride, so I'm not sure how to work around this. There will be degraded quality with the existing implementation

[ficapy]

@ggerganov The results using transformer are as follows.

Good morning. This Tuesday is election day. After months of spirited debate and vigorous campaigning, the time has come for Americans to make important decisions about our nation's future. I encourage all Americans to go to the polls and vote. Election season brings out the spirit of competition between our political parties. And that competition is an essential part of a healthy democracy. But as the campaigns come to a close, Republicans, Democrats, and independents can find common ground on at least one point. Our system of representative democracy is one of America's greatest strengths. The United States was founded on the belief that all men are created equal. Every election day, millions of Americans of all races, religions, and backgrounds step into voting booths throughout the nation. Whether they are rich or poor, old or young, each of them has an equal share in choosing the path that our country will take. And every ballot they cast is a reminder that our founding principles are alive and well. Voting is one of the great privileges of American citizenship, and it has always required brave defenders. As you head to the polls next week, remember the sacrifices that have been made by generations of Americans in uniform to preserve our way of life. From Bunker Hill to Baghdad, the men and women of American Armed Forces have been devoted guardians of our democracy. All of us owe them and their families a special debt of gratitude on election day. Americans should also remember the important example that our elections set throughout the world. Young democracies from Georgia and Ukraine to Afghanistan and Iraq can look to the United States for proof that self-government can endure, and nations that still live under tyranny and oppression can find hope and inspiration in our commitment to liberty. For more than two centuries Americans have demonstrated the ability of free people to choose their own leaders. Our nation has flourished because of its commitment to trusting the wisdom of our citizenry. In this year's election, we will see this tradition continue. And we will be reminded once again that we are blessed to live in a free nation guided by the will of the people. Thank you for listening.

Additionally, according to the author's reply, it seems that timestamps can be generated. huggingface/distil-whisper#13 (comment)

[patrickvonplaten]

Timestamps can also be generated with distil-whisper! In Transformers all you need to do is to set return_token_timestaps=True for generate: https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperForConditionalGeneration.generate.return_token_timestamps

[patrickvonplaten]

In our experiments adding timestamps however didn't improve transcription quality for distil-whisper (even for long-form evaluations).

So we recommend adding timestamps only if you actually need to know the time, not if you're trying to improve quality this way.

[patrickvonplaten]

Amazing that you're already on it! Linking the other issue here: #1423