Add Whisper V3

[AdithyanI]

https://openai.com/blog/new-models-and-developer-products-announced-at-devday

Looks like a drop-in replacement and directly work if we convert it.

[julzaa]

Did you manage to actually run the converted model? I successfully ran convert-pt-to-ggml.py but using the .bin conversion doesn't work, I have this error in the debug console:

runFullImpl: failed to generate timestamp token - skipping one second

It's worth noting that the script works flawlessly with large-v2, and that it seems like n_mels increased from 80 to 128 with v3, maybe it's the cause of the issue? I haven't really looked at how these parameters are handled.

[billylo1]

I managed to convert it to ggml with no errors. But testing it with samples/gb0.wav gave me some strange results...

(base) billylo@pascal whisper.cpp % ./main -l auto -f samples/gb0.wav -m models/ggml-large.bin
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5
whisper_model_load: adding 1610 extra tokens
whisper_model_load: model ctx     = 2951.63 MB
whisper_model_load: model size    = 2951.01 MB
whisper_init_state: kv self size  =   70.00 MB
whisper_init_state: kv cross size =  234.38 MB
whisper_init_state: compute buffer (conv)   =   41.85 MB
whisper_init_state: compute buffer (encode) =  202.52 MB
whisper_init_state: compute buffer (cross)  =    8.89 MB
whisper_init_state: compute buffer (decode) =   59.40 MB
whisper_init_state: Metal context initialized
whisper_init_state: max tensor size =   126.63 MB

system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | COREML = 0 | OPENVINO = 0 |

main: WARNING: model is not multilingual, ignoring language and translation options
main: processing 'samples/gb0.wav' (2037686 samples, 127.4 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.040 --> 00:00:08.080]   bluntlyend his sreen;M,
[00:00:08.080 --> 00:00:10.260]   good
[00:00:10.260 --> 00:00:27.340]   spaceases lack ben s other s begin. ofors a byation HollNature Essence machCMag third Val
^[[A[00:00:29.820 --> 00:00:41.460]   th
[00:00:41.460 --> 00:00:59.680]  host a, th, Python good ofem elite');G. can,ri,
^C