Is it possible to disable the word normalization? #1883

[silvioprog]

Hi.

I have a couple of audios that the authors says: "gonna", "wanna", "let's", but I have it transcribed to "going to", "want to", "let us" etc.

Is it possible to disable this word normalization?

In the Python version, seems like it is possible here: https://github.com/openai/whisper/blob/ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab/whisper/normalizers/english.py#L465

Do we have something similar in the whisper.cpp?

TIA for any help!

[bobqianic]

Do we have something similar in the whisper.cpp?

AFAIK, whisper.cpp transcribes the text as it is, without any word normalization (as of February 21, 2024).

[silvioprog]

Yes I thougth the same, but please notice this transcription below:

% ./main --no-prints --language en --model ~/Downloads/ggml-large-v2-q5_0.bin --file ~/Downloads/audio.wav

In the transcribed result, we will see a going word at [00:08:51.520 --> 00:08:54.560] I get more excited about choosing what taco I'm going to get, but it should be gonna.

I have been using this web site to do the same transcription using the same model (large-v2), and it transcribed accordingly. For example, its result was: (8:51) I get more excited about choosing what taco I'm gonna get.

This is the link to download the audio. And the transcript from `whisper.cpp: whisper.cpp-transcript.txt.

I really would like to keep the words with their original accents without normalizations.

[misutoneko]

Just a quick observation...
I tested your sample with the original openai whisper (small model) and the --suppress_tokens 50364 trick gave me the "gonna" variation.
"I'm going to" is a hallucination that's sometimes seen with non-speech portions unless the --suppress_tokens trick is used.
EDIT: I didn't notice the same effect with "wanna" vs. "want to".
EDIT2: This seems to be model-specific...tried it with medium model and always got "I'm going to". Too bad.

[misutoneko]

Well, you could try that. I just haven't tried that approach. I kinda just randomly stumbled on this stuff and then used whatever works :D
And I'm not sure how much of it can be applied to whisper.cpp anyway.

[silvioprog]

@misutoneko I ran the ./main -h command and couldn't see any parameter to inform suppressed tokens. 😞

@bobqianic I've tried with no-timestamps, but the problem persisted. 😞

[misutoneko]

Yeah there isn't such parameter readily available, it requires a one line change in the source code and then recompilation.
But it looks as though this is not as effective in whisper.cpp as in the python variants, or at least I couldn't get it to handle the non-speech stuff on my previous try.
There is some work going on that may help, so maybe just wait a bit and retry after certain PRs have been merged :D
(hey, no pressure @bobqianic ;)

[christopher-kapic]

Yeah there isn't such parameter readily available, it requires a one line change in the source code and then recompilation.

But it looks as though this is not as effective in whisper.cpp as in the python variants, or at least I couldn't get it to handle the non-speech stuff on my previous try.

There is some work going on that may help, so maybe just wait a bit and retry after certain PRs have been merged :D

(hey, no pressure @bobqianic ;)

Which PRs?

[misutoneko]

There's a bunch of them in the "Fix the decoding" series by bobqianic...still pending btw.
#1768
#1854

[misutoneko]

Looks like suppression doesn't help much, but what about prompting?
What I mean is, list all the possible variants in prompt.
(Just an idea, I haven't used prompting in this manner myself. But maybe?)

EDIT: Tested this idea a bit... yeah actually seems promising, but it only works with very short clips.
(This has been my experience in the past as well.)

I used these samples for testing (these are smaller sections cut out of the previous sample):
samples.zip