Word level time stamps with Whisper 1.6.2 / DTW off - solutions?

[vade]

Hello

Firstly WhisperCPP has been awesome to work with. Im using the Swift API and am experimenting with the word level time stamps.

Im noticing that the timing is quite off, and im curious how the params can be best configured?

Some assumptions on my side:

• Im using Whisper Base.en model file from releases ago. Im assuming I don't need to update the models to get DTW to work?
• Ive had whisper_full_params configured to reduce repetition - per advice in earlier threads. My config prior to enabling DTW was:

        self.params = whisper_full_default_params(WHISPER_SAMPLING_GREEDY)

"en".withCString { en in
           self.params.print_realtime = false
           self.params.print_progress = false
           self.params.print_timestamps = false
           self.params.print_special = false
           self.params.translate = false
           self.params.detect_language = false
           
           // avoid a crash due to shit memory management by yours truly
           if let dupedLeakedEnCStringBullshit = strdup(en)
           {
               self.params.language = UnsafePointer(dupedLeakedEnCStringBullshit)
           }

           self.params.offset_ms = 0
           self.params.no_context = true
           self.params.single_segment = false
           self.params.speed_up = false
           self.params.suppress_blank = true
           self.params.suppress_non_speech_tokens = false
           
           // Remove Repetitions:
           // https://github.com/ggerganov/whisper.cpp/issues/896#issuecomment-1569586018
           // https://github.com/ggerganov/whisper.cpp/issues/471
           self.params.entropy_thold = 2.8

           // Remove Repetitions:
           //https://github.com/ggerganov/whisper.cpp/issues/1511#issuecomment-1816706626
           self.params.n_max_text_ctx = 64 // 0 // unclear which is better

To enable DTW, ive added the following the the above config:

            self.params.token_timestamps = true
            self.params.split_on_word = true
            self.params.max_len = 1

and enabled whisper_context_params as follows:

        cparams.use_gpu = true
//        cparams.flash_attn = true
        cparams.dtw_token_timestamps = true
        cparams.dtw_aheads_preset = WHISPER_AHEADS_BASE_EN;

Im noticing that my the word predictions are pretty consistently off, and that in some cases, im getting bogus or odd time stamp values for reported segments.

• Are non speech tokens supported with the DTW timestamping assumptions?
• Does the DTW addition assume any specific entropy_tholdor n_max_text_ctx settings?
• Is enabling Flash Attention expected to reduce accuracy of some results?

Thanks!

[tannisroot]

I am sorry to ask something off-topic, but where did you find documentation on how to use whisper_full_params? I see them referenced in a few places, but I can't find docs on how to actually apply them (is it an envar? or what?).

[vade]

I looked through how the command line app was altering the params and sort of took some guesses :)