v1.7.2

[ggerganov]

Overview

• Various improvements in the Metal backend
• Fix extra memory usage for large samples
• Remove limit for ggml_context (i.e. more beams and processors are supported)

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M2 Ultra	METAL	tiny	1	1	9.51	1.39	0.41	0.01	83ac284
M2 Ultra	METAL	tiny-q5_0	1	1	9.57	1.41	0.42	0.01	83ac284
M2 Ultra	METAL	tiny-q5_1	1	1	8.74	1.39	0.42	0.01	83ac284
M2 Ultra	METAL	tiny-q8_0	1	1	8.36	1.33	0.41	0.01	83ac284
M2 Ultra	METAL	base	1	1	14.27	1.90	0.63	0.02	83ac284
M2 Ultra	METAL	base-q5_0	1	1	15.50	1.90	0.65	0.02	83ac284
M2 Ultra	METAL	base-q5_1	1	1	15.67	1.88	0.65	0.02	83ac284
M2 Ultra	METAL	base-q8_0	1	1	14.69	1.81	0.63	0.02	83ac284
M2 Ultra	METAL	small	1	1	40.85	3.77	1.43	0.05	83ac284
M2 Ultra	METAL	small-q5_0	1	1	45.99	3.90	1.52	0.05	83ac284
M2 Ultra	METAL	small-q5_1	1	1	46.19	3.83	1.50	0.06	83ac284
M2 Ultra	METAL	small-q8_0	1	1	42.90	3.65	1.46	0.05	83ac284
M2 Ultra	METAL	medium	1	1	109.01	7.59	3.24	0.11	83ac284
M2 Ultra	METAL	medium-q5_0	1	1	126.78	7.55	3.45	0.13	83ac284
M2 Ultra	METAL	medium-q5_1	1	1	127.71	7.39	3.43	0.13	83ac284
M2 Ultra	METAL	medium-q8_0	1	1	115.97	7.21	3.35	0.12	83ac284
M2 Ultra	METAL	medium-dis	1	1	97.74	1.06	0.36	0.01	83ac284
M2 Ultra	METAL	large-v2	1	1	196.99	11.29	5.06	0.20	83ac284
M2 Ultra	METAL	large-v2-q5_0	1	1	233.88	10.83	5.56	0.24	83ac284
M2 Ultra	METAL	large-v2-q5_1	1	1	234.03	10.73	5.46	0.24	83ac284
M2 Ultra	METAL	large-v2-q8_0	1	1	210.83	10.29	5.23	0.22	83ac284
M2 Ultra	METAL	large-v2-dis	1	1	175.37	1.18	0.42	0.02	83ac284
M2 Ultra	METAL	large-v3-turbo	1	1	177.35	1.85	0.73	0.03	83ac284
M2 Ultra	METAL	large-v3-turbo-q5_0	1	1	209.31	1.69	0.80	0.04	83ac284
M2 Ultra	METAL	large-v3-turbo-q8_0	1	1	189.55	1.64	0.75	0.03	83ac284

What's Changed

• Added OpenVino init on state by @sandrohanea in Added OpenVino init on state #2464
• Updating the Quick start by @stsfaroz in Updating the Quick start #2475
• max_length from max_target_positions by @CrispStrobe in max_length from max_target_positions #2477
• Add dtw preset for large-v3-turbo by @rotemdan in Add dtw preset for large-v3-turbo #2481
• make : fix GGML_VULKAN=1 build by @ggerganov in make : fix GGML_VULKAN=1 build #2485
• Add Vulkan notice in README.md by @toboil-features in Add Vulkan notice in README.md #2488
• Fix Ruby binding building by @KitaitiMakoto in Fix Ruby binding building #2484
• Update of README.md by @toboil-features in Update of README.md #2489
• whisper: fix index overflow by @Josscii in whisper: fix index overflow #2505
• ruby : Add Metal support by @KitaitiMakoto in ruby : Add Metal support #2516
• ruby: New segment callback by @KitaitiMakoto in ruby: New segment callback #2506
• ruby : add more APIs by @KitaitiMakoto in ruby : add more APIs #2518
• ruby: fix installation test by @KitaitiMakoto in ruby: fix installation test #2519
• When DTW timestamps are enabled, defer new_segment_callback until after DTW compute step by @jettoblack in When DTW timestamps are enabled, defer new_segment_callback until after DTW compute step #2515
• ci : fix openblas build by @ggerganov in ci : fix openblas build #2511
• whisper : reduce ggml_context usage by @ggerganov in whisper : reduce ggml_context usage #2525
• sync : ggml by @ggerganov in sync : ggml #2528
• passing samples_padded by ref to the threads. by @vinmisra in passing samples_padded by ref to the threads. #2534
• fix ffmpeg v5 build by @stsydow in fix ffmpeg v5 build #2543
• fix: ggml-vulkan logs by @thewh1teagle in fix: ggml-vulkan logs #2547
• Fix the instructions on the Ruby binding by @wilsonsilva in Fix the instructions on the Ruby binding #2548
• whisper.swiftui : add model download list & bench methods by @jhen0409 in whisper.swiftui : add model download list & bench methods #2546
• ruby : Add more API by @KitaitiMakoto in ruby : Add more API #2551
• Fix building workflow for linux/arm64 container by @rai62 in Fix building workflow for linux/arm64 container #2555
• sync : ggml by @ggerganov in sync : ggml #2561
• whisper.swiftui : switch Mac dest to Mac (Designed for iPad) by @jhen0409 in whisper.swiftui : switch Mac dest to Mac (Designed for iPad) #2562
• ci : use local ggml by @ggerganov in ci : use local ggml #2567
• sycl: fix example build by @stsydow in sycl: fix example build #2570

New Contributors

• @stsfaroz made their first contribution in Updating the Quick start #2475
• @CrispStrobe made their first contribution in max_length from max_target_positions #2477
• @toboil-features made their first contribution in Add Vulkan notice in README.md #2488
• @KitaitiMakoto made their first contribution in Fix Ruby binding building #2484
• @Josscii made their first contribution in whisper: fix index overflow #2505
• @jettoblack made their first contribution in When DTW timestamps are enabled, defer new_segment_callback until after DTW compute step #2515
• @vinmisra made their first contribution in passing samples_padded by ref to the threads. #2534
• @stsydow made their first contribution in fix ffmpeg v5 build #2543
• @wilsonsilva made their first contribution in Fix the instructions on the Ruby binding #2548
• @rai62 made their first contribution in Fix building workflow for linux/arm64 container #2555

Full Changelog: v1.7.1...v1.7.2

---

This discussion was created from the release v1.7.2.

[naltmann]

Windows x64 build seems to have failed. Windows cublas build works great though.

[ggerganov]

I noticed, but I don't know how to fix it. PRs welcome.

[sandrohanea]

I think it is a bug in VS2022 17.12 and created: https://developercommunity.visualstudio.com/t/Building-whispercpp-with-latest-VS-caus/10796514

I reproduce the build error locally as well and it also reproduces for previous whisper.cpp versions (e.g. 1.7.1).
Not sure exactly which part of whisper.cpp is causing CL.exe to crash.

If building in debug mode or with older VS, it works as expected.
Here I built windows version for 1.7.2 whisper.cpp inside whisper.net repo, using windows-2019 agent:
https://github.com/sandrohanea/whisper.net/actions/runs/11950394010/job/33311788425

But once I tried to use windows-latest the build crashes with the same exception:
https://github.com/sandrohanea/whisper.net/actions/runs/11950412833/job/33311845173

As a temporary workaround, for whisper.cpp, we can change it to use windows-2019, but I would expect some fix in CL.exe for 17.12.

The issue reproduces when building ggml-cpu on x86 / x64 CPUs (so it is something in that). Cuda / Arm64 and other flavours are not affected.

[sandrohanea]

I see that Llama.cpp is also using VS 16 (2019) https://github.com/ggerganov/llama.cpp/actions/runs/11949299293/job/33308488930
This is why that one builds, but this one is not.

The name of the job is windows-latest-cmake but it is actually using windows-2019:
https://github.com/ggerganov/llama.cpp/blob/1bb30bf28cb5a7adf111bc41c935bdaf128397e7/.github/workflows/build.yml#L731C1-L731C26

[BVLLC1]

Is the latest version working with the latest version of Python?