LLaMA option instead of GPT-2?

[nundys]

How about an option of your (fantastic) llama.cpp repo to swap out GPT-2?

Edit (Georgi): This is now available via the talk-llama example

[ggerganov]

This is now available via 4a0deb8
See the new talk-llama example
Moving to a discussion

[nacmonad]

Hi -- it appears you've .gitignored the models/* directory.

What are some recommended alternatives to finding the 7B/13B/xB ggml-model-q4_0.bin ?

[gourcetools]

I am unable to build talk-llama following the readme instructons at: https://github.com/ggerganov/whisper.cpp/tree/0a2d1210bcb98978214bbf4e100922a413afd39d/examples/talk-llama#building

bigbrain@bigbrain-hardware:~/Desktop/whisper.cpp/examples/talk-llama$ sudo apt-get install libsdl2-dev
[sudo] password for bigbrain: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libsdl2-dev is already the newest version (2.24.0+dfsg-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
bigbrain@bigbrain-hardware:~/Desktop/whisper.cpp/examples/talk-llama$ make talk-llama
g++     talk-llama.cpp   -o talk-llama
talk-llama.cpp:4:10: fatal error: common.h: No such file or directory
    4 | #include "common.h"
      |          ^~~~~~~~~~
compilation terminated.
make: *** [<builtin>: talk-llama] Error 1

i'm able to build whisper and run it.
Ubuntu 22.10

If i grab .h files from the example folder and paste them into talk-llama , i end up with:

bigbrain@bigbrain-hardware:~/Desktop/whisper.cpp/examples/talk-llama$ make talk-llama
g++     talk-llama.cpp   -o talk-llama
In file included from talk-llama.cpp:5:
common-sdl.h:3:10: fatal error: SDL.h: No such file or directory
    3 | #include <SDL.h>
      |          ^~~~~~~
compilation terminated.
make: *** [<builtin>: talk-llama] Error 1

[jonn26]

I believe you should run "make talk-llama" from the whisper.cpp folder:

➜  whisper.cpp git:(master) make talk-llama
I whisper.cpp build info:
I UNAME_S:  Darwin
I UNAME_P:  arm
I UNAME_M:  arm64
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -DGGML_USE_ACCELERATE
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread
I LDFLAGS:   -framework Accelerate
I CC:       Apple clang version 14.0.0 (clang-1400.0.29.202)
I CXX:      Apple clang version 14.0.0 (clang-1400.0.29.202)

c++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread examples/talk-llama/talk-llama.cpp examples/talk-llama/llama.cpp examples/common.cpp examples/common-sdl.cpp ggml.o whisper.o -o talk-llama `sdl2-config --cflags --libs`  -framework Accelerate```

[gourcetools]

Gosh, i'm so stupid.
Thank you, it worked (of course).

Thanks for helping, gonna download the model and it should work properly.
Because by default, the model provided in the example command is not downloaded, maybe the readme for this example should specify that.

Edit: removed stuff mentioning downloading of a model as it goes against repo rules.
Sorry about that. i forget for a second. Have a nice day.

[gourcetools]

I am now getting a "Segmentation fault (core dumped)". I have to find out what's wrong... using different models don't fix it.

[igorbarshteyn]

@ggerganov I tried this recently, and it's running much slower than the updated version of llama.cpp that you have in your main llama.cpp project. It doesn't work with the newer GGML format models. Also, it seems like the code is looking for the Mac "say" command in speak.sh on line 13, even when you comment that out and uncomment the Espeak command on line 10.

Any chance to resync this up with the main llama.cpp branch and fix the pointer in the code to speak.sh, please?

[ggerganov]

Just updated to latest llama.cpp - the performance should be good

[igorbarshteyn]

Thank you!!!

…

On Monday, April 10, 2023, Georgi Gerganov ***@***.***> wrote: Just updated to latest llama.cpp - the performance should be good — Reply to this email directly, view it on GitHub <#672 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXDDDMNQ73DXNECAWIL7FXDXARUCBANCNFSM6AAAAAAWLCZHCY> . You are receiving this because you commented.Message ID: ***@***.***>

[tomqian2022]

@ggerganov thank you for the awesome development work! I find talk-llama fascinating. Will there be support for buffering llama output by certain length (or punctuation token) to trigger the "say" command whenever the buffering condition is met? This can benefit the realtime use for longer generated text. I'd love to build it but i dont know c++...

[ggerganov]

Would be a good improvement - will definitely add it, but it is low on my TODO list, so it will take time before I get to it. Hopefully someone gives it a shot in the meantime

[tail-recursion]

I get permission denied;

EDIT:
Fixed using chmod 755 speak

Georgi: What are you? LLaMA: I am an artificial intelligence assistant. Georgi:sh: ./examples/talk-llama/speak: Permission denied

on M1 Max 64GB.

Installed SDL2, used make in the main directory and make talk-llama and downloaded the whisper model.

When I try execute say 2 "Hello" using bash it works. Also should the speak file be a .sh file? I saw elsewhere maybe it was previously a .sh file.

[igorbarshteyn]

"chmod +x speak" usually solves that for me

…

On Saturday, July 22, 2023, Dale ***@***.***> wrote: I get permission denied; `Georgi: What are you? LLaMA: I am an artificial intelligence assistant. Georgi:sh: ./examples/talk-llama/speak: Permission denied` — Reply to this email directly, view it on GitHub <#672 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXDDDMPCAOJKIVMNHBW3P2DXRNHZZANCNFSM6AAAAAAWLCZHCY> . You are receiving this because you commented.Message ID: ***@***.***>

[edwios]

Hi,

Is there any plan to add RoPE and METAL support for talk-llama, that is, to update to the latest llama.cpp ? It will be great to see this happen.

[scalar27]

Metal is there, at least I think it is!

[anon3345]

Can anyone help me with this issue im having with this part mines keep saying permission denied im on windows nvidia gpu

Georgi: You
LLaMA: Thank you for asking me that question.
Georgi:sh: 1: ./examples/talk-llama/speak: Permission denied
You
LLaMA: Of course, Georgi. Please tell me more about yourself so I can better understand your needs.
Georgi:sh: 1: ./examples/talk-llama/speak: Permission denied

[jboero]

Not sure if you lack permission to the audio device or the model file but some permission is clearly off. Best to step through a debugger if you haven't solved this yet.

[sunnyjocker]

Not working on Windows with cuda builds, the output never stop and it generate random strings which don't make sense. but if it runs with -ng flag, everything is fine. the model runs fine in llama.cpp project. I wonder that's the problem, i tested 7B, 13B, it's the same. Please help. Thanks in advance.
I run it with the following command:
talk-llama -mw ./models/ggml-small.en.bin -ml ./models/13B/ggml-model-q4_0.gguf -s speak.bat -p "Sunny" -t 6 -ng
run with cuda:

run without cuda:

[gavin1818]

Title: transcription quality deteriorates a lot,
I have been using the same Whisper model for the stream transcription the outcome of the transcription is good. I noticed, however, that the same model's transcription quality has deteriorated significantly in the Talk-Llama project, which produced incorrect transcription when transcribing my speech. I'm curious about the reasons behind this discrepancy.

[ggerganov]

Did it work correctly before?

[thesimpleone]

Trying in WIndows
whisper\talk-llama.exe -mw "..\models-whisper\ggml-small.en.bin" -ml "..\models\Meta-Llama-3.1-8B-Instruct.Q4_0.gguf" -p "Georgi" -t 8 -sf "%~DP0speakfile.txt"

I keep getting errors with speak file, any tips?

Georgi:←[1m Testing one, two, three.
LLaMA:←[0m I am functioning correctly. How may I assist you further?
Georgi:'.' is not recognized as an internal or external command,
operable program or batch file.
speak_with_file: failed to speak