Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Using koboldcpp to compare 2 model outputs isn't very efficient. Is there an easy way to load two models at the same time? #1134

Open
Laince1212 opened this issue Sep 22, 2024 · 2 comments

Comments

@Laince1212
Copy link

Laince1212 commented Sep 22, 2024

Sorry I'm a bit of an amateur here, I want to load 2 models at the same time to compare giving them the same exact input and see their outputs are. Currently the only way to do that with Koboldcpp is to load the two models using two instances, use a different port for the second instance, open two browser tabs and copy the input between each after changing the settings for each one to match. It's not very efficient.

I feel like a perfect example would be LMSYS chatbot arena where you can compare two models directly with the same input and same settings to see which outputs better. I feel like if Koboldcpp supported this it would give beginners like me a way to compare models directly and also help anyone that wants to use two models at the same time for any reason not just to compare the two. My request is: A lmsys chatbot arena like UI with the ability to load two models at the same time. Both models should get the same input then either generate outputs one after the other or both generate (probably slower) outputs at the same time.

Finally, Thanks for working on this amazing program, it's the best portable program to run GGUF files and literally the simplest to use. I switched to Koboldcpp after using many other apps and have been using Koboldcpp for months due to its extreme ease of use and portability. I know my request requires alot of work but hopefully it's going to be in a future update for everyone to try out. Having a local chatbot arena koboldcpp feature would be wonderful.

@LostRuins
Copy link
Owner

Hmm not sure how I can implement that easily though. If you are running one model after the other, swapping them out and clicking Retry might work. Otherwise, the dual window thing seems to be the best bet for now

@Laince1212
Copy link
Author

Yeah it's definitely not an easy ask, but anyways thanks for this amazing app. I'm glad that ease of use and portability is the top priority here and it definitely shows. Used it for months now and have had a great experience. Hopefully after the more important issues here, there can be a way to compare models directly. Absolutely marvelous work though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants