SEPIA LLM experiments is a repository to explore LLM application in the SEPIA assistant framework. The focus is on locally run, fast performing LLMs, no Open-AI or Google servers etc.
- SEPIA Web UI for LLM experiments
- llama.cpp server to host models
- llama.cpp - The legendary tool that basically kickstarted on-device LLM experiments by Georgi Gerganov. Includes
llama-server, an easy way to host open LLMs in your network. - Meta's LLaMA - One of the most popular LLMs that is available in many sizes and versions. LLaMA 3.1 8B is going to be our work horse in many cases, due to performance and license.
- LLaMA 3.1 8B Instruct - The original 3.1 8B model, hosted on Huggingface. Amazing performance for the size and ok-ish license. Has tool support.
- LLaMA 3.2 3B Instruct - The smaller 3.2 3B model. Good performance, less resources, worse license ^^.
- Google's Gemma 2 - Interesting models in different sizes from Google.
- Gemma 2 9B it - A small 9B model with very good performance and license (Apache 2.0? I'm not sure tbh).
- Gemma 2 2B it - An even smaller 2B model with solid performance for very restricted devices (Raspberry Pi 5 etc.).
- Mistral 7B - An open 7B model with small size, very good performance and Apache 2.0 license.
- Mistral 7B Instruct v0.3 - The instruction tuned 7B v0.3 model. Solid, fast and open.
- Mistral Nemo 12B - An open 12B model with small size, very good performance, support for tools and Apache 2.0 license.
- Mistral 7B Instruct v0.3 - The instruction tuned 12B model. Solid, fast and open. A cooperation between Mistral and Nvidia.
- Microsoft's Phi 3 - Interesting models in different sizes and formats from Microsoft.
- Phi 3 3.8B - Phi 3 mini with 3.8B parameters, in GGUF format, directly from Microsoft.
- OLMo 7B - A 7B model form Ai2, that was trained with completely transparent and open training data.
- OLMo 7B Instruct - The 7B instruction tuned model.
- TinyLlama 1.1B - A very small, open LLM with 1.1B parameters. Good for experiments on Raspberry Pi etc.
- TinyLlama 1.1 B Chat v1.0 - A fine tuned version for chats (instruction).
PLEASE NOTE: Before you use any of the mentioned models in a commercial environment please check the licenses carefully! It is a bit hard to find out the details sometimes!