Skip to content

Latest commit

 

History

History
31 lines (24 loc) · 3.24 KB

README.md

File metadata and controls

31 lines (24 loc) · 3.24 KB
Raw

SEPIA LLM Experimets

SEPIA LLM experiments is a repository to explore LLM application in the SEPIA assistant framework. The focus is on locally run, fast performing LLMs, no Open-AI or Google servers etc.

Components

  • SEPIA Web UI for LLM experiments
  • llama.cpp server to host models

Resources

  • llama.cpp - The legendary tool that basically kickstarted on-device LLM experiments by Georgi Gerganov. Includes llama-server, an easy way to host open LLMs in your network.
  • Meta's LLaMA - One of the most popular LLMs that is available in many sizes and versions. LLaMA 3.1 8B is going to be our work horse in many cases, due to performance and license.
    • LLaMA 3.1 8B Instruct - The original 3.1 8B model, hosted on Huggingface. Amazing performance for the size and ok-ish license. Has tool support.
    • LLaMA 3.2 3B Instruct - The smaller 3.2 3B model. Good performance, less resources, worse license ^^.
  • Google's Gemma 2 - Interesting models in different sizes from Google.
    • Gemma 2 9B it - A small 9B model with very good performance and license (Apache 2.0? I'm not sure tbh).
    • Gemma 2 2B it - An even smaller 2B model with solid performance for very restricted devices (Raspberry Pi 5 etc.).
  • Mistral 7B - An open 7B model with small size, very good performance and Apache 2.0 license.
  • Mistral Nemo 12B - An open 12B model with small size, very good performance, support for tools and Apache 2.0 license.
    • Mistral 7B Instruct v0.3 - The instruction tuned 12B model. Solid, fast and open. A cooperation between Mistral and Nvidia.
  • Microsoft's Phi 3 - Interesting models in different sizes and formats from Microsoft.
    • Phi 3 3.8B - Phi 3 mini with 3.8B parameters, in GGUF format, directly from Microsoft.
  • OLMo 7B - A 7B model form Ai2, that was trained with completely transparent and open training data.
  • TinyLlama 1.1B - A very small, open LLM with 1.1B parameters. Good for experiments on Raspberry Pi etc.

PLEASE NOTE: Before you use any of the mentioned models in a commercial environment please check the licenses carefully! It is a bit hard to find out the details sometimes!