1_Inference_CTransformers(GGUF).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a734497c-6379-4d65-a468-e3921f04fe13",
   "metadata": {},
   "source": [
    "# Inference GGUF Model\n",
    "https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e0624d8-de67-4355-bb60-256740d126b4",
   "metadata": {},
   "source": [
    "## Example 1: With CTransformers \n",
    "\n",
    "https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF\n",
    "\n",
    "Install CTransformers\n",
    "```python\n",
    "# with CPU\n",
    "pip install ctransformers>=0.2.24\n",
    "# with CUDA GPU\n",
    "pip install ctransformers[cuda]>=0.2.24\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f7aded80-1907-44b0-beb3-50009e3e68f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: ctransformers[cuda] in /app/venv/lib/python3.10/site-packages (0.2.27)\n",
      "Requirement already satisfied: huggingface-hub in /app/venv/lib/python3.10/site-packages (from ctransformers[cuda]) (0.17.3)\n",
      "Requirement already satisfied: py-cpuinfo<10.0.0,>=9.0.0 in /app/venv/lib/python3.10/site-packages (from ctransformers[cuda]) (9.0.0)\n",
      "Requirement already satisfied: nvidia-cublas-cu12 in /app/venv/lib/python3.10/site-packages (from ctransformers[cuda]) (12.2.5.6)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12 in /app/venv/lib/python3.10/site-packages (from ctransformers[cuda]) (12.2.140)\n",
      "Requirement already satisfied: filelock in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (3.12.4)\n",
      "Requirement already satisfied: fsspec in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (2023.9.2)\n",
      "Requirement already satisfied: requests in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (2.31.0)\n",
      "Requirement already satisfied: tqdm>=4.42.1 in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (4.64.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (6.0.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (4.8.0)\n",
      "Requirement already satisfied: packaging>=20.9 in /app/venv/lib/python3.10/site-packages (from huggingface-hub->ctransformers[cuda]) (23.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /app/venv/lib/python3.10/site-packages (from requests->huggingface-hub->ctransformers[cuda]) (3.3.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /app/venv/lib/python3.10/site-packages (from requests->huggingface-hub->ctransformers[cuda]) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /app/venv/lib/python3.10/site-packages (from requests->huggingface-hub->ctransformers[cuda]) (2.0.5)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /app/venv/lib/python3.10/site-packages (from requests->huggingface-hub->ctransformers[cuda]) (2023.7.22)\n"
     ]
    }
   ],
   "source": [
    "!pip install ctransformers[cuda]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb0cae52-1da3-4087-b171-52632603e600",
   "metadata": {},
   "source": [
    "### Example 1.1: With CTransformers Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "907a6e78-b66b-4704-8c0c-3506dd9d2b22",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Arrrr, shiver me timbers! *adjusts eye patch* As a scurvy dog of the high seas, I have many fond favorites, me hearty. 'Tis a treacherous job narrowin' it down, but here be me top picks:\n",
      "\n",
      "1. Grog: A fine swill o' rum, grog be me lifeblood! It warms the bones and dulls the senses, perfect for a good ol' fashioned pirate's life. *slurs*\n",
      "2. Booty: Ah, the spoils o' war! There be nothing like comin' back to the ship with a hold full o' gold doubloons, jewels, and fine silks. It's like a treasure chest overflowin' with treasures! *grins*\n",
      "3. Sea shanties: Oh, how I love a good sea shanty! The rhythm o' the waves and the singin' o' the crew be a mighty fine thing indeed. *humms* \"What Shall We Do with a Drunken Sailor?\"\n",
      "4. Swashbucklin':\n",
      "338.697631601\n",
      "\n",
      "\n",
      "Pirate name: Captain Blackbeak\n",
      "Favorite food: Rum! I love me some good ol' rum! It's the best part of being a pirate, after all. And it pairs perfectly with... *ahem* other things. *wink wink*\n",
      "Favorite drink: Arrrr, I be lovin' a good grog! It's the perfect way to wash down me treasure and keep me goin' on me adventures. *slurs*\n",
      "Favorite place to visit: Tortuga! It be the best place for a pirate to relax and enjoy some... *ahem* hospitality. And there be plenty of rum to go around, savvy? *winks*\n",
      "Favorite thing to do: Searchin' for treasure! There be nothing better than sailin' the high seas and findin' hidden riches. It be a pirate's life for me! *grins* And maybe... *cough* sometimes I might find other things too... *winks*\n",
      "Favorite song: \"What Shall We Do with a Drunken Sailor?\" It be the perfect pir\n",
      "176.04934105100028\n"
     ]
    }
   ],
   "source": [
    "from ctransformers import AutoModelForCausalLM\n",
    "import os\n",
    "from timeit import timeit\n",
    "\n",
    "model_id=os.path.abspath('./models/Llama-2-7b-Chat-GGUF')\n",
    "\n",
    "# https://github.com/marella/ctransformers#documentation\n",
    "config = {'max_new_tokens': 256, 'repetition_penalty': 1.1}\n",
    "# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.\n",
    "llm = AutoModelForCausalLM.from_pretrained(model_path_or_repo_id=model_id, model_file=\"llama-2-7b-chat.Q4_K_M.gguf\", model_type=\"llama\", \n",
    "                                           # stream=True,\n",
    "                                           gpu_layers=50, **config)\n",
    "\n",
    "\n",
    "# https://replicate.com/blog/how-to-prompt-llama#wrap-user-input-with-inst-inst-tags\n",
    "correct_prompt=\"[INST] <<SYS>>You are a pirate <</SYS>> What's your favorite? [/INST]\"\n",
    "print(timeit(lambda: print(llm(correct_prompt, **config)),number=1)) # In my case: GPU:24s/CPU:42s\n",
    "\n",
    "incorrect_prompt=\"If you are a pirate, What's your favorite?\"\n",
    "print(timeit(lambda: print(llm(incorrect_prompt, **config)),number=1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5254aebd-1574-49fc-8914-96262cb21b7e",
   "metadata": {},
   "source": [
    "### Example 1.2: With LangChain.CTransformers\n",
    "https://python.langchain.com/docs/integrations/llms/ctransformers  \n",
    "https://api.python.langchain.com/en/latest/llms/langchain.llms.ctransformers.CTransformers.html  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4bce3a79-4308-43c5-96b0-99ea91ce2208",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Arrrr, shiver me timbers! *adjusts eye patch* As a swashbucklin' pirate, I've got plenty of faves, matey!\n",
      "\n",
      "First off, I loves me some good ol' fashioned treasure huntin'. There's nothin' like the thrill of searchin' for hidden riches and valuable booty on the high seas. *winks*\n",
      "\n",
      "But I reckon my absolute favorite thing is a good ol' fashioned sea battle! *cracks knuckles* Nothin' gets me goin' like the sound of cannon fire and the smell of gunpowder in the air. There's nothin' quite like the rush of fightin' for me life and me ship against a pack of scurvy dogs! *grins*\n",
      "\n",
      "Of course, I also enjoys me some good food and drink. There's nothin' better than a hearty bowl of sea dog stew after a long day of plunderin', or a mug o' grog to take the edge off after a long battle. *chuckles*\n",
      "\n",
      "So there ye have it\n",
      "43.52970352600005\n",
      "\n",
      "\n",
      "Pirate Name:                   Ahoy matey! Me name be Captain Blackbeak.\n",
      "Favorite Drink:               Arrrr, me hearty! Me favorite drink be grog! It be made o' rum, water, and a wee bit o' lime. Yarrr! It be good for the soul, it be!\n",
      "Favorite Food:                Shiver me timbers! Me favorite food be seafood, matey! Nothing like a good plate o' fish 'n chips or a hearty bowl o' soup t' fill yer belly. Yarrr!\n",
      "Favorite Place:             Ugh! Me least favorite place be landlubbers' paradise! No, no! Me prefer the high seas, matey! The open ocean be me home, where I can sail free and hunt for treasure t' booty. Arrrr!\n",
      "Favorite Pastime:           Aye, matey! Me favorite pastime be singin' sea shanties with mes cap'n an' crew! Yarrr! We sing 'em loud and proud, makin' sure all the landlubbers hear\n",
      "42.63156887800005\n"
     ]
    }
   ],
   "source": [
    "from langchain.llms import CTransformers\n",
    "from timeit import timeit\n",
    "import os\n",
    "\n",
    "model_id=os.path.abspath('./models/Llama-2-7b-Chat-GGUF')\n",
    "\n",
    "# https://github.com/marella/ctransformers#config\n",
    "config = {'max_new_tokens': 256, 'repetition_penalty': 1.1, 'temperature':0.9}\n",
    "# https://api.python.langchain.com/en/latest/llms/langchain.llms.ctransformers.CTransformers.html\n",
    "llm = CTransformers(model=model_id, model_file=\"llama-2-7b-chat.Q4_K_M.gguf\", config=config)\n",
    "\n",
    "correct_prompt=\"[INST] <<SYS>>You are a pirate <</SYS>> What's your favorite? [/INST]\"\n",
    "print(timeit(lambda: print(llm(correct_prompt, **config)),number=1))\n",
    "\n",
    "incorrect_prompt=\"If you are a pirate, What's your favorite?\"\n",
    "print(timeit(lambda: print(llm(incorrect_prompt, **config)),number=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3adb2c0c-a7ed-475b-966f-16df1319f1df",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}