week06_llm/practice.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aSWEcS2XKgzi"
      },
      "source": [
        "### Practice: Large Language Models and Their Implications\n",
        "<!-- ![img](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F4470ce74-e595-4750-92a5-5f21f040df6d_577x432.jpeg) -->\n",
        "![img](https://i.imgur.com/QGYa2J8.jpeg)\n",
        "\n",
        "In this notebook, you're gonna play with some of the largest language models on the Internet.\n",
        "\n",
        "_Based on works of: Tim Dettmers, Ruslan Svirschevsky, Artem Chumachenko, Younes Belkada, Felix Marty, Yulian Gilyazev, Gosha Zolotov, Andrey Ishutin,  Elena Volf, Artemiy Vishnyakov, Svetlana Shirokovskih."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jYrxHF8Kgzl"
      },
      "source": [
        "### Part 1: prompt engineering (4 points total)\n",
        "\n",
        "In the assignment, we'll use public APIs that host the 100B+ models for inference. Your task is to prompt-engineer the model into solving a few tasks for you.\n",
        "\n",
        "\n",
        "__Which API?__ You are free to use any publicly available API for general LM -- as long as it's __not a chat assistant__. So, gpt 3.5 is fine, but chatGPT is not. Here's a few options:\n",
        "\n",
        "- BLOOM API - [bigscience/bloom](https://huggingface.co/bigscience/bloom) (on the right; recommended)\n",
        "- OpenAI API (via VPN) - [openai.com/api](https://openai.com/api/)\n",
        "- AI21 Jurrasic API - [ai21.com](https://www.ai21.com/blog/announcing-ai21-studio-and-jurassic-1)\n",
        "\n",
        "These APIs may require you to create a (free) account on their platform. Please note that some APIs also have paid subscriptions. __You do not need to pay them__, this assignment was designed to be solved using free-tier subscriptions. If no APIs work for you, you can also solve these tasks with the 6.7B model that you will find later in this notebook - but this will make the tasks somewhat harder.\n",
        "\n",
        "__Quests:__ you will need to solve 4 problems. For each one, please attach a short __description__ of your solution and a __screenshot__ from the API you use. _[If you use python APIs, show your python code with outputs]_\n",
        "\n",
        "__Example:__ Tony is talking to Darth Vader ([BLOOM API](https://huggingface.co/bigscience/bloom)). Black text is written manually, blue text is generated.\n",
        "<hr>\n",
        "\n",
        "![img](https://i.imgur.com/a1QhKF7.png)\n",
        "<hr>\n",
        "\n",
        "__It is fine to roll back a few times,__ e.g. in the example above, the model first generated Vader lines twice in a row, and we rolled that back. However, if you need more than 1-2 rollbacks per session, you should probably try a different prompt."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHIvIFjsKgzm"
      },
      "source": [
        "__Task 1 (1 pt):__ arange a conversation between any two of the following:\n",
        "\n",
        "- a celebrity or politician of your choice\n",
        "- any fictional character (except Darth Vader)\n",
        "- yourself\n",
        "\n",
        "Compare two setups: a) you prompt with character names only b) you supply additional information (see example)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yJJtTnJEKgzm"
      },
      "outputs": [],
      "source": [
        "# <your code OR writeup with screenshots>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6Bc13ueKgzn"
      },
      "source": [
        "__Please choose task 2a or 2b (1pt)__ depending on your model (you can do both, but you will be awarded points for one of these two tasks).\n",
        "\n",
        "__Task 2a: (for BLOOM or other multilingual model)__ zero-shot translation. Take the first verse of [Edgar Allan Poe's \"Raven\"](https://www.poetryfoundation.org/poems/48860/the-raven) and __translate it into French.__ (You are free to use any other text of at least the same size)\n",
        "\n",
        "Original text: ```\n",
        "Once upon a midnight dreary, while I pondered, weak and weary,\n",
        "Over many a quaint and curious volume of forgotten lore—\n",
        "    While I nodded, nearly napping, suddenly there came a tapping,\n",
        "As of some one gently rapping, rapping at my chamber door.\n",
        "“’Tis some visitor,” I muttered, “tapping at my chamber door—\n",
        "            Only this and nothing more.”\n",
        "```\n",
        "\n",
        "Verify your translation by converting french back into english using a public machine translation service.\n",
        "\n",
        "__Task 2b: (non-BLOOM):__ toxicity classification for [SetFit/toxic_conversations](https://huggingface.co/datasets/SetFit/toxic_conversations). Make the model solve binary classification (toxic vs not toxic) in the few shot mode. For few-shot examples, use 2-3 toxic and 2-3 non-toxic non-toxic examples. Measure accuracy on at least 25 samples. You may need to try several different prompts before you find the one that works."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GygPXoyKKgzo"
      },
      "outputs": [],
      "source": [
        "# <your code OR writeup with screenshots>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iaYweoPsKgzo"
      },
      "source": [
        "\n",
        "__Task 3 (1pt):__ create a prompt and few-shot examples tha make the model __change the gender pronouns__ of the main actor in a given sentence in any direction of your choice. E.g. the doctor took off _his_ mask <-> the doctor took of _her_ mask.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uE-Zv_MoKgzq"
      },
      "outputs": [],
      "source": [
        "# <your code OR writeup with screenshots>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbNrRmgMKgzq"
      },
      "source": [
        "__Task 4 (1pt):__ write a prompt and supply examples such that the model would __convert imperial units to metric units__ (miles -> kilometers; mph -> kph). More specifically, the model should rewrite a given sentence and replace all imperial units with their metric equivalents. After it works with basic distances and speed, try to find complicated examples where it does *not* work.\n",
        "\n",
        "Please note that 1 mile is not equal to 1 km :)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UBxMVGHNKgzr"
      },
      "outputs": [],
      "source": [
        "# <your code OR writeup with screenshots>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZKw-mjuRKgzs"
      },
      "source": [
        "### Part 2: local inference\n",
        "\n",
        "Now, let's try and load the strongest model that can fit a typical Colab GPU (T4 with 16 GB as of spring 2023).\n",
        "\n",
        "Our best candidates are the smaller versions of the best performing open source models:\n",
        "- 7 Bn parameters version of [LLaMA](https://arxiv.org/pdf/2302.13971.pdf) - best for spring 2023, released by Facebook\n",
        "- 7 Bn parameters version of [Falcon](https://falconllm.tii.ae) - close competitor to Llama, released in May 2023 by [Technology Innovation Institute of UAE](https://www.tii.ae).\n",
        "- 6.7 Bn parameters version of [OPT](https://arxiv.org/abs/2205.01068) - top choice in this nomination in 2022, released by Facebook.\n",
        "\n",
        "Beware: while these models are smaller than the ones in API, they're still over 60x larger than the BERT we played with last time. The code below will *just barely* fit into memory, so make sure you don't have anything else loaded. Sometimes you may need to restart runtime for the code to work.\n",
        "\n",
        "It's a good time to restart your kernel and switch to GPU! (Runtime -> Change runtime type)\n",
        "<center><img src=\"https://i.imgur.com/OOfDYzJ.png\" width=240px></center>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7xeRF_hSKgzs"
      },
      "outputs": [],
      "source": [
        "%pip install --quiet bitsandbytes==0.41.1 transformers==4.34.1 accelerate==0.24.0 sentencepiece==0.1.99 optimum==1.13.2 auto-gptq==0.4.2\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import transformers\n",
        "import bitsandbytes as bnb\n",
        "from tqdm.auto import tqdm, trange\n",
        "assert torch.cuda.is_available(), \"you need cuda for this part\"\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VMzFwx29Kgzu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d6bfd145-9c7e-4f68-c619-bf4f63c3c7a6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n"
          ]
        }
      ],
      "source": [
        "model_name = 'TheBloke/Llama-2-13B-GPTQ'\n",
        "\n",
        "# loading Llama tokenizer ...\n",
        "tokenizer = transformers.LlamaTokenizer.from_pretrained(model_name, device_map=device)\n",
        "tokenizer.pad_token_id = tokenizer.eos_token_id\n",
        "\n",
        "# ... and the model itself\n",
        "model = transformers.AutoModelForCausalLM.from_pretrained(\n",
        "    model_name,\n",
        "    device_map='auto',\n",
        "    torch_dtype=torch.float16,\n",
        "    low_cpu_mem_usage=True,\n",
        "    offload_state_dict=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5k2zCgAhG7l5"
      },
      "source": [
        "## Text generation\n",
        "\n",
        "**Comparison of strategies for language model text generation:**\n",
        "\n",
        "| Strategy | Description | Pros & Cons |\n",
        "| --- | --- | --- |\n",
        "| Greedy Search | Chooses the word with the highest probability as the next word in the sequence. | **Pros:** Simple and fast. <br> **Cons:** Can lead to repetitive and incoherent text. |\n",
        "| Sampling with Temperature | Introduces randomness in the word selection. A higher temperature leads to more randomness. | **Pros:** Allows exploration and diverse output. <br> **Cons:** Higher temperatures can lead to nonsensical outputs. |\n",
        "| Nucleus Sampling (Top-p Sampling) | Selects the next word from a truncated vocabulary, the \"nucleus\" of words that have a cumulative probability exceeding a pre-specified threshold (p). | **Pros:** Balances diversity and quality. <br> **Cons:** Setting an optimal 'p' can be tricky. |\n",
        "| Beam Search | Explores multiple hypotheses (sequences of words) at each step, and keeps the 'k' most likely, where 'k' is the beam width. | **Pros:** Produces more reliable results than greedy search. <br> **Cons:** Can lack diversity and lead to generic responses. |\n",
        "| Top-k Sampling | Randomly selects the next word from the top 'k' words with the highest probabilities. | **Pros:** Introduces randomness, increasing output diversity. <br> **Cons:** Random selection can sometimes lead to less coherent outputs. |\n",
        "| Length Normalization | Prevents the model from favoring shorter sequences by dividing the log probabilities by the sequence length raised to some power. | **Pros:** Makes longer and potentially more informative sequences more likely. <br> **Cons:** Tuning the normalization factor can be difficult. |\n",
        "| Stochastic Beam Search | Introduces randomness into the selection process of the 'k' hypotheses in beam search. | **Pros:** Increases diversity in the generated text. <br> **Cons:** The trade-off between diversity and quality can be tricky to manage. |\n",
        "| Decoding with Minimum Bayes Risk (MBR) | Chooses the hypothesis (out of many) that minimizes expected loss under a loss function. | **Pros:** Optimizes the output according to a specific loss function. <br> **Cons:** Computationally more complex and requires a good loss function. |\n",
        "\n",
        "Documentation references:\n",
        "- [reference for `AutoModelForCausalLM.generate()`](https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/text_generation#transformers.GenerationMixin.generate)\n",
        "- [reference for `AutoTokenizer.decode()`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.decode)\n",
        "- Huggingface [docs on generation strategies](https://huggingface.co/docs/transformers/generation_strategies)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generation with HuggingFace"
      ],
      "metadata": {
        "id": "GWm6KDSzMiAf"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gGfyeM-vdq5o",
        "tags": []
      },
      "outputs": [],
      "source": [
        "prompt = 'The first discovered martian lifeform looks like'\n",
        "batch = tokenizer(prompt, return_tensors='pt', return_token_type_ids=False).to(device)\n",
        "print(\"Input batch (encoded):\", batch)\n",
        "\n",
        "output_tokens = model.generate(**batch, max_new_tokens=64, do_sample=True, temperature=0.8)\n",
        "# greedy inference:                                        do_sample=False)\n",
        "# beam search for highest probability:                     num_beams=4)\n",
        "\n",
        "print(\"\\nOutput:\", tokenizer.decode(output_tokens[0].cpu()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P17ehC1sKgzx"
      },
      "source": [
        "#### Low-level code for text generation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LZJvOMbmG7l8",
        "tags": [],
        "outputId": "1fe924b7-3b66-41b9-99c0-16619b064035",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Moscow is the capital of \n",
            "\n",
            "Step #0 candidates:\n",
            "▁Russia   : 0.7616 \n",
            "▁the      : 0.1795 \n",
            "▁Russian  : 0.0218 \n",
            "▁a        : 0.0058 \n",
            "▁not      : 0.0022 \n",
            "\n",
            "Chosen token: Russia\n",
            "\n",
            "Step #1 candidates:\n",
            ".         : 0.3238 \n",
            ",         : 0.3188 \n",
            "▁and      : 0.1845 \n",
            "and       : 0.0554 \n",
            "<0x0A>    : 0.0080 \n",
            "\n",
            "Chosen token: ,\n",
            "\n",
            "Step #2 candidates:\n",
            "▁the      : 0.1961 \n",
            "▁and      : 0.1857 \n",
            "▁located  : 0.0688 \n",
            "▁a        : 0.0603 \n",
            "▁one      : 0.0562 \n",
            "\n",
            "Chosen token: the\n",
            "\n",
            "Step #3 candidates:\n",
            "▁largest  : 0.4282 \n",
            "▁most     : 0.1651 \n",
            "▁country  : 0.0528 \n",
            "▁biggest  : 0.0515 \n",
            "▁world    : 0.0377 \n",
            "\n",
            "Chosen token: world\n",
            "\n",
            "Step #4 candidates:\n",
            "'         : 0.4954 \n",
            "’         : 0.3950 \n",
            "s         : 0.0522 \n",
            "▁largest  : 0.0054 \n",
            "larg      : 0.0047 \n",
            "\n",
            "Chosen token: '\n",
            "\n",
            "Step #5 candidates:\n",
            "s         : 0.9785 \n",
            "sl        : 0.0057 \n",
            "st        : 0.0023 \n",
            "sf        : 0.0020 \n",
            "ss        : 0.0016 \n",
            "\n",
            "Chosen token: s\n",
            "\n",
            "Step #6 candidates:\n",
            "▁largest  : 0.8468 \n",
            "▁biggest  : 0.0521 \n",
            "▁most     : 0.0272 \n",
            "larg      : 0.0189 \n",
            "▁second   : 0.0142 \n",
            "\n",
            "Chosen token: third\n",
            "\n",
            "Step #7 candidates:\n",
            "larg      : 0.3349 \n",
            "-         : 0.3097 \n",
            "most      : 0.1498 \n",
            "▁largest  : 0.0736 \n",
            "▁most     : 0.0605 \n",
            "\n",
            "Chosen token: larg\n",
            "\n",
            "Step #8 candidates:\n",
            "est       : 0.9592 \n",
            "esto      : 0.0138 \n",
            "este      : 0.0073 \n",
            "ests      : 0.0050 \n",
            "es        : 0.0028 \n",
            "\n",
            "Chosen token: est\n",
            "\n",
            "Step #9 candidates:\n",
            "city      : 0.5349 \n",
            "country   : 0.0833 \n",
            "pop       : 0.0525 \n",
            "e         : 0.0521 \n",
            "▁city     : 0.0391 \n",
            "\n",
            "Chosen token: city\n",
            "\n"
          ]
        }
      ],
      "source": [
        "prompt = \"Moscow is the capital of\"\n",
        "# prompt = \"Skippy, a young android, likes to dream about electric\"\n",
        "\n",
        "print(prompt, '\\n')\n",
        "\n",
        "voc = tokenizer.get_vocab()\n",
        "voc_rev = {v:k for k, v in voc.items()}  # reverse vocab for decode\n",
        "\n",
        "for i in range(10):\n",
        "    inputs = tokenizer(prompt, return_tensors='pt', return_token_type_ids=False).to(device)\n",
        "    logits = model.forward(**inputs).logits[0, -1, :]\n",
        "    probs = torch.nn.functional.softmax(logits, dim=-1)\n",
        "    next_token_id = torch.multinomial(probs.flatten(), num_samples=1)\n",
        "\n",
        "    next_token = tokenizer.decode(next_token_id)\n",
        "    prompt += next_token\n",
        "\n",
        "    sorted_probs, sorted_indices = torch.sort(probs, descending=True)\n",
        "    top_tokens = sorted_indices[:5]\n",
        "    print(f\"Step #{i} candidates:\")\n",
        "    for t, p in zip (top_tokens, sorted_probs):\n",
        "        t = voc_rev[t.item()]\n",
        "        print(f\"{t:<10}: {p:.4f} \")\n",
        "\n",
        "    print(f'\\nChosen token: {next_token}', end='\\n\\n', flush=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a686Z2SQKgz0"
      },
      "source": [
        "**Task 5: write code for nucleus sampling generation (2 points)**:\n",
        "\n",
        "Use the `nucleus_sampling()` template below. Look at the detailed generation code above for inspiration. __Please do not use model.generate__.\n",
        "\n",
        "**Bonus task: write code for beam search (3 bonus points)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3WIqDgfBKgz0"
      },
      "outputs": [],
      "source": [
        "from typing import Tuple, List\n",
        "def nucleus_sampling(model, tokenizer, prompt: str, prob: float = 0.5) -> Tuple[str, List[str]]:\n",
        "    \"\"\"generates the next token from the nucleus of tokens with cumulative probability up to param:prob\"\"\"\n",
        "\n",
        "    <YOUR CODE HERE>\n",
        "\n",
        "    # sampled_token should be a string token that was generated\n",
        "    # possible_tokens should be a list of all tokens that have non-zero probability\n",
        "    return sampled_token, possible_tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LzPLQl-HKgz1"
      },
      "outputs": [],
      "source": [
        "# Tests for nucleus sampling\n",
        "test_prompt = \"Elbrus is the highest\"\n",
        "next_token, possible_tokens = nucleus_sampling(model, tokenizer, test_prompt, prob=0.9)\n",
        "print(test_prompt, next_token, possible_tokens)\n",
        "assert next_token in possible_tokens\n",
        "assert 3 <= len(possible_tokens) <= 3\n",
        "assert sorted(possible_tokens) == ['mountain', 'peak', 'point']\n",
        "\n",
        "test_prompt = \"Large language models can learn to\"\n",
        "next_token, possible_tokens = nucleus_sampling(model, tokenizer, test_prompt, prob=0.4)\n",
        "print(test_prompt, next_token, possible_tokens)\n",
        "assert next_token in possible_tokens\n",
        "assert sorted(possible_tokens) == ['be', 'communicate', 'do', 'generate', 'perform', 'predict', 'speak', 'write']\n",
        "assert len(possible_tokens) == 8"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Part 3: Chain-of-thought prompting (4 points total)\n",
        "\n",
        "![img](https://github.com/kojima-takeshi188/zero_shot_cot/raw/main/img/image_stepbystep.png)\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "5ZaQZhPXOPSG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "import random\n",
        "import locale; locale.getpreferredencoding = lambda: \"UTF-8\"\n",
        "!wget https://raw.githubusercontent.com/kojima-takeshi188/zero_shot_cot/2824685e25809779dbd36900a69825068e9f51ef/dataset/AQuA/test.json -O aqua.json\n",
        "data = list(map(json.loads, open(\"aqua.json\")))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N2AmfelTn5en",
        "outputId": "fecfe17f-d07c-40d1-c74f-a99e6b86e392"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-10-27 16:19:33--  https://raw.githubusercontent.com/kojima-takeshi188/zero_shot_cot/2824685e25809779dbd36900a69825068e9f51ef/dataset/AQuA/test.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 130192 (127K) [text/plain]\n",
            "Saving to: ‘aqua.json’\n",
            "\n",
            "aqua.json           100%[===================>] 127.14K  --.-KB/s    in 0.003s  \n",
            "\n",
            "2023-10-27 16:19:34 (46.7 MB/s) - ‘aqua.json’ saved [130192/130192]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Example:\")\n",
        "data[150]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IATXmPfYw8s6",
        "outputId": "6820f54e-3045-4248-db3a-d531f2281d1c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Janice bikes at 10 miles per hour, while Jennie bikes at 20. How long until they have collectively biked 1 mile?',\n",
              " 'options': ['A)1 minute',\n",
              "  'B)2 minutes',\n",
              "  'C)3 minutes',\n",
              "  'D)4 minutes',\n",
              "  'E)5 minutes'],\n",
              " 'rationale': \"Janice's speed = 1/6 miles per minute\\nJennie's speed = 1/3 miles per minute\\nJanice + Jennie's speed= (1/6 + 1/3) = 1/2 miles per minute\\nBoth together will finish the mile in 2 minutes\\ncorrect option is B\",\n",
              " 'correct': 'B'}"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Naive solution\n",
        "\n",
        "Here, we prompt the model to choose an answer to the example above (`data[150]`) out of the options given above. We're using a format that mimics grade school solution textbook.\n",
        "\n",
        "Please note that there are minor formatting changes in options: an extra space and an opening bracket. Those may or may not be important :)"
      ],
      "metadata": {
        "id": "6UcOYQPW8sVq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "EXAMPLE_0SHOT = \"\"\"\n",
        "Question: Janice bikes at 10 miles per hour, while Jennie bikes at 20. How long until they have collectively biked 1 mile?\n",
        "Answer Choices: (A) 1 minute (B) 2 minutes (C) 3 minutes (D) 4 minutes (E) 5 minutes\n",
        "Correct Answer:\n",
        "\"\"\".strip()"
      ],
      "metadata": {
        "id": "KtkkdiJl3-UI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# solving an equation directly\n",
        "batch = tokenizer(EXAMPLE_0SHOT, return_tensors='pt', return_token_type_ids=False).to(device)\n",
        "torch.manual_seed(1337)\n",
        "output_tokens = model.generate(**batch, max_new_tokens=100, do_sample=True, top_p=0.9)\n",
        "print(\"[Prompt:]\\n\" + EXAMPLE_0SHOT)\n",
        "print(\"=\" * 80)\n",
        "print(\"[Generated:]\", tokenizer.decode(output_tokens[0][batch['input_ids'].shape[1]:].cpu()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hyQ_8tJc6nyv",
        "outputId": "4cf634aa-d4db-41a8-e97b-4c97a66ad819"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Prompt:]\n",
            "Question: Janice bikes at 10 miles per hour, while Jennie bikes at 20. How long until they have collectively biked 1 mile?\n",
            "Answer Choices: (A) 1 minute (B) 2 minutes (C) 3 minutes (D) 4 minutes (E) 5 minutes\n",
            "Correct Answer:\n",
            "================================================================================\n",
            "[Generated:] (E) 5 minutes\n",
            "Explanation: Jennie bikes at 20 miles per hour for 2 minutes. She will have travelled 2 miles in this time. Janice also bikes for 2 minutes, but at a slower speed of 10 miles per hour. This means that she will travel 2 miles in 2 times 10 = 20 minutes.\n",
            "Janice and Jennie will have travelled 4 miles collectively,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And here's how you can solve this with few-shot chain-of-thought prompting.\n",
        "\n",
        "You need to chang 3 things\n",
        "- use a new field called **Rationale**, that contains a step-by-step solution to the problem\n",
        "- add several few-shot examples of previously solved problems **with rationales**\n",
        "- change the final prompt so that the model has to generate rationale before answering"
      ],
      "metadata": {
        "id": "suSkiDk28I6C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "EXAMPLE_3SHOT_CHAIN_OF_THOUGHT = \"\"\"\n",
        "Question: The original retail price of an appliance was 60 percent more than its wholesale cost. If the appliance was actually sold for 20 percent less than the original retail price, then it was sold for what percent more than its wholesale cost?\n",
        "Answer Choices: (A) 20% (B) 28% (C) 36% (D) 40% (E) 42%\n",
        "Rationale: wholesale cost = 100;\\noriginal price = 100*1.6 = 160;\\nactual price = 160*0.8 = 128.\\nAnswer: B.\n",
        "Correct Answer: B\n",
        "\n",
        "\n",
        "Question: A grocer makes a 25% profit on the selling price for each bag of flour it sells. If he sells each bag for $100 and makes $3,000 in profit, how many bags did he sell?\n",
        "Answer Choices: (A) 12 (B) 16 (C) 24 (D) 30 (E) 40\n",
        "Rationale: Profit on one bag: 100*1.25= 125\\nNumber of bags sold = 3000/125 = 24\\nAnswer is C.\n",
        "Correct Answer: C\n",
        "\n",
        "\n",
        "Question: 20 marbles were pulled out of a bag of only white marbles, painted black, and then put back in. Then, another 20 marbles were pulled out, of which 1 was black, after which they were all returned to the bag. If the percentage of black marbles pulled out the second time represents their percentage in the bag, how many marbles in total Q does the bag currently hold?\n",
        "Answer Choices: (A) 40 (B) 200 (C) 380 (D) 400 (E) 3200\n",
        "Rationale: We know that there are 20 black marbles in the bag and this number represent 1/20 th of the number of all marbles in the bag, thus there are total Q of 20*20=400 marbles.\\nAnswer: D.\n",
        "Correct Answer: D\n",
        "\n",
        "\n",
        "Question: Janice bikes at 10 miles per hour, while Jennie bikes at 20. How long until they have collectively biked 1 mile?\n",
        "Answer Choices: (A) 1 minute (B) 2 minutes (C) 3 minutes (D) 4 minutes (E) 5 minutes\n",
        "Rationale:\n",
        "\"\"\".strip()"
      ],
      "metadata": {
        "id": "K0F1jYdRvoJW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "batch = tokenizer(EXAMPLE_3SHOT_CHAIN_OF_THOUGHT, return_tensors='pt', return_token_type_ids=False).to(device)\n",
        "torch.manual_seed(1337)\n",
        "output_tokens = model.generate(**batch, max_new_tokens=100, do_sample=True, top_p=0.9)\n",
        "print(\"[Prompt:]\\n\" + EXAMPLE_3SHOT_CHAIN_OF_THOUGHT)\n",
        "print(\"=\" * 80)\n",
        "print(\"[Generated:]\", tokenizer.decode(output_tokens[0][batch['input_ids'].shape[1]:].cpu()))\n",
        "#### NOTE: scroll down for the final answer (below the ======= line)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tn8QoAYcRkHC",
        "outputId": "54ec824a-33c5-4554-b0e9-0e0a9c246994"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Prompt:]\n",
            "Question: The original retail price of an appliance was 60 percent more than its wholesale cost. If the appliance was actually sold for 20 percent less than the original retail price, then it was sold for what percent more than its wholesale cost?\n",
            "Answer Choices: (A) 20% (B) 28% (C) 36% (D) 40% (E) 42%\n",
            "Rationale: wholesale cost = 100;\n",
            "original price = 100*1.6 = 160;\n",
            "actual price = 160*0.8 = 128.\n",
            "Answer: B.\n",
            "Correct Answer: B\n",
            "\n",
            "\n",
            "Question: A grocer makes a 25% profit on the selling price for each bag of flour it sells. If he sells each bag for $100 and makes $3,000 in profit, how many bags did he sell?\n",
            "Answer Choices: (A) 12 (B) 16 (C) 24 (D) 30 (E) 40\n",
            "Rationale: Profit on one bag: 100*1.25= 125\n",
            "Number of bags sold = 3000/125 = 24\n",
            "Answer is C.\n",
            "Correct Answer: C\n",
            "\n",
            "\n",
            "Question: 20 marbles were pulled out of a bag of only white marbles, painted black, and then put back in. Then, another 20 marbles were pulled out, of which 1 was black, after which they were all returned to the bag. If the percentage of black marbles pulled out the second time represents their percentage in the bag, how many marbles in total Q does the bag currently hold?\n",
            "Answer Choices: (A) 40 (B) 200 (C) 380 (D) 400 (E) 3200\n",
            "Rationale: We know that there are 20 black marbles in the bag and this number represent 1/20 th of the number of all marbles in the bag, thus there are total Q of 20*20=400 marbles.\n",
            "Answer: D.\n",
            "Correct Answer: D\n",
            "\n",
            "\n",
            "Question: Janice bikes at 10 miles per hour, while Jennie bikes at 20. How long until they have collectively biked 1 mile?\n",
            "Answer Choices: (A) 1 minute (B) 2 minutes (C) 3 minutes (D) 4 minutes (E) 5 minutes\n",
            "Rationale:\n",
            "================================================================================\n",
            "[Generated:] 10 + 20 = 30 miles per hour, thus the time required for them to bike 1 mile collectively is 1/30th of an hour, which is 1/30th of 60= 2 minutes\n",
            "Answer is B.\n",
            "Correct Answer: B\n",
            "\n",
            "Question: How many different times tables are there in the range of 10 times 10 and 20 times 20?\n",
            "Answer\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "__Task 6 (1 pt)__ write a function that automatically creates chain-of-thought prompts. Follow the instructions from the function docstring."
      ],
      "metadata": {
        "id": "s4px3jv-99-m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "QUESTION_PREFIX = \"Question: \"\n",
        "OPTIONS_PREFIX = \"Answer Choices: \"\n",
        "CHAIN_OF_THOUGHT_PREFIX = \"Rationale: \"\n",
        "ANSWER_PREFIX = \"Correct Answer: \"\n",
        "FEWSHOT_SEPARATOR = \"\\n\\n\\n\"\n",
        "\n",
        "def make_prompt(*, main_question, fewshot_examples):\n",
        "  \"\"\"\n",
        "  Your goal is to produce the same prompt as the EXAMPLE_3SHOT_CHAIN_OF_THOUGHT automatically\n",
        "\n",
        "  For each few-shot question, make sure to follow the following rules:\n",
        "  1. Each question begins with QUESTION_PREFIX, after which you should print the question without leading/traiiling spaces (if any)\n",
        "  2. After the question, provide space-separated options. Each option should be put in double brackets, followed by option text, e.g. \"(A) 146%\"\n",
        "  3. Then, provide the answer as a single letter (A-E)\n",
        "  4. Finally, add trailing newlines from FEWSHOT_SEPARATOR\n",
        "\n",
        "  Your final prompt should contain all fewshot_examples (in order), separated with FEWSHOT_SEPARATOR, then follow with main_question.\n",
        "  The main_question should contain the question and options formatted the same way as in FEWSHOT_EXAMPLES.\n",
        "  After that, you should prompt the model to produce an explanation (rationale) for the answer.\n",
        "\n",
        "  Please make sure your prompt contains no leading/trailing newlines or spaces, same as in EXAMPLE_3SHOT_CHAIN_OF_THOUGHT\n",
        "  \"\"\"\n",
        "\n",
        "  <YOUR CODE HERE>\n",
        "\n",
        "  return <a string that contains the prompt formatted as per instructions above>\n",
        "\n",
        "\n",
        "\n",
        "generated_fewshot_prompt = make_prompt(main_question=data[150], fewshot_examples=(data[30], data[20], data[5]))\n",
        "assert generated_fewshot_prompt == EXAMPLE_3SHOT_CHAIN_OF_THOUGHT, \"prompts don't match\"\n",
        "assert generated_fewshot_prompt != make_prompt(main_question=data[150], fewshot_examples=())\n",
        "assert generated_fewshot_prompt.endswith(make_prompt(main_question=data[150], fewshot_examples=()))\n",
        "\n",
        "print(\"Well done!\")\n",
        "\n",
        "# Hint: if two prompts do not match, you may find it usefull to use https://www.diffchecker.com or similar to find the difference"
      ],
      "metadata": {
        "id": "_ntyFPMt9fyt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "__Task 7 (1 points):__ Evaluate your prompt.\n",
        "\n",
        "Please run the model on the entire dataset and measure it's accuracy.\n",
        "For each question, peak $n=5$ other questions at random to serve as few-shot examples. Make sure not to accidentally sample the main_question among few-shot examples. For scientific evaluation, it is also a good practice to split the data into two parts: one for eval, and another for few-shot examples. However, doing so is optional in this homework.\n",
        "\n",
        "The tricky part is when to stop generating: if you don't control for this, your model can accidentally generate a whole new question - and promptyly answer it :) To make sure you get the correct answer, stop generating tokens when the model is done explaining it's solution. To circumvent this, you need to __stop generating as soon as the model generates Final Answer: [A-E]__\n",
        "To do so, you can either generate manually (see low-level generation above) or use [transformers stopping criteria](https://discuss.huggingface.co/t/implimentation-of-stopping-criteria-list/20040/2), whichever you prefer.\n",
        "\n",
        "If you do everything right, the model should be much better than random. However, please __do not expect miracles__: this is far from the best models, and it will perform much worse than an average human."
      ],
      "metadata": {
        "id": "P7DzQ8hfOcFR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "NUM_SAMPLES = 0    # use this to count how many samples you evaluated\n",
        "NUM_RESPONDED = 0  # how many times did the model produce Correct Answer: (letter) in it's response. use as a sanity check.\n",
        "NUM_CORRECT = 0    # how many times did the model's chosen answer (letter) match the correct answer"
      ],
      "metadata": {
        "id": "oMvj9eCtQwnz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "< A whole lot of your code here >\n",
        "\n",
        "# Optionally, consider inferencing multiple sentences in a batch for faster inference;\n",
        "# If you choose to batch outputs, make sure the results are the same as with batch=1 (using greedy inference)"
      ],
      "metadata": {
        "id": "Zh6gejr0JNuh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Responded %%:\", NUM_RESPONDED / NUM_SAMPLES)\n",
        "print(\"Accuracy (when responded):\", NUM_CORRECT / NUM_RESPONDED)\n",
        "print(\"Accuracy (overall):\", NUM_CORRECT / NUM_SAMPLES)\n",
        "\n",
        "if NUM_RESPONDED / NUM_SAMPLES < 0.9:\n",
        "  print(\"Something is wrong with the evaluation technique (for 5-shot CoT): the model refuses to answer too many questions.\")\n",
        "  print(\"Make sure you generate enough tokens that the model can produce a correct answer.\")\n",
        "  print(\"When in doubt, take a look at the full model output. You can often spot errors there.\")"
      ],
      "metadata": {
        "id": "gxrCJvxIJSjA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "__Task 8 (2 points)__ Experiment time!\n",
        "<img width=200px src=https://www.evolvefish.com/cdn-cgi/image/quality%3D85/assets/images/Apparel/TShirtsWomenCont/Main/EF-APP-CWT-00068(Main).jpg>\n",
        "\n",
        "Your final quest is to use the testbench you've just written to answer one of the following questions:\n",
        "\n",
        "### Option 1: How many shots do you need?\n",
        "\n",
        "How does model accuracy change with the number of fewshot examples?\n",
        "\n",
        "a. check if the model accuracy changes as you increase/decrease the number of \"shots\"\n",
        "\n",
        "b. try to prompt-engineer a model into giving the best rationale __without__ any few-shot examples, i.e. zero-shot\n",
        "\n",
        "For zero-shot mode, feel free to use wild prompt-engineering or modify the inference procedure.\n",
        "\n",
        "### Option 2: Is this prompting tecnique reliable?\n",
        "\n",
        "_Inspired by ongoing research by Anton Voronov, Lena Volf and Max Ryabinin._\n",
        "\n",
        "For this option, you need to check if the model behavior (and hence, accuracy) is robust to perturbations in the input prompt.\n",
        "\n",
        "a. Does the accuracy degrade if you provide wrong answers to few-shot examples? (make sure to modify rationale if it contains answer in the end)\n",
        "\n",
        "b. Does it degrade if you replace question/answer prompts with \"Q\" and \"A\"? What if you write both on the same line? Change few-shot separators?\n",
        "\n",
        "\n",
        "\n",
        "### Option 3: Inference Matters\n",
        "\n",
        "There are many ways to inference the model, not all of them equal.\n",
        "\n",
        "a. check whether greedy inference or beam search affects model generation quality\n",
        "\n",
        "b. implement and evaluate sampling with voting (see explanation below).\n",
        "\n",
        "\n",
        "The voting technique(b) should work as follows: first, you generate k (e.g. 50) \"attempts\" at an answer using nucleus sampling (or a similar technique).\n",
        "Then, you count how many of those attempts chose a particular option (A, B, etc) as the final answer. The option that was chosen most frequently has the most \"votes\", and therefore \"wins\".\n",
        "\n",
        "To speed up voting, you may want to generate these attempts in parallel as a batch. That should be very easy to implement: just run `model.generate` on a list with multiple copies of the same prompt.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "================================================\n",
        "\n",
        "__Common rules:__ You will need to test both hypothes (A and B) in the chosen option. You may choose to replace one of them with your own idea - but please ask course staff in advance (via telegram) if you want full points.\n",
        "\n",
        "Feel free to organize your code and report as you see fit - but please make sure it's readable and the code runs top-to-bottom :)\n",
        "Write a short informal report about what you tried and, in doing so, what did you found. Minimum of 2 paragraphs; more is ok; creative visualizations are welcome.\n",
        "\n",
        "You are allowed (but not required) to prompt the model into generating a report for you --- or helping you write one. However, if you do so, make sure that it is still human-readable :)\n",
        "\n"
      ],
      "metadata": {
        "id": "UZLK2rLiKxbM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# feel free to organize your solution as you see fit"
      ],
      "metadata": {
        "id": "_r6UVDl4NEua"
      },
      "execution_count": 2,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.15"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}