Exercise 7.1: Changing prompt style, tokenizer unchanged

[d-kleine]

Given the solution for exercise 7.1, I would like to ask if this is even a good idea to change the prompt template style without adjusting the tokenizer along the way. In this exercise, we are using the GPT-2 tokenizer

LLMs-from-scratch/ch07/01_main-chapter-code/exercise-solutions.ipynb

Line 181 in 0d4967e

"tokenizer = tiktoken.get_encoding(\"gpt2\")"

but experimenting with Alpaca style and Phi-3 style prompt template styles. But both Alpaca and Phi-3 use the Llama-based tokenizer (that uses different special tokens and a different vocab).

Would that explain why the Alpaca and Phi-3 instruction finetuned models perform so badly (assessed with Ollama using Llama 3)?

[rasbt]

You are absolutely correct there. The Phi-3 template would work, but special tokens like <|user|> would be encoded inefficiently (5 tokens instead of 1) etc.

I am adding some bonus materials via #496 to address that point :)

[d-kleine]

Alright, good to know - thanks a lot!

I think it would be great to add this info to the solution for exercise 7.1 also with #496 that the poor performance of the Alpaca and Phi-3 instruction finetuned models is due to the tokenizer - prompt format mismatch. Currently, there is no explanation on why the models perform so badly.

The score is close to 50, which is in the same ballpark as the score we previously achieved with the Alpaca-style prompts.