Peer reviewed learning - How to get involved

[burtenshaw]

👋 Let's discuss the course

Participtation is open, free, and now!

This course is open and peer reviewed. To get involved with the course open a pull request and submit your work for review. Here are the steps:

1. Fork the repo here
2. Read the material, make changes, do the exercises, add your own examples.
3. Open a PR
4. Get it reviewed and merged

This should help you learn and to build a community-driven course that is always improving.

Reviewing

We will review students work as the course runs in a peer review and voluntary approach.

• Open the PR with your examples on the december-2024 branch of the repo from your fork. @burtenshaw will merge these when the module is finished so future students can learn.
• If you find mistakes or shortcomings in the course, open issues and make PRs. We'll get on it ASAP!

Engagement Time

To make this course flexible and easy to follow, the modules use optional levels, extra reading, and concise written material. This means that you can go as deep or as shallow on a topic as you need, and still take away the basics. The course comes with three levels: 🐢, 🐕, and 🦁.

Here are some guidelines on commitments.

Level	Time	Description
🐢 Skimmer	< 1 hour	Light read, try the examples on easy mode
🐕 Explorer	~ 4 hours	Read everything, do hard examples, submit PR, and get reviews.
🦁 Discoverer	Open-ended	Create your own examples, review others' PRs, contribute improvements to the course

It's really up to you how seriously you take the course, but we're on board with anything.

[lwoollett]

Put this feedback on my PR as well, but didn't see the discussion board:

All in all, great 'first dip into finetuning' for people!

There are some things lacking though in my opinion:
1:
No 'goal' to be aiming towards. The chat formatting stumped me for a good second, as I thought that you were wanting the chats tokenized, not just formatted in a chatml format.
Perhaps a small markdown block showing what is the expected output from process_dataset?
2:
No PR Template. Not sure what sort of general programmer skill level this is aimed at, but you may want to include one here.
3:
bigcode/the-stack-smol was an interesting choice for a secondary dataset, you may want to point people towards https://huggingface.co/docs/trl/v0.12.1/en/sft_trainer#trl.SFTConfig so that they know where to look for setting the dataset text field.
4:
In the decoding conversation cell, I get an error:
TypeError: argument 'ids': Can't extract str to Vec
(Addressed by a PR however, #6 )

Will edit tomorrow morning w/ further thoughts as it's 2am 😴

[burtenshaw]

I added a PR template. Thanks again!

[fairydreaming]

There are some things lacking though in my opinion: 1: No 'goal' to be aiming towards. The chat formatting stumped me for a good second, as I thought that you were wanting the chats tokenized, not just formatted in a chatml format. Perhaps a small markdown block showing what is the expected output from process_dataset?

This! I have a weird feeling that the course creators call the JSON-based conversation representation (list of dictionaries with 'content' and 'role' key/value pairs) a 'chatml' format and they didn't explain this anywhere. Since they also mention a ChatML chat template format this is a source of extreme confusion.

[ash-01xor]

Well I too wish there were a few more things that could be explained or written for clarification

1. A proper PR template as @lwoollett mentions
2. Setup and installation guide.
   • A readme or instruction guide for setup in various devices
   • Also a guide to make PR after completion of examples (either the ones present in each chapter or open ended ones) . I think it was mentioned to push the PR to the December branch somewhere but guess so far the PRs have been directly pushed to the main branch. A bit of clarification or mentioning in the README or a separate file would be great.

[burtenshaw]

Thanks for the feedback @ash-01xor. We'll get to work on this now.

[ash-01xor]

Also @burtenshaw, do you think that it is better to work by forking the repo locally or is it better to do everything in colab?
In that case, we also need to provide a file describing the process of doing the later.

[burtenshaw]

If you want to use Colab, I'd recommend forking the repo, opening the notebooks in colab, and then saving them back to your fork.

Colab has instructions for that here: https://colab.research.google.com/github/googlecolab/colabtools/blob/master/notebooks/colab-github-demo.ipynb

[burtenshaw]

@ash-01xor I've also added a tidy PR template.

[ash-01xor]

Well @burtenshaw , would like to know the plan for saving the model training as well?
Each notebook if I'm not wrong has the cell to train the model and save it in particular folder
trainer.save_model("./sft_output") . It will generate a lot of files and the size of the repo might increase unncessarily.

Considering the number of people that might train and then push a PR is it better to add ./sft_output folder to gitignore ?

I wanted to know before i create a PR for the repo.

[duydl]

Hi. I want to ask if it is possible to add a tool like ReviewNB to make it more convenient to review or learn from other people's PRs as most changes in this course will be made on Jupyter Notebooks? Or is there already another way to do this?

[burtenshaw]

Feel free to open a PR with ReviewNB if you want it. Personally, I use vscode's PR/ notebook integration.

[duydl]

I use GitHub.vscode-pull-request-github too but usually feel dedicated UI like ReviewNB to be more user friendly.
I think the app could only be added by repo owner, but could be wrong.

[ATaylorAerospace]

@burtenshaw This is a great repo. Do you anticipate having any sprints for the In Progress or Planned modules? (especially the planned Vision Language Models and Synthetic Datasets modules)

[burtenshaw]

@ATaylorAerospace Thanks! Totally, we're working on module 2 right now and releasing on December 6th. We have a pr here: #29

Feel free to open a PR if you're interested in contributing to a late module like vlms.

[turbosnail11]

Here are my initial thoughts from a casual python user:

1. The links in the README.md seem like options reading - “For more detailed information, refer to the Chat Templates section.” - but I they are required reading.

2. Not enough explanation as to what needs to be done and why - should I be using the transformer or should I just create my own output? Can I use my own chat template in the transformer?

3. Reviewing the examples should be all you need to complete the assignments but this isn’t the case.

4. What is: src="https://huggingface.co/datasets/HuggingFaceTB/smoltalk/embed/viewer/all/train?row=0” and how does it relate to: load_dataset("HuggingFaceTB/smoltalk", "everyday-conversations")? Are they the same dataset?

5. The ds.map() function isn’t explained nor used in a previous example. Also, why would we want to append the output to the existing data set, I thought the point was to convert it?

6. The way to convert the gsm8k dataset with questions/answer tags instead of user/assistant is nowhere in the “training”.

7. I think the explanation order of the tokenizer could be improved. Makes more sense to Apply Template/Tokenize/Decode as opposed to Template/Decode/Tokenize.

8. Not explained what the ChatML template is.

9. Do not use short hand descriptions for a training manual when people are not familiar with the terms.

10. When creating an exercise, you should include an example of the expected output.

11. Explain TRL and some if it’s features (specifically the ones we’ll be using).

12. Explain how we can use datasets on Huggingface and what some of them are.

13. Explain how we can pull resources, connect to models directly from Huggingface.

[schorndorfer]

I saw on bluesky there is a discord channel but not having luck finding where it's been posted.

[turbosnail11]

I saw on bluesky there is a discord channel but not having luck finding where it's been posted.

Click the discord button on this page: https://github.com/huggingface/smol-course

[SuperMohit]

I tried to run SFT from Google collab, training runs on wandb with default system resources. The process crashed twice. I don't see the anything in the logs. Any idea? Should I use GPU?

CPU count	1
Logical CPU count	2

[souzatharsis]

Hi, @burtenshaw. I am not sure the DPO Finetuning tutorial generates valid models.

I ran the dpo finetuning example notebook exactly as provided in the course:

https://github.com/huggingface/smol-course/blob/main/2_preference_alignment/notebooks/dpo_finetuning_example.ipynb

However, the corresponding fine-tuned model only responds with assistant's empty messages.

In this example, the generated fine-tuned model is "thatupiso/SmolLM2-FT-DPO2":

I noticed Training Loss goes to 0 right after the first epoch:

[burtenshaw]

@souzatharsis thanks for reporting. We have an open issue on the default parameters. I'll work on that today. For now, could you try setting the model to dtype torch.float32.

If the problem remains, please open an issue. 🙂

[souzatharsis]

Perfect. It works now. @burtenshaw

I have implemented a process that given an alignment "policy document":

1. generates synthetic preference data (inspired by Meta AI's self-taught evaluators)
2. train a DPO-based aligned fine-tuned model
3. automatically evaluates base and aligned models using LLM-as-a-judge

The output is a policy-aligned model, preference data and evals.

I was wondering whether there is a place in the course to share python notebooks like this. So we can learn from feedback as well as help others learn from practical implementations. If yes, I would love to know practical steps to share it.

Thanks again for the great course!

[SuperMohit]

@souzatharsis can you please share your notebook?

[burtenshaw]

Nice work @souzatharsis!

Yes. You can share your work from module 2 by opening a PR on the December-2024 branch. You can find instructions in this discussion, and when you create the PR.

Looking forward to it!