repository_details.txt

#Directories:
-> llama3_prompting - contains code Notebooks for all prompting strategies using Llama3-8B Instruct model
-> mistral_promoting - contains code Notebooks for all prompting strategies using Mistral-7B Instruct model
-> phi3_prompting - contains code Notebooks for all prompting strategies using Phi-3 mini-4K Instruct model
-> gemma_prompting - contains code Notebooks for all prompting strategies using Gemma-7B Instruct model
-> results - contains results of all prompting techniques

#Structure
---------------------------------------------------------------------------------------------------------------
_llama3_prompting
|
|_llama3_role_all.ipynb -- Role prompting on all datasets using Llama3-8B Instruct (run complete)
|
|_llama3_zcot_all.ipynb -- Zero shot chain of thought prompting on all datasets using Llama3-8B Instruct
---------------------------------------------------------------------------------------------------------------
_phi3_prompting
|
|_phi3_role_all.ipynb -- Role prompting on all datasets using Phi-3 mini-4K Instruct (run complete)
|
|_phi3_zcot_all.ipynb -- Zero shot chain of thought prompting on all datasets using Phi-3 mini-4K Instruct
---------------------------------------------------------------------------------------------------------------
_mistral_prompting
|
|_mistral_role_all.ipynb -- Role prompting on all datasets using Mistral-7B Instruct Instruct (run complete)
|
|_mistral_zcot_all.ipynb -- Zero shot chain of thought prompting on all datasets Mistral-7B Instruct Instruct
---------------------------------------------------------------------------------------------------------------
_gemma_prompting
|
|_Gemma_role_all.ipynb -- Role prompting on all datasets using Gemma-7B Instruct Instruct (run complete)
|
|_Gemma_zcot_all.ipynb -- Zero shot chain of thought prompting on all datasets Gemma-7B Instruct Instruct
---------------------------------------------------------------------------------------------------------------
_results
|
|_Role_prompting____quantitaive_analysis.txt (run complete)
|                 |_qualitative_analysis.txt (run complete)
|
|
---------------------------------------------------------------------------------------------------------------
# General Information

**Each Notebook implements the prompting technique on all the four different tasks.**
** All Model instances are created using Hugging face API with 4-bit quantisation in 'nf4' format**
** All Evaluation metric instances are created using Hugging Face API except the Multiclass Accuracy instance which was created using pytorch.**
** All Dataset instances are created using Hugging Face API with corresponding splits needed for evaluation.**


Datasets: 
Question-Answering Dataset --  BoolQ (validation split - 3270 samples)
Reasoning Dataset -- COMMONSENSEQA (validation split - 1221 samples)
Transaltion Dataset -- iwslt2017-en-fr dataset (validation split - 890 samples)
Summarisation dataset -- samsum dataset (test split - 819 samples)


 General Notebook Structure:
 1. All the necessary libraries installation is carried out in the first cell.
 2. The initial cells will import required all the required libraries and also load the metrics needed for evaluation.
 3. A cell is entirely dedicated for loading the model and setting the configuration of the model.
 4. The next few cells will load all the datasets needed for evaluation.
 5. From now onwards , a Text header will indicate the task the prompting strategy is being evaluated on, i.e, four tasks:
--Question-Answering Dataset 
--Reasoning Dataset
--Transaltion Dataset
--Summarisation dataset

General Task structure:
1.The first cell in each task will contain the function for prompt formulation.
2.The second cell processes the corresponding dataset associated with the task is evaluated using map functionality, currently 'batch size =1',
we recommend to set it according to your capacity of GPU for paralell processing.
3.The third and fourth cells consists the functionality for post processing the generated text from the model and storing the final answers in suitable
data structures for evaluation using the metric (for corresponding task).
4.The fifth cell is for computing the score of the metric.
5.The next two cells correspond to qualitative analysis of the prompt.