tutorials/adapted_named_entity_recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Token Classification with Adapt𝒪r\n",
    "\n",
    "This tutorial demonstrates a usage of Adapt𝒪r library for **Named Entity Recognition**, a Token Classification task.\n",
    "\n",
    "In this scenario, Adapt𝒪r's Token Classification objective also conveniently deals with labels' per-token **alignment**, a reoccurring problem with identification of entities using the models based on sub-words tokenizers, e.g. Transformers.\n",
    "\n",
    "This tutorial will show you:\n",
    "\n",
    "1. The convenience of objective-centric training and evaluation, even in the single-objective token classification scenario.\n",
    "2. The potential of a simple domain adaptation using Transformers, merely utilising the data of the training domain, that can be easily extended further with bigger corporas, matching the domain of application.\n",
    "3. How can the states and methods of instances of Adapt𝒪r's objects be used for easy debugging or analyses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SB2KDo6-XP7l",
    "outputId": "c4b6be6a-af86-42f9-d433-d13e2c94d7aa"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "!pip install --upgrade adaptor==0.1.4 datasets==2.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset resolution\n",
    "\n",
    "We will use publicly-available CoNLL 2003 dataset for Named Entity Recognition, annotated on the news articles from Reuters Corpus. See e.g. [paperswithcode.com](https://paperswithcode.com/dataset/conll-2003) for more info."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 140,
     "referenced_widgets": [
      "311ac0b4f7124968a10da068c6b080c7",
      "41660ae55a3a4a24939e3a9c5921405d",
      "60e20867027a4aad9ee017dd87aab365",
      "5df842530194491bafa52bf7c7fd3661",
      "2950d13507d34371984f224c1fb4519e",
      "8146bb6de82d409c87241b973d63e3d9",
      "5712a7090a194bdca94d864d62b55ab3",
      "a50ccd63ffe84d268dd02d98ddd8fbae",
      "014ff61d5011457a96ff4d9e1bf3f1b4",
      "614690118e484425b438f1580dc72ac2",
      "f30b0650f0884912bbe71aa25573789c"
     ]
    },
    "id": "bHixc84Bdqn_",
    "outputId": "22c9e72d-17c5-4b31-eab9-fca6f1ccd040"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset conll2003 (/home/xstefan3/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74fe8793a0614fc0932df17c37cddca3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /home/xstefan3/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee/cache-fc8ea9d3f4217ca3.arrow\n",
      "Loading cached processed dataset at /home/xstefan3/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee/cache-8dba4962de85cab3.arrow\n",
      "Loading cached processed dataset at /home/xstefan3/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee/cache-0306437327e5464f.arrow\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "conll_dataset = load_dataset(\"conll2003\")\n",
    "conll_dataset_nonempty = conll_dataset.filter(lambda sample: sample[\"tokens\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transferring data to Adaptor's `TokenClassification` Objective format should be always easily-doable: for all existing objectives, the input format consists of a list of texts & labels, or a paths to a `.txt` files with one sample / label per line. See the base `Objective` docs.\n",
    "\n",
    "If a different format suits your case better, all you might need is to override `_per_split_iterators(self, split: str) -> Tuple[Iterable[str], Iterable[str]]` for a selected top-level objective. This method returns a tuple of Iterables of samples and their aligned labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TJ1GNFVBez0O",
    "outputId": "ea271f0f-34cb-4d9f-f981-6b9579e69be5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EU rejects German call to boycott British lamb .',\n",
       " 'Peter Blackburn',\n",
       " 'BRUSSELS 1996-08-22',\n",
       " 'The European Commission said on Thursday it disagreed with German advice to consumers to shun British lamb until scientists determine whether mad cow disease can be transmitted to sheep .',\n",
       " \"Germany 's representative to the European Union 's veterinary committee Werner Zwingmann said on Wednesday consumers should buy sheepmeat from countries other than Britain until the scientific advice was clearer .\"]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conll_texts = [\" \".join(sample_tokens) for sample_tokens in conll_dataset_nonempty[\"train\"][\"tokens\"]]\n",
    "conll_val_texts = [\" \".join(sample_tokens) for sample_tokens in conll_dataset_nonempty[\"validation\"][\"tokens\"]]\n",
    "conll_texts[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "^^ this is a default input format of TokenClassification objective\n",
    "\n",
    "To let the tuned transformer model remember the mapping of CoNLL's integer labels to their actual identifiers, we only need to map the integers to these identifiers and separate them with \" \", (spaces). This way, each label correspond to a single word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e59CsQ8HhCpj",
    "outputId": "b3b8f659-48d9-413f-96fb-823ef415b14d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-ORG O B-MISC O O O B-MISC O O',\n",
       " 'B-PER I-PER',\n",
       " 'B-LOC O',\n",
       " 'O B-ORG I-ORG O O O O O O B-MISC O O O O O B-MISC O O O O O O O O O O O O O O',\n",
       " 'B-LOC O O O O B-ORG I-ORG O O O B-PER I-PER O O O O O O O O O O O B-LOC O O O O O O O']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_map = {0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
    "label_map_sorted = dict(())\n",
    "\n",
    "conll_labels = [\" \".join(label_map[tag_id] for tag_id in sample) for sample in conll_dataset_nonempty[\"train\"][\"ner_tags\"]]\n",
    "conll_val_labels = [\" \".join(label_map[tag_id] for tag_id in sample) for sample in conll_dataset_nonempty[\"validation\"][\"ner_tags\"]]\n",
    "\n",
    "conll_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "^^ this is a default format of the labels of TokenClassification objective\n",
    "\n",
    "### Training setup\n",
    "\n",
    "Using Adaptor, we compose our training process from the selected `Objective`s. This is fairly straight with a single fine-tuning Objective.\n",
    "In addition, we can use an arbirary set of compatible `Evaluator`s. You can also define your own Evaluators (as shown later below).\n",
    "\n",
    "Adaptor Objectives provide high-level interface, expecting both the input texts and labels in a form of:\n",
    "* either a `List[str]`, with the texts and labels of the matching length\n",
    "* or a paths to a `.txt` files with one sample / label per line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KOzboGA5g8od",
    "outputId": "b2086f48-66fb-45e5-a50e-a1c3f734ed11"
   },
   "outputs": [],
   "source": [
    "from adaptor.objectives.classification import TokenClassification\n",
    "from adaptor.evaluators.token_classification import MeanPerCategoryFScore\n",
    "from adaptor.lang_module import LangModule\n",
    "\n",
    "lang_module = LangModule(\"bert-base-cased\")\n",
    "\n",
    "ner_evaluators = [MeanFScore()]\n",
    "\n",
    "ner_objective = TokenClassification(lang_module,\n",
    "                                    batch_size=8,\n",
    "                                    texts_or_path=conll_texts,\n",
    "                                    labels_or_path=conll_labels,\n",
    "                                    val_texts_or_path=conll_val_texts[:1000],\n",
    "                                    val_labels_or_path=conll_val_labels[:1000],\n",
    "                                    val_evaluators=ner_evaluators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we are done with the datasets, objectives and evaluators, we set up the `AdaptationArguments`. These are a small extension of 🤗 's [TrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer?highlight=launch#transformers.TrainingArguments), the extra parameters are documented with [AdaptationArguments definition](https://github.com/gaussalgo/adaptor/blob/db33e6e439babc68fe801a8946d87116ff44f170/adaptor/utils.py#L77)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gA5opjLokI5A",
    "outputId": "1830835e-46f4-42d5-c9b6-99bc7352636e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "comet_ml is installed but `COMET_API_KEY` is not set.\n"
     ]
    }
   ],
   "source": [
    "from adaptor.utils import StoppingStrategy, AdaptationArguments\n",
    "\n",
    "training_arguments = AdaptationArguments(output_dir=\"ner_training_checkpoints\",\n",
    "                                         stopping_strategy=StoppingStrategy.FIRST_OBJECTIVE_CONVERGED,\n",
    "                                         do_train=True,\n",
    "                                         do_eval=True,\n",
    "                                         gradient_accumulation_steps=4,\n",
    "                                         evaluation_strategy=\"steps\",\n",
    "                                         # log_level=\"critical\",\n",
    "                                         logging_steps=200,\n",
    "                                         eval_steps=500,\n",
    "                                         num_train_epochs=10,\n",
    "                                         save_steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we define a `Schedule`, defining an order of application of selected `Objective`s. If our training is a single-objective, we can pick any Schedule available - it makes no difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "Q6-GmxlbgzAW"
   },
   "outputs": [],
   "source": [
    "from adaptor.schedules import SequentialSchedule\n",
    "\n",
    "schedule = SequentialSchedule(objectives=[ner_objective], args=training_arguments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define a main object called `Adapter`: this is again merely a small adjustment of 🤗  [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer?highlight=launch#transformers.Trainer), that takes care of data iteration according to selected `Schedule`, collection of `Objective`s' logs or applying selected multi-objective early-stopping strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "2kbzF8z6l9NZ"
   },
   "outputs": [],
   "source": [
    "from adaptor.adapter import Adapter\n",
    "\n",
    "adapter = Adapter(lang_module, schedule, training_arguments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having the identical iterface, we initialize the training in the same way as with 🤗 Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FNXdUxq1mIVr",
    "outputId": "f2e4d8d0-512b-4e06-b160-6c2cac0224fb"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 17550\n",
      "  Num Epochs = 10\n",
      "  Instantaneous batch size per device = 1\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 4\n",
      "  Gradient Accumulation steps = 4\n",
      "  Total optimization steps = 43870\n",
      "Converged objectives: []\n",
      "TokenClassification:  46%|████▌     | 799/1755 [01:03<01:06, 14.38batches/s, epoch=1, loss=0.408, split=train]  Converged objectives: []\n",
      "TokenClassification:  46%|████▌     | 803/1755 [01:03<01:30, 10.57batches/s, epoch=1, loss=0.223, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.3426, 'learning_rate': 4.9772053795304315e-05, 'train_TokenClassification_loss': 0.34259614028735086, 'train_TokenClassification_num_batches': 800, 'epoch': 0.05}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  91%|█████████ | 1600/1755 [02:06<00:13, 11.54batches/s, epoch=1, loss=0.22, split=train]   Converged objectives: []\n",
      "TokenClassification:  91%|█████████▏| 1602/1755 [02:07<00:22,  6.71batches/s, epoch=1, loss=0.0128, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.1504, 'learning_rate': 4.954410759060862e-05, 'train_TokenClassification_loss': 0.15091142149188091, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.09}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification: 100%|██████████| 1755/1755 [02:21<00:00, 14.14batches/s, epoch=1, loss=0.00926, split=train]Converged objectives: []\n",
      "TokenClassification: 1756batches [02:21, 12.41batches/s, epoch=1, loss=0.00254, split=train]                     \n",
      "TokenClassification:  14%|█▍        | 243/1755 [00:19<01:50, 13.68batches/s, epoch=2, loss=0.571, split=train]  Evaluating...\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 125\n",
      "  Batch size = 1\n",
      "TokenClassification: 1000batches [00:25, 39.86batches/s, epoch=2, loss=0.00392, split=eval]0.0117, split=train]\n",
      "Converged objectives: []\n",
      "TokenClassification:  14%|█▍        | 247/1755 [01:32<3:13:01,  7.68s/batches, epoch=2, loss=0.0133, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.11448673903942108, 'eval_runtime': 25.0907, 'eval_samples_per_second': 318.844, 'eval_steps_per_second': 318.844, 'eval_TokenClassification_loss': 0.11448674616022618, 'eval_TokenClassification_num_batches': 1000, 'eval_TokenClassification_MeanPerCategoryFScore': 0.1062371446113906, 'epoch': 0.11}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  37%|███▋      | 642/1755 [02:03<01:18, 14.18batches/s, epoch=2, loss=0.108, split=train]    Converged objectives: []\n",
      "TokenClassification:  37%|███▋      | 648/1755 [02:04<01:47, 10.34batches/s, epoch=2, loss=0.0857, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0911, 'learning_rate': 4.931616138591293e-05, 'train_TokenClassification_loss': 0.09676574244932272, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.14}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  82%|████████▏ | 1443/1755 [03:10<00:27, 11.22batches/s, epoch=2, loss=0.0208, split=train]  Converged objectives: []\n",
      "TokenClassification:  82%|████████▏ | 1446/1755 [03:10<00:43,  7.06batches/s, epoch=2, loss=0.0185, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0727, 'learning_rate': 4.9088215181217234e-05, 'train_TokenClassification_loss': 0.07120190719582024, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.18}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification: 100%|██████████| 1755/1755 [03:38<00:00, 14.12batches/s, epoch=2, loss=0.0022, split=train]  Converged objectives: []\n",
      "TokenClassification: 1756batches [03:38,  8.05batches/s, epoch=2, loss=0.00155, split=train]                    \n",
      "TokenClassification:  28%|██▊       | 488/1755 [00:39<01:17, 16.44batches/s, epoch=3, loss=0.00385, split=train] Converged objectives: []\n",
      "Evaluating...\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 125\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0464, 'learning_rate': 4.886026897652155e-05, 'train_TokenClassification_loss': 0.05340490972591215, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:   0%|          | 0/125 [03:32<?, ?batches/s, epoch=2, loss=-1, split=eval]\n",
      "TokenClassification: 1000batches [00:24, 40.08batches/s, epoch=3, loss=0.000766, split=eval].00385, split=train]\n",
      "Converged objectives: []\n",
      "tokenizer config file saved in ner_training_checkpoints/checkpoint-1000/TokenClassification/tokenizer_config.json\n",
      "Special tokens file saved in ner_training_checkpoints/checkpoint-1000/TokenClassification/special_tokens_map.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.032869964838027954, 'eval_runtime': 24.9592, 'eval_samples_per_second': 320.524, 'eval_steps_per_second': 320.524, 'eval_TokenClassification_loss': 0.07367835350268433, 'eval_TokenClassification_num_batches': 2000, 'eval_TokenClassification_MeanPerCategoryFScore': 0.11058395881654728, 'epoch': 0.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in ner_training_checkpoints/checkpoint-1000/TokenClassification/config.json\n",
      "Model weights saved in ner_training_checkpoints/checkpoint-1000/TokenClassification/pytorch_model.bin\n",
      "TokenClassification:  73%|███████▎  | 1287/1755 [03:26<00:46, 10.09batches/s, epoch=3, loss=0.0202, split=train]   Converged objectives: []\n",
      "TokenClassification:  74%|███████▎  | 1291/1755 [03:26<00:59,  7.79batches/s, epoch=3, loss=0.000707, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0391, 'learning_rate': 4.863232277182585e-05, 'train_TokenClassification_loss': 0.04019417429540772, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification: 1756batches [04:06, 15.25batches/s, epoch=3, loss=0.00049, split=train]                      Converged objectives: []\n",
      "TokenClassification: 1756batches [04:06,  7.11batches/s, epoch=3, loss=0.00049, split=train]\n",
      "TokenClassification:  19%|█▉        | 332/1755 [00:26<02:01, 11.70batches/s, epoch=4, loss=0.781, split=train]   Converged objectives: []\n",
      "TokenClassification:  19%|█▉        | 334/1755 [00:26<03:26,  6.87batches/s, epoch=4, loss=0.215, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0333, 'learning_rate': 4.840437656713016e-05, 'train_TokenClassification_loss': 0.03447006914763188, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.32}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  42%|████▏     | 732/1755 [00:56<01:11, 14.35batches/s, epoch=4, loss=0.000473, split=train]Evaluating...\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 125\n",
      "  Batch size = 1\n",
      "TokenClassification:   0%|          | 0/125 [03:58<?, ?batches/s, epoch=3, loss=-1, split=eval]\n",
      "TokenClassification: 1000batches [00:25, 39.77batches/s, epoch=4, loss=0.000602, split=eval].000473, split=train]\n",
      "TokenClassification: 100%|██████████| 1755/1755 [03:57<00:00, 14.76batches/s, epoch=7, loss=0.000133, split=train]Converged objectives: []\n",
      "TokenClassification: 1756batches [03:57,  7.40batches/s, epoch=7, loss=0.000174, split=train]                     \n",
      "TokenClassification:  29%|██▉       | 508/1755 [00:39<01:19, 15.59batches/s, epoch=8, loss=0.000146, split=train]Converged objectives: []\n",
      "TokenClassification:  29%|██▉       | 512/1755 [00:40<02:06,  9.82batches/s, epoch=8, loss=0.000339, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0067, 'learning_rate': 4.6352860724868935e-05, 'train_TokenClassification_loss': 0.0077938092411204704, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.73}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  75%|███████▍  | 1308/1755 [01:43<00:29, 15.38batches/s, epoch=8, loss=0.000492, split=train]Converged objectives: []\n",
      "TokenClassification:  75%|███████▍  | 1310/1755 [01:44<00:47,  9.35batches/s, epoch=8, loss=0.0107, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0077, 'learning_rate': 4.612491452017324e-05, 'train_TokenClassification_loss': 0.0075396578505533395, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.77}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  97%|█████████▋| 1707/1755 [02:18<00:04, 10.91batches/s, epoch=8, loss=0.000394, split=train]Evaluating...\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 125\n",
      "  Batch size = 1\n",
      "TokenClassification:   0%|          | 0/125 [03:52<?, ?batches/s, epoch=7, loss=-1, split=eval]\n",
      "TokenClassification: 1000batches [00:25, 39.74batches/s, epoch=8, loss=0.000123, split=eval]0.102, split=train]   \n",
      "Converged objectives: []\n",
      "TokenClassification:  97%|█████████▋| 1710/1755 [03:31<06:46,  9.04s/batches, epoch=8, loss=0.000161, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.006507412530481815, 'eval_runtime': 25.1699, 'eval_samples_per_second': 317.84, 'eval_steps_per_second': 317.84, 'eval_TokenClassification_loss': 0.02726664596444581, 'eval_TokenClassification_num_batches': 7000, 'eval_TokenClassification_MeanPerCategoryFScore': 0.11259841086486444, 'epoch': 0.8}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification: 1756batches [03:35, 13.49batches/s, epoch=8, loss=0.000184, split=train]                     Converged objectives: []\n",
      "TokenClassification: 1756batches [03:35,  8.16batches/s, epoch=8, loss=0.000184, split=train]\n",
      "TokenClassification:  20%|██        | 352/1755 [00:28<02:14, 10.40batches/s, epoch=9, loss=0.0921, split=train]  Converged objectives: []\n",
      "TokenClassification:  20%|██        | 354/1755 [00:29<03:29,  6.69batches/s, epoch=9, loss=0.0188, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0072, 'learning_rate': 4.5896968315477554e-05, 'train_TokenClassification_loss': 0.007918790022467875, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.82}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  66%|██████▌   | 1152/1755 [01:30<00:46, 12.98batches/s, epoch=9, loss=0.00713, split=train] Converged objectives: []\n",
      "TokenClassification:  66%|██████▌   | 1156/1755 [01:31<01:02,  9.54batches/s, epoch=9, loss=0.000517, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0052, 'learning_rate': 4.566902211078185e-05, 'train_TokenClassification_loss': 0.005010867099506868, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.87}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification: 1756batches [02:15, 16.91batches/s, epoch=9, loss=9.97e-5, split=train]                      Converged objectives: []\n",
      "TokenClassification: 1756batches [02:15, 12.98batches/s, epoch=9, loss=9.97e-5, split=train]\n",
      "TokenClassification:  11%|█         | 195/1755 [00:13<01:58, 13.11batches/s, epoch=10, loss=9.25e-5, split=train] Scheduler reached the given maximum number of epochs for all objectives. Triggering termination.\n",
      "Scheduler reached a termination condition: ALL_OBJECTIVES_NUM_EPOCHS\n",
      "Evaluating...\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 125\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0069, 'learning_rate': 4.5441075906086166e-05, 'train_TokenClassification_loss': 0.0064484594378245675, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.91}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:   0%|          | 0/125 [03:20<?, ?batches/s, epoch=8, loss=-1, split=eval]\n",
      "TokenClassification:   1%|          | 1/125 [00:00<00:03, 36.52batches/s, epoch=10, loss=0.000128, split=eval]\n",
      "TokenClassification:  11%|█         | 196/1755 [00:24<01:58, 13.11batches/s, epoch=10, loss=0.000163, split=train]Scheduler reached the given maximum number of epochs for all objectives. Triggering termination.\n",
      "Scheduler reached a termination condition: ALL_OBJECTIVES_NUM_EPOCHS\n",
      "tokenizer config file saved in ner_training_checkpoints/checkpoint-4000/TokenClassification/tokenizer_config.json\n",
      "Special tokens file saved in ner_training_checkpoints/checkpoint-4000/TokenClassification/special_tokens_map.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.00012769266322720796, 'eval_runtime': 0.0302, 'eval_samples_per_second': 265.095, 'eval_steps_per_second': 265.095, 'eval_TokenClassification_loss': 0.027262769524894145, 'eval_TokenClassification_num_batches': 7001, 'eval_TokenClassification_MeanPerCategoryFScore': 0.11273161205067099, 'epoch': 0.91}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in ner_training_checkpoints/checkpoint-4000/TokenClassification/config.json\n",
      "Model weights saved in ner_training_checkpoints/checkpoint-4000/TokenClassification/pytorch_model.bin\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "Scheduler reached the given maximum number of epochs for all objectives. Triggering termination.\n",
      "Scheduler reached a termination condition: ALL_OBJECTIVES_NUM_EPOCHS\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 1951.3271, 'train_samples_per_second': 89.939, 'train_steps_per_second': 22.482, 'train_loss': 0.0467430756688118, 'train_TokenClassification_loss': 0.0064484594378245675, 'train_TokenClassification_num_batches': 1000, 'epoch': 0.91}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=4000, training_loss=0.0467430756688118, metrics={'train_runtime': 1951.3271, 'train_samples_per_second': 89.939, 'train_steps_per_second': 22.482, 'total_flos': 0.0, 'train_loss': 0.0467430756688118})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adapter.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training terminates when the selected `StoppingStrategy` is satisfied. There is a slightly larger list of options to pick from, to cover the wider variety of multi-objective situations. See the [StoppingStrategy options](https://github.com/gaussalgo/adaptor/blob/db33e6e439babc68fe801a8946d87116ff44f170/adaptor/utils.py#L19)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display logs\n",
    "\n",
    "Objectives have **states** that remember their most-recent evaluations, that you can directly use to verify Objective's convergence, or accuracy\n",
    "\n",
    "However, it might be more convenient to use **external logging** tool like *Tensorboard*, *Comet.ml*, or *Weights & Biases*, that will also track other info for you. You can (and are encouraged to) use any logging tools supported by 🤗  Transformers; see [here](https://huggingface.co/docs/transformers/main_classes/logging)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([<adaptor.evaluators.token_classification.MeanFScore object at 0x7f720bdb9a20>, 'loss'])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner_objective.evaluations_history['eval'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3AAAAGbCAYAAACF9nK/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4hUlEQVR4nO3de3xc1X3v/e9vrpJmJNm6WL4SS9hcDAECjk0CBQcSCpwGJ6dwQvr0gaScOk1C2zw5Pae0PU3zoknbpKehTUsvziENgZwCIU3jpzhQAojcwBgTbsYY5AvYxhdk2bJ118ys88fskUdCskfSntkz0uf9es1r9uzZM/qNtTya76y11zLnnAAAAAAA5S8UdAEAAAAAgMIQ4AAAAACgQhDgAAAAAKBCEOAAAAAAoEIQ4AAAAACgQkSCLmCspqYmt3Tp0qDLeIfe3l4lEomgy8AMQXuCX2hL8BPtCX6hLcFPs7E9bdmypdM51zzefWUX4JYuXapnn3026DLeob29XWvWrAm6DMwQtCf4hbYEP9Ge4BfaEvw0G9uTmb0x0X0MoQQAAACACkGAAwAAAIAKQYADAAAAgApBgAMAAACACkGAAwAAAIAKQYADAAAAgApBgAMAAACACkGAAwAAAIAKQYADAAAAgApBgAMAAACACkGAAwAAAIAKQYADAAAAgApBgAMAAACACkGAAwAAAIAKQYArQP9QWnuOZ9Q7mAq6FAAAAACzGAGuAFveOKI//lm/Xt7XHXQpAAAAAGYxAlwBGpMxSdLh3qGAKwEAAAAwmxHgCtCYIMABAAAACB4BrgBzcwGuZzDgSgAAAADMZgS4AkTDISWi0uEeeuAAAAAABIcAV6C6mOlwLz1wAAAAAIJDgCtQXczUSQ8cAAAAgAAR4ApUGzPOgQMAAAAQKAJcgerixiyUAAAAAAJFgCtQXcx0tG9Yw+lM0KUAAAAAmKUIcAWqi5kk6Qi9cAAAAAACQoArUK0X4BhGCQAAACAoBLgC5XrgWAsOAAAAQFAIcAU60QPHTJQAAAAAgkGAK1CuB4614AAAAAAEhQBXoJqoFAmxFhwAAACA4BQU4MzsajPbbmYdZnbbOPdfZmbPmVnKzK7P23+BmT1lZlvN7EUz+5ifxZdSyEwNiRjnwAEAAAAIzCkDnJmFJd0p6RpJKyR93MxWjDnsTUmfkPR/xuzvk3STc+4cSVdL+mszmzPNmgPTmIxzDhwAAACAwEQKOGaVpA7n3E5JMrP7JK2V9EruAOfcbu++UatcO+dey9t+y8wOSWqWdHS6hQehKRnjHDgAAAAAgSkkwC2StCfv9l5Jqyf7g8xslaSYpB3j3LdO0jpJamlpUXt7+2Sfvuh6enqU6hnQvqOZsqwPlaWnp4d2BF/QluAn2hP8QluCn2hPoxUS4KbNzBZIukfSzc65zNj7nXPrJa2XpJUrV7o1a9aUoqxJaW9v14rT5+nFZ95UOdaHytLe3k47gi9oS/AT7Ql+oS3BT7Sn0QqZxGSfpCV5txd7+wpiZnWSHpL0R865pydXXnlpSMTUO5RW/1A66FIAAAAAzEKFBLjNkpabWauZxSTdKGlDIU/uHf99Sd92zj049TLLQ1MyJonFvAEAAAAE45QBzjmXknSrpEckbZP0gHNuq5ndbmbXSZKZvdfM9kq6QdI/mdlW7+H/RdJlkj5hZs97lwuK8UJKoTERlySWEgAAAAAQiILOgXPObZS0ccy+L+Rtb1Z2aOXYx90r6d5p1lg2GumBAwAAABCgghbyRlZTMtsDx1ICAAAAAIJAgJuEkR44AhwAAACAABDgJqEmFlF1NKzDPQyhBAAAAFB6BLhJakzGdLiXHjgAAAAApUeAm6TGZFyd9MABAAAACAABbpIaEzF10QMHAAAAIAAEuElqTMSYxAQAAABAIAhwk9SYjOtw76Ccc0GXAgAAAGCWIcBNUlMypuG007GBVNClAAAAAJhlCHCTdGItOCYyAQAAAFBaBLhJakzEJYmlBAAAAACUHAFukuiBAwAAABAUAtwkNSWzPXCdzEQJAAAAoMQIcJM0tybXA0eAAwAAAFBaBLhJikVCqquKqKuXIZQAAAAASosANwVNybg6mcQEAAAAQIkR4KagMRljEhMAAAAAJUeAm4LGRJxz4AAAAACUHAFuChqTMdaBAwAAAFByBLgpaEzGdaRvSKl0JuhSAAAAAMwiBLgpaErG5Jx0pG846FIAAAAAzCIEuCloTGQX8z7MUgIAAAAASogANwWNSRbzBgAAAFB6BLgpaMoFOCYyAQAAAFBCBLgpaMgNoWQtOAAAAAAlRICbgjnVUYWMIZQAAAAASosANwWhkKkhEWcSEwAAAAAlRYCboqZkTJ30wAEAAAAoIQLcFDUmY5wDBwAAAKCkCHBT1JiIMwslAAAAgJIiwE1RtgeOAAcAAACgdAhwU9SUjKtnMKWB4XTQpQAAAACYJQhwU9SYyC7m3cUwSgAAAAAlQoCbogYvwDGMEgAAAECpEOCmqDEZlyR1shYcAAAAgBIhwE1RU5IeOAAAAAClRYCbolwPHGvBAQAAACgVAtwUJWJhxSMh1oIDAAAAUDIEuCkyMzUl4+qkBw4AAABAiRDgpoHFvAEAAACUEgFuGhoTMR1mFkoAAAAAJUKAm4bGZJweOAAAAAAlQ4CbhmwP3JCcc0GXAgAAAGAWIMBNQ2MypqFURj2DqaBLAQAAADALEOCmoTGRWwuOYZQAAAAAio8ANw2NyZgkMZEJAAAAgJIoKMCZ2dVmtt3MOszstnHuv8zMnjOzlJldP+a+m83sde9ys1+Fl4OmZLYHrpMeOAAAAAAlcMoAZ2ZhSXdKukbSCkkfN7MVYw57U9InJP2fMY9tkPQnklZLWiXpT8xs7vTLLg8jPXAEOAAAAAAlUEgP3CpJHc65nc65IUn3SVqbf4Bzbrdz7kVJmTGP/WVJjzrnupxzRyQ9KulqH+ouCw2JXIBjCCUAAACA4osUcMwiSXvybu9VtketEOM9dtHYg8xsnaR1ktTS0qL29vYCn750enp6xq2rOiK9sH2n2sP7Sl8UKtZE7QmYLNoS/ER7gl9oS/AT7Wm0QgJc0Tnn1ktaL0krV650a9asCbagcbS3t2u8uuY/266qOXVas+bC0heFijVRewImi7YEP9Ge4BfaEvxEexqtkCGU+yQtybu92NtXiOk8tiI0JmLq6uUcOAAAAADFV0iA2yxpuZm1mllM0o2SNhT4/I9IusrM5nqTl1zl7ZsxGhIxJjEBAAAAUBKnDHDOuZSkW5UNXtskPeCc22pmt5vZdZJkZu81s72SbpD0T2a21Xtsl6Q/VTYEbpZ0u7dvxmhMxlkHDgAAAEBJFHQOnHNuo6SNY/Z9IW97s7LDI8d77DclfXMaNZa1pmR2CGU64xQOWdDlAAAAAJjBClrIGxNrTMSUcdLRPoZRAgAAACguAtw0NSbjkqTDTGQCAAAAoMgIcNPUmMwu5t3JYt4AAAAAiowAN01NuR44ZqIEAAAAUGQEuGlqTGR74A7TAwcAAACgyAhw0zSnJqaQicW8AQAAABQdAW6awiHT3JqYOglwAAAAAIqMAOeDxmSMIZQAAAAAio4A54PGRJxJTAAAAAAUHQHOB43JGOvAAQAAACg6ApwPmpJx1oEDAAAAUHQEOB80JmI6PpDSYCoddCkAAAAAZjACnA8avcW8WUoAAAAAQDER4HzQmMwt5k2AAwAAAFA8BDgfNHkBjvPgAAAAABQTAc4HDQmGUAIAAAAoPgKcDxhCCQAAAKAUCHA+qI1HFAuH1NnLEEoAAAAAxUOA84GZZRfzpgcOAAAAQBER4HySDXD0wAEAAAAoHgKcTxoTcR1mEhMAAAAARUSA8wlDKAEAAAAUGwHOJ03JuDp7BuWcC7oUAAAAADMUAc4njYmYBlMZ9Q6lgy4FAAAAwAxFgPPJ/PoqSdJbR/sDrgQAAADATEWA80lbU1KStPPtnoArAQAAADBTEeB8srSpRpK0s7M34EoAAAAAzFQEOJ/UVkXVXBvXrrcJcAAAAACKgwDno7amBD1wAAAAAIqGAOejtuaEdhHgAAAAABQJAc5HrU0JdfUO6WgfC3oDAAAA8B8BzkcjM1HSCwcAAACgCAhwPmptTkgSE5kAAAAAKAoCnI9Oa6hROGScBwcAAACgKAhwPoqGQzqtoUY7O1nMGwAAAID/CHA+a21KaCdDKAEAAAAUAQHOZ21NCe0+3KtMxgVdCgAAAIAZhgDns9bmhAaGM9p/bCDoUgAAAADMMAQ4n7U2MRMlAAAAgOIgwPns9ObsWnC7mMgEAAAAgM8IcD6bVxtXTSysHfTAAQAAAPAZAc5nZqbWpgRrwQEAAADwHQGuCNqakwQ4AAAAAL4jwBVBa1NCe4/0aTCVDroUAAAAADMIAa4I2poSyjjpzcN9QZcCAAAAYAYhwBVBW3N2KYGdDKMEAAAA4KOCApyZXW1m282sw8xuG+f+uJnd792/ycyWevujZna3mb1kZtvM7A98rr8sLfXWgtvJTJQAAAAAfHTKAGdmYUl3SrpG0gpJHzezFWMOu0XSEefcMkl3SPqKt/8GSXHn3LslXSTpU7lwN5PVVUXVlIyzFhwAAAAAXxXSA7dKUodzbqdzbkjSfZLWjjlmraS7ve0HJV1pZibJSUqYWURStaQhScd8qbzMtTWzlAAAAAAAf0UKOGaRpD15t/dKWj3RMc65lJl1S2pUNsytlbRfUo2k/8851zX2B5jZOknrJKmlpUXt7e2TexUl0NPTM6m6qoYH9YtDqbJ8LQjeZNsTMBHaEvxEe4JfaEvwE+1ptEIC3HSskpSWtFDSXEk/MbMfOed25h/knFsvab0krVy50q1Zs6bIZU1ee3u7JlPXdtuhH//wVb1n1SWqr4kWrzBUpMm2J2AitCX4ifYEv9CW4Cfa02iFDKHcJ2lJ3u3F3r5xj/GGS9ZLOizp1yQ97Jwbds4dkvQzSSunW3QlaGtOSpJ2HWYYJQAAAAB/FBLgNktabmatZhaTdKOkDWOO2SDpZm/7ekmPO+ecpDclXSFJZpaQdLGkV/0ovNy1jsxEyUQmAAAAAPxxygDnnEtJulXSI5K2SXrAObfVzG43s+u8w+6S1GhmHZI+Lym31MCdkpJmtlXZIPjPzrkX/X4R5ei0hhqFQ8ZEJgAAAAB8U9A5cM65jZI2jtn3hbztAWWXDBj7uJ7x9s8GsUhIS+ZWs5g3AAAAAN8UtJA3pqa1KcFi3gAAAAB8Q4ArotampHZ39iqTcUGXAgAAAGAGIMAVUVtzQv3DaR08PhB0KQAAAABmAAJcEbWNzETJMEoAAAAA00eAK6LcWnBMZAIAAADADwS4Imqpi6s6GtYueuAAAAAA+IAAV0Rmlp2JspPFvAEAAABMHwGuyNqaEyzmDQAAAMAXBLgia2tKaE9Xn4ZSmaBLAQAAAFDhCHBF1tqcUMZJb3bRCwcAAABgeghwRdbW5M1EyUQmAAAAAKaJAFdkS7214DgPDgAAAMB0EeCKrL46qqZkjB44AAAAANNGgCuBtqYkPXAAAAAApo0AVwLZteAIcAAAAACmhwBXAq3NCXX2DOrYwHDQpQAAAACoYAS4EmjLTWTCeXAAAAAApoEAVwJtzcxECQAAAGD6CHAlsKShRiGTdr7dE3QpAAAAACoYAa4E4pGwljTUMJEJAAAAgGkhwJVIa1OCIZQAAAAApoUAVyK5AOecC7oUAAAAABWKAFcibc1J9Q2ldfDYYNClAAAAAKhQBLgSyS0lsLOTiUwAAAAATA0BrkRacwGOteAAAAAATBEBrkTm11WpOhpmIhMAAAAAU0aAK5FQyLSUmSgBAAAATAMBroTamhIs5g0AAABgyghwJXTm/Fq90dWn7r7hoEsBAAAAUIEIcCW0qrVBzknPvtEVdCkAAAAAKhABroQuWDJHsXBIm3YR4AAAAABMHgGuhKqiYV2wZA4BDgAAAMCUEOBKbFVrg17e162ewVTQpQAAAACoMAS4Elvd1qB0xum5N44EXQoAAACACkOAK7ELT5urcMi0adfhoEsBAAAAUGEIcCWWiEd07qJ6PcN5cAAAAAAmiQAXgItbG/TCnm4NDKeDLgUAAABABSHABWBVa4OG0hn94s2jQZcCAAAAoIIQ4AKwcmmDzMR5cAAAAAAmhQAXgPrqqM6eX8d5cAAAAAAmhQAXkNVtDXruzSMaSmWCLgUAAABAhSDABWR1a4MGhjN6ad/RoEsBAAAAUCEIcAF579IGSdLTOxlGCQAAAKAwBLiANCbjWj4vyXlwAAAAAApGgAvQ6rYGPbu7S6k058EBAAAAODUCXIBWtTaqdyitV/YfC7oUAAAAABWgoABnZleb2XYz6zCz28a5P25m93v3bzKzpXn3nWdmT5nZVjN7ycyqfKy/oq1uzZ4Ht4nz4AAAAAAU4JQBzszCku6UdI2kFZI+bmYrxhx2i6Qjzrllku6Q9BXvsRFJ90r6LefcOZLWSBr2rfoK11JXpaWNNdrEeXAAAAAAClBID9wqSR3OuZ3OuSFJ90laO+aYtZLu9rYflHSlmZmkqyS96Jx7QZKcc4edc2l/Sp8ZVrU2aPPuLmUyLuhSAAAAAJS5SAHHLJK0J+/2XkmrJzrGOZcys25JjZLOkOTM7BFJzZLuc859dewPMLN1ktZJUktLi9rb2yf5Moqvp6enKHXVDw6ru39Y33noCS2p5ZTE2aJY7QmzD20JfqI9wS+0JfiJ9jRaIQFuus9/qaT3SuqT9JiZbXHOPZZ/kHNuvaT1krRy5Uq3Zs2aIpc1ee3t7SpGXad39ekbLz2hdEOr1lzS6vvzozwVqz1h9qEtwU+0J/iFtgQ/0Z5GK6TLZ5+kJXm3F3v7xj3GO++tXtJhZXvrfuyc63TO9UnaKOnC6RY9kyxpqNGiOdV6ZjfnwQEAAAA4uUIC3GZJy82s1cxikm6UtGHMMRsk3extXy/pceeck/SIpHebWY0X7C6X9Io/pc8cq1ob9MyuLmX/yQAAAABgfKcMcM65lKRblQ1j2yQ94Jzbama3m9l13mF3SWo0sw5Jn5d0m/fYI5K+pmwIfF7Sc865h3x/FRVudWuDOnuGtOPt3qBLAQAAAFDGCjoHzjm3Udnhj/n7vpC3PSDphgkee6+ySwlgAqty68HtOqxl85IBVwMAAACgXDHtYRlobUqouTauZ1gPDgAAAMBJEODKgJlpVWuDNu3kPDgAAAAAEyPAlYmLWxt04NiA9nT1B10KAAAAgDJFgCsTq1obJUlP7zoccCUAAAAAyhUBrkwsn5fU3Joo58EBAAAAmBABrkyEQqb3Lm3QJnrgAAAAAEyAAFdGVrU2aE9Xv946ynlwAAAAAN6JAFdGLm7Lnge3eTfDKAEAAAC8EwGujJy9oE618Yie3kmAAwAAAPBOBLgyEg6ZVrc16Mnth5TOsB4cAAAAgNEIcGXmugsW6a3uAT21g8lMAAAAAIxGgCszV61oUV1VRN/dsifoUgAAAACUGQJcmamKhnXdBQv18MsH1N0/HHQ5AAAAAMoIAa4M3XDREg2mMvr3F98KuhQAAAAAZYQAV4bOW1yv5fOS+u6ze4MuBQAAAEAZIcCVITPTDSsX6/k9R9Vx6HjQ5QAAAAAoEwS4MvWR9yxSOGT67hZ64QAAAABkEeDK1LzaKn3gzGb963P7lEpngi4HAAAAQBkgwJWx6y9aorePD+rHr78ddCkAAAAAygABroxdcdY8NSRiTGYCAAAAQBIBrqzFIiGtvWChfrTtoI70DgVdDgAAAICAEeDK3A0XLdFw2ukHz+8LuhQAAAAAASPAlbkVC+t0zsI6ZqMEAAAAQICrBDdctFhb3zqmV946FnQpAAAAAAJEgKsAay9YpGjY9N0te4IuBQAAAECACHAVYG4ipg+e3aIfPP+WhlKsCQcAAADMVgS4CnHDysXq6h3S468eCroUAAAAAAEhwFWIy5Y3a15tXA8yjBIAAACYtQhwFSISDumjFy7SE9vf1qHjA0GXAwAAACAABLgKcsNFi5XOOP3bL1gTDgAAAJiNCHAVZNm8Wl2wZI4e3LJXzrmgywEAAABQYgS4CnPDysV67WCPXtzbHXQpAAAAAEqMAFdhPnz+QsUjIX1n0xtBlwIAAACgxAhwFaauKqqPrzpN33tun14/eDzocgAAAACUEAGuAv3OlctVEw3rL374atClAAAAACghAlwFakjE9JkPLNNjrx7Sz3d0Bl0OAAAAgBIhwFWoT16yVAvrq/RnG7cpk2FGSgAAAGA2IMBVqKpoWP/96jP18r5j+sELrAsHAAAAzAYEuAq29vxFOndRnf7XI69pYDgddDkAAAAAiowAV8FCIdMfXnu29h3t1z//bHfQ5QAAAAAoMgJchXv/6U268qx5+vsnOtTVOxR0OQAAAACKiAA3A9x2zVnqHUrp64+9HnQpAAAAAIqIADcDLG+p1Y2rTtO9T7+hXZ29QZcDAAAAoEgIcDPE5z64XLFISF9hcW8AAABgxiLAzRDzaqv0W5efroe3HtDm3V1BlwMAAACgCAhwM8h//aVWtdTF9eWHtsk5FvcGAAAAZpqCApyZXW1m282sw8xuG+f+uJnd792/ycyWjrn/NDPrMbPf86lujKMmFtF/+9CZen7PUT300v6gywEAAADgs1MGODMLS7pT0jWSVkj6uJmtGHPYLZKOOOeWSbpD0lfG3P81ST+cfrk4lV+9aLHOml+rrz68XYMpFvcGAAAAZpJCeuBWSepwzu10zg1Juk/S2jHHrJV0t7f9oKQrzcwkycw+ImmXpK2+VIyTCodMf3Dt2Xqzq0/f/OnuoMsBAAAA4KNIAccskrQn7/ZeSasnOsY5lzKzbkmNZjYg6fclfUjShMMnzWydpHWS1NLSovb29kLrL5menp6yrGsiF84L66/+41XV97yhhUlOdSw3ldaeUL5oS/AT7Ql+oS3BT7Sn0QoJcNPxRUl3OOd6vA65cTnn1ktaL0krV650a9asKXJZk9fe3q5yrGsi51w0qKvueFL3vxHX937rfYqECXHlpNLaE8oXbQl+oj3BL7Ql+In2NFohn+r3SVqSd3uxt2/cY8wsIqle0mFle+q+ama7JX1O0h+a2a3TKxmFaK6N6/a15+qFPUe1/ic7gy4HAAAAgA8KCXCbJS03s1Yzi0m6UdKGMcdskHSzt329pMdd1i8555Y655ZK+mtJf+ac+zt/Ssep/Mp5C3Ttu+frrx99XdsPHA+6HAAAAADTdMoA55xLSbpV0iOStkl6wDm31cxuN7PrvMPuUvactw5Jn5f0jqUGUHpmpj9de65qqyL6ve++oOF0JuiSAAAAAExDQefAOec2Sto4Zt8X8rYHJN1wiuf44hTqwzQ1JuP60kfO1ae/85z+sX2HfvvK5UGXBAAAAGCKmNliFrjm3Qv04fMX6uuPv65X3joWdDkAAAAApogAN0vcft05qq+O6fe++4KGUgylBAAAACoRAW6WmJuI6c8+eq5e2X9Mdz7REXQ5AAAAAKaAADeLXHXOfP3n9yzSnU906OV93UGXAwAAAGCSCHCzzJ98+Bw1JLJDKQdT6aDLAQAAADAJBLhZpr4mqr/41Xfr1QPH9bePMZQSAAAAqCQEuFnoirNadMNFi/UPT+7Qlje6gi4HAAAAQIEIcLPU//yVFVo0p1qfvvc5HTw2EHQ5AAAAAApAgJul6quj+sZNK9UzmNKn7tmigWHOhwMAAADKHQFuFjtzfq2+9l/O1/N7juqP/+1lOeeCLgkAAADASRDgZrmrz12g37limb67Za/u/vnuoMsBAAAAcBIEOOhzHzxDHzy7RX/60Db9fEdn0OUAAAAAmAABDgqFTHd87Hy1NiX02e88pz1dfUGXBAAAAGAcBDhIkmqrolr//16kVMZp3T1b1DeUCrokAAAAAGMQ4DCirTmpv/34e/TqgWP6Hw++yKQmAAAAQJkhwGGUNWfO0+9ffZb+/cX9+ocndwRdDgAAAIA8BDi8w6cua9N15y/UXz6yXU+8eijocgAAAAB4CHB4BzPTV371PK1YUKffue8Xenlfd9AlAQAAABABDhOojoW1/qaVqquK6te+8bRe2HM06JIAAACAWY8AhwktmlOt+z91seprovr1/71JW944EnRJAAAAwKxGgMNJLZ5bowc+9T41JmO66a5N2ry7K+iSAAAAgFmLAIdTWlBfrfs/9T611Ffpprue0VM7DgddEgAAADArEeBQkJa6Kt2/7n1a0lCtT37rGf309c6gSwIAAABmHQIcCtZcG9e//ObFWtqY0G/cvVnt21liAAAAACglAhwmpTGZDXFntCS17ttb9KNXDgZdEgAAADBrEOAwaXMTMX3nlot19oJa/da9W/Twy/uDLgkAAACYFQhwmJL6mqju+a+rdd7ien3mO8/p64+9rnTGBV0WAAAAMKMR4DBldVVR3XPLal13/kJ97dHX9Il/fkadPYNBlwUAAADMWAQ4TEsiHtEdH7tAf/6f361Nu7p07d/8RE/vZJkBAAAAoBgIcJg2M9PHV52mf/vMJUrEI/q1bzytO5/oUIYhlQAAAICvCHDwzYqFdfr/f/tS/afzFuovH9muT35rs7p6h4IuCwAAAJgxCHDwVTIe0ddvvEBf+si5emrnYV37Nz/Rs7u7gi4LAAAAmBEIcPCdmenXL36X/vXT71c8GtLH1j+te57aHXRZAAAAQMUjwKFozl1Ur3//7Ut1cVuDvrxxmwZT6aBLAgAAACoaAQ5FVVsV1Sfe36qB4Yx+8ebRoMsBAAAAKhoBDkW3uq1B4ZDpZx2dQZcCAAAAVDQCHIquriqq8xfX66cEOAAAAGBaCHAoiUuXNemFPUd1bGA46FIAAACAikWAQ0lcsqxJGSc9veNw0KUAAAAAFYsAh5J4z2lzVR0N6+cEOAAAAGDKCHAoiVgkpFWtDZwHBwAAAEwDAQ4lc+myJnUc6tGB7oGgSwEAAAAqEgEOJXPJsiZJYjkBAAAAYIoIcCiZs+bXqjERI8ABAAAAU0SAQ8mEQqb3L2vSTzs65ZwLuhwAAACg4hDgUFKXLmvUoeOD6jjUE3QpAAAAQMUpKMCZ2dVmtt3MOszstnHuj5vZ/d79m8xsqbf/Q2a2xcxe8q6v8Ll+VJjceXDMRgkAAABM3ikDnJmFJd0p6RpJKyR93MxWjDnsFklHnHPLJN0h6Sve/k5JH3bOvVvSzZLu8atwVKbFc2u0tLGG8+AAAACAKSikB26VpA7n3E7n3JCk+yStHXPMWkl3e9sPSrrSzMw59wvn3Fve/q2Sqs0s7kfhqFyXLGvS0zu7NJzOBF0KAAAAUFEiBRyzSNKevNt7Ja2e6BjnXMrMuiU1KtsDl/Orkp5zzg2O/QFmtk7SOklqaWlRe3t7ofWXTE9PT1nWVYnmDKbUM5jStzY8oeVzw0GXEwjaE/xCW4KfaE/wC20JfqI9jVZIgJs2MztH2WGVV413v3NuvaT1krRy5Uq3Zs2aUpQ1Ke3t7SrHuirRBX1D+vsXHlVf7Wlas2Z50OUEgvYEv9CW4CfaE/xCW4KfaE+jFTKEcp+kJXm3F3v7xj3GzCKS6iUd9m4vlvR9STc553ZMt2BUvjk1MZ27sJ7z4AAAAIBJKiTAbZa03MxazSwm6UZJG8Ycs0HZSUok6XpJjzvnnJnNkfSQpNuccz/zqWbMAJcsa9Iv9hxR72Aq6FIAAACAinHKAOecS0m6VdIjkrZJesA5t9XMbjez67zD7pLUaGYdkj4vKbfUwK2Slkn6gpk9713m+f4qUHEuXdak4bTTM7u7gi4FAAAAqBgFnQPnnNsoaeOYfV/I2x6QdMM4j/uSpC9Ns0bMQCuXzlUsEtLPXu/UB84k0wMAAACFKGghb8BvVdGw3rt0Lgt6AwAAAJNAgENgLlnWpFcPHNfbx9+xsgQAAACAcRDgEJhLlzVJkn6+g144AAAAoBAEOATmnIX1qq+OspwAAAAAUCACHAITDpnef3qjfvp6p5xzQZcDAAAAlD0CHAJ1ybImvdU9oN2H+4IuBQAAACh7BDgEKnceHLNRAgAAAKdGgEOg3tVYo0VzqvWz1wlwAAAAwKkQ4BAoM9Oly5r08x2dSmc4Dw4AAAA4GQIcAvf+ZY06NpDS1re6gy4FAAAAKGsEOATu/adzHhwAAABQCAIcAtdcG9fZC+r0b7/Yp6FUJuhyAAAAgLJFgENZ+PyHztBrB3v0T0/uCLoUAAAAoGwR4FAWPrSiRf/pvAX628c79PrB40GXAwAAAJQlAhzKxhc/fI5q4mH9/vdeZEZKAAAAYBwEOJSN5tq4vvArK/Tcm0d1z1O7gy4HAAAAKDsEOJSVj75nkS4/o1lffWS79nT1BV0OAAAAUFYIcCgrZqYvf/RcmaQ//P5Lco6hlAAAAEAOAQ5lZ/HcGv3+NWfpJ6936nvP7Qu6HAAAAKBsEOBQln599bu08l1z9af//ooOHR8IuhwAAACgLBDgUJZCIdNXrj9P/cNpfXHD1qDLAQAAAMoCAQ5l6/TmpH73yuXa+NIBPfzygaDLAQAAAAJHgENZW3dZm1YsqNMf/+BldfcNB10OAAAAECgCHMpaNBzSV68/T129Q/qzjduCLgcAAAAIFAEOZe/cRfX6zV9q0/3P7tHGl/YHXQ4AAAAQGAIcKsLnPrhcZy+o02e+85w+fe8W7T3CIt8AAACYfQhwqAhV0bC+/5n367//8pl6YvshffBrT+pvH3tdA8PpoEsDAAAASoYAh4pRFQ3rsx9Ypsf+2xpdcdY8/dWjr+mqO36sx7YdDLo0AAAAoCQIcKg4i+ZU6+//n4t07y2rFQ2bbrn7Wf3GtzZrd2dv0KUBAAAARUWAQ8W6dHmTfvi7l+mPrj1bm3Ye1lV3/Fj/65Ht6uwZDLo0AAAAoCgiQRcATEcsEtJvXtam6y5YqL/44av6uyc6dGd7hy46ba4+tKJFH1zRotObk0GXCQAAAPiCAIcZoaWuSnd87AJ96vI2PfzyAT36ykH9+Q9f1Z//8FW1NSf0oRUt+tDZLXrPaXMVDlnQ5QIAAABTQoDDjHLW/DqdNb9On/vgGdp3tF+PbTuoR185qLt+skv/9ORONSZiuuKsebrm3fN1ybImxSPhoEsGAAAACkaAw4y1aE61bnrfUt30vqU6NjCsJ7e/rUdfOaiHtx7Qd7fsVW08oivOnqdrzl2gy89oVnWMMAcAAIDyRoDDrFBXFdWHz1+oD5+/UEOpjH62o1MPv3RA//HKAf3g+bdUHQ3rirPm6epz5+sDZ81TMs5/DQAAAJQfPqVi1olFQvrAmfP0gTPn6cvpc7VpV5c2vrRfj2w9qIde2q9YJKTLljdpzZnzdPkZzVrSUBN0yQAAAIAkAhxmuUg4pEuWNemSZU26fe252vLGEW18ab8efeWgfrTtkCSprTmhy5Y36/Izm3VxayNDLQEAABAYAhzgCYdMq1obtKq1QX/y4RXa8Xavfvza23rytbf1L8+8qW/9fLdikZBWtzbo8jOadeG75qo2HlF1LKyaWEQ1sbDikZDMmOUSAAAAxUGAA8ZhZlo2L6ll85L6jUtbNTCc1jO7uvTka2/rx6+9rS89tG3cx4VMqo6GVe0FuppY2At4YVVHT+w7fGhQmwdfVU0songkpHDIFAmHFAlZ9hI2RULe7XBI1dGwauJhJfKeNxGPEBgBAABmGQIcUICqaFiXndGsy85oliTtO9qvV/cfU99QWv1DafUNpdQ3nNtOe/tT2evh7O2u3n71D6XUO5TW8b6UHntzhzJuenWFTNlQFw9rfn21Fs+t1pK5NVo8t9q7ZLerogz7BAAAmAkIcMAULJpTrUVzqqf8+Pb2dl1++eUaTGU0lM4onXYazmSUzjil0k6pjFMqnfGunRcCs4Gwd9C7HkqpbzAbDo8PDOvAsQFt3det/9h6QMPp0cmwuTau+XVVkqThdPbnpDPez0w7DXu3Q2ZqqYtrQX2VFtRXa359lRbUV2l+fZUWercJgwAAAMEhwAEBMTNVRcO+B6JMxunQ8UHtPdKnvUf6R64PHBtQyEzhkCkaNoVDIUVD5g3fzA7ZTGUyOtA9oL1H+vXsG0d0tG/4Hc9fXx1VYzKmpkRcDYmYGpMxNSZiakzG1ZiMqSERUzwS1mAqrcFURoPDGQ2m0hpKZbK3U9nbzkkR7+fnLiHLDiENeUNJm5JxLZpbrYX11UweAwAAIAIcMOOEQqb5Xq/ZyqXTe67+obT2d/frQPeA9ncP6MCxAR08NqDDvUM63DOoHW/3aPPuIXX1DclNczjoqTQkYlo0p1oL51RpodcDOr++SlWRsMJhUzQUGgmj4dCJ25LUO5RSz0BKxwdTOj4wrJ6BlHoGUzo+kL0MpzPeuYURJeKjr3PnHMYiIWUyUsY5ZZyTc7lt7zrjFAmHxpz3GB6Z5KY6Gh6pB4VzLtsDfaw/+7vrH05rYDijgeG0t53W4HBGA6ns9nA625McMo18KRD2vhQIe/tDZkp7v8eMy/6MTCbvd+myzxGLhBQLhxQNhxSL5K5NsXBY0bApGgkpGgopEjZFwyFFw9lzVqNee4yEsz//VLJfqoRoHwCAghDgAEyoOhZWW3NSbc3Jkx6Xzjgd7RvS4d4hdfYMaiiVUVU0G3rikZDikbB3nd2ORUIyyz4u7ZzS6ex1JnfbGzp66Pig3jrar33e5a2j/drV2aufvt6p3qH0tF5byKRkPKLaqqiiYRs5n7F3KDXtcxMnEvcCQThsI72PkVzwzOuJHG/im1yYzN2XCwwd+4bV/fw+xcKh7EQ4Ycv+jJDJeeHESSOB0yl7Le815oLNiZ5QnQg9ZiO15R83EoZCUthMGSelMplRw3+H0yeG6abSblTgyp0rOur2cFrH+od1bCCl7v5hHe8fVnf/sI4NDL9jSPBMZaaR4BfxQl0kbIpHwmpMxtScjKupNj7qurk2puZkleYkoqqJhhUJh4J+GQCAIiPAAZi2cMi8IZRxndFS69vzLm1KjLvfOadj/SntP9avoVT2XMFc6Ms/ty+VcZKcEvHISFirrcpu18TC487g6ZzTYCoz6lzD3sG0htMZL9Rkh7+G8npzzLtOpd07JrTpH0qNTGTTP5T2gk1ezaOus8FnIG/im9y5j7nJcsYNly8979u/eamdmLk1rLqqqOqqo6qvjuq0hhrVVUVUX31iX21VticzO/Q4+2VAbjs3HDkazgbX3JcDGe/fN+31mqYz2R62sBdCLe/3GDKTeQE245yGUhkNpzMj14Op7O8nt28o7YXWdN52JqMhb18q7eR08vDpnEbOdU1lMiPtY9i7nTsH9nDPkN7s6tOWN46ctMc7Gj4xNLvau1TFwqqOhlRbFVVTMq7mZExNtXE1JXOX7O3a+PgfCfK/WBn7pcvIvrz7cr3R2V5LUzwcVjRiI18yAEFJZ5wGU9kv/3Lv3eHc/33vvb1c5d5nsqclZLzTEtIjpyY4700h/61h7PtE9m/VxH/DQpY9xSIWCY2MQMhdF+v/rnNOxwZSOtA9oLfyR/x092tgOJOtMWQ6dHBQGztfUDh04n077I2mSKXdifdO70vEdCa7LzeiYtQM33kjdXJfmH32imWaV1tVlNdYDAQ4ABXHzFRfE1V9TbQoz537ANzo+7NPTy5c9g+lNex9yP/Zz5/ShStXadgLDMOZjIZT2T9elvtQouwfQJNG7ZOyvXHpjN7xATw95kN75h3bGjk2ZBpZBiPXaxQJnfhDGQ2HRkJWbjhpNlhkPxiU84emcpRKZ9TVO6RDxwfV2TOot48P6mjf8Egv56hhpl5PZ/9QWnu6+vSLN4+oq3do3C8CYpGQwsrIHn9YqcyJHnE/h0eHTNmhqOGQQqH8D5D5QTqvnXpNw2R529615dp0XgAf+XB64rlyPdBOzuuVzu3zPvC67P+PWP4QWG/obCSUGyqb/aknnsvr3X7H8+aG5XpDc0d6vU/8vFHhOhryAnZ45MuJeCQ08v8p1wM+9raTsh/eJ/gwPzicHvkgm/8l0ajbeV8SpDJjt098CM7/tw6Fsr+LkPcPH/LeS/J/F6bszu6j/fqH7U+N/H7GjjSIhEIj5zvnvhzLDWMeO0Tdudx7j7zhzk7p/O2M9wWNy37Bkjv3Ov/fY9D7su9kLL8Neq9j1Osa2c4LPmO+DMqNXsiNUAiPel/0hlrnvTdqzO9y7PnjudeQPkXtxZb7vxuPhJSIR7KXCU49iEVCXoDKTNj2uvuHtb+7X/u7B9Q3ZkSNmdScjKsmFlbG++KtfyCtjuOdI8Pcc21j7L/x2GAWDpnSTtkvTseZJC7X9j95Savk3/fPRVdQgDOzqyX9jaSwpP/tnPuLMffHJX1b0kWSDkv6mHNut3ffH0i6RVJa0u845x7xrXoAmEXGm/hmXk1Iy+adfIgrZpZIOKR5dVWaVze1b4vTGacub7jzyOV49vbON97U0tOWjHwozZ9UaOxw2/H25T68pjIZDaecBvN6MPOvh9KZvA/p+eciatRtaUzY8l5DLlTmAlL+Y0duex/8bJyQIeV6HXL/JvJ6QLNfjPQMprxv9U/skzT6ucZ53tw5j7kP8GPDjnPS28cHNeCF7FzgHkplpvS7PJncUOpcz0N4ZG3REx9sc+de5q9DWhUNjaxDmjsvM5sdTgSr/FA6XojN/X6cJJeRUi4zanh8xuWPPsgo432JlO1dORGixvYOmReQcr0w4dx9udcTMtVVRbLD9qNjhvBH84bw60RYzD//daTd5L0uufx250YC/OigOSZcZrLH5l7jsBcYhvN66vuHs9eSVOXVW1cdHXW6QTya/bIjV3vc6xnLvY7csbHI6HNoTXnbed+P5dc7fkjWyP/RwXT2y8Ch3P/ZVK4H8MRySbnZsPd3D6h3MLtMUt9gSkPpTN5atuO3vWRVVGe01OryM+admO16TpXm11drXm18JNzmtLe3a82aNT7+D6lspwxwZhaWdKekD0naK2mzmW1wzr2Sd9gtko4455aZ2Y2SviLpY2a2QtKNks6RtFDSj8zsDOfc9E5eAQAAUxIOmZpr42qujb/jvvb2g1qzZkUAVc1uuaF9/UNpDaQyyowZVp0/zDqVPtGTN/IhPhoe9WG+HHq2sx+43xdoDcBMVUgP3CpJHc65nZJkZvdJWispP8CtlfRFb/tBSX9n2XeOtZLuc84NStplZh3e8z3lT/kAAACVLRwyb6IizmwBcGqFvFMskrQn7/ZeSasnOsY5lzKzbkmN3v6nxzx20dgfYGbrJK2TpJaWFrW3txdYfun09PSUZV2oTLQn+IW2BD/RnuAX2hL8RHsarSy+6nHOrZe0XpJWrlzpynGMK2Nv4SfaE/xCW4KfaE/wC20JfqI9jVbInKD7JC3Ju73Y2zfuMWYWkVSv7GQmhTwWAAAAAFCAQgLcZknLzazVzGLKTkqyYcwxGyTd7G1fL+lxl50yaoOkG80sbmatkpZLesaf0gEAAABgdjnlEErvnLZbJT2i7DIC33TObTWz2yU965zbIOkuSfd4k5R0KRvy5B33gLITnqQkfZYZKAEAAABgago6B845t1HSxjH7vpC3PSDphgke+2VJX55GjQAAAAAAFTaEEgAAAABQBghwAAAAAFAhCHAAAAAAUCEIcAAAAABQIQhwAAAAAFAhCHAAAAAAUCEIcAAAAABQIQhwAAAAAFAhCHAAAAAAUCHMORd0DaOY2duS3gi6jnE0SeoMugjMGLQn+IW2BD/RnuAX2hL8NBvb07ucc83j3VF2Aa5cmdmzzrmVQdeBmYH2BL/QluAn2hP8QluCn2hPozGEEgAAAAAqBAEOAAAAACoEAa5w64MuADMK7Ql+oS3BT7Qn+IW2BD/RnvJwDhwAAAAAVAh64AAAAACgQhDgAAAAAKBCEOAKYGZXm9l2M+sws9uCrgflw8x2m9lLZva8mT3r7Wsws0fN7HXveq6338zs6147etHMLsx7npu94183s5vz9l/kPX+H91gr/atEsZjZN83skJm9nLev6O1nop+ByjVBW/qime3z3p+eN7Nr8+77A69dbDezX87bP+7fOzNrNbNN3v77zSzm7Y97tzu8+5eW6CWjSMxsiZk9YWavmNlWM/tdbz/vTZi0k7Qn3p+mwznH5SQXSWFJOyS1SYpJekHSiqDr4lIeF0m7JTWN2fdVSbd527dJ+oq3fa2kH0oySRdL2uTtb5C007ue623P9e57xjvWvMdeE/Rr5uJr+7lM0oWSXi5l+5noZ3Cp3MsEbemLkn5vnGNXeH/L4pJavb9x4ZP9vZP0gKQbve1/lPRpb/szkv7R275R0v1B/1twmXZbWiDpQm+7VtJrXpvhvYmLn+2J96dpXOiBO7VVkjqcczudc0OS7pO0NuCaUN7WSrrb275b0kfy9n/bZT0taY6ZLZD0y5Iedc51OeeOSHpU0tXefXXOuadd9t3n23nPhRnAOfdjSV1jdpei/Uz0M1ChJmhLE1kr6T7n3KBzbpekDmX/1o37987rHblC0oPe48e2y1xbelDSlYwUqGzOuf3Ouee87eOStklaJN6bMAUnaU8T4f2pAAS4U1skaU/e7b06ecPD7OIk/YeZbTGzdd6+Fufcfm/7gKQWb3uitnSy/XvH2Y+ZrRTtZ6KfgZnnVm9Y2zfzhqNNti01SjrqnEuN2T/qubz7u73jMQN4Q87eI2mTeG/CNI1pTxLvT1NGgAOm51Ln3IWSrpH0WTO7LP9O79tF1urAlJSi/dBGZ7R/kHS6pAsk7Zf0V4FWg4piZklJ35P0Oefcsfz7eG/CZI3Tnnh/mgYC3Kntk7Qk7/Zibx8g59w+7/qQpO8r28V/0BsiIu/6kHf4RG3pZPsXj7MfM1sp2s9EPwMziHPuoHMu7ZzLSPqGsu9P0uTb0mFlh8VFxuwf9Vze/fXe8ahgZhZV9sP2d5xz/+rt5r0JUzJee+L9aXoIcKe2WdJyb4abmLInQW4IuCaUATNLmFltblvSVZJeVrZ95GbbulnSD7ztDZJu8mbsulhStzdU5BFJV5nZXG8IwVWSHvHuO2ZmF3tjtm/Key7MXKVoPxP9DMwguQ/Cno8q+/4kZX//N3oztLVKWq7spBLj/r3zekKekHS99/ix7TLXlq6X9Lh3PCqU935xl6Rtzrmv5d3FexMmbaL2xPvTNAU9i0olXJSdYek1ZWe/+aOg6+FSHhdlZ0J6wbtszbUNZcdXPybpdUk/ktTg7TdJd3rt6CVJK/Oe6zeUPVG3Q9In8/avVPZNbYekv5NkQb9uLr62oX9RdujIsLLj9m8pRfuZ6GdwqdzLBG3pHq+tvKjsB5kFecf/kdcutitvdtuJ/t5573fPeG3su5Li3v4q73aHd39b0P8WXKbdli5Vdujii5Ke9y7X8t7Exef2xPvTNC65/zAAAAAAgDLHEEoAAAAAqBAEOAAAAACoEAQ4AAAAAKgQBDgAAAAAqBAEOAAAAACoEAQ4AAAAAKgQBDgAAAAAqBD/F2JIhXWqRCcmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "total_logs = len(ner_objective.evaluations_history[\"eval\"][\"loss\"])\n",
    "training_steps = ner_objective.num_steps\n",
    "\n",
    "index = range(0, training_steps, training_steps // total_logs)\n",
    "\n",
    "pd.Series(ner_objective.evaluations_history[\"eval\"][\"loss\"], index=index).plot(figsize=(15, 7), grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the trained model\n",
    "\n",
    "After the training finishes, we persist the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5. save the trained lang_module (with all heads)\n",
    "adapter.save_model(\"entity_detector_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls entity_detector_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is slightly different with Adaptor's `Adapter` as with 🤗  Transformers: instead of a single model, the `save_model` will produce a separate model for each `Objective`. Even though the most parameters of the trained multi-head model are shared, thanks to this, you can entirely get rid of the dependency on Adaptor, as each of the models can be reloaded using Transformers' `AutoModelForXY.from_pretrained(\"save_model_path/objective_name\")`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "eeCHo7AVccz4"
   },
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "\n",
    "ner_model = AutoModelForTokenClassification.from_pretrained(\"entity_detector_model/TokenClassification\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"entity_detector_model/TokenClassification\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Objectives also take care of the correct configuration of the persisted model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'B-LOC',\n",
       " 1: 'B-MISC',\n",
       " 2: 'B-ORG',\n",
       " 3: 'B-PER',\n",
       " 4: 'I-LOC',\n",
       " 5: 'I-MISC',\n",
       " 6: 'I-ORG',\n",
       " 7: 'I-PER',\n",
       " 8: 'O'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner_model.config.id2label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model usage\n",
    "\n",
    "The reloaded model can be used rightaway as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tokenizer(\"Is there any Abraham Lincoln here?\", return_tensors=\"pt\")\n",
    "outputs = ner_model(**inputs)\n",
    "ner_tags = [ner_model.config.id2label[label_id.item()] for label_id in outputs.logits[0].argmax(-1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "To obtain a representative evaluation of our freshly-trained model, we'll evaluate it in a held-out (test) set of CoNLL.\n",
    "\n",
    "To avoid a boilerplate implementation of Wordpiece-to-labels alignment, we'll also use the TokenClassification objective here, but if you prefer to completely avoid Adaptor in evaluation, take a look at [how it's done](https://github.com/gaussalgo/adaptor/blob/db33e6e439babc68fe801a8946d87116ff44f170/adaptor/objectives/classification.py#L14) in TokenClassfication and implement it in a standalone.\n",
    "\n",
    "Additionally, we'll see that defining a custom `Evaluator` in Adaptor is usually fairly simple. Here, we define a custom `AverageAccuracy` measure and report the results of our TokenClassification Objective on this new measure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TJ1GNFVBez0O",
    "outputId": "ea271f0f-34cb-4d9f-f981-6b9579e69be5"
   },
   "outputs": [],
   "source": [
    "conll_test_texts = [\" \".join(sample_tokens) for sample_tokens in conll_dataset_nonempty[\"test\"][\"tokens\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e59CsQ8HhCpj",
    "outputId": "b3b8f659-48d9-413f-96fb-823ef415b14d"
   },
   "outputs": [],
   "source": [
    "# we load the data again from CoNLL, so the mapping is different than in our Objective -> we transform them to universal identifier format\n",
    "\n",
    "label_map = {0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
    "\n",
    "conll_test_labels = [\" \".join(label_map[tag_id] for tag_id in sample) for sample in conll_dataset_nonempty[\"test\"][\"ner_tags\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Evaluators\n",
    "\n",
    "Now we define a custom evaluation measure that we'll report on. There is a good bunch of different approaches on how to aggregate token-level predictions. Our `AverageAccuracy` implements a global token-level accuracy.\n",
    "\n",
    "When implementing a new Evaluator in Adaptor, it's a good idea to take a look at existing evaluator of the same category and its superclass, and utilise the methods of the highest level that are still useful for you.\n",
    "\n",
    "In this case, we find that `TokenClassificationEvaluator` superclass has correct `compatible_heads` and a convenient implementation of `_collect_token_predictions` method. All that's left to do is to produce the desired statistics on the flattened token-level predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptor.evaluators.token_classification import TokenClassificationEvaluator\n",
    "\n",
    "class AverageAccuracy(TokenClassificationEvaluator):\n",
    "\n",
    "    def __call__(self, \n",
    "                 model: \"torch.nn.Module\",\n",
    "                 tokenizer: \"PreTrainedTokenizer\",\n",
    "                 dataset: \"AdaptationDataset\",\n",
    "                 ignored_index: int = -100) -> float:\n",
    "\n",
    "        expected, actual = self._collect_token_predictions(model, dataset)\n",
    "\n",
    "        num_matching = sum(e_i == a_i for e_i, a_i in zip(expected, actual) if e_i != ignored_index)\n",
    "        num_all = sum(e_i != ignored_index for e_i in expected)\n",
    "\n",
    "        return num_matching / num_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a custom Evaluator, we initialise the `TokenClassification` objective identically as when training, we just replace the `val_texts_or_path` and `val_labels_or_path` with the test resources and add an instance of `AverageAccuracy` to a list of evaluators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptor.lang_module import LangModule\n",
    "from adaptor.objectives.classification import TokenClassification\n",
    "\n",
    "ner_evaluators = [AverageAccuracy()]\n",
    "\n",
    "lang_module = LangModule(\"entity_detector_model/TokenClassification\")\n",
    "\n",
    "\n",
    "test_ner_objective = TokenClassification(lang_module,\n",
    "                                         batch_size=8,\n",
    "                                         texts_or_path=[],\n",
    "                                         labels_or_path=[],\n",
    "                                         val_texts_or_path=conll_test_texts[:1000],\n",
    "                                         val_labels_or_path=conll_test_labels[:1000],\n",
    "                                         val_evaluators=ner_evaluators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `per_objective_log` for evaluating on a single objective. During the training, this method is called by `Schedule`, but can be conveniently used also for computing test evaluation. Feel free to take a look at its [documentation](https://github.com/gaussalgo/adaptor/blob/db33e6e439babc68fe801a8946d87116ff44f170/adaptor/objectives/objective_base.py#L135) in base Objective implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  34%|███▎      | 42/125 [00:07<00:15,  5.43batches/s, epoch=0, loss=-1, split=eval]\n",
      "TokenClassification: 100%|██████████| 125/125 [00:18<00:00,  5.36batches/s, epoch=0, loss=-1, split=eval]"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'eval_TokenClassification_loss': 0,\n",
       " 'eval_TokenClassification_num_batches': 0,\n",
       " 'eval_TokenClassification_AverageAccuracy': 0.9574836867862969}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluations = test_ner_objective.per_objective_log(\"eval\")\n",
    "\n",
    "evaluations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9574836867862969"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluations[\"eval_TokenClassification_AverageAccuracy\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that means that our model correctly assigned appx. 95% of tokens into one of 9 categories. Not bad."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaptation\n",
    "\n",
    "Finally, we'll take a look at how can we **improve** a quality of NER model with **adaptation**, using Adaptor.\n",
    "\n",
    "In addition to the final task, we'll first adapt the model using unsupervised `MaskedLanguageModeling`. This objective is commonly used approach to pre-train the encoders like BERT on a vast amounts of unsupervised texts and makes them able to fit the supervised tasks **faster** afterwards. Continuing in such pre-training is also the most basic adaptation scenario.\n",
    "\n",
    "We'll show how to do this using Adaptor here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l21Uz52JdnIE",
    "outputId": "0867135a-9fbe-44eb-eb89-4602fef7a634"
   },
   "outputs": [],
   "source": [
    "from adaptor.lang_module import LangModule\n",
    "\n",
    "lang_module = LangModule(\"bert-base-cased\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We proceed identically as in the former case, but we add a `MaskedLanguageModeling` objective, fitted on the same training data into the `SequentialSchedule`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oPpW5SkDeexE",
    "outputId": "6227d62c-dc88-4dd1-f4ec-c126c9b19400"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "from adaptor.objectives.MLM import MaskedLanguageModeling\n",
    "\n",
    "mlm_adaptation = MaskedLanguageModeling(lang_module,\n",
    "                                        batch_size=32,\n",
    "                                        texts_or_path=conll_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though `TokenClassification` objective is identical to the previous case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForTokenClassification: ['cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForTokenClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "These layers of the loaded TOKEN_CLASSIFICATION were not merged: ['weight', 'bias']\n"
     ]
    }
   ],
   "source": [
    "from adaptor.objectives.classification import TokenClassification\n",
    "from adaptor.evaluators.token_classification import MeanFScore\n",
    "\n",
    "ner_evaluators = [MeanFScore(decides_convergence=True)]\n",
    "\n",
    "ner_objective = TokenClassification(lang_module,\n",
    "                                    batch_size=8,\n",
    "                                    texts_or_path=conll_texts,\n",
    "                                    labels_or_path=conll_labels,\n",
    "                                    val_texts_or_path=conll_val_texts[:1000],\n",
    "                                    val_labels_or_path=conll_val_labels[:1000],\n",
    "                                    val_evaluators=ner_evaluators)\n",
    "\n",
    "training_objectives = [mlm_adaptation, ner_objective]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gA5opjLokI5A",
    "outputId": "1830835e-46f4-42d5-c9b6-99bc7352636e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "comet_ml is installed but `COMET_API_KEY` is not set.\n"
     ]
    }
   ],
   "source": [
    "from adaptor.utils import StoppingStrategy, AdaptationArguments\n",
    "\n",
    "training_arguments = AdaptationArguments(output_dir=\"ner_training_checkpoints\",\n",
    "                                         stopping_strategy=StoppingStrategy.FIRST_OBJECTIVE_CONVERGED,\n",
    "                                         do_train=True,\n",
    "                                         do_eval=True,\n",
    "                                         gradient_accumulation_steps=4,\n",
    "                                         evaluation_strategy=\"steps\",\n",
    "                                         # log_level=\"critical\",\n",
    "                                         logging_steps=200,\n",
    "                                         eval_steps=500,\n",
    "                                         num_train_epochs=10,\n",
    "                                         save_steps=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Total number of train samples: 28082\n",
      "Total number of eval samples: 1000\n"
     ]
    }
   ],
   "source": [
    "from adaptor.schedules import SequentialSchedule\n",
    "\n",
    "schedule = SequentialSchedule(objectives=training_objectives, args=training_arguments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adaptor.adapter import Adapter\n",
    "\n",
    "adapter = Adapter(lang_module, schedule, training_arguments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adapter.train()  # the training is quite long here, so we cleaned the outputs logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tokenizer config file saved in mlm_adapted_ner_model/MaskedLanguageModeling/tokenizer_config.json\n",
      "Special tokens file saved in mlm_adapted_ner_model/MaskedLanguageModeling/special_tokens_map.json\n",
      "Configuration saved in mlm_adapted_ner_model/MaskedLanguageModeling/config.json\n",
      "Model weights saved in mlm_adapted_ner_model/MaskedLanguageModeling/pytorch_model.bin\n",
      "tokenizer config file saved in mlm_adapted_ner_model/TokenClassification/tokenizer_config.json\n",
      "Special tokens file saved in mlm_adapted_ner_model/TokenClassification/special_tokens_map.json\n",
      "Configuration saved in mlm_adapted_ner_model/TokenClassification/config.json\n",
      "Model weights saved in mlm_adapted_ner_model/TokenClassification/pytorch_model.bin\n"
     ]
    }
   ],
   "source": [
    "adapter.save_model(\"mlm_adapted_ner_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a main evaluation metric, we again use our newly-implemented `AverageAccuracy`, and proceed identically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Didn't find file mlm_adapted_ner_model/TokenClassification/added_tokens.json. We won't load it.\n",
      "loading file mlm_adapted_ner_model/TokenClassification/vocab.txt\n",
      "loading file mlm_adapted_ner_model/TokenClassification/tokenizer.json\n",
      "loading file None\n",
      "loading file mlm_adapted_ner_model/TokenClassification/special_tokens_map.json\n",
      "loading file mlm_adapted_ner_model/TokenClassification/tokenizer_config.json\n",
      "Objective TokenClassification will use TOKEN_CLASSIFICATION head of TokenClassification objective\n"
     ]
    }
   ],
   "source": [
    "ner_evaluators = [AverageAccuracy()]\n",
    "\n",
    "adapted_lang_module = LangModule(\"mlm_adapted_ner_model/TokenClassification\")\n",
    "\n",
    "test_ner_objective = TokenClassification(adapted_lang_module,\n",
    "                                         batch_size=8,\n",
    "                                         texts_or_path=[],\n",
    "                                         labels_or_path=[],\n",
    "                                         val_texts_or_path=conll_test_texts[:1000],\n",
    "                                         val_labels_or_path=conll_test_labels[:1000],\n",
    "                                         share_other_objective_head=ner_objective,\n",
    "                                         val_evaluators=ner_evaluators)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification:  98%|█████████▊| 123/125 [00:03<00:00, 41.30batches/s, epoch=0, loss=-1, split=eval]"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9633462479608483"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TokenClassification: 100%|██████████| 125/125 [00:14<00:00, 41.30batches/s, epoch=0, loss=-1, split=eval]"
     ]
    }
   ],
   "source": [
    "evaluations = test_ner_objective.per_objective_log(\"eval\")\n",
    "evaluations[\"eval_TokenClassification_AverageAccuracy\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we got a gain of **0.6%** of accuracy!\n",
    "\n",
    "It's not much, but you can also say, that thanks to adaptation, we've **eliminated more than 10% of the remaining error** of the model for free, without any additional data. Sounds like a free lunch, right?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****************\n",
    "In this tutorial, we have shown the benefits of objective-centric approach to training language models on a common task of Named Entity Recognition. You could have seen that transferring from conventional single-objective training to multi-objective adaptation is a three-liner with the initialisation of one extra objective.\n",
    "\n",
    "In the more interesting scenarios, you might want to try to use **external datasets** and **auxiliary tasks** from other **similar domain**. In the low-resource scenarios, it might also be interesting to try to train multi-head model on **complementary supervised data**."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "adaptor_example_NER.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "014ff61d5011457a96ff4d9e1bf3f1b4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2950d13507d34371984f224c1fb4519e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "311ac0b4f7124968a10da068c6b080c7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_41660ae55a3a4a24939e3a9c5921405d",
       "IPY_MODEL_60e20867027a4aad9ee017dd87aab365",
       "IPY_MODEL_5df842530194491bafa52bf7c7fd3661"
      ],
      "layout": "IPY_MODEL_2950d13507d34371984f224c1fb4519e"
     }
    },
    "41660ae55a3a4a24939e3a9c5921405d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8146bb6de82d409c87241b973d63e3d9",
      "placeholder": "​",
      "style": "IPY_MODEL_5712a7090a194bdca94d864d62b55ab3",
      "value": "100%"
     }
    },
    "5712a7090a194bdca94d864d62b55ab3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5df842530194491bafa52bf7c7fd3661": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_614690118e484425b438f1580dc72ac2",
      "placeholder": "​",
      "style": "IPY_MODEL_f30b0650f0884912bbe71aa25573789c",
      "value": " 3/3 [00:00&lt;00:00, 43.07it/s]"
     }
    },
    "60e20867027a4aad9ee017dd87aab365": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a50ccd63ffe84d268dd02d98ddd8fbae",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_014ff61d5011457a96ff4d9e1bf3f1b4",
      "value": 3
     }
    },
    "614690118e484425b438f1580dc72ac2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8146bb6de82d409c87241b973d63e3d9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a50ccd63ffe84d268dd02d98ddd8fbae": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f30b0650f0884912bbe71aa25573789c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}