tutorials/simple_sequence_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c9f30c1a-8962-4236-b7aa-3718d08548da",
   "metadata": {
    "id": "c9f30c1a-8962-4236-b7aa-3718d08548da"
   },
   "source": [
    "# Simple Sequence Classification using Adaptor\n",
    "\n",
    "This notebook demonstrates a simple, single-objective training pipelines using Adaptor. You will see how easy it is to exchange one objective for another, when, e.g. you want to experiment with both (i) traditional and (ii) generative models.\n",
    "\n",
    "This notebook contains the following parts:\n",
    "\n",
    "1. Training of Sequence Classification model\n",
    "2. Training of Generative model\n",
    "3. Evaluation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42d0299e-e0b4-4e36-8b1d-2322b68cab54",
   "metadata": {
    "id": "42d0299e-e0b4-4e36-8b1d-2322b68cab54"
   },
   "outputs": [],
   "source": [
    "!pip install matplotlib datasets adaptor==0.2.1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b487bae2-bbf2-44c4-9ea6-70126e319792",
   "metadata": {
    "tags": [],
    "id": "b487bae2-bbf2-44c4-9ea6-70126e319792"
   },
   "source": [
    "## 1. Training of Sequence Classification model\n",
    "\n",
    "Remember, Transformer-based language models map Sequence classification to a token classification of `[CLS]` or `<s>`.\n",
    "\n",
    "![image.jpg](https://www.researchgate.net/profile/Juan-Pablo-Usuga-Cadavid/publication/353419108/figure/fig1/AS:1052388236476416@1627920316521/Example-of-a-trained-BERT-for-text-classification_W640.jpg)\n",
    "\n",
    "[[image source]](https://www.researchgate.net/publication/353419108_Using_deep_learning_to_value_free-form_text_data_for_predictive_maintenance)\n",
    "\n",
    "1. In this settings, when we **train** or update the model to maximise the chance of predicting correct catePrediction using 'gaussalgo/xlm-roberta-base_CSFD-sk' classifiergory on this single token.\n",
    "\n",
    "2. During the **inference** (i.e. the actual usage of the model), we look only at model's prediction at this single token. This is also what `transformers.AnyModelForSequenceClassification` does internally."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab407ee-eedc-4843-9d58-a372ce76b86a",
   "metadata": {
    "id": "6ab407ee-eedc-4843-9d58-a372ce76b86a"
   },
   "source": [
    "## Construction of Training pipeline\n",
    "\n",
    "Below, you will see how to configure training pipeline in Adaptor. In the single-objective training scenarios like this, consider also using vanilla Transformers, such as in the [Sequence Classification tutorial](https://huggingface.co/docs/transformers/tasks/sequence_classification).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f45c9f88-bb44-4e74-9828-e6acebbfc8ba",
   "metadata": {
    "id": "f45c9f88-bb44-4e74-9828-e6acebbfc8ba"
   },
   "source": [
    "#### 2. Training with Adaptor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29b4c188-de9a-49fc-8e76-1d8455e9eb3f",
   "metadata": {
    "id": "29b4c188-de9a-49fc-8e76-1d8455e9eb3f"
   },
   "source": [
    "First we pick the base model. At this point we do not care about the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed1eb187-19dd-4115-ba05-89288f53726a",
   "metadata": {
    "id": "ed1eb187-19dd-4115-ba05-89288f53726a"
   },
   "outputs": [],
   "source": [
    "from adaptor.lang_module import LangModule\n",
    "\n",
    "language_module = LangModule(\"xlm-roberta-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524193ff-3067-440f-8fd9-2e9e1b4b93f8",
   "metadata": {
    "id": "524193ff-3067-440f-8fd9-2e9e1b4b93f8"
   },
   "source": [
    "Second, we choose the objective that we want to fine-tune the model for. The objective will take care of configuring the model correctly. We just give it our desired inputs and outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf15ea36-655a-4fa3-b164-980379f786b3",
   "metadata": {
    "id": "bf15ea36-655a-4fa3-b164-980379f786b3",
    "outputId": "aca2910c-e708-4c10-da93-9d0de650c00e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at xlm-roberta-base were not used when initializing XLMRobertaForSequenceClassification: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'roberta.pooler.dense.bias', 'roberta.pooler.dense.weight', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight']\n",
      "- This IS expected if you are initializing XLMRobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing XLMRobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of XLMRobertaForSequenceClassification were not initialized from the model checkpoint at xlm-roberta-base and are newly initialized: ['classifier.out_proj.bias', 'classifier.dense.bias', 'classifier.dense.weight', 'classifier.out_proj.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from adaptor.objectives.classification import SequenceClassification\n",
    "from adaptor.evaluators.sequence_classification import SequenceAccuracy\n",
    "\n",
    "model_quality_evaluator = SequenceAccuracy(decides_convergence=True)\n",
    "\n",
    "training_objective = SequenceClassification(lang_module=language_module,\n",
    "                                            texts_or_path=dataset[\"train\"]['comment'],\n",
    "                                            labels_or_path=dataset[\"train\"][\"rating_int\"],\n",
    "                                            val_texts_or_path=dataset[\"validation\"]['comment'],\n",
    "                                            val_labels_or_path=dataset[\"validation\"][\"rating_int\"],\n",
    "                                            val_evaluators=[model_quality_evaluator],\n",
    "                                            batch_size=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc718cca-6ae1-4bed-a846-57ffe43780bf",
   "metadata": {
    "id": "dc718cca-6ae1-4bed-a846-57ffe43780bf"
   },
   "source": [
    "The training process is configured through a possibly large set of arguments. Adaptor works with all arguments of HuggingFace's [TrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32bae738-67fb-465d-81c6-a9efae3e138d",
   "metadata": {
    "id": "32bae738-67fb-465d-81c6-a9efae3e138d"
   },
   "outputs": [],
   "source": [
    "from adaptor.utils import AdaptationArguments, StoppingStrategy\n",
    "\n",
    "args = AdaptationArguments(output_dir=\"./output_dir\",\n",
    "                           learning_rate=2e-5,\n",
    "                           warmup_steps=1000,\n",
    "                           stopping_strategy=StoppingStrategy.FIRST_OBJECTIVE_CONVERGED,\n",
    "                           do_train=True,\n",
    "                           do_eval=True,\n",
    "                           log_level=\"critical\",\n",
    "                           logging_steps=100,\n",
    "                           eval_steps=200,\n",
    "                           evaluation_strategy=\"steps\",\n",
    "                           save_steps=200,\n",
    "                           save_total_limit=6,\n",
    "                           stopping_patience=5,\n",
    "                           num_train_epochs=20,\n",
    "                           gradient_accumulation_steps=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f784884b-576e-403c-b6c3-6cdb35b3ebc1",
   "metadata": {
    "id": "f784884b-576e-403c-b6c3-6cdb35b3ebc1"
   },
   "source": [
    "Adaptor allows training for multiple objectives at once, that are applied in a chosen Schedule. Multi-objective training is useful for more complex scenarios, but for our case, we suffice with a single objective, so a selection of Schedule does not really matter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76343b11-b1e4-47b7-a647-fc4d99264aac",
   "metadata": {
    "id": "76343b11-b1e4-47b7-a647-fc4d99264aac",
    "outputId": "f35f424c-0854-45d6-d971-10cee8f175cd"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Total number of train samples: 25000\n",
      "Total number of eval samples: 2500\n"
     ]
    }
   ],
   "source": [
    "from adaptor.schedules import SequentialSchedule\n",
    "from adaptor.adapter import Adapter\n",
    "\n",
    "parallel_schedule = SequentialSchedule(objectives=[training_objective], args=args)\n",
    "\n",
    "adapter = Adapter(lang_module=language_module,\n",
    "                  schedule=parallel_schedule,\n",
    "                  args=args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91dc5c37-8ecf-491f-9001-e8b9867198c1",
   "metadata": {
    "id": "91dc5c37-8ecf-491f-9001-e8b9867198c1"
   },
   "source": [
    "After all the configuration, we are ready to run the training and wait for the trained model.\n",
    "\n",
    "Given the `stopping_strategy=StoppingStrategy.FIRST_OBJECTIVE_CONVERGED` and `stopping_patience=1`, the training will terminate after first evaluation, where `model_quality_evaluator` (i.e. `SequenceAccuracy`) does not improve over one evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b409841-c3d5-47fe-95bd-cba617bc8219",
   "metadata": {
    "scrolled": true,
    "tags": [],
    "id": "6b409841-c3d5-47fe-95bd-cba617bc8219",
    "outputId": "57920d34-abd7-4c24-c5e1-2b2ccbbe9cd7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/.local/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "Converged objectives: []\n",
      "SequenceClassification:  13%|█▎        | 800/6250 [01:15<09:04, 10.01batches/s, epoch=1, loss=1.64, split=train]Converged objectives: []\n",
      "SequenceClassification:  13%|█▎        | 802/6250 [01:15<09:14,  9.83batches/s, epoch=1, loss=1.79, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.8057, 'learning_rate': 2.0000000000000003e-06, 'train_SequenceClassification_loss': 1.8056869113445282, 'train_SequenceClassification_num_batches': 800, 'epoch': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  26%|██▌       | 1600/6250 [02:41<07:26, 10.41batches/s, epoch=1, loss=1.89, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.7941, 'learning_rate': 4.000000000000001e-06, 'train_SequenceClassification_loss': 1.7950378651618957, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [00:20<00:00, 30.71batches/s, epoch=1, loss=1.87, split=eval]in]\n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:18<00:00, 35.98batches/s, epoch=1, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.7883870601654053, 'eval_runtime': 20.3519, 'eval_samples_per_second': 30.71, 'eval_steps_per_second': 30.71, 'eval_SequenceClassification_loss': 1.7883872732162476, 'eval_SequenceClassification_num_batches': 625, 'eval_SequenceClassification_SequenceAccuracy': 0.1964, 'epoch': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  38%|███▊      | 2400/6250 [04:55<06:09, 10.41batches/s, epoch=1, loss=1.84, split=train]   Converged objectives: []\n",
      "SequenceClassification:  38%|███▊      | 2402/6250 [04:55<06:49,  9.40batches/s, epoch=1, loss=1.71, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.771, 'learning_rate': 6e-06, 'train_SequenceClassification_loss': 1.7740453741550446, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.02}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  51%|█████     | 3199/6250 [06:21<04:57, 10.25batches/s, epoch=1, loss=1.39, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.6261, 'learning_rate': 8.000000000000001e-06, 'train_SequenceClassification_loss': 1.649385324239731, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.03}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:19<00:00,  3.13batches/s, epoch=1, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:20<00:00, 30.99batches/s, epoch=1, loss=1.65, split=eval]in]\n",
      "SequenceClassification: 100%|█████████▉| 622/625 [00:18<00:00, 35.54batches/s, epoch=1, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.5000486373901367, 'eval_runtime': 20.167, 'eval_samples_per_second': 30.991, 'eval_steps_per_second': 30.991, 'eval_SequenceClassification_loss': 1.6442179903507232, 'eval_SequenceClassification_num_batches': 1250, 'eval_SequenceClassification_SequenceAccuracy': 0.356, 'epoch': 0.03}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  64%|██████▍   | 3999/6250 [08:36<03:30, 10.67batches/s, epoch=1, loss=1.79, split=train]  Converged objectives: []\n",
      "SequenceClassification:  64%|██████▍   | 4001/6250 [08:36<04:08,  9.04batches/s, epoch=1, loss=1.44, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.4952, 'learning_rate': 1e-05, 'train_SequenceClassification_loss': 1.5068887263536452, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.03}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  77%|███████▋  | 4800/6250 [09:56<01:26, 16.79batches/s, epoch=1, loss=1.82, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.4331, 'learning_rate': 1.2e-05, 'train_SequenceClassification_loss': 1.4371108202934264, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.04}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:15<00:00,  3.20batches/s, epoch=1, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:10<00:00, 59.23batches/s, epoch=1, loss=1.11, split=eval] \n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:19<00:00, 36.85batches/s, epoch=1, loss=-1, split=eval]rain]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.3178272247314453, 'eval_runtime': 10.5531, 'eval_samples_per_second': 59.224, 'eval_steps_per_second': 59.224, 'eval_SequenceClassification_loss': 1.5354210516452789, 'eval_SequenceClassification_num_batches': 1875, 'eval_SequenceClassification_SequenceAccuracy': 0.4364, 'epoch': 0.04}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  90%|████████▉ | 5599/6250 [12:01<01:09,  9.34batches/s, epoch=1, loss=1.21, split=train]  Converged objectives: []\n",
      "SequenceClassification:  90%|████████▉ | 5603/6250 [12:01<01:05,  9.83batches/s, epoch=1, loss=1.04, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.3826, 'learning_rate': 1.4e-05, 'train_SequenceClassification_loss': 1.3834207997322083, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.04}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 6250/6250 [13:10<00:00,  9.68batches/s, epoch=1, loss=1.71, split=train] Converged objectives: []\n",
      "SequenceClassification: 100%|██████████| 6250/6250 [13:10<00:00,  7.90batches/s, epoch=1, loss=1.08, split=train]\n",
      "SequenceClassification:   2%|▏         | 149/6250 [00:15<09:12, 11.04batches/s, epoch=2, loss=0.803, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.3303, 'learning_rate': 1.6000000000000003e-05, 'train_SequenceClassification_loss': 1.3414764949083329, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.05}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:19<00:00,  3.14batches/s, epoch=1, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:20<00:00, 30.77batches/s, epoch=2, loss=1.22, split=eval] ] \n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:18<00:00, 36.04batches/s, epoch=2, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.2793632745742798, 'eval_runtime': 20.3154, 'eval_samples_per_second': 30.765, 'eval_steps_per_second': 30.765, 'eval_SequenceClassification_loss': 1.471406610417366, 'eval_SequenceClassification_num_batches': 2500, 'eval_SequenceClassification_SequenceAccuracy': 0.4548, 'epoch': 0.05}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  15%|█▌        | 950/6250 [02:27<08:30, 10.39batches/s, epoch=2, loss=2.01, split=train]   Converged objectives: []\n",
      "SequenceClassification:  15%|█▌        | 952/6250 [02:27<08:23, 10.52batches/s, epoch=2, loss=1.24, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.3165, 'learning_rate': 1.8e-05, 'train_SequenceClassification_loss': 1.3210068408548832, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  28%|██▊       | 1749/6250 [03:55<07:10, 10.46batches/s, epoch=2, loss=0.829, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.2726, 'learning_rate': 2e-05, 'train_SequenceClassification_loss': 1.277403223991394, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:20<00:00,  3.12batches/s, epoch=2, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:19<00:00, 31.30batches/s, epoch=2, loss=1.65, split=eval] ]  \n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:19<00:00, 37.23batches/s, epoch=2, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.256230115890503, 'eval_runtime': 19.9682, 'eval_samples_per_second': 31.3, 'eval_steps_per_second': 31.3, 'eval_SequenceClassification_loss': 1.428371317024231, 'eval_SequenceClassification_num_batches': 3125, 'eval_SequenceClassification_SequenceAccuracy': 0.444, 'epoch': 0.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  41%|████      | 2550/6250 [05:33<03:32, 17.38batches/s, epoch=2, loss=1.76, split=train]  Converged objectives: []\n",
      "SequenceClassification:  41%|████      | 2552/6250 [05:33<04:01, 15.31batches/s, epoch=2, loss=1.76, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.2568, 'learning_rate': 1.9993579454253612e-05, 'train_SequenceClassification_loss': 1.2581012021899223, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.07}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  54%|█████▎    | 3350/6250 [06:24<02:48, 17.22batches/s, epoch=2, loss=0.779, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.2173, 'learning_rate': 1.9987158908507226e-05, 'train_SequenceClassification_loss': 1.2179782185554504, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.08}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [02:08<00:00,  4.86batches/s, epoch=2, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:10<00:00, 59.08batches/s, epoch=2, loss=1.05, split=eval] \n",
      "SequenceClassification:  99%|█████████▉| 618/625 [00:09<00:00, 69.43batches/s, epoch=2, loss=-1, split=eval]rain] Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.1869587898254395, 'eval_runtime': 10.5805, 'eval_samples_per_second': 59.071, 'eval_steps_per_second': 59.071, 'eval_SequenceClassification_loss': 1.3881358908136685, 'eval_SequenceClassification_num_batches': 3750, 'eval_SequenceClassification_SequenceAccuracy': 0.5168, 'epoch': 0.08}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  66%|██████▋   | 4149/6250 [07:44<01:52, 18.69batches/s, epoch=2, loss=1.18, split=train]   Converged objectives: []\n",
      "SequenceClassification:  66%|██████▋   | 4154/6250 [07:44<01:49, 19.13batches/s, epoch=2, loss=1.21, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.1949, 'learning_rate': 1.9980738362760837e-05, 'train_SequenceClassification_loss': 1.1925911034345626, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.08}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  79%|███████▉  | 4948/6250 [08:35<01:25, 15.27batches/s, epoch=2, loss=0.822, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.1322, 'learning_rate': 1.9974317817014448e-05, 'train_SequenceClassification_loss': 1.128819655776024, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.09}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [02:00<00:00,  5.21batches/s, epoch=2, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:10<00:00, 59.13batches/s, epoch=2, loss=0.909, split=eval]\n",
      "SequenceClassification:  99%|█████████▉| 618/625 [00:09<00:00, 69.44batches/s, epoch=2, loss=-1, split=eval]rain] Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.1504303216934204, 'eval_runtime': 10.5734, 'eval_samples_per_second': 59.111, 'eval_steps_per_second': 59.111, 'eval_SequenceClassification_loss': 1.3541779587779725, 'eval_SequenceClassification_num_batches': 4375, 'eval_SequenceClassification_SequenceAccuracy': 0.5268, 'epoch': 0.09}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  92%|█████████▏| 5750/6250 [09:53<00:30, 16.55batches/s, epoch=2, loss=0.376, split=train] Converged objectives: []\n",
      "SequenceClassification:  92%|█████████▏| 5752/6250 [09:53<00:35, 14.06batches/s, epoch=2, loss=0.567, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.1184, 'learning_rate': 1.996789727126806e-05, 'train_SequenceClassification_loss': 1.1163887090831994, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|█████████▉| 6249/6250 [10:23<00:00, 14.84batches/s, epoch=2, loss=0.687, split=train]Converged objectives: []\n",
      "SequenceClassification: 100%|██████████| 6250/6250 [10:23<00:00, 10.02batches/s, epoch=2, loss=0.627, split=train]\n",
      "SequenceClassification:   5%|▍         | 299/6250 [00:18<06:58, 14.21batches/s, epoch=3, loss=0.815, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0957, 'learning_rate': 1.9961476725521673e-05, 'train_SequenceClassification_loss': 1.0892368990331889, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [01:56<00:00,  5.37batches/s, epoch=2, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:10<00:00, 59.20batches/s, epoch=3, loss=0.877, split=eval]\n",
      "SequenceClassification:  99%|█████████▉| 618/625 [00:09<00:00, 69.18batches/s, epoch=3, loss=-1, split=eval]ain] Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0905945301055908, 'eval_runtime': 10.5584, 'eval_samples_per_second': 59.195, 'eval_steps_per_second': 59.195, 'eval_SequenceClassification_loss': 1.3212300370931624, 'eval_SequenceClassification_num_batches': 5000, 'eval_SequenceClassification_SequenceAccuracy': 0.5508, 'epoch': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  18%|█▊        | 1099/6250 [01:35<04:31, 18.96batches/s, epoch=3, loss=1.79, split=train]  Converged objectives: []\n",
      "SequenceClassification:  18%|█▊        | 1103/6250 [01:35<05:59, 14.30batches/s, epoch=3, loss=0.781, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0702, 'learning_rate': 1.9955056179775283e-05, 'train_SequenceClassification_loss': 1.0723766617029906, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.11}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  30%|███       | 1899/6250 [02:23<05:11, 13.98batches/s, epoch=3, loss=0.99, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0789, 'learning_rate': 1.9948635634028894e-05, 'train_SequenceClassification_loss': 1.073704716950655, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.12}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [01:55<00:00,  5.43batches/s, epoch=3, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:10<00:00, 59.27batches/s, epoch=3, loss=1.07, split=eval] \n",
      "SequenceClassification:  99%|█████████▉| 618/625 [00:09<00:00, 69.91batches/s, epoch=3, loss=-1, split=eval]rain]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0545318126678467, 'eval_runtime': 10.5456, 'eval_samples_per_second': 59.267, 'eval_steps_per_second': 59.267, 'eval_SequenceClassification_loss': 1.2915969106515248, 'eval_SequenceClassification_num_batches': 5625, 'eval_SequenceClassification_SequenceAccuracy': 0.5744, 'epoch': 0.12}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  43%|████▎     | 2700/6250 [03:43<04:08, 14.29batches/s, epoch=3, loss=0.386, split=train]  Converged objectives: []\n",
      "SequenceClassification:  43%|████▎     | 2702/6250 [03:43<03:57, 14.94batches/s, epoch=3, loss=0.69, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0297, 'learning_rate': 1.9942215088282505e-05, 'train_SequenceClassification_loss': 1.047823493182659, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.12}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  56%|█████▌    | 3499/6250 [04:50<04:40,  9.81batches/s, epoch=3, loss=1, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.03, 'learning_rate': 1.993579454253612e-05, 'train_SequenceClassification_loss': 1.0191282953172922, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [02:15<00:00,  4.61batches/s, epoch=3, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 34.37batches/s, epoch=3, loss=1.21, split=eval] n]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:17<00:00, 40.04batches/s, epoch=3, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.034177541732788, 'eval_runtime': 18.1862, 'eval_samples_per_second': 34.367, 'eval_steps_per_second': 34.367, 'eval_SequenceClassification_loss': 1.265854975129366, 'eval_SequenceClassification_num_batches': 6250, 'eval_SequenceClassification_SequenceAccuracy': 0.5808, 'epoch': 0.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  69%|██████▉   | 4299/6250 [06:55<03:14, 10.03batches/s, epoch=3, loss=0.556, split=train]  Converged objectives: []\n",
      "SequenceClassification:  69%|██████▉   | 4302/6250 [06:55<04:04,  7.97batches/s, epoch=3, loss=1.12, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9672, 'learning_rate': 1.992937399678973e-05, 'train_SequenceClassification_loss': 0.9752445418387652, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  82%|████████▏ | 5100/6250 [08:17<01:27, 13.09batches/s, epoch=3, loss=0.711, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9491, 'learning_rate': 1.992295345104334e-05, 'train_SequenceClassification_loss': 0.9467247564047575, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.14}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:09<00:00,  3.30batches/s, epoch=3, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:17<00:00, 35.11batches/s, epoch=3, loss=1.14, split=eval] in]\n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:16<00:00, 40.60batches/s, epoch=3, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0453429222106934, 'eval_runtime': 17.8016, 'eval_samples_per_second': 35.109, 'eval_steps_per_second': 35.109, 'eval_SequenceClassification_loss': 1.245808419204842, 'eval_SequenceClassification_num_batches': 6875, 'eval_SequenceClassification_SequenceAccuracy': 0.5732, 'epoch': 0.14}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  94%|█████████▍| 5899/6250 [10:20<00:27, 12.68batches/s, epoch=3, loss=1.25, split=train]   Converged objectives: []\n",
      "SequenceClassification:  94%|█████████▍| 5903/6250 [10:20<00:31, 11.17batches/s, epoch=3, loss=0.591, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9317, 'learning_rate': 1.991653290529695e-05, 'train_SequenceClassification_loss': 0.9310446998402476, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.15}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 6250/6250 [10:56<00:00, 11.18batches/s, epoch=3, loss=1.73, split=train] Converged objectives: []\n",
      "SequenceClassification: 100%|██████████| 6250/6250 [10:56<00:00,  9.52batches/s, epoch=3, loss=0.436, split=train]\n",
      "SequenceClassification:   7%|▋         | 449/6250 [00:44<09:11, 10.52batches/s, epoch=4, loss=1.24, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9136, 'learning_rate': 1.9910112359550565e-05, 'train_SequenceClassification_loss': 0.9135694723203778, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.15}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:06<00:00,  3.36batches/s, epoch=3, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.53batches/s, epoch=4, loss=0.876, split=eval] \n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:17<00:00, 39.50batches/s, epoch=4, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0503120422363281, 'eval_runtime': 18.6407, 'eval_samples_per_second': 33.529, 'eval_steps_per_second': 33.529, 'eval_SequenceClassification_loss': 1.2295170531610649, 'eval_SequenceClassification_num_batches': 7500, 'eval_SequenceClassification_SequenceAccuracy': 0.5776, 'epoch': 0.15}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  20%|█▉        | 1249/6250 [02:49<07:58, 10.45batches/s, epoch=4, loss=2.28, split=train]  Converged objectives: []\n",
      "SequenceClassification:  20%|██        | 1253/6250 [02:50<08:54,  9.34batches/s, epoch=4, loss=1.07, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9005, 'learning_rate': 1.9903691813804176e-05, 'train_SequenceClassification_loss': 0.906367786437273, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.16}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  33%|███▎      | 2049/6250 [04:12<06:17, 11.14batches/s, epoch=4, loss=0.559, split=train]Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9216, 'learning_rate': 1.9897271268057787e-05, 'train_SequenceClassification_loss': 0.9126941991522908, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.17}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:08<00:00,  3.31batches/s, epoch=4, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.35batches/s, epoch=4, loss=1.46, split=eval] in]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:17<00:00, 35.13batches/s, epoch=4, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.088841438293457, 'eval_runtime': 18.7421, 'eval_samples_per_second': 33.347, 'eval_steps_per_second': 33.347, 'eval_SequenceClassification_loss': 1.2186958487639061, 'eval_SequenceClassification_num_batches': 8125, 'eval_SequenceClassification_SequenceAccuracy': 0.5748, 'epoch': 0.17}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  46%|████▌     | 2849/6250 [06:19<05:23, 10.52batches/s, epoch=4, loss=0.805, split=train]  Converged objectives: []\n",
      "SequenceClassification:  46%|████▌     | 2851/6250 [06:19<06:15,  9.05batches/s, epoch=4, loss=0.762, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8983, 'learning_rate': 1.9890850722311397e-05, 'train_SequenceClassification_loss': 0.8973047889471054, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.17}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  58%|█████▊    | 3650/6250 [07:36<04:18, 10.06batches/s, epoch=4, loss=1.81, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8727, 'learning_rate': 1.988443017656501e-05, 'train_SequenceClassification_loss': 0.8773357503265142, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.18}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:05<00:00,  3.36batches/s, epoch=4, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:16<00:00, 38.92batches/s, epoch=4, loss=0.902, split=eval]n]\n",
      "SequenceClassification: 100%|█████████▉| 622/625 [00:14<00:00, 44.36batches/s, epoch=4, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.164727807044983, 'eval_runtime': 16.0588, 'eval_samples_per_second': 38.919, 'eval_steps_per_second': 38.919, 'eval_SequenceClassification_loss': 1.2148409901802029, 'eval_SequenceClassification_num_batches': 8750, 'eval_SequenceClassification_SequenceAccuracy': 0.5696, 'epoch': 0.18}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  71%|███████   | 4449/6250 [09:27<03:04,  9.78batches/s, epoch=4, loss=1.25, split=train]   Converged objectives: []\n",
      "SequenceClassification:  71%|███████   | 4453/6250 [09:28<03:00,  9.96batches/s, epoch=4, loss=0.296, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8894, 'learning_rate': 1.9878009630818622e-05, 'train_SequenceClassification_loss': 0.886393307954073, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.19}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  84%|████████▍ | 5249/6250 [10:49<01:28, 11.31batches/s, epoch=4, loss=0.807, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.81, 'learning_rate': 1.9871589085072233e-05, 'train_SequenceClassification_loss': 0.838916339289397, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.19}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [02:56<00:00,  3.54batches/s, epoch=4, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.82batches/s, epoch=4, loss=0.982, split=eval]in]\n",
      "SequenceClassification: 100%|█████████▉| 622/625 [00:17<00:00, 37.40batches/s, epoch=4, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0787484645843506, 'eval_runtime': 18.479, 'eval_samples_per_second': 33.822, 'eval_steps_per_second': 33.822, 'eval_SequenceClassification_loss': 1.20576815628767, 'eval_SequenceClassification_num_batches': 9375, 'eval_SequenceClassification_SequenceAccuracy': 0.5832, 'epoch': 0.19}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  97%|█████████▋| 6050/6250 [12:51<00:19, 10.20batches/s, epoch=4, loss=0.857, split=train]  Converged objectives: []\n",
      "SequenceClassification:  97%|█████████▋| 6052/6250 [12:52<00:20,  9.56batches/s, epoch=4, loss=0.714, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7877, 'learning_rate': 1.9865168539325844e-05, 'train_SequenceClassification_loss': 0.7906617993675172, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|█████████▉| 6249/6250 [13:12<00:00, 10.79batches/s, epoch=4, loss=0.204, split=train]Converged objectives: []\n",
      "SequenceClassification: 100%|██████████| 6250/6250 [13:12<00:00,  7.88batches/s, epoch=4, loss=0.274, split=train]\n",
      "SequenceClassification:  10%|▉         | 600/6250 [01:00<09:44,  9.67batches/s, epoch=5, loss=0.193, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.788, 'learning_rate': 1.9858747993579458e-05, 'train_SequenceClassification_loss': 0.7754975991696119, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:05<00:00,  3.37batches/s, epoch=4, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.50batches/s, epoch=5, loss=0.726, split=eval] \n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:17<00:00, 40.49batches/s, epoch=5, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0855845212936401, 'eval_runtime': 18.6592, 'eval_samples_per_second': 33.496, 'eval_steps_per_second': 33.496, 'eval_SequenceClassification_loss': 1.1982566833805293, 'eval_SequenceClassification_num_batches': 10000, 'eval_SequenceClassification_SequenceAccuracy': 0.6012, 'epoch': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  22%|██▏       | 1399/6250 [03:05<07:24, 10.92batches/s, epoch=5, loss=1.63, split=train]  Converged objectives: []\n",
      "SequenceClassification:  22%|██▏       | 1403/6250 [03:05<06:57, 11.61batches/s, epoch=5, loss=0.517, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7722, 'learning_rate': 1.985232744783307e-05, 'train_SequenceClassification_loss': 0.7749191367141902, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.21}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  35%|███▌      | 2200/6250 [04:28<08:15,  8.18batches/s, epoch=5, loss=0.863, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8283, 'learning_rate': 1.984590690208668e-05, 'train_SequenceClassification_loss': 0.8308469288982451, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:09<00:00,  3.30batches/s, epoch=5, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 34.04batches/s, epoch=5, loss=0.671, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:17<00:00, 42.15batches/s, epoch=5, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.1354830265045166, 'eval_runtime': 18.3611, 'eval_samples_per_second': 34.039, 'eval_steps_per_second': 34.039, 'eval_SequenceClassification_loss': 1.1574501658227294, 'eval_SequenceClassification_num_batches': 10000, 'eval_SequenceClassification_SequenceAccuracy': 0.5692, 'epoch': 0.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  48%|████▊     | 2999/6250 [06:34<05:20, 10.16batches/s, epoch=5, loss=0.554, split=train]  Converged objectives: []\n",
      "SequenceClassification:  48%|████▊     | 3003/6250 [06:35<05:25,  9.99batches/s, epoch=5, loss=0.654, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8034, 'learning_rate': 1.983948635634029e-05, 'train_SequenceClassification_loss': 0.7967601801268757, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  61%|██████    | 3799/6250 [07:56<04:20,  9.40batches/s, epoch=5, loss=0.569, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7657, 'learning_rate': 1.98330658105939e-05, 'train_SequenceClassification_loss': 0.787080108102411, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:10<00:00,  3.29batches/s, epoch=5, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.27batches/s, epoch=5, loss=0.831, split=eval] n]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:17<00:00, 40.83batches/s, epoch=5, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.1050034761428833, 'eval_runtime': 18.7841, 'eval_samples_per_second': 33.273, 'eval_steps_per_second': 33.273, 'eval_SequenceClassification_loss': 1.1327598391219973, 'eval_SequenceClassification_num_batches': 10000, 'eval_SequenceClassification_SequenceAccuracy': 0.5924, 'epoch': 0.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  74%|███████▎  | 4600/6250 [09:59<01:51, 14.82batches/s, epoch=5, loss=1.15, split=train]   Converged objectives: []\n",
      "SequenceClassification:  74%|███████▎  | 4602/6250 [09:59<02:11, 12.49batches/s, epoch=5, loss=0.623, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7677, 'learning_rate': 1.982664526484751e-05, 'train_SequenceClassification_loss': 0.7652335932441056, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.24}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  86%|████████▋ | 5400/6250 [11:04<00:48, 17.44batches/s, epoch=5, loss=0.762, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6952, 'learning_rate': 1.9820224719101126e-05, 'train_SequenceClassification_loss': 0.7126810678839683, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.24}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [02:49<00:00,  3.69batches/s, epoch=5, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 34.07batches/s, epoch=5, loss=1.15, split=eval] in]\n",
      "SequenceClassification: 100%|█████████▉| 622/625 [00:17<00:00, 41.27batches/s, epoch=5, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.1131144762039185, 'eval_runtime': 18.355, 'eval_samples_per_second': 34.051, 'eval_steps_per_second': 34.051, 'eval_SequenceClassification_loss': 1.1199652938313782, 'eval_SequenceClassification_num_batches': 10000, 'eval_SequenceClassification_SequenceAccuracy': 0.5916, 'epoch': 0.24}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  99%|█████████▉| 6199/6250 [13:06<00:05,  9.17batches/s, epoch=5, loss=0.368, split=train]  Converged objectives: []\n",
      "SequenceClassification:  99%|█████████▉| 6203/6250 [13:07<00:04, 10.05batches/s, epoch=5, loss=0.512, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6894, 'learning_rate': 1.9813804173354736e-05, 'train_SequenceClassification_loss': 0.6986253183111548, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.25}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|█████████▉| 6249/6250 [13:11<00:00,  9.69batches/s, epoch=5, loss=0.186, split=train]Converged objectives: []\n",
      "SequenceClassification: 100%|██████████| 6250/6250 [13:12<00:00,  7.89batches/s, epoch=5, loss=0.192, split=train]\n",
      "SequenceClassification:  12%|█▏        | 749/6250 [01:14<09:55,  9.24batches/s, epoch=6, loss=0.74, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6872, 'learning_rate': 1.9807383627608347e-05, 'train_SequenceClassification_loss': 0.6885251641608775, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:03<00:00,  3.41batches/s, epoch=5, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.59batches/s, epoch=6, loss=0.714, split=eval] ]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 38.55batches/s, epoch=6, loss=-1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.1804156303405762, 'eval_runtime': 18.6061, 'eval_samples_per_second': 33.591, 'eval_steps_per_second': 33.591, 'eval_SequenceClassification_loss': 1.1137810640845447, 'eval_SequenceClassification_num_batches': 10000, 'eval_SequenceClassification_SequenceAccuracy': 0.5896, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  25%|██▍       | 1549/6250 [03:21<08:14,  9.51batches/s, epoch=6, loss=0.514, split=train]  Converged objectives: []\n",
      "SequenceClassification:  25%|██▍       | 1552/6250 [03:22<09:45,  8.03batches/s, epoch=6, loss=0.819, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6924, 'learning_rate': 1.9800963081861958e-05, 'train_SequenceClassification_loss': 0.6893863456472754, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification:  38%|███▊      | 2349/6250 [04:43<07:33,  8.59batches/s, epoch=6, loss=0.66, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.686, 'learning_rate': 1.9794542536115572e-05, 'train_SequenceClassification_loss': 0.7037654118854553, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "SequenceClassification: 100%|██████████| 625/625 [03:09<00:00,  3.29batches/s, epoch=6, loss=-1, split=eval]\n",
      "SequenceClassification: 100%|██████████| 625/625 [00:18<00:00, 33.74batches/s, epoch=6, loss=0.854, split=eval] \n",
      "SequenceClassification: 100%|█████████▉| 624/625 [00:17<00:00, 42.28batches/s, epoch=6, loss=-1, split=eval]Objective `SequenceClassification` convergence metric `SequenceAccuracy` did not improve for 5 eval steps. History: [0.5692, 0.5924, 0.5916, 0.5896, 0.5532]\n",
      "Converged objectives: ['SequenceClassification']\n",
      "Scheduler reached a termination condition: FIRST_OBJECTIVE_CONVERGED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.3328857421875, 'eval_runtime': 18.5229, 'eval_samples_per_second': 33.742, 'eval_steps_per_second': 33.742, 'eval_SequenceClassification_loss': 1.1185720396701246, 'eval_SequenceClassification_num_batches': 10000, 'eval_SequenceClassification_SequenceAccuracy': 0.5532, 'epoch': 0.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Objective `SequenceClassification` convergence metric `SequenceAccuracy` did not improve for 5 eval steps. History: [0.5692, 0.5924, 0.5916, 0.5896, 0.5532]\n",
      "Converged objectives: ['SequenceClassification']\n",
      "Scheduler reached a termination condition: FIRST_OBJECTIVE_CONVERGED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 3983.5237, 'train_samples_per_second': 627.585, 'train_steps_per_second': 78.448, 'train_loss': 1.0587789444696336, 'train_SequenceClassification_loss': 0.7037654118854553, 'train_SequenceClassification_num_batches': 1000, 'epoch': 0.27}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=4200, training_loss=1.0587789444696336, metrics={'train_runtime': 3983.5237, 'train_samples_per_second': 627.585, 'train_steps_per_second': 78.448, 'total_flos': 0.0, 'train_loss': 1.0587789444696336})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adapter.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ef2f71-a322-478a-9ffc-ad4114e1b7f2",
   "metadata": {
    "id": "24ef2f71-a322-478a-9ffc-ad4114e1b7f2"
   },
   "source": [
    "#### Evaluations report\n",
    "\n",
    "We'll briefly take a look at the dynamics of the training process.\n",
    "\n",
    "We see that in our case, evaluation loss is more reliable early-stopping criterion than classfication accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "331820af-4701-4392-909e-1f25c80665c9",
   "metadata": {
    "tags": [],
    "id": "331820af-4701-4392-909e-1f25c80665c9",
    "outputId": "18ed5f04-dbc7-40d9-9144-73ef6bfe2be2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAAESCAYAAACCZc2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5uElEQVR4nO3deXwU9f0/8NfuZo9sks1J7oQc3FeAYEJAPDkUFfGoKLRgVKpCWjW1Rb79yaFWtFqktQhqRVotilqVKhQMgSCSQJCAIEeAJOQg933vbnbn90eShZBzw25md/N6Ph55wM5+ZvbzzmSzr8x85jMSQRAEEBER0aAmFbsDREREJD4GAiIiImIgICIiIgYCIiIiAgMBERERgYGAiIiIwEBAREREAJzE7kBfGI1GFBYWws3NDRKJROzuEBER2Q1BEFBXV4fAwEBIpd0fB7CLQFBYWIiQkBCxu0FERGS38vPzERwc3O3zdhEI3NzcALQWo9FoLLJNvV6P7777DrNnz4ZcLrfINsXGmmyfo9UDsCZ7wZrsgzVqqq2tRUhIiOmztDt2EQjaTxNoNBqLBgK1Wg2NRuNQP0isybY5Wj0Aa7IXrMk+WLOm3k65c1AhERERMRAQERERAwERERGBgYCIiIjAQEBEREToZyDYuHEjwsLCoFKpEBsbi/T09B7bV1dXY/ny5QgICIBSqcSIESOwa9eufnWYiIiILM/syw63b9+OxMREbN68GbGxsdiwYQPmzJmDzMxM+Pr6dmqv0+kwa9Ys+Pr64osvvkBQUBByc3Ph4eFhif4TEVEf5JQ3oFYndi/IlpkdCNavX4+lS5ciPj4eALB582bs3LkTW7ZswQsvvNCp/ZYtW1BZWYnU1FTTNZVhYWE9voZWq4VWqzU9rq2tBdB6faZerze3y11q346ltmcLWJPtc7R6ANZky4xGAfszy/BhWi6O5FRBJZPBa1gRZo8LELtrFuEo++lq1qipr9uSCIIg9HWjOp0OarUaX3zxBebPn29avmTJElRXV2PHjh2d1pk7dy68vLygVquxY8cODBkyBAsXLsSKFSsgk8m6fJ01a9Zg7dq1nZZv27YNarW6r90lIhqUtAYgvUyClCIpyps7T0YzO8iIO0OMkPLWMINCY2MjFi5ciJqamh4n9zPrCEF5eTkMBgP8/Pw6LPfz88O5c+e6XCc7Oxv79u3DokWLsGvXLly8eBHLli2DXq/H6tWru1xn5cqVSExMND1un3Zx9uzZFp2pMCkpCbNmzXKoGa5Yk21ztHoA1mRLimqa8dHhPGw/XoDa5hYAgJvKCQ9PCcaDk/zxp89T8X2xFN9dlqJZPQR/eXACPNT2U9+17HU/9cQaNbUfZe+N1acuNhqN8PX1xXvvvQeZTIbo6GhcvnwZb7zxRreBQKlUQqlUdloul8stvtOtsU2xsSbb52j1AKxJTCfyq/HBDznYdaoIBmPrQd8wbzXip4fjwehguCidoNfr8UC4EfOmT8D/++8ZfH+hAve/exibFkVjXJC7yBVcH3vZT+awZE193Y5ZgcDHxwcymQwlJSUdlpeUlMDf37/LdQICAiCXyzucHhg9ejSKi4uh0+mgUCjM6QIREQFoMRjx3ZkSfPBDDo7lVpmWT43wwuM3RuC2Ub6QdXFO4N6JgRgT5ImnPj6GvMpGPLApFa/eNx4PRHd/F7yB0qw34J+pl3D0UhWcpBI4ySSQy6Rt/5dCLpPASdr2r0wCKQRkF0hQcDAHSrlTa9u2dTq0lUqhcJLCRSmDWuEEV6UT1AoZXJROUDpJe53jf7AwKxAoFApER0cjOTnZNIbAaDQiOTkZCQkJXa4zffp0bNu2DUaj0XQf5vPnzyMgIIBhgIjITLXNenx2NB9bUy+hoKoJACCXSXBPVCAemx7ep7/2xwRq8E3CjXh2+3HszyzD7z7/CSfyq/Hi3WOgcBr46WkEQcDOU0VYt+scLlc3mbm2DDvzL/T7tWVSCVzawoGL0sn0/9bgcPVyJ7go25+TwVXphCFuSowPcneYQGH2KYPExEQsWbIEU6ZMQUxMDDZs2ICGhgbTVQeLFy9GUFAQ1q1bBwB4+umn8fe//x3PPPMMfvOb3+DChQt49dVX8dvf/taylRARObD8ykZ8eOgSPvsxH/Xa1vEBnmo5FsUOxeK4ofDVqMzanrtajg+W3IC/Jl/AX5Mv4KPDuThdWIN3FkXD3928bV2PkwXVeOmbM/ix7SiHv0aFx28Mh0ohQ4vBiBaDAL2x7V+DEXqD0LrcKECrb0FObh4CAoNgECRoMXZ8Xm9aX4CuxYhGXQsatC1o0BrQpDcAAAxGAbXNLaYxF+a6e0IA/vJQFJROXQ+StydmB4IFCxagrKwMq1atQnFxMSZOnIjdu3ebBhrm5eWZjgQAQEhICPbs2YPnnnsOEyZMQFBQEJ555hmsWLHCclUQETkgQRDwY24VPjiYg+/OFKNteACG+brisenhuH9yEFTy/n8QSaUSPDdrBKJC3PHspyeQkVeNu98+iI0LJyM2wttCVXStuKYZf95zDl9mXAYAOMtlePLmCPz6pgioFX37aNLr9di16xLmzh1v9vl2g1FoCwgGNFwVFBq0LW2PDWjUtaBe24JGnQH12o5tGnUtOFNUi29PFqG8Xov3Fk+BRmXf4xj6NagwISGh21MEKSkpnZbFxcXh8OHD/XkpIqJBR28wYtepInzwQw5OFtSYls8Y7oPHbwzHTcOHQGrBawZvG+WHb35zI5786BjOFddh4T+O4P/mjsZj08Msfji8SWfA+wezsSkly/RX+v2TgvD7O0YiwN3Zoq/VE5lUAjeVHG7X8SF+6GI5nvzoGA5nV+KhzWnYGh8zoEdXLM3qVxkQEVHvjEYBx/Or8O3JIuw8WYTSutbJ2RROUtw/KQiP3RiOEX5uVnv9od4u+GrZdKz88iS+PlGIl789gxP51Xj9gfF9/ou9J4Ig4L8/FeL1/51DYU0zACB6qCdevHsMJoZ4XPf2xTB9mA+2PzkVj354FOeK63D/O4fwr8djMMzXevvJmhgIiByYwSjgclUTssrqTV8tBgGjAjQY0/blbsfXodu79hCw82Qxdp0qQnFts+k5H1clFscNxaLYUHi7dr4M2xqcFTK8tWAiJoZ44JWdZ/HNT4U4X1yHzb+KRriPS7+3ezyvCi99ewbH86oBAEEeznjhzlG4e0KA3Q/IGxvoji+fnoYlH6Yju6wBD2xKwz+WTMENYV5id81sDAREDqBJZ7jqQ7+h9d/SeuSUN0DbYuxx3SAPZ4wO0GBMoAZjAtwwJsAdIV7Oov+iNhoFix4WtxVXh4D//VyEoporIcBV6YRZY/wwd3wAbhrhI8pANYlEgkenh2NskDuW/TsDmSV1mPf2D3hrwUTMHOPX+wauUljdhD/vPoevTxQCANQKGZbdEoknZkRc19gHWxPipcZ/npqGx/55FMfzqvHLfxzBXx+ehDvGdX05vq1iICCyE4IgoLxeh6yyelwsverDv7S+x0u1FE5SRPi4IHKIKyKHuEAikeBsUS3OFNWioKoJl6tbv/aevTK/iJvSCaMDNBgd4NYWFNwx3M/VIr/EBUFAdaMeRTXNKKltRlFNM4prmlBs+n8zimubUdfcAheFDB5qBTTOcng4y+GhlsPdWQ53tRwezgq4X73sqv+7Kp1EDzRXaw0B1dh5sqjHEDBjuI/NfFDeEOaFnb+5Ecu3ZeDopSo88a8f8ZvbhuHZmSO6nN/gao26Frx7IBvvfp+FZr0REgnw4ORgPD9nJPzMvBrCXni6KLDtian4zSfHsfdsCZb9+xjW3jsOv5o6VOyu9RkDAZGNOpZbhaOXKpFVWo+LbX/x93RplJeLApFD2j/4XRHp64JhQ9wQ5Onc7S/wmiY9zrWFgzOFtThbXIvzxfWo07Yg/VIl0i9VmtrKpBJEDnFpPdXQFhJGB7hBo7xyVZHBKKC8XnvlQ76mGUW1zSipafuwr239wO/tqEW7Bp0BDboms69Nl0kl8HC+Eh7cTYFCgSAPZwR7OiPES40QTzU0ztYJD+0hYNepIuw61TkEzBzti7smBNpUCLiWr0aFbUun4k87z2Jr6iW8ve8ifiqowd8enggPded5ZIxGAV+fuIw/7840nf6ICfPCi3ePwfhg+54NsS+cFTJs/uVkvLjjND5Jz8OLX/+M4pomPD97pE0F1O4wEBDZmNyKBryy8yySzpR0ek4iAUI81Rjm63rlw9+3NQB4uZg/0Ze7sxyxEd4dLjHTG4zIKqtvDQhXhYWqRj3Ol9TjfEm96RAwAPi5KaE0yrDu9AGU1etMU+f2xstFAX+NCgHuKvi5qxCgUcHfvfUrwF0FD7UC9c0tqG7So7pRh5omfetXo75tWdvjJp3p/9VNeuhajDAYBVQ06FDR0Pv9ft2UTgj2UiPE0xnBnmqEeDkjQKNAYQPQoG2BhxmXs10dAv53qsg0eA64EgJaTwcMsdkQcC25TIo188ZiYogHXvjyJL4/X4a73/4Bm3/ZccrjY7mVeOmbM/ip7aqIYE9n/N/c0bhznL9dfBhaipNMilfvG4cAdxXWJ53Hxv1ZKKnVYt394yGXDfykT+ZgICCyEfXaFmzcfxEfHMyBzmCETCrB7DF+GOWvaQ0Avi4I83ax+geJXCbFKH8NRvlfuZGYIAgorm3uFBIuVTSipE4LQAKgdVS8VAL4aVTwa/uw93dXwV/T/kHvDH+NCr4aZZ/q8OnHYLpmvQHVjXpUN+lM4aE9SFQ06HC5ugn5lY0oqGpCeb0WddoWnC1qrasjJ7x+ch+8XBSmsBDs5YwQT7XpCEOQhzOUTtIrpwOuCQEuCtlVYwLsJwR0Zf6kIIz0d8NTHx9DbkXrlMd/um88pkZ44fXdmfjmp9aQ6KKQIeG24YifHmbX9V4PiUSC394+HL5uSvzx65/xxbEClNdrsXHhZLgobfdj13Z7RjRIGI0Cvjp+Ga/vPme61GzGcB+sunsMhlvxMjNzSCQSBLg7I8DdGbePvjKwrF7bgtMFVdhzIA1zb52GYC9X+Lgq4CTiX0IquQz+7rI+XQ/epDOgoKo1HOS3/1vZiLzKBuSU1qKxRYLKBh0qG3Smv3yv5ap0Ms0cCLR+IM4c44e7HCAEXGt0gAb/XX4jnvvsBPadK8Xzn/8EuUwCvUGARAIsmBKCxNkj4OvmmOMEzPVwTCiGuCmxfFsGUjLL8Mj7h7Hl0Rv6FXQHAgMBkYiO51VhzTdn8FN+NQBgqLcaL941BreP9rWLw6yuSidMDvVAsbeAqGB3u7vjnLNChuF+bp2CV+sMeLsw47ZZKK5rQUFVI/KrrhxZKKhqRH5lIxraZrBrDwFzxwfgZgcLAddyV8vxj8VT8Pa+i9iQfB56g4CpEa3jBMYGOv44AXPdPtoPnyydise2HsXJgho8sCkV/3osBkO9+38Zp7UwEBCJoKS2Gev3nsaXx1unbXVRyPCb21sPszrCnOiOwk0lh5ebGmMCNZ2ea79aoqSueUBO5dgSqVSCZ2YOx00jfNCoM2BapLddBFixTAr1xH/a5irIrWjE/e+k4sP4GzAh2EPsrnVg2yMciByMVm9A0mUJZv/1kCkM/CI6GPt/fwueujmSYcCOSCQSeLooMMpfM6jCwNUmhXpi+jAfhoE+iBjiiv88PQ1jAzWoaNDh4fcOY39mqdjd6oCBgGgACIKA3T8X4463U/FtngyNOgMmhXpgx/LpeOMXUTznSjQI+LqpsP3JOMwY3npk5Yl//ojPf8wXu1smPGVAZGWZxXV46dvTOHSxAgDgLhfw4r0T8EB0CP+yIhpkXJVO+GDJDVjxn5P46vhl/P6Lkyit02LZLZGi/z5gICCykupGHdYnncfHh3NhFFpnDHx8+lCEN13AvVH2P4c7EfWPwkmKv/wiCn4aFTYfyMIbezJRXNOMNfPGitovBgIiC2sxGLEtPQ/rk86julEPALhjrD/+eNdo+LvJsWvXBZF7SERik0oleOHOUfDXKLH22zP46HAuSuua8eYD40TrEwMBkQUduliOtd+cxvmSegDASD83rL5nDKYN8wHQejkbEVG7R6eHY4ibCs9tP4E9p0tQXqfFA77i9IWBgMgC8ioa8addZ7DndOt0wx5qOX43awQeiQkVdZIeIrJ9d00IgLerAkv/9SOO5VWjvFKGB+f1bQpwS2IgILoOjboWvLM/C+8dzIaupXW64V9NHYpnZw7v8uYvRERdmRrhjS+emobH/3kUc/3rRRljxEBA1A+CIODbk0V4dddZ013spg/zxqq7x2Kkv21MN0xE9mWkvxv2/HY69n63W5TXZyAgMtO54lqs+e9pHM5uvTVwkIczXrx7DOaM9eOVA0R0XRRO4p1iZCAg6qOaRj3e2nseHx3OhcEoQOkkxdO3ROKpmyMH7Ux1ROQ4GAiIemEwCvjsx3y8sScTlQ06AMCd4/zxf3NHI8RLLXLviIgsg4GAqAcZeVVYveM0Tl1uvfXtMF9XrLlnLG4c7iNyz4iILIuBgKgLpXXNeP1/mfhPRgEAwE3phGdmDseSaWGQ8zJCInJADAREV9EbjPhn6iVs2HsB9doWAMCD0cFYcccoDHFTitw7IiLrYSAganPwQhnW/Pc0ssoaAABRwe5YM28sJoV6itwzIiLrYyCgQS+/shGv7Lwyy6C3iwJ/uGMkfhEdAqmUlxES0eDAQECDVpPOgE0HsvDugSxor5pl8LlZI+DuLBe7e0REA4qBgAYdQRCw++divLLzLC5XNwEA4iK8sWYeZxkkosGLgYAGBUEQcKG0HkeyK7DzVJFplsFAdxX+eNcYzB3vz1kGiWhQYyAgh2Q0CjhbXIsj2ZVIz6lE+qVK06RCQOv0oE/dFIGnbomEWsG3ARERfxOSQ2gxGHGmqDUAHMmpQHpOJWqbWzq0UcmliB7qiZgwb9w/OYizDBIRXYWBgOyS3mDEyYIaHMmpwJHsShzLrTLNG9DORSFDdJgXYsO9MDXCC+ODPES9cQgRkS1jICC7oG0xIiO/9S//IzmtAaBJb+jQxk3lhJgwL8RGeCE23BtjAzVw4qyCRER9wkBANqtZb8C7Kdn45rQMvz+6D7oWY4fnPdVyxIS3fvjHRnhhlL8GMs4bQETULwwEZLNe/PpnfH6sAIAEgBE+rkrERnhhargXYsK9MdzXlRMHERFZCAMB2aTThTX4ou3GQveFGfDUvJswIsCdlwYSEVkJT7CSzREEAX/aeRaCANw93h+3BAiIGOLCMEBEZEUMBGRzks+WIjWrAgonKZ6fPVzs7hARDQoMBGRT9AYjXv3fWQDA4zeGI8jDWeQeERENDgwEZFM+Sc9DdlkDvF0UWHZLpNjdISIaNPoVCDZu3IiwsDCoVCrExsYiPT2927Zbt26FRCLp8KVSqfrdYXJcNU16bNh7AQDw7KwRcFPxjoNERAPF7ECwfft2JCYmYvXq1cjIyEBUVBTmzJmD0tLSbtfRaDQoKioyfeXm5l5Xp8kxvbP/IiobdBjm64pHbggRuztERIOK2Zcdrl+/HkuXLkV8fDwAYPPmzdi5cye2bNmCF154oct1JBIJ/P39+/waWq0WWq3W9Li2thYAoNfrodfrze1yl9q3Y6nt2QJ7rim/qhFbDuUAAFbMGQ7BaIDeaLDrmrriaPUArMlesCb7YI2a+rotiSAIQl83qtPpoFar8cUXX2D+/Pmm5UuWLEF1dTV27NjRaZ2tW7fiiSeeQFBQEIxGIyZPnoxXX30VY8eO7fZ11qxZg7Vr13Zavm3bNqjVvCGNI9p6XorjFVKMcDdi2WgjeIUhEZFlNDY2YuHChaipqYFGo+m2nVlHCMrLy2EwGODn59dhuZ+fH86dO9flOiNHjsSWLVswYcIE1NTU4M0338S0adNw+vRpBAcHd7nOypUrkZiYaHpcW1uLkJAQzJ49u8dizKHX65GUlIRZs2ZBLneMc9X2WtPxvGocT0uHRAL85ZfTMcrfzfScvdbUHUerB2BN9oI12Qdr1NR+lL03Vp+pMC4uDnFxcabH06ZNw+jRo/Huu+/i5Zdf7nIdpVIJpVLZablcLrf4TrfGNsVmTzUJgoB1e84DAB6KDsH4EK8u29lTTX3haPUArMlesCb7YMma+rodswYV+vj4QCaToaSkpMPykpKSPo8RkMvlmDRpEi5evGjOS5OD2nmqCMfzqqFWyPC72SPE7g4R0aBlViBQKBSIjo5GcnKyaZnRaERycnKHowA9MRgMOHXqFAICAszrKTmcZr0Br+9uPdX05E2R8NXwclQiIrGYfcogMTERS5YswZQpUxATE4MNGzagoaHBdNXB4sWLERQUhHXr1gEAXnrpJUydOhXDhg1DdXU13njjDeTm5uKJJ56wbCVkd/6Zegn5lU3w16iw9KZwsbtDRDSomR0IFixYgLKyMqxatQrFxcWYOHEidu/ebRpomJeXB6n0yoGHqqoqLF26FMXFxfD09ER0dDRSU1MxZswYy1VBdqeyQYe/7289bfT8nJFQK3jjTSIiMfXrt3BCQgISEhK6fC4lJaXD47feegtvvfVWf16GHNhf955HXXMLxgZqcP+kILG7Q0Q06PFeBjTgLpbW4+MjeQCAP941GlIpJx0gIhIbAwENuNf+dxYGo4CZo/0wLdJH7O4QEREYCGiApWaVY+/ZUjhJJVg5d5TY3SEiojYMBDRgDEYBr3x7FgCwKDYUkUNcRe4RERG1YyCgAfNlRgHOFNXCTeWEZ2ZyEiIiIlvCQEADolHXgje/ywQA/Oa2YfByUYjcIyIiuhoDAQ2I97/PQUmtFiFezlgyLUzs7hAR0TUYCMjqSmqbsflAFgBgxR2joHSSidwjIiK6FgMBWd1fvstEk96AyaEeuGs872FBRGSLGAjIqs4U1uLzYwUAgP939xhIJJyEiIjIFjEQkNUIgoA/7ToDQQDunhCAyaGeYneJiIi6wUBAVrM/sxSHLlZAIZNixR2chIiIyJYxEJBVtBiMeHXXOQBA/I1hCPFSi9wjIiLqCQMBWcUnR/NxsbQeXi4KLL91mNjdISKiXjAQDEKldc04kl2Bem2LVbZf26zHW0nnAQDPzhwOjUpuldchIiLLcRK7AzRwMovr8P7BbOw4cRl6gwAnqQSTQj0wfZgPbhzmg6gQD8hl158R39mfhcoGHSKGuOCRmFAL9JyIiKyNgcDBCYKAQxcr8N7BbHx/vsy03NtFgYoGHY5eqsLRS1XYsPcCXBQyxEZ4mwLCCD9Xsy8TzK9sxJZDOQCAP84dbZGAQURE1sdA4KB0LUZ8e7IQ7x/MwdmiWgCAVALcOS4AT8wIx6RQT+RVNOJQVjl+uFiO1IvlqGrUY9+5Uuw7VwoA8HFV4sZh3pjWFhACPZx7fd039mRC12LEtEhv3DbK16o1EhGR5TAQOJjGFuC9gzn46HA+imubAQBqhQwPTQnBY9PDEep9ZbR/qLcaod6heCQmFEajgDNFtUjNKscPFyuQnlOB8notvj5RiK9PFAIAInxcMH2YD6YP80FchDfc1R3HBhzPq8J/fyqERAL88a7RnISIiMiOMBA4iIKqRnxwMBvbjsmgNV4AAAxxU+LRaWFYFBsKD3XPdxeUSiUYF+SOcUHu+PVNkdC2GJCRW41DF1uPIJwsqEZ2eQOyyxvw0eFcSCXA+GAPTI/0xo3DfDB5qCde2XkWAPDA5GCMDXS3es1ERGQ5DAR27mRBNd4/mINdp4pgMAoAJBju64Jf3xSJeRMD+30jIaWTDHGR3oiL9Mbzc0aipkmPI9kVpoCQVdaAn/Kr8VN+Nd5JyYJCJoXOYISzXIbnZ4+0bJFERGR1DAR2yGgUsD+zFO99n40jOZWm5dMivTBBUYbER6ZBoej5iIC53J3lmD3WH7PH+gMAimqacOjilYBQVqcFADx1cyT83VUWfW0iIrI+BgI70qw34Kvjl/GPg9nIKmsAADhJJZgXFYjHZ4RjxBA1du3aNSDn7gPcnfFgdDAejA6GIAi4WFqP/KpG3DKCAwmJiOwRA4EdqGzQ4aO0XHx0+BLK63UAADelExZODcWj08IQ4N46+l+v14vSP4lEguF+bhju5ybK6xMR0fVjILBx732fhfVJ59GsNwIAgjycET89DAtuCIEbZwAkIiILYSCwYRdK6rDuf+cgCMD4IHcsvSkCc8f5w4mT/RARkYUxENiwDckXIAjAnLF+2PzLaF7XT0REVsM/NW3UueJa7DxZBAB4btYIhgEiIrIqBgIbtSGpdXKhuyYEYJS/RuTeEBGRo2MgsEGnC2uw+3QxJBLg2duHi90dIiIaBBgIbNCGva1HB+6ZEMhL+YiIaEAwENiYUwU1SDpTAqkE+C2PDhAR0QBhILAxG/aeBwDcOzEIw3xdRe4NERENFgwENuREfjWSz5VCJpXw6AAREQ0oBgIb8lZS69GB+yYFIdzHReTeEBHRYMJAYCOO5VbhwPmy1qMDt/HoABERDSwGAhvRPnbgwcnBCPVWi9wbIiIabBgIbEB6TiUOXiiHk1SChNuGid0dIiIahBgIbED72IGHbghBiBePDhAR0cBjIBBZWlYF0rIrIJdJsPxWHh0gIiJx9CsQbNy4EWFhYVCpVIiNjUV6enqf1vv0008hkUgwf/78/ryswxEEAW+1jR14+IZQBHk4i9wjIiIarMwOBNu3b0diYiJWr16NjIwMREVFYc6cOSgtLe1xvUuXLuH555/HjBkz+t1ZR5OaVYH0nEoonKRYdmuk2N0hIqJBzOxAsH79eixduhTx8fEYM2YMNm/eDLVajS1btnS7jsFgwKJFi7B27VpERERcV4cdhSAIWN82dmBhTCgC3Hl0gIiIxONkTmOdTodjx45h5cqVpmVSqRQzZ85EWlpat+u99NJL8PX1xeOPP46DBw/2+jparRZardb0uLa2FgCg1+uh1+vN6XK32rdjqe2Z6+DFchzLrYLSSYqlNw61SD/ErskaHK0mR6sHYE32gjXZB2vU1NdtmRUIysvLYTAY4Ofn12G5n58fzp071+U6P/zwAz744AOcOHGiz6+zbt06rF27ttPy7777Dmq1ZUfhJyUlWXR7fSEIwFs/ywBIEDekBT8eTLbo9sWoydocrSZHqwdgTfaCNdkHS9bU2NjYp3ZmBQJz1dXV4Ve/+hXef/99+Pj49Hm9lStXIjEx0fS4trYWISEhmD17NjQajUX6ptfrkZSUhFmzZkEul1tkm32Vcr4MuYePQyWXYt3im+HjqrTIdsWsyVocrSZHqwdgTfaCNdkHa9TUfpS9N2YFAh8fH8hkMpSUlHRYXlJSAn9//07ts7KycOnSJdxzzz2mZUajsfWFnZyQmZmJyMjOg+mUSiWUys4fknK53OI73Rrb7IkgCHh7fzYAYHFcGAI8LX9Hw4GuaSA4Wk2OVg/AmuwFa7IPlqypr9sxa1ChQqFAdHQ0kpOvHOI2Go1ITk5GXFxcp/ajRo3CqVOncOLECdPXvHnzcOutt+LEiRMICQkx5+UdQvLZUpwsqIFaIcOTN3GAJRER2QazTxkkJiZiyZIlmDJlCmJiYrBhwwY0NDQgPj4eALB48WIEBQVh3bp1UKlUGDduXIf1PTw8AKDT8sHg6isLlkwLg7eFThUQERFdL7MDwYIFC1BWVoZVq1ahuLgYEydOxO7du00DDfPy8iCVcgLEruw5XYIzRbVwUcjw6xk8OkBERLajX4MKExISkJCQ0OVzKSkpPa67devW/ryk3TMaBdMdDeOnh8PTRSFyj4iIiK7gn/IDZPfpYpwrroOb0glPzAgXuztEREQdMBAMAINRMN3R8LEbw+Gh5tEBIiKyLQwEA2DnqSJcKK2HRuWEx27k0QEiIrI9DARWZjAK+Gvb2IEnZkTA3dmxrpUlIiLHwEBgZd/8VIissga4O8sRPz1M7O4QERF1iYHAiloMRvw1+QIA4Nc3RcBNxaMDRERkmxgIrOjrE4XIKW+Ap1qOJdPCxO4OERFRtxgIrERvMOLtfa1HB568ORKuSqveR4qIiOi6MBBYyVcZl5Fb0QhvFwUWxw0VuztEREQ9YiCwAl2LEX9rOzrw9C2RUCt4dICIiGwbA4EV/CejAAVVTfBxVWJRLI8OEBGR7WMgsDBtiwF/33cRALDslkg4K2Qi94iIiKh3DAQW9tmPBbhc3QQ/jRILY0PF7g4REVGfMBBYULPegI2mowPDoJLz6AAREdkHBgIL2n40H8W1zQhwV2HBDSFid4eIiKjPGAgspMVgxDsprUcHlt/KowNERGRfGAgs5OTlGpTUaqFROeGhKTw6QERE9oWBwELSsioAAFMjvKFw4reViIjsCz+5LORwdmsgmBbpLXJPiIiIzMdAYAHaFgOOXqoEAMRF+ojcGyIiIvMxEFjAT/k1aNYb4e2iwAg/V7G7Q0REZDYGAgswjR+I9IZEIhG5N0REROZjILCA1KxyAEBcBMcPEBGRfWIguE7NegOO51UDAOI4oJCIiOwUA8F1ysitgs5ghJ9GiQgfF7G7Q0RE1C8MBNcptW38QFwExw8QEZH9YiC4Tmlt8w/wdAEREdkzBoLr0KBtwU/51QCAaZx/gIiI7BgDwXU4eqkSLUYBQR7OCPFSi90dIiKifmMguA5pnK6YiIgcBAPBdWifkIjjB4iIyN4xEPRTbbMeP1+uAcBAQERE9o+BoJ/SsythFIBwHxcEuDuL3R0iIqLrwkDQT+3zD0zldMVEROQAGAj6ifMPEBGRI2Eg6IeqBh3OFtUC4A2NiIjIMTAQ9MPhtqMDw31dMcRNKXJviIiIrh8DQT/wdAERETkaBoJ+aJ9/gBMSERGRo2AgMFNpXTMulNZDIgFiwxkIiIjIMTAQmOlwdiUAYLS/Bp4uCpF7Q0REZBn9CgQbN25EWFgYVCoVYmNjkZ6e3m3bL7/8ElOmTIGHhwdcXFwwceJEfPTRR/3usNg4XTERETkiswPB9u3bkZiYiNWrVyMjIwNRUVGYM2cOSktLu2zv5eWFP/7xj0hLS8PJkycRHx+P+Ph47Nmz57o7L4a0rHIAvNyQiIgci9mBYP369Vi6dCni4+MxZswYbN68GWq1Glu2bOmy/S233IL77rsPo0ePRmRkJJ555hlMmDABP/zww3V3fqAV1TThUkUjpBIgJsJL7O4QERFZjJM5jXU6HY4dO4aVK1ealkmlUsycORNpaWm9ri8IAvbt24fMzEy8/vrr3bbTarXQarWmx7W1rZMA6fV66PV6c7rcrfbtmLO9H863HgUZF6iBs8y8dQdCf2qydY5Wk6PVA7Ame8Ga7IM1aurrtiSCIAh93WhhYSGCgoKQmpqKuLg40/I//OEPOHDgAI4cOdLlejU1NQgKCoJWq4VMJsM777yDxx57rNvXWbNmDdauXdtp+bZt26BWq/vaXYv790Up0sukuD3QiHlDjaL1g4iIqK8aGxuxcOFC1NTUQKPRdNvOrCME/eXm5oYTJ06gvr4eycnJSExMREREBG655ZYu269cuRKJiYmmx7W1tQgJCcHs2bN7LMYcer0eSUlJmDVrFuRyea/tBUHA6385CKAZi2ZOwYzhPhbphyWZW5M9cLSaHK0egDXZC9ZkH6xRU/tR9t6YFQh8fHwgk8lQUlLSYXlJSQn8/f27XU8qlWLYsGEAgIkTJ+Ls2bNYt25dt4FAqVRCqew8JbBcLrf4Tu/rNvMqGlFY0wwnqQSxkUMglw9IluoXa3yfxOZoNTlaPQBrshesyT5Ysqa+bsesQYUKhQLR0dFITk42LTMajUhOTu5wCqE3RqOxwxgBe5CW3Xp1QVSIB1yUthsGiIiI+sPsT7bExEQsWbIEU6ZMQUxMDDZs2ICGhgbEx8cDABYvXoygoCCsW7cOALBu3TpMmTIFkZGR0Gq12LVrFz766CNs2rTJspVYGacrJiIiR2Z2IFiwYAHKysqwatUqFBcXY+LEidi9ezf8/PwAAHl5eZBKrxx4aGhowLJly1BQUABnZ2eMGjUKH3/8MRYsWGC5KqxMEASktk9IxPkHiIjIAfXr2HdCQgISEhK6fC4lJaXD41deeQWvvPJKf17GZmSXN6C0TguFTIrJQz3F7g4REZHF8V4GfdB+umDyUA+o5DKRe0NERGR5DAR9YLp/QYTtXWpIRERkCQwEvRAEAYez2wYUDuP4ASIickwMBL04X1KPigYdnOUyRAV7iN0dIiIiq2Ag6EVq290Np4R5QuHEbxcRETkmfsL1wjR+gPMPEBGRA2Mg6IHBKOBITiUAzj9ARESOjYGgB2eLalHTpIer0gnjg9zF7g4REZHVMBD0oP10QUy4F5xk/FYREZHj4qdcD9KyOV0xERENDgwE3WgxGJHePn6AAwqJiMjBMRB049TlGtRrW+DuLMfoAI3Y3SEiIrIqBoJutJ8uiA33gkwqEbk3RERE1sVA0A3OP0BERIMJA0EXdC1GHL3UOn5gWiRvaERERI6PgaALJ/Kr0aw3wttFgRF+rmJ3h4iIyOoYCLrQfrpgaqQ3JBKOHyAiIsfHQNCFtOzWGxpx/gEiIhosGAiu0aw3ICO3GgAHFBIR0eDBQHCNjNwq6AxG+GmUiPBxEbs7REREA4KB4BpXT1fM8QNERDRYMBBcI5XzDxAR0SDEQHCVBm0LfsqvBsD5B4iIaHBhILjKj7lVaDEKCPJwRoiXWuzuEBERDRgGgqukZrVdbsjTBURENMgwEFzlcNv4gWkMBERENMgwELSpbdbj1OUaADxCQEREgw8DQZv07EoYBSDcxwUB7s5id4eIiGhAMRC0aZ9/YCqnKyYiokGIgaBNGucfICKiQYyBAEBVgw5nimoB8IZGREQ0ODEQADiS03p0YLivK4a4KUXuDRER0cBjIACnKyYiImIgwJXxA5x/gIiIBqtBHwjK67W4UFoPiQSIDWcgICKiwWnQB4LD2ZUAgFH+Gni6KETuDRERkTgYCHKqAPB0ARERDW6DPhAcyWk9QsDLDYmIaDAb1IGgWgtcqmiEVALERHiJ3R0iIiLRDOpAcKFWAgAYH+QOjUoucm+IiIjEM7gDQU1rIJjK8QNERDTI9SsQbNy4EWFhYVCpVIiNjUV6enq3bd9//33MmDEDnp6e8PT0xMyZM3tsP5DajxBMi/QRuSdERETiMjsQbN++HYmJiVi9ejUyMjIQFRWFOXPmoLS0tMv2KSkpeOSRR7B//36kpaUhJCQEs2fPxuXLl6+789cjv6oRlVoJnKQSTBnqKWpfiIiIxGZ2IFi/fj2WLl2K+Ph4jBkzBps3b4ZarcaWLVu6bP/vf/8by5Ytw8SJEzFq1Cj84x//gNFoRHJy8nV3/noczm693HBCsDtclE6i9oWIiEhsZn0S6nQ6HDt2DCtXrjQtk0qlmDlzJtLS0vq0jcbGRuj1enh5dT+qX6vVQqvVmh7X1rbeiVCv10Ov15vT5W6lZpUDAGKGultsm2Jrr8NR6gEcryZHqwdgTfaCNdkHa9TU121JBEEQ+rrRwsJCBAUFITU1FXFxcablf/jDH3DgwAEcOXKk120sW7YMe/bswenTp6FSqbpss2bNGqxdu7bT8m3btkGtVve1u90SBGD1MRlq9BIsH2PACPc+fwuIiIjsSmNjIxYuXIiamhpoNJpu2w3osfLXXnsNn376KVJSUroNAwCwcuVKJCYmmh7X1taaxh70VExf5ZQ3oObwIcgkAh6/91a4qbvviz3R6/VISkrCrFmzIJc7xmWUjlaTo9UDsCZ7wZrsgzVqaj/K3huzAoGPjw9kMhlKSko6LC8pKYG/v3+P67755pt47bXXsHfvXkyYMKHHtkqlEkqlstNyuVxukW9QUwswZagH6qsr4aZWOcwPUjtLfZ9siaPV5Gj1AKzJXrAm+2DJmvq6HbMGFSoUCkRHR3cYENg+QPDqUwjX+vOf/4yXX34Zu3fvxpQpU8x5SauICvHAJ0/E4KnRRrG7QkREZBPMPmWQmJiIJUuWYMqUKYiJicGGDRvQ0NCA+Ph4AMDixYsRFBSEdevWAQBef/11rFq1Ctu2bUNYWBiKi4sBAK6urnB1dbVgKeaTSER9eSIiIpthdiBYsGABysrKsGrVKhQXF2PixInYvXs3/Pz8AAB5eXmQSq8ceNi0aRN0Oh0efPDBDttZvXo11qxZc329JyIiIovo16DChIQEJCQkdPlcSkpKh8eXLl3qz0sQERHRABrU9zIgIiKiVgwERERExEBAREREDARERESEAZ6psL/aZ1fu62xLfaHX69HY2Ija2lqHmdCCNdk+R6sHYE32gjXZB2vU1P7Z2dudCuwiENTV1QEAQkJCRO4JERGRfaqrq4O7u3u3z5t1cyOxGI1GFBYWws3NDRILzSbUfn+E/Px8i9wfwRawJtvnaPUArMlesCb7YI2aBEFAXV0dAgMDO8wTdC27OEIglUoRHBxslW1rNBqH+UFqx5psn6PVA7Ame8Ga7IOla+rpyEA7DiokIiIiBgIiIiIaxIFAqVRi9erVXd5m2V6xJtvnaPUArMlesCb7IGZNdjGokIiIiKxr0B4hICIioisYCIiIiIiBgIiIiBgIiIiICAwEREREBAcPBBs3bkRYWBhUKhViY2ORnp7eY/vPP/8co0aNgkqlwvjx47Fr164B6mnv1q1bhxtuuAFubm7w9fXF/PnzkZmZ2eM6W7duhUQi6fClUqkGqMe9W7NmTaf+jRo1qsd1bHkfAUBYWFinmiQSCZYvX95le1vbR99//z3uueceBAYGQiKR4Ouvv+7wvCAIWLVqFQICAuDs7IyZM2fiwoULvW7X3PeiJfVUk16vx4oVKzB+/Hi4uLggMDAQixcvRmFhYY/b7M/PriX1tp8effTRTv274447et2ure4nAF2+ryQSCd54441utynmfurL7+zm5mYsX74c3t7ecHV1xQMPPICSkpIet9vf92BfOGwg2L59OxITE7F69WpkZGQgKioKc+bMQWlpaZftU1NT8cgjj+Dxxx/H8ePHMX/+fMyfPx8///zzAPe8awcOHMDy5ctx+PBhJCUlQa/XY/bs2WhoaOhxPY1Gg6KiItNXbm7uAPW4b8aOHduhfz/88EO3bW19HwHA0aNHO9STlJQEAPjFL37R7Tq2tI8aGhoQFRWFjRs3dvn8n//8Z/ztb3/D5s2bceTIEbi4uGDOnDlobm7udpvmvhctraeaGhsbkZGRgRdffBEZGRn48ssvkZmZiXnz5vW6XXN+di2tt/0EAHfccUeH/n3yySc9btOW9xOADrUUFRVhy5YtkEgkeOCBB3rcrlj7qS+/s5977jl88803+Pzzz3HgwAEUFhbi/vvv73G7/XkP9pngoGJiYoTly5ebHhsMBiEwMFBYt25dl+0feugh4a677uqwLDY2VnjyySet2s/+Ki0tFQAIBw4c6LbNhx9+KLi7uw9cp8y0evVqISoqqs/t7W0fCYIgPPPMM0JkZKRgNBq7fN6W9xEA4auvvjI9NhqNgr+/v/DGG2+YllVXVwtKpVL45JNPut2Oue9Fa7q2pq6kp6cLAITc3Nxu25j7s2tNXdW0ZMkS4d577zVrO/a2n+69917htttu67GNLe2na39nV1dXC3K5XPj8889Nbc6ePSsAENLS0rrcRn/fg33lkEcIdDodjh07hpkzZ5qWSaVSzJw5E2lpaV2uk5aW1qE9AMyZM6fb9mKrqakBAHh5efXYrr6+HkOHDkVISAjuvfdenD59eiC612cXLlxAYGAgIiIisGjRIuTl5XXb1t72kU6nw8cff4zHHnusx7t02vo+apeTk4Pi4uIO+8Dd3R2xsbHd7oP+vBfFVlNTA4lEAg8Pjx7bmfOzK4aUlBT4+vpi5MiRePrpp1FRUdFtW3vbTyUlJdi5cycef/zxXtvayn669nf2sWPHoNfrO3zPR40ahdDQ0G6/5/15D5rDIQNBeXk5DAYD/Pz8Oiz38/NDcXFxl+sUFxeb1V5MRqMRzz77LKZPn45x48Z1227kyJHYsmULduzYgY8//hhGoxHTpk1DQUHBAPa2e7Gxsdi6dSt2796NTZs2IScnBzNmzEBdXV2X7e1pHwHA119/jerqajz66KPdtrH1fXS19u+zOfugP+9FMTU3N2PFihV45JFHerzTnLk/uwPtjjvuwL/+9S8kJyfj9ddfx4EDB3DnnXfCYDB02d7e9tM///lPuLm59Xp43Vb2U1e/s4uLi6FQKDoFz94+p9rb9HUdc9jF7Y+po+XLl+Pnn3/u9VxYXFwc4uLiTI+nTZuG0aNH491338XLL79s7W726s477zT9f8KECYiNjcXQoUPx2Wef9Sn527oPPvgAd955JwIDA7ttY+v7aDDR6/V46KGHIAgCNm3a1GNbW//Zffjhh03/Hz9+PCZMmIDIyEikpKTg9ttvF7FnlrFlyxYsWrSo1wG4trKf+vo7W2wOeYTAx8cHMpms02jNkpIS+Pv7d7mOv7+/We3FkpCQgG+//Rb79+9HcHCwWevK5XJMmjQJFy9etFLvro+HhwdGjBjRbf/sZR8BQG5uLvbu3YsnnnjCrPVseR+1f5/N2Qf9eS+KoT0M5ObmIikpyez70Pf2syu2iIgI+Pj4dNs/e9lPAHDw4EFkZmaa/d4CxNlP3f3O9vf3h06nQ3V1dYf2vX1Otbfp6zrmcMhAoFAoEB0djeTkZNMyo9GI5OTkDn+NXS0uLq5DewBISkrqtv1AEwQBCQkJ+Oqrr7Bv3z6Eh4ebvQ2DwYBTp04hICDACj28fvX19cjKyuq2f7a+j6724YcfwtfXF3fddZdZ69nyPgoPD4e/v3+HfVBbW4sjR450uw/6814caO1h4MKFC9i7dy+8vb3N3kZvP7tiKygoQEVFRbf9s4f91O6DDz5AdHQ0oqKizF53IPdTb7+zo6OjIZfLO3zPMzMzkZeX1+33vD/vQXM77ZA+/fRTQalUClu3bhXOnDkj/PrXvxY8PDyE4uJiQRAE4Ve/+pXwwgsvmNofOnRIcHJyEt58803h7NmzwurVqwW5XC6cOnVKrBI6ePrppwV3d3chJSVFKCoqMn01Njaa2lxb09q1a4U9e/YIWVlZwrFjx4SHH35YUKlUwunTp8UooZPf/e53QkpKipCTkyMcOnRImDlzpuDj4yOUlpYKgmB/+6idwWAQQkNDhRUrVnR6ztb3UV1dnXD8+HHh+PHjAgBh/fr1wvHjx00j7l977TXBw8ND2LFjh3Dy5Enh3nvvFcLDw4WmpibTNm677Tbh7bffNj3u7b0oZk06nU6YN2+eEBwcLJw4caLDe0ur1XZbU28/u2LWVFdXJzz//PNCWlqakJOTI+zdu1eYPHmyMHz4cKG5ubnbmmx5P7WrqakR1Gq1sGnTpi63YUv7qS+/s5966ikhNDRU2Ldvn/Djjz8KcXFxQlxcXIftjBw5Uvjyyy9Nj/vyHuwvhw0EgiAIb7/9thAaGiooFAohJiZGOHz4sOm5m2++WViyZEmH9p999pkwYsQIQaFQCGPHjhV27tw5wD3uHoAuvz788ENTm2trevbZZ031+/n5CXPnzhUyMjIGvvPdWLBggRAQECAoFAohKChIWLBggXDx4kXT8/a2j9rt2bNHACBkZmZ2es7W99H+/fu7/Dlr77PRaBRefPFFwc/PT1AqlcLtt9/eqc6hQ4cKq1ev7rCsp/eitfVUU05OTrfvrf3793dbU28/u2LW1NjYKMyePVsYMmSIIJfLhaFDhwpLly7t9MFuT/up3bvvvis4OzsL1dXVXW7DlvZTX35nNzU1CcuWLRM8PT0FtVot3HfffUJRUVGn7Vy9Tl/eg/0laXtBIiIiGsQccgwBERERmYeBgIiIiBgIiIiIiIGAiIiIwEBAREREYCAgIiIiMBAQERERGAiIiIgIDAREREQEBgIiIiICAwEREREB+P/6DHFoLyC9zAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Evaluation Accuracy\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "pd.Series(training_objective.evaluations_history[\"eval\"][model_quality_evaluator]).plot(figsize=(6, 3), grid=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db170748-29d7-4daf-a1a8-b71fd1ef1e25",
   "metadata": {
    "tags": [],
    "id": "db170748-29d7-4daf-a1a8-b71fd1ef1e25",
    "outputId": "0303f873-7616-40bd-f666-877f4599a78d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAAESCAYAAACCZc2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8WklEQVR4nO3deVhU9f4H8PcZGAaGVUAUZHUlFRHXTEvcI6O4tlh2U7PFyla6Ldx7U7z1a73tmW0qWWFlid6WexM3UHMJZXJHEZR9FwZmYBiY8/sDmSLWwYEzM7xfz8MjZ/ie73w+HAbennPmHEEURRFERETUp8mkLoCIiIikx0BAREREDARERETEQEBERERgICAiIiIwEBAREREYCIiIiAiAvdQFdIXBYEBBQQFcXV0hCILU5RAREVkNURRRXV0NPz8/yGTt7wewikBQUFCAgIAAqcsgIiKyWrm5ufD392/361YRCFxdXQE0NePm5maWOfV6PbZv3465c+dCLpebZU6psSfrYGs92Vo/AHuyFuypa9RqNQICAox/S9tjFYGg+TCBm5ubWQOBUqmEm5ubTf0gsSfLZ2s92Vo/AHuyFuzJNJ0dcjf5pMLU1FRER0fDz88PgiBg69atna7z5ZdfIjw8HEqlEr6+vli2bBnKy8tNfWoiIiLqISYHAo1Gg/DwcKxZs6ZL4/fv34/Fixfj3nvvxcmTJ7F582YcPnwY999/v8nFEhERUc8w+ZBBVFQUoqKiujz+wIEDCA4OxmOPPQYACAkJwfLly/Hqq6+a+tRERETUQ3r8HIIpU6bg73//O3766SdERUWhpKQE3377LW644YZ219HpdNDpdMZltVoNoOnYil6vN0tdzfOYaz5LwJ6sg631ZGv9AOzJWrAn0+bsjCCKotjdJxEEAUlJSYiJielw3ObNm7Fs2TLU1dWhoaEB0dHR+O6779o9YSI+Ph6rV69u9XhiYiKUSmV3yyUiIupztFotFi1ahKqqqg5PzO/xQHDq1CnMnj0bTz75JObNm4fCwkI8/fTTmDhxItatW9fmOm3tIQgICEBZWZlZ32WQnJyMOXPm2NTZqezJ8tlaT7bWD8CerAV76hq1Wg1vb+9OA0GPHzJ4+eWXMXXqVDz99NMAgDFjxsDZ2RnXXnstXnzxRfj6+rZaR6FQQKFQtHpcLpeb7RuUWVIDVbmAG8w4p6Uw5/fJUrAny2dr/QDsyVqwp87n6ooev5eBVqttdalEOzs7AE2XU5TCb7mVuOH9X5B4XgZ1re0ceyIiIuoukwNBTU0NVCoVVCoVACA7OxsqlQo5OTkAgLi4OCxevNg4Pjo6Glu2bMHatWuRlZWF/fv347HHHsOkSZPg5+dnni5MFDbIHcP6u0DXKCDxcK4kNRAREVkSkwNBWloaIiIiEBERAQCIjY1FREQEVq5cCQAoLCw0hgMAWLp0Kd588028//77GD16NG677TaMGDECW7ZsMVMLppPJBDxwbTAAIOFADur0jZLVQkREZAlMPocgMjKyw139CQkJrR579NFH8eijj5r6VD1qfthAvPTDcZRr6rE5LRd3TwmWuiQiIiLJ9Pg5BJbK3k6GmX4GAMBHqVloaDRIXBEREZF0+mwgAIDJ/UV4OsuRd6kWPxwrlLocIiIiyfTpQOBgByydEgQAWLvnPAwGad71QEREJLU+HQgA4K5JAXBR2COjuBq7zpRIXQ4REZEk+nwgcHOS466rAwEAH+zJlOzaCERERFLq84EAAO6dFgIHexmO5lTicHaF1OUQERH1OgYCAD6ujrhtvD8AYG3KeYmrISIi6n0MBJc9cN1gyARgT0YpThZUSV0OERFRr2IguCzIyxk3jmm6lPKHKVkSV0NERNS7GAj+4MHpQwAAPx4rwIUyjcTVEBER9R4Ggj8Y6eeGGSP6wyA2Xb2QiIior2Ag+JOHIocCAL47kocSdZ3E1RAREfUOBoI/mRTiiQlB/VDfaMC6fdlSl0NERNQrGAja8FBk07kEXxy8iCqtXuJqiIiIeh4DQRtmhvpgxABXaOob8fnBC1KXQ0RE1OMYCNogCIJxL8GG/RdQW98ocUVEREQ9i4GgHTeO8UWApxPKNfX4Ji1X6nKIiIh6FANBO+ztZHjguqa9BB+nZkHfaJC4IiIiop7DQNCB28b7w9vFAfmVtfj+twKpyyEiIuoxDAQdcJTbYdm0EADA2j3nYTDw1shERGSbGAg68derg+CqsMe5khrsPFMidTlEREQ9wuRAkJqaiujoaPj5+UEQBGzdurXD8UuXLoUgCK0+Ro0a1d2ae5Wboxx/nRIEAPhgTyZEkXsJiIjI9pgcCDQaDcLDw7FmzZoujX/nnXdQWFho/MjNzYWnpyduu+02k4uVyj1Tg+FgL0N6TiUOZVdIXQ4REZHZ2Zu6QlRUFKKioro83t3dHe7u7sblrVu34tKlS7jnnnvaXUen00Gn0xmX1Wo1AECv10OvN8+VA5vn6cp8/RztcOs4PyQezsOaXecwPsDNLDWYmyk9WQv2ZPlsrR+APVkL9mTanJ0RxCvYBy4IApKSkhATE9PldaKjo6HT6bB9+/Z2x8THx2P16tWtHk9MTIRSqexOqVesrA54Md0OIgT8LawBAS6SlEFERGQSrVaLRYsWoaqqCm5u7f+H1uQ9BFeioKAA//3vf5GYmNjhuLi4OMTGxhqX1Wo1AgICMHfu3A6bMYVer0dycjLmzJkDuVzepXVUjcfw/bEinBIHYfkN4Wapw5y605OlY0+Wz9b6AdiTtWBPXdO8l70zvRoIPvvsM3h4eHS6R0GhUEChULR6XC6Xm32jmzLnwzOG4ftjRfjfyWLkVdUjxNvZrLWYS098n6TGniyfrfUDsCdrwZ46n6sreu1th6IoYv369bj77rvh4ODQW09rVlf5umFmqA8MIvBx6nmpyyEiIjKbXgsEKSkpyMzMxL333ttbT9kjHr5806PvjuSjWF0ncTVERETmYXIgqKmpgUqlgkqlAgBkZ2dDpVIhJycHQNPx/8WLF7dab926dZg8eTJGjx59ZRVLbEKwJyYG90N9owHr9mVLXQ4REZFZmBwI0tLSEBERgYiICABAbGwsIiIisHLlSgBAYWGhMRw0q6qqwnfffWf1eweaPRw5FADw5cGLqNLazttdiIio7zL5pMLIyMgOr9aXkJDQ6jF3d3dotVpTn8piRY7oj9CBrjhTVI3PDlzAY7OGSV0SERHRFeG9DLpBEAQ8dPlcgg37s6Gtb5C4IiIioivDQNBN88N8EeipxCWtHl//mit1OURERFeEgaCb7O1keOC6wQCAT1KzoG80SFwRERFR9zEQXIFbx/vD20WBgqo6bFMVSF0OERFRtzEQXAFHuR3unRYCAPgw5TwMBt4amYiIrBMDwRX669WBcHW0R2ZJDZJPF0tdDhERUbcwEFwhV0c57r46CADwwZ7zHb4lk4iIyFIxEJjBPVNDoLCX4bfcShzIKpe6HCIiIpMxEJhBf1cFbp8QAABYu4c3PSIiIuvDQGAmD1w3GHYyAXvPleGX82VSl0NERGQSBgIzCfBU4s5JTXsJ/pl0ArqGRokrIiIi6joGAjN6el4o+rsqkFWm4aEDIiKyKgwEZuTuJMeq6JEAgA92n8f50hqJKyIiIuoaBgIzmx/mi8gR/VHfaMA/ko7zbYhERGQVGAjMTBAEvHDzaDjKZTiYVYEtR/OlLomIiKhTDAQ9IMBTicdnDQcAvPjjKVRo6iWuiIiIqGMMBD3kvmtDMGKAKy5p9Xj5p9NSl0NERNQhBoIeIreT4aUFYQCAzUfycOA8r2BIRESWi4GgB40P6odFkwMBAP/YepzXJiAiIovFQNDDnp0XCm8XBbJKNfgoJUvqcoiIiNpkciBITU1FdHQ0/Pz8IAgCtm7d2uk6Op0O//jHPxAUFASFQoHg4GCsX7++O/VaHXelHCsvX5vg/d2ZyOK1CYiIyAKZHAg0Gg3Cw8OxZs2aLq9z++23Y+fOnVi3bh0yMjKwadMmjBgxwtSntlrRY3xx3fD+qG8w4J9bT/DaBEREZHHsTV0hKioKUVFRXR7/v//9DykpKcjKyoKnpycAIDg4uMN1dDoddDqdcVmtVgMA9Ho99Hq9qSW3qXkec83XmVXzR+CGrHL8cr4c36blIGasn9mfo7d76g3syfLZWj8Ae7IW7Mm0OTsjiFfw31VBEJCUlISYmJh2xzz88MM4e/YsJkyYgM8//xzOzs646aab8MILL8DJyanNdeLj47F69epWjycmJkKpVHa3XMkl5wv4IccOzvYi/jG2Ec5yqSsiIiJbp9VqsWjRIlRVVcHNza3dcSbvITBVVlYW9u3bB0dHRyQlJaGsrAwPP/wwysvLsWHDhjbXiYuLQ2xsrHFZrVYjICAAc+fO7bAZU+j1eiQnJ2POnDmQy3vnL/PsBgMy1h7AuRIN0sUgvHTDKLPOL0VPPY09WT5b6wdgT9aCPXVN8172zvR4IDAYDBAEAV9++SXc3d0BAG+++SZuvfVWfPDBB23uJVAoFFAoFK0el8vlZt/oPTFn+88FvLxgDG798AA2H8nHreMDMHmwVw88T+/11FvYk+WztX4A9mQt2FPnc3VFj7/t0NfXF4MGDTKGAQC46qqrIIoi8vLyevrpLc6EYE/cOanp2gR/T+K1CYiIyDL0eCCYOnUqCgoKUFPz+9vtzp49C5lMBn9//55+eov03PWh8HZxwPlSDT7mtQmIiMgCmBwIampqoFKpoFKpAADZ2dlQqVTIyckB0HT8f/HixcbxixYtgpeXF+655x6cOnUKqampePrpp7Fs2bJ2Tyq0de5KOZ6/senaBO/tzsSFMo3EFRERUV9nciBIS0tDREQEIiIiAACxsbGIiIjAypUrAQCFhYXGcAAALi4uSE5ORmVlJSZMmIC77roL0dHRePfdd83UgnW6KdwP1w7z5rUJiIjIIph8UmFkZGSHf7wSEhJaPRYaGork5GRTn8qmCYKAF2NGY+5bqdiXWYZtqgLERAySuiwiIuqjeC8DCQV5OePRmUMBAC/8cAqV2nqJKyIior6KgUBiD1w3BEN9XFCuqcer/zsjdTlERNRHMRBIzMFehpf+EgYA2HQ4F79eqJC4IiIi6osYCCzApBBP3DExAADw9y3HUd9gkLgiIiLqaxgILMRzUaHwcnbAuZIafLKX1yYgIqLexUBgITyUDvjnjVcBAN7deQ4Xy3ltAiIi6j0MBBYkZuwgTB3qBR2vTUBERL2MgcCCNF2bIAwO9jLsPVeG//xWIHVJRETURzAQWJgQb2c8OuP3axNUafUSV0RERH0BA4EFemD6YAzp74yymnq8wmsTEBFRL2AgsEAKe7s/XJsgB0cu8toERETUsxgILNTkwV64fULT7aHjthyHrqFR4oqIiMiWMRBYsLioq+Dp7ICzxTV4YOMR1NYzFBARUc9gILBg/Zwd8N6dEXCS2yHlbCmWrD+M6jqeZEhERObHQGDhpg71xuf3ToKrwh6HL1Rg0SeHUKHhXRGJiMi8GAiswIRgT2x64Gp4OjvgeH4VFn50AMXqOqnLIiIiG8JAYCVGD3LHN8unYKCbI86V1OC2Dw8gt0IrdVlERGQjGAisyFAfF2x+cAqCvJTIqdDi1g9/wbniaqnLIiIiG8BAYGUCPJXYvHwKhg9wQbFah4UfH8SJ/CqpyyIiIivHQGCFfNwc8fUDUzDG3x0Vmnrc+fFB/HqBFy8iIqLuMzkQpKamIjo6Gn5+fhAEAVu3bu1w/J49eyAIQquPoqKi7tZMaHpL4pf3TcakEE9U6xpw97pD2HuuTOqyiIjISpkcCDQaDcLDw7FmzRqT1svIyEBhYaHxw8fHx9Snpj9xdZRj47JJmDGiP+r0Biz/Mh2qckHqsoiIyArZm7pCVFQUoqKiTH4iHx8feHh4mLwedcxRboeP7p6AJ79W4cfjhUg4K0Noej4WTgqWujQiIrIiJgeC7ho7dix0Oh1Gjx6N+Ph4TJ06td2xOp0OOp3OuKxWqwEAer0eer15rtTXPI+55pOSAOCNW0fD0R74Lr0Qz245iRpdIxZfHSh1aVfMlrZTM1vrydb6AdiTtWBPps3ZGUEURbG7TyIIApKSkhATE9PumIyMDOzZswcTJkyATqfDp59+is8//xyHDh3CuHHj2lwnPj4eq1evbvV4YmIilEpld8u1eaIIJF2UIaWw6UjQ/IBGzBkkQuBRBCKiPkur1WLRokWoqqqCm5tbu+N6PBC0Zfr06QgMDMTnn3/e5tfb2kMQEBCAsrKyDpsxhV6vR3JyMubMmQO5XG6WOaWm1+uxfXsyzjoMxQepFwAA900LxjNzh0Gw0lRgq9vJlnqytX4A9mQt2FPXqNVqeHt7dxoIeu2QwR9NmjQJ+/bta/frCoUCCoWi1eNyudzsG70n5pSSIABPzhkOT1cnvPjjaXy67wK0egNeuHk07GTWGQoA29tOgO31ZGv9AOzJWrCnzufqCkmuQ6BSqeDr6yvFU/cZ9107GK8sCIMgAImHchD7jQr6RoPUZRERkYUyeQ9BTU0NMjMzjcvZ2dlQqVTw9PREYGAg4uLikJ+fj40bNwIA3n77bYSEhGDUqFGoq6vDp59+il27dmH79u3m64LadMekQDgr7PHk1ypsUxVAo2vE+4si4Ci3k7o0IiKyMCYHgrS0NMyYMcO4HBsbCwBYsmQJEhISUFhYiJycHOPX6+vr8dRTTyE/Px9KpRJjxozBjh07WsxBPSc63A/OCjs89MVR7DhdjGUJv+KTxRPgrJDkaBEREVkok/8qREZGoqPzEBMSElosP/PMM3jmmWdMLozMZ2boACTcMwn3ffYrfjlfjr+uO4SEpZPgrrStY25ERNR9vJdBHzFliBe+vP9quDvJkZ5TiYUf8/bJRET0OwaCPmRsgAe+Xn41vF0UOFNUjblvpeKT1Cw08GRDIqI+j4Ggjwkd6IYtD12DySGeqNU34v9+Oo2YD/bzFspERH0cA0EfFOilxKb7r8art4TBzdEeJ/LVuOn9ffi/H09BW98gdXlERCQBBoI+SiYTsHBiIHY8NR03jvGFQQQ+2ZuNuW+lIuVsqdTlERFRL2Mg6ON8XB3x/qJxWL90AgZ5OCHvUi2WrD+Mx79KR1mNrvMJiIjIJjAQEICmtyZuf/I63DstBDIB2KYqwOw3U7A5LbfDt5kSEZFtYCAgI2eFPZ6/cSS2rpiKkb5uqNTq8fS3x7Dok0PILtNIXR4REfUgBgJqZYy/B7Y9MhVxUaFwlMtwIKsc895OxZrdmbwfAhGRjWIgoDbJ7WRYPn0Itj8xHdcO80Z9gwGv/5yBG9/dh6M5l6Quj4iIzIyBgDoU6KXExmWT8PbCsfB0dkBGcTVuWfsLVm07geo6vdTlERGRmTAQUKcEQUBMxCDsiJ2OW8b5QxSBzw5cxJw3U7H9ZJHU5RERkRkwEFCXeTo74I3bw/HlfZMR5KVEkboOD3x+BA99cQTF6jqpyyMioivAQEAmmzrUGz8/cR0eihwCO5mA/54owuw3U/DFwYswGPgWRSIia8RAQN3iKLfDs9eH4odHpyE8wAPVdQ3459YTWLD2FxzLq5S6PCIiMhEDAV2Rq3ybbpa0KnokXBT2UOVW4uY1+xG35RgqNPVSl0dERF3EQEBXzE4m4J6pIdj11HQsiBgEUQQ2Hc5F5Ou7sfHABd5emYjICjAQkNn4uDnizYVj8e2DUzDS1w3qugas3HYSN763D4ezK6Quj4iIOsBAQGY3IdgT3z86DS/EjIa7kxxniqpx+0cH8MRX6Xw3AhGRhWIgoB5hJxNw99VB2P23SNw5KRCCAGxVFWDmv/fgo5TzqG/gYQQiIkticiBITU1FdHQ0/Pz8IAgCtm7d2uV19+/fD3t7e4wdO9bUpyUr5ensgJcXhOE/K6YhItADmvpGvPzfM7j+nVSkni2VujwiIrrM5ECg0WgQHh6ONWvWmLReZWUlFi9ejFmzZpn6lGQDwvzd8d2D1+D1W8fA28UBWaUaLF5/GMs/T0NuhVbq8oiI+jx7U1eIiopCVFSUyU/04IMPYtGiRbCzszNprwLZDplMwG0TAjBv9EC8nXwOnx24gJ9PFmNPRikejhyK5dMHw1FuJ3WZRER9ksmBoDs2bNiArKwsfPHFF3jxxRc7Ha/T6aDT6YzLarUaAKDX66HXm+eGOs3zmGs+S2AtPTnZAXHXD8MtEQPxwo9ncDD7Et7acRab03LwjxtCMSu0PwRBAGA9PZnC1nqytX4A9mQt2JNpc3ZGEEWx29eaFQQBSUlJiImJaXfMuXPnMG3aNOzduxfDhw9HfHw8tm7dCpVK1e468fHxWL16davHExMToVQqu1suWSBRBFTlArZelKGyvikEXOVhwIJgA3ycJC6OiMgGaLVaLFq0CFVVVXBzc2t3XI/uIWhsbMSiRYuwevVqDB8+vMvrxcXFITY21risVqsREBCAuXPndtiMKfR6PZKTkzFnzhzI5XKzzCk1a+1pPoAndA34MDUb6/ZfwOlKGV47bod7rgnC/VMD8EvKbqvrqSPWup3aY2v9AOzJWrCnrmney96ZHg0E1dXVSEtLQ3p6Oh555BEAgMFggCiKsLe3x/bt2zFz5sxW6ykUCigUilaPy+Vys2/0nphTatbYk4dcjuduGImFk4Kw+vuT2JNRio/3XsC23woxs7+AuTI7q+upM9a4nTpia/0A7MlasKfO5+qKHg0Ebm5uOH78eIvHPvjgA+zatQvffvstQkJCevLpyQqFeDtjw9KJ2Hm6BP/64RRyKrTYpLbD/nf349GZw/CXiEGwt+PlM4iIzM3kQFBTU4PMzEzjcnZ2NlQqFTw9PREYGIi4uDjk5+dj48aNkMlkGD16dIv1fXx84Ojo2OpxomaCIGD2yAGYNswbG/ZlYc3ODORU1OLpb4/h/d2ZeGTGUAYDIiIzM/k3alpaGiIiIhAREQEAiI2NRUREBFauXAkAKCwsRE5OjnmrpD7JUW6H+6YFY+W4Rjwzbxg8nR1wsVyLp789hllvpmBzWi5vnEREZCYm7yGIjIxER29MSEhI6HD9+Ph4xMfHm/q01Icp7ID7p4Vg6dTB+PzARXyUmmUMBtxjQERkHvwNSlZD6WCP5dOHYN+zMxAXFco9BkREZsRAQFaHwYCIyPwYCMhqMRgQEZkPAwFZveZgsPcZBgMiou5iICCb4axgMCAi6i4GArI5DAZERKZjICCb1VEwmPlGCjYdzkF9A4MBERHAQEB9QFvBIKdCi7gtxzH99d347JcLqNM3Sl0mEZGkGAioz2gOBvuenYF/zr8KPq4KFFbVYdV/TuLa13bjk9QsaHQNUpdJRCQJBgLqc5QO9rjv2sFIfWYGXogZjUEeTiit1uH/fjqNaa/uwvu7zkFdp5e6TCKiXsVAQH2Wo9wOd18dhN1/i8Rrt4xBkJcSl7R6/Hv7WUx9ZRfe3J6BS5p6qcskIuoVDATU5znYy3D7xADsjJ2Od+4Yi2E+Lqiua8C7uzIx9dVdePmn0yiprpO6TCKiHsVAQHSZvZ0MN48dhJ+fuA5r7xqHkb5u0NY34qPULFz76m7E/+ckCqtqpS6TiKhHMBAQ/YlMJiAqzBc/PjYN65dOwNgAD+gaDEj45QKue2034rYcR065VuoyiYjMyuTbHxP1FYIgYGboAMwY4YP9meV4b9c5HMquwKbDOfgmLRc3j/XDw5FDMdTHRepSiYiuGAMBUScEQcC0Yd6YNswbh7Mr8P7uTKSeLcWWo/lISs/H/DBfrJgxFFf5ukldKhFRtzEQEJlgUognNoZMwm+5lXhvVyZ2nC7GD8cK8cOxQswK9cG900IwZYgXBEGQulQiIpMwEBB1Q3iABz5dMgGnCtRYsycTPx0vxM4zJdh5pgShA12xbGoIbhrrB0e5ndSlEhF1CU8qJLoCI/3csGbROOyInY67rw6Ck9wOZ4qq8cx3x3DNK7vwxvYMlKj5lkUisnwMBERmMKS/C16IGY2DcbMQFxWKQR5OqNDU473L1zJ48msVjuVVSl0mEVG7TA4EqampiI6Ohp+fHwRBwNatWzscv2/fPkydOhVeXl5wcnJCaGgo3nrrre7WS2TR3JVyLJ8+BClPR+KDu8ZhQlA/6BtFJKXn46b39+PWtb/gvyeK0ChKXSkRUUsmn0Og0WgQHh6OZcuWYcGCBZ2Od3Z2xiOPPIIxY8bA2dkZ+/btw/Lly+Hs7IwHHnigW0UTWTp7OxluCPPFDWG+OJZXiQ37L+CHYwVIu3gJaRcvoZ+DHYrcs3HX5BC4K+VSl0tEZHogiIqKQlRUVJfHR0REICIiwrgcHByMLVu2YO/evQwE1CeM8ffAWwvH4rmoUHxx8CK+OHgRl7R6vPbzOby3Kwu3jvfH0qnBGNKf1zMgIun0+rsM0tPT8csvv+DFF19sd4xOp4NOpzMuq9VqAIBer4deb5670DXPY675LAF7smyeTnZ4bMZg3DPZD69/sxtHqt1wtkSDzw9exOcHL2L6MG8suSYQ06zsbYu2tI2asSfrwJ5Mm7MzgiiK3T6aKQgCkpKSEBMT0+lYf39/lJaWoqGhAfHx8Xj++efbHRsfH4/Vq1e3ejwxMRFKpbK75RJZFFEEzqkFpBQKOHlJgIimEDDQScR0XwMmeItw4LsWiegKabVaLFq0CFVVVXBza/8Car0WCLKzs1FTU4ODBw/iueeew/vvv48777yzzbFt7SEICAhAWVlZh82YQq/XIzk5GXPmzIFcbhvHcNmTdWirp4vlWmw8mIPvjuZDU98IAPBwkuO28YNw67hBGNzfWcqSO9RXtpG1Y0/WoSd6UqvV8Pb27jQQ9Nohg5CQEABAWFgYiouLER8f324gUCgUUCgUrR6Xy+Vm3+g9MafU2JN1+GNPQwe6418xYfjb9aHYnJaHhF+ykVtRi0/2XcAn+y5gXKAHbpsQgBvH+MLV0TK/D7a+jWwFe7IO5uypq/NIcqVCg8HQYg8AETVxc5Tj3mkhWHpNMHacLsbXv+ZiT0YJjuZU4mhOJVZ/fxJRo31x2wR/XB3iBZnMes41ICLLZnIgqKmpQWZmpnE5OzsbKpUKnp6eCAwMRFxcHPLz87Fx40YAwJo1axAYGIjQ0FAATdcx+Pe//43HHnvMTC0Q2R47mYB5owZi3qiBKFHXYUt6Pjan5eJ8qQZJ6U03VfLv54Rbx/vjlnH+CPDkuTVEdGVMDgRpaWmYMWOGcTk2NhYAsGTJEiQkJKCwsBA5OTnGrxsMBsTFxSE7Oxv29vYYMmQIXn31VSxfvtwM5RPZPh83Rzw4fQiWXzcY6bmV2JyWhx9+K0DepVq8veMc3t5xDtcM8cJtE/xx/ShfOPFMRCLqBpMDQWRkJDo6DzEhIaHF8qOPPopHH33U5MKIqCVBEDAusB/GBfbDyhtH4ueTRdh8JBf7M8vxy/mmj5WKk7gx3Be3jg/AuEAPq3r7IhFJi3c7JLJCTg52iIkYhJiIQci7pMV3R/Lx7dFc5FbUYtPhXGw6nIsh/Z1x6/gALBg3CAPcHKUumYgsHAMBkZXz76fE47OH4dGZQ3EouwKbj+Tiv8eLcL5Ug1f/dwav/3wG04f3x20TAjDrKh8o7HlIgYhaYyAgshEymYApQ7wwZYgXVt+kx0/HC7E5LQ9pFy9hd0YpdmeUwkMpx03hfpgzcgAmhXgyHBCREQMBkQ1ydZRj4cRALJwYiKzSGnx7JA/fHc1DsVqHjQcuYuOBi3B2sMO0Yd6YFToAkaH94ePKwwpEfRkDAZGNG9zfBc9cH4qn5o7A3nOl+O/xIuzKKEFptQ4/nyzGzyeLAQBj/N0xY4QPZl3lg9F+7rzGAVEfw0BA1EfYyQREjvBB5AgfGAwiThaosfNMMXafKcFveVU4dvnjnZ3n0N9VgRkj+mNm6ABMG+YNFwV/VRDZOr7KifogmUxAmL87wvzd8cTs4SiprsOejFLsOl2CvedKUVqtwzdpefgmLQ9yOwFXD/Yy7j0I8rLc+yoQUfcxEBARfFwdcfuEANw+IQC6hkb8mn0JO88UY9eZElws12LvuTLsPVeGf/1wCkP6O2NmqA9mhg7AhOB+UpdORGbCQEBELSjsm042nDbMGytvHImsMg12nynBztMl+PVCBc6XanC+NBuf7M2Gq6M9rh3qhX61AsIraxHc37ZuMEPUlzAQEFG7BEHAkP4uGNLfBfddOxjqOj32ni3DrjMl2JNRgnJNPX46UQzADl++sReDPJwwMbgfJoV4YVJIPwzp78KrJRJZCQYCIuoyN0c55o/xxfwxvmg0iPgtrxI7ThbhxyPnkaeVIb+yFvmqWmxVFQAAvJwdMCG4HyYGe2JyiBeu8nWFvZ1M4i6IqC0MBETULXaypnsrhPm6YET9WUyfNQsnCjU4fKECh7PLkZ5TiXJNfYu3Njo72GF8sCcmXd6LMMbfHY5yXhyJyBIwEBCRWTgr7I3nHgBAfYMBx/MrcTj7En69UIFfL1Sguq4BqWdLkXq2FADgYCdDeIA7JoV4YmKwJ8YH9YOrI89DIJICAwER9QgHexnGB3lifJAnHsIQNBpEZBRV43B2OX69cAmHsitQVqPDrxcu4dcLlwCch0wARvq5YWKwJyYFe2LEQFcEeip5mIGoFzAQEFGvsJMJGOnnhpF+blg6NQSiKOJCuRaHs8uNexFyKrQ4ka/GiXw1Nuy/AKBpL0KwtxJDfVwwtL8Lhvi4GE90dHLg4QYic2EgICJJCIKAEG9nhHg7Y+HEQABAUVVdi3MQzpfWoE5vwNniGpwtrvnT+sAgD6cWQaH5837ODlK0RGTVGAiIyGIMdHfETeF+uCncDwBgMIjIr6xFZmkNzpfUILP5o7QGlVo98i7VIu9SLfZklLaYx8vZwRgQhvS/HBR8XODn7si3QRK1g4GAiCyWTCYgwFOJAE8lZozwafG18hqdMRxkltQ0XTCppAb5lbUo19SjPLsCh7MrWqzjJLeDp7MDXB3t4aKwh6ujPVwd5XBxvPy5omnZyV7A2QoBnlkV6OfiaBzr4mjPW0aTzWIgICKr5OWigJeLApMHe7V4XKNrQFapBpml1U1BoUSDzNIaXCjToFbfiPzK2i4+gx0+zUhr9aiDvQxuxkAhh4vCHgPdHRHoqWz68Gr618dVwb0RZFUYCIjIpjgr7I03bvojfaMBeZdqUVWrR3WdHjV1Daiua0C1rqHFco2uAVW19cgrKoOdkws0ukZU1+mhqW8E0PR2yrKaepTV1HdYh6NchoB+TeEg4HJYCPL6fZnXXyBLY3IgSE1Nxeuvv44jR46gsLAQSUlJiImJaXf8li1bsHbtWqhUKuh0OowaNQrx8fGYN2/eldRNRGQSuZ0MId5du1OjXq/HTz/9hBtumAq5vOm6CI0GETW6psDwxwChrtOjoLIOORUa5FRokVOhRUFlHer0BpwrqcG5kpo2n8PHVdFij8IfP/pz7wJJwORAoNFoEB4ejmXLlmHBggWdjk9NTcWcOXPw0ksvwcPDAxs2bEB0dDQOHTqEiIiIbhVNRNTb7GQC3J3kcHeSA3DqcKy+0YCCylpjQMip0CK3QouL5VrklGtRrWtASbUOJdU6pF281Gp9R7kMgzyc4OvuBF93R/i6O2KguxN8PZo+93VzgpuTPUMDmZXJgSAqKgpRUVFdHv/222+3WH7ppZewbds2fP/99wwERGST5HYyBHk5I8ir9R4JURRRVas3BoWL5U1h4fe9C7Wo0xsu31VS0+5zKB3sMLA5LLg5wc/D0bjcHCTcneQMDdRlvX4OgcFgQHV1NTw9Pdsdo9PpoNPpjMtqtRpA0248vV5vljqa5zHXfJaAPVkHW+vJ1voBer4nZ7mAqwY446oBrQND096FOhRU1aKoSocidR0Kq+pQpK4zLl/S6qGtb0RWqQZZHYQGR7kMA92aQoKPiwMcqgVE1umg7JGueh9/9kybszOCKIpid59EEIROzyH4s9deew2vvPIKzpw5Ax8fnzbHxMfHY/Xq1a0eT0xMhFJpKz/KRETdU98IVNUDlfUCKuuBynrgkk4wPnZJB2ga2t4zMNBJxK0hBgxz7/avfrIyWq0WixYtQlVVFdzc3Nod16uBIDExEffffz+2bduG2bNntzuurT0EAQEBKCsr67AZU+j1eiQnJ2POnDnGk4asHXuyDrbWk631A9hGTzp9I4qqdSiqqkNRVR2yy2qQsD/bGBRuDBuI564fjgFujhJX2n22sJ3+rCd6UqvV8Pb27jQQ9Nohg6+++gr33XcfNm/e3GEYAACFQgGFQtHqcblcbvaN3hNzSo09WQdb68nW+gGsuye5XI6hSkcMHdD09ku9Xg9/bSZOCCHY9GsufjhehN0ZpXhyznAsuSYYciu+gZQ1b6f2mLOnrs7TKz8BmzZtwj333INNmzZh/vz5vfGURET0J0p7ID76KvznkWmICPSApr4RL/54GvPf3YuDWeVSl0cSMzkQ1NTUQKVSQaVSAQCys7OhUqmQk5MDAIiLi8PixYuN4xMTE7F48WK88cYbmDx5MoqKilBUVISqqirzdEBERCYZPcgd3z14DV67ZQw8nR1wtrgGd3x8EI9/lY5idZ3U5ZFETA4EaWlpiIiIML5lMDY2FhEREVi5ciUAoLCw0BgOAODjjz9GQ0MDVqxYAV9fX+PH448/bqYWiIjIVDKZgNsnBmDXU9Px16sDIQjANlUBZr2Rgk/3ZkHfaJC6ROplJp9DEBkZiY7OQ0xISGixvGfPHlOfgoiIeomH0gEvxoRh4YRAPL/tBFS5lXjxx9P4Ji0X/7p5NK7+070iyHZZ71kkRERkNmH+7tjy0DV49ZYw9FPKjYcRnvgqHSU8jNAnMBAQERGApsMICycGYvffInHX5KbDCFtVBZj5RgrW7ctGAw8j2DQGAiIiasFD6YD/+0sYtq2YinB/d9ToGvDCD6cw/919OMR3I9gsBgIiImrTGH8PJD08FS8vCIOHUo6M4mos/PggnvxahZJqHkawNQwERETULplMwJ2TArH7qUjcOanpMEJSej5m/TsF63kYwaYwEBARUaf6OTvg5QVhSHp4Ksb4u6Na14B//XAK17+zF5/uzeIeAxvAQEBERF02NqDpMMJLf2k6jJBZUoMXfzyNq1/aiSXrD2ObKh+19Y1Sl0nd0Ou3PyYiIutmJxOwaHIg5of54j+/5eO7o/lQ5VYi5WwpUs6WwtnBDteP9sWCcYNw9WAv2MnavvMiWRYGAiIi6hZ3pRx3TwnG3VOCkVVag63p+UhS5SO3ohbfHc3Dd0fzMNDNETdH+GFBhD9GDHSVumTqAAMBERFdscH9XRA7dwSenDMcaRcvYcvRfPx4rABF6jp8lJKFj1KyMMrPDX+JGISbxvrBx9V6b7tsqxgIiIjIbARBwMRgT0wM9sSq6JHYfaYEW9LzsSejBCcL1DhZoMZLP53GtcP6Y8G4QZg7ciCcHOykLpvAQEBERD3EUW6HqDBfRIX54pKmHj8cK8CW9Hyk57Q83yAqzBcLIprON5DxfAPJMBAQEVGP6+fsYDzfILtMg6T0fCSl5yG3ohbfHsnDt0fy4OvuiJvHDsJN4X4YMdCVJyP2MgYCIiLqVSHezoidMxxPzh6GtIuXkJSejx9+K0BhVR0+TDmPD1POw0luh1BfV4z0dcMoP3eM9HPDiAGuPLzQgxgIiIhIEm2db/Dd0XzsPVeKWn0j0nMqkZ5TaRwvE5pOXhzl54aRvm4Y7uOMGr109dsaBgIiIpKcwr7p2gXXj/ZFo0FEdpkGpwrVOFWgvvxvFcpq6pFZUoPMkhpsUxVcXtMe72akYNQgd4z0dcPIy2Eh0FPJ8xFMxEBAREQWxU4mYKiPC4b6uOCmcD8AgCiKKK3W4eQfQsLJ/CpcLNeguFqH4jMl2HWmxDiHi8IeV10+5DDSzw2hA93g6+4IT2cH2NvxIr1tYSAgIiKLJwgCfNwc4ePmiBkjfAAAer0eW77/CcHh1+BsSdMehZMFapwpqkaNrgG/XriEXy9c+tM8QD+lA/q7KODt6gBvF8XlzxVNn7sq4O3S9HVzhQdRFFGnN6Cyth6VWj0qtXpUNX9e23L5kkYHh1oZbrjiZzUdAwEREVktRztgXKAHJg/pb3ysodGArDLN73sSCqpwtrgG5TU6GESgQlOPCk09Moo7nlsQAE+lQ4ug8PvnTYFBW99o/ENfVatHpfaPf+h//7y+oet3hQxxleZQBwMBERHZFHs7GYYPcMXwAa6IiRhkfLzRIOKSth5lNTqUVutQVqNDWXU9Smt0KKvWodT4eD0qNE3hoVxTj3JNPTKKq6+8LpkAD6Uc7k5yeCgd4OEkh7tSDg8nB/RTyuGhlMPFQYYLp9Kv+Lm6VZ+pK6SmpuL111/HkSNHUFhYiKSkJMTExLQ7vrCwEE899RTS0tKQmZmJxx57DG+//fYVlExERGQ6O5kAb5em/92HDux4bKNBRIWmKTy0CBA19cbPL2nroXSwh4dT0x9zD6XD5T/2TX/kf//j3/Q1Zwc7CELH//vX6/X4Kc9KAoFGo0F4eDiWLVuGBQsWdDpep9Ohf//++Oc//4m33nqrW0USERH1JjuZgP6uTYcH+gqTA0FUVBSioqK6PD44OBjvvPMOAGD9+vWmPh0RERH1Aos8h0Cn00Gn0xmX1Wo1gKZdKXq9ea5C0TyPueazBOzJOthaT7bWD8CerAV7Mm3OzgiiKIrdfRJBEDo9h+CPIiMjMXbs2E7PIYiPj8fq1atbPZ6YmAilUtmNSomIiPomrVaLRYsWoaqqCm5ubu2Os8g9BHFxcYiNjTUuq9VqBAQEYO7cuR02Ywq9Xo/k5GTMmTMHcrncLHNKjT1ZB1vrydb6AdiTtWBPXdO8l70zFhkIFAoFFIrWJ3LI5XKzb/SemFNq7Mk62FpPttYPwJ6sBXvqfK6u4PUbiYiIyPQ9BDU1NcjMzDQuZ2dnQ6VSwdPTE4GBgYiLi0N+fj42btxoHKNSqYzrlpaWQqVSwcHBASNHjrzyDoiIiOiKmRwI0tLSMGPGDONy87H+JUuWICEhAYWFhcjJyWmxTkREhPHzI0eOIDExEUFBQbhw4UKXnrP5vMeuHgfpCr1eD61WC7VabTO7mtiTdbC1nmytH4A9WQv21DXNfzs7ew/BFb3LoLfk5eUhICBA6jKIiIisVm5uLvz9/dv9ulUEAoPBgIKCAri6unZ62ceuan7nQm5urtneuSA19mQdbK0nW+sHYE/Wgj11jSiKqK6uhp+fH2Sy9k8dtMh3GfyZTCbrMNVcCTc3N5v5QWrGnqyDrfVka/0A7MlasKfOubu7dzqG7zIgIiIiBgIiIiLqw4FAoVBg1apVbV4AyVqxJ+tgaz3ZWj8Ae7IW7Mm8rOKkQiIiIupZfXYPAREREf2OgYCIiIgYCIiIiIiBgIiIiMBAQERERLDxQLBmzRoEBwfD0dERkydPxuHDhzscv3nzZoSGhsLR0RFhYWH46aefeqnSzr388suYOHEiXF1d4ePjg5iYGGRkZHS4TkJCAgRBaPHh6OjYSxV3Lj4+vlV9oaGhHa5jydsIAIKDg1v1JAgCVqxY0eZ4S9xGqampiI6Ohp+fHwRBwNatW1t8XRRFrFy5Er6+vnBycsLs2bNx7ty5Tuc19fVoLh31o9fr8eyzzyIsLAzOzs7w8/PD4sWLUVBQ0OGc3fnZNafOttHSpUtb1Xf99dd3Oq9U2wjovKe2XleCIOD1119vd04pt1NXfmfX1dVhxYoV8PLygouLC2655RYUFxd3OG93X39dYbOB4Ouvv0ZsbCxWrVqFo0ePIjw8HPPmzUNJSUmb43/55RfceeeduPfee5Geno6YmBjExMTgxIkTvVx521JSUrBixQocPHgQycnJ0Ov1mDt3LjQaTYfrubm5obCw0Phx8eLFXqq4a0aNGtWivn379rU71tK3EQD8+uuvLfpJTk4GANx2223trmNp20ij0SA8PBxr1qxp8+uvvfYa3n33XXz44Yc4dOgQnJ2dMW/ePNTV1bU7p6mvR3PqqB+tVoujR4/i+eefx9GjR7FlyxZkZGTgpptu6nReU352za2zbQQA119/fYv6Nm3a1OGcUm4joPOe/thLYWEh1q9fD0EQcMstt3Q4r1TbqSu/s5988kl8//332Lx5M1JSUlBQUIAFCxZ0OG93Xn9dJtqoSZMmiStWrDAuNzY2in5+fuLLL7/c5vjbb79dnD9/fovHJk+eLC5fvrxH6+yukpISEYCYkpLS7pgNGzaI7u7uvVeUiVatWiWGh4d3eby1bSNRFMXHH39cHDJkiGgwGNr8uqVvIwBiUlKScdlgMIgDBw4UX3/9deNjlZWVokKhEDdt2tTuPKa+HnvKn/tpy+HDh0UA4sWLF9sdY+rPbk9qq6clS5aIN998s0nzWMo2EsWubaebb75ZnDlzZodjLGk7/fl3dmVlpSiXy8XNmzcbx5w+fVoEIB44cKDNObr7+usqm9xDUF9fjyNHjmD27NnGx2QyGWbPno0DBw60uc6BAwdajAeAefPmtTtealVVVQAAT0/PDsfV1NQgKCgIAQEBuPnmm3Hy5MneKK/Lzp07Bz8/PwwePBh33XUXcnJy2h1rbduovr4eX3zxBZYtW9bhXTotfRv9UXZ2NoqKilpsB3d3d0yePLnd7dCd16OUqqqqIAgCPDw8Ohxnys+uFPbs2QMfHx+MGDECDz30EMrLy9sda23bqLi4GD/++CPuvffeTsdaynb68+/sI0eOQK/Xt/ieh4aGIjAwsN3veXdef6awyUBQVlaGxsZGDBgwoMXjAwYMQFFRUZvrFBUVmTReSgaDAU888QSmTp2K0aNHtztuxIgRWL9+PbZt24YvvvgCBoMB11xzDfLy8nqx2vZNnjwZCQkJ+N///oe1a9ciOzsb1157Laqrq9scb03bCAC2bt2KyspKLF26tN0xlr6N/qz5e23KdujO61EqdXV1ePbZZ3HnnXd2eKc5U392e9v111+PjRs3YufOnXj11VeRkpKCqKgoNDY2tjnemrYRAHz22WdwdXXtdPe6pWyntn5nFxUVwcHBoVXw7OzvVPOYrq5jCqu4/TG1tGLFCpw4caLTY2FTpkzBlClTjMvXXHMNrrrqKnz00Ud44YUXerrMTkVFRRk/HzNmDCZPnoygoCB88803XUr+lm7dunWIioqCn59fu2MsfRv1JXq9HrfffjtEUcTatWs7HGvpP7t33HGH8fOwsDCMGTMGQ4YMwZ49ezBr1iwJKzOP9evX46677ur0BFxL2U5d/Z0tNZvcQ+Dt7Q07O7tWZ2sWFxdj4MCBba4zcOBAk8ZL5ZFHHsEPP/yA3bt3w9/f36R15XI5IiIikJmZ2UPVXRkPDw8MHz683fqsZRsBwMWLF7Fjxw7cd999Jq1n6duo+XttynbozuuxtzWHgYsXLyI5Odnk+9B39rMrtcGDB8Pb27vd+qxhGzXbu3cvMjIyTH5tAdJsp/Z+Zw8cOBD19fWorKxsMb6zv1PNY7q6jilsMhA4ODhg/Pjx2Llzp/Exg8GAnTt3tvjf2B9NmTKlxXgASE5Obnd8bxNFEY888giSkpKwa9cuhISEmDxHY2Mjjh8/Dl9f3x6o8MrV1NTg/Pnz7dZn6dvojzZs2AAfHx/Mnz/fpPUsfRuFhIRg4MCBLbaDWq3GoUOH2t0O3Xk99qbmMHDu3Dns2LEDXl5eJs/R2c+u1PLy8lBeXt5ufZa+jf5o3bp1GD9+PMLDw01etze3U2e/s8ePHw+5XN7ie56RkYGcnJx2v+fdef2ZWrRN+uqrr0SFQiEmJCSIp06dEh944AHRw8NDLCoqEkVRFO+++27xueeeM47fv3+/aG9vL/773/8WT58+La5atUqUy+Xi8ePHpWqhhYceekh0d3cX9+zZIxYWFho/tFqtccyfe1q9erX4888/i+fPnxePHDki3nHHHaKjo6N48uRJKVpo5amnnhL37NkjZmdni/v37xdnz54tent7iyUlJaIoWt82atbY2CgGBgaKzz77bKuvWcM2qq6uFtPT08X09HQRgPjmm2+K6enpxrPuX3nlFdHDw0Pctm2beOzYMfHmm28WQ0JCxNraWuMcM2fOFN977z3jcmevR6n6qa+vF2+66SbR399fVKlULV5bOp2u3X46+9mVsqfq6mrxb3/7m3jgwAExOztb3LFjhzhu3Dhx2LBhYl1dXbs9SbmNOuupWVVVlahUKsW1a9e2OYclbaeu/M5+8MEHxcDAQHHXrl1iWlqaOGXKFHHKlCkt5hkxYoS4ZcsW43JXXn/dZbOBQBRF8b333hMDAwNFBwcHcdKkSeLBgweNX5s+fbq4ZMmSFuO/+eYbcfjw4aKDg4M4atQo8ccff+zlitsHoM2PDRs2GMf8uacnnnjC2P+AAQPEG264QTx69GjvF9+OhQsXir6+vqKDg4M4aNAgceHChWJmZqbx69a2jZr9/PPPIgAxIyOj1desYRvt3r27zZ+15roNBoP4/PPPiwMGDBAVCoU4a9asVr0GBQWJq1atavFYR69HqfrJzs5u97W1e/fudvvp7GdXyp60Wq04d+5csX///qJcLheDgoLE+++/v9UfdkvaRp311Oyjjz4SnZycxMrKyjbnsKTt1JXf2bW1teLDDz8s9uvXT1QqleJf/vIXsbCwsNU8f1ynK6+/7hIuPyERERH1YTZ5DgERERGZhoGAiIiIGAiIiIiIgYCIiIjAQEBERERgICAiIiIwEBAREREYCIiIiAgMBERERAQGAiIiIgIDAREREQH4f1lrJlVRPmKHAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Evaluation loss\n",
    "pd.Series(training_objective.evaluations_history[\"eval\"][\"loss\"]).plot(figsize=(6, 3), grid=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0df7f5b8-0370-4fbd-a507-ffbd0c4739cc",
   "metadata": {
    "tags": [],
    "id": "0df7f5b8-0370-4fbd-a507-ffbd0c4739cc",
    "outputId": "6a3245f2-f150-48ab-fe61-79f0956b4350"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAAESCAYAAACCZc2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA3c0lEQVR4nO3deVyVdf7//8d1DoeDKAcFWQUEc8EVcSeztFzGirK9rLQsm8oWc6b55cxk+plmbJpPTeMn03ZrSitLbZpWxjUVU1DM3BcSkU1QOOwcOOf3h0njV01Q4Bzgeb/duOm5uJbX4dWRZ9fyfhsul8uFiIiItGomdxcgIiIi7qdAICIiIgoEIiIiokAgIiIiKBCIiIgICgQiIiKCAoGIiIgAXu4uoC6cTidZWVn4+flhGIa7yxEREWk2XC4XxcXFhIeHYzKd+zxAswgEWVlZREZGursMERGRZuvIkSNERESc8/vNIhD4+fkBJ9+MzWZrkH06HA6++eYbxo4di8ViaZB9ysVRTzyL+uF51BPP0xx6YrfbiYyMrP1dei7NIhCcukxgs9kaNBD4+vpis9k8tomtjXriWdQPz6OeeJ7m1JPzXXLXTYUiIiKiQCAiIiIKBCIiIoICgYiIiKBAICIiIlxAIFi3bh2JiYmEh4djGAYrVqw47zbvv/8+cXFx+Pr6EhYWxpQpUygoKLiQekVERKQR1DsQlJaWEhcXx/z58+u0/oYNG5g0aRL33XcfO3fuZOnSpWzevJmpU6fWu9iG9NXOXI6WgqPG6dY6REREPEG9xyEYP34848ePr/P6ycnJREdH89hjjwEQExPDr3/9a/7617/W99ANpqyqmsc/3I7T5cVLu1bRM8xG3042+nbyp08nf7qH+GEx62qKiIi0Ho0+MFFCQgK///3v+eKLLxg/fjx5eXl8/PHHXH311efcprKyksrKytrXdrsdODkAhMPhuOiajhWVMyiqPd9nnqCi2sn2I4VsP1JY+32L2SA21I/e4Tb6hNsY3LkDXYLaXvRx5Zed6m1D9FgunvrhedQTz9McelLX2gyXy+W60IMYhsHy5cuZMGHCL663dOlSpkyZQkVFBdXV1SQmJvLJJ5+cc1Sn2bNnM2fOnDOWL168GF9f3wst9wxOFxRUwJFS4+RXCWSWGpTXnDmaU3Q7FwkhTuIDXVjNDVaCiIhIoyorK2PixIkUFRX94mi/jR4Idu3axejRo3niiScYN24c2dnZPPnkkwwePJg333zzrNuc7QxBZGQk+fn5DTp0cVJSEmPGjDktmLhcLjJOlLMry84PWXZ2HLWz+ccT1DhP/pjaWs1c2zeM2wZ1ok+4TbMvNqBz9UTcQ/3wPOqJ52kOPbHb7XTs2PG8gaDRLxnMnTuX4cOH8+STTwLQr18/2rZty4gRI3j22WcJCws7Yxur1YrVaj1jucViafAf+Nn22TXEm64h/lwXf/J1XnEFn6Qe5cMtGfxYUMaHKZl8mJJJzzAbtw+OZEL/Tvj7euZ/CM1RY/RZLpz64XnUE8/jyT2pa12NfudcWVnZGfMvm80nz7lfxMmJJhXs58NDIy9h9W9HsmTqMCb0D8fby8TubDvP/GsnQ/7yH574MI3vMwvdXaqIiMgFqXcgKCkpIS0tjbS0NADS09NJS0sjIyMDgJkzZzJp0qTa9RMTE1m2bBkLFizg0KFDbNiwgccee4whQ4YQHh7eMO+iiRiGQcIlgbx0ezybf38VsxN7ERvqR2W1k+XbjnLzwmQOHitxd5kiIiL1Vu9AkJKSQnx8PPHxJ8+nz5gxg/j4eGbNmgVAdnZ2bTgAuOeee3jxxRd5+eWX6dOnD7fccgs9evRg2bJlDfQW3KO9rzf3DI/hy8dHsGLacPpHtqeq2sk7G390d2kiIiL1Vu97CEaOHPmLp/oXLVp0xrJHH32URx99tL6HahYMw6B/ZHt+N64HE9/4jo9TM/ntuB7YfDzzWpKIiMjZaPSdBpJwSSDdQ9pRVlXD0pRMd5cjIiJSLwoEDcQwDCZfGg3AOxt/rH1MUUREpDlQIGhAN8R3wubjRcbxMtbszXN3OSIiInWmQNCAfL29uH1IFACLdHOhiIg0IwoEDezuYZ0xGfDt/nwO5BW7uxwREZE6USBoYJEBvozuGQLAOxsPu7kaERGRulEgaAT3DI8G4JOtmRSVe+4MWCIiIqcoEDSChC6B9Ajx++kRxCPuLkdEROS8FAgagWEYtWcJ3k0+rEcQRUTE4ykQNJIJ/Tvh38ZCxvEyVu/RI4giIuLZFAgaSRtvM7cPiQT0CKKIiHg+BYJGdOoRxPUH8tmfq0cQRUTEcykQNKKIDr6M7RUK6CyBiIh4NgWCRnbq5sJlW49SVKZHEEVExDMpEDSyoTEBxIb6Ue6o4SM9gigiIh5KgaCRGYbBvT+dJXgnWbMgioiIZ1IgaALX9+9Ee18LmSfKWbk7193liIiInEGBoAn4WMzcoVkQRUTEgykQNJG7fnoEcePBAvbm6BFEERHxLAoETaRT+zaM661HEEVExDMpEDShey6NBmD5tkwKy6rcW4yIiMh/USBoQkN+egSxwuHkX9uz3F2OiIhILQWCJmQYBrcOOjm/wcepmW6uRkRE5GcKBE3s+v7heJkMvs8s0s2FIiLiMRQImlhgOytX9QwG4JOtOksgIiKeQYHADW4eePKywbKtR3HUON1cjYiIiAKBW4zsEURgW2/ySypZt++Yu8sRERGpfyBYt24diYmJhIeHYxgGK1asOO82lZWV/OEPf6Bz585YrVaio6N56623LqTeFsFiNjEhvhOgmwtFRMQz1DsQlJaWEhcXx/z58+u8za233srKlSt588032bt3L0uWLKFHjx71PXSLcvPACAD+szuXE6Uak0BERNzLq74bjB8/nvHjx9d5/a+++oq1a9dy6NAhAgICAIiOjv7FbSorK6msrKx9bbfbAXA4HDgcjvqWfFan9tNQ+6uvrh3b0Dvcj51ZxSzfeoS7h0W5pQ5P4u6eyOnUD8+jnnie5tCTutZmuFyuC56P1zAMli9fzoQJE865zsMPP8y+ffsYNGgQ//znP2nbti3XXXcdf/rTn2jTps1Zt5k9ezZz5sw5Y/nixYvx9fW90HI9zrpsg09+NBPR1sWT/WrcXY6IiLRAZWVlTJw4kaKiImw22znXq/cZgvo6dOgQ69evx8fHh+XLl5Ofn8/DDz9MQUEBb7/99lm3mTlzJjNmzKh9bbfbiYyMZOzYsb/4ZurD4XCQlJTEmDFjsFgsDbLP+hpWWsW//raWzFLoMmAEsaF+bqnDU3hCT+Rn6ofnUU88T3Poyamz7OfT6IHA6XRiGAbvv/8+/v7+ALz44ovcfPPNvPLKK2c9S2C1WrFarWcst1gsDf4Db4x91lVIewtXxYbw1c4cPt2eQ9/IALfU4Wnc2RM5k/rhedQTz+PJPalrXY3+2GFYWBidOnWqDQMAPXv2xOVykZmpO+xP3Vy4Ik1jEoiIiPs0eiAYPnw4WVlZlJSU1C7bt28fJpOJiIiIxj68x7uiRxAd21nJL6li7V6NSSAiIu5R70BQUlJCWloaaWlpAKSnp5OWlkZGRgZw8vr/pEmTatefOHEigYGB3HvvvezatYt169bx5JNPMmXKlHPeVNiaWMwmbogPB2Bp6hE3VyMiIq1VvQNBSkoK8fHxxMfHAzBjxgzi4+OZNWsWANnZ2bXhAKBdu3YkJSVRWFjIoEGDuPPOO0lMTGTevHkN9Baav5t+umywcnceBSWV51lbRESk4dX7psKRI0fyS08qLlq06IxlsbGxJCUl1fdQrUZsqI2+nfzZcbSIf23P4t7hMe4uSUREWhnNZeAhTt1cqKGMRUTEHRQIPMR1ceF4m03szLKzK6tuz4yKiIg0FAUCD9GhrTejewUDOksgIiJNT4HAg/z3mARV1RqTQEREmo4CgQe5vNvJMQmOl1axZm+eu8sREZFWRIHAg3iZTdw4oBOgywYiItK0FAg8zKnLBqv2aEwCERFpOgoEHqZ7iB9xEf5UO12sSMtydzkiItJKKBB4II1JICIiTU2BwAMl/jQmwe5sOxsP5ru7HBERaQUUCDxQe19vbhp48ubC6R+kcaxY9xKIiEjjUiDwUH+8phddg9uRV1zJ4x9so8Z57vkjRERELpYCgYdqa/Vi4V0D8PU2s/FgAX9P2ufukkREpAVTIPBgXYP9eO6mfgC8vPoAq/bkurkiERFpqRQIPNx1ceFMTugMwBMfbufI8TI3VyQiIi2RAkEz8PtrehIX2Z6icgfTFm+lsrrG3SWJiEgLo0DQDFi9zMyfGE97XwvfZxbxp3/vcndJIiLSwigQNBMRHXx56bb+GAa8tymDFduOurskERFpQRQImpGRPYJ5dFRXAGYu28G+3GI3VyQiIi2FAkEz8/jo7lzWtSPljhoefC+Vkspqd5ckIiItgAJBM2M2Gfzj9v6E2nw4dKyUmct24HJp0CIREbk4CgTNUGA7K/PvjMfLZPDZ9izeTT7s7pJERKSZUyBopgZ2DmDm1T0BePbzXWw4oEmQRETkwikQNGNThkdzTd8wHDUu7ntnCxsVCkRE5AIpEDRjhmHw4m1xjOoRRIXDyZR3tmi6ZBERuSAKBM2c1cvMwrsH/hwKFikUiIhI/dU7EKxbt47ExETCw8MxDIMVK1bUedsNGzbg5eVF//7963tY+QVWLzML7hrIyP8KBckHC9xdloiINCP1DgSlpaXExcUxf/78em1XWFjIpEmTuOqqq+p7SKkDH4uZhQoFIiJygeodCMaPH8+zzz7LDTfcUK/tHnzwQSZOnEhCQkJ9Dyl1dCoUXNE9iHJHDVMWbWHTIYUCERE5P6+mOMjbb7/NoUOHeO+993j22WfPu35lZSWVlZW1r+12OwAOhwOHw9EgNZ3aT0Ptz1OYgfm39+PhJWms21/AvW9v5vW7BzA0JsDdpZ1XS+1Jc6V+eB71xPM0h57UtbZGDwT79+/nqaee4ttvv8XLq26Hmzt3LnPmzDlj+TfffIOvr2+D1peUlNSg+/MU1wVAnr+JPUUwZdEWft2zhq42d1dVNy21J82V+uF51BPP48k9KSsrq9N6jRoIampqmDhxInPmzKF79+513m7mzJnMmDGj9rXdbicyMpKxY8diszXMbzWHw0FSUhJjxozBYrE0yD49za/G1fDQ4jS+PVDAm/utvH53PEOiPfdMQWvoSXOifnge9cTzNIeenDrLfj6NGgiKi4tJSUlh27ZtPPLIIwA4nU5cLhdeXl588803XHnllWdsZ7VasVqtZyy3WCwN/gNvjH16CovFwuuTBzP13RS+3Z/P1H9u4+WJ8VwZG+Lu0n5RS+5Jc6R+eB71xPN4ck/qWlejjkNgs9nYsWMHaWlptV8PPvggPXr0IC0tjaFDhzbm4YWTNxq+PmkQI7p1pKyqhimLUngxaR81Tk2IJCIiP6v3GYKSkhIOHDhQ+zo9PZ20tDQCAgKIiopi5syZHD16lHfffReTyUSfPn1O2z44OBgfH58zlkvj8bGYeXPyYJ79fBfvJh9m3sr9bD9SyD9u7097X293lyciIh6g3mcIUlJSiI+PJz4+HoAZM2YQHx/PrFmzAMjOziYjI6Nhq5SL5u1l4n+u78OLt8bhYzGxdt8xEl9ezw9Hi9xdmoiIeIB6B4KRI0ficrnO+Fq0aBEAixYtYs2aNefcfvbs2aSlpV1guXKxbhwQwbKHhhMV4MuR4+XctGAjH6dmurssERFxM81l0Ar1Crfx2SOXMapHEJXVTn67dDt/XLGDyuoad5cmIiJuokDQSvn7Wnhz8mCmj+6GYcB7mzK47dVNZBeVu7s0ERFxAwWCVsxkMpg+ujtvTR6MzceLtCOFXDtvvWZLFBFphRQIhFGxwfz70RH0CrNRUFrFXW98x0PvpbJ6Tx7VNU53lyciIk1AgUAAiAr05ZOHLuWmARE4XfDlDzncu2gLw/+6iue/2kN6fqm7SxQRkUakQCC12nibeeHWOL58fARThsfQwddCrr2SV9YcZNT/ruGWhRv5KOUIpZXV7i5VREQaWJPMdijNS88wG7MSe/HU+FhW7s7lo5QjrN13jC0/nmDLjyeY/a+dXNsvjFsHRTKwcwcMw3B3ySIicpEUCOScvL1MjO8bxvi+YeQUVbBsWyZLUzJJzy/lo5RMPkrJ5Pr+4fz1pn74WMzuLldERC6CLhlInYT6+/DwyK6s+s0VLH0wgZsHRuBlMvg0LYtbFiaTVajHFUVEmjMFAqkXwzAYHB3A/94Sxz/vG0oHXws7jhZx3csbSPnxuLvLExGRC6RAIBcs4ZJA/vXIZcSG+pFfUskdr2/ig82ax0JEpDlSIJCLEhngy7KHL+XqvqE4alw8tWwHz3z6Aw6NXyAi0qwoEMhF8/X2Yv7EAfxmTHcA3kk+zKQ3N3O8tMrNlYmISF0pEEiDMAyDR6/qxmt3D6Stt5nkQwVc9/J6dmfb3V2aiIjUgQKBNKixvUNZPm04nQN9yTxRzo2vbOTLHdnuLktERM5DgUAaXPcQPz6dNpzLunak3FHDQ+9v5f53UvhiRzYVDk2xLCLiiTQwkTSK9r7eLLp3MH/5Yg9vbUjnP7tz+c/uXPx8vLi6Txg3DOjEkOgATCaNcigi4gkUCKTReJlNzErsxR1DIlm+7Sgrth0lq6iCD1OO8GHKEcL9fbg+vhM3xHeie4ifu8sVEWnVFAik0XUL8eN3v4rlt2N7sPnH46zYdpTPd2STVVTBgjUHWbDmIL3CbFwfF4pN8yaJiLiFAoE0GZPJYFiXQIZ1CWT2db1ZtSeP5duOsmZvHruy7ezKthNgNdM1vpDBXYLcXa6ISKuiQCBu4WMxc3XfMK7uG8aJ0io+35HNq2sPcuREOXe8sYWnxsdy32UxmklRRKSJ6CkDcbsObb25a1hnPn04gf6BTqqdLp79fDdT302lsEyDG4mINAUFAvEYfj5e3NPNyezEnnibTfxndy7XzFvP1owT7i5NRKTFUyAQj2IYcOeQSJY9fCnRgb4cLSzn1oXJvPHtIVwul7vLExFpsRQIxCP16eTPZ49exjX9wv7rEkKKLiGIiDQSBQLxWH4+Fl6+I55nJ/TB28vEf3bn6RKCiEgjUSAQj2YYBncN68yyh06/hPDcl3vIL6l0d3kiIi1GvQPBunXrSExMJDw8HMMwWLFixS+uv2zZMsaMGUNQUBA2m42EhAS+/vrrC61XWqlTlxCu/ekSwsK1Bxn+3Cqe+fQHMk+Uubs8EZFmr96BoLS0lLi4OObPn1+n9detW8eYMWP44osvSE1NZdSoUSQmJrJt27Z6Fyutm5+Phf+7I55X7x5IXIQ/ldVO3kk+zMi/reE3H23nQF6xu0sUEWm26j0w0fjx4xk/fnyd13/ppZdOe/2Xv/yFTz/9lM8++4z4+Pj6Hl5aOcMwGNc7lLG9Qth4sID5qw+w8WABn2zNZNm2TMb1CuXhUZfQL6K9u0sVEWlWmnykQqfTSXFxMQEBAedcp7KyksrKn68P2+12ABwOBw6Ho0HqOLWfhtqfXLz69mRIZ3+G3DOQ7ZlFvLounaTdeXy1M4evduYw/JJAHrw8hqExHTTa4QXSZ8TzqCeepzn0pK61Ga6LeLjbMAyWL1/OhAkT6rzN888/z3PPPceePXsIDg4+6zqzZ89mzpw5ZyxfvHgxvr6+F1qutHDZZbDyqInUfAMnJ0PAgEAnd3dzolmWRaS1KisrY+LEiRQVFWGz2c65XpMGgsWLFzN16lQ+/fRTRo8efc71znaGIDIykvz8/F98M/XhcDhISkpizJgxWCyWBtmnXJyG6smRE2W8uf4wH6Vm4qhx8eDlMfxmTLcGrLR10GfE86gnnqc59MRut9OxY8fzBoImu2TwwQcfcP/997N06dJfDAMAVqsVq9V6xnKLxdLgP/DG2KdcnIvtSZdgf/58Yz8GdA7gN0u3s3BdOt1CbNw0MKIBq2w99BnxPOqJ5/HkntS1riYZh2DJkiXce++9LFmyhGuuuaYpDinCTQMjeGjkJQDMXLaDLT8ed3NFIiKeq96BoKSkhLS0NNLS0gBIT08nLS2NjIwMAGbOnMmkSZNq11+8eDGTJk3ihRdeYOjQoeTk5JCTk0NRUVHDvAORX/Dk2B6M6x1CVY2TX/8zlSPHNWaBiMjZ1DsQpKSkEB8fX/vI4IwZM4iPj2fWrFkAZGdn14YDgNdee43q6mqmTZtGWFhY7dfjjz/eQG9B5NxMJoO/39af3uE2jpdWcd87Wyiu8Ny7gUVE3KXe9xCMHDnyF2edW7Ro0Wmv16xZU99DiDQoX28v3pg8iOtf3sC+3BIeW7KNNyYPxqxHD0REamkuA2kVwvzb8PqkQVi9TKzee4w/f77b3SWJiHgUBQJpNeIi2/Pirf0BeGtDOou/y/jlDUREWhEFAmlVrukXxowx3QGY9ekPbDyQ7+aKREQ8gwKBtDqPXtmV6/uHU+108eB7qRw6VuLukkRE3E6BQFodwzD46039iI9qj72imvveSaGwrMrdZYmIuJUCgbRKPhYzr909iE7t25CeX8roF9fy8PupvLk+ne1HCnHUON1doohIk2ry2Q5FPEWQn5U37xnEXW98R35JFV/syOGLHTkAtLGY6R/ZnsHRHRgYHcCAqPb4+XjmsKQiIg1BgUBatdhQG+v/vyv5PrOIlMPHSfnxBKmHT1BU7iD5UAHJhwoAMBnQI9TG2F4hTBkeg7+vwoGItCwKBNLq+VjMDIkJYEhMAABOp4sDx0pI+fFEbUjIOF7G7mw7u7PtvLU+nftGxDDlshhsOmsgIi2EAoHI/8NkMuge4kf3ED8mDo0CIM9ewYaD+by69hB7cop56T/7eWt9OlNHdOGe4dG6nCAizZ5uKhSpg2CbDzfER/DFYyOYP3EA3YLbYa+o5oWkfYx4fjWvrDlAaWW1u8sUEblgCgQi9WAyGVzTL4yvpl/OvDvi6RLUlsIyB89/tZcRz6/m1bUHKatSMBCR5keBQOQCmE0G18WFk/TEFfz9tjiiA305XlrF3C/3cPnzq/m/lfs5kKcBj0Sk+dA9BCIXwWwyuCE+gsR+4SzfdpR5q/Zz5Hg5LyTt44WkfXQJasu43qGM6x1Kv07+mDTDooh4KAUCkQbgZTZxy6BIJsR34tO0LD7bnsXGg/kcOlbKgjUHWbDmIKE2H8b0CmFc71CGdgnAYtYJOhHxHAoEIg3IYjZx88AIbh4Ygb3CwZq9x/h6Zw5r9uSRY6/gn5sO889Nh7H5eHFVzxD6dPKnusZJVbUTR42TqhpX7d8dPy2vqnES5u/DpIRoIgN83f0WRaSFUiAQaSQ2HwvXxYVzXVw4FY4akg8W8PXOHJJ25VJQWsXybUdZvu1onff39oYfmRDfiYdHXkKXoHaNWLmItEYKBCJNwMdiZlRsMKNig/nzDS62ZpwgaVcu2UUVeJtNeHsZWMwmLGYT3l4//Wk28PYyYTaZWL0nj/UH8vk4NZNlWzO5pl8400ZdQmyozd1vTURaCAUCkSZmNhkMjg5gcHRAnbe577IYtmacYP6qA6zck8dn20/epzCmVwiPjOpKXGT7xitYRFoF3dUk0kwMiOrAm/cM5vPHLuOavmEYBiTtyuX6+RuY9NZmNqcfd3eJItKM6QyBSDPTO9yf+XcO4EBeMa+sOcinaVms23eMdfuOMTQmgN/9qgcDO9f97IOICOgMgUiz1TXYjxdv7c/q34zkjiFRWMwG36Uf56YFyUx9N4X9ucXuLlFEmhEFApFmLirQl7k39mXtk6O4fXAkpp8uJYx7aR1PLt1OVmG5u0sUkWZAgUCkhQhv34bnburHN09czrjeIThdsDQ1k5H/u4a/fLGbwrIqd5coIh5MgUCkheka7Merdw9i2cOXMiQmgKpqJ6+tO8SI51czf/UByqtq3F2iiHggBQKRFmpAVAc+fGAYb98zmNhQP4orqvnb13u54m+rWbI5A5fL5e4SRcSD1DsQrFu3jsTERMLDwzEMgxUrVpx3mzVr1jBgwACsVitdu3Zl0aJFF1CqiNSXYRiMig3mi8dG8Pfb4ojo0Ia84kpmLtvBI0u2aapmEalV70BQWlpKXFwc8+fPr9P66enpXHPNNYwaNYq0tDSmT5/O/fffz9dff13vYkXkwph+mpVx5W+u4A9X98RiNvj8+2xuWZjMUd10KCJcwDgE48ePZ/z48XVef+HChcTExPDCCy8A0LNnT9avX8/f//53xo0bV9/Di8hFsHqZmXp5F+Ii2/PQe6nszLJz/cvrWXjXQAbVY+REEWl5Gn1gouTkZEaPHn3asnHjxjF9+vRzblNZWUllZWXta7vdDoDD4cDhcDRIXaf201D7k4unnjSd+Ag/PnlwKA++n8aenGLueH0TcxJ7csvAiNp11A/Po554nubQk7rW1uiBICcnh5CQkNOWhYSEYLfbKS8vp02bNmdsM3fuXObMmXPG8m+++QZf34ad/jUpKalB9ycXTz1pOlOi4P0qE9uPm/j9il18tekHJkQ7MRs/r6N+eB71xPN4ck/KysrqtJ5HDl08c+ZMZsyYUfvabrcTGRnJ2LFjsdkaZnY3h8NBUlISY8aMwWKxNMg+5eKoJ+5xvdPF/LWHmLfqIOtyTFS37cg/bo2jrQX1w8PoM+J5mkNPTp1lP59GDwShoaHk5uaetiw3NxebzXbWswMAVqsVq9V6xnKLxdLgP/DG2KdcHPWk6c0YG0uv8PbM+CiNjQePc/Nr37FgYn9A/fBE6onn8eSe1LWuRh+HICEhgZUrV562LCkpiYSEhMY+tIjUw6/6hPLJQ5fSqX0bDheUcctr3/HDCeP8G4pIi1DvQFBSUkJaWhppaWnAyccK09LSyMjIAE6e7p80aVLt+g8++CCHDh3id7/7HXv27OGVV17ho48+4oknnmiYdyAiDaZnmI1/PTKcoTEBlFbW8PoeM9M/+p4f80vdXZqINLJ6B4KUlBTi4+OJj48HYMaMGcTHxzNr1iwAsrOza8MBQExMDJ9//jlJSUnExcXxwgsv8MYbb+iRQxEPFdjOynv3D+XuoZEAfL4jh9EvruWPK3aQZ69wc3Ui0ljqfQ/ByJEjf3HI07ONQjhy5Ei2bdtW30OJiJtYzCZmXduT0PJ0NpeHsnZ/Pu9tyuCT1KNMuSyaBy6/BP82nnm9VEQujOYyEJFzimgLb0wawAcPDGNAVHvKHTXMX32Qy59fzWvrDlLh0ERJIi2FAoGInNewLoF88tClvD5pEN1D2lFU7uAvX+xh5N/W8MHmDKprnO4uUUQukkeOQyAinscwDMb0CuHK2GCWbzvK35P2cbSwnKeW7eDVdYdI7BfGqNhg4iLaYzLp6QSR5kaBQETqxWwyuHlgBIlxYby3KYP5qw+Qnl/KvFUHmLfqAIFtvbmiRxBXxgYzoluQ7jUQaSYUCETkgli9zNx3WQy3Dorg6525rN6Tx7p9xygorWLZ1qMs23oUs8lgUOcOXBkbzJWxwXQNbodh6OyBiCdSIBCRi+LnY+HmgRHcPDACR42TlB9PsHpvHqv25HEgr4Tv0o/zXfpx5n65h4gObfhV71DG9w0lPrKDLi2IeBAFAhFpMBaziYRLAkm4JJDfX92TI8fLWLXnZDhIPlRA5oly3lifzhvr0wmxWflV71B+1SeMITEBmBUORNxKgUBEGk1kgC+TL41m8qXRlFVV8+3+fL76IYf/7Mol117JO8mHeSf5MB3beTOmVyhX9w1lWJdALGY9ACXS1BQIRKRJ+Hp7Ma53KON6h1JZXcOGA/l8uSOHb3blkl9SxZLNGSzZnEF7XwtXxgbTK8xGVIAvUYG+RAX44utdv3+uSiurySos52hhOfklVcSG+tE73KZ7GETOQYFARJqc1cvMlbEhXBkbwl9qnCQfLODLH3L4ZmfOzzclcvS0bTq2s9L5p3Bw6ivM34fjZVUcPVH+0y//CrIKy8kqKqewzHHGcYP9rFzRPYhRscFc1q0jNh89ASFyigKBiLiVxWzi8u5BXN49iD9d35stP57g2/3HOFxQRsbxk19F5Q7ySyrJL6kk9fCJOu/bz8eLTu3b4N/GwveZReQVV7I0NZOlqZmYTQYDO3dgVI9gRvYIIjbUT2cPpFVTIBARj+H1Xzcl/reiMgcZx8s4fLyUjONlHDlexuGCMnLsFQS29Sa8fZvar4if/gxr73PaGYDK6ho2px9nzd5jrN6bx6FjpWxOP87m9OP89as9hNp8GBUbxPg+YQzv2lE3OUqro0AgIh7P39dCX19/+kb4X/A+rF5mRnQLYkS3IJ6+thcZBWWs2ZfH6p+egMixV7Bk8xGWbD5CqM2HGwZ04qYBEXQNbteA70TEcykQiEirFBXoy6SEaCYlRFPhqGHToQL+szuXz7Znk2OvYMGagyxYc5D+ke25aWAE1/ULx9/3/PccuFwu8kuq2J9bzOHjZQyODlCokGZBgUBEWj0fi5mRPYIZ2SOYp6/txcrdeXySmsmafcdIO1JI2pFC/vTZLsb0CuGmgZ24vFsQXmYTJ0qr2JdbzL68EvblFLMvt5j9eSUcL62q3bfVy8Q/bo/nV31C3fgORc5PgUBE5L9Yvcxc3TeMq/uGkVdcwb/Ssvg4NZM9OcV8viObz3dk07GdN4ZhcKy48qz7MAyICvCljcXMnpxiHno/lTnX9WZSQnTTvhmRelAgEBE5h2A/H+4f0YX7LothZ5adT7Zm8mlaFvklP58B6NS+Dd1D2tE91I/uwX50D/Gja3A72nibqa5x8vSnO1myOYNZn+4kq7CC343roSGbxSMpEIiInIdhGPTp5E+fTv7MHN+TlMPH8fX2omtwO9pZz/3PqJfZxF9u6EO4vw8vJO1j4dqD5BSV8/zNcXh7aTRG8SwKBCIi9eDtZeLSSzrWeX3DMHj0qm6E+vswc9kOVqRlcaykkgV3DdTASOJRFFFFRJrALYMiefOewfh6m9lwoIBbFyaTU1Th7rJEaikQiIg0kSu6B/HRrxPo2M7KnpxibnxlA/tyi91dlgigQCAi0qT6dPJn+cOX0iWoLVlFFdy8YCObDhW4uywRBQIRkaYWGeDLJw9eyoCo9tgrqpn05mb+/X2Wu8uSVk6BQETEDTq09Wbx1GGM7RVCVY2TR5dsY9GGdHeXdVYulwuXy+XuMqSRKRCIiLiJj8XMgrsGcvewzrhcMPuzXTz/1R6P+uVbWlnNpLc2c9lfV/PZ9iyPqk0algKBiIgbmU0G/3N9b347tjsAr6w5yJMff4+jxunmyqDCUcP976Tw7f58jhaW8+iSbdz3TgpHC8vdXZo0AgUCERE3MwyDR67sxl9v6ovJgI9TM3ng3RTKqqrdVlNVtZOH3ksl+VAB7axe3HNpNBazwao9eYx9cS1vb0inxqmzBS3JBQWC+fPnEx0djY+PD0OHDmXz5s2/uP5LL71Ejx49aNOmDZGRkTzxxBNUVOj5WxGR/3bb4Cheu3sQVi8Tq/ceY+Lr3502UVJTqa5x8vgH21i99xg+FhNv3TOY2df15svHRzCocwdKq2qY89kublqwkT059iavTxpHvQPBhx9+yIwZM3jmmWfYunUrcXFxjBs3jry8vLOuv3jxYp566imeeeYZdu/ezZtvvsmHH37I73//+4suXkSkpRndK4TFU4fi38ZC2pFCbl64kcwTZU12fKfTxZMff8+XP+TgbTbx+qRBDIkJAKBrsB8f/TqBZyf0wc/qRdqRQq6dt56/fb2HCkdNk9UojaPegeDFF19k6tSp3HvvvfTq1YuFCxfi6+vLW2+9ddb1N27cyPDhw5k4cSLR0dGMHTuWO+6447xnFUREWquBnQP45KEEwv19OHSslBtf2cju7Mb/P3GXy8UfVvzA8m1H8TIZzL9zACO6BZ22jslkcNewziTNuIJxvUOodrqYv/og4//xLckHNZ5Cc1avuQyqqqpITU1l5syZtctMJhOjR48mOTn5rNtceumlvPfee2zevJkhQ4Zw6NAhvvjiC+6+++5zHqeyspLKyp+nFbXbT34QHA4HDoejPiWf06n9NNT+5OKpJ55F/XCvzh18+GDqEO57N5X9eaXc+moyL9/aB2icnrhcLv785V6WbM7AZMD/3tyXkd0CznmsQF8zL98ex9c7c/mfz/eQnl/KHa9v4tq+oSR0CaBHqB/df5r1sSVrDp+TutZmuOrxDElWVhadOnVi48aNJCQk1C7/3e9+x9q1a/nuu+/Out28efP47W9/i8vlorq6mgcffJAFCxac8zizZ89mzpw5ZyxfvHgxvr6+dS1XRKTZK6uG1/eYOVRsYDZc3BjtJD7QRdsGnhfp3xkmko6ePGk88ZIahgbX/YbBsmr4LMPExtzTTzobuAjygXBfF+FtXYT7nvx7gBUMzQDdZMrKypg4cSJFRUXYbLZzrtfogWDNmjXcfvvtPPvsswwdOpQDBw7w+OOPM3XqVJ5++umzHudsZwgiIyPJz8//xTdTHw6Hg6SkJMaMGYPFohnHPIF64lnUD89R4ahhxtIdJO0+ea+WAfSL8OeKbh0Z0S2Qvp38MZsu/DfsgrWHePE/BwCYfW0sdw6NuqD9bDtSyNc7c9mTU8Le3GLyS85+Q2Q7qxd9wv24Mb4T4/uE4GNpvmcRmsPnxG6307Fjx/MGgnpdMujYsSNms5nc3NzTlufm5hIaGnrWbZ5++mnuvvtu7r//fgD69u1LaWkpDzzwAH/4wx8wmc68jcFqtWK1Ws9YbrFYGvwH3hj7lIujnngW9cP9LBYLC+8exPxV+1iycT/ZZQbbM4vYnlnEvNUH6eBrYUS3IEb2COLy7kF0bHfmv5/n8ub69Now8PurY7nnsksuuM4hXYIY0uXnew6OFVeyJ8fOnuxidufY2Z1dzIG8Ykoqq9mUfoJN6Sf4y1d7uXlABHcO60xMx7YXfGx38+TPSV3rqlcg8Pb2ZuDAgaxcuZIJEyYA4HQ6WblyJY888shZtykrKzvjl77ZfDINasQrEZG6MZsMHrqiC51L9xA//EqS00+wZu8x1u/P50SZg39tz+Jf20/Oh9C3kz+9wmyYTAZmE5gMA5NhYDad/DIMMBsG9goH723KAGD66G48cPmFh4GzCfKzEuQXdNqNiY4aJwePlbBydx6Lv8vgaGE5b6xP54316QzvGshdQzszulcIFrOGyWlq9QoEADNmzGDy5MkMGjSIIUOG8NJLL1FaWsq9994LwKRJk+jUqRNz584FIDExkRdffJH4+PjaSwZPP/00iYmJtcFARETqLszfh9sGR3Hb4CgcNU62ZRSydl8ea/YeY2eWnR1Hi9hxtKjO+/v15V14/KpujVjxzyxmE7GhNmJDbTx4xSWs3ZfH+5syWLU3jw0HCthwoIAgPyu3D47k9iFRdGrfpknqkgsIBLfddhvHjh1j1qxZ5OTk0L9/f7766itCQkIAyMjIOO2MwB//+EcMw+CPf/wjR48eJSgoiMTERP785z833LsQEWmlLGYTQ2ICGBITwJPjYskrruDbffnk2CtwOl3UuFw4nS6cLmr/XvPTcpcLeoXbuGVgBIYb7vIzmwyujA3hytgQMk+U8cHmI3yw5QjHiiv5v1UHmL/6AFf1DOHPE/oQbPNp8vpam3oHAoBHHnnknJcI1qxZc/oBvLx45plneOaZZy7kUCIiUg/Bfj7cNDDC3WXUW0QHX347rgePXdWNpF25vLfpMMmHCkjalcvBvBIWTx1GqL9CQWPSRRoREfEY3l4mrukXxpIHhvHV9BF0at+GQ/ml3PZasiZVamQKBCIi4pFiQ218+OthRAa04XBBGbe9msyR4003jHNro0AgIiIeK6KDLx8+kEB0oC+ZJ8q5/bVNZBQoFDQGBQIREfFo4e3b8MEDCXTp2JajheXc9loy6fml7i6rxVEgEBERjxfq78MHDwyja3A7sosquO3VZA4eK3F3WS2KAoGIiDQLwTYflkwdRo8QP/KKK7nt1U3szy12d1kthgKBiIg0G0F+VhZPHUpsqB/5JZXc/tom9uYoFDQEBQIREWlWAttZWTJ1GL3DbRSUVnH7a8nsyrK7u6xmT4FARESanQ5tvVl8/zDiIvw5Uebgjtc38ey/d7F6bx6lldXuLq9ZuqCRCkVERNzN39fCP+8fyuS3NrMto7B2kiQvk0F8VHsuvaQjw7t2pH9ke7y93Pv/v3nFFVQ6nHVa1+plcstQzQoEIiLSbNl8LCyZOoxvduWyYX8+Gw7mk3minC0/nmDLjyf4x8r9+HqbGRITwPBLOpJwSSA9Qv0afTbFojIHGw/m8+2BfNbvzyejHgMqDYkJ4KNfJzRidWenQCAiIs2aj8XMdXHhXBcXDkBGQRkbDuaz4UA+yQcLKCitYs3eY6zZeww4OTxyjxA/eofb6BVuo1eYjdgwG+2sF/4rsbLaSUpGAesPnJySesfRIpyun79vMk7WWdf34w4KBCIi0qJEBfoSFRjFHUOicDpd7M0tZsOBfDYeLGBL+nGKK6vPmCLaMCA6sC29wk6GhO4hfljMBjVOF9U/zRBZ7XRRXeM87XVRaSWf7zbxVMoqyv+fSwJdg9txWdeOjOjWkaFdAi8qcDQFz65ORETkIphMBj3DbPQMs3H/iC44nS4yT5SzM6uIXdl2dmXZ2ZllJ8deQXp+Ken5pXy+I7u+RwGcdGxn5bKugVzWLYjLunZsdrMzKhCIiEirYTIZP51B8GV837Da5QUllezOLq4NCgePleBygZfZhJfJwGwyTvuzdrkBXvaj3HftZfSJ6IBhGG58dxdHgUBERFq9wHZWLutm5bJuHeu1ncPh4IsvMokN9WvWYQA0DoGIiIigQCAiIiIoEIiIiAgKBCIiIoICgYiIiKBAICIiIigQiIiICM1kHAKX6+SA0HZ7w8137XA4KCsrw263Y7FYGmy/cuHUE8+ifnge9cTzNIeenPrdeep36bk0i0BQXFwMQGRkpJsrERERaZ6Ki4vx9/c/5/cN1/kigwdwOp1kZWXh59dwI0HZ7XYiIyM5cuQINputQfYpF0c98Szqh+dRTzxPc+iJy+WiuLiY8PBwTKZz3ynQLM4QmEwmIiIiGmXfNpvNY5vYWqknnkX98Dzqiefx9J780pmBU3RToYiIiCgQiIiISCsOBFarlWeeeQar1eruUuQn6olnUT88j3rieVpST5rFTYUiIiLSuFrtGQIRERH5mQKBiIiIKBCIiIiIAoGIiIigQCAiIiK00kAwf/58oqOj8fHxYejQoWzevNndJbUa69atIzExkfDwcAzDYMWKFad93+VyMWvWLMLCwmjTpg2jR49m//797im2lZg7dy6DBw/Gz8+P4OBgJkyYwN69e09bp6KigmnTphEYGEi7du246aabyM3NdVPFLduCBQvo169f7ch3CQkJfPnll7XfVy/c77nnnsMwDKZPn167rCX0pdUFgg8//JAZM2bwzDPPsHXrVuLi4hg3bhx5eXnuLq1VKC0tJS4ujvnz55/1+88//zzz5s1j4cKFfPfdd7Rt25Zx48ZRUVHRxJW2HmvXrmXatGls2rSJpKQkHA4HY8eOpbS0tHadJ554gs8++4ylS5eydu1asrKyuPHGG91YdcsVERHBc889R2pqKikpKVx55ZVcf/317Ny5E1Av3G3Lli28+uqr9OvX77TlLaIvrlZmyJAhrmnTptW+rqmpcYWHh7vmzp3rxqpaJ8C1fPny2tdOp9MVGhrq+tvf/la7rLCw0GW1Wl1LlixxQ4WtU15engtwrV271uVyneyBxWJxLV26tHad3bt3uwBXcnKyu8psVTp06OB644031As3Ky4udnXr1s2VlJTkuuKKK1yPP/64y+VqOZ+RVnWGoKqqitTUVEaPHl27zGQyMXr0aJKTk91YmQCkp6eTk5NzWn/8/f0ZOnSo+tOEioqKAAgICAAgNTUVh8NxWl9iY2OJiopSXxpZTU0NH3zwAaWlpSQkJKgXbjZt2jSuueaa037+0HI+I81itsOGkp+fT01NDSEhIactDwkJYc+ePW6qSk7JyckBOGt/Tn1PGpfT6WT69OkMHz6cPn36ACf74u3tTfv27U9bV31pPDt27CAhIYGKigratWvH8uXL6dWrF2lpaeqFm3zwwQds3bqVLVu2nPG9lvIZaVWBQER+2bRp0/jhhx9Yv369u0tp1Xr06EFaWhpFRUV8/PHHTJ48mbVr17q7rFbryJEjPP744yQlJeHj4+PuchpNq7pk0LFjR8xm8xl3fubm5hIaGuqmquSUUz1Qf9zjkUce4d///jerV68mIiKidnloaChVVVUUFhaetr760ni8vb3p2rUrAwcOZO7cucTFxfGPf/xDvXCT1NRU8vLyGDBgAF5eXnh5ebF27VrmzZuHl5cXISEhLaIvrSoQeHt7M3DgQFauXFm7zOl0snLlShISEtxYmQDExMQQGhp6Wn/sdjvfffed+tOIXC4XjzzyCMuXL2fVqlXExMSc9v2BAwdisVhO68vevXvJyMhQX5qI0+mksrJSvXCTq666ih07dpCWllb7NWjQIO68887av7eEvrS6SwYzZsxg8uTJDBo0iCFDhvDSSy9RWlrKvffe6+7SWoWSkhIOHDhQ+zo9PZ20tDQCAgKIiopi+vTpPPvss3Tr1o2YmBiefvppwsPDmTBhgvuKbuGmTZvG4sWL+fTTT/Hz86u95unv70+bNm3w9/fnvvvuY8aMGQQEBGCz2Xj00UdJSEhg2LBhbq6+5Zk5cybjx48nKiqK4uJiFi9ezJo1a/j666/VCzfx8/OrvafmlLZt2xIYGFi7vEX0xd2PObjD//3f/7mioqJc3t7eriFDhrg2bdrk7pJajdWrV7uAM74mT57scrlOPnr49NNPu0JCQlxWq9V11VVXufbu3eveolu4s/UDcL399tu165SXl7sefvhhV4cOHVy+vr6uG264wZWdne2+oluwKVOmuDp37uzy9vZ2BQUFua666irXN998U/t99cIz/Pdjhy5Xy+iL4XK5XG7KIiIiIuIhWtU9BCIiInJ2CgQiIiKiQCAiIiIKBCIiIoICgYiIiKBAICIiIigQiIiICAoEIiIiggKBiIiIoEAgIiIiKBCIiIgI8P8DOWdT4Hy1/O8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training loss\n",
    "pd.Series(training_objective.evaluations_history[\"train\"][\"loss\"]).plot(figsize=(6, 3), grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd0b70fa-5f1a-498e-a02b-81b72c83cc1f",
   "metadata": {
    "tags": [],
    "id": "bd0b70fa-5f1a-498e-a02b-81b72c83cc1f"
   },
   "source": [
    "## 7. Training of Generative model\n",
    "\n",
    "Interestingly, generation is just a special case of token classification: In each prediction step, out of all possible tokens in the model vocabulary, we want to pick the one that best fits the input and previously-generated text.\n",
    "\n",
    "\n",
    "1. In this settings, when we train the model, we update it maximise the chance of predicting correct category on this single token.\n",
    "\n",
    "2. During the inference (i.e. the actual usage of the model), the model conditions next token prediction based on both (i) input text, and  (ii) it's previously-generated text.\n",
    "\n",
    "Notice, that in `adaptor`, **the whole training setup for generation is identical to sequence classification above**, thanks to the objective-centric design of experiments.\n",
    "\n",
    "The only things that change are the base model and training objective: we exchange `SequenceClassification` objective for `Sequence2Sequence`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6b1a0f2-cc70-46dd-82bf-72277f44bff4",
   "metadata": {
    "id": "b6b1a0f2-cc70-46dd-82bf-72277f44bff4"
   },
   "outputs": [],
   "source": [
    "!pip install sentencepiece protobuf==3.20.0  # extra dependences required for generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4df1218-66cf-4f27-b156-b6c84f49f11d",
   "metadata": {
    "id": "b4df1218-66cf-4f27-b156-b6c84f49f11d",
    "outputId": "45c2989e-191f-4603-b6ab-5856123d1c7c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/.local/lib/python3.8/site-packages/transformers/convert_slow_tokenizer.py:434: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from adaptor.lang_module import LangModule\n",
    "\n",
    "language_module = LangModule(\"google/mt5-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92bcac29-dad5-41da-8967-1fba93de87d3",
   "metadata": {
    "id": "92bcac29-dad5-41da-8967-1fba93de87d3"
   },
   "source": [
    "Second, we choose the objective that we want to fine-tune the model for. The objective will take care of configuring the model correctly. We just give it our desired inputs and outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ec9dbb0-28ce-4eef-adb4-e7f94fface07",
   "metadata": {
    "id": "8ec9dbb0-28ce-4eef-adb4-e7f94fface07"
   },
   "outputs": [],
   "source": [
    "from adaptor.objectives.seq2seq import Sequence2Sequence\n",
    "\n",
    "promt = \" Kolko hviezdičiek si zaslúži táto recenzia? Možnosti: 1, 2, 3, 4, 5.\"\n",
    "\n",
    "# note that we also have a myriad of Generative Evaluators in Adaptor, \n",
    "# but for simplicity, we only use eval loss as early-stopping criterion in this tutorial\n",
    "# Evaluators can be found in adaptor.evaluators.generative, and used identically to SequenceClassification\n",
    "\n",
    "training_objective = Sequence2Sequence(lang_module=language_module,\n",
    "                                       texts_or_path=[promt + review for review in dataset[\"train\"]['comment']],\n",
    "                                       labels_or_path=[str(x) for x in dataset[\"train\"][\"rating_int\"]],\n",
    "                                       val_texts_or_path=[promt + review for review in dataset[\"validation\"]['comment']],\n",
    "                                       val_labels_or_path=[str(x) for x in dataset[\"validation\"][\"rating_int\"]],\n",
    "                                       batch_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cebf2c-6c81-46ce-84f8-e5eb2d8f0380",
   "metadata": {
    "id": "12cebf2c-6c81-46ce-84f8-e5eb2d8f0380"
   },
   "source": [
    "The training process is configured through a possibly large set of arugments. You can read through each of them in [TrainingArgs documentation](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06ef7263-2e6c-42dd-abc1-cdaec7668881",
   "metadata": {
    "id": "06ef7263-2e6c-42dd-abc1-cdaec7668881"
   },
   "outputs": [],
   "source": [
    "from adaptor.utils import AdaptationArguments, StoppingStrategy\n",
    "\n",
    "args = AdaptationArguments(output_dir=\"./output_dir_generative\",\n",
    "                           learning_rate=2e-5,\n",
    "                           warmup_steps=1000,\n",
    "                           stopping_strategy=StoppingStrategy.FIRST_OBJECTIVE_CONVERGED,\n",
    "                           do_train=True,\n",
    "                           do_eval=True,\n",
    "                           log_level=\"critical\",\n",
    "                           logging_steps=100,\n",
    "                           eval_steps=200,\n",
    "                           evaluation_strategy=\"steps\",\n",
    "                           save_steps=200,\n",
    "                           save_total_limit=11,\n",
    "                           stopping_patience=10,\n",
    "                           num_train_epochs=20,\n",
    "                           gradient_accumulation_steps=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c21df6-44e3-44f1-9f00-3e6ba9fe2868",
   "metadata": {
    "id": "87c21df6-44e3-44f1-9f00-3e6ba9fe2868"
   },
   "source": [
    "Adaptor allows training for multiple objectives at once, that are applied in a chosen Schedule. Multi-objective training is useful for more complex scenarios, but for our case, we suffice with a single objective, so a selection of Schedule does not really matter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2efa3d0-f4bc-4ca4-a9e3-a430cf7c8384",
   "metadata": {
    "id": "d2efa3d0-f4bc-4ca4-a9e3-a430cf7c8384",
    "outputId": "e2c66f18-d9aa-4de1-96da-e850f1b3a1cd"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Total number of train samples: 25000\n",
      "Total number of eval samples: 2500\n"
     ]
    }
   ],
   "source": [
    "from adaptor.schedules import SequentialSchedule\n",
    "from adaptor.adapter import Adapter\n",
    "\n",
    "parallel_schedule = SequentialSchedule(objectives=[training_objective], args=args)\n",
    "\n",
    "adapter = Adapter(lang_module=language_module,\n",
    "                  schedule=parallel_schedule,\n",
    "                  args=args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a72e0cd7-464b-470d-82c2-89e142af7d69",
   "metadata": {
    "id": "a72e0cd7-464b-470d-82c2-89e142af7d69"
   },
   "source": [
    "After all the configuration, we are ready to run the training and wait for the trained model.\n",
    "\n",
    "Given the `stopping_strategy=StoppingStrategy.FIRST_OBJECTIVE_CONVERGED` and `stopping_patience=1`, the training will terminate after first evaluation, where `model_quality_evaluator` (i.e. `SequenceAccuracy`) does not improve over one evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37079d64-8d00-4dcc-a5ed-061655433f01",
   "metadata": {
    "scrolled": true,
    "tags": [],
    "id": "37079d64-8d00-4dcc-a5ed-061655433f01",
    "outputId": "26be2ee4-f3ef-4fa0-c62f-63326e49db70"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/.local/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "Converged objectives: []\n",
      "Sequence2Sequence:  13%|█▎        | 3200/25000 [07:04<44:01,  8.25batches/s, epoch=1, loss=11.7, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  13%|█▎        | 3202/25000 [07:04<53:40,  6.77batches/s, epoch=1, loss=17, split=train]    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 17.9087, 'learning_rate': 2.0000000000000003e-06, 'train_Sequence2Sequence_loss': 16.13142704963684, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  26%|██▌       | 6400/25000 [13:55<36:35,  8.47batches/s, epoch=1, loss=17, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 17.1405, 'learning_rate': 4.000000000000001e-06, 'train_Sequence2Sequence_loss': 17.9901912689209, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [01:25<00:00, 27.76batches/s, epoch=1, loss=16.1, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 13.747293472290039, 'eval_runtime': 85.0848, 'eval_samples_per_second': 29.382, 'eval_steps_per_second': 29.382, 'eval_Sequence2Sequence_loss': 13.888874187469483, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  38%|███▊      | 9600/25000 [22:30<30:15,  8.48batches/s, epoch=1, loss=5.41, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  38%|███▊      | 9602/25000 [22:30<40:07,  6.40batches/s, epoch=1, loss=17.7, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 15.2123, 'learning_rate': 6e-06, 'train_Sequence2Sequence_loss': 13.12563066482544, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.02}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  51%|█████     | 12800/25000 [29:24<23:46,  8.55batches/s, epoch=1, loss=9.41, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 13.1265, 'learning_rate': 8.000000000000001e-06, 'train_Sequence2Sequence_loss': 11.312626147270203, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.03}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [15:28<00:00,  2.69batches/s, epoch=1, loss=14.1, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [01:29<00:00, 27.67batches/s, epoch=1, loss=7.91, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 9.292762756347656, 'eval_runtime': 89.5283, 'eval_samples_per_second': 27.924, 'eval_steps_per_second': 27.924, 'eval_Sequence2Sequence_loss': 9.237446851730347, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.03}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  64%|██████▍   | 16000/25000 [38:01<17:41,  8.48batches/s, epoch=1, loss=6.95, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  64%|██████▍   | 16003/25000 [38:01<19:42,  7.61batches/s, epoch=1, loss=7.07, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 10.1095, 'learning_rate': 1e-05, 'train_Sequence2Sequence_loss': 7.4681835770606995, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.03}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  77%|███████▋  | 19199/25000 [43:14<08:29, 11.38batches/s, epoch=1, loss=6.23, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.9675, 'learning_rate': 1.2e-05, 'train_Sequence2Sequence_loss': 5.620758986473083, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.04}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [13:50<00:00,  3.01batches/s, epoch=1, loss=8.7, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [01:00<00:00, 42.41batches/s, epoch=1, loss=3.8, split=eval] in]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 4.800583362579346, 'eval_runtime': 60.8677, 'eval_samples_per_second': 41.073, 'eval_steps_per_second': 41.073, 'eval_Sequence2Sequence_loss': 4.826601564884186, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.04}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  90%|████████▉ | 22399/25000 [49:39<04:14, 10.23batches/s, epoch=1, loss=2.18, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  90%|████████▉ | 22402/25000 [49:40<04:34,  9.46batches/s, epoch=1, loss=0.0848, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.6865, 'learning_rate': 1.4e-05, 'train_Sequence2Sequence_loss': 2.250028187036514, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.04}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [53:50<00:00,  9.07batches/s, epoch=1, loss=0.833, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [53:50<00:00,  7.74batches/s, epoch=1, loss=1.11, split=train] \n",
      "Sequence2Sequence:   2%|▏         | 600/25000 [01:00<36:15, 11.21batches/s, epoch=2, loss=1.72, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.6736, 'learning_rate': 1.6000000000000003e-05, 'train_Sequence2Sequence_loss': 1.288987749814987, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.05}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:35<00:00,  3.59batches/s, epoch=1, loss=4.07, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2497/2500 [01:00<00:00, 41.67batches/s, epoch=2, loss=0.806, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.9550275206565857, 'eval_runtime': 61.0507, 'eval_samples_per_second': 40.95, 'eval_steps_per_second': 40.95, 'eval_Sequence2Sequence_loss': 1.0189049816131592, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.05}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  15%|█▌        | 3799/25000 [07:30<30:52, 11.45batches/s, epoch=2, loss=0.444, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  15%|█▌        | 3803/25000 [07:30<36:24,  9.70batches/s, epoch=2, loss=0.744, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.2113, 'learning_rate': 1.8e-05, 'train_Sequence2Sequence_loss': 0.9597227975726128, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  28%|██▊       | 7000/25000 [12:58<38:41,  7.75batches/s, epoch=2, loss=0.829, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0595, 'learning_rate': 2e-05, 'train_Sequence2Sequence_loss': 0.9232580676674843, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:58<00:00,  3.48batches/s, epoch=2, loss=0.949, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [01:09<00:00, 35.87batches/s, epoch=2, loss=0.687, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.8977074027061462, 'eval_runtime': 69.7066, 'eval_samples_per_second': 35.865, 'eval_steps_per_second': 35.865, 'eval_Sequence2Sequence_loss': 0.9306386995315552, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  41%|████      | 10199/25000 [20:00<26:26,  9.33batches/s, epoch=2, loss=0.786, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  41%|████      | 10203/25000 [20:00<26:15,  9.39batches/s, epoch=2, loss=0.84, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0496, 'learning_rate': 1.9993579454253612e-05, 'train_Sequence2Sequence_loss': 0.9891065150499344, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.07}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  54%|█████▎    | 13399/25000 [25:31<25:28,  7.59batches/s, epoch=2, loss=1.2, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.114, 'learning_rate': 1.9987158908507226e-05, 'train_Sequence2Sequence_loss': 1.4632992073893547, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.08}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:33<00:00,  3.32batches/s, epoch=2, loss=0.893, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [02:21<00:00, 15.79batches/s, epoch=2, loss=0.931, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.8781482577323914, 'eval_runtime': 141.1936, 'eval_samples_per_second': 17.706, 'eval_steps_per_second': 17.706, 'eval_Sequence2Sequence_loss': 0.8995459002256393, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.08}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  66%|██████▋   | 16599/25000 [34:46<12:47, 10.95batches/s, epoch=2, loss=1.03, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  66%|██████▋   | 16603/25000 [34:46<12:45, 10.97batches/s, epoch=2, loss=2.86, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.1752, 'learning_rate': 1.9980738362760837e-05, 'train_Sequence2Sequence_loss': 1.0303605556488038, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.08}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  79%|███████▉  | 19800/25000 [39:56<08:57,  9.68batches/s, epoch=2, loss=0.651, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.1018, 'learning_rate': 1.9974317817014448e-05, 'train_Sequence2Sequence_loss': 1.2170063734054566, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.09}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [14:24<00:00,  2.89batches/s, epoch=2, loss=0.737, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2497/2500 [01:00<00:00, 42.61batches/s, epoch=2, loss=0.777, split=eval]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.9093708992004395, 'eval_runtime': 60.3291, 'eval_samples_per_second': 41.439, 'eval_steps_per_second': 41.439, 'eval_Sequence2Sequence_loss': 0.8524070379137992, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.09}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  92%|█████████▏| 23000/25000 [46:17<02:49, 11.80batches/s, epoch=2, loss=1.36, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  92%|█████████▏| 23002/25000 [46:18<03:07, 10.67batches/s, epoch=2, loss=1.23, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.1251, 'learning_rate': 1.996789727126806e-05, 'train_Sequence2Sequence_loss': 0.8924560755491256, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|█████████▉| 24999/25000 [49:28<00:00, 12.83batches/s, epoch=2, loss=1.14, split=train] Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [49:28<00:00,  8.42batches/s, epoch=2, loss=0.611, split=train]\n",
      "Sequence2Sequence:   5%|▍         | 1199/25000 [01:55<36:57, 10.73batches/s, epoch=3, loss=1.25, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9416, 'learning_rate': 1.9961476725521673e-05, 'train_Sequence2Sequence_loss': 0.9576057404279709, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:27<00:00,  3.64batches/s, epoch=2, loss=0.759, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [01:00<00:00, 42.74batches/s, epoch=3, loss=0.881, split=eval]]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.8875173926353455, 'eval_runtime': 60.2358, 'eval_samples_per_second': 41.504, 'eval_steps_per_second': 41.504, 'eval_Sequence2Sequence_loss': 0.9013218656182289, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  18%|█▊        | 4400/25000 [08:19<35:08,  9.77batches/s, epoch=3, loss=0.704, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  18%|█▊        | 4402/25000 [08:20<45:23,  7.56batches/s, epoch=3, loss=0.362, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8777, 'learning_rate': 1.9955056179775283e-05, 'train_Sequence2Sequence_loss': 0.9964094690978527, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.11}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  30%|███       | 7600/25000 [13:30<28:35, 10.14batches/s, epoch=3, loss=1.48, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8518, 'learning_rate': 1.9948635634028894e-05, 'train_Sequence2Sequence_loss': 0.98110511302948, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.12}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:35<00:00,  3.60batches/s, epoch=3, loss=0.859, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [01:00<00:00, 41.94batches/s, epoch=3, loss=0.625, split=eval]]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.7786861062049866, 'eval_runtime': 60.4968, 'eval_samples_per_second': 41.325, 'eval_steps_per_second': 41.325, 'eval_Sequence2Sequence_loss': 0.747508175522089, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.12}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  43%|████▎     | 10800/25000 [20:13<22:13, 10.65batches/s, epoch=3, loss=0.48, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  43%|████▎     | 10802/25000 [20:13<25:38,  9.23batches/s, epoch=3, loss=1.13, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.8176, 'learning_rate': 1.9942215088282505e-05, 'train_Sequence2Sequence_loss': 0.856459392607212, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.12}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  56%|█████▌    | 13999/25000 [25:21<17:50, 10.28batches/s, epoch=3, loss=0.751, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.801, 'learning_rate': 1.993579454253612e-05, 'train_Sequence2Sequence_loss': 0.738645076006651, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:50<00:00,  3.52batches/s, epoch=3, loss=0.61, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [01:00<00:00, 41.97batches/s, epoch=3, loss=0.66, split=eval]  ] Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6833826899528503, 'eval_runtime': 60.4829, 'eval_samples_per_second': 41.334, 'eval_steps_per_second': 41.334, 'eval_Sequence2Sequence_loss': 0.6239492813497782, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  69%|██████▉   | 17199/25000 [31:43<13:49,  9.41batches/s, epoch=3, loss=0.475, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  69%|██████▉   | 17203/25000 [31:44<13:49,  9.40batches/s, epoch=3, loss=0.784, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7796, 'learning_rate': 1.992937399678973e-05, 'train_Sequence2Sequence_loss': 0.6439758151769638, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  82%|████████▏ | 20399/25000 [36:53<08:32,  8.98batches/s, epoch=3, loss=0.658, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7586, 'learning_rate': 1.992295345104334e-05, 'train_Sequence2Sequence_loss': 0.7579818919301033, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.14}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:31<00:00,  3.61batches/s, epoch=3, loss=0.701, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [01:00<00:00, 41.53batches/s, epoch=3, loss=0.522, split=eval] Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6365209221839905, 'eval_runtime': 60.4646, 'eval_samples_per_second': 41.347, 'eval_steps_per_second': 41.347, 'eval_Sequence2Sequence_loss': 0.5349153718724847, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.14}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  94%|█████████▍| 23599/25000 [43:22<02:14, 10.40batches/s, epoch=3, loss=0.433, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  94%|█████████▍| 23603/25000 [43:22<02:29,  9.35batches/s, epoch=3, loss=1.02, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7269, 'learning_rate': 1.991653290529695e-05, 'train_Sequence2Sequence_loss': 0.6378240719437599, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.15}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|█████████▉| 24999/25000 [45:37<00:00, 12.76batches/s, epoch=3, loss=0.403, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [45:37<00:00,  9.13batches/s, epoch=3, loss=0.662, split=train]\n",
      "Sequence2Sequence:   7%|▋         | 1799/25000 [02:52<36:38, 10.55batches/s, epoch=4, loss=1.28, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7174, 'learning_rate': 1.9910112359550565e-05, 'train_Sequence2Sequence_loss': 0.8308555901050567, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.15}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:36<00:00,  3.59batches/s, epoch=3, loss=0.624, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2496/2500 [01:00<00:00, 42.83batches/s, epoch=4, loss=1.12, split=eval]  ]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6226826906204224, 'eval_runtime': 60.3031, 'eval_samples_per_second': 41.457, 'eval_steps_per_second': 41.457, 'eval_Sequence2Sequence_loss': 0.5227230403013527, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.15}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  20%|█▉        | 4999/25000 [09:37<28:54, 11.53batches/s, epoch=4, loss=0.279, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  20%|██        | 5003/25000 [09:37<33:51,  9.84batches/s, epoch=4, loss=0.557, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6995, 'learning_rate': 1.9903691813804176e-05, 'train_Sequence2Sequence_loss': 0.7266338691115379, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.16}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  33%|███▎      | 8199/25000 [15:16<23:13, 12.05batches/s, epoch=4, loss=0.401, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6957, 'learning_rate': 1.9897271268057787e-05, 'train_Sequence2Sequence_loss': 0.6796964161098004, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.17}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:24<00:00,  3.36batches/s, epoch=4, loss=0.699, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2497/2500 [01:00<00:00, 41.53batches/s, epoch=4, loss=1.04, split=eval]  ]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5963655114173889, 'eval_runtime': 60.7486, 'eval_samples_per_second': 41.153, 'eval_steps_per_second': 41.153, 'eval_Sequence2Sequence_loss': 0.48524558486416935, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.17}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  46%|████▌     | 11400/25000 [21:56<21:09, 10.71batches/s, epoch=4, loss=0.217, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  46%|████▌     | 11402/25000 [21:56<24:30,  9.25batches/s, epoch=4, loss=4.58, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6896, 'learning_rate': 1.9890850722311397e-05, 'train_Sequence2Sequence_loss': 0.6386201687157154, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.17}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  58%|█████▊    | 14600/25000 [27:10<18:46,  9.23batches/s, epoch=4, loss=1.62, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6769, 'learning_rate': 1.988443017656501e-05, 'train_Sequence2Sequence_loss': 0.8141349509358407, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.18}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:54<00:00,  3.50batches/s, epoch=4, loss=0.598, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2497/2500 [01:00<00:00, 42.10batches/s, epoch=4, loss=1.07, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6001282930374146, 'eval_runtime': 60.5808, 'eval_samples_per_second': 41.267, 'eval_steps_per_second': 41.267, 'eval_Sequence2Sequence_loss': 0.487833188418299, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.18}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  71%|███████   | 17800/25000 [33:39<10:43, 11.18batches/s, epoch=4, loss=1.54, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  71%|███████   | 17802/25000 [33:40<12:09,  9.87batches/s, epoch=4, loss=0.58, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6756, 'learning_rate': 1.9878009630818622e-05, 'train_Sequence2Sequence_loss': 0.6737121604382992, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.19}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  84%|████████▍ | 21000/25000 [38:53<06:16, 10.62batches/s, epoch=4, loss=0.306, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6585, 'learning_rate': 1.9871589085072233e-05, 'train_Sequence2Sequence_loss': 0.6223988311365247, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.19}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:42<00:00,  3.56batches/s, epoch=4, loss=0.594, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2496/2500 [01:00<00:00, 42.18batches/s, epoch=4, loss=0.967, split=eval]  ]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5861856937408447, 'eval_runtime': 60.5055, 'eval_samples_per_second': 41.319, 'eval_steps_per_second': 41.319, 'eval_Sequence2Sequence_loss': 0.4468530134856701, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.19}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  97%|█████████▋| 24199/25000 [45:36<01:10, 11.31batches/s, epoch=4, loss=0.618, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  97%|█████████▋| 24201/25000 [45:36<01:21,  9.79batches/s, epoch=4, loss=0.277, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6455, 'learning_rate': 1.9865168539325844e-05, 'train_Sequence2Sequence_loss': 0.5100643190555274, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [46:54<00:00,  9.71batches/s, epoch=4, loss=0.0989, split=train] Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [46:54<00:00,  8.88batches/s, epoch=4, loss=0.622, split=train] \n",
      "Sequence2Sequence:  10%|▉         | 2399/25000 [03:54<33:43, 11.17batches/s, epoch=5, loss=0.79, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6468, 'learning_rate': 1.9858747993579458e-05, 'train_Sequence2Sequence_loss': 0.5149811970070004, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:55<00:00,  3.49batches/s, epoch=4, loss=0.8, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [00:59<00:00, 41.41batches/s, epoch=5, loss=0.378, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5601460933685303, 'eval_runtime': 59.788, 'eval_samples_per_second': 41.814, 'eval_steps_per_second': 41.814, 'eval_Sequence2Sequence_loss': 0.4408376425690949, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  22%|██▏       | 5599/25000 [10:26<28:16, 11.44batches/s, epoch=5, loss=0.423, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  22%|██▏       | 5603/25000 [10:26<30:39, 10.54batches/s, epoch=5, loss=0.469, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6352, 'learning_rate': 1.985232744783307e-05, 'train_Sequence2Sequence_loss': 0.7465759068727493, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.21}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  35%|███▌      | 8799/25000 [15:38<27:25,  9.84batches/s, epoch=5, loss=0.0388, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.644, 'learning_rate': 1.984590690208668e-05, 'train_Sequence2Sequence_loss': 0.6023554718121886, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:44<00:00,  3.55batches/s, epoch=5, loss=0.658, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [01:01<00:00, 41.07batches/s, epoch=5, loss=0.714, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5537274479866028, 'eval_runtime': 61.2439, 'eval_samples_per_second': 40.82, 'eval_steps_per_second': 40.82, 'eval_Sequence2Sequence_loss': 0.42096569160930813, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  48%|████▊     | 12000/25000 [22:05<20:19, 10.66batches/s, epoch=5, loss=0.952, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  48%|████▊     | 12002/25000 [22:05<22:23,  9.68batches/s, epoch=5, loss=0.121, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6388, 'learning_rate': 1.983948635634029e-05, 'train_Sequence2Sequence_loss': 0.5808817652985454, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  61%|██████    | 15199/25000 [27:25<14:39, 11.14batches/s, epoch=5, loss=0.579, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6176, 'learning_rate': 1.98330658105939e-05, 'train_Sequence2Sequence_loss': 0.5324886912247166, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [11:46<00:00,  3.54batches/s, epoch=5, loss=0.583, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.27batches/s, epoch=5, loss=0.723, split=eval]  ]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5530375838279724, 'eval_runtime': 59.3352, 'eval_samples_per_second': 42.134, 'eval_steps_per_second': 42.134, 'eval_Sequence2Sequence_loss': 0.41948916090419514, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  74%|███████▎  | 18400/25000 [33:57<11:28,  9.58batches/s, epoch=5, loss=0.694, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  74%|███████▎  | 18402/25000 [33:57<13:39,  8.05batches/s, epoch=5, loss=0.125, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6197, 'learning_rate': 1.982664526484751e-05, 'train_Sequence2Sequence_loss': 0.5873832687211689, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.24}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  86%|████████▋ | 21600/25000 [39:28<05:25, 10.44batches/s, epoch=5, loss=0.351, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6004, 'learning_rate': 1.9820224719101126e-05, 'train_Sequence2Sequence_loss': 0.5783053252846002, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.24}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:03<00:00,  3.46batches/s, epoch=5, loss=0.566, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.52batches/s, epoch=5, loss=0.22, split=eval]    Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5415412783622742, 'eval_runtime': 59.261, 'eval_samples_per_second': 42.186, 'eval_steps_per_second': 42.186, 'eval_Sequence2Sequence_loss': 0.3954517652653158, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.24}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  99%|█████████▉| 24799/25000 [46:14<00:16, 12.25batches/s, epoch=5, loss=0.327, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  99%|█████████▉| 24803/25000 [46:15<00:17, 11.22batches/s, epoch=5, loss=0.258, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6034, 'learning_rate': 1.9813804173354736e-05, 'train_Sequence2Sequence_loss': 0.5259017555043102, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.25}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|█████████▉| 24999/25000 [46:31<00:00, 11.07batches/s, epoch=5, loss=0.0068, split=train]Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [46:31<00:00,  8.96batches/s, epoch=5, loss=0.952, split=train] \n",
      "Sequence2Sequence:  12%|█▏        | 2999/25000 [05:04<41:45,  8.78batches/s, epoch=6, loss=0.46, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6117, 'learning_rate': 1.9807383627608347e-05, 'train_Sequence2Sequence_loss': 0.7263975866138935, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:07<00:00,  3.44batches/s, epoch=5, loss=0.65, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.34batches/s, epoch=6, loss=0.104, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5392204523086548, 'eval_runtime': 59.1984, 'eval_samples_per_second': 42.231, 'eval_steps_per_second': 42.231, 'eval_Sequence2Sequence_loss': 0.40796574177918954, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  25%|██▍       | 6200/25000 [12:01<26:33, 11.80batches/s, epoch=6, loss=0.0174, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  25%|██▍       | 6202/25000 [12:02<29:22, 10.66batches/s, epoch=6, loss=0.315, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5915, 'learning_rate': 1.9800963081861958e-05, 'train_Sequence2Sequence_loss': 0.5904795465990901, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  38%|███▊      | 9400/25000 [17:34<26:13,  9.92batches/s, epoch=6, loss=1.09, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.602, 'learning_rate': 1.9794542536115572e-05, 'train_Sequence2Sequence_loss': 0.5218310073018074, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:29<00:00,  3.33batches/s, epoch=6, loss=0.64, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 41.92batches/s, epoch=6, loss=0.0644, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5270799994468689, 'eval_runtime': 59.729, 'eval_samples_per_second': 41.856, 'eval_steps_per_second': 41.856, 'eval_Sequence2Sequence_loss': 0.3850528732431121, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  50%|█████     | 12600/25000 [24:30<20:50,  9.92batches/s, epoch=6, loss=0.522, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  50%|█████     | 12603/25000 [24:31<20:55,  9.87batches/s, epoch=6, loss=0.919, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.6061, 'learning_rate': 1.9788121990369183e-05, 'train_Sequence2Sequence_loss': 0.45653727781027553, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.28}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  63%|██████▎   | 15800/25000 [30:04<15:23,  9.96batches/s, epoch=6, loss=0.989, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5984, 'learning_rate': 1.9781701444622793e-05, 'train_Sequence2Sequence_loss': 0.6076782478019596, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.28}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:29<00:00,  3.33batches/s, epoch=6, loss=0.536, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2497/2500 [01:00<00:00, 41.70batches/s, epoch=6, loss=0.976, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5285273194313049, 'eval_runtime': 60.1461, 'eval_samples_per_second': 41.565, 'eval_steps_per_second': 41.565, 'eval_Sequence2Sequence_loss': 0.3991472888714634, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.28}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  76%|███████▌  | 19000/25000 [36:59<11:01,  9.07batches/s, epoch=6, loss=0.968, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  76%|███████▌  | 19003/25000 [36:59<11:25,  8.74batches/s, epoch=6, loss=0.615, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5773, 'learning_rate': 1.9775280898876404e-05, 'train_Sequence2Sequence_loss': 0.41512585487216713, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.29}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  89%|████████▉ | 22200/25000 [42:32<04:42,  9.93batches/s, epoch=6, loss=0.782, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5764, 'learning_rate': 1.9768860353130018e-05, 'train_Sequence2Sequence_loss': 0.4847982835024595, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.29}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:27<00:00,  3.34batches/s, epoch=6, loss=0.591, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.06batches/s, epoch=6, loss=0.0612, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5239563584327698, 'eval_runtime': 59.3078, 'eval_samples_per_second': 42.153, 'eval_steps_per_second': 42.153, 'eval_Sequence2Sequence_loss': 0.37656585180317054, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.29}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|█████████▉| 24999/25000 [48:35<00:00, 10.35batches/s, epoch=6, loss=0.0808, split=train]   Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:35<00:00,  8.58batches/s, epoch=6, loss=0.0695, split=train]\n",
      "Sequence2Sequence:   2%|▏         | 400/25000 [00:41<49:57,  8.21batches/s, epoch=7, loss=0.357, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:   2%|▏         | 402/25000 [00:42<55:18,  7.41batches/s, epoch=7, loss=0.277, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5622, 'learning_rate': 1.976243980738363e-05, 'train_Sequence2Sequence_loss': 0.4356877865735441, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.3}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  14%|█▍        | 3600/25000 [06:13<34:46, 10.26batches/s, epoch=7, loss=0.668, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5741, 'learning_rate': 1.975601926163724e-05, 'train_Sequence2Sequence_loss': 0.5757703617215156, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.31}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:16<00:00,  3.40batches/s, epoch=6, loss=0.581, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.57batches/s, epoch=7, loss=0.136, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5329573154449463, 'eval_runtime': 59.1827, 'eval_samples_per_second': 42.242, 'eval_steps_per_second': 42.242, 'eval_Sequence2Sequence_loss': 0.3857708898675628, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.31}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  27%|██▋       | 6800/25000 [13:05<33:47,  8.98batches/s, epoch=7, loss=0.832, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  27%|██▋       | 6803/25000 [13:06<35:08,  8.63batches/s, epoch=7, loss=0.269, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5604, 'learning_rate': 1.9749598715890854e-05, 'train_Sequence2Sequence_loss': 0.5301136564346962, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.31}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  40%|███▉      | 9999/25000 [18:37<23:11, 10.78batches/s, epoch=7, loss=0.533, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5775, 'learning_rate': 1.9743178170144464e-05, 'train_Sequence2Sequence_loss': 0.7959721550461836, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.32}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:24<00:00,  3.36batches/s, epoch=7, loss=0.582, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [00:59<00:00, 40.64batches/s, epoch=7, loss=0.149, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5236142873764038, 'eval_runtime': 59.7858, 'eval_samples_per_second': 41.816, 'eval_steps_per_second': 41.816, 'eval_Sequence2Sequence_loss': 0.3973331648739986, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.32}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  53%|█████▎    | 13199/25000 [25:26<25:02,  7.86batches/s, epoch=7, loss=0.0325, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  53%|█████▎    | 13202/25000 [25:26<23:50,  8.25batches/s, epoch=7, loss=1.72, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5611, 'learning_rate': 1.9736757624398075e-05, 'train_Sequence2Sequence_loss': 0.7791984749026597, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.33}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  66%|██████▌   | 16399/25000 [30:57<13:01, 11.01batches/s, epoch=7, loss=0.605, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5711, 'learning_rate': 1.9730337078651686e-05, 'train_Sequence2Sequence_loss': 0.6143085643649101, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.33}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:19<00:00,  3.38batches/s, epoch=7, loss=0.645, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.41batches/s, epoch=7, loss=0.0698, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5071893334388733, 'eval_runtime': 59.3963, 'eval_samples_per_second': 42.09, 'eval_steps_per_second': 42.09, 'eval_Sequence2Sequence_loss': 0.3698523764451966, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.33}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  78%|███████▊  | 19600/25000 [37:50<08:53, 10.13batches/s, epoch=7, loss=0.573, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  78%|███████▊  | 19602/25000 [37:50<09:45,  9.22batches/s, epoch=7, loss=0.145, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5538, 'learning_rate': 1.97239165329053e-05, 'train_Sequence2Sequence_loss': 0.6059135699593753, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.34}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  91%|█████████ | 22800/25000 [43:30<03:06, 11.80batches/s, epoch=7, loss=1.05, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5512, 'learning_rate': 1.971749598715891e-05, 'train_Sequence2Sequence_loss': 0.5801600774750113, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.35}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:32<00:00,  3.32batches/s, epoch=7, loss=0.563, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:58<00:00, 42.76batches/s, epoch=7, loss=0.732, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5192049741744995, 'eval_runtime': 58.9591, 'eval_samples_per_second': 42.402, 'eval_steps_per_second': 42.402, 'eval_Sequence2Sequence_loss': 0.38003518845303913, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.35}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:36<00:00,  9.04batches/s, epoch=7, loss=0.0131, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:36<00:00,  8.57batches/s, epoch=7, loss=0.0995, split=train]\n",
      "Sequence2Sequence:   4%|▍         | 1000/25000 [01:38<42:37,  9.38batches/s, epoch=8, loss=0.083, split=train] Converged objectives: []\n",
      "Sequence2Sequence:   4%|▍         | 1003/25000 [01:38<41:52,  9.55batches/s, epoch=8, loss=0.878, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5457, 'learning_rate': 1.971107544141252e-05, 'train_Sequence2Sequence_loss': 0.4755964858457446, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.35}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  17%|█▋        | 4200/25000 [07:08<35:11,  9.85batches/s, epoch=8, loss=1.73, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5596, 'learning_rate': 1.9704654895666132e-05, 'train_Sequence2Sequence_loss': 0.5862939998507499, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.36}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:14<00:00,  3.40batches/s, epoch=7, loss=0.519, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.30batches/s, epoch=8, loss=0.148, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5198851227760315, 'eval_runtime': 59.3985, 'eval_samples_per_second': 42.089, 'eval_steps_per_second': 42.089, 'eval_Sequence2Sequence_loss': 0.37279789358435667, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.36}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  30%|██▉       | 7400/25000 [13:48<29:30,  9.94batches/s, epoch=8, loss=0.334, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  30%|██▉       | 7403/25000 [13:49<30:49,  9.51batches/s, epoch=8, loss=1.71, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5529, 'learning_rate': 1.9698234349919746e-05, 'train_Sequence2Sequence_loss': 0.33956901472993195, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.36}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  42%|████▏     | 10600/25000 [19:20<26:04,  9.20batches/s, epoch=8, loss=0.259, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5615, 'learning_rate': 1.9691813804173357e-05, 'train_Sequence2Sequence_loss': 0.4332905065268278, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.37}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:11<00:00,  3.42batches/s, epoch=8, loss=0.521, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.67batches/s, epoch=8, loss=0.715, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5131158232688904, 'eval_runtime': 59.3613, 'eval_samples_per_second': 42.115, 'eval_steps_per_second': 42.115, 'eval_Sequence2Sequence_loss': 0.37863726761075667, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.37}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  55%|█████▌    | 13799/25000 [26:04<19:17,  9.68batches/s, epoch=8, loss=0.585, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  55%|█████▌    | 13803/25000 [26:04<18:51,  9.90batches/s, epoch=8, loss=1.02, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.547, 'learning_rate': 1.9685393258426968e-05, 'train_Sequence2Sequence_loss': 0.5277315009385347, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.38}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  68%|██████▊   | 17000/25000 [31:39<13:54,  9.59batches/s, epoch=8, loss=0.573, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5503, 'learning_rate': 1.967897271268058e-05, 'train_Sequence2Sequence_loss': 0.49954956378787757, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.38}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:18<00:00,  3.38batches/s, epoch=8, loss=0.567, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.93batches/s, epoch=8, loss=0.654, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5158900618553162, 'eval_runtime': 59.3458, 'eval_samples_per_second': 42.126, 'eval_steps_per_second': 42.126, 'eval_Sequence2Sequence_loss': 0.373412733253208, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.38}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  81%|████████  | 20200/25000 [38:29<07:29, 10.68batches/s, epoch=8, loss=1.39, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  81%|████████  | 20202/25000 [38:30<08:34,  9.32batches/s, epoch=8, loss=0.479, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5371, 'learning_rate': 1.9672552166934192e-05, 'train_Sequence2Sequence_loss': 0.7105328477453441, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.39}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  94%|█████████▎| 23399/25000 [43:52<02:43,  9.81batches/s, epoch=8, loss=0.855, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5349, 'learning_rate': 1.9666131621187803e-05, 'train_Sequence2Sequence_loss': 0.5205264747200999, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.4}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:13<00:00,  3.41batches/s, epoch=8, loss=0.504, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 43.00batches/s, epoch=8, loss=0.0224, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5281214714050293, 'eval_runtime': 59.0845, 'eval_samples_per_second': 42.312, 'eval_steps_per_second': 42.312, 'eval_Sequence2Sequence_loss': 0.3893917181767756, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.4}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [47:50<00:00,  8.84batches/s, epoch=8, loss=0.00228, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [47:50<00:00,  8.71batches/s, epoch=8, loss=0.17, split=train]   \n",
      "Sequence2Sequence:   6%|▋         | 1599/25000 [02:43<38:14, 10.20batches/s, epoch=9, loss=1.52, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:   6%|▋         | 1603/25000 [02:44<39:21,  9.91batches/s, epoch=9, loss=0.375, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5409, 'learning_rate': 1.9659711075441414e-05, 'train_Sequence2Sequence_loss': 0.3952503086067736, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.4}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  19%|█▉        | 4799/25000 [08:11<31:23, 10.72batches/s, epoch=9, loss=0.373, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5353, 'learning_rate': 1.9653290529695025e-05, 'train_Sequence2Sequence_loss': 0.5716028929455206, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.41}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:09<00:00,  3.43batches/s, epoch=8, loss=0.508, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [00:59<00:00, 42.68batches/s, epoch=9, loss=0.0832, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5132700204849243, 'eval_runtime': 59.4986, 'eval_samples_per_second': 42.018, 'eval_steps_per_second': 42.018, 'eval_Sequence2Sequence_loss': 0.37770710997865536, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.41}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  32%|███▏      | 8000/25000 [14:47<22:34, 12.55batches/s, epoch=9, loss=1.66, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  32%|███▏      | 8002/25000 [14:47<25:52, 10.95batches/s, epoch=9, loss=0.424, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.533, 'learning_rate': 1.964686998394864e-05, 'train_Sequence2Sequence_loss': 0.6118331732344814, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.42}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  45%|████▍     | 11199/25000 [20:20<22:54, 10.04batches/s, epoch=9, loss=0.921, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5387, 'learning_rate': 1.964044943820225e-05, 'train_Sequence2Sequence_loss': 0.60723997905734, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.42}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:08<00:00,  3.43batches/s, epoch=9, loss=0.557, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.46batches/s, epoch=9, loss=0.669, split=eval]   ]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5135796070098877, 'eval_runtime': 59.2113, 'eval_samples_per_second': 42.222, 'eval_steps_per_second': 42.222, 'eval_Sequence2Sequence_loss': 0.37837944358718234, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.42}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  58%|█████▊    | 14400/25000 [27:05<19:03,  9.27batches/s, epoch=9, loss=0.232, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  58%|█████▊    | 14403/25000 [27:06<18:22,  9.61batches/s, epoch=9, loss=1, split=train]    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5397, 'learning_rate': 1.963402889245586e-05, 'train_Sequence2Sequence_loss': 0.5801524210721254, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.43}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  70%|███████   | 17600/25000 [32:35<13:08,  9.38batches/s, epoch=9, loss=1.37, split=train]    Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5328, 'learning_rate': 1.962760834670947e-05, 'train_Sequence2Sequence_loss': 0.5299173167906701, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.44}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:14<00:00,  3.40batches/s, epoch=9, loss=0.442, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 41.13batches/s, epoch=9, loss=0.0297, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5034902691841125, 'eval_runtime': 59.516, 'eval_samples_per_second': 42.006, 'eval_steps_per_second': 42.006, 'eval_Sequence2Sequence_loss': 0.3722729913392686, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.44}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  83%|████████▎ | 20800/25000 [39:24<06:37, 10.58batches/s, epoch=9, loss=0.0106, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  83%|████████▎ | 20802/25000 [39:24<07:35,  9.21batches/s, epoch=9, loss=0.0276, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5131, 'learning_rate': 1.9621187800963085e-05, 'train_Sequence2Sequence_loss': 0.5429276508511975, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.44}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  96%|█████████▌| 24000/25000 [45:03<01:46,  9.35batches/s, epoch=9, loss=0.0665, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5065, 'learning_rate': 1.9614767255216696e-05, 'train_Sequence2Sequence_loss': 0.5721983339637517, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.45}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:28<00:00,  3.34batches/s, epoch=9, loss=0.404, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.47batches/s, epoch=9, loss=0.706, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5150699019432068, 'eval_runtime': 59.2128, 'eval_samples_per_second': 42.221, 'eval_steps_per_second': 42.221, 'eval_Sequence2Sequence_loss': 0.3666959664868773, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.45}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:03<00:00,  9.06batches/s, epoch=9, loss=0.0265, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:03<00:00,  8.67batches/s, epoch=9, loss=0.812, split=train] \n",
      "Sequence2Sequence:   9%|▉         | 2199/25000 [03:51<45:03,  8.43batches/s, epoch=10, loss=0.0174, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:   9%|▉         | 2202/25000 [03:51<44:50,  8.47batches/s, epoch=10, loss=1.01, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5281, 'learning_rate': 1.9608346709470306e-05, 'train_Sequence2Sequence_loss': 0.48698622044175865, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.45}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  22%|██▏       | 5400/25000 [09:28<31:46, 10.28batches/s, epoch=10, loss=0.706, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5218, 'learning_rate': 1.960192616372392e-05, 'train_Sequence2Sequence_loss': 0.6729309821268543, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.46}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:28<00:00,  3.34batches/s, epoch=9, loss=0.463, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.91batches/s, epoch=10, loss=0.0379, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.528730571269989, 'eval_runtime': 59.4233, 'eval_samples_per_second': 42.071, 'eval_steps_per_second': 42.071, 'eval_Sequence2Sequence_loss': 0.385827737479558, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.46}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  34%|███▍      | 8600/25000 [16:17<29:20,  9.31batches/s, epoch=10, loss=0.191, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  34%|███▍      | 8603/25000 [16:17<29:29,  9.27batches/s, epoch=10, loss=0.468, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5162, 'learning_rate': 1.959550561797753e-05, 'train_Sequence2Sequence_loss': 0.4432920694351196, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.47}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  47%|████▋     | 11799/25000 [21:57<22:18,  9.86batches/s, epoch=10, loss=0.958, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.53, 'learning_rate': 1.9589085072231142e-05, 'train_Sequence2Sequence_loss': 0.6654968202812597, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.47}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:28<00:00,  3.34batches/s, epoch=10, loss=0.62, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.25batches/s, epoch=10, loss=0.693, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5064353346824646, 'eval_runtime': 59.6871, 'eval_samples_per_second': 41.885, 'eval_steps_per_second': 41.885, 'eval_Sequence2Sequence_loss': 0.3757906282806653, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.47}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  60%|██████    | 15000/25000 [28:47<16:32, 10.08batches/s, epoch=10, loss=0.761, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  60%|██████    | 15002/25000 [28:47<18:10,  9.17batches/s, epoch=10, loss=1.75, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5033, 'learning_rate': 1.9582664526484753e-05, 'train_Sequence2Sequence_loss': 0.5067023937823251, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.48}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  73%|███████▎  | 18199/25000 [34:10<10:35, 10.70batches/s, epoch=10, loss=0.115, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5146, 'learning_rate': 1.9576243980738367e-05, 'train_Sequence2Sequence_loss': 0.3965223783161491, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.49}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:13<00:00,  3.41batches/s, epoch=10, loss=0.483, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.73batches/s, epoch=10, loss=0.0298, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5249765515327454, 'eval_runtime': 59.447, 'eval_samples_per_second': 42.054, 'eval_steps_per_second': 42.054, 'eval_Sequence2Sequence_loss': 0.39318603200707003, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.49}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  86%|████████▌ | 21400/25000 [41:01<06:01,  9.95batches/s, epoch=10, loss=0.583, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  86%|████████▌ | 21402/25000 [41:01<06:55,  8.65batches/s, epoch=10, loss=0.0114, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4981, 'learning_rate': 1.9569823434991978e-05, 'train_Sequence2Sequence_loss': 0.4352576333330944, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.49}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  98%|█████████▊| 24600/25000 [46:40<00:39, 10.12batches/s, epoch=10, loss=0.7, split=train]     Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5059, 'learning_rate': 1.9563402889245588e-05, 'train_Sequence2Sequence_loss': 0.39349839659407737, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.5}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:29<00:00,  3.34batches/s, epoch=10, loss=0.446, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.86batches/s, epoch=10, loss=0.0244, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.520451545715332, 'eval_runtime': 59.2704, 'eval_samples_per_second': 42.18, 'eval_steps_per_second': 42.18, 'eval_Sequence2Sequence_loss': 0.37552610839542466, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.5}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:35<00:00, 10.20batches/s, epoch=10, loss=0.0857, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:35<00:00,  8.57batches/s, epoch=10, loss=0.0259, split=train]\n",
      "Sequence2Sequence:  11%|█         | 2799/25000 [04:50<33:33, 11.02batches/s, epoch=11, loss=0.29, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  11%|█         | 2802/25000 [04:50<38:48,  9.53batches/s, epoch=11, loss=0.631, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5152, 'learning_rate': 1.95569823434992e-05, 'train_Sequence2Sequence_loss': 0.6208932337845908, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.51}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  24%|██▍       | 6000/25000 [10:19<29:36, 10.70batches/s, epoch=11, loss=0.0411, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4941, 'learning_rate': 1.955056179775281e-05, 'train_Sequence2Sequence_loss': 0.3747291999403387, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.51}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:15<00:00,  3.40batches/s, epoch=10, loss=0.496, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.06batches/s, epoch=11, loss=0.0319, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5053517818450928, 'eval_runtime': 59.5323, 'eval_samples_per_second': 41.994, 'eval_steps_per_second': 41.994, 'eval_Sequence2Sequence_loss': 0.3802253436569299, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.51}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  37%|███▋      | 9200/25000 [17:06<28:48,  9.14batches/s, epoch=11, loss=0.292, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  37%|███▋      | 9203/25000 [17:06<28:14,  9.32batches/s, epoch=11, loss=0.441, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5151, 'learning_rate': 1.954414125200642e-05, 'train_Sequence2Sequence_loss': 0.913335249107331, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.52}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  50%|████▉     | 12400/25000 [22:22<20:51, 10.07batches/s, epoch=11, loss=0.573, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5055, 'learning_rate': 1.953772070626003e-05, 'train_Sequence2Sequence_loss': 0.4158824751153588, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.52}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:02<00:00,  3.46batches/s, epoch=11, loss=0.517, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.62batches/s, epoch=11, loss=0.645, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5081944465637207, 'eval_runtime': 59.1503, 'eval_samples_per_second': 42.265, 'eval_steps_per_second': 42.265, 'eval_Sequence2Sequence_loss': 0.3985707842015108, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.52}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  62%|██████▏   | 15599/25000 [29:13<15:06, 10.37batches/s, epoch=11, loss=0.372, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  62%|██████▏   | 15602/25000 [29:14<17:43,  8.84batches/s, epoch=11, loss=0.0146, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5082, 'learning_rate': 1.9531300160513645e-05, 'train_Sequence2Sequence_loss': 0.32458888047840445, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.53}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  75%|███████▌  | 18800/25000 [34:52<11:38,  8.88batches/s, epoch=11, loss=0.015, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4858, 'learning_rate': 1.9524879614767256e-05, 'train_Sequence2Sequence_loss': 0.46332801144453695, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.54}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:30<00:00,  3.33batches/s, epoch=11, loss=0.43, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.82batches/s, epoch=11, loss=0.594, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5175153613090515, 'eval_runtime': 59.2053, 'eval_samples_per_second': 42.226, 'eval_steps_per_second': 42.226, 'eval_Sequence2Sequence_loss': 0.3807108040136154, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.54}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  88%|████████▊ | 22000/25000 [41:52<05:43,  8.73batches/s, epoch=11, loss=0.541, split=train]      Converged objectives: []\n",
      "Sequence2Sequence:  88%|████████▊ | 22002/25000 [41:52<06:32,  7.64batches/s, epoch=11, loss=1.56, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4846, 'learning_rate': 1.9518459069020867e-05, 'train_Sequence2Sequence_loss': 0.5482988751027733, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.54}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [47:01<00:00,  9.75batches/s, epoch=11, loss=0.00605, split=train] Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [47:02<00:00,  8.86batches/s, epoch=11, loss=0.144, split=train]  \n",
      "Sequence2Sequence:   1%|          | 200/25000 [00:21<44:08,  9.36batches/s, epoch=12, loss=0.343, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4824, 'learning_rate': 1.9512038523274477e-05, 'train_Sequence2Sequence_loss': 0.47327965890290213, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.55}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:30<00:00,  3.33batches/s, epoch=11, loss=0.38, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.48batches/s, epoch=12, loss=0.0306, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5046790242195129, 'eval_runtime': 59.3408, 'eval_samples_per_second': 42.129, 'eval_steps_per_second': 42.129, 'eval_Sequence2Sequence_loss': 0.37417812409192264, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.55}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  14%|█▎        | 3400/25000 [07:06<40:16,  8.94batches/s, epoch=12, loss=0.227, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  14%|█▎        | 3401/25000 [07:07<44:41,  8.06batches/s, epoch=12, loss=0.914, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5018, 'learning_rate': 1.950561797752809e-05, 'train_Sequence2Sequence_loss': 0.4678455790504813, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.56}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  26%|██▋       | 6600/25000 [12:40<28:40, 10.70batches/s, epoch=12, loss=0.396, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4929, 'learning_rate': 1.9499197431781702e-05, 'train_Sequence2Sequence_loss': 0.3709556887246436, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.56}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:19<00:00,  3.38batches/s, epoch=12, loss=0.446, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.79batches/s, epoch=12, loss=0.713, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4892161190509796, 'eval_runtime': 59.2241, 'eval_samples_per_second': 42.213, 'eval_steps_per_second': 42.213, 'eval_Sequence2Sequence_loss': 0.3665177660973131, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.56}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  39%|███▉      | 9800/25000 [19:22<29:45,  8.51batches/s, epoch=12, loss=1.13, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  39%|███▉      | 9803/25000 [19:22<27:52,  9.09batches/s, epoch=12, loss=0.0112, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5053, 'learning_rate': 1.9492776886035313e-05, 'train_Sequence2Sequence_loss': 0.3760595821891911, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.57}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  52%|█████▏    | 12999/25000 [24:57<18:18, 10.93batches/s, epoch=12, loss=0.0808, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4889, 'learning_rate': 1.9486356340288927e-05, 'train_Sequence2Sequence_loss': 0.5273703459650279, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.58}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:16<00:00,  3.39batches/s, epoch=12, loss=0.405, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.75batches/s, epoch=12, loss=0.077, split=eval]   ]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5077356696128845, 'eval_runtime': 59.1835, 'eval_samples_per_second': 42.242, 'eval_steps_per_second': 42.242, 'eval_Sequence2Sequence_loss': 0.37432838380113026, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.58}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  65%|██████▍   | 16200/25000 [31:44<15:00,  9.77batches/s, epoch=12, loss=0.0847, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  65%|██████▍   | 16202/25000 [31:45<17:19,  8.46batches/s, epoch=12, loss=0.708, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.496, 'learning_rate': 1.9479935794542538e-05, 'train_Sequence2Sequence_loss': 0.4600994316395372, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.58}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  78%|███████▊  | 19400/25000 [37:14<09:30,  9.81batches/s, epoch=12, loss=0.0225, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4809, 'learning_rate': 1.947351524879615e-05, 'train_Sequence2Sequence_loss': 0.4438044151291251, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.59}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:17<00:00,  3.39batches/s, epoch=12, loss=0.466, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.58batches/s, epoch=12, loss=0.0763, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5054042339324951, 'eval_runtime': 59.3612, 'eval_samples_per_second': 42.115, 'eval_steps_per_second': 42.115, 'eval_Sequence2Sequence_loss': 0.3638694470819246, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.59}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  90%|█████████ | 22599/25000 [44:05<04:01,  9.96batches/s, epoch=12, loss=0.0988, split=train]  Converged objectives: []\n",
      "Sequence2Sequence:  90%|█████████ | 22603/25000 [44:05<04:14,  9.41batches/s, epoch=12, loss=0.146, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4807, 'learning_rate': 1.946709470304976e-05, 'train_Sequence2Sequence_loss': 0.3785831667264574, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:16<00:00,  9.24batches/s, epoch=12, loss=0.00232, split=train] Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:16<00:00,  8.63batches/s, epoch=12, loss=0.322, split=train]  \n",
      "Sequence2Sequence:   3%|▎         | 800/25000 [01:23<44:40,  9.03batches/s, epoch=13, loss=0.396, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4737, 'learning_rate': 1.9460674157303373e-05, 'train_Sequence2Sequence_loss': 0.48623442770913244, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:24<00:00,  3.36batches/s, epoch=12, loss=0.391, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.57batches/s, epoch=13, loss=0.553, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5129628777503967, 'eval_runtime': 59.5483, 'eval_samples_per_second': 41.983, 'eval_steps_per_second': 41.983, 'eval_Sequence2Sequence_loss': 0.3666597532480591, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  16%|█▌        | 4000/25000 [08:13<40:46,  8.58batches/s, epoch=13, loss=0.205, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  16%|█▌        | 4003/25000 [08:13<41:39,  8.40batches/s, epoch=13, loss=0.0983, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4954, 'learning_rate': 1.9454253611556984e-05, 'train_Sequence2Sequence_loss': 0.48100812174379826, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.61}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  29%|██▉       | 7200/25000 [13:44<31:23,  9.45batches/s, epoch=13, loss=0.0375, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4879, 'learning_rate': 1.9447833065810595e-05, 'train_Sequence2Sequence_loss': 0.4292648277012631, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.61}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:20<00:00,  3.37batches/s, epoch=13, loss=0.295, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:58<00:00, 43.11batches/s, epoch=13, loss=0.0934, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5119818449020386, 'eval_runtime': 58.8604, 'eval_samples_per_second': 42.473, 'eval_steps_per_second': 42.473, 'eval_Sequence2Sequence_loss': 0.36383356753074625, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.61}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  42%|████▏     | 10400/25000 [20:39<23:38, 10.29batches/s, epoch=13, loss=1.3, split=train]      Converged objectives: []\n",
      "Sequence2Sequence:  42%|████▏     | 10402/25000 [20:40<28:01,  8.68batches/s, epoch=13, loss=0.458, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4861, 'learning_rate': 1.9441412520064206e-05, 'train_Sequence2Sequence_loss': 0.4191356441937387, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.62}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  54%|█████▍    | 13599/25000 [26:03<18:22, 10.34batches/s, epoch=13, loss=0.528, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4784, 'learning_rate': 1.943499197431782e-05, 'train_Sequence2Sequence_loss': 0.5468687702901661, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:19<00:00,  3.38batches/s, epoch=13, loss=0.455, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 41.96batches/s, epoch=13, loss=0.0733, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5147397518157959, 'eval_runtime': 59.1669, 'eval_samples_per_second': 42.253, 'eval_steps_per_second': 42.253, 'eval_Sequence2Sequence_loss': 0.3721724234933208, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  67%|██████▋   | 16799/25000 [32:58<15:19,  8.92batches/s, epoch=13, loss=0.459, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  67%|██████▋   | 16802/25000 [32:58<16:11,  8.44batches/s, epoch=13, loss=0.1, split=train]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4749, 'learning_rate': 1.942857142857143e-05, 'train_Sequence2Sequence_loss': 0.39462841373169794, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  80%|████████  | 20000/25000 [38:30<09:06,  9.15batches/s, epoch=13, loss=0.559, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4779, 'learning_rate': 1.942215088282504e-05, 'train_Sequence2Sequence_loss': 0.3715055366512388, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.64}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:27<00:00,  3.35batches/s, epoch=13, loss=0.365, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:58<00:00, 42.83batches/s, epoch=13, loss=0.0553, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5091077089309692, 'eval_runtime': 59.0218, 'eval_samples_per_second': 42.357, 'eval_steps_per_second': 42.357, 'eval_Sequence2Sequence_loss': 0.3622108953912175, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.64}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  93%|█████████▎| 23200/25000 [45:15<03:16,  9.14batches/s, epoch=13, loss=0.349, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  93%|█████████▎| 23203/25000 [45:16<03:18,  9.03batches/s, epoch=13, loss=0.58, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4676, 'learning_rate': 1.9415730337078652e-05, 'train_Sequence2Sequence_loss': 0.5174128056329209, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.65}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:23<00:00,  9.39batches/s, epoch=13, loss=0.00691, split=train] Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:23<00:00,  8.61batches/s, epoch=13, loss=0.114, split=train]  \n",
      "Sequence2Sequence:   6%|▌         | 1400/25000 [02:24<38:11, 10.30batches/s, epoch=14, loss=0.0106, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4614, 'learning_rate': 1.9409309791332266e-05, 'train_Sequence2Sequence_loss': 0.45616555340820925, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.65}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:17<00:00,  3.39batches/s, epoch=13, loss=0.297, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.56batches/s, epoch=14, loss=0.0403, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5207175612449646, 'eval_runtime': 59.3334, 'eval_samples_per_second': 42.135, 'eval_steps_per_second': 42.135, 'eval_Sequence2Sequence_loss': 0.3794464306027658, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.65}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  18%|█▊        | 4600/25000 [09:06<34:20,  9.90batches/s, epoch=14, loss=0.24, split=train]      Converged objectives: []\n",
      "Sequence2Sequence:  18%|█▊        | 4602/25000 [09:07<39:48,  8.54batches/s, epoch=14, loss=1.32, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4761, 'learning_rate': 1.9402889245585877e-05, 'train_Sequence2Sequence_loss': 0.27794278897345065, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.66}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  31%|███       | 7799/25000 [14:36<29:20,  9.77batches/s, epoch=14, loss=0.814, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4778, 'learning_rate': 1.9396468699839487e-05, 'train_Sequence2Sequence_loss': 0.36681154281977796, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.67}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:11<00:00,  3.42batches/s, epoch=14, loss=0.328, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.60batches/s, epoch=14, loss=0.0429, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.512718141078949, 'eval_runtime': 59.2895, 'eval_samples_per_second': 42.166, 'eval_steps_per_second': 42.166, 'eval_Sequence2Sequence_loss': 0.36688949639614293, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.67}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  44%|████▍     | 11000/25000 [21:12<18:09, 12.85batches/s, epoch=14, loss=0.153, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  44%|████▍     | 11002/25000 [21:13<22:01, 10.59batches/s, epoch=14, loss=0.0743, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4761, 'learning_rate': 1.9390048154093098e-05, 'train_Sequence2Sequence_loss': 0.3861128713237122, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.67}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  57%|█████▋    | 14199/25000 [26:48<17:57, 10.03batches/s, epoch=14, loss=0.0322, split=train]  Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4701, 'learning_rate': 1.9383627608346712e-05, 'train_Sequence2Sequence_loss': 0.5288792096194811, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.68}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:11<00:00,  3.42batches/s, epoch=14, loss=0.348, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 42.54batches/s, epoch=14, loss=0.0206, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5064195990562439, 'eval_runtime': 59.2337, 'eval_samples_per_second': 42.206, 'eval_steps_per_second': 42.206, 'eval_Sequence2Sequence_loss': 0.36742951164022086, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.68}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  70%|██████▉   | 17399/25000 [33:38<11:13, 11.28batches/s, epoch=14, loss=0.236, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  70%|██████▉   | 17402/25000 [33:38<13:51,  9.13batches/s, epoch=14, loss=0.00703, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4665, 'learning_rate': 1.9377207062600323e-05, 'train_Sequence2Sequence_loss': 0.5853470032467157, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.68}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  82%|████████▏ | 20599/25000 [38:50<07:01, 10.44batches/s, epoch=14, loss=0.807, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4468, 'learning_rate': 1.9370786516853934e-05, 'train_Sequence2Sequence_loss': 0.49325691343983635, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.69}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:02<00:00,  3.46batches/s, epoch=14, loss=0.33, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [01:01<00:00, 42.44batches/s, epoch=14, loss=0.0286, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5058165788650513, 'eval_runtime': 61.6052, 'eval_samples_per_second': 40.581, 'eval_steps_per_second': 40.581, 'eval_Sequence2Sequence_loss': 0.36956959562106023, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.69}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  95%|█████████▌| 23800/25000 [45:22<01:37, 12.36batches/s, epoch=14, loss=0.299, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  95%|█████████▌| 23802/25000 [45:23<02:03,  9.73batches/s, epoch=14, loss=0.000817, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4551, 'learning_rate': 1.9364365971107544e-05, 'train_Sequence2Sequence_loss': 0.44054210537287875, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.7}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [47:24<00:00, 10.49batches/s, epoch=14, loss=0.0234, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [47:24<00:00,  8.79batches/s, epoch=14, loss=0.101, split=train] \n",
      "Sequence2Sequence:   8%|▊         | 2000/25000 [03:26<39:53,  9.61batches/s, epoch=15, loss=0.805, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4627, 'learning_rate': 1.935794542536116e-05, 'train_Sequence2Sequence_loss': 0.4921399026643485, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.7}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:01<00:00,  3.47batches/s, epoch=14, loss=0.379, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.38batches/s, epoch=15, loss=0.669, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5068882703781128, 'eval_runtime': 59.3718, 'eval_samples_per_second': 42.108, 'eval_steps_per_second': 42.108, 'eval_Sequence2Sequence_loss': 0.3671030857101687, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.7}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  21%|██        | 5199/25000 [10:16<25:11, 13.10batches/s, epoch=15, loss=0.725, split=train]    Converged objectives: []\n",
      "Sequence2Sequence:  21%|██        | 5203/25000 [10:17<27:46, 11.88batches/s, epoch=15, loss=0.0023, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4568, 'learning_rate': 1.935152487961477e-05, 'train_Sequence2Sequence_loss': 0.34417617645813153, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.71}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  34%|███▎      | 8400/25000 [15:46<28:48,  9.60batches/s, epoch=15, loss=0.491, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4615, 'learning_rate': 1.934510433386838e-05, 'train_Sequence2Sequence_loss': 0.48778395596600604, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.72}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:19<00:00,  3.38batches/s, epoch=15, loss=0.334, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.35batches/s, epoch=15, loss=0.7, split=eval]     Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5138648152351379, 'eval_runtime': 59.6039, 'eval_samples_per_second': 41.944, 'eval_steps_per_second': 41.944, 'eval_Sequence2Sequence_loss': 0.37122122727636453, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.72}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  46%|████▋     | 11600/25000 [22:29<22:58,  9.72batches/s, epoch=15, loss=0.123, split=train]   Converged objectives: []\n",
      "Sequence2Sequence:  46%|████▋     | 11602/25000 [22:29<24:38,  9.06batches/s, epoch=15, loss=0.55, split=train] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4811, 'learning_rate': 1.9338683788121994e-05, 'train_Sequence2Sequence_loss': 0.5794610660756007, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.72}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  59%|█████▉    | 14800/25000 [28:01<17:35,  9.67batches/s, epoch=15, loss=0.00273, split=train] Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4589, 'learning_rate': 1.9332263242375605e-05, 'train_Sequence2Sequence_loss': 0.4520539697899949, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.73}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:15<00:00,  3.40batches/s, epoch=15, loss=0.362, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2498/2500 [00:58<00:00, 42.01batches/s, epoch=15, loss=0.0395, split=eval]  Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5128594040870667, 'eval_runtime': 58.9628, 'eval_samples_per_second': 42.4, 'eval_steps_per_second': 42.4, 'eval_Sequence2Sequence_loss': 0.38117097136870143, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.73}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  72%|███████▏  | 18000/25000 [34:51<11:40,  9.99batches/s, epoch=15, loss=0.87, split=train]      Converged objectives: []\n",
      "Sequence2Sequence:  72%|███████▏  | 18002/25000 [34:51<13:30,  8.63batches/s, epoch=15, loss=0.165, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4612, 'learning_rate': 1.9325842696629215e-05, 'train_Sequence2Sequence_loss': 0.41338120492873714, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.74}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  85%|████████▍ | 21200/25000 [40:20<05:55, 10.68batches/s, epoch=15, loss=0.319, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4399, 'learning_rate': 1.9319422150882826e-05, 'train_Sequence2Sequence_loss': 0.47086584819771815, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.74}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:18<00:00,  3.39batches/s, epoch=15, loss=0.3, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2499/2500 [00:59<00:00, 42.93batches/s, epoch=15, loss=0.829, split=eval]   n]Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5155004858970642, 'eval_runtime': 59.1815, 'eval_samples_per_second': 42.243, 'eval_steps_per_second': 42.243, 'eval_Sequence2Sequence_loss': 0.37860950291757034, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.74}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  98%|█████████▊| 24399/25000 [47:00<00:58, 10.24batches/s, epoch=15, loss=0.906, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  98%|█████████▊| 24402/25000 [47:00<01:04,  9.21batches/s, epoch=15, loss=0.385, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4483, 'learning_rate': 1.931300160513644e-05, 'train_Sequence2Sequence_loss': 0.4644367385888472, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.75}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:02<00:00, 10.75batches/s, epoch=15, loss=0.0247, split=train]  Converged objectives: []\n",
      "Sequence2Sequence: 100%|██████████| 25000/25000 [48:02<00:00,  8.67batches/s, epoch=15, loss=0.073, split=train] \n",
      "Sequence2Sequence:  10%|█         | 2600/25000 [04:35<40:10,  9.29batches/s, epoch=16, loss=0.259, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.456, 'learning_rate': 1.930658105939005e-05, 'train_Sequence2Sequence_loss': 0.4327113742008805, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.76}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:17<00:00,  3.39batches/s, epoch=15, loss=0.412, split=eval]\n",
      "Sequence2Sequence: 100%|█████████▉| 2497/2500 [00:59<00:00, 43.34batches/s, epoch=16, loss=0.756, split=eval]   Converged objectives: []\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5141530632972717, 'eval_runtime': 59.4825, 'eval_samples_per_second': 42.029, 'eval_steps_per_second': 42.029, 'eval_Sequence2Sequence_loss': 0.3772306936941823, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.76}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  23%|██▎       | 5800/25000 [11:12<33:19,  9.60batches/s, epoch=16, loss=0.269, split=train]     Converged objectives: []\n",
      "Sequence2Sequence:  23%|██▎       | 5803/25000 [11:13<35:00,  9.14batches/s, epoch=16, loss=0.0074, split=train]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4478, 'learning_rate': 1.9300160513643662e-05, 'train_Sequence2Sequence_loss': 0.46690354589372873, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.76}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence:  36%|███▌      | 8999/25000 [16:42<26:59,  9.88batches/s, epoch=16, loss=0.256, split=train]   Converged objectives: []\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.4538, 'learning_rate': 1.9293739967897272e-05, 'train_Sequence2Sequence_loss': 0.5165918341255746, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.77}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sequence2Sequence: 100%|██████████| 2500/2500 [12:07<00:00,  3.44batches/s, epoch=16, loss=0.457, split=eval]\n",
      "Sequence2Sequence: 100%|██████████| 2500/2500 [00:59<00:00, 41.30batches/s, epoch=16, loss=0.0375, split=eval]  Objective `Sequence2Sequence` convergence metric `loss` did not improve for 10 eval steps. History: [0.3794464306027658, 0.36688949639614293, 0.36742951164022086, 0.36956959562106023, 0.3671030857101687, 0.37122122727636453, 0.38117097136870143, 0.37860950291757034, 0.3772306936941823, 0.37459440535254546]\n",
      "Converged objectives: ['Sequence2Sequence']\n",
      "Scheduler reached a termination condition: FIRST_OBJECTIVE_CONVERGED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5148823261260986, 'eval_runtime': 59.5736, 'eval_samples_per_second': 41.965, 'eval_steps_per_second': 41.965, 'eval_Sequence2Sequence_loss': 0.37459440535254546, 'eval_Sequence2Sequence_num_batches': 100, 'epoch': 0.77}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Objective `Sequence2Sequence` convergence metric `loss` did not improve for 10 eval steps. History: [0.3794464306027658, 0.36688949639614293, 0.36742951164022086, 0.36956959562106023, 0.3671030857101687, 0.37122122727636453, 0.38117097136870143, 0.37860950291757034, 0.3772306936941823, 0.37459440535254546]\n",
      "Converged objectives: ['Sequence2Sequence']\n",
      "Scheduler reached a termination condition: FIRST_OBJECTIVE_CONVERGED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 44469.4849, 'train_samples_per_second': 224.873, 'train_steps_per_second': 7.027, 'train_loss': 1.267830918629964, 'train_Sequence2Sequence_loss': 0.5165918341255746, 'train_Sequence2Sequence_num_batches': 20, 'epoch': 0.77}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=12000, training_loss=1.267830918629964, metrics={'train_runtime': 44469.4849, 'train_samples_per_second': 224.873, 'train_steps_per_second': 7.027, 'total_flos': 0.0, 'train_loss': 1.267830918629964})"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adapter.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Training logs\n",
    "\n",
    "Note that generative approach took much longer to converge: While discriminative training converged in roughly 2 hours, generative approach took almost 12hours. However, mT5-base (600m) is also larger than XLM-RoBERTa-base (110m)."
   ],
   "metadata": {
    "id": "KOWr3FOtMykE"
   },
   "id": "KOWr3FOtMykE"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b54201ac-827a-4ead-827a-20f0bfdbf818",
   "metadata": {
    "tags": [],
    "id": "b54201ac-827a-4ead-827a-20f0bfdbf818",
    "outputId": "c18f096a-42be-446d-c89c-c94202dee822"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgAAAAESCAYAAACLjm33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAsqElEQVR4nO3df3QU9b038Pfs7uwkm+wmJCGbLCSIFkV++gOlXGzFKz8eVCr33nr1YG2OPaetGqs0faxyn6Kgtah9LpeqHNDec6X3tIi194LWKpIHES5VfkMFfyDRiAgkIUiySTbZnd35Pn/MzubnLtkwm0123q9zcpKdncx888lm857vfOc7khBCgIiIiCzFlu4GEBER0eBjACAiIrIgBgAiIiILYgAgIiKyIAYAIiIiC2IAICIisiAGACIiIgtypLsBPWmahlOnTsHtdkOSpHQ3h4iIaNgQQqClpQU+nw82W+Jj/CEXAE6dOoWysrJ0N4OIiGjYOnHiBEaPHp1wnSEXANxuNwC98R6Px5RtqqqKLVu2YO7cuZBl2ZRtZhLWJzHWJz7WJjHWJzHWJ76B1sbv96OsrCz2vzSRIRcAjG5/j8djagBwuVzweDx8kfWB9UmM9YmPtUmM9UmM9YnvQmvTn1PoHARIRERkQQwAREREFpR0ANixYwcWLFgAn88HSZKwadOmuOvec889kCQJq1atuoAmEhERkdmSDgBtbW2YOnUqVq9enXC9jRs3YteuXfD5fANuHBEREaVG0oMA58+fj/nz5ydc5+TJk/jJT36Ct99+GzfffPOAG0dERESpYfpVAJqm4a677sJDDz2EiRMnnnf9YDCIYDAYe+z3+wHoIyBVVTWlTcZ2zNpepmF9EmN94mNtEmN9EmN94htobZJZ3/QA8PTTT8PhcOCBBx7o1/orVqzA8uXLey3fsmULXC6XKW062wGc6ZBQ/3o1vNmmbDIjVVdXp7sJQxrrEx9rkxjrkxjrE1+ytQkEAv1e19QAsH//fvzmN7/BgQMH+j2N75IlS1BVVRV7bExiMHfuXNPmAfg/m47gjx+fwr3fvgh3z7nUlG1mElVVUV1djTlz5vBa3D6wPvGxNomxPomxPvENtDZGL3p/mBoA/ud//gcNDQ0oLy+PLYtEIvjZz36GVatW4Ysvvuj1PYqiQFGUXstlWTbtBVHi0Q/7vw6ofJElYGbNMxHrEx9rkxjrkxjrE1+ytUlmXVMDwF133YXZs2d3WzZv3jzcdddduPvuu83cVVKKPXrAqPcHz7MmERGRNSQdAFpbW1FTUxN7XFtbi0OHDqGgoADl5eUoLCzstr4syygpKcFll1124a0doJFuPQCcaWUAICIiAgYQAPbt24cbbrgh9tg4f19RUYF169aZ1jAzFedGA0BLKM0tISIiGhqSDgCzZs2CEKLf6/d13n+wGacAGluDiGgCdlv/BigSERFlKkvcC6AwxwkJApoAzvI0ABERkTUCgN0mwR0dGNnQwgBARERkiQAAAB6n/rne35HehhAREQ0BlgkAeU593AJ7AIiIiCwUADzRUwDsASAiIrJSAIieAmAPABERkYUCQOwUAGcDJCIisk4A8MSuAuApACIiIssEAPYAEBERdbJMADB6AM5EZwMkIiKyMssEALcTkCQgogl83cZ7AhARkbVZJgDYJX1KYIDjAIiIiCwTAABgZPSugBwHQEREVmepAGDcFZA9AEREZHXWCgBuPQDUsweAiIgszlIBIHYKgD0ARERkcZYKALFTAOwBICIii7NWAIj2ANTzfgBERGRx1goA0R6AM7wjIBERWZy1AoDbGAMQhMbZAImIyMKSDgA7duzAggUL4PP5IEkSNm3aFHtOVVU8/PDDmDx5MnJycuDz+fD9738fp06dMrPNA2ZMBBTWBM4FOBsgERFZV9IBoK2tDVOnTsXq1at7PRcIBHDgwAEsXboUBw4cwH//93/j6NGj+M53vmNKYy+U02HrMhsgxwEQEZF1OZL9hvnz52P+/Pl9PpeXl4fq6upuy55//nlce+21+PLLL1FeXj6wVppopFvB2bYQ6v0duLzUk+7mEBERpUXSASBZzc3NkCQJ+fn5fT4fDAYRDHYejfv9fgD66QRVVU1pg7EdVVVR7HbikzrgdFPAtO0Pd13rQ72xPvGxNomxPomxPvENtDbJrC8JIQY8Gk6SJGzcuBELFy7s8/mOjg7MnDkT48ePxx/+8Ic+11m2bBmWL1/ea/n69evhcrkG2rS41tfYsPuMDTeXRTB3NAcCEhFR5ggEAli0aBGam5vh8STu5U5ZD4Cqqvjnf/5nCCGwZs2auOstWbIEVVVVscd+vx9lZWWYO3fueRufTFuqq6sxZ84cfCJ/gd1najHCdxFuuulyU7Y/3HWtjyzL6W7OkMP6xMfaJMb6JMb6xDfQ2hi96P2RkgBg/PM/fvw43nnnnYT/yBVFgaIovZbLsmz6C0KWZZSO0HsVGltVvuB6SEXNMwnrEx9rkxjrkxjrE1+ytUlmXdMDgPHP/9ixY9i2bRsKCwvN3sUFid0QiPcDICIiC0s6ALS2tqKmpib2uLa2FocOHUJBQQFKS0vx3e9+FwcOHMAbb7yBSCSCuro6AEBBQQGcTqd5LR+gYk8WAN4PgIiIrC3pALBv3z7ccMMNscfG+fuKigosW7YMr7/+OgDgiiuu6PZ927Ztw6xZswbeUpMYPQBnWoIQQkCSpDS3iIiIaPAlHQBmzZqFRBcOXMBFBYNiZDQAhCIamgIqRuSkv1eCiIhosFnqXgAAoDjsGOHSB0lwNkAiIrIqywUAACh26+MA6nlXQCIisihrBgBP510BiYiIrMiaASDaA9DASwGJiMiirBkAjB4AXgpIREQWZckA4HUbpwDYA0BERNZkyQBgTAZUzx4AIiKyKEsGAK+HPQBERGRtlgwAsUGA/uCQn7iIiIgoFSwZAIzZAINhDf72cJpbQ0RENPgsGQCyZDvyso3ZAHkagIiIrMeSAQDocltgDgQkIiILsmwA8Ho4GRAREVmXZQNAsZvTARMRkXVZNwB4eEMgIiKyLusGAPYAEBGRhVk3AMTuB8AeACIish7LBoDOQYDsASAiIuuxbACInQLgbIBERGRBFg4Aeg9AuxpBS5CzARIRkbUkHQB27NiBBQsWwOfzQZIkbNq0qdvzQgg8+uijKC0tRXZ2NmbPno1jx46Z1V7TZDvtcGc5AOi9AERERFaSdABoa2vD1KlTsXr16j6ff+aZZ/Dss89i7dq12L17N3JycjBv3jx0dAy9wXadVwIMvbYRERGlkiPZb5g/fz7mz5/f53NCCKxatQq/+MUvcOuttwIA/vM//xNerxebNm3CHXfccWGtNZnXk4XPzrSxB4CIiCwn6QCQSG1tLerq6jB79uzYsry8PEyfPh3vv/9+nwEgGAwiGOz8B+z3+wEAqqpCVVVT2mVsp+f2inKcAIDTTQHT9jUcxasP6Vif+FibxFifxFif+AZam2TWNzUA1NXVAQC8Xm+35V6vN/ZcTytWrMDy5ct7Ld+yZQtcLpeZzUN1dXW3x62NNgA27P7gE/j8H5m6r+GoZ32oO9YnPtYmMdYnMdYnvmRrEwgE+r2uqQFgIJYsWYKqqqrYY7/fj7KyMsydOxcej8eUfaiqiurqasyZMweyLMeW1793HO+cOoqcIh9uummKKfsajuLVh3SsT3ysTWKsT2KsT3wDrY3Ri94fpgaAkpISAEB9fT1KS0tjy+vr63HFFVf0+T2KokBRlF7LZVk2/QXRc5sl+XoPQ2NriC8+pKbmmYT1iY+1SYz1SYz1iS/Z2iSzrqnzAIwdOxYlJSXYunVrbJnf78fu3bsxY8YMM3dlCi/vB0BERBaVdA9Aa2srampqYo9ra2tx6NAhFBQUoLy8HIsXL8Yvf/lLjBs3DmPHjsXSpUvh8/mwcOFCM9ttCuOOgLwfABERWU3SAWDfvn244YYbYo+N8/cVFRVYt24dfv7zn6OtrQ0/+tGP0NTUhOuuuw6bN29GVlaWea02iTEPQFsogtZgGLlK2odEEBERDYqk/+PNmjUr4dz5kiTh8ccfx+OPP35BDRsMOYoDuYoDrcEwGvwdyB2Zm+4mERERDQrL3gvAUMxxAEREZEEMAB49ANRzHAAREVkIA0D0roBn2ANAREQWYvkA4GUPABERWZDlA4DRA8AxAEREZCUMAOwBICIiC7J8ACiJTgZUz1sCExGRhVg+AHijAaCuuSPh/AZERESZxPIBoCRPDwDtagT+jnCaW0NERDQ4LB8AsmQ78rL1uyfxngBERGQVlg8AQOc4gDoGACIisggGAADevM5xAERERFbAAADA6+algEREZC0MAOgcCMhTAEREZBUMAOi8FJBzARARkVUwAKDrZEDsASAiImtgAED3yYCIiIisgAEAgDdPHwTY2BpEOKKluTVERESpxwAAoChHgcMmQRPAmVaOAyAioszHAADAZpNQHLsUkAGAiIgyn+kBIBKJYOnSpRg7diyys7NxySWX4IknnhjyN9rhZEBERGQlDrM3+PTTT2PNmjX43e9+h4kTJ2Lfvn24++67kZeXhwceeMDs3ZnG6+aVAEREZB2mB4D33nsPt956K26++WYAwEUXXYSXX34Ze/bsMXtXpuJkQEREZCWmB4C/+7u/w4svvohPP/0Ul156Kf72t79h586dWLlyZZ/rB4NBBIOd5939fj8AQFVVqKpqSpuM7STaXlGOfkfAuqaAafsdLvpTHytjfeJjbRJjfRJjfeIbaG2SWV8SJp+c1zQN//Iv/4JnnnkGdrsdkUgETz75JJYsWdLn+suWLcPy5ct7LV+/fj1cLpeZTUto7xkJv6+x49I8DZUTeCkgERENP4FAAIsWLUJzczM8Hk/CdU0PABs2bMBDDz2EX//615g4cSIOHTqExYsXY+XKlaioqOi1fl89AGVlZWhsbDxv4/tLVVVUV1djzpw5kGW5z3Xe//wsvv/SflxclIO3H5xpyn6Hi/7Ux8pYn/hYm8RYn8RYn/gGWhu/34+ioqJ+BQDTTwE89NBDeOSRR3DHHXcAACZPnozjx49jxYoVfQYARVGgKEqv5bIsm/6CSLTNUQW5AICGlqBlX4ipqHkmYX3iY20SY30SY33iS7Y2yaxr+mWAgUAANlv3zdrtdmja0O5WN+4H0BoMozUYTnNriIiIUsv0HoAFCxbgySefRHl5OSZOnIiDBw9i5cqV+MEPfmD2rkyVozjgVhxoCYZR7+9A7sjcdDeJiIgoZUwPAM899xyWLl2K++67Dw0NDfD5fPjxj3+MRx991Oxdmc6bl4WWhlbUN3fgEgYAIiLKYKYHALfbjVWrVmHVqlVmbzrlvB4FNQ2tnAuAiIgyHu8F0EXstsAMAERElOEYALowBgLW834ARESU4RgAujCmA+YdAYmIKNMxAHRR7OYpACIisgYGgC46ewAYAIiIKLMxAHRhjAFoaAkiopk6QzIREdGQwgDQRVGuEzYJiGgCZ1s5DoCIiDIXA0AXDrsNI936fQk4EJCIiDIZA0APnAuAiIisgAGgBwYAIiKyAgaAHjgZEBERWQEDQA+8FJCIiKyAAaCH4uggQJ4CICKiTMYA0AN7AIiIyAoYAHowxgDUcQwAERFlMAaAHrzRHgB/RxjtoUiaW0NERJQaDAA9uBUHXE47AJ4GICKizMUA0IMkSZwLgIiIMh4DQB+8HmM6YAYAIiLKTAwAfeBAQCIiynQpCQAnT57E9773PRQWFiI7OxuTJ0/Gvn37UrGrlDAGAvIUABERZSqH2Rs8d+4cZs6ciRtuuAFvvfUWRo4ciWPHjmHEiBFm7yplvG49ADTwjoBERJShTA8ATz/9NMrKyvDSSy/Flo0dO9bs3aRUCXsAiIgow5keAF5//XXMmzcPt912G7Zv345Ro0bhvvvuww9/+MM+1w8GgwgGO4+0/X4/AEBVVaiqakqbjO30d3tFLr0sdc3tprVhKEu2PlbD+sTH2iTG+iTG+sQ30Noks74khBBJbf08srL0o+eqqircdttt2Lt3Lx588EGsXbsWFRUVvdZftmwZli9f3mv5+vXr4XK5zGxav30dBJYfcMAuCfzf6RHYpLQ0g4iIKCmBQACLFi1Cc3MzPB5PwnVNDwBOpxPTpk3De++9F1v2wAMPYO/evXj//fd7rd9XD0BZWRkaGxvP2/j+UlUV1dXVmDNnDmRZPv/6EQ0Tlv0/AMCuR2ahMMdpSjuGqmTrYzWsT3ysTWKsT2KsT3wDrY3f70dRUVG/AoDppwBKS0sxYcKEbssuv/xy/Nd//Vef6yuKAkVRei2XZdn0F0R/tynLQFGuE42tIZwNhFGSn2NqO4aqVNQ8k7A+8bE2ibE+ibE+8SVbm2TWNf0ywJkzZ+Lo0aPdln366acYM2aM2btKKWM2QE4GREREmcj0APDTn/4Uu3btwq9+9SvU1NRg/fr1ePHFF1FZWWn2rlKqczIgXgpIRESZx/QAcM0112Djxo14+eWXMWnSJDzxxBNYtWoV7rzzTrN3lVKcDIiIiDKZ6WMAAOCWW27BLbfckopND5rOyYAYAIiIKPPwXgBxlOTpAxPZA0BERJmIASAOL28IREREGYwBIA5jOmBeBUBERJmIASAO4yqAcwEVHWokza0hIiIyFwNAHHnZMpwOvTxnWngpIBERZRYGgDgkSeqcC4CnAYiIKMMwACRQwoGARESUoRgAEvByICAREWUoBoAEvO7oXADsASAiogzDAJBA7FJADgIkIqIMwwCQQOyOgOwBICKiDMMAkIDRA3Da357mlhAREZmLASABX342AOB0Uwcimkhza4iIiMzDAJCA163AbpMQ1gQaWngagIiIMgcDQAIOuy02F8DJczwNQEREmYMB4DxGj9BPA5xsYgAgIqLMwQBwHqOiAeAr9gAQEVEGYQA4j9H57AEgIqLMwwBwHkYPAMcAEBFRJmEAOI9R+S4A7AEgIqLMwgBwHl17AITgXABERJQZUh4AnnrqKUiShMWLF6d6VylRGp0NsF2N4FxATXNriIiIzJHSALB371688MILmDJlSip3k1JZsh0jo3cF5DgAIiLKFI5Ubbi1tRV33nknfvvb3+KXv/xl3PWCwSCCwc677fn9fgCAqqpQVXOOuI3tDHR7vrwsnGkJ4nhjC8Z7Xaa0aSi50PpkOtYnPtYmMdYnMdYnvoHWJpn1JZGiE9sVFRUoKCjAv/3bv2HWrFm44oorsGrVql7rLVu2DMuXL++1fP369XC5hsY/23Wf2nDwrA0Lx0Rwg4/jAIiIaGgKBAJYtGgRmpub4fF4Eq6bkh6ADRs24MCBA9i7d+95112yZAmqqqpij/1+P8rKyjB37tzzNr6/VFVFdXU15syZA1mWk/7+w/ZPcXDnF8jzjcVNN403pU1DyYXWJ9OxPvGxNomxPomxPvENtDZGL3p/mB4ATpw4gQcffBDV1dXIyso67/qKokBRlF7LZVk2/QUx0G2WF+YAAE43BzP6RZqKmmcS1ic+1iYx1icx1ie+ZGuTzLqmB4D9+/ejoaEBV111VWxZJBLBjh078PzzzyMYDMJut5u925QaxdkAiYgow5geAG688UYcPny427K7774b48ePx8MPPzzs/vkDXeYCYAAgIqIMYXoAcLvdmDRpUrdlOTk5KCws7LV8uDB6AJoCKtqCYeQoKbt4goiIaFBwJsB+cGfJ8GTp//TZC0BERJlgUA5l33333cHYTUqNGuGC/7QfJ8+141KvO93NISIiuiDsAegn4zTAV+wBICKiDMAA0E+jeVtgIiLKIAwA/cRLAYmIKJMwAPRT522BA2luCRER0YVjAOgn9gAQEVEmYQDoJ6MHoN4fRDAcSXNriIiILgwDQD8V5jiRJevlOt3UkebWEBERXRgGgH6SJAk+ngYgIqIMwQCQhNg4AF4KSEREwxwDQBKMuQA4GRAREQ13DABJGD3CBYA9AERENPwxACSh81JAzgVARETDGwNAEmKTAfEUABERDXMMAEkwegBON3Ugook0t4aIiGjgGACS4PVkwWGTENYEGlo4FwAREQ1fDABJsNsklORlAeBAQCIiGt4YAJLEewIQEVEmYABIkjEQ8Cv2ABAR0TDGAJCk0ewBICKiDMAAkKTYpYDsASAiomHM9ACwYsUKXHPNNXC73SguLsbChQtx9OhRs3eTNqPyo7MBsgeAiIiGMdMDwPbt21FZWYldu3ahuroaqqpi7ty5aGtrM3tXadG1B0AIzgVARETDk8PsDW7evLnb43Xr1qG4uBj79+/Ht7/97V7rB4NBBIPB2GO/3w8AUFUVqqqa0iZjO2Zsb6TLDgBoVyNoaA6gIMd5wdtMNzPrk4lYn/hYm8RYn8RYn/gGWptk1pdEig9ja2pqMG7cOBw+fBiTJk3q9fyyZcuwfPnyXsvXr18Pl8uVyqYN2NJ9dvhVCf97chhlueluDRERkS4QCGDRokVobm6Gx+NJuG5KA4CmafjOd76DpqYm7Ny5s891+uoBKCsrQ2Nj43kb31+qqqK6uhpz5syBLMsXvL3vvrAbf/uqGc/fMRXzJnpNaGF6mV2fTMP6xMfaJMb6JMb6xDfQ2vj9fhQVFfUrAJh+CqCryspKHDlyJO4/fwBQFAWKovRaLsuy6S8Is7Y5usCFv33VjLqWUEa9aFNR80zC+sTH2iTG+iTG+sSXbG2SWTdlAeD+++/HG2+8gR07dmD06NGp2k1acC4AIiIa7kwPAEII/OQnP8HGjRvx7rvvYuzYsWbvIu04FwAREQ13pgeAyspKrF+/Hq+99hrcbjfq6uoAAHl5ecjOzjZ7d2nB+wEQEdFwZ/o8AGvWrEFzczNmzZqF0tLS2Mcrr7xi9q7SJtYDwABARETDVEpOAWQ6owegKaCiLRhGjpLSsZRERESm470ABsCdJcOTpf/TZy8AERENRwwAAzRqRPSeABwISEREwxADwAAZpwG+OhdIc0uIiIiSxwAwQKOjAwG/4ikAIiIahhgABih2KSBPARAR0TDEADBAo3kpIBERDWMMAANkzAVw/GwAR042IxTW0twiIiKi/uMF7ANUFr0K4Ou2EG55bidku4RxxW5M9Hn0j1F5+MbIXHiyZdhtUppbS0RE1B0DwACNyHHiiYWT8OYHp/HhqWb4O8L46LQfH53249X9netJEuBWHMhzycjL1j/ys53Id8kodmfB61HgzcuCN/r1CJcTNpuEiCZwtjWIOn8HTjd3oN7fgbrmDnzdFsKkUXn4X5NKUJTb+y6KRERE/cEAcAHu+uYY3PXNMRBC4Ktz7fjwlB8fnWrGh6f8+PCUH3X+DggB+DvC8HeEcQLnHy8g2yXkZcs4F1AR0eLMqrj3BB597Qi+eXEhbppcyjBARERJYwAwgSRJKCtwoazAhf81qSS2PBTW4O9Q0RRQ0dyuwt+uf24KhPB1QMWZlg7U+4Oo9+tH+I2tIagRgcbWEADAJgEj3QpKPFnwerJQkpeFXMWBnTWN+OCrZrz32Vm899nZbmFgzgQvvJ6sdJWCiIiGCQaAFHI6bCjKVfp9dK5GNJxpCeJcIITCHAVFuU447L3Haf4cwJdnA3jzyGm8efh0tzDwi01HMK44F9eNK8J13yjC9IsLkct7FRARUQ/8zzCEyHYbfPnZ8OWf/7bJ5YUu3HP9Jbjn+ktw4usA3jx8Gm8eqcMHXzXhWEMrjjW04qW/fgGHTcJV5SMw8xtFmH5xASb4PPBkyYPw0xAR0VDGAJABygpc+PH1l+DH11+CpkAI7312FjtrGrHzWCO+/DqAPV98jT1ffB1bf0yhK3q1Qh4m+jy4rNiVxtYTEVE6MABkmHyXEzdNLsVNk0sB6KcKdtY04q81jTh0ogknm9px/GwAx88G8Obhutj3ZdntePyDd+F02PQPu/5ZttuQLdtRkONEQY4TI3KcKIx+XZCjX80gQYIWvQ20EIAmBIzhi3nZMopynchVHJAkXg5JRDRUMABkuPJCFxYVlmPR9HIAwLm2UPQqhWYciX6ubWxDR0RCR1soZe1QYuMhnCjKVVCY60S2bIfdZoPdBthsEhw2CXZJgt2mhw+X045spx2u6Ee27IDLaYfTYYMkARKk6Gf9cksAsNtsyFHsyFUcyJbtDB1ERHEwAFjMiBynPkBwXFFsWVNrO159YwtmzPwWhGRHKBJBMKwhFP1oVyP4ui3U7eNs9HNzuwqg85+wTZKiX+v/eM8FQgiE9O2dbGof1KmTbRKQoziQqziQo+jhQY0IhMIRhCKdP18orEHVBBSHLbZujtOuf1YcyHbYcPKUDX95+RDUiIjVJhjWEAxHYJMk5CoOuBQHchU7XE59ny6nHbLdhrCmIRwRUCMCYU3TP0c0hDWh7z+iQY22x/gcEQI5TgfcWTI8WQ54smW4sxxwZzmQq8jQhL4NNSKgahrUsIAa0aBqGiREw5QRquzGYxuEENCEQERDdBsCESGgaQICAnZJgi0axGzRbdijv8sONYKOcAQdqqZ/rWpoD6k4cdKGzf6/IcvpgGyXYj1HRk+SBKDrBa1CdP8dGfuz240A2Plh6/o4+rXNJkEIgUAogkAogvZQGO2q8bX+u5VtNsgOCQ6b0ZOlf223SQiGI2gPafrPEoqgXY2gQ9Vfo0aPV7bTjizZjixZf5wl2yEB0AQQEQJCCEQ0AS3a4xUKa2iPbsvYXiAUQSAYht9vx0tf7YbLqYfSLKdd34dsh+KwxX5u43dkt+lBNvY7tOuf5Wj7HXa9LrHfn6a/BiJa52ObTYLTLkG22+Cw6z+/067/XiSpe0+dECL6WP+d6H/H+t+whOjfdDRHq9HtG69p4/UshIDdpu9Hju7HYZcg2/WfKRTWXzPtagTBWI00BIIqPv/Shs+3fYZsRY71PDodNigOG2xdAnzPi6KNdgtEP/fofTTeh/SfBbFtSZLU6+9Af4xoDbv/XYU1/W80HNH3Y2xPkqTYgYixTN9v54EJYu+H3dczvtd4v7xlqi92f5nBxABAyFEcKM4GLitxQ5bNHyAYCIVxtjWEM61BnG0NobE1iLOtQQTD+j9BTTPeVPQ/SuMfY3sogkAorL+xd3mDD4Y1wPijh/5GAOhfhyMCbaFw7A2tpSOMlo5wv9oZCmsJ1rUBjQ1mlCMD2XD4XH26GzGESTgZaE53I4YwG7ac/CzdjUirKaPzGQAoM7mcDrgKHCgrGJzBhsbRYVswjJZgGG3BMFqDYbSHInDYO8c3KF3GOzjsEoJhLbZuIBhBW0j/uqU9hE8+/hhXTJkElyJDiX6v4tBPR0Q0gUAojNagvs+2kL7PtmAEakQ/qnTYJMgOG2SbBIdxdGSzdTtalu0SlOhjm01CWzAMf3sYLR0qWjrC8Ec/t3SEYbch9rPIdin2tSM67XQ4epTW9ahQ1QQkIHZk7bAZR/mAXZIgSfoMlF2PcI0jJADIkvWfWZFtyHLoR8WyTeDoxx9h/ISJ0IQU61np2qPRU9fTMkbgM0Kgsf+uyzSjLQKxZTYbYqeEjCP2bKcdLtkO2WHrPIrr0tOiRjRoQkCJtl3/Plv0SF8/Gg+FNXSEtVjPQLtqhM6IfvRn04/ijB4So4fCabfpbTDaEt2m0yawd+9eTLnyaoQ0vRelPRRBRzTgdoQj0KLhN9LjI6x1P9o2foeR6GO78VqK9hQYR9t2SQ+/apcaGB+hiH6o3PMI1BY7ZAUQPao2ejf0o2v9NWDszxHtUXF06ZEIdzl61o+Yo71dmoDTYUN2l96UbNkORbbDaQdqa79A6ehyhDVEXz+RWM+YpvV87XR/3LV3Qup2tK0fEGhd2t75s+h/A8bPb7dJsR4Pm4RYD0bPvytHl94TowfBONAw/kaM959oGWO9XZ29LV17XES0jQIj3emZyI0BgDKOJEmx7vtiE7anqirebP4IN11blpIekuFMVVW8ee5D3PTNctamD6qqouWYwOzLi1mfPqiqijff/Bw33TSB9UmDlN0NcPXq1bjooouQlZWF6dOnY8+ePanaFRERESUpJQHglVdeQVVVFR577DEcOHAAU6dOxbx589DQwHOoREREQ0FKTgGsXLkSP/zhD3H33XcDANauXYu//OUv+I//+A888sgj3dYNBoMIBoOxx36/H4DeNaSqqintMbZj1vYyDeuTGOsTH2uTGOuTGOsT30Brk8z6khBdRi+YIBQKweVy4U9/+hMWLlwYW15RUYGmpia89tpr3dZftmwZli9f3ms769evh8vFGeqIiIj6KxAIYNGiRWhubobH40m4ruk9AI2NjYhEIvB6vd2We71efPLJJ73WX7JkCaqqqmKP/X4/ysrKMHfu3PM2vr9UVUV1dTXmzJnDgSZ9YH0SY33iY20SY30SY33iG2htjF70/kj7VQCKokBRel8CIcuy6S+IVGwzk7A+ibE+8bE2ibE+ibE+8SVbm2TWNX0QYFFREex2O+rru08MUl9fj5KSErN3R0RERANgeg+A0+nE1Vdfja1bt8bGAGiahq1bt+L+++8/7/cbQxKS6cY4H1VVEQgE4Pf7mTL7wPokxvrEx9okxvokxvrEN9DaGP87+zO8LyWnAKqqqlBRUYFp06bh2muvxapVq9DW1ha7KiCRlpYWAEBZWVkqmkZERJTxWlpakJeXl3CdlASA22+/HWfOnMGjjz6Kuro6XHHFFdi8eXOvgYF98fl8OHHiBNxut2l3cjMGFp44ccK0gYWZhPVJjPWJj7VJjPVJjPWJb6C1EUKgpaUFPp/vvOuafhngUOT3+5GXl9evyyKsiPVJjPWJj7VJjPVJjPWJbzBqk7KpgImIiGjoYgAgIiKyIEsEAEVR8Nhjj/U53wCxPufD+sTH2iTG+iTG+sQ3GLWxxBgAIiIi6s4SPQBERETUHQMAERGRBTEAEBERWRADABERkQUxABAREVmQJQLA6tWrcdFFFyErKwvTp0/Hnj170t2ktNixYwcWLFgAn88HSZKwadOmbs8LIfDoo4+itLQU2dnZmD17No4dO5aexg6yFStW4JprroHb7UZxcTEWLlyIo0ePdluno6MDlZWVKCwsRG5uLv7pn/6p110vM9WaNWswZcoUeDweeDwezJgxA2+99VbseSvXpqennnoKkiRh8eLFsWVWrs+yZcsgSVK3j/Hjx8eet3JtDCdPnsT3vvc9FBYWIjs7G5MnT8a+fftiz6fqvTnjA8Arr7yCqqoqPPbYYzhw4ACmTp2KefPmoaGhId1NG3RtbW2YOnUqVq9e3efzzzzzDJ599lmsXbsWu3fvRk5ODubNm4eOjo5Bbung2759OyorK7Fr1y5UV1dDVVXMnTsXbW1tsXV++tOf4s9//jNeffVVbN++HadOncI//uM/prHVg2f06NF46qmnsH//fuzbtw9///d/j1tvvRUffvghAGvXpqu9e/fihRdewJQpU7ott3p9Jk6ciNOnT8c+du7cGXvO6rU5d+4cZs6cCVmW8dZbb+Gjjz7Cv/7rv2LEiBGxdVL23iwy3LXXXisqKytjjyORiPD5fGLFihVpbFX6ARAbN26MPdY0TZSUlIhf//rXsWVNTU1CURTx8ssvp6GF6dXQ0CAAiO3btwsh9FrIsixeffXV2Doff/yxACDef//9dDUzrUaMGCH+/d//nbWJamlpEePGjRPV1dXi+uuvFw8++KAQgq+dxx57TEydOrXP56xeGyGEePjhh8V1110X9/lUvjdndA9AKBTC/v37MXv27Ngym82G2bNn4/33309jy4ae2tpa1NXVdatVXl4epk+fbslaNTc3AwAKCgoAAPv374eqqt3qM378eJSXl1uuPpFIBBs2bEBbWxtmzJjB2kRVVlbi5ptv7lYHgK8dADh27Bh8Ph8uvvhi3Hnnnfjyyy8BsDYA8Prrr2PatGm47bbbUFxcjCuvvBK//e1vY8+n8r05owNAY2MjIpFIr9sQe71e1NXVpalVQ5NRD9YK0DQNixcvxsyZMzFp0iQAen2cTify8/O7rWul+hw+fBi5ublQFAX33HMPNm7ciAkTJrA2ADZs2IADBw5gxYoVvZ6zen2mT5+OdevWYfPmzVizZg1qa2vxrW99Cy0tLZavDQB8/vnnWLNmDcaNG4e3334b9957Lx544AH87ne/A5Da92bHBX03UQaqrKzEkSNHup2nJOCyyy7DoUOH0NzcjD/96U+oqKjA9u3b092stDtx4gQefPBBVFdXIysrK93NGXLmz58f+3rKlCmYPn06xowZgz/+8Y/Izs5OY8uGBk3TMG3aNPzqV78CAFx55ZU4cuQI1q5di4qKipTuO6N7AIqKimC323uNKK2vr0dJSUmaWjU0GfWweq3uv/9+vPHGG9i2bRtGjx4dW15SUoJQKISmpqZu61upPk6nE9/4xjdw9dVXY8WKFZg6dSp+85vfWL42+/fvR0NDA6666io4HA44HA5s374dzz77LBwOB7xer6Xr01N+fj4uvfRS1NTUWP61AwClpaWYMGFCt2WXX3557DRJKt+bMzoAOJ1OXH311di6dWtsmaZp2Lp1K2bMmJHGlg09Y8eORUlJSbda+f1+7N692xK1EkLg/vvvx8aNG/HOO+9g7Nix3Z6/+uqrIctyt/ocPXoUX375pSXq0xdN0xAMBi1fmxtvvBGHDx/GoUOHYh/Tpk3DnXfeGfvayvXpqbW1FZ999hlKS0st/9oBgJkzZ/a65PjTTz/FmDFjAKT4vfmChhAOAxs2bBCKooh169aJjz76SPzoRz8S+fn5oq6uLt1NG3QtLS3i4MGD4uDBgwKAWLlypTh48KA4fvy4EEKIp556SuTn54vXXntNfPDBB+LWW28VY8eOFe3t7Wlueerde++9Ii8vT7z77rvi9OnTsY9AIBBb55577hHl5eXinXfeEfv27RMzZswQM2bMSGOrB88jjzwitm/fLmpra8UHH3wgHnnkESFJktiyZYsQwtq16UvXqwCEsHZ9fvazn4l3331X1NbWir/+9a9i9uzZoqioSDQ0NAghrF0bIYTYs2ePcDgc4sknnxTHjh0Tf/jDH4TL5RK///3vY+uk6r054wOAEEI899xzory8XDidTnHttdeKXbt2pbtJabFt2zYBoNdHRUWFEEK/3GTp0qXC6/UKRVHEjTfeKI4ePZreRg+SvuoCQLz00kuxddrb28V9990nRowYIVwul/iHf/gHcfr06fQ1ehD94Ac/EGPGjBFOp1OMHDlS3HjjjbF//kJYuzZ96RkArFyf22+/XZSWlgqn0ylGjRolbr/9dlFTUxN73sq1Mfz5z38WkyZNEoqiiPHjx4sXX3yx2/Opem+WhBDiwvoQiIiIaLjJ6DEARERE1DcGACIiIgtiACAiIrIgBgAiIiILYgAgIiKyIAYAIiIiC2IAICIisiAGACIiIgtiACAiIrIgBgAiIiILYgAgIiKyoP8P+pU1Ey4jfYEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Evaluation loss\n",
    "pd.Series(training_objective.evaluations_history[\"eval\"][\"loss\"]).plot(figsize=(6, 3), grid=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78366658-99df-4222-a2f6-4f847205949a",
   "metadata": {
    "tags": [],
    "id": "78366658-99df-4222-a2f6-4f847205949a",
    "outputId": "ff31054a-a5fc-4090-b1f7-74063865e364"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAESCAYAAAB+cOZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/t0lEQVR4nO3de3xT9f0/8NfJtU3b9N6mhbbcCuV+FSwyhXHHoaBDLTqYc/idX/hOv/3qFKcORFc3N7wy2fxOcV9FkN+ETadILQIyyqWFqgUppfQCbdPSljZN0yZpcn5/lAZKT0oCSUOT1/Px6ENyLp988jZtXvmczzlHEEVRBBEREdFVyHzdASIiIuobGBqIiIjIJQwNRERE5BKGBiIiInIJQwMRERG5hKGBiIiIXMLQQERERC5R+LoDnmC321FVVYWwsDAIguDr7hAREfUZoiiiubkZiYmJkMl6Hkvwi9BQVVWFpKQkX3eDiIiozzp79iz69+/f4zZ+ERrCwsIAdLxgrVbrsXatVit27dqFOXPmQKlUeqzdvo51cY61kca6SGNdnGNtpHmjLgaDAUlJSY7P0p74RWjoPCSh1Wo9Hho0Gg20Wi3ftJdhXZxjbaSxLtJYF+dYG2nerIsrh/c5EZKIiIhcwtBARERELmFoICIiIpcwNBAREZFLGBqIiIjIJQwNRERE5BKGBhc1tVpx7oLJ190gIiLyGYYGFy19+yBm/GEP6oxmX3eFiIjIJxgaXNBibsfxKgOsNhEVDRxtICKiwMTQ4IIzdS2OfxtarT7sCRERke+4HRr27duHhQsXIjExEYIgYMeOHV3WC4Ig+fPyyy87bXPNmjXdtk9LS3P7xXhLyflLoaG5rd2HPSEiIvIdt0NDS0sLxo4diw0bNkiur66u7vLzzjvvQBAE3H333T22O3LkyC777d+/392uec3locHQxpEGIiIKTG7fsGr+/PmYP3++0/U6na7L43/84x+YMWMGBg0a1HNHFIpu+94ouoSGVo40EBFRYPLqXS5ramrwr3/9C++9995Vty0uLkZiYiKCgoKQnp6OrKwsJCcnS25rNpthNl86i8FgMADouPuX1eq5kYDOtk7XGh3Lmkxmjz5HX9T5+gO9DlJYG2msizTWxTnWRpo36uJOW4IoiuK1PpEgCNi+fTsWLVokuf73v/89XnrpJVRVVSEoKMhpO59//jmMRiOGDRuG6upqrF27FpWVlSgsLJS8v/eaNWuwdu3abss3b94MjUZzrS9HUrsdeOKQHHZ03DL0lng77hlk9+hzEBER+YrJZMLSpUvR1NQErVbb47ZeDQ1paWmYPXs23njjDbfabWxsREpKCtavX4+HHnqo23qpkYakpCTU1dVd9QW7w2q14v/+kY2sby4NyCwco8P6JWM89hx9kdVqRXZ2NmbPns373F+BtZHGukhjXZxjbaR5oy4GgwExMTEuhQavHZ74+uuvUVRUhK1bt7q9b0REBIYOHYrTp09Lrler1VCr1d2WK5VKj7+59K1Cl8dGs41v4Iu8UW9/wdpIY12ksS7OsTbSPFkXd9rx2nUa/vrXv2LixIkYO3as2/sajUaUlJQgISHBCz1zT01rx39D1R35ysBTLomIKEC5HRqMRiMKCgpQUFAAACgtLUVBQQEqKioc2xgMBmzbtg0///nPJduYOXMm3nzzTcfjxx9/HHv37kVZWRkOHDiAxYsXQy6XIyMjw93ueVzNxZGGcUkRAIBmnnJJREQByu3DE3l5eZgxY4bjcWZmJgBg+fLl2LRpEwBgy5YtEEXR6Yd+SUkJ6urqHI/PnTuHjIwM1NfXIzY2FtOmTcPBgwcRGxvrbvc8rjM0jE+OwP7TdTzlkoiIApbboWH69Om42tzJhx9+GA8//LDT9WVlZV0eb9myxd1u9Aq7XUTtxcMTE5IjAfDiTkREFLh474keVBvaYLELUMoFjOzXMaPUZLGh3cZTLomIKPAwNPSg80qQKVEaRGpUjuW8/wQREQUihoYedIaGwbEhUMpl0KjkAHiIgoiIAhNDQw9KzndcPnpwbCgAICyoYwoIRxqIiCgQMTT04PKRBgDQBnVcAMPQypEGIiIKPAwNPbgyNHSONPACT0REFIgYGpyoN5pxwdQxojAwpuMmWNrgiyMNnNNAREQBiKHBic7bYUepRWhUHSMMPDxBRESBjKHBidMXJ0HGB1+6kBUnQhIRUSBjaHBCLggYEK1BgubSMh6eICKiQMbQ4MR9k5OR/dg03JF86eqPlw5PcKSBiIgCD0PDVQjCpX9fOjzBkQYiIgo8DA1u4OEJIiIKZAwNbtB2XqeBhyeIiCgAMTS4IezinIZmM0caiIgo8DA0uCE8mCMNREQUuBga3OAYaWizQhTFq2xNRETkXxga3NB5yqVdBFosNh/3hoiIqHe5HRr27duHhQsXIjExEYIgYMeOHV3W//SnP4UgCF1+5s2bd9V2N2zYgAEDBiAoKAhTpkzB4cOH3e2a1wUpZVDKO87B5KWkiYgo0LgdGlpaWjB27Fhs2LDB6Tbz5s1DdXW14+fDDz/ssc2tW7ciMzMTv/nNb3D06FGMHTsWc+fORW1trbvd8ypBEC47RMF5DUREFFgU7u4wf/58zJ8/v8dt1Go1dDqdy22uX78eK1aswIMPPggA2LhxI/71r3/hnXfewVNPPeVuF71KG6RAQ4uF12ogIqKA43ZocMWePXsQFxeHyMhI/PCHP8QLL7yA6OhoyW0tFgvy8/OxevVqxzKZTIZZs2YhNzdXch+z2Qyz2ex4bDAYAABWqxVWq+c+zDvburzNzqtCNhjbPPpcfYlUXagDayONdZHGujjH2kjzRl3cacvjoWHevHm46667MHDgQJSUlODpp5/G/PnzkZubC7lc3m37uro62Gw2xMfHd1keHx+PkydPSj5HVlYW1q5d2235rl27oNFoJPa4PtnZ2Y5/m40yADLsP5iHtpLAPoPi8rpQV6yNNNZFGuviHGsjzZN1MZlMLm/r8dBw3333Of49evRojBkzBoMHD8aePXswc+ZMjzzH6tWrkZmZ6XhsMBiQlJSEOXPmQKvVeuQ5gI70lZ2djdmzZ0Op7JjL8FlTAU411WJQ2kgsmJLssefqS6TqQh1YG2msizTWxTnWRpo36tI5Wu8KrxyeuNygQYMQExOD06dPS4aGmJgYyOVy1NTUdFleU1PjdF6EWq2GWq3utlypVHrlzXV5u+EaFQCgxWIP+Deyt+rtD1gbaayLNNbFOdZGmifr4k47Xr9Ow7lz51BfX4+EhATJ9SqVChMnTkROTo5jmd1uR05ODtLT073dPbdpefYEEREFKLdDg9FoREFBAQoKCgAApaWlKCgoQEVFBYxGI5544gkcPHgQZWVlyMnJwZ133okhQ4Zg7ty5jjZmzpyJN9980/E4MzMTb7/9Nt577z18//33eOSRR9DS0uI4m+JGwjtdEhFRoHL78EReXh5mzJjheNw5t2D58uV466238O233+K9995DY2MjEhMTMWfOHKxbt67L4YSSkhLU1dU5Ht977704f/48nnvuOej1eowbNw47d+7sNjnyRtB59oSBIw1ERBRg3A4N06dP7/G+C1988cVV2ygrK+u2bNWqVVi1apW73el1nYcneEVIIiIKNLz3hJsuHZ7gSAMREQUWhgY3dR6eaOacBiIiCjAMDW66dHiCIw1ERBRYGBrcpA3unAjJkQYiIgosDA1u6rzLpaXdjjarzce9ISIi6j0MDW4KUysgCB3/5gWeiIgokDA0uEkmExCq4iEKIiIKPAwN16DztEuONBARUSBhaLgGjqtC8gJPREQUQBgargFvWkVERIGIoeEa8LRLIiIKRAwN14D3nyAiokDE0HANLl1KmocniIgocDA0XINLN63iSAMREQUOhoZrwLMniIgoEDE0XIOIYBUAoL7F4uOeEBER9R6GhmswMDYEAHDmfIuPe0JERNR7GBquwZDYUABAZWMrjGZOhiQiosDA0HANIkNUiAlVAwBKao0+7g0REVHvcDs07Nu3DwsXLkRiYiIEQcCOHTsc66xWK5588kmMHj0aISEhSExMxLJly1BVVdVjm2vWrIEgCF1+0tLS3H4xvSk1rmO0oZihgYiIAoTboaGlpQVjx47Fhg0buq0zmUw4evQonn32WRw9ehQff/wxioqKcMcdd1y13ZEjR6K6utrxs3//fne71qtS4ztDQ7OPe0JERNQ7FO7uMH/+fMyfP19yXXh4OLKzs7sse/PNNzF58mRUVFQgOTnZeUcUCuh0One74zOdIw2nazjSQEREgcHt0OCupqYmCIKAiIiIHrcrLi5GYmIigoKCkJ6ejqysLKchw2w2w2w2Ox4bDAYAHYdHrFbPXTuhsy2pNgdGBwMATtU0e/Q5+4Ke6hLoWBtprIs01sU51kaaN+riTluCKIritT6RIAjYvn07Fi1aJLm+ra0Nt9xyC9LS0vDBBx84befzzz+H0WjEsGHDUF1djbVr16KyshKFhYUICwvrtv2aNWuwdu3abss3b94MjUZzrS/HLc1W4Jk8BQSI+P1kG1TyXnlaIiIijzKZTFi6dCmampqg1Wp73NZrocFqteLuu+/GuXPnsGfPnqt25HKNjY1ISUnB+vXr8dBDD3VbLzXSkJSUhLq6Oree52qsViuys7Mxe/ZsKJXKLutEUcSUl/bggsmKHY/cjJGJnnveG11PdQl0rI001kUa6+IcayPNG3UxGAyIiYlxKTR45fCE1WrFPffcg/LycuzevdvtD/KIiAgMHToUp0+fllyvVquhVqu7LVcqlV55czlrNzUuDIfLGlDW0IZxKdEef94bnbfq7Q9YG2msizTWxTnWRpon6+JOOx6/TkNnYCguLsaXX36J6Gj3P0yNRiNKSkqQkJDg6e551BCeQUFERAHE7dBgNBpRUFCAgoICAEBpaSkKCgpQUVEBq9WKH//4x8jLy8MHH3wAm80GvV4PvV4Pi+XSfRpmzpyJN9980/H48ccfx969e1FWVoYDBw5g8eLFkMvlyMjIuP5X6EWOazXwDAoiIgoAbh+eyMvLw4wZMxyPMzMzAQDLly/HmjVr8M9//hMAMG7cuC77ffXVV5g+fToAoKSkBHV1dY51586dQ0ZGBurr6xEbG4tp06bh4MGDiI2Ndbd7vSo1rmOS5mle4ImIiAKA26Fh+vTp6GnupCvzKsvKyro83rJli7vduCF0XuCpvMEEc7sNagVPoSAiIv/Fe09ch7gwNcKCFLDZRZTVmXzdHSIiIq9iaLgOgiBcdg8KToYkIiL/xtBwnTrnNXAyJBER+TuGhuvUOa+BkyGJiMjfMTRcp8E8PEFERAGCoeE6dc5pKK1rgdVm93FviIiIvIeh4TolhgdDo5LDahNRXs8zKIiIyH8xNFwnmUxA/8iO22TXGNp83BsiIiLvYWjwgEiNCgDQ0GK5ypZERER9F0ODB3SGhgsmhgYiIvJfDA0eEBlyMTS0WH3cEyIiIu9haPCAqJCOe5FzpIGIiPwZQ4MHcE4DEREFAoYGD4gK4ZwGIiLyfwwNHsCJkEREFAgYGjyAEyGJiCgQMDR4QBTnNBARUQBgaPCAyItnT7RabWi12HzcGyIiIu9gaPCAULUCSrkAgPMaiIjIf7kdGvbt24eFCxciMTERgiBgx44dXdaLoojnnnsOCQkJCA4OxqxZs1BcXHzVdjds2IABAwYgKCgIU6ZMweHDh93tms8IgoAIToYkIiI/53ZoaGlpwdixY7FhwwbJ9b///e/x+uuvY+PGjTh06BBCQkIwd+5ctLU5v5nT1q1bkZmZid/85jc4evQoxo4di7lz56K2ttbd7vlM57wGToYkIiJ/5XZomD9/Pl544QUsXry42zpRFPHqq6/imWeewZ133okxY8bgb3/7G6qqqrqNSFxu/fr1WLFiBR588EGMGDECGzduhEajwTvvvONu93ymc15DA0caiIjITyk82VhpaSn0ej1mzZrlWBYeHo4pU6YgNzcX9913X7d9LBYL8vPzsXr1ascymUyGWbNmITc3V/J5zGYzzGaz47HBYAAAWK1WWK2e+6bf2ZYrbUYEd4SGOkOrR/twI3KnLoGGtZHGukhjXZxjbaR5oy7utOXR0KDX6wEA8fHxXZbHx8c71l2prq4ONptNcp+TJ09K7pOVlYW1a9d2W75r1y5oNJpr6XqPsrOzr7qNoU4GQIbD35xAdEOhx/twI3KlLoGKtZHGukhjXZxjbaR5si4mk8nlbT0aGnrL6tWrkZmZ6XhsMBiQlJSEOXPmQKvVeux5rFYrsrOzMXv2bCiVyh63PfllMQ7UlCK2XwoWLBjusT7ciNypS6BhbaSxLtJYF+dYG2neqEvnaL0rPBoadDodAKCmpgYJCQmO5TU1NRg3bpzkPjExMZDL5aipqemyvKamxtHeldRqNdRqdbflSqXSK28uV9qNCQsGADS22QLmDe6tevsD1kYa6yKNdXGOtZHmybq4045Hr9MwcOBA6HQ65OTkOJYZDAYcOnQI6enpkvuoVCpMnDixyz52ux05OTlO97kROW6PzatCEhGRn3J7pMFoNOL06dOOx6WlpSgoKEBUVBSSk5Px2GOP4YUXXkBqaioGDhyIZ599FomJiVi0aJFjn5kzZ2Lx4sVYtWoVACAzMxPLly/HpEmTMHnyZLz66qtoaWnBgw8+eP2vsJfw9thEROTv3A4NeXl5mDFjhuNx59yC5cuXY9OmTfjVr36FlpYWPPzww2hsbMS0adOwc+dOBAUFOfYpKSlBXV2d4/G9996L8+fP47nnnoNer8e4ceOwc+fObpMjb2S80yUREfk7t0PD9OnTIYqi0/WCIOD555/H888/73SbsrKybstWrVrlGHnoi6JCGBqIiMi/8d4THtJ5e+w2q503rSIiIr/E0OAhISo5VPKOcvKqkERE5I8YGjxEEATHpaR5BgUREfkjhgYP4mRIIiLyZwwNHsTTLomIyJ8xNHiQ4wwKhgYiIvJDDA0edOn22LwrGxER+R+GBg+K0nCkgYiI/BdDgwdFcCIkERH5MYYGD+JVIYmIyJ8xNHhQ51UhG1o4p4GIiPwPQ4MHcU4DERH5M4YGD4rQdJ49Yenxpl5ERER9EUODB3XOabC029Fq5U2riIjIvzA0eJBGJYdKcfGmVTxEQUREfoahwYMEQbhsXgMnQxIRkX9haPAwxxkUPO2SiIj8DEODh0VqeHtsIiLyTx4PDQMGDIAgCN1+Vq5cKbn9pk2bum0bFBTk6W71mkhe4ImIiPyUwtMNHjlyBDbbpTMHCgsLMXv2bCxZssTpPlqtFkVFRY7HgiB4ulu9htdqICIif+Xx0BAbG9vl8UsvvYTBgwfjtttuc7qPIAjQ6XSe7opPcE4DERH5K4+HhstZLBa8//77yMzM7HH0wGg0IiUlBXa7HRMmTMBvf/tbjBw50un2ZrMZZrPZ8dhgMAAArFYrrFbPnbXQ2ZY7bYYHyQEA9c1mj/blRnItdQkUrI001kUa6+IcayPNG3Vxpy1B9OKlCz/66CMsXboUFRUVSExMlNwmNzcXxcXFGDNmDJqamvCHP/wB+/btw/Hjx9G/f3/JfdasWYO1a9d2W75582ZoNBqPvgZ3Ha0T8F6xHIPDRPxyFC/wRERENzaTyYSlS5eiqakJWq22x229Ghrmzp0LlUqFTz75xOV9rFYrhg8fjoyMDKxbt05yG6mRhqSkJNTV1V31BbvDarUiOzsbs2fPhlKpdGmf7yqbcNfGQ4gOUeHgU9M91pcbybXUJVCwNtJYF2msi3OsjTRv1MVgMCAmJsal0OC1wxPl5eX48ssv8fHHH7u1n1KpxPjx43H69Gmn26jVaqjVasl9vfHmcqfdYQkRAID6FgtarCIiLk6M9Efeqrc/YG2ksS7SWBfnWBtpnqyLO+147ToN7777LuLi4nD77be7tZ/NZsN3332HhIQEL/XMu0LUCiSGd5wyerrW6OPeEBEReY5XQoPdbse7776L5cuXQ6HoOpixbNkyrF692vH4+eefx65du3DmzBkcPXoUDzzwAMrLy/Hzn//cG13rFYPjQgEAxQwNRETkR7xyeOLLL79ERUUFfvazn3VbV1FRAZnsUla5cOECVqxYAb1ej8jISEycOBEHDhzAiBEjvNG1XpEaF4avi+s40kBERH7FK6Fhzpw5cDa/cs+ePV0ev/LKK3jllVe80Q2fGXJxpIGhgYiI/AnvPeEFDA1EROSPGBq8oDM0VDa2wmRp93FviIiIPIOhwQuiQlSIung56TPnW3zcGyIiIs9gaPCSIbGdZ1A0+7gnREREnsHQ4CVD4jmvgYiI/AtDg5d0jjQwNBARkb9gaPASnkFBRET+hqHBSzpDQ3m9CZZ2u497Q0REdP0YGrwkITwIISo52u0iyut5BgUREfV9DA1eIggCD1EQEZFfYWjwosEMDURE5EcYGrzIMdJwnqGBiIj6PoYGL+Jpl0RE5E8YGryoc6Sh5LwRdrv0XT+JiIj6CoYGL0qO0kAll6HNakdlY6uvu0NERHRdGBq8SCGXISVaAwAoreNpl0RE1LcxNHhZnFYNAKgzmn3cEyIiouvD0OBlMaEdoaHeaPFxT4iIiK4PQ4OXdYYGjjQQEVFf5/HQsGbNGgiC0OUnLS2tx322bduGtLQ0BAUFYfTo0fjss8883S2fiQ5VAQDqONJARER9nFdGGkaOHInq6mrHz/79+51ue+DAAWRkZOChhx7CsWPHsGjRIixatAiFhYXe6Fqv40gDERH5C6+EBoVCAZ1O5/iJiYlxuu1rr72GefPm4YknnsDw4cOxbt06TJgwAW+++aY3utbrYhwjDQwNRETUtym80WhxcTESExMRFBSE9PR0ZGVlITk5WXLb3NxcZGZmdlk2d+5c7Nixw2n7ZrMZZvOlD2GDwQAAsFqtsFqt1/8CLups63rajAiSA+gIDZ7smy95oi7+irWRxrpIY12cY22keaMu7rQliKLo0UsVfv755zAajRg2bBiqq6uxdu1aVFZWorCwEGFhYd22V6lUeO+995CRkeFY9qc//Qlr165FTU2N5HOsWbMGa9eu7bZ88+bN0Gg0nnsxHtBgBtYeVUAuiPjjFBsEwdc9IiIiusRkMmHp0qVoamqCVqvtcVuPjzTMnz/f8e8xY8ZgypQpSElJwUcffYSHHnrII8+xevXqLqMTBoMBSUlJmDNnzlVfsDusViuys7Mxe/ZsKJXKa2rDbLVh7dEc2EQB0344G+HB19bOjcQTdfFXrI001kUa6+IcayPNG3XpHK13hVcOT1wuIiICQ4cOxenTpyXX63S6biMKNTU10Ol0TttUq9VQq9XdliuVSq+8ua6nXaVSiTC1As3mdjS22RGj9Z83v7fq7Q9YG2msizTWxTnWRpon6+JOO16/ToPRaERJSQkSEhIk16enpyMnJ6fLsuzsbKSnp3u7a70mJqzzAk+cDElERH2Xx0PD448/jr1796KsrAwHDhzA4sWLIZfLHXMWli1bhtWrVzu2f/TRR7Fz50788Y9/xMmTJ7FmzRrk5eVh1apVnu6az8TwWg1EROQHPH544ty5c8jIyEB9fT1iY2Mxbdo0HDx4ELGxsQCAiooKyGSXssrUqVOxefNmPPPMM3j66aeRmpqKHTt2YNSoUZ7ums9Eh1wcaWjhSAMREfVdHg8NW7Zs6XH9nj17ui1bsmQJlixZ4umu3DBiwi6ONDQzNBARUd/Fe0/0gs6RhvM8PEFERH0YQ0Mv4ERIIiLyBwwNvSAmhJeSJiKivo+hoRc4RhpaeHiCiIj6LoaGXuC40yUnQhIRUR/G0NALoi9ep6HFYkOrxebj3hAREV0bhoZeEKZWQKXoKDXnNRARUV/F0NALBEHgZEgiIurzGBp6yaXTLjkZkoiI+iaGhl4SzZEGIiLq4xgaeknnGRQ87ZKIiPoqhoZe0nl44jxPuyQioj6KoaGXdB6e4EgDERH1VQwNvSQ2jBd4IiKivo2hoZd03umyvoWhgYiI+iaGhl4SE9Z59gQPTxARUd/E0NBLOkcaLpgsaLfZfdwbIiIi9zE09JKoEBVkAiCKQIOJow1ERNT3MDT0ErlMQFTnBZ6aGRqIiKjv8XhoyMrKwk033YSwsDDExcVh0aJFKCoq6nGfTZs2QRCELj9BQUGe7prPcTIkERH1ZR4PDXv37sXKlStx8OBBZGdnw2q1Ys6cOWhpaelxP61Wi+rqasdPeXm5p7vmc5cmQzI0EBFR36PwdIM7d+7s8njTpk2Ii4tDfn4+br31Vqf7CYIAnU7n0nOYzWaYzZc+eA0GAwDAarXCarVeQ6+ldbblqTYjg5UAgNqmVo/2s7d5ui7+hLWRxrpIY12cY22keaMu7rTl8dBwpaamJgBAVFRUj9sZjUakpKTAbrdjwoQJ+O1vf4uRI0dKbpuVlYW1a9d2W75r1y5oNJrr7/QVsrOzPdJOc50MgAyHvz0JXdMJj7TpS56qiz9ibaSxLtJYF+dYG2merIvJZHJ5W0EURdFjz3wFu92OO+64A42Njdi/f7/T7XJzc1FcXIwxY8agqakJf/jDH7Bv3z4cP34c/fv377a91EhDUlIS6urqoNVqPdZ/q9WK7OxszJ49G0ql8rrb27j3DP745WksHp+I3981ygM99A1P18WfsDbSWBdprItzrI00b9TFYDAgJiYGTU1NV/0M9epIw8qVK1FYWNhjYACA9PR0pKenOx5PnToVw4cPx5///GesW7eu2/ZqtRpqtbrbcqVS6ZU3l6fajQ/vGAW5YLL6xS+Bt+rtD1gbaayLNNbFOdZGmifr4k47XgsNq1atwqeffop9+/ZJjhb0RKlUYvz48Th9+rSXeucb8eEdZ4ScrjVCFEUIguDjHhEREbnO42dPiKKIVatWYfv27di9ezcGDhzodhs2mw3fffcdEhISPN09n7ppQCTUChnOXWhFUU2zr7tDRETkFo+HhpUrV+L999/H5s2bERYWBr1eD71ej9bWVsc2y5Ytw+rVqx2Pn3/+eezatQtnzpzB0aNH8cADD6C8vBw///nPPd09n9KoFJg2JAYAkH28xse9ISIico/HQ8Nbb72FpqYmTJ8+HQkJCY6frVu3OrapqKhAdXW14/GFCxewYsUKDB8+HAsWLIDBYMCBAwcwYsQIT3fP52aPiAcA7DrB0EBERH2Lx+c0uHIyxp49e7o8fuWVV/DKK694uis3pJnD4yEI3+G7yiZUN7UiITzY110iIiJyCe890ctiw9SYkBwJAPiSow1ERNSHMDT4AA9REBFRX8TQ4AOdoeHgmXoY2niJVCIi6hsYGnxgcGwoBsWGwGoTsbfovNv7myztyC9vcGn+CBERkacwNPhI52hDtpuHKERRxH/8Xz7ufisXG/ee8UbXiIiIJDE0+MicER139PzqZC0s7XaX99tRUImvi+sAAOuzi3CiyuCV/hEREV2JocFHxidFICZUjWZzO94/WO7SPhdaLFj36fcAgJhQNaw2EZkfFcDcbnNsU2c0I/tEDaw214MIERGRKxgafEQmE/DwrR2X2F73rxP4/Lvqq+wBZH3+PRpaLBgaH4pP/usWRIeocFLfjFe/LIbdLuLDwxX44R/2YMXf8vBfm4/BZuecByIi8hyGBh9a8YNBWDolGaIIPLqlAAfP1Dvd9uCZenyUdw4AkHXXaCSEB+PFxaMBAH/eW4JFf/o3Vn/8HQxt7QCAncf1eGZHISdLEhGRx3j11tjUM0EQsO7OUahrNmPXiRqseC8PLy8Zi6HxoUiMCIZMEPB9tQFHKy7gnX+XAgCWTknGxJQoAMC8UTrcNaEfPj5aiW/PNUGjkiNz9lDowoPwyw+P4cPDFYgJVeF/5gyDKIoorzfhu8omVDSYcO6CCWcbWmFutyFUrUCIWoEIjRJzR+pwy+AYyGS8AycREXXF0OBjcpmA1zPGY9lfD+NwWQN+8X6+Y51SLsBquzRSEBumxpNz07rsv+aOkagzWqANUmD1guHoF9FxWWpDazue3v4d3th9GofONKC4thkXTFe/JsT7ByswKCYED9ycgh9P6g9tEO9jT+Qth0obsPF7GYZOMmJ4v0hfd4foqhgabgBBSjneXjYJ6/51At+ea8S5C60wWWyw2kREaJQYnxSB8cmRuHtif4Rrun6Ia4OU+NvPJndrc+mUZDS0mPGHXadwuKwBAKCSyzAiUYtBMSFIitIgKUqDYKUcLeZ2GM3tOFNnxI5jVThT14LnPz2Bl78owo/GJCBjSjLGJ0VAENwbffjqZC3+lluGpCgN7hyXiAnJkS61UdnYir8dKMMwXRgWjk2EUs6jaK44d8GEAyX10KjkCFUrEBakxMhELYKUcl93jSS0WW341d8LUdUkw5PbC7HjP6dxhI9ueAwNN4hwjRJ/WDIWQMe1GBpNVhjN7egfGez2h3WnlTOGIDEiGE2tVoxPjsTwhDCoFT1/gDw1fzi2H6vE/+WW4VSNEdvyz2Fb/jmkXfwAnzdKh+QINUQRyC+/gL8fq8aBknqMTQrH/VNSkD4oGo2tVjz/yXHsKKhytPu33HL0jwzGnBE6DNOFYkhcKAbHhiJCo3Js02a14X+/PoM3vzqNNmvH2R9/3HUKD986CPdMSkKwyvUPP1EU8e6/y7DrhB4/npiEO8d5PnzYLk4+tbTbfT4qc6CkDv/xt3w0m9u7LI8LU+PxOcNw98T+kPvwA+lYRSPKjT57+hvSewfKUNXUBgD49pwB2/LP4t6bkn3cK6KeCaIfzJQzGAwIDw9HU1MTtFqtx9q1Wq347LPPsGDBAiiVgTVML4oi8ssv4MPDZ/Hpt1UwX3YticGxIWhuNqK2rfuH0IBoDZrb2lHfYoFMAB64OQVGczu+KNSjxWLrtn2ERomU6BCkRGnwzblGlNebAABjkyJQecGEOqMFABCmVmB4ghZDdaEYptPilsHRGBQbKtl3o7kdT2z7Bp8X6h3L+kcG45Hpg7FgVALCg5XX/Y2u5LwRj2/7BscqGgEAoWoFlk5JxgOT++PYv3df9T1jsrTD0m7vEpqu1SffVOF/PvoGFpsdqXGhiApRobmtHXpDGxpaOuqXpgvDL2emIl4bBLVCBrVChvBgJSJDVF4dyWm12PDiZyfw/sEKAMCPJ/TDsz8a2W3ETEqb1YZzF0wwtLVDIRMglwlQymWI0CgRHaK+7hAkiiL+X/45bPjqNNIHx+C5H43oEky/PdeIF/71PQbHhuKXM4dc8x1pjeZ2nKgyYGJKpKPPF1osuPXlr9Dc1o4hWhGnDQKiQlTY/T+3eeQ94Q8C+e9vT7xRF3c+QxkaesA3bYemVis++64aOwv1OFBS55hnoVHJcfvoBMweEY+vi+uw/VgljBe/6Q6LD8PvfjwG45IiAHR8eOw+WYsjZQ0oOW9ESa3R8S3rcvFaNZ5eMBx3jE2Eud2ObXln8ed9Z3DuQmu3bSemRGLJxP6YPyoBKoUM7XY7KhpM+OWHx1ByvgVKuYAfT0xC9gm9I3wAgCAA4cFKRIeoMDAmFKnxoRgSGwqT1YYTVQZ8X21ARYMJcpng+ICNDVMjTafFMF0YGk1WvPrlKZjb7QhTKxCnVaPkfAsAQCET0F9jx8yxAzBlUAwmpkQiOlTteG5DmxXv7C/FX78uhcVmR+bsoXho2kAoXPjgrqg34X/3n0FxjREp0RoMig2Bsa0dr+8+DQBYMFqH9feMcxyOMLfb8H+55Xg9p9hxVo2UCI0SieHBmDMyHneO64eBMSGw2uzYW3Qefz96DkU1zfj1guGYOTy+x/6dbTDBaG5HbJgaURoVTlQb8OiWY47adIoJVWPtHSOxYLSuyyiaKIr48vtabDpQijPnW6A3tMHZXyeZAESFqJAQHozU+FAMjQ/DkNhQyGSAyWKDyWJDk8mK2uY21Dab0dBiQWpcGKYPi8XkgVGobGzFr7d/h4NnGhxtDk/Q4q37JyAlWoNNB8rw28++d7zX1QoZfjp1AB6ZPtitD/XCyiY88kE+zja0YsawWLyeMR5hQUo8/8kJvPPvUqTpwrAi5QI2loajuLYFP7k5BesWjeqxTZtdRMHZRhRWNiE1LhQTUiKv+RCUoc2KeqMFKVGaXjk0UljZhNK6FkwfFouwq4zMtbSasXPnTtzxo2v/+9tus+NwWQMKK5twosqAE9UGqBVy/DAtDrNHxGNkovaaR3LrjGbs/r4WX35fg1arDXdP6I8Fozv+Fnlac5sVrdaOL13t1nbk5OTgzgVzoQ0J8kj7DA0ewtDQnaHNit0n9Mg/egyZ981GZOilb18mSzs+/bYabVYb7rsp+aq/PC3mdlQ0mFBe34LyehOUchnuuSkJoequR83abXYU1TTjVE0zivRGfFfZiNySevR0GQqdNgh/emACJiRHotViw5YjFfjr/lLJ8HGtfpAag9/dPQY6bRC+KqrFX/adwaHShm7bpURrMD4pArFhamw9crbbB/jofuH43d1jEBakwJGyBhwpuwBDqxWDY0OQGh+G6FAVth45i0++qXL6mn86dQCe/dEIyW/fjSYLNnx1GnuKzqOt3QZLux3mdjsMrVbJ9kb100Lf1NYtaK2en4YVPxjk+CMriiKOVxmw67geO4/rcarm0vEHuUyAKIqwix2HSH531ygcyzuET2vCcaauI0SMTNRiefoA3DEuEdVNbVj7yXHsueJeLCEqOSJDVLDbRbTbRVhtdjS2Wp2GCVcEK+Ww2UVYbHYEKWX4yc0p2H6sEnVGC8KCFBiXFOG46uqs4XEwtLY75gWFqhVYODYBd0/oj4kpl+botFpsMLRZERuqhuzia//w8Fms+eR4lyu+psaF4rmFI/CzTUdgtYl4d/lEGE4dQvTwm/HAO3mQCcA/V03DqH7hXfrcbrNj14kafF6ox9fF59F42aRmlVyGcckRSNOFQSGTQSkXoFbKcdOASNw8KFpyJKnRZMHbX5/Bu/8ug8ligzZIgfHJkZiQHImE8CCEBikQqlYgNkyN1LjQLqH2+2oDdhyrxIlqA2SCAMXFEaDEiGCk6cIwVBeGgTEh0KjkUMplsNrs+OK4Hpv+XYa88gsAOoLqf9w6GMunpkCj6vr7Xtvchrf3ncH7B8thabdhRGI4xvSPwITkSNw+JsGlgGRut+Hv+ZXYuLcEFQ0mp9slhgdhfHIkhunCMEwXBpVChiJ9M07pm1HZ2Ipbh8Zi+dQBjr9JdruIL47r8e6/y3CkvKHb+zA2TI2lk5MxLTUGyVEaxF78wlDeYMLxqiacrG6G1W6HUiaDUi5DsEqGeG0QdNogxGuDYDS3o6qxFVWNrahoaEVxbTOKa4zQG7p/wcpaPBIZUwZctRauYGjwEIYGaTdCXWoMbdh+rBLb8s52+yY7Y1gsXl4yFjGXfcPvZGm3o7HVgkaTFbUGM0rOG1Fc24zTtUYEKeUYkaDF8AQthsR1HPowt9svDpO3okhvwEl9MxpaLFg6JRlLJyd3+5ZSUtOEdz7Zi/aIFBytaERxbfcD+UPiQvHYrFSYzDa88K8TPY4CXOm2obFYMFqHysY2nDlvhL6pDQvHJmJZeorb35hsdhGNJgvqWyz49lwT/vlNFf59us5xUbCYUBUWjeuH5rZ2bM07CwC4e0J//Hhif2SfqMEXx/WobLwUwhQyAWFBii5n6cwdGY+su8YgTCXgs88+w8w58/D2/nJs3FviOOQVHqxEq8UGi80OpVzAz6YNxNyROqREaRAVour2utptdjSYLKhrtqCiwYTimmacqjXizHkjZIIAjUoOjUqOsCAl4sLUiNcGQRuswLGKRnxVVIsag9lRyxcWjUJSlAb6pjas3HwU+Rc/1JRyAU8vGI6fTh0AANhTdB6/23kSJ/XNjn4MiO7o39kLrTjf3NGmSiFDcpQGoWoFCs42AgBmDY/HT6cOwP9sK3A8d+fz/+9Pxjt+lzL/XyE++aYKyVEa3HtTEmaPiEdylAbb8s/h7X1nunz4aYMUGJsUgSJ9M2qbL7V5pQiNErOHx2NiSiREAO12EZUXWvHBwXLH/BeFTEB7Dwlco5JjTP9wpOm0OHimvksNrkYuEyAXBFguXqFWKRcQFxbkeN/EhKowa3g8okJUiApR4dyFVnx4uKLL4dDLJUUFY83CkV1Gvc43m5FffgGNJguaWq1oaLHgHwVVjg/aSI0SNw+KxshELUYkatHQYkX2CT32njrvmDvVkwiNEit+MAgJ4UF4a09Jl9/pkYlazB4RDwECPjhU3u3/hUohg1ImSB6addfl3wdEUUTW4lG4j6Hh2jA09K4bqS6iKMJobu/4xiMXoJDJfDrh78raNLVa8e25RhyraHQMy/5oTKKjj7WGNjz3j+PYeVwPpVzA6H7hmDQgCrGh6ouBxoizDSbcNDAKj9w2uNs3UE8732zGnqJaRIeq8IPUWCjlMoiiiPcOlOH5T090G5kIUspw29BYzB2pw8y0eIRrlLDa7Kg3WmC12R0Tea+sy4UWCz7KO4v/O1juGP25dWgsfrNwBAY7maviCaIo4vvqZrTb7RjdL7xLILHa7Hgl+xTyyy/g6QXDMfbiobVOdruIg6X1+Ht+JT4vrIbpig8CQUCXb55ymYAn5g7Df9zaMTpTY2jDw3/LwzfnmiATgM8e/QEGRwc76lJvsuH2179GfculER6VQuYYqYjUKHHvTcmYOTwO45MioLj4/6a0rgUHzzSguqkVVpvoCFV7is475rRISdOFIXP2UMxIi0ORvhn55RfwzblGNJqsaG6zormtHecutDoOOTr6JJfhh2lx+GFaHGQyAe02Oyw2O8rqTCiqMaBI39xllAroCAf3T0nB/VOSERWiwj8KqvBaTrHTUYDxyRFYedtAlBceQUzqBJzQG7GjoNIRumYNj8PY/hH48mQtvrkYzq6k0wbh4VsHIWNysuQk6jarDUfKGvB9dceXgSJ9M6w2O4bGh2FYfBi0wUq8d6DMMTLWKSxIgQenDsC9k5Mdp7gDHV9Idh7X4+/551By3ojqpjZHAFcpZBiuC8OIRC1CVArHqFmLuWPukb6pDTUGM0LUCiRGBCExPBj9IoORGheK1PgwpMaHOiZa++2chg0bNuDll1+GXq/H2LFj8cYbb2Dy5O6nBnbatm0bnn32WZSVlSE1NRW/+93vsGDBApeei6Ghd7Euzl1rbWoMbQgPVt7Qp0fuO3Uev9xyDAAwMy0ec0bG49bUWJfOanFWF5tdxNfF5yGXCZg2JOaajy/3thZzO/aeOg9R7Pj2mxSpgTZYiarGVpTXd1w8bUz/CIxI7Pr3qM1qw9v7ziA5WoM7x/XrVpd6oxlfHK9B9gk9/l1SD0u7Hf0igrHiBwNxz01J3Ybye9J5PP+LQv3FOTodhy5UChnmjtRh3kjdVecx2OwiSs4bcaziAr6vbkZqfCh+NDrxqhNZ26w2mK12x+GweG1Qt8OVVpsdOwv1KK1rQUOLBRdMFogicM+kJNwyJBrt7e1damM0t+ONnGL8dX9pt5GRNF0Y+kUEQxusRHiwEiMStbhzXOJVzxa7GptdxCffVOGtPSVobrPi/ptT8JP0FJfOlGq32VHd1IY2qw0DYkI8NuHY16HBK6dcbt26FZmZmdi4cSOmTJmCV199FXPnzkVRURHi4uK6bX/gwAFkZGQgKysLP/rRj7B582YsWrQIR48exahRPU8KIvIH8VrPTGjypluHxuLQ0zM9OpojlwmYPqz734QbXYhagQWjE7ot77z+iTNBSjn+a2aq0/XRoeqOQ19TktFibkdpXQuG6cKu6QNHIZdh6uAYTB0c4/a+neQyAUPjwzA0Psyt/YKUcgQp5QiH8w81pVyGhWMTXW4zVN1xAbslk/rjtZzTMFttmHFxxMNbvz9ymYBF4/th0fh+bu+rkMt6fC/0VV4512r9+vVYsWIFHnzwQYwYMQIbN26ERqPBO++8I7n9a6+9hnnz5uGJJ57A8OHDsW7dOkyYMAFvvvmmN7pHRNdIrZD79PBPIAlRKzCqXzgvbnaFIXFheCNjPP6ybBIyJif3icDtTzw+0mCxWJCfn4/Vq1c7lslkMsyaNQu5ubmS++Tm5iIzM7PLsrlz52LHjh2S25vNZpjNlyacGAwGAB3DNlbr1S+V7KrOtjzZpj9gXZxjbaSxLtJYF+dYG2neqIs7bXk8NNTV1cFmsyE+vus53fHx8Th58qTkPnq9XnJ7vV4vuX1WVhbWrl3bbfmuXbug0Xh+OCg7O9vjbfoD1sU51kYa6yKNdXGOtZHmybqYTM5PS71Sn7yM9OrVq7uMTBgMBiQlJWHOnDkenwiZnZ2N2bNnc8LfZVgX51gbaayLNNbFOdZGmjfq0jla7wqPh4aYmBjI5XLU1NR0WV5TUwOdTie5j06nc2t7tVoNtbr7OfhKpdIrby5vtdvXsS7OsTbSWBdprItzrI00T9bFnXY8PsNGpVJh4sSJyMnJcSyz2+3IyclBenq65D7p6eldtgc6hl6cbU9ERES9zyuHJzIzM7F8+XJMmjQJkydPxquvvoqWlhY8+OCDAIBly5ahX79+yMrKAgA8+uijuO222/DHP/4Rt99+O7Zs2YK8vDz85S9/8Ub3iIiI6Bp4JTTce++9OH/+PJ577jno9XqMGzcOO3fudEx2rKiogEx2aZBj6tSp2Lx5M5555hk8/fTTSE1NxY4dO3iNBiIiohuI1yZCrlq1CqtWrZJct2fPnm7LlixZgiVLllzTc3Ve1NKdyRyusFqtMJlMMBgMPKZ2GdbFOdZGGusijXVxjrWR5o26dH52unKB6D559sSVmps7bqCSlJTk454QERH1Tc3NzQgP7/n+Nn5xwyq73Y6qqiqEhYV59Nr1nadynj171qOncvZ1rItzrI001kUa6+IcayPNG3URRRHNzc1ITEzsMnVAil+MNMhkMvTv399r7Wu1Wr5pJbAuzrE20lgXaayLc6yNNE/X5WojDJ14UXMiIiJyCUMDERERuYShoQdqtRq/+c1vJK8+GchYF+dYG2msizTWxTnWRpqv6+IXEyGJiIjI+zjSQERERC5haCAiIiKXMDQQERGRSxgaiIiIyCUMDUREROQShgYnNmzYgAEDBiAoKAhTpkzB4cOHfd2lXpWVlYWbbroJYWFhiIuLw6JFi1BUVNRlm7a2NqxcuRLR0dEIDQ3F3XffjZqaGh/12DdeeuklCIKAxx57zLEskOtSWVmJBx54ANHR0QgODsbo0aORl5fnWC+KIp577jkkJCQgODgYs2bNQnFxsQ973DtsNhueffZZDBw4EMHBwRg8eDDWrVvX5QZBgVCbffv2YeHChUhMTIQgCNixY0eX9a7UoKGhAffffz+0Wi0iIiLw0EMPwWg09uKr8I6eamO1WvHkk09i9OjRCAkJQWJiIpYtW4aqqqoubfRKbUTqZsuWLaJKpRLfeecd8fjx4+KKFSvEiIgIsaamxtdd6zVz584V3333XbGwsFAsKCgQFyxYICYnJ4tGo9GxzS9+8QsxKSlJzMnJEfPy8sSbb75ZnDp1qg973bsOHz4sDhgwQBwzZoz46KOPOpYHal0aGhrElJQU8ac//al46NAh8cyZM+IXX3whnj592rHNSy+9JIaHh4s7duwQv/nmG/GOO+4QBw4cKLa2tvqw59734osvitHR0eKnn34qlpaWitu2bRNDQ0PF1157zbFNINTms88+E3/961+LH3/8sQhA3L59e5f1rtRg3rx54tixY8WDBw+KX3/9tThkyBAxIyOjl1+J5/VUm8bGRnHWrFni1q1bxZMnT4q5ubni5MmTxYkTJ3Zpozdqw9AgYfLkyeLKlSsdj202m5iYmChmZWX5sFe+VVtbKwIQ9+7dK4pix5tYqVSK27Ztc2zz/fffiwDE3NxcX3Wz1zQ3N4upqalidna2eNtttzlCQyDX5cknnxSnTZvmdL3dbhd1Op348ssvO5Y1NjaKarVa/PDDD3ujiz5z++23iz/72c+6LLvrrrvE+++/XxTFwKzNlR+MrtTgxIkTIgDxyJEjjm0+//xzURAEsbKystf67m1SgepKhw8fFgGI5eXloij2Xm14eOIKFosF+fn5mDVrlmOZTCbDrFmzkJub68Oe+VZTUxMAICoqCgCQn58Pq9XapU5paWlITk4OiDqtXLkSt99+e5fXDwR2Xf75z39i0qRJWLJkCeLi4jB+/Hi8/fbbjvWlpaXQ6/VdahMeHo4pU6b4fW2mTp2KnJwcnDp1CgDwzTffYP/+/Zg/fz6AwK5NJ1dqkJubi4iICEyaNMmxzaxZsyCTyXDo0KFe77MvNTU1QRAEREREAOi92vjFXS49qa6uDjabDfHx8V2Wx8fH4+TJkz7qlW/Z7XY89thjuOWWWzBq1CgAgF6vh0qlcrxhO8XHx0Ov1/ugl71ny5YtOHr0KI4cOdJtXSDX5cyZM3jrrbeQmZmJp59+GkeOHMEvf/lLqFQqLF++3PH6pX63/L02Tz31FAwGA9LS0iCXy2Gz2fDiiy/i/vvvB4CArk0nV2qg1+sRFxfXZb1CoUBUVFTA1AnomDf15JNPIiMjw3Gny96qDUMDXdXKlStRWFiI/fv3+7orPnf27Fk8+uijyM7ORlBQkK+7c0Ox2+2YNGkSfvvb3wIAxo8fj8LCQmzcuBHLly/3ce9866OPPsIHH3yAzZs3Y+TIkSgoKMBjjz2GxMTEgK8NucdqteKee+6BKIp46623ev35eXjiCjExMZDL5d1mu9fU1ECn0/moV76zatUqfPrpp/jqq6/Qv39/x3KdTgeLxYLGxsYu2/t7nfLz81FbW4sJEyZAoVBAoVBg7969eP3116FQKBAfHx+QdQGAhIQEjBgxosuy4cOHo6KiAgAcrz8Qf7eeeOIJPPXUU7jvvvswevRo/OQnP8F///d/IysrC0Bg16aTKzXQ6XSora3tsr69vR0NDQ0BUafOwFBeXo7s7GzHKAPQe7VhaLiCSqXCxIkTkZOT41hmt9uRk5OD9PR0H/asd4miiFWrVmH79u3YvXs3Bg4c2GX9xIkToVQqu9SpqKgIFRUVfl2nmTNn4rvvvkNBQYHjZ9KkSbj//vsd/w7EugDALbfc0u203FOnTiElJQUAMHDgQOh0ui61MRgMOHTokN/XxmQyQSbr+udWLpfDbrcDCOzadHKlBunp6WhsbER+fr5jm927d8Nut2PKlCm93ufe1BkYiouL8eWXXyI6OrrL+l6rjcemVPqRLVu2iGq1Wty0aZN44sQJ8eGHHxYjIiJEvV7v6671mkceeUQMDw8X9+zZI1ZXVzt+TCaTY5tf/OIXYnJysrh7924xLy9PTE9PF9PT033Ya9+4/OwJUQzcuhw+fFhUKBTiiy++KBYXF4sffPCBqNFoxPfff9+xzUsvvSRGRESI//jHP8Rvv/1WvPPOO/3utEIpy5cvF/v16+c45fLjjz8WY2JixF/96leObQKhNs3NzeKxY8fEY8eOiQDE9evXi8eOHXOcAeBKDebNmyeOHz9ePHTokLh//34xNTXVL0657Kk2FotFvOOOO8T+/fuLBQUFXf4mm81mRxu9URuGBifeeOMNMTk5WVSpVOLkyZPFgwcP+rpLvQqA5M+7777r2Ka1tVX8z//8TzEyMlLUaDTi4sWLxerqat912keuDA2BXJdPPvlEHDVqlKhWq8W0tDTxL3/5S5f1drtdfPbZZ8X4+HhRrVaLM2fOFIuKinzU295jMBjERx99VExOThaDgoLEQYMGib/+9a+7/MEPhNp89dVXkn9Xli9fLoqiazWor68XMzIyxNDQUFGr1YoPPvig2Nzc7INX41k91aa0tNTp3+SvvvrK0UZv1EYQxcsuSUZERETkBOc0EBERkUsYGoiIiMglDA1ERETkEoYGIiIicglDAxEREbmEoYGIiIhcwtBARERELmFoICIiIpcwNBAREZFLGBqIiIjIJQwNRERE5JL/Dy6pcX1AmRG1AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training loss\n",
    "pd.Series(training_objective.evaluations_history[\"train\"][\"loss\"]).plot(figsize=(6, 3), grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "68450343-0980-4dbf-b8f8-8062a7c07a0a"
   },
   "source": [
    "## Evaluation\n",
    "\n",
    "We will take a look whether it's worth to take the **generative** approach, that takes larger model and longer to train, compared to traditional sequence classification.\n",
    "\n",
    "Below, we evaluate both approaches by loading the training checkpoints where the trainings early-stopped and evaluate them accordingly, for classification accuracy.\n",
    "\n",
    "Note that, for prediction, we no longer depend on Adaptor: We can use original HuggingFace models, as if we trained with pure `transformers` library. Therefore, no dependency on Adaptor is needed when you deploy the final model."
   ],
   "id": "68450343-0980-4dbf-b8f8-8062a7c07a0a"
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 1. Classification"
   ],
   "metadata": {
    "id": "0GJCPdPbTK5s"
   },
   "id": "0GJCPdPbTK5s"
  },
  {
   "cell_type": "code",
   "source": [
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
    "\n",
    "# pick the 5-th-last checkpoint (with optimal evaluation accuracy) for the trained objective - note that the specific checkpoint might differ in your setup \n",
    "tokenizer = AutoTokenizer.from_pretrained(\"./train_dir/checkpoint-1600/SequenceClassification\")  \n",
    "classifier = AutoModelForSequenceClassification.from_pretrained(\"./train_dir/checkpoint-1600/SequenceClassification\")"
   ],
   "metadata": {
    "id": "3j9MiCssR2UZ"
   },
   "id": "3j9MiCssR2UZ",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "background_save": true
    },
    "id": "662fd205-37eb-4fc8-9f66-b19c4aa8952f",
    "outputId": "32ee5958-ba61-46ec-dd5f-7a201adf6235"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2500/2500 [18:00<00:00,  2.31it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "predictions = []\n",
    "\n",
    "for review, score in tqdm(zip(test_df[\"comment\"].values, test_df[\"rating_int\"].values), total=len(test_df)):\n",
    "    inputs = tokenizer(review, return_tensors=\"pt\", padding=True, truncation=True)\n",
    "    outputs = classifier(**inputs)\n",
    "    output_label_id = outputs.logits.argmax(-1)[0].item()\n",
    "\n",
    "    prediction = classifier.config.id2label[output_label_id]\n",
    "    predictions.append(prediction)"
   ],
   "id": "662fd205-37eb-4fc8-9f66-b19c4aa8952f"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "background_save": true
    },
    "id": "e9f18919-bd9e-47b2-8711-9daa68269aac",
    "outputId": "69fc00c3-158b-4d9a-a4d6-d88ba9a3aba4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction using 'xlm-roberta-base' classifier; accuracy: 0.5736, average error: 0.6412\n"
     ]
    }
   ],
   "source": [
    "true_labels = test_df.rating_int\n",
    "\n",
    "correct_predictions = sum([pred == true for pred, true in zip(predictions, true_labels)])\n",
    "incorrect_predictions = sum([pred != true for pred, true in zip(predictions, true_labels)])\n",
    "\n",
    "accuracy = correct_predictions / (correct_predictions+incorrect_predictions)\n",
    "average_error = sum(abs(pred - true) for pred, true in zip(predictions, true_labels)) / len(predictions)\n",
    "\n",
    "print(\"Prediction using '%s' classifier; accuracy: %s, average error: %s\" % (classifier.config.model_type, accuracy, average_error))"
   ],
   "id": "e9f18919-bd9e-47b2-8711-9daa68269aac"
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 2. Generation\n",
    "\n",
    "Evaluation pipeline for generation slightly differs from classification, but the resulting performance is easily comparable."
   ],
   "metadata": {
    "id": "sMkRtd2rTgEB"
   },
   "id": "sMkRtd2rTgEB"
  },
  {
   "cell_type": "code",
   "source": [
    "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n",
    "\n",
    "# again, pick the 5-th-last checkpoint (with optimal eval loss) for the trained objective - note that the specific checkpoint might differ in your setup \n",
    "tokenizer = AutoTokenizer.from_pretrained(\"./train_dir_generative/checkpoint-3400/Sequence2Sequence\")\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(\"./train_dir_generative/checkpoint-3400/Sequence2Sequence\")"
   ],
   "metadata": {
    "id": "DKDpsVaBT7tv"
   },
   "id": "DKDpsVaBT7tv",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "background_save": true
    },
    "id": "c397e787-dc01-46fc-b905-d86026167f89",
    "outputId": "9ea541a7-c392-4be7-cb23-2bfd6e4cc05f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/2500 [00:00<?, ?it/s]/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py:1288: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n",
      "100%|██████████| 2500/2500 [40:24<00:00,  1.03it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "promt = \" Kolko hviezdičiek si zaslúži táto recenzia? Možnosti: 1, 2, 3, 4, 5.\"\n",
    "\n",
    "predictions = []\n",
    "for review in tqdm(test_df[\"comment\"]):\n",
    "    input_text = promt + review\n",
    "    inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "    outputs = model.generate(**inputs)\n",
    "    response_text = tokenizer.decode(outputs[0], skip_special_tokens=True).strip()\n",
    "    predictions.append(response_text)\n",
    "    \n",
    "true_labels = test_df.rating_int"
   ],
   "id": "c397e787-dc01-46fc-b905-d86026167f89"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "background_save": true
    },
    "id": "cbc3b22c-8dc2-4a7e-aef5-79608891337c",
    "outputId": "33559fed-914a-4878-ae15-3c87193b3c8d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction using 'mt5' classifier; accuracy: 0.6016, average error: 0.6132\n"
     ]
    }
   ],
   "source": [
    "correct_predictions = sum([pred == str(true) for pred, true in zip(predictions, true_labels)])\n",
    "incorrect_predictions = sum([pred != str(true) for pred, true in zip(predictions, true_labels)])\n",
    "\n",
    "accuracy = correct_predictions / (correct_predictions+incorrect_predictions)\n",
    "average_error = sum(abs(int(pred) - true) for pred, true in zip(predictions, true_labels)) / len(predictions)\n",
    "\n",
    "print(\"Prediction using '%s' classifier; accuracy: %s, average error: %s\" % (model.config.model_type, accuracy, average_error))"
   ],
   "id": "cbc3b22c-8dc2-4a7e-aef5-79608891337c"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Conclusions\n",
    "\n",
    "We have obtained ca. +3% of accuracy, and ca. -3% of absolute error by using the larger generative mt5 model. Perhaps using similarly-large classification model, the gains would be even larger, so the comparison is not perfectly fair, but shows that both traditional and generative approach can perform comparably in the single-objective settings.\n",
    "\n",
    "Importantly, we have seen that training the model for a completely different approach (classification vs. generation) is fairly simple in Adaptor - you just import a different `Objective` with identical interface and everything else works out-of-box."
   ],
   "metadata": {
    "id": "XrgfThULUeCW"
   },
   "id": "XrgfThULUeCW"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "colab": {
   "provenance": []
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}