fastai_imgc_plantpathology_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "fastai_imgc_plantpathology_2.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyPdE7ojFTUz5/Ffc2ftMINZ",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/moarshy/PyTorch_fastai_Projects/blob/main/fastai_imgc_plantpathology_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OcZuumzfVWK7"
      },
      "source": [
        "The idea of this notebook is to use fastai with timm's efficientnet model. \n",
        "\n",
        "We will use Plant Pathology dataset from Kaggle and see what scoring we can achieve\n",
        "\n",
        "Tecniques used in this image classification\n",
        "* transfer learning\n",
        "* efficientnet architecture from timm\n",
        "* stratified kfold crossvalidation\n",
        "* MixUp\n",
        "* label smoothing\n",
        "* TTA\n",
        "\n",
        "\n",
        "In this notebook, our results were private score: 0.96454 and public score: 0.96877. There is still room for further improvement"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0y3p5OMtWFiN",
        "outputId": "808d0f02-0d62-48f5-f723-5eff7f3994f3"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAZGWAlmWIki"
      },
      "source": [
        "!pip uninstall fastai -q -y\n",
        "!pip install fastai -q"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VW-IvspHW8he"
      },
      "source": [
        "!pip install timm -q"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jfpzYecqXjSL"
      },
      "source": [
        "! pip install iterative-stratification > /dev/null -q"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dpLDrn4fWT7d"
      },
      "source": [
        "import fastai\n",
        "from fastai.vision.all import *\n",
        "from iterstrat.ml_stratifiers import MultilabelStratifiedKFold\n",
        "from sklearn.metrics import roc_auc_score\n",
        "\n",
        "import pandas as pd\n",
        "from tqdm.notebook import tqdm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fpi5LrIkYjFT"
      },
      "source": [
        "from timm import create_model\n",
        "import timm\n",
        "from fastai.vision.learner import _update_first_layer\n",
        "from fastai.vision.learner import _add_norm\n",
        "import torch.nn as nn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GT6YM1eAWYKa"
      },
      "source": [
        "# Experiment set-up\n",
        "\n",
        "* transfer learning --> densnet201\n",
        "* img_size 600\n",
        "* batch size 16\n",
        "* mixup\n",
        "* stratified kfold\n",
        "* oversampling\n",
        "* label smoothing\n",
        "* tta"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ruu2p7BWWia"
      },
      "source": [
        "PATH = Path('/content/drive/MyDrive/colab_notebooks/fastai/Plant Pathology')\n",
        "DATA_DIR = Path('/content/drive/MyDrive/colab_notebooks/fastai/Plant Pathology/data')\n",
        "IMG_DIR = PATH/'data/images'\n",
        "TRAIN_CSV = 'train.csv'\n",
        "TEST_CSV = 'test.csv'\n",
        "SS_CSV = 'sample_submission.csv'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XK22i2BqXF0z"
      },
      "source": [
        "SEED = 42\n",
        "set_seed(42)\n",
        "N_FOLDS = 5\n",
        "IMG_SIZE = 128\n",
        "BS = 64"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dlIGkotSXHIe"
      },
      "source": [
        "train_df = pd.read_csv(DATA_DIR/TRAIN_CSV)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "vOCPK2iJXIZM",
        "outputId": "d0f41176-6deb-41fb-f840-98307ecfa1bf"
      },
      "source": [
        "train_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>image_id</th>\n",
              "      <th>healthy</th>\n",
              "      <th>multiple_diseases</th>\n",
              "      <th>rust</th>\n",
              "      <th>scab</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Train_0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Train_1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Train_2</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Train_3</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Train_4</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  image_id  healthy  multiple_diseases  rust  scab\n",
              "0  Train_0        0                  0     0     1\n",
              "1  Train_1        0                  1     0     0\n",
              "2  Train_2        1                  0     0     0\n",
              "3  Train_3        0                  0     1     0\n",
              "4  Train_4        1                  0     0     0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ujb-BB2_XQ6-"
      },
      "source": [
        "LABELS = ['healthy', 'multiple_diseases', 'rust', 'scab']\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NrYBh5y8XJmz"
      },
      "source": [
        "train_df['label'] = train_df[LABELS].idxmax(1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y2YI8Hp0XPLj"
      },
      "source": [
        "# Oversampling and dataloader"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCBHWhxmXOl2"
      },
      "source": [
        "#create folds\n",
        "train_df['fold'] = -1\n",
        "\n",
        "strat_kfold = MultilabelStratifiedKFold(n_splits=N_FOLDS, random_state=SEED, shuffle=True)\n",
        "for i, (_, test_index) in enumerate(strat_kfold.split(train_df.image_id.values, train_df.iloc[:,1:].values)):\n",
        "    train_df.iloc[test_index, -1] = i\n",
        "    \n",
        "train_df['fold'] = train_df['fold'].astype('int')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pabfRHn_Xp23"
      },
      "source": [
        "def get_data(fold):\n",
        "    \n",
        "    train_df_no_val = train_df[train_df['fold'] != {fold}]     #training set\n",
        "    train_df_just_val = train_df[train_df['fold'] == {fold}]   #validation set\n",
        "\n",
        "    train_df_bal = pd.concat(\n",
        "                   [train_df_no_val[train_df_no_val['label'] != \"multiple_diseases\"], train_df_just_val] +\n",
        "                   [train_df_no_val[train_df_no_val['label'] == 'multiple_diseases']] * 6\n",
        "                   ).sample(frac=1.0, random_state=SEED).reset_index(drop=True)\n",
        "\n",
        "    datablock = DataBlock(\n",
        "                blocks=(ImageBlock, CategoryBlock(vocab=LABELS)),\n",
        "                getters=[\n",
        "                ColReader('image_id', pref=IMG_DIR, suff='.jpg'),\n",
        "                ColReader('label')\n",
        "                ],\n",
        "                splitter=IndexSplitter(train_df_bal.loc[train_df_bal.fold==fold].index),\n",
        "                item_tfms=Resize(IMG_SIZE),\n",
        "                batch_tfms=[*aug_transforms(size=IMG_SIZE, max_rotate=30., min_scale=0.75, flip_vert=True, do_flip=True), \n",
        "                        Normalize.from_stats(*imagenet_stats)]\n",
        "                )\n",
        "    return datablock.dataloaders(source=train_df_bal, bs=BS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BGiMn7kqX-Fe"
      },
      "source": [
        "def comp_metric(preds, targs, labels=range(len(LABELS))):\n",
        "    # One-hot encode targets\n",
        "    targs = np.eye(4)[targs]\n",
        "    return np.mean([roc_auc_score(targs[:,i], preds[:,i]) for i in labels])\n",
        "\n",
        "def healthy_roc_auc(*args):\n",
        "    return comp_metric(*args, labels=[0])\n",
        "\n",
        "def multiple_diseases_roc_auc(*args):\n",
        "    return comp_metric(*args, labels=[1])\n",
        "\n",
        "def rust_roc_auc(*args):\n",
        "    return comp_metric(*args, labels=[2])\n",
        "\n",
        "def scab_roc_auc(*args):\n",
        "    return comp_metric(*args, labels=[3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VBxV-F16edKl"
      },
      "source": [
        "# timm Model\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Low_vRU1ep3u"
      },
      "source": [
        "class Model(nn.Module):\n",
        "    def __init__(self, model_arch, n_class, pretrained=False):\n",
        "        super().__init__()\n",
        "        self.model = timm.create_model(model_arch, pretrained=pretrained)\n",
        "        n_features = self.model.classifier.in_features\n",
        "        self.model.classifier = nn.Linear(n_features, n_class)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.model(x)\n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_0sR03Aae1dK",
        "outputId": "a4ee0495-23df-465a-db85-8cf701e9d0b6"
      },
      "source": [
        "model = Model('tf_efficientnet_b3_ns', len(LABELS), pretrained=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading: \"https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b3_ns-9d44bf68.pth\" to /root/.cache/torch/hub/checkpoints/tf_efficientnet_b3_ns-9d44bf68.pth\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rN_4Gt2rfszU"
      },
      "source": [
        "dls = get_data(0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wGvBluPhfpPF"
      },
      "source": [
        "opt_func = partial(Adam, lr=1e-3, wd=0.005, eps=1e-8)\n",
        "learn = Learner(dls, model, loss_func=LabelSmoothingCrossEntropy(),\n",
        "                opt_func=opt_func,\n",
        "                metrics=[accuracy, AccumMetric(comp_metric, flatten=False)],\n",
        "                cbs=MixUp()).to_fp16()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "id": "MZNWFZMrf-VT",
        "outputId": "e7036be9-4ed1-46cd-eec4-dc6ee6c50414"
      },
      "source": [
        "learn.lr_find()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SuggestedLRs(lr_min=0.0019054606556892395, lr_steep=0.001737800776027143)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhcdZ3v8fe3lt63LJ19I5Gwhy0swxpGBxAQEK4O6jgOo4OOPo5eRwZn5pmL9zqOjl4dB5dBrmJGr4IbKigR9MoSZFgSDJCwGbLRnXS23rururqrvvePqg5t6DX06XOq6/N6nn66q+pUnW+fdOpTv3N+i7k7IiJSumJhFyAiIuFSEIiIlDgFgYhIiVMQiIiUOAWBiEiJUxCIiJS4RNgFTNTs2bN92bJlYZchIlJUNm7ceMDdG4d7rOiCYNmyZWzYsCHsMkREioqZ7RzpMZ0aEhEpcQoCEZESpyAQESlxCgIRkRKnIBARKXEKAhGREqcgEBEpAvdvaeHl/d2BvLaCQEQk4nI550Pfe4ofbmgK5PUVBCIiEbevq4/+rLNwRmUgr68gEBGJuOb2XgAWNSgIRERKUlNbCoBFahGIiJSm5vZ8EOjUkIhIiWpuSzGjKklVWTDzhCoIREQirrk9FVhrABQEIiKR19yWYmFAF4pBQSAiEmnuTlNbioUNVYHtQ0EgIhJhbb39pPqzOjUkIlKqmgtdR3VqSESkRB0aTKYWgYhIaQp6MBkoCEREIq25PUV1WZz6ymRg+1AQiIhEWHNbfgyBmQW2DwWBiEiENbcHO4YAAgwCM1tsZg+Y2XNmtsXMPjLMNmZmt5jZVjN7xsxOC6oeEZFi1NQW7KhigGAmrsgbAP7W3Z8ys1pgo5n9yt2fG7LNm4GjC19nAf9R+C4iUvK6+wboSPUHOpgMAmwRuPsed3+q8HMX8Dyw8LDNrgK+7XmPAQ1mNj+omkREismhMQQBtwim5BqBmS0DTgUeP+yhhcArQ2438dqwwMxuMLMNZrZh//79QZUpIhIpg2MIivYawSAzqwF+DHzU3TuP5DXc/TZ3X+3uqxsbGye3QBGRiGqegjEEEHAQmFmSfAh8193vGmaTZmDxkNuLCveJiJS8pvYUZfEYjTXlge4nyF5DBnwTeN7dvzjCZncDf17oPXQ20OHue4KqSUSkmDS3pVjQUEEsFtwYAgi219C5wLuBZ81sU+G+fwCWALj7rcC9wGXAVqAXuD7AekREikrQC9IMCiwI3P0RYNQYc3cHPhRUDSIixay5LcWaY4K/LqqRxSIiEZTuz7Kvqy/wMQSgIBARiaQ9HWkg+DEEoCAQEYmkqViQZpCCQEQkgqZiQZpBCgIRkQhqbksRM5hXXxH4vhQEIiIR1NSeYm5dBcl48G/TCgIRkQhqbgt+HYJBCgIRkQh6pbWXJTOD7zoKCgIRkcjpG8iypzPNYgWBiEhpam5L4Y5aBCIipWpXa77r6JJZCgIRkZL0SiEIlqpFICJSmna19lKeiNFYG+w6BIMUBCIiEbOr0GMov6xL8BQEIiIRs/Pg1HUdBQWBiEikuDuvtPZOWddRUBCIiERKa0+GnkxWLQIRkVJ1qOuogkBEpDQNBsHSKRpDAAoCEZFIGRxDsGiGgkBEpCTtau1lTm05lWXxKdungkBEJEJ2TeGso4MUBCIiEbJriscQgIJARCQypnr66UEKAhGRiJjq6acHKQhERCJiqqefHqQgEBGJiKmefnqQgkBEJCKmevrpQQoCEZGImOrppwcpCEREImKqp58epCAQEYmAMKafHqQgEBGJgDCmnx6kIBARiYAwpp8epCAQEYmAMKafHqQgEBGJgDCmnx6kIBARiYBdrb00TvH004MUBCIiEdDcnmJhQ2Uo+1YQiIhEQHNbikUzplkQmNntZrbPzDaP8Hi9md1jZk+b2RYzuz6oWkREoiyXc3a3p1k43YIAWAtcOsrjHwKec/eTgTXAF8ysLMB6REQiaX93H5lsjkXT7dSQuz8MtI62CVBr+Uk1agrbDgRVj4hIVDW1pYBwegxBuNcIvgIcB+wGngU+4u654TY0sxvMbIOZbdi/f/9U1igiErimtnzX0el4amgslwCbgAXAKcBXzKxuuA3d/TZ3X+3uqxsbG6eyRhGRwDW351sEpdhr6HrgLs/bCmwHjg2xHhGRUDS3pZhRlaS6PBHK/sMMgl3AGwHMbC5wDLAtxHpERELR1JYK7bQQQGDxY2Z3kO8NNNvMmoCbgSSAu98KfApYa2bPAgbc5O4HgqpHRCSqmttTrGisDm3/gQWBu79jjMd3AxcHtX8RkWLg7jS3pbjg6PCuf2pksYhIiNp6+0n1Z0MbVQwKAhGRUIXddRQUBCIioWpuC7frKCgIRERCNTiGYHFIo4pBQSAiEqqmthQ15QnqKsMZQwAKAhGRUDW15dchyE+7Fg4FgYhIiJrbwx1MBgoCEZFQNbX1htp1FBQEIiKh6Uz305UeCLXHECgIRERCc6jrqFoEIiKlqTnkBWkGKQhEREJyaFSxTg2JiJSm5vYU5YkYs2vCXa5dQSAiEpLBrqNhjiEABYGISGgGB5OFTUEgIhKS5rZU6GMIQEEgIhKKVCbLwZ6MWgQiIqWquT3fYyjsrqOgIBARCUVTRAaTwTiDwMyqzSxW+HmlmV1pZslgSxMRmb52t6cBWFBEp4YeBirMbCFwP/BuYG1QRYmITHctnWnMYE5tediljDsIzN17gWuAr7n724ATgitLRGR629eZZlZ1Ocl4+Gfoxx0EZvZHwLuAXxTuiwdTkojI9NfSmWZeffitARh/EHwU+HvgJ+6+xcyWAw8EV5aIyPTW0pFmXl1F2GUAMK5FMt39IeAhgMJF4wPu/jdBFiYiMp3t6+rjtKUzwi4DGH+voe+ZWZ2ZVQObgefM7MZgSxMRmZ76BrK09mQi0yIY76mh4929E7gaWAccRb7nkIiITNC+zj4A5tYV1zWCZGHcwNXA3e7eD3hwZYmITF97O/NjCOYWWYvg68AOoBp42MyWAp1BFSUiMp3tPdQiiEYQjPdi8S3ALUPu2mlmFwVTkojI9NZSaBEU1TUCM6s3sy+a2YbC1xfItw5ERGSC9nWmKUvEaKiKxkw94z01dDvQBby98NUJfCuookREprOWzjRz68pDX5ls0LhODQEr3P3aIbf/p5ltCqIgEZHpLkqDyWD8LYKUmZ03eMPMzgVSwZQkIjK97evqY06EgmC8LYIPAN82s/rC7TbgPcGUJCIyfbk7LR1p/vjYOWGXcsh4ew09DZxsZnWF251m9lHgmSCLExGZbrr6Bkj1ZyMzmAwmuEKZu3cWRhgDfCyAekREprW9HdEaTAavb6nKaFzuFhEpIlEbTAavLwhGnWLCzG43s31mtnmUbdaY2SYz22JmD72OWkREikLUBpPBGNcIzKyL4d/wDRhroc21wFeAb4/w2g3A14BL3X2XmUXnyomISECiNs8QjBEE7l57pC/s7g+b2bJRNnkncJe77ypsv+9I9yUiUiz2dqapq0hQWRadRR7DXCxzJTDDzB40s41m9ucjbWhmNwxOb7F///4pLFFEZHK1dKSZVx+d1gCEGwQJ4HTgcuAS4J/MbOVwG7r7be6+2t1XNzY2TmWNIiKTam9XX6ROC0G4QdAE3OfuPe5+AHgYODnEekREAre3I60gGOJnwHlmljCzKuAs4PkQ6xERCVQ25+zv7ovUYDIY/xQTE2ZmdwBrgNlm1gTcDCQB3P1Wd3/ezH5JfnRyDviGu4/Y1VREpNgd7O4jm/NIdR2FAIPA3d8xjm0+D3w+qBpERKIkioPJINxTQyIiJaUlgmMIQEEgIjJlBgeTqfuoiEiJ2tuZJmYwq7os7FL+gIJARGSKtHSkaawtJxGP1ltvtKoREZnGojiYDBQEIiJTJoqDyUBBICIyZfZ2pSM3mAwUBCIiUyLdn6W9tz9yg8lAQSAiMiX2RXQwGSgIRESmxJ6OFKAgEBEpWYOjihc0KAhERErS7vbBUcVjrfI79RQEIiJToKUjRW1FgprywOb6PGIKAhGRKbC7I82CCLYGQEEgIjIl9nSkmB/B6wOgIBARmRItHWnmR2zW0UEKAhGRgPUNZDnQnWG+Tg2JiJSmvR35wWRRW4dgkIJARCRguwuDyXSxWESkRLV0RHNlskEKAhGRgA22CHSxWESkRLV0pKmrSFAdwcFkoCAQEQnc7vY0CxqieX0AFAQiIoFr6UxF9voAKAhERAK3pz0d2TEEoCAQEQlUuj/LwZ5MZC8Ug4JARCRQewvrECgIRERK1J6OwQVpdGpIRKQkDS5RqYvFIiIlarBFoFNDIiIlak97mvrKJFVl0RxMBgoCEZFA7YnwOgSDFAQiIgHa05FSEIiIlLKWjjTzI9xjCBQEIiKBOTSYrE4tAhGRkjS4DoFaBCIiJaoYuo6CgkBEJDB7Ir4gzaDAOraa2e3AFcA+dz9xlO3OAP4LuM7dfxRUPY9vO8jn73uRxTOrWDyziiUzqzhmbi0nLqzDzILarYiUsFdbBNE+NRTkCIe1wFeAb4+0gZnFgX8F7g+wDgCy7sRjxhPbW/nppmbc8/cvmlHJ5avm85ZVCzhhgUJBRCbPno4UDVVJKsviYZcyqsCCwN0fNrNlY2z2YeDHwBlB1THonBWzOWfFbAAyAzl2t6fYuLONnz+zm2+u387XH9rGgvoKVi+byRnLZrB62UwWzaikIhknGR/+DFpHbz87Dvaws7WXjt4MM6vLaawtZ3ZNGTOqyqgsi1OeiGFm5HJOe6qf/V197O/q40D34FeG9t4MmWyOgayTzeUTqrIsTnVZnKryRP57WYLq8jiVZQlmVpUxt66cufUV1JYnxgyvVCZLS2eaPR0pOlMD5NzJeX5fZkYiZsTMiMeMeAxilr+diBnV5QlqKhLUViQoj8fp6uunKz1AZ6qfTDZHPGYkYjHiMSOVydLam6GtJ0NrT4bezACp/iypTI7+bI6aigT1lUkaKpPUViSJx8AK+wLIuePuZHMQM4jHjGQ8/9o5dzIDOfqzTn82/3rZnDOQyz8nHouRjOdrLk/GqRmsuzy/PGBN4Xt1eZzyRLT/U8r00dIR7XUIBoU25tnMFgJvBS5iCoJgqLJEjGWzq1k2u5prT19EW0+G+7a0sH7rAR7bdpC7n979B9snYkZFMk48ZsQs/0aZyeboSg+Mua94zKhKxkn1ZxkovMkPlYwbDVVllCdiJGJGIh7D3Un35+jJDNDblyWTzY34+hXJGHEzsoU39mzOScRjlMXzb4wDOR9XnUGoKotTmYwXwtTo7svSkcrQn33tcZhK5YkYdZVJ6iuT1FUkaKgqo6EqyYyqMmZUJZlVU87smnygz6zOB3plMh/G8ZhajDJ+u9ujP6oYQgwC4EvATe6eG+sTrZndANwAsGTJkkkvZEZ1GdeduYTrzlyCu9PUlmLDzlYOdmdIZbKk+rOk+3OHPknn3ImbsWhGFUtnVbFsdjUNVUlaezIc6MqwvztNe28/vZksvZkBejNZqsriNNaU01hbweyaMhpry5lVU05dxdif6DMDOVKZLD2ZAXr6BmjtydDSmWZfZx/7utLkPB9W8cIn+4Hcq5+aDZhTV8G8ugrm11dQX5U8tF3+k3j+E/hALkcux6FAcXf6s05vZoDuvgE60wP09Wepq0hSW5GgtiJJeTJ2KHz6szkqk3FmVuffPOsrkySGaUm5O6n+LF3pgUOtEndwh9iQ1ojjDGTzn/gHCi2PZDxG2ZDATMbzv7Nh+Rpy+VZVuj9LT98AXX0DdKXzx6y7L/+9p3BfR6qfznQ/Hal+9namebGli7beDL2Z7Kj/FrOqyzhmXi3Hzqvj2Hm1zKkrp7ZwTOoqkjTWliss5JCWzjSnLmkIu4wxhRkEq4E7C2+Cs4HLzGzA3X96+IbufhtwG8Dq1asD/ThpZocuKE/UnNoKmDf5NZUl8m+A9VXJyX/xKWZmVJUlApmAq5LXf8pncADQgcLpu7be/vwHgUyW3kyW5vZeXmzp4o4ndpHqf21oJGLGohmVLJ6Z/5CworGGN8yp4eg5tcytK9c1qBKyubmD1p4MK+fWhl3KmEILAnc/avBnM1sL/Hy4EBCZShXJOAsbKlk4xgCgbM5pauvlQHeGrnT/oVZGc3uKXa29NLX2cvem3XQOOS1XW55g5bxaVs6t5Zi5Naxa3MCqhfXDtpyk+H3zke1Ul8V562kLwy5lTEF2H70DWAPMNrMm4GYgCeDutwa1X5GpEI8ZS2dVs3RW9YjbuDv7u/vYuq+brfu6eWlvFy/t7ebeZ/dwxxP9QD4czl4xi/OPns3pS2ewcm7tiJ0TpHjs7Uxzz9O7efcfLaWuIvot+SB7Db1jAtv+RVB1iITFzJhTW8Gc2opDPdYgHxD7uvrYsKONR7buZ/3vD/Cr5/YC+dOAx82r5cSF9Vy4spHzj26MfNdDea1v/9cOsu5cf85RY24bBdFdKUFkmjIz5tZVcPmq+Vy+aj4Auw72sqmpnc3NHTzb1MHPNu3mu4/voiIZ48KVjVxywjwuPXFepBc3kbxUJst3H9/FxcfPZcmsiV9rDIP+qkQiYMmsKpbMquLKkxcA0J/N8fi2Vu5/roX7t+zlvi17ufnuLVx72iLeedaSorgAWap+/FQT7b39vO/85WGXMm7mHm6f7olavXq1b9iwIewyRKZMLuc8uaOV7z2xi3XPtpDJ5jhj2QyuOW0Rl500n/rK6J+DLhW5nPOmf3uImvIEP/vQuZHqJWZmG9199bCPKQhEisfB7j5+uLGJH2x4hW37eyiLx3jjcXN451lLOO8NsyP1xlOKfvPCXv5y7Qb+/bpTuOqUaPUWGi0IdGpIpIjMqinnAxeu4P0XLOfZ5g5+8rtm7nl6N+s2t3DeG2bz95cdywkL6sMus2T956M7mV9fwWUnzQ+7lAlRPzWRImRmrFrUwM1vOYFHP/FG/scVx7N5dwdXfPkRPv7Dp2luT4VdYsnJDOR4bNtBLj1xXtF1AVaLQKTIlSVi/OV5R3HtaYv46oNbWfvbHfz0d8289dSFfGDNClY01oRdYknYvLuDvoEcZy6bGXYpE1ZcsSUiI6qvSvIPlx3HAzeu4c/OXso9z+zmTV98iA9+dyMvtHSGXd60t3FHGwCnL5sRciUTpyAQmWYWNlTyyStP4JGb/pgPrlnB+pcO8OZ/X89H7/wdOw/2hF3etPXkjlaWzarKzzlWZBQEItPU7JpybrzkWNbfdBHvv2AFv9zSwhu/8BD/+JNn2a1rCJPK3dmws43TlxbfaSFQEIhMew1VZXzizcfy8I0X8Y4zl/D9J19hzecf5J9+ulmBMEm2HeihtSfDGUV4WggUBCIlY05dBZ+6+kQevHEN/231Iu58ctehQNjbmQ67vKK2YUcrAKuL8EIxKAhESs6iGVX8y1tP4oGP5wPhjid2ccHnHuAz9z5PW08m7PKK0pM72phRlWRF48iz0UaZgkCkRA0Gwm/+dg2XnzSf29Zv44LPPcBXH9hK/yjLo8prbdzZxuplM4t2ZLeCQKTELZlVxRf/9BR++ZELOGv5LD5/34u85cuP8ExTe9ilFYX9XX1sP9BTtNcHQEEgIgXHzKvlG+9ZzdfffTqtPRmu/upv+Zd7nyc1xjrOpW7jzuK+PgAKAhE5zCUnzONXH7uQPz1jMbc9vI03fuFBfrapmWKboHKqPLmjjfJEjBOLeI4nBYGIvEZ9ZZLPXLOK799wNg1VZXzkzk1c8x+P8tSutrBLi5wNO1o5eXEDZYnifTst3spFJHBnLZ/FPR8+j89du4qmthTXfO1R3nP7Ezzw4j5yObUQejMDbN7dWdTXB0CTzonIGOIx4+1nLOayVfP51iPb+c5jO7n+W09y1OxqrjtjMfPqK0jGYyTjMebVVXDSouI9RTJRm15pJ5vzor4+AAoCERmnmvIEH37j0bz/wvx0FWt/u53PrHvhNdutOaaRf7jsuGm/nGYu59z77B7M4LQlahGISAkpS8S48uQFXHnyAlo60vRmBujPOv3ZHI++fIAv/2Yrl37pYa47cwn//U0raawtD7vkSbf9QA83/fgZntjeypUnLyj65UIVBCJyxObV/+FMmycurOdtpy/m3//f7/m/j+3k7k27+cCFy3nvecupLIuHVOXk6c/m+OYj2/m3X71EWSLGv157Em9fvTjssl43rVksIoHYtr+bz657gfuf28u8ugo+fskxvPXUhcRjxTf6Nptzfvq7Zm75ze/ZebCXi4+fy6euPpG5dcUz5bQWrxeR0DyxvZVP/+I5nm7qYEFhPd/LV83nlMUNkZ+SIZdzfv7sHr7065fYtr+HExbU8bcXr+SiY+ZEvvbDKQhEJFS5nHPflhZ+/FQTD720n/6ss7ChkvdfuJx3nrmERATX+P2vlw/y6XufY3NzJyvn1vCxP1nJxcfPI1aELRpQEIhIhHSk+vn1c3v5/pOv8MSOVo6bX8f/uuoEzohIF8yt+7r47LoX+PXz+1hQnz+lddUpxXlKaygFgYhEjruzbnML//zz59jdkebqUxbwd5cey4KGylDq2bK7g689+DLrnt1DdVmCD170Bq4/dxkVyeK/yA2jB4F6DYlIKMyMy06az5pjGvnaAy9z28PbWLe5hevPPYq/XrMikC6ZB7v7WLe5hW37e6ipSFBXkaCyLM79W/by0Ev7qS1P8P4LV/C+845iVs306/Y6ErUIRCQSmtp6+eL9L/GTTc3UVya54YLlXHLCPJbPrn5dF2a7+wa495k93PPMbh59+SDZnFORjJHuf3XNhVnVZfzleUfxZ2cvLfoxASPRqSERKRpbdnfw2XUvsP73BwBYUF/B+Uc3cvLiBmZUJamvStJQWUZZwsjm8l07HWduXQWzqssOhcbOgz2sfXQHP9zQRHffAEtmVnHFqvlcsWoBx82vJefQkxmgOz3AzOqyaXMKaCQKAhEpOrsO9rJ+637Wv3SA3758gK70wJjPqS1PcFRjNVVlcR7f3krcjMtXzec95yzj1CLorhokBYGIFLWBbI59XX10pPpp7+2nI5WhP+vEzIjHwB32dKTZcbCH7Qd62N/Vx8XHz+VdZy8tqkFfQdLFYhEpaol4jAUNlaH1KJruojeKQ0REppSCQESkxCkIRERKnIJARKTEBRYEZna7me0zs80jPP4uM3vGzJ41s0fN7OSgahERkZEF2SJYC1w6yuPbgQvd/STgU8BtAdYiIiIjCKz7qLs/bGbLRnn80SE3HwMWBVWLiIiMLCrXCN4LrAu7CBGRUhT6gDIzu4h8EJw3yjY3ADcUbnabWQvQMWST+iG3h/t58Pts4MARljr0dSfy+HD3j1Tv0NvDbXOk9Y9V+2jbjFTfcLdHO/YQXP1HeuwPv61jP/7axnr8SI/90J+j9P92orUP/Tkqx37piK/k7oF9AcuAzaM8vgp4GVg5wde9baTbw/085PuG1/G73HYkjw93/0j1DlfzZNQ/Vu0Tqf9Ij32Q9R/psR9nzTr2U3jsh6s/Cv9vJ1r7VPztvJ5jf/hXaKeGzGwJcBfwbnd/aYJPv2eU28P9fPj2R2Ks1xjp8eHuH6neobdH22aixvP88dY/nY794bd17MeuYbyPH+mxH/pzlOqfaO3j2fdYgjz2fyCwSefM7A5gDflm0V7gZiAJ4O63mtk3gGuBnYWnDPgIEyJNYk0bgt5HkFR/eIq5diju+ou5diiO+oPsNfSOMR5/H/C+oPY/gmLvoqr6w1PMtUNx11/MtUMR1F9001CLiMjkikr3URERCYmCQESkxCkIRERKnIKgwMzON7NbzewbZvbo2M+IFjOLmdmnzezLZvaesOuZCDNbY2brC8d/Tdj1HAkzqzazDWZ2Rdi1TISZHVc47j8ys78Ou56JMrOrzez/mNn3zezisOuZKDNbbmbfNLMfhVnHtAiCkWY6NbNLzexFM9tqZp8Y7TXcfb27fwD4OfCfQdZ7uMmoH7iK/HxN/UBTULUebpJqd6AbqGAKa4dJqx/gJuAHwVQ5vEn6u3++8Hf/duDcIOs93CTV/1N3/yvgA8CfBlnv4Sap/m3u/t5gKx2HIx2xF6Uv4ALgNIaMYgbi5EctLwfKgKeB44GTyL/ZD/2aM+R5PwBqi61+4BPA+wvP/VGR1R4rPG8u8N0iPPZ/AlwH/AVwRTHVXnjOleTn+npnsR37Ic/7AnBaEdc/Zf9nh/sKfa6hyeDDz3R6JrDV3bcBmNmdwFXu/hlg2OZ7YbRzh7t3BVjua0xG/WbWBGQKN7PBVfuHJuvYF7QB5UHUOZJJOvZrgGry/+FTZnavu+eCrBsm79i7+93A3Wb2C+B7wVX8mv1OxrE34LPAOnd/KtiK/9Ak/+2HaloEwQgWAq8Mud0EnDXGc94LfCuwiiZmovXfBXzZzM4HHg6ysHGYUO1mdg1wCdAAfCXY0sZlQvW7+z8CmNlfAAemIgRGMdFjvwa4hnwA3xtoZeMz0b/7DwNvAurN7A3ufmuQxY3DRI//LODTwKlm9veFwJhy0zkIJszdbw67hiPl7r3kg6zouPtd5IOsqLn72rBrmCh3fxB4MOQyjpi73wLcEnYdR8rdD5K/vhGqaXGxeATNwOIhtxcV7isWxVx/MdcOxV1/MdcOqj8U0zkIngSONrOjzKyM/MW8u0OuaSKKuf5irh2Ku/5irh1UfzjCvFI9iVfv7wD28GrXyfcW7r8MeIn8Vfx/DLvO6Vh/Mdde7PUXc+2qP1pfmnRORKTETedTQyIiMg4KAhGREqcgEBEpcQoCEZESpyAQESlxCgIRkRKnIJBpwcy6p3h/k7JmRWEthg4z22RmL5jZ/x7Hc642s+MnY/8ioCAQGZaZjToPl7ufM4m7W+/upwCnAleY2VjrAlxNfqZTkUmhIJBpy8xWmNkvzWyj5VdAO7Zw/1vM7HEz+52Z/drM5hbu/6SZfcfMfgt8p3D7djN70My2mdnfDHnt7sL3NYXHf1T4RP/dwtTImNllhfs2mtktZvbz0ep19xSwifwMlpjZX5nZk2b2tJn92MyqzOwc8usHfL7Qilgx0u8pMl4KApnObgM+7O6nAx8Hvla4/xHgbHc/FbgT+LshzzkeeJO7v6fJ+fgAAAHjSURBVKNw+1jyU2SfCdxsZslh9nMq8NHCc5cD55pZBfB14M2F/TeOVayZzQCO5tVpxO9y9zPc/WTgefJTGDxKfu6aG939FHd/eZTfU2RcNA21TEtmVgOcA/yw8AEdXl30ZhHwfTObT34Vqe1Dnnp34ZP5oF+4ex/QZ2b7yK+idvhymk+4e1Nhv5uAZeSX3tzm7oOvfQdwwwjlnm9mT5MPgS+5e0vh/hPN7J/Jr9NQA9w3wd9TZFwUBDJdxYD2wrn3w30Z+KK7311YmOWTQx7rOWzbviE/Zxn+/8x4thnNene/wsyOAh4zsx+4+yZgLXC1uz9dWPRmzTDPHe33FBkXnRqSacndO4HtZvY2yC9paGYnFx6u59U54t8TUAkvAsuHLGU45sLqhdbDZ4GbCnfVAnsKp6PeNWTTrsJjY/2eIuOiIJDposrMmoZ8fYz8m+d7C6ddtgBXFbb9JPlTKRuBA0EUUzi99EHgl4X9dAEd43jqrcAFhQD5J+Bx4LfAC0O2uRO4sXCxewUj/54i46JpqEUCYmY17t5d6EX0VeD37v5vYdclcji1CESC81eFi8dbyJ+O+nrI9YgMSy0CEZESpxaBiEiJUxCIiJQ4BYGISIlTEIiIlDgFgYhIiVMQiIiUuP8P6TsYrN+LXMAAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 292
        },
        "id": "-JrakdsEguoh",
        "outputId": "c737f08f-14b8-475b-f79d-20cd59a5e4e3"
      },
      "source": [
        "learn.fine_tune(5, base_lr=0.0017378, freeze_epochs=2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.251482</td>\n",
              "      <td>0.834706</td>\n",
              "      <td>0.756545</td>\n",
              "      <td>0.895529</td>\n",
              "      <td>02:04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.054975</td>\n",
              "      <td>0.789331</td>\n",
              "      <td>0.821990</td>\n",
              "      <td>0.875768</td>\n",
              "      <td>01:26</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.834447</td>\n",
              "      <td>0.643697</td>\n",
              "      <td>0.853403</td>\n",
              "      <td>0.931786</td>\n",
              "      <td>01:26</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.809841</td>\n",
              "      <td>0.645811</td>\n",
              "      <td>0.874346</td>\n",
              "      <td>0.921489</td>\n",
              "      <td>01:25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.796481</td>\n",
              "      <td>0.610828</td>\n",
              "      <td>0.887435</td>\n",
              "      <td>0.937501</td>\n",
              "      <td>01:25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.781504</td>\n",
              "      <td>0.648869</td>\n",
              "      <td>0.884817</td>\n",
              "      <td>0.915322</td>\n",
              "      <td>01:25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.759213</td>\n",
              "      <td>0.616056</td>\n",
              "      <td>0.890052</td>\n",
              "      <td>0.931546</td>\n",
              "      <td>01:26</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HkG8DZokieXD"
      },
      "source": [
        "learn.unfreeze()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CPysZ6nQknMl"
      },
      "source": [
        "# Trainging with efficientnet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDNaDXkbk-ub"
      },
      "source": [
        "IMG_SIZE = 600\n",
        "BS = 16"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJj0zVzlkt1g"
      },
      "source": [
        "def get_learner(fold_num, model):\n",
        "\n",
        "    data = get_data(fold_num)\n",
        "    opt_func = partial(Adam, lr=1e-3, wd=0.005, eps=1e-8)\n",
        "    learn = Learner(\n",
        "        data, model, loss_func=LabelSmoothingCrossEntropy(),\n",
        "        opt_func = opt_func,\n",
        "        metrics=[accuracy, AccumMetric(comp_metric, flatten=False)],\n",
        "        cbs=MixUp()).to_fp16()\n",
        "\n",
        "    return learn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "dCb4sEyKkmns",
        "outputId": "1aa02963-9bb2-4e81-e980-1d97da46a936"
      },
      "source": [
        "for i in range(N_FOLDS):\n",
        "    print(f'Fold {i} results')\n",
        "    savemodel = SaveModelCallback(monitor='valid_loss', every_epoch=True, fname=f'{DATA_DIR}/model/model_efficientnet_best_fold_{i}')\n",
        "    learn = get_learner(fold_num=i, model=model)\n",
        "\n",
        "    learn.fit_one_cycle(3)\n",
        "    learn.unfreeze()\n",
        "    learn.fit_one_cycle(6, slice(1e-6, 1e-3), cbs=savemodel)\n",
        "    \n",
        "    learn.recorder.plot_loss()\n",
        "    \n",
        "    learn.save(f'{DATA_DIR}/model/model_efficientnet_fold_{i}')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fold 0 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.643899</td>\n",
              "      <td>0.423214</td>\n",
              "      <td>0.976440</td>\n",
              "      <td>0.987116</td>\n",
              "      <td>02:07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.653210</td>\n",
              "      <td>0.388196</td>\n",
              "      <td>0.986911</td>\n",
              "      <td>0.992105</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.626883</td>\n",
              "      <td>0.386649</td>\n",
              "      <td>0.986911</td>\n",
              "      <td>0.997562</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.642750</td>\n",
              "      <td>0.416256</td>\n",
              "      <td>0.984293</td>\n",
              "      <td>0.995817</td>\n",
              "      <td>02:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.666391</td>\n",
              "      <td>0.464986</td>\n",
              "      <td>0.939791</td>\n",
              "      <td>0.990527</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.652677</td>\n",
              "      <td>0.425493</td>\n",
              "      <td>0.973822</td>\n",
              "      <td>0.987699</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.636264</td>\n",
              "      <td>0.420738</td>\n",
              "      <td>0.973822</td>\n",
              "      <td>0.988795</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.615271</td>\n",
              "      <td>0.411800</td>\n",
              "      <td>0.984293</td>\n",
              "      <td>0.994817</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.602413</td>\n",
              "      <td>0.404939</td>\n",
              "      <td>0.984293</td>\n",
              "      <td>0.994872</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Fold 1 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.665327</td>\n",
              "      <td>0.397150</td>\n",
              "      <td>0.981675</td>\n",
              "      <td>0.999438</td>\n",
              "      <td>02:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.655469</td>\n",
              "      <td>0.383951</td>\n",
              "      <td>0.989529</td>\n",
              "      <td>0.999787</td>\n",
              "      <td>02:16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.618138</td>\n",
              "      <td>0.376811</td>\n",
              "      <td>0.989529</td>\n",
              "      <td>0.999925</td>\n",
              "      <td>02:15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.618753</td>\n",
              "      <td>0.381934</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999975</td>\n",
              "      <td>02:14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.639478</td>\n",
              "      <td>0.430898</td>\n",
              "      <td>0.968586</td>\n",
              "      <td>0.995581</td>\n",
              "      <td>02:13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.639131</td>\n",
              "      <td>0.413696</td>\n",
              "      <td>0.971204</td>\n",
              "      <td>0.998638</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.630620</td>\n",
              "      <td>0.383500</td>\n",
              "      <td>0.989529</td>\n",
              "      <td>0.999541</td>\n",
              "      <td>02:12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.632871</td>\n",
              "      <td>0.388269</td>\n",
              "      <td>0.981675</td>\n",
              "      <td>0.999150</td>\n",
              "      <td>02:11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.608948</td>\n",
              "      <td>0.382307</td>\n",
              "      <td>0.984293</td>\n",
              "      <td>0.999383</td>\n",
              "      <td>02:10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Fold 2 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.671980</td>\n",
              "      <td>0.401519</td>\n",
              "      <td>0.981675</td>\n",
              "      <td>0.999425</td>\n",
              "      <td>02:10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.634348</td>\n",
              "      <td>0.371006</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>0.999317</td>\n",
              "      <td>02:10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.622128</td>\n",
              "      <td>0.368189</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>0.999574</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.610940</td>\n",
              "      <td>0.378448</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999603</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.648033</td>\n",
              "      <td>0.415220</td>\n",
              "      <td>0.976440</td>\n",
              "      <td>0.997154</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.635492</td>\n",
              "      <td>0.381836</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999356</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.625655</td>\n",
              "      <td>0.386691</td>\n",
              "      <td>0.984293</td>\n",
              "      <td>0.999291</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.607153</td>\n",
              "      <td>0.378458</td>\n",
              "      <td>0.989529</td>\n",
              "      <td>0.998864</td>\n",
              "      <td>02:07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.597830</td>\n",
              "      <td>0.374053</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999425</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Fold 3 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.635362</td>\n",
              "      <td>0.393685</td>\n",
              "      <td>0.984334</td>\n",
              "      <td>0.999153</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.634572</td>\n",
              "      <td>0.382443</td>\n",
              "      <td>0.989556</td>\n",
              "      <td>0.999780</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.612179</td>\n",
              "      <td>0.365901</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.999924</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.599160</td>\n",
              "      <td>0.368581</td>\n",
              "      <td>0.989556</td>\n",
              "      <td>0.999959</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.658362</td>\n",
              "      <td>0.408666</td>\n",
              "      <td>0.976501</td>\n",
              "      <td>0.996489</td>\n",
              "      <td>02:07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.626428</td>\n",
              "      <td>0.373978</td>\n",
              "      <td>0.986945</td>\n",
              "      <td>0.999983</td>\n",
              "      <td>02:07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.642327</td>\n",
              "      <td>0.381586</td>\n",
              "      <td>0.986945</td>\n",
              "      <td>0.999839</td>\n",
              "      <td>02:10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.615132</td>\n",
              "      <td>0.368494</td>\n",
              "      <td>0.992167</td>\n",
              "      <td>0.999924</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.609591</td>\n",
              "      <td>0.365360</td>\n",
              "      <td>0.994778</td>\n",
              "      <td>0.999924</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Fold 4 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.646600</td>\n",
              "      <td>0.371056</td>\n",
              "      <td>0.994778</td>\n",
              "      <td>0.999952</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.642292</td>\n",
              "      <td>0.374897</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.997265</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.619978</td>\n",
              "      <td>0.366672</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.997353</td>\n",
              "      <td>02:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.607602</td>\n",
              "      <td>0.372398</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.997161</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.633818</td>\n",
              "      <td>0.420553</td>\n",
              "      <td>0.976501</td>\n",
              "      <td>0.997408</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.644183</td>\n",
              "      <td>0.384016</td>\n",
              "      <td>0.992167</td>\n",
              "      <td>0.999168</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.622727</td>\n",
              "      <td>0.398131</td>\n",
              "      <td>0.992167</td>\n",
              "      <td>0.997105</td>\n",
              "      <td>02:07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.607728</td>\n",
              "      <td>0.389593</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.997161</td>\n",
              "      <td>02:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.594803</td>\n",
              "      <td>0.385305</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.997161</td>\n",
              "      <td>02:07</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOyddXhUR9fAf3c1m924u5BAEiBYcKcUKFKnSp229G371e2tUaHubpQKpS3WFkrR4hIkAsQ9Ie6bbHY3a/f744YNKVKqtLz7e548uXvvzJ25dubMmTNnBFEUceHChQsXZy+yM10BFy5cuHDx1+IS9C5cuHBxluMS9C5cuHBxluMS9C5cuHBxluMS9C5cuHBxlqM40xX4Jf7+/mJ0dPSZroYLFy5c/KtIS0trFEUx4ETH/nGCPjo6mgMHDpzparhw4cLFvwpBEMpPduy0TDeCIEwTBCFfEIQiQRAePsHx1wVByOz6KxAEofWYY9cJglDY9Xfd77sEFy5cuHDxe/lVjV4QBDnwLnAuUAnsFwRhlSiKOUfTiKJ4zzHp7wQGdW37Ak8CKYAIpHXlbflTr8KFCxcuXJyU09HohwFFoiiWiKJoAb4BLjhF+iuBr7u2pwIbRVFs7hLuG4Fpf6TCLly4cOHit3E6gj4MOHLM78qufcchCEIUEANs/i15BUG4RRCEA4IgHGhoaDidertw4cKFi9Pkz3avvAJYLoqi/bdkEkXxI1EUU0RRTAkIOOGgsQsXLly4+J2cjqCvAiKO+R3ete9EXEG32ea35nXhwoULF38BpyPo9wPxgiDECIKgQhLmq36ZSBCEBMAH2HPM7vXAFEEQfARB8AGmdO1z4cKFCxd/E7/qdSOKok0QhDuQBLQc+FQUxWxBEJ4GDoiieFToXwF8Ix4T91gUxWZBEJ5BaiwAnhZFsfnPvQQXZytp5c3k1bbjrpLjr1MzNt5l1nPh4vcg/NPi0aekpIiuCVMu2s1W+s/f0GNf3jPTcFPKT5heFEUsdgdqxYmPu3BxtiMIQpooiiknOuaKdePiH8nPufXH7cuu1p80/bK0Svo8to7qVtNfWS0XLv6VuAS9izOCKIoUNxgob+o44fHUkiZ0agWPTk/EV6sC4O3NRSxPq+RoL3Rrfj3XL9pHVpWe97YUAfD6xgLsjn9WL9WFizPNPy7WjYuzn4NHWlmRXskXe6TQHEtuHs6waF9aTVaeXJXNiFg/UkuaGBHrx83jYrl5XCx3fp3B6oPVbM1vIDZAy6acOt7bWgxAZYuJimYjIGn2MkHgiVlJyGXCSU09Llz8L+Gy0bv4W9GbrAx4SrK9+7graTFauXBgKNH+Wt7YVNgj7eMzk7hpTAwAhk4bW/PruWNJBqFeblTrzYzrHYCPu5IfMqvx06rYdO94Xlqfx9f7pDl6UX7ubL1/AoIg/L0X6cLFGcBlo3fxj2FzXh0At4yLZedDk7h6eCTrsmtJLWkCYOF1KUzvH4wgwJg4f2c+nVrBzORQRsT6Uq03AzC6lx/PXtiP5y/uz/e3j8ZHq2Le+F7OPOVNRjadwNbvwsX/Gi5B7+Jv5eARPTq1goenJaBVK7hkSDhmq4PUkmbG9w7gnMQg3r5yMFvum0CfYI/j8v9nQpxze3JSEB5uSq4cFkmErzsAUX5ayl6Ywfe3jybMW8Nd32TQ3GH5267PhYt/Ii5B7+IvxWy18/nuMjptUlSM0sYOov3dkckkc8rAcG+CPd0ACPWW/stlAtH+2hOeb1zvAEqem07ZCzPoFaA7abkDI7xZdMNQTFY7C3eW/JmX5MLFvw6XoHfxl7IyvYonV2Xz/tZi1mfXsq2ggSi/biEukwlMTgoEOG1b+tFG4tfoHeTBef2C+WxXGWWNJ/buceHifwGXoHfxl1LVKnnDvLGpkFu/TAPAS6PskeamMbEATOwT+KeX/8h5ichkAhNe2crH20tcAt/F/yQurxsXfxmGThsTX9lKQ3snY+P9mdAnELvDwfT+IYT7uPdIa3eIyE9TU/+t3L/sIMvTKp2/n5yVxPWjolm0q4yUaB+Sw73/knJduPg7OZXXjcuP3sVfxjf7Kmho72Te+F48fF7CKdP+VUIe4PEZSWzKraPVaAXgqdU5NHdYeHuzNMkq9ZFzCPZy+8vKd+HiTOMy3bj4S/ghs4pn1+QC8NC0Pme0Ll7uSnY8OJH5s5L4YM5gAKeQB/h0V+mZqpoLF38LLo3exZ+GxebgmR9zmDs2hnVZtQCckxD4j5iw5OGm5PrRMdjsDp6YmUSQpxvjevtz8xcHSC1pwmp3sDK9kqHRvsSewpvHhYt/Iy5B7+IPU6s389j3WXTa7OwobGRTbh01ejMhXm68fsXAM129HijkMm7smm0LMDTal7c3FzHzrZ3k17UDsPvhSYR6a85UFV24+NNxmW5cnJSyxg4+2VFCXZv5lOmeX5vLptw6dhQ2AlDTNXN1WIwvnm7KU2U941wxLBJvd6VTyANc8v5u/mlOCi5c/BFcGr2LE2KzO7jx8/2UNHTw3tZiZg8JJz7Ig5IGAzuLGnn2wn4kh3uzo7CBHzKr6RvqSXZ1GwBXD4+kf5gXY+L9f6WUM0+Yt4bFNw3nzq8zmDUgFFEUeXtzEZlHWhkU6XOmq+fCxZ+Cy73SxQk5XKln1js7uWVcLJty6ig5gf/5R9cMYUdBA5vTq/n5icmsz65lQp/A4/zk/03oTVbGvLCZpFBPvr115JmujgsXp40rqJmL0ya1pIldRY0cKJdWfLxhdDQ/3zeez28cxth4f24ZF8vau8ailAvsXVPErXtb+dbqjnltGeOPmNF2On5TeY2V7dSXt/0Vl/K78NIouWNSHHtLm3nyhyyXCcfFWYFLo3fRg+iH1wDgr1PjppSx86FJx6URRZGvU8tJ+aECd3p61CiC3Am+ZwgOs422zUdQhWlxH3DyGa/vztsMwK1vjKN9XRnaIUGoIo4PZgZgymlC7qlCFX7i438WRfXtTH5tOwBr/m8MfUO9jkuzt6SJJ1dlM2tAKKklTTx9QT9iThKf58/miz1lrD5YzXMX9Sc+6K+9Fy7+Pbg0ehenxdrDNc7tRkMnw2P8qGiroFRfyhO7niC9Lh1rg5Ha5/cx7sdK3BF4CiPmq3o789nqjNj1nVTP34NheyXNX+cj2h3YDcdHkGyukcxBMuDI62l0pNbQvCz/hFq0w2Kn6Ysc6t/JxNb0x5YL7LR38p9N/yGrMeuEx3sF6Lh+VDQAi1PLT5jms91l5NW28/L6fHYUNvLKhvw/VKfTpaCunSd+yGZ/WQtL9lX8LWW6+Pdz1g7GijYHppwmNP39/xF+3P8GbvsqHYCx8f40GkyM6mtgxndznMc3lW9iqe0daOsS2mo5M6YnEpcchNlNhbWmA/3aUlrX9IwW2fDxYSxlbfhe0Qf3gZJ2L4oiK15KQwBG6eQo9RZsPmqoN2HIbMBjkJTOYbLhMNswptU5z9e+owqfC+P4vRxuOMyOqh3kNOWw9fKtxx0XBIH55/dFJgh8uquUG0bH0DvIg8oWI/d8m8nY+AA25NQxMMIbvcmKWiFjzaEazutXzczk0N9dr9NhX6lkUksI9mBdVi2Pz0g67SBvLv53OS2NXhCEaYIg5AuCUCQIwsMnSXOZIAg5giBkC4Kw5Jj9dkEQMrv+Vv1ZFT8ZaeXN6E1W2n6uoHlJHuaClr+6yH88doedDuupg3nVdrlEju8dwEfXJqOLfZf5B+Y5j7vJ3RBFEUNBI5p+fijmRRN0/xAuHB4pHe/tg25cGKpoT0yHJDdL7cgQNAMDsJRJNnj92lIcRisNFe3k7anFYrIxONYDP4UMo0NkbamBdrtIy9J8OivaaPjkMNVP7aH2xf20bZK0V1WMJ6bDDViajXxf9D05TTmndQ/0nXraLe3UdtRyw/obAGgyN5FWl4bRKgVeazH3fFfumBSHSiHjtQ0FVDQZuerjvewva+G1jQXo1Aq+mjucLfdPYMFF/aT0SzJ4dUO+c3GVE9FutmKz/7ZxjGPZU9yEl0bJzWNjqdGbyaxs/d3ncvG/w69q9IIgyIF3gXOBSmC/IAirRFHMOSZNPPAIMFoUxRZBEI41yppEUfxbZs3Y7A4ueX8PCcEeLPbyBUA02v6Oov/RfHL4E97JfIeXxr3EOxnvMCV6CncNvqtHmswjksC4a3I8GfUZ5DXnAfDWxLdQy9VEeEZweGsqOpuGNM9c7ts2nylRU3hi5BN4qSUbtiAI+F7am5ZVxbj390c7NBiH2YYqRIvgpqD1h2Jq3j/I94VtyIAENxnhzWbk3mrsY8KIyGzkUG0Hw2x2Gt47eNx1GMMd+E+KpPHTLOpeSePTqA8odatiRMgIHhn+CLFesSe8fovdwuzVs2k2NzM1emqPY9evu55k/2R6+/ZmecFyHhz6IHMS50jXolVx9+R4XlqXz7rsWjzdFMzoH8KawzW4q+Ro1dLnMyTKl8U3Defl9XnO0Apb759wXEx9URTpP38DE/oE8NkNw077+emNVh757hBeGhVrDtdwblIQk5OC0KkVvLwun6/mDndp9S5Oyelo9MOAIlEUS0RRtADfABf8Is3NwLuiKLYAiKJ4RtZvM1mlxS3yattxmKVte5trdaHPcz4H4MHtD1LRXsGygmXYHXanLby8qYN5i9NQygX6hnqSVp+GXJCTelUqEyMnMipsFBEeESSXRlGlqueRpgUAbCjfwEv7X+pRluipoqW/P/JEPwBkbgo8xkfQGeFBdocNscGEVgYxahm9PA1UJb+D7hpfEseEMfOOASRN8SbDexciUt2qVQ0AfBK4kncSV9AR7sD99t6IosiYtkFEekSSWpPKA9seQN+pP+7ay/Rl3Ln5Tmo6aui0d7KqeBX+Gn/WXbKOR4Y9AsChxkMsL1gOwEv7X2JzxWZn/tvG92Js13yAa0ZG8dxF/QGYO7ZnozIm3p8f7hjDtgcmIBPgu4yq4+pypFkaW9ia38DYlzZz+1fpNLR3/urze29bET8druXrLpv8i5ck46VR8tiMRPaUNPHlScYRXLg4yukI+jDgyDG/K7v2HUtvoLcgCLsEQUgVBGHaMcfcBEE40LX/wj9Y31Nistid23a99AHp15bSuvZ/N2hVfnM+7ZZ2ZveeTbxPPLFeseg79UxcOpGpK6Zy9ZqruXTNeagC1qNVy1Ar5KTVpZHgm4BW2a2RiqKI0GxHiHHHIrPSz08yVxxqOMRD2x/i/YPvA7BzWSEbF+bw47sH2bAwG7vVgc1qZ9uSfGqsksliuFZBiFaBybsAQ/AB9uVPpbziYwoKF9Aqm4vXsI+p1BZRZLazRiikdZQa2QgftlRuYcLSCYz/eTLZbkVMNYxm+bSlPJDyAIUthTyb+iz6Tj3Ppj5Lqb6Ur3K/Ytb3s9hdvZtbkm/hjQlvoJareXLkk4Tpwrgq8SoOzDnA1Oip3DvkXtLnpOOv8ee1tNcwWAyA1Et54ZJkzkkI5JoR0Xi5KylccB43jo4+4f2O8tMyKNKHrfk9dZ2fDtcw7uUtAAyK9EYUYc3hGl5Ym3fK57cirZIPt5WgUsgAB+cPCMZXqwLg8qERTOgTwPNrc8mocJkoXZycP2swVgHEAxOAcGC7IAj9RVFsBaJEUawSBCEW2CwIwmFRFIuPzSwIwi3ALQCRkZG/uxImqx0fBDoQnYIewLCtEu/zYnqkFUURS1kbqggPBMXZ6XxktpmZv3s+HioP7h5yNx5KD2yijRvX3UhmQyZ0gkahwSy2oPbfwt0TLqDd0k5aXRrXJF3jPI+10UTrD0WIZhsJcf15OfxlxoaP5eNDH7MwayFlbWUAXNfneor2S/bputI26krbsNscBEToqCnSM/6qPmgtNthUAQ4RU1z3xKqiohd61N006AB5P4URWDeU3ZtN3PjkTWQ2ZlLcWsyVCVdiDtPgt8kL++E2rh1+LY2mRhZlL0IhU/BjyY98m/8tAGG6MO5LuY9zo84FYGfYTtwU3SGJ1XI1r4x/xfn7PwP/w9N7nubrrHcZ5OHO0pojPD3qaRZeP9SZRik/9fsyLj6AN34uoL7NTKCnG8UNBv7TNdAd5q3hm1tGoFbIeXFdHu9vLWbOiEgSQzypL2xlS1ULFrWMzXn1TO0bxA8HqwH48JohLDh0NcqAgcAQABpNjTx1QW8ueGcfl324h5W3jaZ/+PGuoC5cnI6grwIijvkd3rXvWCqBvaIoWoFSQRAKkAT/flEUqwBEUSwRBGErMAjoIehFUfwI+AgkP/rfcR0AGM02VuNBOjb4lbN0Futp/OQwAH7XJaHpMjWcTbyb+S5ZTVm8MfENPFWeACgFJYumLULfqcdP44fBYmDk19IM0GrbXh7avgKAMaFjADAXttC4sNsNURngzrQYqcN2cfzFrC5ZTae9E32nnp1Z+7GY7fSfEE5HayclmQ2UZEh/YX186DdO6gg6GkyYDjaw3y2VIKDTAfU2gahejzMy4lzyC56isXE93hfup3jXf/Cti8JYKOfbmd/SZGoiWBtMW1sWeW3PYCmahW74HK5JuoZF2Yv4seRHAAYEDKDR1MgLY19gYGD3ENGxQv5EzO49m6X5S/Fu/AR9q4PDtW6sLR3JRfEXnfZ9v2BgKG/8XMAXe8q5f2ofHvk6k0lWFaMDvUgeEkRHgxl1iJbbJ8axPK2Si97bTbRVxuwONRZEPvPoxCaI7ClpAETunNQHtUcRdcZaNpSvw+54gUXZi3gz/U2CtcEsnruY2e9nMOudnXxybQqTk4JOu66/hiiKLq+1s4DTUWX3A/GCIMQIgqACrgB+6T3zPZI2jyAI/kimnBJBEHwEQVAfs380cHpuEr8Dc7tkjx/c1X7Jfbs/atHRU/Lbm7sDdbVtqjju+L8ZURTJbcplZ9VORoeN5pzIc3ocV8gU+Gmkhs1uV2Mtvw9/eX82lm/EXt7BTbIrGBk6ElN+c08hH65DHdOtMUZ6RrL+kvVsuGQDCkFBfqnkVpk0JoTz5vWn+Px1qDyrSbjsZsZdK3fms83yofPWALKNB7CKAtGDVrO4PZrPi7egVocQGXETAEGKZsLP+Qmtt4qitHqUMiXB2mBEUSQn937avQ9Q7v8ibWml+OHNmxPfBMBDqePtUXex9uI1PYT86TIxNAVfhWRmusDbwsGid2hs2nra+aP9tUxJCuLL1HLWZdXiX2hkSIccc6mBfcuL+fqpvaz78DCmehOvzuxLikLN7A41ACoEbml34z9tGpI90tDGL+Cb+mu4deOtzvPftOEm3kyXrrW2o5Y7dlzOnefrCfPW8NrGgt98vSfCYbHT8GkW1U/u/sPzFlyceX5V0IuiaAPuANYDucBSURSzBUF4WhCE87uSrQeaBEHIAbYAD4ii2AQkAgcEQTjYtf+FY711/mysbT0HtpTB3TZmh6nb+8ZSZaBlZSEAHuPDsVYZMOw8fvDs76K+zczd32Rw42f7KW/6/WuafnToI7Ias9hYvpHLfryMotYiQtS9Sf+F/dZS04Exo56mxTnkvJWG2RjA+bEXEdboy+OFc7k0exzG9Hqnm6Qqxgv/G/sRdMcgZGp5j3MpZArcle4M0KZQUiHdQ68Ad6wOKxsb1pI/SBJImUVPUVC4gNLG/Zz/w/lcuH02WrmIRh1Akl8SN/S7gbS6NN5MfxODIoTXmmPJMMqJtKUTPSKXipxmjIZWGho20dS0hY6OQoI8L8ChNFKx5wuqn04lpSSez4ZfxTPB9aSnX0FV1Te/+R7a7Ub66D8CwCdgBvFuDsaqyzh48CZaWlJPmddhd2DukFaxmjs2Fr3Jyrwv04h1KPAK1BDWp3vJwuKMBpYu2M/h93OZ2Ch9hopkb2LGhDjTTKhOwceqxWyXBO0nUz5hcuRk0uqktXeXzVrG06OepsncxIqyj7hocDB5tW3oTVbazVY+3FZMXm3P8BK2RhPmIul9EO0ilpru983ebsFa14G5sIWaZ/fSWdCCaHFgzmv+1ftmdVgp1ZfyXuZ72B32X03/d5LfnE9hS+GZrsYZ5bRs9KIo/gT89It9TxyzLQL3dv0dm2Y30P+PV/P0sBqsPX7LvVTObYfRilwr2YQ7UrtngHpOi8Za20H71iPoxoadkW7qIysP83OeNHhX3tTB5UMjuHF0DIpfsQUfS5m+jLcz3ubtjLed9miAL7YKLOrYTeGC81DIBKw1HdS/leE8HgHMifbnlpR+rNlci15uQKfU0bJM0gzVvbwIuDn5lGUXp9czdOOVACh0oFTLKW4tAyApPBaszYgdmRzpyERfvYFOWwfBChFPuQyNKgCAKxOuZEPZBhZmLWRh1kIA0t0GM9ytFlH+JdqQ89i5+TXkumwAdLoEEgc9h2FXPnVJX2AMzqe0vQKLo/vZ5hc8gYdHX7y8Tl+rb23tDr+RnLiAHPck3st8g4sDvDh0+E6Sev2El58PLTVGgmI8nWmrC1v47tUMBJnAlJv6kjI4gGhfDbb6TjR2GDg5kviUQBorDXQabegbTOxe0b3K1cX3DyYoxhNBJpA/UcmTX73MuJLLuTLzMS66bxCqEAf+Oj8S/RKJ8ozivJjz6OPbhwTfBNwUbjy4/UFC4lpwiDDgqQ3O8y5Lq+S9qwfz/E+5vHhJMo7Ps7E1mJDplDg6rCCCpr8/jk47nb+Yc+J3TRKtPxZjLmhBN7qn/4VDdJDTlENfv77kNucyd/1c2q1SqOdGUyM6lY55yfNwV/ZcG9jmsKGQ/X3zNLdUbOG+bfcR6B7I2ovX/s+aoc6qUUh7R09Brx3cbau01UmTYuwGC9ZGSUNSx3ohCALqOG8cRhui6e/3ubc7RPaWNnPhwFBuHR9LcUMHz/2UxxubfpsGsqq425q2sXwjKUEp3KR4nUkdSQBkV7fRsaemh5A/yrwyC5afahhuGkBLLwsBV/VF7ik1knJP9a+WffBnySmrxrOYjuHS8MtjOx7iXE8rMUgeT6Wd0qumsVZyR6iWh0PM9NW5o1RKWq5MkPH82OeZFDGJ5IBkls1axpKZ39Av8SXkCggd8YlTyAP07/cOcrkbvRLvBqDddy8Kkx+Rlv9j9OhdCILUqB88dDP5BU/jcPR8N05Ga+t+AKIib0Wh8CAxci4HzT7kyIZhszXz84oFbP5xHmsWLqGqfC87d43BZDrCnu8ks5XoENnzfTGdHTZusem40qBG46UiLiWQZWXf8nHjm5T6HCJpUhC3fzCJwVOjiE72JyTOG5lchiAIVNnKKfE95KzTd69mkLG0FpvVjodCGljv49u9PGN/f0mXUmqqGd87AA0QioAWKK03MOX17WzJb2DMcz9ja5DefYfB6hzHMh1uxFpt6BFjSBXtiaavH+6DgzDnt1D99B7q3kqn9tUD2FrMrC1dy5VrruT878/n8h8vdwp5gGUFy1iUtYjbNt3G1iNbsdqtZNRn8G3etwz6chD9P+//p2rYDtHBjsodWOwWajtqaTI10WhqZH3Zeu7ddi/uSneqDFU8sP0Bqg3Vf1q5/ybOqhAI4jHxVMyI2IPcCbx9IPXvZtK0ONe5DeA+KBCf2VKMFrmXJMzsbRZk7n9NiF1RFEktaWZotE8PTT3zSAuGThtj4gO4dEg488b14r/fHebTXaVcMzKK1QeruXZkdJd7XTf6Tj1quRo3hRvpdel8fPhjzo06F0+VJ98VfMelnXOIP6zkUgTsGDmwv4qAQ5Kf+cuCmbxANQV17Xyn88XfYKMjtQY5AkNGjEHT2xf1nYNoWpyL+6DjA5KJokhp2ds0NmwiNOwa6ssDGDg5gv26L9hrqCKxNBx/SxYzfKw4rE3SvQ25j1U165mpOkSMTDIJifYOQsOudJ43VBfKm5Pe7FGWj88wRo7cREPjJlqa08jeUYmHbgDu7pIXlb//ZPr1exsvr8EYV7Vh3FmPcowvI4avZU/qZKzWZiorP8dNHURUVLedu+LIIurr1hAZOZfyik/om/Qyzc27aWjciJfXYOLiHsThEPl6/j4u0N3G5/Y3mC/vR0C/1QAEJpdTmJeKXVnDkSOf0VI3nqh+fiSNCWXdh4dZeP8OALyD3Jnxn2Q+zv+Ajw5JJqEVhdKA9/Dg4bw47UX8NL2ke+SwU95eToOpAbPSwOWvJyM2qNnzfTEBOY1suX8H5jgfZt6R3EMzDdOF4any5EDdft4dMA59A4gtkhmzE5F5dFCGgwFIZref+3lyWUoE+s0VKC/shTK/Be2wEORaJY5OG4YdVbh1OSd4jA2j/ecKHEYbdrMVwSFQvyaf97WSO+1Rj6vx4eN5eNjDmG1mFucuZm/NXtLr00nfnM51Sdc553IcZcHeBXwy5ZPfrd3nN+cTpgtDp9KxuWIz92y956RpP5v6Gd/kf8Pq4tXkNOWwaOoigrSnN2Btspmw2C3OSYH/Vs4qjV74hUZ+pMWIIkCD3E8alD0q5AEUgRqErtmERwW9Tf/rk1d+Cz/n1jHp1a10dNpIK2/hyo9TiXt0rXPiy47CBi55fw8quYxRvaQPy0er4uZxsRgtdma8tYNn1+TS+7G17C4tp7hV0pZ3Vx5g7DfjmPPTHJI/T+a6ddcB8ODQB5k/aj5r4pYzYI/KGVnyadyZtq8Z0WznNUwoBwfy/e2jeWR6AhE3J+N7eR+QC8i0StRxkoYt91AReNsA3Hp3L75hsTSTkXk9NTXLKC19k3ZDNkWFzyNiJCDKg7n951LbUcuD2x8kTOVAkHV3228dOI83z/uOPv0kAeHpkcyQIUsJCjzvV++jTKYiKHA6CQmPEx70EIXbEqgt0dPeNaAeFDgdN3UwulGhYBMx57fg7h7D4EFLiIi4EU/PARyp/AKHQ3o/2toOUVj4LPq2DA5n3U5bWwZ7UieTX/AEHR2FBARMoa3RRHF6Pe1NZjzKw5m7+xWq99xKzf5rEUUBtXcNnTaph1FZ+S02exPR/f2IHRjA0FnRALSrWgi70cyPTSv56NBHTI+ZzpLpS3h8xOMoZAr21u5lwV5p8tnWI1sZ+OVALvj+AtLr0pEJMnzUPviHaBkVosFfISNRI6c9r5l9q3vOCxEEgQvjLkSVbqZ1WQEYrLgPDMCmVaJGYBE6tuDJc36+WIEVDXoaAty4oLWRIW9t5/mWVqdZU6ZW4Dk5ClWYtG6uzE2BbSEzwrgAACAASURBVE4A7/VazvQ+t7PS92csWS2o6kQuib+EdZesI847jrn95xLuEU6cTxzzR81n6ayl/HDhD/i5+fUQ8hqFhkD3QNLq0lhVvAqzzcz92+7v0SP9NY60HeHS1ZcyZcUUjFYjS/OXAhCgCSBQ01MxWTx9MXE+cTw24jE+nvIxTaYmLl51MTsqd5xWWfM2zuPc5editZ9ej/CfylkVpnj32weIrOrqmiJSOTeJUXHSrMaGjw/RWdw9c9L3yj7O8Lk2fSe1z+8DuUDYM6OdDcAf4au95Tz6neSxEuGrYVZyKO9tlQS1Si5j58MT+WbfEV7bWMD3t49mYET3QJ0oikx4ZSvlTUbnPs+41xCV9QRoAtDYE6mwbO9R3nNjnmNWr1kA7HplL1GNFuoG+tFL59ZjoHmezMjKp889rodgazEj0yiQuZ1Yw7JaW8nIvJb29m7zyYDkjzl46FZaCsdzzgVv4hOsZduRbdyx+Q7uDzIxIHgcYWFXoVR64+3dHT3Vbjcik7khCL9dz7CYbXzx3910HhPaotfgQPzCtAyZGkXtC/tQRHoScE2iU+utq/+JrKw7AQgLuJoOSwH6tgwkPwPQaKIwmcpRKLyIjb0bN85j6bOS660gE9DolBjbLJSFZvL4Pbfy+d6HSJKtBaC5YDK+vTdJ5VjCSOr3IO9kraQpy0GLey1N2u57v/bitYR7hANQbajmo0MfsaJwBctmLeO5vc+RUd9tVouTx7DsvG/p2FdL+7bKnvfAIaKdGUPQOMnrefMXuTRWtpFgbKZN3kbtcC1tjXZMuz3wkMEkT0mIiwIU93Ln+qJqelsUjDMryUkycFC+iG8vegcl3sT4a3u8G0arkct+vIzWzlbuHHgnSzIW80zWrXjjSdDtg1CHnjpM8tL8pTyT+gwpQSksmrZIqococtmPl2GxW5gaPdU52a6XVy8WjF1AX7++J3/+dgu3bbqNfbX7euyP9oxm9UWrnb9XF69G36nn6sSre/R+tldu5/afbwdgWvQ0shqzmBg5kUt7X3pcCI0yfRmzvpe+KY1Cw6dTP6Wff79TXu+Z5H8mTLHG0P3xyxBoMHRisTl4anU2jhnReF/cHfFQ7tFte5brugZt7SLWasOfUpejQh6kqe/vbS1GIRP4+b7x2BwOblucTna1niBPdQ8hD5KGdm6i1LWMC9RxTl93RKU0WNtgaqDCsh1bewLG8rkMU7zCnPC3mdVrFg6HyLb8erwazeT4KRlyRRJe02MIuHcIOz0FNmFl4tio44Q8gMLH7aRC3mbrICPz+h5CPiLiBvz9JyEzTsW71zbaLcvJzr6XPioDc/rMJlwtR6dLICBgcg8hDyCXu/8uIQ+gclPQZ3gwADKZgEqjoDi9nn2rS/nsv7ux+mloz2pk57cFiA6RDn0nPt7jnPmrGr6iVb+fyMhbSEx4Ea02nv793sXTI5moyFtoyBnvFPIaDyXTbunHnGdHMui/GtZFLWJd9Ro+rtpKrVVAdMhoLphMR600DhKkqqKp4C78zLuoDzqAZ2Antw24DQ+VBwm+CU4hD5KZ6t6Ue3GTuzF79Wwy6jOI8YohyjMKgIcKr6X25QNOIa8M0RLy2HBkXipUMoHONaW0ZtZjbLOQu7sGY2UHvjYdLa1+1K1yx7RbEsAGATYa7fwn/FUui7+f+91uJbDXS5xvVOHjkJFSKkenaOTW1S8x9Y3tDF2wice7FnoHSK1JpbytnAWjF3B5wuW8N+sDOmPkyOwCDW9lItpOHaDtkvhLuGvwXTzd+zEcndL3KQgC1/W9jhJ9Ce8ffJ+UoBTuHHQnjeZGFqQu6JHfbDPzxK4nWFe2Dofo4NOsT9lXu4+b+9+Mr5sUz2pgwECeGPlEj3yzes1iTtKc4wZfx4SNYXjIcADWla2j0lDJlzlfcumqS1lbupb3D77PwQYpztKxDa/JZuKuzXfhEH9/QLozyVml0Rc+vpNSm40kUbJFrpsRRriPO/MWp3HxoDBeu3wgnaV6WlYWEnjHQGTqbsFmqTJQ/3YGntOi8ZwQcbIiTosfMqu46xvJTBTq5cYNo2NY8FMuAGUvzODOrzNY3TXjcWi0D8vmjTruHIZOG5ty6jh/QCjvHXyXDw99iLxjKHatNFhoqroSW9sAZ/oJfQLYmt/ABSh5AA2682PxHtXTU+Los/4tngdVVV+Tl/8YAMFBFxIcfCE6XQIqlT8Om8jXz68iZMx9zvSCIMfPdzyNTZsZNPBLfH2Pv7Y/irnDyoE1ZQydGY3KTUFRej2p3xfT1mgmXCkwRKsg3WjjiEW63mGzYogZXkbV8g009l6KzKohSfkR+7YL9PdQ0uquRNHXH7vNwc6l0iBh7+FBnHtDt2YpiiKX/3g5uc3Sc5QhMqBlIMPzbkSQd4LGQp/pktOZKChxdwvFZCpn9KgdqNRdDdMJGrf5u+ezonAFUZ5RPD/meUJ0Idyyai5vZXTfU785iWj6ST1Te4eV5c/spZ/dgZdcwKaScaDZgloGg9wV/NxmxeCAQr80igPTcfdTMHr3HKo9ikkLX0+tRwkTi6+iT8Nw5/mtsk5W9HuNqvpLcZilxmjWgFAem5HI8pKFfHToI1KvSkWj0NDQ3kluVj2xa8rBJqLp54ffnKRTPi97h5WaZ1KRuSsI+e9wBIUMq8PKzJUzqe6o5oPJHzA6bDRfZH/Bywde5qvpX/FZ9mfcMfAONpRv4N3MdwFI9E0ktzmXMWFjeO+c98huymZJ7hLmj5qPSq46ZR1+yTsZ79Da2crU6Knsr93v7FWAFKn1rUlv8Vraa1S2V7J4+mJePfAqO6p28Or4V5kSPeU3lfV3cSqN/qwR9A6zjer5e1ihtHGJVRLg88LghlEx3LfsILMGhPL2lYNOeY66N9KQ6VQEzP39HqFmq53+89djtYskBHvw6mUDiPLTcuNn++kT5MEzF/ZDb7Qy4ZUttBitzB4SzsuzB5z0fEarkSkrpjA4cDB9hP/j1e0bUPns5rmxCxAdcoxWO1vz6vk5r57eyPgUybYaeNdgVCF/bMUjfdtBDhy4GICQ4EtITHyxRyNx8Ocj7FxWSMrl2YTGeRIYOI2duyTBrlT6Mmb0LmSy3/YB/hG2LM6jtcrAEJMFmdlOrdXB3g5JMx1/eTze68vIkR8h3isKZXNPzSzPbMfqgI5ADXEjQuiV4IMuVNfDjJfXnMeb6W+iEBQ8NfopDjccZoB6KG5aJQ67iMVxiIzM63E4uicYJSa+SHDQBZSUvI5Ol0Bw8Pk9yrU77DSbmwlwD3DuMxU207QwG92oUAQ3OZ7nRvW47631Rn56K5PhFhvqo/VTyBAEsF0Vx4aSn7lo0hQONhzk7i1307d2DGNLZwNQM3o/bhkR+BiD+XrgswS1RzOp4moqNUVkJGeyYvbbXPFRKoer9MhlAr2Tv0WhbmX1RT9gd4gMfmYjepOV5fNGEptaj/FwI/7X98Xe2ol2aPAJn4thXw2tK7tdSRUBGtSxXlimeFJlqGJw0GBAcjCYtmIadtGOyWZCQEAmyLCL0jP0Unvx0NCHmBE744SN5h+hydTEurJ1xHvH8+xeKVYSwDmR5/DGxDewO+xcvEr6FlaevxK5TH6q050RTiXozx6vG4fI7mAV6R02LukaN8mqamNtVi0ASvmva7HqOB8MqdWIVjuC8vc9yPzadqx2kTevGMj5A0KdH+jSYxaa9nJX8sl1Q3l/axF3TDr1AhrfFX2HvlPPjf1uZEBAL8J9LuCbfYOZ3i8ct646Xjk0ggeWH6JvhuTh4n1Vwh8W8gC1Nd8BMHDAp/j5jT/+eIkeD183hk+807kvJvr/qKv/kaEpK/9WIQ8wcU4CIHlPta4tJTijnnFjA9m+o4a0ZYWc46nE1BbCluZORukU6I55JxLcpHupCtLgOySQ2hf30+6hwu/qBESLA/3aUuLmJPL+5G7Nb3xEz3viTgrjxh5g67bunkBu7kM0lG+m0bgeAKtNT0R4dxwhuUzeQ8gDONqkF1g3KhSFv+a46/QOdOeih1PI/L6ESF8VyvI2LOXtKEK0hCUGcXPiVQBMCJ9AP79+VLvlMPOqBH5ckEdK47lUm1qJm+yDvqOBxNheTBiRwNavBJQHlaSqcnloWh+e/jGHFmMnR4w52Or60fvRtViOiaP/ZWo5L43qhTGzwTl7WrQ70I3oXnjFYbIhqOUYdlahCHJH6a/BlN2ErcGErcGEf5IfQX0GO9N7qb24IO4Cvsr5ir6mXuRrytCqdSycshCz3UyYLgx/jf/JX4A/gJ/Gj6sTrwbg+THP8/COh0kJTuHxEY8D0nO6qf9NPLrzUbKashgQcHLl7J/IWSPoZe5KNvspqHbI8BgawV6bFba3sSlXCrLV+Su2RAB1tCeGnVVYa40nXbf016hskbS53kEepzSRDIny4ZPrhp70+FFWFa+in18/+nr3x9DSycz+IVwwsKdJxn6knUetKsyocB8ShC454CRn+22YTOV46PqeUMgD1Je3ERjt2WNfbOxdxMbedcL0fxdyTxXeM2Mx5zbjm9/CtBuSSP9KihIpeKgwNXey3SoSrVMSMiyYQA8V7iIYUquxlLXRtr4MAEe7hYYPuv3ZO9Lr8To36pRli3rox+fUtHyL2ArNMT/RaFyPpi0et8gAiotfwdMzGS/PnoLCWm/E1mBC09fPGVpb5nnyhlKjUzGyq2Gz1nbQ9FUumv7dQlAUHYgOM1/P/Nq5b+jMTvb/KGmqyYNj2RKyBS+VF0q5kvLyWtgZTd7mBlLc3Nlwz3iyG7K54iczMdoB5HYJ+YRgD5JCPNmQU4d4STI+F8djKmihsaQF8YdiRJuIKkyHKbcZw/ZKVJEe2OpNTucH/YYyZO7KrmUjCwi4JRllYLd31nlR04jaqGFs+2CyfUuJ9YzF82sTvpfGo9T8PWvy9vXv22Ng9yjjw8cjF+SsKlrVQ9CvLl7NsOBhp+2yeSY4awQ9SNEr3VVyvKZEMwVIKKghr1aayFHfZuZAWTNfppbz+mUDT7hQgyJIeuGs9b9P0OfXtnP7kq4ohT7Ha2K/FaPVSH5zPjcm3MTyFw7QXC1NVx94biRDpkXhplViymum6TNpkFQZpsN7Zk/PgSM5zVQXtzJsZswJG57OznoyD96I1r0XMrkb7u6xBAXOxG7vwGiqwEOXeMK6mTustDWa6Tv2lxGr/xnItUr85iTQ+EkWwSoZE8eFYs6oZ/g1iYSVtdF/fDgqTc/XX+GjpmlxLsbMBnRjwtCNCaP2hW7vjvafK1BHefZwOT0We4eVxkVZ2BpE/IXL8J7Vi/rsEZRHvohf4fkE9hlLjvl2srLuZOCAz1C2BWHOaULmoXSaNnyvTJAaGhnIVFIvQxTttLSkotXGoVYfL0yUwVqC7+vusTscVtIzrsZgyGfQoC+cjUrK9GgaKtppa9LjFWxApfJFLpc8cqbPGcIn/RdyaHkD/AQHNpTiNl3qIX5+1VW0GdyJ9nNHLhPYkFPHyowqDlfpGTosmOU2My9mlbDIzQvhx57LSFoq2lGG69D0l5QPrynRALjFe1P/wSFalhcQcHN/Zw+6T3sUfu2Slt+3OQZaRKyCgZbvigicd2a1aC+1FxfHX8zSgqUMDR7KtJhpVLRV8N+d/6WPTx+Wn7/8jNbvVJxdgt5iR6PqNrmEeLk5Bf3+shbmLU6n0dDJA1P7EO7jflx+ha8G5ALWOmOP/aIoIlodzg+v3dKOh6pnQ2C22rn6k1QmocAOeLr98YlX2U3ZYJOhXdOX5rrumCSZGytoazAx7aa+tK4qRu6jxnNiJG59/ZD9QnhtWJiNucOKXC4jOtmPnF012DrttDWZmDgnkXr9QgyGXAyGXGee4uLuxUQCA7uXFqgvb6PxiAGZQnAOWgZG/b6ez9+BOtYbRaB7V4hlO+4pQfgm+BKU4HuS9N2TYnSjQlF4q/G9OgGFtxuiQ6T5q1xavi8i+IEUHEab0/f8KC0rC7E1mFBGeOB3dSIKbzXuKVcQabqY+gOH6Pi8laiJD5KvvIvUvecSln4vusae4SXqV+yl07cK9+YETKZKDmfdQXu75AXk7z+ZAckfAtI72dCwAZXKj1Z9GlGRtzgb8nZDDnq9FA/nwIGL8fObSHzcf9FqYxk7R8b+A9ezuytsT0zMXcRE344gyLkx+QYW2hbRuLgBT1MAlZs6iRgTQaB7IMco3QyKlLzEDh5pxU0h58lVkqLxtJeDhZ5asDtQ9/ZB6a9BFe2FMlh7nMuyMkiLbGoklu9LaNtUgVdXGHFrrfTthTw6HGttB3JPFeb8FvQ/ldK6qhivGbEIp2GG/at4aNhD7KraxYPbHyRIG+RcyjK/JZ+6jroeWr0oivxY8iNtljanWehMcXYJeqsdf113dzehuBMvg4rtQdDcYaHRIE2IKm8ynlDQC3IBZaA71pqeLpb1a/Kw7mwkdP5IikwlXLLqEh4f8TiX9bnMmWbPhhImGuAupPM2fpaNx6QI1JE9TRungyiKVLRXcOP6G4loS8BcB2Mvj6f/hHAajxg4vK2S3N01NB+qx95sxu+aRDR9u7wy7A7qy9oJ6eVF2eFGZ5CtvatK2Luqp7a1+IndJJy/C6VbP4o3T8UrZhde0ZIEUCpCUKs9CQzoXnrv589znb2KowRE/nMFvSAT8L2sN/XvSB5Q2mEnHiw8isxdidd50VhrjSi6Ip+69+82g3mcE0nrd0VUPbITkHqAXlOi0PT1R3SIdJZI8zR8Z/dG4S2578pUcmQqDX5XJtDw0SHY4kVQ2PXU9f2MqsGvEWqYS0Dz+QgqGZ19iilrehS7woCmoxeFqUeQFnUDjSaSxsZNlJa+Q3T07dTWfk9O7v3Ouvl4D8PLaxDV1csoLJJcFON6PUhR8Us0NW2hqWkLyf0/oOLIoh7XXFr6Jhq3cEJCLkYmyLh5yE2092vniY9eJy57DJcqbzjuPgV6uBHhq2HJvgrWZ9cilwlcMyKKz3aXcUuYgJtGziCllUdGn3j8yWp3sC6rlju/z2RleDDCnmo0AwOx1nag7+oRyD1UyD1Uzm39T6UYdlej7uXlfNfPBGq5ms+mfca0ldO4du21AKhkKmSCjPcOvsdTo55ypv0i5wteOSCtddDfvz/JAaeOGfVXclb50RstNtxV3W2XrtFKvE3OjH7BBB0Ts+XYiUi/RBXhQWeFnprq73A4rDiMVqw7pSn7uZkZHKiVPIIW7F3gdFcUrXa8a99idvJCHHKpMTHnNdP4SRZtbYfIyLgOm639xAUeQ3pdOq8eeJXR34xm5nczAfAxSdEM44cGIQgCAZEeDJ8Vi1wmUN/V3bd7u9Gh76Sp2sBP7x5i5ctprP8kizXvHkLjqeLSh1MYPDWSQedGMmVuX/wiLUSMf4WE2beAOpe6vGCMDQnU7LuJ2vSrqMu4gsNLniYufCmentLLmbGhgubqDrwCNQiC5LI4884BqP+ikBF/FqpwD4IfGorftUmn1eh6jI+QZgqfALc+Uk9AUMuR+7phqzPS9GUuppwmLBVtiCYbPpf36WFzPoo61gu/a5NQhukI0lzI0Pg1+PqMp8ZjEW6zVXhfHkt++8PYFQYC/aZj0hYjihYCA2cwYXwOgwctQSZTUVL6Oql7p1Jx5NMe5y8r/0AKT533MDZbOypVIFFRtzJpYhH+fpMAyM9/kvb2bJRKX+RyHYGB01EqfSkpeR2Ho3tWuIfag1fmPYZ/hA7ZrlCsncdHo3zq/L7Ut3WSUdHKk7OSeHym9Heoqo195S18uL0E+y9CfxstNg6UNRP/6Fru/FryUb+jsg6HQ6T+zXRavs0HcI41NBk6SS1pQuauxG+OZEI82pj+FiqajMz9/AD7Sn89CuexlDd1nDCabIguhPkj5xOqlQae7025lxmxM1hZuJL/7vgvOU051HbUSt9y6Gg0Cg1L8pb85nr/mZxdGr3F7vREOdZtVNcpYjxmmcHy5hOHAjYay8mRvY6PTxQ1eR9RceRTQhue4mhHsWZXIWVDSohvSKHYL53Mhkw0xcH42+w0x0jBPT3ah+NRKtlLbaKezMz/w2proaJiIbGxd5+07kfaj3Dj+hudrmRHGa2ZiJunCs0xPRWtt5rhfbzxqDbQHLmejJ3Poi8dg75stDNN0YF63LRKLv/vULTeanQBjdTWriQ29h7aZF/Q2prvTNtWIflUB0R6oFZcjKGlEzCy/ZsCLrpvMHarg91djcr0eclofdSoNf+eV0fh44bC59QLjpzWebzVhDw6HJlWCQ6RzrI2Wpbm0/xNnmSeUMrQJJ18ARtNkl+P432DXmZP6jkUFb1IVORcHA4T/fq+RVDQDAyGO3CIVjx0SQiCDLk8hKSk18jKugOjUZphHRZ2FfFxj1FZ+RlFxS+xc9cI57kHDZTCDgiCwIABH9Pensu+/ZLyEBl5F7Ex/wdAQ8NGDh2eR0vLPvz8xjrzy5Uyxl3em5WvpLPx02wmX5/Etm/ySZ4YQVC0J5MSgjjw2GRsDhFd13yUa0dG8dH2Yuq6woUfrGxlcKQ0nlHdauL8d3Y5e9UPTUvA213JIysP83q4gnva5QitnQTdOcjpafTod1msy67lldkDuHRIOOp4b8x5zYgzY9mUW88PmVW8dcWgky6MXtrYwesbC9hR2ECL0cq2gnruntybm8bEOOWEvcNKy7IC3AcGoEkOwOoQ+WBbMS1GC4t2leGnVXHgscnHjW9dFH8RF8ZdSKm+lGivaJrNzeyt2cvqktWsLlnNDX1vQETkniH3sLJwJUsLlnJFnysI0YackUHbf8/XehocHYwFekyRlxlstJul325yGfVlbSfMX17xCWbP9TT2krr4ho4cymqWYHBPwtPNi4SGCLw26vBqGkWcLZr3v36DOWU3oFPL4RwFyG3YhpUjDgtHkRZGccgT2GxS6NfSsrfx8xuHl9fgE5a9t2YvdtHOh5M/RCFTcNOGm7hMuBmhyJvgvp6Iokh5xUcEBkzF3T2aqGhPOmpaaY5djUZhQuNXgtw2gqSRCQTFeGJo6cSuWUVl/Wb6eD9Jadlb1NWtRqH0prV1L9HRdxAYMIWKvFryWgSUajmzH05BkAl0Gq0cWFtO5sYKitPr2fipZIecflt/fEP/Hs+HfypHzQnIBNzivPG/qR91b2diqWhHOyLkuHj9p0Kl8iM8/FrKyt6lpSUVudwdPz9pFq9Od3yvIjBgGqNGbkOvTyM75168vYYil6uJiLiJqupvMJmkGEoDB36OTte7R14Pj0RiYu6mrOxdfLy7J0v5+EiNQ0HhM6R4LnVGEwUIifMmeVI4hzZX8vE9UsiNgr11XPf8KHQ+bk5haWgx01JjJDzRhw33jMdmdzB0wSYW7ihl4JXS+a77dJ9TyE9ODOS2CVIgt/zadj7bXcZqQWDz7WOca0hY7Q62Fkizwe9fdhBvjZKRyQG0rCikaXUx83cXEIGcHYnVjB8kOQSYrXbuW3qQwrp2nhsRy/rcOjJzGtAB5w4Po9Vo5eX1+ewva2bR9UPJqmpDm9WEKq8Zc14zyj01rFDaeK2o1nkPmjosFNUbiA863kQpCAKx3pLzg7/Gnw/P/ZA5P82hpbOFRdmSiSzOO44rE65kSd4Srll7DVqlltSrTr2uwV/BWSXojccMxna0dndFZUY7E/sEsCW/gYtUOsIPdtBYacA/XNcjv6UrKJpV2/2gNaoWtrvnMlp+GWAmxO5DopudyaZJaFsngRocMgvIpbz1LaupZzWapP9n78zjo6rO//8+s2QySSb7vpGwJhBCNhZlEUVkURF/iliXr6VVqmJRW9uitupXraXVfltoRWstVSuIFhe0pVKQfVPClgQIhEAgG9n3zCSZmfP74w5JgLBngXDer1deM/fec+59biCfsz3nefpgt1USlP09fIpHc/jGJ0jfOYOUlGX4+Z7qVtnkaOLjgx8T4B7AiKCRNDXaWTluLaveWkfUzX/iuvFv0th4lNzc31Fe/g0pSctwlDXS6J+Nw62esIDHOFHxFiGj5+DXbx5uXoMIDQ7k2+9+A4C7ewRlZZof9+HDvwEEUZEP4ebmz8BhcaxnA4NGhrYumJk8jIy8PZacHSV8/Y7mIz1geAjRQ3pfusXLxRjiid/0/jQdqcZ7wsXnOw4KvJm8vDdxOq2kpX6KwXD2NQ8hBGZzJGZzJJ5eg/DyHACATmcgNeVjSkq+wtd3BBZLx/FY+sb+mNiYJ07pnRoMFkymUBobc9m67UaShv0dH58krNZ87I4GkqcK6ioCObq3vLXO8t/uJCjaQv/UYAaNDGXjskOt12+fO4zowQFc1y+Af2cWo9MJNrt61L+9ayjX9wvE0i7Uxs8nD8Jk1PGXDUd4Y3Nuq0fc1twKbC1OXr87kRe/3MfDH6Tz6cOjCOhjwba1mI/Rfk/VH+dy3OxOdFwA3x6t5N+ZxTyAG6Ff5vEQkGI3c6TJCWsrSY3yIm1KHGs/P8yKr3L4fE8Bv2400iwl1sQgzJllTEMwwsefdQk+6E16Fq49zN1/2MTHvgF4VjeTHWrihoeTcGs3wpYOJ6Vv7cWnny8bZm4gszyT3OpcvE3e6HV6YnximNhnIquPraahpYHPcz5nSOAQYn1iMeq6Z+qz9+yMdUr6PreSJycM4OmJAzm+r4Kv/qTFrDANsHDfj5OpqG/mn3/YhamsmYEjQ/D0NpF2awxGk5739r2Hd/6H+JryAHCrD8fQ5EODWw1m45/Z82UFOiDZQ0+km7a0IaWk3C6pNVZiv+UZipr1hLudOvWS4vFvGr4qp2rwfygNW4apOYqRSf8+JfvVN8e+4an1T/HLkb8keEciOXsysETuxN2vCO/obYSG3IHFeyg5Oa8CELLv++jsZloG51Np+i/jxu4i79jb5OUtAs6+X8DHJ42amnS8vYcxPO2z1vO2hhaM7nr0pyU6Ob6vgv1bikmbGnNGo6joHKSUmkD7jcDddO7F4q6iru4A0nUTMQAAIABJREFUtbV7OZr3JwACAsZTVNSWnStl2D8pzQmjpsyKl7876/6R3XrtsUU38refbKTZpv2/jxjky/SnU2hstvPCin0s39kWkG33ryZiRlBbbsU/zBNDOw+5l7/az+ItR/nbQ2mY3fQ888le6prs7Hj+ZvYV1fDwO9+RWqcj183BnQY3fKO9GTssFIcrkufWsSEckw6Wbj7KZ1godjjxdgqCjTr2tAuHMf6BQWxfchCbhL5uOoZ66NnW0ESB3cFuE7zi5YneLnHr74vnXQNY8tEXjD4egk+7PrHdw0DA+Ci8xkQgWxzUrc1vjUkUPDcZt/Az/1ZanC3sK9/H79N/z54yzTkg1DOUPt59CDIH8UjiI2cEVbtYromdsdYWh5Ytx6DD3uIgZ0dJ67WmnDoK9pQzcEQoZjc9TrQhKEBglBdrTV/w5p4/87+BOuoKk5D2Ibjnh1EfmInPwNUc3aRtDAqM8GTXlmI8dOCuE2yos1Mf7MDLWk9fYGO9nhG+QSwtqeaxkHD6e9yJ78hByGMCdk+GGj2lcUs4vPQdou+7i6KqpfiHjGFn6U7MwsjQmjGs23KQPjf/BbP/sVb7T5Ss4ETJCoQw4N4cTcmQ91qvBQdMQa8306/v04SH3cXRvDcpLm7z540b9Gv0ejNNTSUEBU3i6NEFREc/csrvzt2z415F9JAA1YvvYoQQZ4RF6G4slngslnicsoVDh16iqGgZBoOFmJg5HD48n6PH/0BQ5ERq9MvxCL6VaU/N4Ms/amKVva2YZpuDiT8cTMnRWrI2FlJfZUNv0PHTiQOpy6wkJdSHhFui8fUw8tHL31FV3ICnjxvTf5KCr2vvyrNT41iZWcwP39c6eTF+Zv4wZhA6hyS1jz+PhAYiK2pJbjZQO9KPH38/CSEE31ibGbSmiJs2lbJGtrBaaAvuO1skGboGbknOIHFfAtERfcg4Xk/xP3OY5GNks83BUHc9zQKOjnyHOM8S3P77K7yfTsSRW411dS77vn6BfnEZ1PqmUVo7jypfAx9mFvFEsxnDyqPUf1sMTomjqgm9nwlHVROlC3cT+MhQ3PudGqjQqDOSFJzET9N+yoP/0XZGn2jQZg4yyzIps5bx7i3vdtm/ca/p0ZfXN/HccxsY1mxg6PhIMtdrLWz0EH+O79NW22+fO4yv38mixXZqrzsz7bfM6KstNpYenMBr5u342oKY0TCMIckrqMi+heC4g1w3+jOW/ioLa20zTUJyzOBkpUczz3scInLsn8msv5W/Va1D59Rx3+4X8Wr2Zcw9A0hIC6ZufQFVu/ZyZNzPTnm2EEYKWkyE6RuwlsZht1nw6aNt0jGbY0hOep+t27SdqQkJf8L2r2aKAt/B4V9HVNSDREU9hE7X5lEkpYO6+gN4evSltm4fvj6plxwpUnFt0dxcybbtE+jb92kiwmei05lI3zmTmhrt71Gv98ThaCCmz+OEBPyI95/9Dr1Bh6PFyazfjaGlyc6yl7/D3qKNKqPi/cg/oK1R3fnTZPasyefo3nL6JgWRf6CSqMH+TPlRW1yptdklzP57Ov1a9NxhMyGdksSbIgmJ8Wb14v3ozHp8fE00Wx3c/78jyc17haKMQTRtCifGrMPimpJqMBtYU2xl8NR0nF5/waNxEDGZL2J5bBiHX/uOQNcUpUTSMHgHhZGLADiR/iAR0dOoqv0Mbyqgf1tKxsKtsxk58SH26lv4+fIMPosOI/i45tThc1tfvEaHU/qn3bQUNSBMekLmJqP3NiGMZ/7tlTSUEOQRxPHa44R5hfHh+sWszF3Jr+3PEDMzFfeASxs9XxM9em93I8OatdcpabfYOuXRoZQdr+ez13fy7z9n4HRKPMN3g9RhreiHs9mLW9zbdrE2NHmAh6TaXIpHqLaJIyDuvziA/IJ3iXjwAX704U6ka5ozwquIyOv/DEBw9gh0QRsJbIjAq1lr0Tf8O4Mh4ybjO60flhsiEcXlFGZ/iqHRF0OTL9XRawnX1yMEeIa0bVoaMvgPBAdPQaczMmL4l7S01ODndx1F+duJ812A35gBHf4ehNDj7ZqjPX0toDfR0tJCQUEBNputp03pctzd3YmMjMRo7Nr5XDc3f8aNTUeItimVxKFvUVX9LU6HlcDAm9m9+wHyji2ivPwbIof+gIIMfwKjvPDwdgPcGDI2gr1rtdSS+QeqcPc0otMLPv99W8jfm/4njr1rC9jxr6Mc2nGCgcNDKT1WS4qfhfl+IZTk1SJdeQ4z1rZN/dz+o6Eg4Ou/r2D3hhrqDB9CABxpfocjzU4sOqh3gqxuIXpIAH5RR6mogkaPgzTaj+JfHUdIiAeOcistblVU9VlFVeTXgJaDODj5YxobVuMd3LZGF7jp91gnLiY0bQk7t1QTnLCaX6XE8Mvj0/jfsdcxUG/Ea7QW0yr4x8k07imj6uODnHg9Hb2fiaBHh2HwOTUdZ4hnCFJKQgotCJ8WJq1OYBIJWIWNPcd3MSpgHJ1NrxH69jHW7e1cKQ1GPWH9fPAL9aDqRCOevjqixmgtuJQGKnLvxiuwLbJedbP2j9Lftz8Tb7iDw7v+CIAQBo4de4dXt4cjhQ+fPnY9uSWlBNTPba1bW2jhkSNvIFzbEzb0/Zgbjsxk/coMbp6WjN7HRIzPbEL1M7BmlZN1cCf9c+/kiKimskXiO/HX6A16HI56goImoXMt1FgsWpAsZ2ML0mrHEHT54RWudgoKCrBYLMTExPTqhM9SSioqKigoKCA2NrbLn9de5EET//ZZwEaM+JKysjVkH/wlXnHPE+32NCMn3ofVWkhp6b8JS3HSFLiAAO/vkbViAjf9z2B8QzxZ8cfdVBY18OCr12HyMJJ0cxR5GeV8894BvPzc+XLBHhyukUDssEBihwUR3MfC7v8ep7bCyvBbYwmKbWHf/meIvWUr7XelRA2rJn+vH3Wu5SmfiGP0ueELysrX4O8/lrq6/eSNfp6qLf8loO52bOMzKTYsRQptM2H/fj/H4p3I7t0PYPI+gcV4P3Uty/C2DCXhuWk0tSSzbftNhCRpmaxiAqv4aeBuFh14gJ9O/wm64jriw7TYVh5JQeg9DDRmltOYXkLZoj2E/CQNe2kjFR9lYwzzxGtUGPVbirBlt/n16zwNVE90J23YdXQFvUboAXR6gdMhT/G4OYlviCb0Ubf8sPWcxEFg/2WnlKtyhTiuPDSXnxUXMtI3gaTgLLwi/kFdwfcYEbKeX403Yy14jSi9B41AVNQPiAj9ET62OnasPoLTqj3Pa1gLRWWHEVv7MOF22SpI7gP8sPcxsil3F/9TfjsD0BZmw8buQWcU2O316PVnJuW2u1LndYZP+NWOzWbr9SIP2hx+QEAAZWVlPW1KKyeTyeze833o+zpVTaWcOFxCWdnXrWUqat9n3OxowqO0vR0z5qVRdaIRb5ePvJu7gdueGMY/nt/K529o8aFiEgMJjPJi5O1ti5I3z9Ji3UvpZPu3k1v3ENhtFkp23UfEde8wdHIRyTfcSF1jOuUFjbR4z6eyyp3oqFnExMyhqekE+9J/QU3YBmrCNgDg45NCRMT9BAfdgl7vgZROwkLvQzo8GZzwC0rLrsPN6I/eqMPD2IeEIQsoLPqEqn0PUXJiLcEJmdw7aCm/+8zOxsLreWV6Ag+O0sJJuw/yx32QP54pIZS9k0Hx/O/QW4w4Km04Km3Y9leABL2vCUd1E+gg/FfX0Rb3s/O5IKEXQkwGFgB64F0p5fwOytwDvISWW36vlPI+1/mHgF+6ir0qpXz/9LqdhdGkp6nR3upDf8sP28LFBkVbyNvXNgzM409s2L+Fh4Yso64+gIaDkwlNXcKoAZOIjb6dVw9UcKyikT36h/Bxq+XWZE+SvVKYEvsN2OBkU6LXe9Gv7zPo9SbSpgYyYHgo1rpmAiK9uM84ksdzniM8qz/fbTjIyPFxrc8vqC9gecBqfL0DmXZEa8Vte8pxHxKA9bta6kpP4HfXAC1JQ0kDhiAP7FWa0Ov9ldDDxSVQuZq5Et/TaPQlNWUph3Nfp6BA+5MODLgJu6OB4ODJlJd9Q07OKxw+/BqxMXOJiXmsNVxGReVmjh/7K9HRP6Tv2D0c3jgA74AAbn38zBABtXVZ5OUtQq8z09iYS1jYDOIGvcLa9w9gLanE4v0tR47+kcjISoprPwDX5ucRw79oTSBvNPowYtznFH23guzGn2IwWEhJ/vCUtS0hdAwe/Err8em5jENCbiMk5DZIhi8XBlC0ZTyWcQt4aMgyGp3hfLLDhwdH9eH1VdnsL6rl1TuHEtHXB5+psdRtKsReasWcGEhTbg3OhhYCH07A1NeXymXZmIcG4nBKvthdyJ3JEWfdAHY5nFfohTaWexOYCBQAO4QQX0op97crMwB4FhgtpawSQgS7zvsDLwJpaA3ATlfdqk5/E8Dgpm8V+Tt+moDeawtSTkcIQdLEaGprNLcwi2UoY6NG8+cN2tDpu8IBROSOpzp3PNN/lczoP7XlY212mCizBvHBtjwyw/4fN0c2c3PiKCIjHsBmK8bdPeKU3rdPkBmfdlMrP575IB8e3wjLJYMTY7C4RLqgrgBzky9yTxoraOHWWAu13xzDllOFNVPzSTbF+iDcdFQuO4jX9eGtScxVj15xJaDXezBo4ItUVKzHaj1Ov37PtG70sliGULlzM1I6OHL0DzidNvr1ewYpJdnZv8Rmy6eyajO6QIidMJiYsLc6fMaJE1+07gEJC72L+LjXEELH+PuHkDrFRmXjGOrq9lBQ8AEAUZHfd0VhPXWaS+gEEaOm414ZiMUr7hSRv1gSxkbwn79UMtL/JQpq7uXRoQv4a8b9vLFSx9ubyhkZls7cpVY+eXQclnGRWMZFYq+wovMwUre5EGd9M+79/ahubMbv3jh0OsGzn2Xy0XfH8TYbmTi483fOXkiPfgRwWEp5BEAIsQy4A9jfrswjwJsnBVxKWeo6PwlYLaWsdNVdDUwGPqILMLbbldhi+JiDB/6ITu9OcNBkjG56Em9xIyMDBg18CR8fH3a+eD9PrN1KYcgJ/hPUgo9Oh9exitZ79A/24nBpPZt+fiP3/GUbO4vA2/I8j/UfAYDZfP6Ug4ODBnN42GtEbBpI1oYCRt3Rjz3f5JNjOsa4o1pQtH4pQQQmBlD9RS7WzHLMw4Kwl1up3ZDfuompflsRbn280VnczohQqeh+qqurWbp0KY8//vhF1Zs6dSpLly7F19f3/IWvElJTPsFmKzxlN6+vTyo3jNtLWdlqSktXcuz4X/HxSUGnM2Gz5RMaeictLVWYTKEUsYwmj1dobv41bm6nBiyrrc3ExzuZlJSlpySyMbjp8Qv1xKvlhzgcDeTn/x2DwYeBA391TlsD/Mdc9vvGJAbg5WfiwAYdEx75iu923MbsxA+AD3hnolZmVV4x/Z6r58/3JXNbYjiGAK3zdzKfQa2thTG/XcfscX3xNBn46Lvj/L+UiC4RebgwoY8A8tsdFwAjTyszEEAIsQVteuclKeXXZ6l7RgBzIcRsYDZAdPTF7y48icGtbUG2uUUT7KysJxDCSN/YJ9HrtV+2u1nLi6nTCfLr8hngP4DH7x3Gj/6xk4++07aRv/NgKklRvpTXNxPl78FHj4zinzvzmZIQdtF2TRk2gcN7c2AVZBw7gD3bE4glEhgzYwAJN0Qga5uBXAwhHvhOjaU5v46KDzUvHMuNUdSty6c5rxZzgvJrvxKorq5m0aJFZwi93W7HYDj7n9XKlSu72rRux2QKwmQ6M9mNweBFWNidWCxDKK9Yy94Mbf+GXu9F/34/x2QKBsDTsz85Oa+Sta8Bi9dg+vR5FDc3f4qLP6emJp3IyIfOmq3MaPRm4IBfEhZ2N4LumeLS6XUkTYxm8yc5lB+NRlTNQfq9eUqZSTFrSQnO4MONP2LsgAfxMbd5TEkp2XSonPomO/+3+hBGCd5SMKJaUF3aiG8HQfEul87qGhqAAcB4IBLYKIS44MSrUsp3gHdA86O/ZCNcsTdihwUi2wUHk7KF3CP/h7//aAwGC25GTSztTjsFdQVMiJ7AhLhgLO4Gsk/UEeln5pYh2i7FYG9tmiQm0JOfTYrjUpg5aCbLZvyT2g+tkN22I9YQ2cSwCa5Rgb87Yb8a1Rrj3N3ihjHSC4TAcmOUloKttBFTv97TE+ws/verfewv6jh+0aUyONybF28fctbr8+bNIzc3l6SkJIxGI+7u7vj5+ZGdnc2hQ4eYPn06+fn52Gw2nnzySWbPng1ATEwM6enp1NfXM2XKFMaMGcPWrVuJiIhgxYoVmM29z6PKy2sg/fs/6wq/AYPjf9cq8gDRUbOorztA8YlPqaraRnNzOf37P8uB7Gfx8UkjNub8oyaL16X9bV4qCeMiyFibz6p3snDYk4B3mPiT3RQVf0BY2F3o9Z5w7C2S/Fey8JtR/Oq2wTTZHRw5XM1z7++iNsCgeQq2OHmsxh03vY6KXRW0TL683bFn40KEvhBoP0cR6TrXngLgWyllC3BUCHEITfgL0cS/fd31l2rs+XA6JX6hHkx6OIHM/Qtazw8a9AoHD75IZeUmvL2TKWks4ffpv+em6JuwSzt9vPtg0OtIjPRhy+EKYgI6N3CXh9GDHwx/iGMe5az+fDd27wZ0x3247d5Rp5Rrn8hC6ATBjw0DnUAIgd9dA2g+VovniIsfUSg6n/nz55OVlcWePXtYv349t956K1lZWa0ukIsXL8bf3x+r1crw4cO56667CAg4dTSWk5PDRx99xF//+lfuuecePv30Ux544IGeeJ0up0/0w0RH/ZDm5rJTRP4koWF3Ula+Bru9hhMlX9HcXIGULcTH/eaM6ZwrAb1Bx6jp/fjvu/tw9zJiq2/BaH2I8Te05QhoabYynPcoqn+YQc8/Q4zVnWmN7tyMnj0NLYyIDyDyiA2hs+ITswXv6O9w83kD6DhW0eVwIUK/AxgghIhFE+57gftOK/MF8D3g70KIQLSpnCNALvCaEOJk7rVb0BZtuwSH3YlPsAd6ow6brYiAgPHExsyhVgRx2Ab93bVh4heHv+DrvK/5Ou9rYrxjuLnPzQAEeGoLNMOifM71mEumz5BAHh4y8YLLi3axZ0x9vDH1ufgkJtcC5+p5dxcjRow4xc994cKFfP65llw9Pz+fnJycM4Q+NjaWpKQkAFJTU8nLy+s2e3sCIUSHIg/g73cdN4zbRXNzOd/tuIPKqs2EBN+Gp2fX9HA7g/6pwej0gvD+vnz4wnYObC0mJjGQpkY7FYX1bHg7jagbVhPuW8iiCT+j+MgYatIfAiCp2QB76/HwcaP/HRtpNmpu3m5uXTNiP6/QSyntQogngFVo8++LpZT7hBAvA+lSyi9d124RQuwHHMDPpJQVAEKIV9AaC4CXTy7MdgVOuxO9QZuns9trcTMOxccnhYyCTWyoNxDkpiMx8Cbq89oSPj867NHWtIAPXd+Ho+UN/GB0129MUfQuPD3bRoHr169nzZo1bNu2DQ8PD8aPH9/hDl6Tqc3zQ6/XY7Vau8XWKxk3t0BGjvg3jY1H8fbu2Ryx50MIQb9kreFKvCmS9H/nsfiZza1Z3cBE3b7fYRz+MHo3K2F9N1PsnEVInwjiWgw4HU6iU46TffifICE0+F7MrvXDzuaC5uillCuBlaede6Hddwn8xPVzet3FwOLTz3cFDkdLq9A7HFb0em1R40jNETKtBnwZi1cDvL9f8/t9f/L7JAcnt9ZP7ePPVz++/FV5Re/HYrFQV9dx1rCamhr8/Pzw8PAgOzub7du7P/741YzR6IuPT/L5C15BpE2Nwexl5ERuDX5hnhiMeuKuC8XkaeSrvzyHKfIPGD0rmThyKUMT/khtbSYWy2B27/kFJrcQhg9fgZtbx7mMO4Ne46dnteYTNPw5dLZHgASX0GsLW0drtFCmmwo3kVut7azTCz0pIR0nAVEozkdAQACjR48mISEBs9lMSEibW9zkyZN5++23iY+PZ9CgQYwaNeocd1L0BvR6HYk3RpF445ku17c8eDdfLuhPeMoGKireYf2GU/1UBvR/rktFHnqR0Nsw48SO3fI3nM77kLIZnatHX91U3VquqKEIgPcmv9cTZip6EUuXdpwH1GQy8Z///KfDayfn4QMDA8nKymo9/8wzz3RYXnH1Y/Zy457nhgPDqahIY2/Gw6dcDw6e2uU29BqhN+hNVB9PITh+FQ5HPUBrj76+uf6UskumLunRjOwKheLa4mQYi8DAG0lL+4zmpjJ8fVNpbMzD3b3rPel6jdB7Gb1wNlkQQtLUpCUVOTlHX9dSR4RXBI8Oe5SpsVNx03e8+UKhUCi6Gp92i8w+Pn7nKNl59BqhB5BNmvuhzaZNz+h1Wo++rrmOxKBEpvef3mO2KRQKRU/Ra1IPOe0SR7OWmeXYsb8AbT36+uZ6vN2UD7pCobg26TVC77A7cTRpvszVNZrbvl5vRkpJXXMdXkaV3FqhUFyb9B6hdzhbe/Qn0es9sDls2KUdLzcl9AqF4tqk1wi93qCjPv7UTbd6vbnV40ZN3Sh6Ei8vraNRVFTE3Xff3WGZ8ePHk56e3p1mKa4Reo3Qu7kbCL3em8O2tlfS6cxU2jTxV0KvuBIIDw9n+fLlPW2G4hqjV3ndJIekcM8Od0Z7tjDDv4Uiax1H6jQPnFgfFb+m1/KfeXAis3PvGToUppyRMbOVefPmERUVxZw5cwB46aWXMBgMrFu3jqqqKlpaWnj11Ve54447TqmXl5fHbbfdRlZWFlarlVmzZrF3717i4uJUrBtFl9GrhH6g30ASAhIYHH4dT2e+wyT9h1jtVgzCQF+fKzcKnuLqY+bMmTz11FOtQv/JJ5+watUq5s6di7e3N+Xl5YwaNYpp06adNefrW2+9hYeHBwcOHCAjI4OUFBWSQ9E19Cqh1+v0fHSblqVwzfE1fJ2nZaXv79sfo954rqqKq5lz9Ly7iuTkZEpLSykqKqKsrAw/Pz9CQ0N5+umn2bhxIzqdjsLCQkpKSggNDe3wHhs3bmTu3LkAJCYmkpiodmsruoZeJfTtifCKaA1mNjhgcA9bo+iNzJgxg+XLl3PixAlmzpzJkiVLKCsrY+fOnRiNRmJiYjoMT6xQdDe9ZjH2dPxMbVuL+/n260FLFL2VmTNnsmzZMpYvX86MGTOoqakhODgYo9HIunXrOHbs2Dnrjxs3rjUwWlZWFhkZGecsr1BcKr22R3+SxKBEHox/sKfNUPRChgwZQl1dHREREYSFhXH//fdz++23M3ToUNLS0oiLO3ce08cee4xZs2YRHx9PfHw8qamp3WS54lqj1wq9p1HbJXtf3H1qfl7RZWRmtnn7BAYGsm3btg7L1ddr+zliYmJawxObzWaWLVvW9UYqrnl6rdA/kfwEZqOZW/rc0tOmKBQKRY/Sa4Xex+TDT1LPyGyoUCgU1xy9djFWoVAoFBpK6BUKhaKXc0FCL4SYLIQ4KIQ4LISY18H17wshyoQQe1w/D7e75mh3/svONF6hUCgU5+e8c/RCCD3wJjARKAB2CCG+lFLuP63ox1LKJzq4hVVKmXT5pioUCoXiUriQHv0I4LCU8oiUshlYBtxxnjoKRa+murqaRYsWXXS9qVOnUl1d3QUWKRRn50KEPgLIb3dc4Dp3OncJITKEEMuFEFHtzrsLIdKFENuFEB0mbRVCzHaVSS8rK7tw6xWKHuJsQm+3289Zb+XKlfj6+naVWQpFh3SWe+VXwEdSyiYhxI+A94GbXNf6SCkLhRB9gbVCiEwpZW77ylLKd4B3ANLS0mQn2aS4Rvjtd78luzK7U+8Z5x/HL0b84qzX582bR25uLklJSRiNRtzd3fHz8yM7O5tDhw4xffp08vPzsdlsPPnkk8yePRvQNkylp6dTX1/PlClTGDNmDFu3biUiIoIVK1ZgNps79T0UCriwHn0h0L6HHuk614qUskJK2eQ6fBdIbXet0PV5BFgPJF+GvQrFFcH8+fPp168fe/bs4fXXX2fXrl0sWLCAQ4cOAbB48WJ27txJeno6CxcupKKi4ox75OTkMGfOHPbt24evry+ffvppd7+G4hrhQnr0O4ABQohYNIG/F7ivfQEhRJiUsth1OA044DrvBzS6evqBwGjgd51lvEIBnLPn3V2MGDGC2Ni25DYLFy7k888/ByA/P5+cnBwCAgJOqRMbG0tSkuankJqaSl5eXrfZq7i2OK/QSyntQogngFWAHlgspdwnhHgZSJdSfgnMFUJMA+xAJfB9V/V44C9CCCfa6GF+B946CsVVj6enZ+v39evXs2bNGrZt24aHhwfjx4/vMFyxyWRq/a7X61WGKUWXcUFz9FLKlcDK08690O77s8CzHdTbCgy9TBsViisOi8VCXV1dh9dqamrw8/PDw8OD7Oxstm/f3s3WKRSn0mtj3SgUXUlAQACjR48mISEBs9lMSEhI67XJkyfz9ttvEx8fz6BBgxg1alQPWqpQgJDyynJySUtLk+np6T1thuIK58CBA8THx/e0Gd3Gtfa+iotHCLFTSpnW0TUV60ahUCh6OUroFQqFopejhF6hUCh6OUroFQqFopejhF6hUCh6OUroFQqFopejhF6h6Aa8vLwAKCoq4u677+6wzPjx41GuxYquQAm9QtGNhIeHs3z58p42Q3GNoXbGKq56Trz2Gk0HOjdMsSk+jtDnnjvr9Xnz5hEVFcWcOXMAeOmllzAYDKxbt46qqipaWlp49dVXueOOU3P05OXlcdttt5GVlYXVamXWrFns3buXuLg4FetG0WUooVcoLoGZM2fy1FNPtQr9J598wqpVq5g7dy7e3t6Ul5czatQopk2bhhCiw3u89dZbeHh4cODAATIyMkhJSenOV1BcQyihV1z1nKvn3VUkJydTWlpKUVERZWVl+Pn5ERoaytNPP83GjRvR6XQUFhZSUlJCaGhoh/fYuHEjc+fOBSAxMZHExMTufAXFNYQSeoXiEpniD9/sAAAgAElEQVQxYwbLly/nxIkTzJw5kyVLllBWVsbOnTsxGo3ExMR0GJ5Yoehu1GKsQnGJzJw5k2XLlrF8+XJmzJhBTU0NwcHBGI1G1q1bx7Fjx85Zf9y4cSxduhSArKwsMjIyusNsxTWI6tErFJfIkCFDqKurIyIigrCwMO6//35uv/12hg4dSlpaGnFxcees/9hjjzFr1izi4+OJj48nNTX1nOUViktFhSlWXJVca2F7r7X3VVw8KkyxQqFQXMMooVcoFIpejhJ6hUKh6OUooVcoFIpezgUJvRBishDioBDisBBiXgfXvy+EKBNC7HH9PNzu2kNCiBzXz0OdabxCoVAozs953SuFEHrgTWAiUADsEEJ8KaXcf1rRj6WUT5xW1x94EUgDJLDTVbeqU6xXKBQKxXm5kB79COCwlPKIlLIZWAbccZ46J5kErJZSVrrEfTUw+dJMVSiuHKqrq1m0aNFF15s6dSrV1dVdYJFCcXYuROgjgPx2xwWuc6dzlxAiQwixXAgRdTF1hRCzhRDpQoj0srKyCzRdoeg5zib0drv9nPVWrlyJr69vV5mlUHRIZ+2M/Qr4SErZJIT4EfA+cNOFVpZSvgO8A9qGqU6ySXGNsOmTQ5Tn13fqPQOjvBh7z8CzXp83bx65ubkkJSVhNBpxd3fHz8+P7OxsDh06xPTp08nPz8dms/Hkk08ye/ZsAGJiYkhPT6e+vp4pU6YwZswYtm7dSkREBCtWrMBsNnfqeygUcGE9+kIgqt1xpOtcK1LKCillk+vwXSD1QusqFFcj8+fPp1+/fuzZs4fXX3+dXbt2sWDBAg4dOgTA4sWL2blzJ+np6SxcuJCKiooz7pGTk8OcOXPYt28fvr6+fPrpp939GoprhAvp0e8ABgghYtFE+l7gvvYFhBBhUspi1+E04IDr+yrgNSGEn+v4FuDZy7ZaoWjHuXre3cWIESOIjY1tPV64cCGff/45APn5+eTk5BAQEHBKndjYWJKSkgBITU0lLy+v2+xVXFucV+illHYhxBNooq0HFksp9wkhXgbSpZRfAnOFENMAO1AJfN9Vt1II8QpaYwHwspSysgveQ6HoUTw9PVu/r1+/njVr1rBt2zY8PDwYP358h+GKTSZT63e9Xq8yTCm6jAuao5dSrgRWnnbuhXbfn+UsPXUp5WJg8WXYqFBccVgsFurq6jq8VlNTg5+fHx4eHmRnZ7N9+/Zutk6hOBUVplihuAQCAgIYPXo0CQkJmM1mQkJCWq9NnjyZt99+m/j4eAYNGsSoUaN60FKFQoUpVlylXGthe6+191VcPCpMsUKhUFzDKKFXKBSKXo4SeoVCoejlKKFXKBSKXo4SeoVCoejlKKFXKBSKXo4SeoWiG/Dy8gKgqKiIu+++u8My48ePR7kWK7oCJfQKRTcSHh7O8uXLe9oMxTWG2hmruOpZ9947lB470qn3DO7Tlxu/P/us1+fNm0dUVBRz5swB4KWXXsJgMLBu3TqqqqpoaWnh1Vdf5Y47Ts3Rk5eXx2233UZWVhZWq5VZs2axd+9e4uLiVKwbRZehhF6huARmzpzJU0891Sr0n3zyCatWrWLu3Ll4e3tTXl7OqFGjmDZtGkKIDu/x1ltv4eHhwYEDB8jIyCAlJaU7X0FxDaGEXnHVc66ed1eRnJxMaWkpRUVFlJWV4efnR2hoKE8//TQbN25Ep9NRWFhISUkJoaGhHd5j48aNzJ07F4DExEQSExO78xUU1xBK6BWKS2TGjBksX76cEydOMHPmTJYsWUJZWRk7d+7EaDQSExPTYXhihaK7UYuxiq5BSmjp3XPOM2fOZNmyZSxfvpwZM2ZQU1NDcHAwRqORdevWcezYsXPWHzduHEuXLgUgKyuLjIyM7jBbcQ2ihF7R+RTuhMWT4LVw+Of3oWhPT1vUJQwZMoS6ujoiIiIICwvj/vvvJz09naFDh/LBBx8QFxd3zvqPPfYY9fX1xMfH88ILL5CamnrO8grFpaLCFCs6j7oT8M3LsGcJeAZB3K2Q9Rk01ULfG2HMUxB7A5xlcfJiuNbC9l5r76u4eM4VpljN0SsunxYbbF8Em34P9ia4fi6M+xm4e8PElyH979r1D+6A8GQY8zTE3QY6fU9brlBcEyihV1w6UkL2v+C/v4SqPBh0K9zyCgT0ayvj7qP15Ec+Cns/gq0L4ZP/Af9+MPpJGHYvGExnfYRCobh81By94tI4kQXv3w4fPwAGMzz4BXxv6aki3x6jO6TNgifSYcZ7YLLAV3Phj4mwZQHYarvVfIXiWkL16BUXR0MFrHsVdr6n9danvgGps0B/gf+VdHoYcicMng5H1sPmP8DqF2Dj72H4D2HUY+AV3JVvoFBccyihV1wYjhbY8S6s/w001cPwR2D8PPDwv7T7CQH9btR+CnfBlj9qor/tTUi+H67/Mfj37dx3UCiuUS5o6kYIMVkIcVAIcVgIMe8c5e4SQkghRJrrOEYIYRVC7HH9vN1Zhiu6kZw18Nb18PU8iEiFx7bA1N9dusifTkQK3POBNq0z7F7Y/SH8KRX+OQuK93bOMxSKa5jzCr0QQg+8CUwBBgPfE0IM7qCcBXgS+Pa0S7lSyiTXz6OdYLOiuyjPgSUzYMld4LTD9z6GBz6D4C5y8wvsD9MWwlOZWo8+ZzX8ZRz84044ulFb/L1CqK6uZtGiRRddb+rUqVRXV3eBRQrF2bmQHv0I4LCU8oiUshlYBtzRQblXgN8Cas/31Y61Gr5+DhaNguPbYeIr8Pi3MGhyp/jAnxdLqOaW+XQWTHixbeH3rzfB/i/B6ex6G87D2YTebrefs97KlSvx9fXtKrMUig65kDn6CCC/3XEBMLJ9ASFEChAlpfy3EOJnp9WPFULsBmqBX0opN53+ACHEbGA2QHR09EWYr+hUnA7Y9T6sfRUaKyHlQbjpVz23OGr2hbE/gVGPw96lsGUhfPIgBAyAcW+BdILQUf1VLs1FDZ36aLdwT3xvP4sHEVqY4tzcXJKSkjAajbi7u+Pn50d2djaHDh1i+vTp5OfnY7PZePLJJ5k9Wwu8FhMTQ3p6OvX19UyZMoUxY8awdetWIiIiWLFiBWazuVPfQ6GATnCvFELogP8DftrB5WIgWkqZDPwEWCqE8D69kJTyHSllmpQyLSgo6HJNUlwKRzfBX26Afz0NgYPgRxtg2p+uDA8Yozuk/QB+vBPu/jsYzVpDVLIf6kt6ZEpn/vz59OvXjz179vD666+za9cuFixYwKFDhwBYvHgxO3fuJD09nYULF1JRUXHGPXJycpgzZw779u3D19eXTz/9tLtfQ3GNcCE9+kIgqt1xpOvcSSxAArDeFXc7FPhSCDFNSpkONAFIKXcKIXKBgYCKcXClUJUH//0VHPgSfKI0IR1yZ/dM0VwsOj0k/D/Nvsw92kar2iJ8h+vBM1ALu6A39ohpI0aMIDY2tvV44cKFfP755wDk5+eTk5NDQEDAKXViY2NJSkoCIDU1lby8vG6zV3FtcSFCvwMYIISIRRP4e4H7Tl6UUtYAgSePhRDrgWeklOlCiCCgUkrpEEL0BQYAnZsKSHFpNNXD5v+DrX/WBPTGX8L1T2i95SsdIbRefuAAaG7QevX1JVBfCh4B2iikm3fbenp6tn5fv349a9asYdu2bXh4eDB+/PgOwxWbTG026vV6lWFK0WWcV+illHYhxBPAKkAPLJZS7hNCvAykSym/PEf1ccDLQogWwAk8KqWs7AzDFZeI0wkZH8Oal6D+BCTO1BY8fSJ62rJLw81T87dvsUFDKTRWQGM5uPtpgu/m0SWPtVgs1NXVdXitpqYGPz8/PDw8yM7OZvv27V1ig0JxoVzQhikp5Upg5WnnXjhL2fHtvn8KqInHK4X8HfD1L7QwwuEpMPMfEDWip63qHIzu4ButeezUl2lib6vSQi14hYCbV6dORwUEBDB69GgSEhIwm82EhIS0Xps8eTJvv/028fHxDBo0iFGjRnXacxWKS0GFKb4WqC3SevAZH4NXKNz8ktaT1129oY7OG7bXaYeGcmgo074bPTTBd/e5MtcfzoMKU6w4HypM8bVKixW2/kkLLeB0wNifwpifgMmrpy3renQGrXfvGQzWCm3+vuqoNnfvFQJmPxBXb0OnUFwMSuh7I1LC/i/gvy9AzXGIn6aFD/aL6WnLuh+dTvPG8QgEa5Um+NXHobYYvFznVVx8RS9HCX1vo3gv/GceHN8KIQkw/V8QO7anrep5hNBi85j9oKlO89KpLYK6kh53zVQouhol9L2F+jJY+zLs+ocmaLf9AVIe6rHealNuLvWbNmHq1x/3hCEY/Px6xI4zEELLfOXufcW4ZioUXY0S+qsdezN8+zZsfB1aGrVwATf8XAsf0AM4amooe/NNqpYsBYej9bwxIgL3hATcE4ZgTkjAfcgQ9N5nbJLuXk5xzSxpc800+2nz+FfDngKF4gJQQn+1IiUc+hpWPQ+VuTDgFpj0mraJqCfMcTio/udyyhYswFFdje899xDw8A9pKSzCti8La1YWtqx91K1a1VrH2Cca85AE3BMSMA9NwBQ/GL2X5zme0kUY3cG3D1jC2lwzrVVg8na5ZnpelZ46CsVJlNBfjZRmw6pnIXetFuDr/uUwYGKPmdPw3XeUvPYbmrKz8UhLI+T553B3uQK6RUXhOaotBp69qgrb/v3YsvZhy8qkcfduale6tmgIgVvfvpgThuDuagDc4+PQdVegL72btnHMEtLmmlmR43LNDAaDu+apo9OD0F+U+Ht5eVFfX09RURFz585l+fLlZ5QZP348b7zxBmlpHXrIKRSXjBL6q4nGSlg/X8v05OYFk34DIx7psUXElsJCSl5/g7qvv8YQHkbEH/+AZdIkxDkE0ODnh9fo0XiNHt16zl5ejm3fvtZef/3WrdSscG241ukw9e9/yrSPadCgrn2xDl0z884sJ3Sa4Otcn0KvNQKnH59047TVEh7ow/KP/gGOZlcZnRotKLocJfRXAw477Pw7rPs12Gog9ftw4/Oat0gP4GxspOLdd6n422IQgsC5PybgBz9A5+5+SfczBAbidcMNeN1wQ+u5lpJSbPuysGVp0z7169ZR89ln2kWjEfubf6bZ2xud2czqLVs4UV7eGa/WSmhoKFOmTGlzzWyu1zZeOZ0gHcx74WWiwsOY8/CD4HTw0vw/YNDrWLd5O1XVtbTYW3j1549zx6Tx2g2lEypzycsv4raHniRr7T+xWm3M+slL7N2fQ1z/WKy1lVB1DCp82zUSrs+mOti9RNvp6+6tfZq8XT8WbT1BNRiKs6CE/kondx18/SyUHYCYsTB5PoQm9IgpUkpq/72S0jfewH7iBN633krwMz/FGBZ2Rtkmp5Piphb6uLuds4d/NowhwRhDbsJy002tz7YXF2PN1MS/RKfDWVuLo6oKe00NzsZGhE6n9aZ1urbvnYEQmpi2Y+aDP+Spp55izjPPA/DJv9awatUq5j7rg7e3N+Xl5YwaNYpp//MEQkqt5x4wAOoM2gjMJ4q33n8TD28/DqRvJCMzi5Txt2nPcjrA2QzS4Uqy4tTWDFY9fnYbdQaX+FvA5HNag2BpaxBMLo8jr2CwhIN3mHZONRK9GiX0VyoVufDfX8LBldpC4cwPIe62HvuDtGbto+S117Du2oX74MFE/P4NPFJTzygnpeTr8hr+N7eIPGsziV5mHo4K4o5gX0yXIbxCCIzh4RjDw/GedAsVBw5giotDNjczNSoKp9WK02pFWq3IkxmohA6d2R2d2Ywwm7VPt0treE4nOTmZ0tJSioqKKCsrw8/Pj9DQUJ5++mk2btyITqejsLCQkrIKQkNDtUomL01shRZWeeO3u5k7dy74RJI4JpLExEQtXk/QaVNT0glVB+DJvVrP3larfTbVQVNNB+dc32uLTj3naO74ZYyemuBbwsA7vONPrxDQK7m4WlH/clcatlrY9AZsW6QtDk54UXOZNF7atMjlYi8vp/SPf6Tm08/Q+/sT9uor+Nx5J0J/pn9+Vl0jLxwuYmt1PQM93HmubxjLT1Qx98BxXj5cxP9EBPBQeCAhps5ZUxBCIEwmMJnQ+/gAWkMjm5tbRd9ptWKvqgJX4g+h0yHcXaJ/shG4RPGfMWMGy5cv58SJE8ycOZMlS5ZQVlbGzp07MRqNxMTEdBie+OJf1DU6udydzfYm7f+XrUbbO1BX7No01u7z2Dbt09lypg2ewVqD4B3hagTC2kYFraMDS8fPVvQoSuivJPZ+rPXiG0oh6X6Y8IK2KNgDyOZmKv/xIeWLFuFsbsZ/1iwCH3sUveXMP+Sy5hbmHylmaXElfkY9vxkYyYNhARh0gh9HB7Opqp6/FpTxh7wS/nSslGnBvjwcGUSyd+eHED4p/jqTCVy5WaWUyKamU8W/sqI1M5XQ6xHumuif7P0Lo/G84j9z5kweeeQRysvL2bBhA5988gnBwcEYjUbWrVvHsWPHzll/3LhxLF26lJtuuomsrCwyMjI655dwNgwmLeyDV5CWiP1sOJ3anoK6Ii1UxMnP2iLte0Uu5G3SGozTcbOcY3TgahC8glXYiW5GCf2VRNVRrdd23zKIOHNapDuQUlK/fj2l839L87FjeN1wA8HzfoGpXfakk9gcTv5aUMaCYyXYnE5mRwXxVFQQVfv38sV7Cyg5cpgBI65j6IRJfDB0IHnWZhYXlvFRcSWfllSR5u3Bw5FB3Brki1HXdVNSQghNyN3dwbVDVzqdreLvtNqQViv2itPE3yX8rVM/xlNHIkOGDKGuro6IiAjCwsK4//77uf322xk6dChpaWnExcWd067HHnuMWbNmER8fT3x8PKkdTIX1CDpdW4MQNuzs5ZobOx4VnPw8uknLeeA8LWG60GsdmLONCk5+uvXAnopeigpTfCXhaNEW1XpoHr4pN5eS38ynYfNm3GJjCXl2Hl7jxp1RTkrJv8pqeDm3iHxbM5MCvXk2MgDrd5vY/Z+vqCouxMvPn/BBgzmyewf2piaComMYOmES8WNvpMVk5uMTlfytoIyj1mZC3Yx8PyKAB8IDCXS7sL5HV4TtlU4n0taE09qI06aJv9PWBGh/IzqzGb2vL3ofH4She/tIV22YYqdT249w+ujgZINwslFoqj2zrsmng9FBu4bAO8IVlE5FIYVzhylWQq/AUVtL+ZtvUrlkKTqzmcA5j+N///0I45lz6XvrGnkxp5DtNQ3Ee7rzi0Az7ptXk7VuNU2NDYT2H0jK1DsYOPJ69AYjTY2NZG/ZQMY3X1N6NBeD0Y2Bo0YzdMIkwgYNZl1VPe/ml7G+qg6TTvD/Qvx4ODKIIV7n3iTVXcInnU6cNhvOhgacNTU4bTZAoLN4offxQW+xdLhe0dlctUJ/oTTVn3t0UFvsSgTvOLWeztDmAusZoMUr8gjUXI89XMeegW3nzH69dtpICb2iQ84IWzBjBkFPzsVwWhJrgJKmFn5zpJiPT1TibzQw291B9Pp/cTT9WxAwcNQYUqZMI3zg2acrSo4cJnPtKg5sXk+z1Yp/eCRDJ0xi8LibKNCb+FtBGZ+cqMLqdHKdrycPRwYxOdAHfQcjnJ4SPqfNhqO6GkdNDbKlBXQ69N7e6H180Hl5dYpHT0f0eqG/EJwObfPa6aOChlJoqGiLVdRQoXkjdYjQxL5V/E9vHAK1oIDtG4erJMidEnrFGZwrbEF7rA4nf8kvZeHxUlqcTu6UjQxZ8yn1hw/i7mUh8ebJJN1yK5YAbfOWlJIjR46wefNmCgoKGDx4MGlpaURGRraKYIvNxsFtm8hYu4riQ9noDQb6D9fm8r0HDmZZSTWLC8vJtzUT6W7kBxFB3Bfmj6+xbbqkp4VPSomzsRFHdTXO2lqkw4EwGLRevo+PtqDbiaLf0+971WFvBmulFsqisdz16WoMWs+1axwaKzQ31o5w8zpzZODhf1rjEOBqNAJdLrTdP/2qhF7RyulhC0J+/vMOwxZIKVlRWs0ruUUUNrUwsqmG4as+wVx0jIDIaFKmTiN+zHiMJs3t0+l0cvDgQTZt2kRRURFeXl7ExsZy8OBBmpubCQ0NZfjw4QwdOhQ3N7fW55QfzyNz7X/Zv3EttoZ6fEJCGXrjLcTdMIEtDj3vFpSztboes07HjFA/fhgZxCBP9ytK+KTTibO+Xuvp19WBlAg3t9b5fJ3p8nuEV9L79kqcTrBVt2sQytsahMYOGoyGcnA0dXwvvVu7UUJAu4bgtM9Onk5SQq9whS34GxV/+xsIQcAjD2thCzoIGLartoEXc4rYUdtAtLWO0d98SmRhLn2T00iZegfRCcNaGwaHw0FmZiabN2+mvLwcPz8/Ro8ezbBhwzAajdhsNjIzM9mxYwelpaWYTCaGDRtGWloawcHBrc+0NzeT891WMr9ZRf7+TIROR7/UEQydMImGfvH8vUjz1GlySm7ws/ALg43kIYO7bKrkUpEOB46aGhw1NTgbGoDOWcRVQn+FIaWWz+D0kUFHo4WTjUNHC84AuJLieARA5AiY/uYlmXTZQi+EmAwsAPTAu1LK+WcpdxewHBgupUx3nXsW+CHgAOZKKVd1VPckSug7l4sJW1Dc1MyvDxexvLQaS5OV67d9TfKxAySOn0Dy5NvxCw1vLdvS0sLu3bvZsmULNTU1BAcHM3bsWAYPHoy+g8VJKSX5+fns2LGD/fv343A46NOnD2lpacTHx2NoJ4BVxYVkrv0vWevXYK2twRIQRMKNNxM+9iZW2HS8V1TOb812ogYMJNDNgL/R0OE8fk/jbGnB6ZrPv9xFXCX0vQB785nif8pUUjn4RMHk1y7p9pcl9EIIPXAImAgUADuA70kp959WzgL8G3ADnpBSpgshBgMfASOAcGANMFDK05fO21BC33m0D1tgGhxP6PPPdxi2oNHh5E+H81lUWIHD6SR17xYmFmZz3YTJJNx4MyaPNn9mm83Gjh072L59Ow0NDURGRjJ27FgGDhx4wb3rhoYGdu/eTXp6OtXV1Xh6epKcnExqaip+7TJROewt5KZ/S8Y3qziWuQeA2KRU4m+aRKOHBe/Y/jQ6nOgE+BsNBBoNmPTd42pXXV3N0qVLefzxc8SfacfJRdzbZ87k77/5Db6+vhe1iKuEXnE+LlforwNeklJOch0/CyCl/M1p5f4IrAZ+BjzjEvpTygohVrnute1sz1NCf/mcHrYg+OmnOgxb4JSSfxzI5beFFVQaTAzKzeTeiuNMvGkCfVOHo2s3b1hfX8/27dvZsWMHTU1N9OvXj7Fjx9KnT59Lnj5xOp3k5uaSnp7OoUOHkFIyYMAAhg8fTv/+/dG184+uKS3h/7d37tFxFGei/1X39Dx6Hno/bBlsGWT8AANGtoltCDFsMK+QEBPy2l0IJ9xkkwubw92FcENONjnJJlk2yeXcHLIs4Wx2b4BlQ7IBx2AesbEdTLBwjJEtvzGxhW1JtkbS9PS8uuv+0a3R6GXLsmVp5P6d06erq6ql+qSZ76v6quqr5rUv07z2FRKdx7nqfz7gKL5IlE4b4jkLKSHqU6ny+4ioypi6dQ4cOMDNN99Mc3Nzv/xcLtdvdDKQ0U7ieore42ScSNGPxGFYBxwseD4ELC6sIIRYAJwnpfydEOLvBrz75oB364Zo4D3APQDnn3/+CJrkMRT5sAWPPYadSlF+551U/s2XB4UtkLbNqre38N3DXRyIVlATP8ZDyTY+8xcfpmp6/x2w8XicN954gy1btpDL5Zg7dy7Lli1j6tSpnC6KotDQ0EBDQwPxeJwtW7awZcsWnnrqKUpKSmhsbOTyyy8nEolQUl3D0jv+kg+t/CzvbW0inrUx4p0Q7yTe/QSp3HtYQqFLSg5KUARoQuBTBIJTV/jRyBxmzXp42PIHH3yQffv2cdlll6FpGsFgkLKyMnbu3Mnu3bv5+Mc/zsGDB0mlUtx3333cc889ANTX19PU1EQim+WG225j6aJFvLFpE1MrK3n20UfRY7EzOokLkLMl76fS7DJS7DJS7Eum0VWF2oBGjV+jJqBR6/dRE9Co0HwoE9AN5nF6nPb2PiGEAvwIuHO0P0NK+TjwODg9+tNt07mGlJLE66/T9o/f7wtb8MADBGb2V9qZlMna19fxz20Jtk1rIOILcK9xmHs/uoRISf8zZtvb29m4cSPvvvsuAPPnz2fZsmVUVo5NDPzS0lKWL1/Ohz/8YXbu3MnmzZt57bXXWLt2bX6J5vTp01FUlQuuWExLSwtV02dg9vRwNG5jZTIIIQiqKrbqIychbUsyUqIJgSbEGe3hf//736e5uZmtW7eybt06brrpJpqbm6l3Q0U8+eSTlJeXY5omCxcu5JOf/CQVA/Yn7Nmzh6effponfvELPnX77bzQ1MSnr7+eXFsbuba2fpO4I8GSkgNmn0Lf3avYzTRpu+9rNTWgkbJtjmcHe1B9Amr8GtV+zTEEBUag1s2r9muUa+qEmwj3GJ6RKPpW4LyC52luXi9R4GJgnfuPrwWeF0J8bATvepwm6f37nbAFGzbgr6/nvMf/ZVDYgq62o2x6eTVPdBi8OW8xTKniL5UUD1/bSCzQPypma2srGzZsYOfOnfh8PhYuXMiSJUsoGaGyOV1UVWXevHnMmzeP9vZ2mpqa2Lp1K83NzVRVVdHY2MillzrxV1SfRqSsnPkLfkjGTGJ2d5NOGkgp0YIhRKyEblWjy7JBQommUqn5CI+BW2fRokV5JQ/w6KOP8pvf/AaAgwcPsmfPnkGKvr6+nssuuwyAKxobOdjRQaC+vt8kbvbwYbKHj5CLd9L1wiqi1y5HhkIcMNN5Rb5rGIU+LahxkR7imvIYs8IBLgqHmKUHCPscl1zatmnL5DiaznIkneVoJuukM1mOpnPsN9NsiieI5wYbBL8QVAd81OZHBM69Jm8gnLISn2cQJgIjUfSbgQYhRD2Okv408NneQillF5Dv5gkh1tHnozeBp4QQP8KZjG0A3jpzzZ0yN68AABgVSURBVD936Re2IBik+sEHKP/sZxHuGnUpJa0t22l68Xme70qxftF1JOpiXB9S+M6lF3F+qM8tIKXkvffeY+PGjezfv59AIMDVV1/N4sWLCYfHL7BUVVUVN9xwA9deey3Nzc00NTXx4osv8uqrr7JixQoymQx+N8RwQA8T0MNYuRypRA9mdxfZtiPoikIsWoKph+nMWXRlLYKqoFLTKNPUM+amKPw7rVu3jldffZVNmzah6zrXXHPNkOGKAwWuGVVVMU0TAEXTUKqqUCsrSaVSmAmDRE8PX255n/e7XuVgbR2ZgvmWaUGNWXqQD5dHuSgcHKTQhyOgKJwX9HNe0H/CeqZl05Y3ArkCY+AYh91Gig2dPXTnBm84Ciqib3Tg16gN+PLGoHd0UBvQiI7xnMq5zkkVvZQyJ4T4KrAGZ3nlk1LK7UKIbwNNUsrnT/DudiHEs8AOIAd85UQrbjxOjrQs4r96jvaf/GTIsAW5bJZdb6zn7dW/5U8pi3VX3czhiinMD2l8d84MFpb0KSTbttm9ezcbNmygtbWVcDjMddddR2NjI8FRHgs4Fvj9fhYsWMCCBQtobW1l8+bNZLNZOjo60DSNcDhMMBhEURRUn49waRl6SSnZlEmyu5t0dxy1q5PaYJBstJQu6eNQKsPhNFT4fVRoPvynGBgrGo3S09MzZFlXVxdlZWXous7OnTt58803h6zXi5SSnC1JWTZH01lStk3Ktknb0gmmGdDpCYbYuWgJ9e1HWbjxNc5/fz8zjR4uuXw+U2+6geAlM8dMUYZUhemhANNDJ54zMCyLtnSOo5m+EYJzd4xDi2Gy9niWhDXYIIQUpZ8RyM8dBDRq/D7HMPi1kxovj6EZkY9eSrkaWD0g75vD1L1mwPN3ge+Osn0eBSQ3b+bI9/6RdEsLocYrqH3oIYJz5wJgxDvZ+vJqtr36IodtePMjn2Db1Auo9fv4vxdM5baasnzv1bIsmpub2bhxI+3t7ZSWlnLTTTflJxYnMnV1ddTV1bFjxw5isRiGYRCPxxFCoOs64XAYn8+HEAJ/SMcf0rEtC9Pt5Yv2I5QpCsRKMAI6bekcbZkcpT7HraOPsGdZUVHB0qVLufjiiwmFQtTU1OTLVqxYwc9+9jPmzJnDRRddxJVXXgm4cfGB7myOY+ksWSnZZZikbUlbJksyZ3EknUVTBEFFENVUAopCUBX4AhpbrnH+1/atyzHWr6fr+RdIPP0UB37xb/inTyd2yy2U3HIz/unTx+RvfzLCqkq9rlKvn9ggJHLWICPQO0I4ks7yTk+SI+kcpj3YIIRVpc9d5BqBMtcdF1FVIj73riqEC9IRVcU3hqGwJzreztgiINvaytFHHqHnxZfwTZlCzd//HdEVKxBCcHT/Xras/i0739hASlHYdcOn+f3UWSiK4CvnV/M351cTdof5vZuc3njjDeLxONXV1Sxbtox58+YNuclpItO73FBKSSaTwTCMvHvE7/fne/mFSltKSTadwuzuJmX0IG2JCIZIRUvoFiq2dHqvVX4fJb7Ru3WklGRs6fbM++5p26bw69ar0IOK4ih0RRBUlVMK4mZ1d9Pz8st0vbCK5FtvgZQEL51Pyc23ELvxhiED1E1kpGVhGwZWdzddPQZHehIcSZocNjMczVoctW3apaBN+GjX/LQHgqRHeMRhQBF9BkFViPjUAQbCSYfdskhBWTj/Tl+dibY6yQuBUKTYpsmxf31iUNgC/H72bt7Elhefp3XnDtSQzvGbPs2vay6kw7JZWVPGQzOnMNX1vaZSKZqamti0aVN+k9OyZcuYNWtWv7XqxcRQis+yLJLJJIZhYNs2iqLke/kDDZltWaSMhOPLT6dBUcjGykj4g2Qk+BRBheajQlPRhvkbnapCd5T5iRX6qcjb24ZsOkU6aWD8+c90vvYq8Y1/wPyglazmQ7lgJmrDhciaGrLZLOmkQSZlEtTDhMvKCZeWOZebjpSVE4rGnMPVTwEpJTKZxEoksBMJ7J4erISBnehx4gAlEtg9CTfdg50wnDpGX76dSGAnkyf/ZYqCEomgRiKIcJi0hEQqTU8mQ1LzYwZDJINBkgHnbgZDJANBzGgUMxrDjEQx9TBmSCcZCJL0+0n6NAzFR1II5Aj/L7raZxwGjSAKjUiB4RjOiISU018V5in6IkNKSffq1bT9kxu24MYbqf5f92PFYrz7+zX8ac0qejraKamuwXfj7fxH+XS2J9NcEdP5zoV1LHD98IZh8Oabb/LWW2+RTqeZOXMmV111FTNmzCj6ia8TbSCSUpJKpUgmk6TTTuCpYDCIrusEAoFBsvf28s1ED9K2yekRzHCUJApCQKlPpVzzYUl5RhW6lNIJiGbbzqEntlWQtvul9+7fz/vr1pBOGqSTSTJm0lHaSRM5XNTFvl+Ez5b4/X6CJaUEyiscwxDvJJ00BlVXFIVQUCcUDBLS/IQUHwEEQVsSzFr402n8Zhp/IoE0DEdpJxJOYLCToOg6SjSaV9RK7xWNoIYjblkYNRpFCbv5ETc/HEGNhBG6PuTnV0rpjAbicax4F1ZXHNuNO2R1dbl5hVc8n0/WOSPXFoK03583EslgCDMaI11WTqqsDDNWihmLYUYimHrENRYBklqApKZhKCqGUDAkJNzPyUhQheOWaoyFeerSC0b0zkBOd8OUx1nE3L6do9/tC1tQ98g/YdbW8Prq37B9/e/JpdOcN28+F/3Vl/h3vYrfdXRTZ9n8bO50bq0uRQhBPB5n06ZNvP322+RyOebMmcOyZcuoqxu0V21SIoQgFAoRCoXI5XIkk0mSySSpVApVVQmHw4RCoXwvXwsE0aqCRCoqSScSJHu68LUfRvdppKOlxKWfzoI155oiHDeATyEA+JH4pUTYFnbO6qeoE71pOVCBW8gRKAEhBEJRsHI5Ep3HCehhSmtq8Yd0d6WRnk/7dfce0gm4aS0QxNqxg+5Vv6P7pZewu3ehlpaiRKPYiQQZwyCNTcrnI62ppH0qac1HWouT9vmIaypHfSoZnzo49K4OgUiU0HmVhIIhdD1MOBJDLykhUlZBpKqKSFUN0SlTCJRXoITDY3pIixAC1TUgTJs24veklEjTLDAI8QFGwY1X1NWFdeRgP2MhT3T4u6Yhy8rIVFaRrqgkVV5OqrSMVD9jESYZdI2F38+U0NjMkXk9+glE249/wrHHH0ctK6Pyb+8jfsEM/rRmFQfe2YKqacxZdg0NH72ZX8oQ/3qwHVUI7p1ezf84rxpdVWhvb+cPf/hD/pDp+fPns3TpUqqqqsZZsjPPqYYEsG2bVCqFYRhk3d5bKBQiHA6jDXEQeDadxuzpJpXoxrIl2aCOKm3UXBZhWYwoGKCioCgKwr2GTqv988XgOqOR17IsDMMgkUjk74nubuK7dtH954OEbJsyv0a5rlMejRGORZ1etNvDzveiIxHUcBhbUUh2xzE6OzHixzE6O0l0HnfS8U6MzuMY8ThGvBPbyg1qjz+kuy6iMsKl5UTce6HLKFxaTiAcLqrRpp1KYXV19x89xOPDjCDcUUa8a1gXVfDii6n/1X+Nqi1ej75ICFx4IdHPf562Sy5i49pX6HzuEOGycpZ+6vPMu24FvzUs7t9/mI5sD5+qLeOhmVOpDWh88MEHvLBhAy0tLfh8PhobG1myZAmlpaUn/6XnCL3+el3XyWazGIaBaZqYponP58v38nvnLLRAAC1QRbS8gpSRIJ00EMKHCPgLlLA6rHIWyplfF57L5fop7hPde9fkD0TTNELlZRiGgWVZkExCMone001lZSUVUlIZCFCpqlTqOqWxGEJVUYFoeSXR8hPvjJa2jZnoKVD+BQbBNRJH9+1hX/w4ufTgeO6qprkGoIxIaTl6aRmR3jmEvJEoJxSL9YvFNF4o7qHzWk31ySsXIDMZrO7uQW4lRdfHpJ1ej34CsfmFX/PHX/+nc/bqBQ3O2atXLmVTT4pv7mllh5FicUmYf7iwjkujIQ4cOMDGjRvZt28fgUCARYsWsXjxYiKRyHiLMuaciSBftm1jmiaGYZDL5fJLNHVdP2vLTG3XlWPbNpZlDflsWRbvvfceL7300pA/w+/3E4lECIfDJ733btKybZt4PE5HR0f+OnbsGB0dHRhGn99eURQqKiqoqKigsrIyf1VUVBAa4iyDkSKlJGOargFwRwV5o9BnJIzO46SMxKD3haKgl5QSLi0jGIniD4YI6DpaMIQ/FHKX1obwB0P4dd2598vX8QeDpzzpPJHxevRFgqqqTL90AVfc+DGmNMzmPTPD3S0Heamjm2lBjcfnzeDmyhh79uzh589u4NChQxN2k1MxoCgK4XAYXdfJZDL5FTuGYeD3+9F1ndAojgQcSnHX1NRw6NAhWltbefDBB3niiSccX31BR2vlypU8/PDDXHqpc7CLoiioqoqmafj9fj7ykY8MqcBHY5QURaG8vJzy8nJmzZrVr8w0zSENwO7du7ELJlzD4XA/5d9rAEpLS0+6msvZzezMJZRPPbE/PZfJuIbAdRkVjA6MzuOkkgbJeCdpM0nWNEmbSeQIJoaB/oZhxEaiIF2QPxFGGMPh9egnIF3ZHD9+/yg/P9SBXxHcN72Gu6eWs6+lhY0bN9LW1kZpaSlLlizh8ssvn/CbnMaCsQrb27tEM5lMYllW3uXTq/ALlXdhj7uwJz7Ud6qhoYG9e/eiqo5PvleJF6avv/56fvCDH7Bo0aJBinIihCm2LIvOzs5+yr/3KnQVqao65AigsrKyX9iHsUJKSS6bIWuaZFzFnzVNMqmCtJkkbZpkU0m3jknWTJJJmWSS7t2tZ+UGzzkMhc8f6FP+QR2/PtAo6EMbiQJDEgiHCYZHNyL3evRFQs6W/L/Dx/jhe4fpzFp8Zko5959XyeGWHTzx38/Q2dlJVVUVn/jEJ7j44ouLbpPTWPHwnkM0J4b2SY+WeeEQ3zi/sm8iM9HfffC9732PqVOn8oUvfAFFUXjkkUfQNI2NGzfS1dVFNpvlW9/6Frfeeiuq6gT2mjJlSr849qZpctddd/HOO+8we/ZsUqkUPp9vwu5tUFU1r7gHYhjGIANw5MgRWlpa+hm+aDQ6pAGIxWJnTG4hBJo/gOYPoJec/jyVlcvmlX7G7DMAhcYg4xqSvjpOeeL4cTKpQ/n3cplhzpl1qb2ggc9978en3eaBeIp+AnHfzj/z3NFOPlQa5hvnV5Pa2czTq54lkUhQV1fH9ddfX9SbnIoJIZy198FgkFwuRzqd7udOufPOO7n//vt56KGHAFi1ahVr1qzhgQceIBaL0dHRwZVXXsnKlSuHdf089thj6LpOS0sL27ZtY8GCBWdTxDNKOBwmHA4zfUD4hVwulx8FFF7btm3L73EAZ5K4V+kXjgYqKir6HSY/Hqg+jVBUIxSNnfbPsi1rgEHobzQCobGZjPUU/QTi7rpKro0GKdmznVfWPEcqlaK+vp7bbruN+vr6olp2djb5TsPI10yPBp/PN+jUqMbGRtra2vjggw9ob2+nrKyM2tpavva1r7F+/XoURaG1tZWjR49SW1s75M9dv3499957L+AshZ0/f/6YyjEe+Hw+qqqqBi3xlVLmRwGFV2tr66BTu0pKSgYZgMrKSqLRaNF9JxRVJRiOjNo9M1o8RT+BMLZtoWX9erLZLLNnz+aqq646ZzY5FSO33347v/rVrzhy5Ah33HEHv/zlL2lvb+ftt99G0zRmzJgxZHhiD8e9EolEiEQizJgxo19ZNpvl+PHjgyaEt27dSiaTydfz+/1UVFQQiUTw+Xxompa/TvR8srLJOGL2FP0EIhgMMnfuXJYuXUp19amty/U4+9xxxx188YtfpKOjg9dff51nn32W6upqNE1j7dq1vP/++yd8/+qrr+app55i+fLlNDc35ze6netomkZNTU2/iKDgjAJ6enqGXA6azWbzVy6XI5vN9lshdCr0rnQaiWE41efxMiqeop9ALFy4kIULF453MzxGyLx58+jp6aGuro4pU6bwuc99jltuuYVLLrmExsZGZs+efcL3v/zlL3PXXXcxZ84c5syZwxVXXHGWWl6cCCGIxWLEYjFmzpx50vqWZfVT/AMNwWiex9qoTJ06lZUrV47qZ50IT9F7eJwGvWfqAlRWVrJp06Yh6/Wu2pkxY0beBx0KhXjmmWfGvpHnKKqqnrWVaWfKqIzVbnZP0Xt4eHicJmfTqIyGyTfr4OHh4eHRD0/RexQtE21X91hxrsjpMXZ4it6jKAkGgxw7dmzSK0EpJceOHfPiGHmcFp6P3qMomTZtGocOHaK9vX28mzLmBINBpp3CQRoeHgMZkaIXQqwA/g+gAk9IKb8/oPxLwFcAC0gA90gpdwghZgAtwC636ptSyi+dmaZ7nMtomkZ9ff14N8PDoyg4qaIXQqjAT4G/AA4Bm4UQz0spdxRUe0pK+TO3/seAHwEr3LJ9UsrLzmyzPTw8PDxGykh89IuAvVLK/VLKDPAMcGthBSlld8FjGJjcjlMPDw+PImIkir4OOFjwfMjN64cQ4itCiH3AD4F7C4rqhRB/EkK8LoS4aqhfIIS4RwjRJIRoOhd8rh4eHh5nkzM2GSul/CnwUyHEZ4FvAH8NHAbOl1IeE0JcAfy3EGLegBEAUsrHgccBhBDtQogTBwkZTCXQcdpCTDw8uYqLySjXZJQJJqdc04crGImibwXOK3ie5uYNxzPAYwBSyjSQdtNvuz3+WcCwR0hJKauGKxsOIUTTcCerFDOeXMXFZJRrMsoEk1eu4RiJ62Yz0CCEqBdC+IFPA88XVhBCNBQ83gTscfOr3MlchBAzgQZg/5louIeHh4fHyDhpj15KmRNCfBVYg7O88kkp5XYhxLeBJinl88BXhRDXAVmgE8dtA3A18G0hRBawgS9JKY+PhSAeHh4eHkMzIh+9lHI1sHpA3jcL0vcN895zwHOn08AR8vhZ+B3jgSdXcTEZ5ZqMMsHklWtIxGTfQu7h4eFxruPFuvHw8PCY5HiK3sPDw2OSU9SKXgixQgixSwixVwjx4Hi351QRQjwphGgTQjQX5JULIV4RQuxx72VuvhBCPOrKuk0IsWD8Wj48QojzhBBrhRA7hBDbhRD3ufnFLldQCPGWEOIdV65/cPPrhRB/dNv/n+7KNIQQAfd5r1s+YzzbfzKEEKq7sXGV+1z0cgkhDggh3hVCbBVCNLl5Rf05HC1Fq+gLYvDcAMwFPiOEmDu+rTpl/o2+mEC9PAi8JqVsAF5zn8GRs8G97sHdqzAByQH3SynnAlcCX3H/L8UuVxpYLqW8FLgMWCGEuBL4AfBjKeWFOCvO7nbr3w10uvk/dutNZO7DCUDYy2SR6yNSyssK1swX++dwdEgpi/ICPgSsKXj+OvD18W7XKOSYATQXPO8CprjpKcAuN/0vwGeGqjeRL+C3OAHxJo1cgA5sARbj7K70ufn5zyTOcuQPuWmfW0+Md9uHkWcajtJbDqwCxCSR6wBQOSBv0nwOT+Uq2h49I4zBU4TUSCkPu+kjQI2bLjp53WH95cAfmQRyue6NrUAb8AqwD4hLKXNulcK25+Vyy7uAirPb4hHzE+Dvcfa6gNPOySCXBF4WQrwthLjHzSv6z+Fo8A4emcBIKaUQoijXvwohIjh7KP5WStkthMiXFatcUkoLuEwIUQr8Bpg9zk06bYQQNwNt0glRcs14t+cMs0xK2SqEqAZeEULsLCws1s/haCjmHv2pxuApFo4KIaYAuPc2N79o5BVCaDhK/pdSyl+72UUvVy9SyjiwFselUSqE6O0wFbY9L5dbXgIcO8tNHQlLgY8JIQ7gxKlajnPIULHLhZSy1b234RjmRUyiz+GpUMyK/qQxeIqU5+kLIfHXOD7u3vy/clcHXAl0FQxBJwzC6br/HGiRUv6ooKjY5apye/IIIUI48w4tOAp/pVttoFy98q4Efi9d5+9EQkr5dSnlNCnlDJzv0O+llJ+jyOUSQoSFENHeNPBRoJki/xyOmvGeJDidC7gR2I3jK/3f492eUbT/aZxQzlkcn+DdOP7O13ACw70KlLt1Bc4qo33Au0DjeLd/GJmW4fhGtwFb3evGSSDXfOBPrlzNwDfd/JnAW8Be4L+AgJsfdJ/3uuUzx1uGEch4DbBqMsjltv8d99reqx+K/XM42ssLgeDh4eExySlm142Hh4eHxwjwFL2Hh4fHJMdT9B4eHh6THE/Re3h4eExyPEXv4eHhMcnxFL2Hh4fHJMdT9B4eHh6TnP8PQLI7t+aD59cAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qs6kMSDilWOc"
      },
      "source": [
        "# Prediction - no tta"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z2H-EecnXCG1"
      },
      "source": [
        "dls = get_data(0)\n",
        "test_df = pd.read_csv(DATA_DIR/TEST_CSV)\n",
        "test_dl = dls.test_dl(test_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "6JhWZRaRXIWM",
        "outputId": "255f8189-6d9a-45df-9e2d-7f815059b720"
      },
      "source": [
        "preds = []\n",
        "for i in range(N_FOLDS):\n",
        "  learn = get_learner(i, model=model)\n",
        "  learn.load(f'{DATA_DIR}/model/model_efficientnet_fold_{i}')\n",
        "\n",
        "  test_preds, _ = learn.get_preds(dl=test_dl)\n",
        "  preds.append(test_preds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eLvBwoF4XmES"
      },
      "source": [
        "test_preds= np.mean(np.stack(preds), axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XAW0bRK5YM1h"
      },
      "source": [
        "sub_df = pd.concat([test_df, pd.DataFrame(test_preds, columns=LABELS)], axis=1)\n",
        "sub_df = sub_df[['image_id', 'healthy', 'multiple_diseases', 'rust', 'scab']]\n",
        "sub_df.to_csv(f'{DATA_DIR}/submission_efficient_notta.csv', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 117
        },
        "id": "MfOKupdiYTMB",
        "outputId": "e0a3c67e-f3b3-4fb7-ed3f-67afe8cc5adf"
      },
      "source": [
        "#preds with tta\n",
        "preds = []\n",
        "for i in range(N_FOLDS):\n",
        "  learn = get_learner(i, model=model)\n",
        "  learn.load(f'{DATA_DIR}/model/model_efficientnet_fold_{i}')\n",
        "\n",
        "  test_preds, _ = learn.tta(dl=test_dl)\n",
        "  preds.append(test_preds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      \n",
              "    </div>\n",
              "    \n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      \n",
              "    </div>\n",
              "    \n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      \n",
              "    </div>\n",
              "    \n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      \n",
              "    </div>\n",
              "    \n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      \n",
              "    </div>\n",
              "    \n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tpNDTCZfbCCn"
      },
      "source": [
        "test_preds= np.mean(np.stack(preds), axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eO34M92dbIPH"
      },
      "source": [
        "sub_df = pd.concat([test_df, pd.DataFrame(test_preds, columns=LABELS)], axis=1)\n",
        "sub_df = sub_df[['image_id', 'healthy', 'multiple_diseases', 'rust', 'scab']]\n",
        "sub_df.to_csv(f'{DATA_DIR}/submission_efficient_tta.csv', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ir53Ay6OgZZC"
      },
      "source": [
        "# Training with efficientnet v2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6RYPfJndgYjf"
      },
      "source": [
        "IMG_SIZE = 700\n",
        "BS = 16\n",
        "#increase oversampling of multiple_disease to 4x\n",
        "#no weight decay"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qCTRK-O2bLfY"
      },
      "source": [
        "def get_learner(fold_num, model):\n",
        "\n",
        "    data = get_data(fold_num)\n",
        "    opt_func = partial(Adam, lr=1e-3, wd=0.0005, eps=1e-8)\n",
        "    learn = Learner(\n",
        "        data, model, loss_func=LabelSmoothingCrossEntropy(),\n",
        "        opt_func = opt_func,\n",
        "        metrics=[accuracy, AccumMetric(comp_metric, flatten=False)],\n",
        "        cbs=MixUp()).to_fp16()\n",
        "\n",
        "    return learn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "O5bxvSc6kjLH",
        "outputId": "df3c9da5-9834-4a3c-cdc3-827d99f60226"
      },
      "source": [
        "for i in range(N_FOLDS):\n",
        "    print(f'Fold {i} results')\n",
        "    savemodel = SaveModelCallback(monitor='valid_loss', every_epoch=False, fname=f'{DATA_DIR}/model/model_efficientnet2_best_fold_{i}')\n",
        "    learn = get_learner(fold_num=i, model=model)\n",
        "\n",
        "    learn.fit_one_cycle(3)\n",
        "    learn.unfreeze()\n",
        "    learn.fit_one_cycle(6, slice(1e-6, 1e-3), cbs=savemodel)\n",
        "    \n",
        "    learn.recorder.plot_loss()\n",
        "    \n",
        "    learn.save(f'{DATA_DIR}/model/model_efficientnet2_fold_{i}')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fold 0 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.625193</td>\n",
              "      <td>0.415521</td>\n",
              "      <td>0.973822</td>\n",
              "      <td>0.996661</td>\n",
              "      <td>02:46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.623968</td>\n",
              "      <td>0.376997</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999420</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.612945</td>\n",
              "      <td>0.361072</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.610796</td>\n",
              "      <td>0.376784</td>\n",
              "      <td>0.986911</td>\n",
              "      <td>0.999165</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.636635</td>\n",
              "      <td>0.424619</td>\n",
              "      <td>0.971204</td>\n",
              "      <td>0.994717</td>\n",
              "      <td>02:40</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.629351</td>\n",
              "      <td>0.401584</td>\n",
              "      <td>0.979058</td>\n",
              "      <td>0.998731</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.612558</td>\n",
              "      <td>0.386442</td>\n",
              "      <td>0.986911</td>\n",
              "      <td>0.997863</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.600456</td>\n",
              "      <td>0.384830</td>\n",
              "      <td>0.986911</td>\n",
              "      <td>0.998093</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.597878</td>\n",
              "      <td>0.383988</td>\n",
              "      <td>0.986911</td>\n",
              "      <td>0.998020</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 0.3767843246459961.\n",
            "Fold 1 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.605751</td>\n",
              "      <td>0.374851</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.618145</td>\n",
              "      <td>0.362542</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.612321</td>\n",
              "      <td>0.358521</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.619260</td>\n",
              "      <td>0.377972</td>\n",
              "      <td>0.989529</td>\n",
              "      <td>0.999560</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.637759</td>\n",
              "      <td>0.458525</td>\n",
              "      <td>0.950262</td>\n",
              "      <td>0.997835</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.646755</td>\n",
              "      <td>0.386873</td>\n",
              "      <td>0.984293</td>\n",
              "      <td>0.999291</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.619988</td>\n",
              "      <td>0.389662</td>\n",
              "      <td>0.979058</td>\n",
              "      <td>0.997654</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.606409</td>\n",
              "      <td>0.380043</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>0.997448</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.592532</td>\n",
              "      <td>0.374567</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>0.998738</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 0.37797197699546814.\n",
            "Better model found at epoch 5 with valid_loss value: 0.3745674788951874.\n",
            "Fold 2 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.631075</td>\n",
              "      <td>0.410633</td>\n",
              "      <td>0.973822</td>\n",
              "      <td>0.999493</td>\n",
              "      <td>02:37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.627063</td>\n",
              "      <td>0.364313</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>0.999635</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.614307</td>\n",
              "      <td>0.361746</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>0.999681</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.602997</td>\n",
              "      <td>0.371205</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999676</td>\n",
              "      <td>02:37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.624997</td>\n",
              "      <td>0.369295</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>0.998726</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.608586</td>\n",
              "      <td>0.369879</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>0.999552</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.615457</td>\n",
              "      <td>0.367999</td>\n",
              "      <td>0.994764</td>\n",
              "      <td>0.999903</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.600474</td>\n",
              "      <td>0.366119</td>\n",
              "      <td>0.992147</td>\n",
              "      <td>0.999937</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.599410</td>\n",
              "      <td>0.361222</td>\n",
              "      <td>0.997382</td>\n",
              "      <td>0.999905</td>\n",
              "      <td>02:37</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 0.3712051808834076.\n",
            "Better model found at epoch 1 with valid_loss value: 0.369295209646225.\n",
            "Better model found at epoch 3 with valid_loss value: 0.3679991066455841.\n",
            "Better model found at epoch 4 with valid_loss value: 0.3661193549633026.\n",
            "Better model found at epoch 5 with valid_loss value: 0.36122193932533264.\n",
            "Fold 3 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.619965</td>\n",
              "      <td>0.398012</td>\n",
              "      <td>0.986945</td>\n",
              "      <td>0.999659</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.616255</td>\n",
              "      <td>0.359967</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.608235</td>\n",
              "      <td>0.370480</td>\n",
              "      <td>0.989556</td>\n",
              "      <td>0.999809</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.591724</td>\n",
              "      <td>0.358228</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.631921</td>\n",
              "      <td>0.436083</td>\n",
              "      <td>0.966057</td>\n",
              "      <td>0.997247</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.628987</td>\n",
              "      <td>0.357444</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.602148</td>\n",
              "      <td>0.367809</td>\n",
              "      <td>0.994778</td>\n",
              "      <td>0.999939</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.586641</td>\n",
              "      <td>0.357375</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.586488</td>\n",
              "      <td>0.358056</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 0.35822752118110657.\n",
            "Better model found at epoch 2 with valid_loss value: 0.3574444353580475.\n",
            "Better model found at epoch 4 with valid_loss value: 0.3573746979236603.\n",
            "Fold 4 results\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.601366</td>\n",
              "      <td>0.379640</td>\n",
              "      <td>0.992167</td>\n",
              "      <td>0.999608</td>\n",
              "      <td>02:37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.614294</td>\n",
              "      <td>0.361146</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.999640</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.611544</td>\n",
              "      <td>0.359048</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.999560</td>\n",
              "      <td>02:38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='5' class='' max='6' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      83.33% [5/6 13:16<02:39]\n",
              "    </div>\n",
              "    \n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.602020</td>\n",
              "      <td>0.364529</td>\n",
              "      <td>0.992167</td>\n",
              "      <td>0.999840</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.611387</td>\n",
              "      <td>0.390654</td>\n",
              "      <td>0.994778</td>\n",
              "      <td>0.999480</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.622628</td>\n",
              "      <td>0.373143</td>\n",
              "      <td>0.994778</td>\n",
              "      <td>0.999272</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.619307</td>\n",
              "      <td>0.365483</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.999440</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.586180</td>\n",
              "      <td>0.372691</td>\n",
              "      <td>0.997389</td>\n",
              "      <td>0.999280</td>\n",
              "      <td>02:39</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>\n",
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "                background: #F44336;\n",
              "            }\n",
              "        </style>\n",
              "      <progress value='13' class='' max='95' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      13.68% [13/95 00:20<02:12 0.5874]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 0.3645288348197937.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yijPavROkx2X"
      },
      "source": [
        "# Prediciton with efficientnet 2 best model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zb61ss27wA1c"
      },
      "source": [
        "dls = get_data(0)\n",
        "test_df = pd.read_csv(DATA_DIR/TEST_CSV)\n",
        "test_dl = dls.test_dl(test_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "id": "fI0ibwnlwFEL",
        "outputId": "39308a6e-b74a-4f76-8e5a-c858258e6e3c"
      },
      "source": [
        "preds = []\n",
        "for i in range(N_FOLDS):\n",
        "  learn = get_learner(i, model=model)\n",
        "  learn.load(f'{DATA_DIR}/model/model_efficientnet2_best_fold_{i}')\n",
        "\n",
        "  test_preds, _ = learn.get_preds(dl=test_dl)\n",
        "  preds.append(test_preds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/fastai/learner.py:54: UserWarning: Saved filed doesn't contain an optimizer state.\n",
            "  elif with_opt: warn(\"Saved filed doesn't contain an optimizer state.\")\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/fastai/learner.py:54: UserWarning: Saved filed doesn't contain an optimizer state.\n",
            "  elif with_opt: warn(\"Saved filed doesn't contain an optimizer state.\")\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/fastai/learner.py:54: UserWarning: Saved filed doesn't contain an optimizer state.\n",
            "  elif with_opt: warn(\"Saved filed doesn't contain an optimizer state.\")\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/fastai/learner.py:54: UserWarning: Saved filed doesn't contain an optimizer state.\n",
            "  elif with_opt: warn(\"Saved filed doesn't contain an optimizer state.\")\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/fastai/learner.py:54: UserWarning: Saved filed doesn't contain an optimizer state.\n",
            "  elif with_opt: warn(\"Saved filed doesn't contain an optimizer state.\")\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OVHP0SXqwTI9"
      },
      "source": [
        "test_preds= np.mean(np.stack(preds), axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DWVqz1esxB0L"
      },
      "source": [
        "sub_df = pd.concat([test_df, pd.DataFrame(test_preds, columns=LABELS)], axis=1)\n",
        "sub_df = sub_df[['image_id', 'healthy', 'multiple_diseases', 'rust', 'scab']]\n",
        "sub_df.to_csv(f'{DATA_DIR}/submission_efficient2_notta.csv', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HdIAwRnbWZ3k"
      },
      "source": [
        "# Understanding which class we are performing well in"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XNv31zqkshPX"
      },
      "source": [
        "IMG_SIZE = 700\n",
        "BS = 8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ykXvXEA4s_Z"
      },
      "source": [
        "def get_data(fold):\n",
        "    \n",
        "    train_df_no_val = train_df[train_df['fold'] != {fold}]     #training set\n",
        "    train_df_just_val = train_df[train_df['fold'] == {fold}]   #validation set\n",
        "\n",
        "    train_df_bal = pd.concat(\n",
        "                   [train_df_no_val[train_df_no_val['label'] != \"multiple_diseases\"], train_df_just_val] +\n",
        "                   [train_df_no_val[train_df_no_val['label'] == 'multiple_diseases']] * 6\n",
        "                   ).sample(frac=1.0, random_state=SEED).reset_index(drop=True)\n",
        "\n",
        "    datablock = DataBlock(\n",
        "                blocks=(ImageBlock, CategoryBlock(vocab=LABELS)),\n",
        "                getters=[\n",
        "                ColReader('image_id', pref=IMG_DIR, suff='.jpg'),\n",
        "                ColReader('label')\n",
        "                ],\n",
        "                splitter=IndexSplitter(train_df_bal.loc[train_df_bal.fold==fold].index),\n",
        "                item_tfms=Resize(IMG_SIZE),\n",
        "                batch_tfms=[*aug_transforms(size=IMG_SIZE, max_rotate=30., min_scale=0.75, flip_vert=True, do_flip=True),\n",
        "                            Brightness(max_lighting=0.6, p=0.5),\n",
        "                            Contrast(max_lighting=0.6, p=0.5),\n",
        "                            Saturation(max_lighting=0.6, p=0.5),\n",
        "                            RandomErasing(),\n",
        "                            Normalize.from_stats(*imagenet_stats)]\n",
        "                )\n",
        "    return datablock.dataloaders(source=train_df_bal, bs=BS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YCkYWz3djJqE"
      },
      "source": [
        "class Model(nn.Module):\n",
        "    def __init__(self, model_arch, n_class, pretrained=False):\n",
        "        super().__init__()\n",
        "        self.model = timm.create_model(model_arch, pretrained=pretrained)\n",
        "        n_features = self.model.classifier.in_features\n",
        "        self.model.classifier = nn.Linear(n_features, n_class)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.model(x)\n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5G3gJcOmio-M"
      },
      "source": [
        "def get_learner(fold_num, model):\n",
        "\n",
        "    data = get_data(fold_num)\n",
        "    opt_func = partial(Adam, lr=1e-3, wd=0.0005, eps=1e-8)\n",
        "    learn = Learner(\n",
        "        data, model, loss_func=LabelSmoothingCrossEntropy(),\n",
        "        opt_func = opt_func,\n",
        "        metrics=[accuracy, \n",
        "                 AccumMetric(comp_metric, flatten=False),\n",
        "                 AccumMetric(healthy_roc_auc, flatten=False),\n",
        "                 AccumMetric(multiple_diseases_roc_auc, flatten=False),\n",
        "                 AccumMetric(rust_roc_auc, flatten=False),\n",
        "                 AccumMetric(scab_roc_auc, flatten=False)],\n",
        "        cbs=MixUp()).to_fp16()\n",
        "\n",
        "    return learn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TixCsD7vjRtr"
      },
      "source": [
        "model = Model('tf_efficientnet_b3_ns', len(LABELS), pretrained=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pm84Bcgbi7Vd"
      },
      "source": [
        "learn = get_learner(0, model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        },
        "id": "__LeIkFZjzOY",
        "outputId": "aa82d492-8d32-4c41-8477-764dcc8f3c6f"
      },
      "source": [
        "learn.fit_one_cycle(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>healthy_roc_auc</th>\n",
              "      <th>multiple_diseases_roc_auc</th>\n",
              "      <th>rust_roc_auc</th>\n",
              "      <th>scab_roc_auc</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.050382</td>\n",
              "      <td>0.726290</td>\n",
              "      <td>0.777533</td>\n",
              "      <td>0.928690</td>\n",
              "      <td>0.987608</td>\n",
              "      <td>0.778741</td>\n",
              "      <td>0.992009</td>\n",
              "      <td>0.956403</td>\n",
              "      <td>03:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.930570</td>\n",
              "      <td>0.590497</td>\n",
              "      <td>0.881057</td>\n",
              "      <td>0.973847</td>\n",
              "      <td>0.983763</td>\n",
              "      <td>0.936737</td>\n",
              "      <td>0.996285</td>\n",
              "      <td>0.978603</td>\n",
              "      <td>03:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.817150</td>\n",
              "      <td>0.557378</td>\n",
              "      <td>0.933921</td>\n",
              "      <td>0.971314</td>\n",
              "      <td>0.987193</td>\n",
              "      <td>0.915543</td>\n",
              "      <td>0.999853</td>\n",
              "      <td>0.982666</td>\n",
              "      <td>03:07</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "id": "TznPLH6ykbmY",
        "outputId": "aef820cb-bda8-48cb-e70f-1bc424f25823"
      },
      "source": [
        "learn.fit_one_cycle(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>comp_metric</th>\n",
              "      <th>healthy_roc_auc</th>\n",
              "      <th>multiple_diseases_roc_auc</th>\n",
              "      <th>rust_roc_auc</th>\n",
              "      <th>scab_roc_auc</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.964456</td>\n",
              "      <td>0.588492</td>\n",
              "      <td>0.878855</td>\n",
              "      <td>0.970957</td>\n",
              "      <td>0.980693</td>\n",
              "      <td>0.919075</td>\n",
              "      <td>0.999413</td>\n",
              "      <td>0.984648</td>\n",
              "      <td>03:09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.858294</td>\n",
              "      <td>0.602769</td>\n",
              "      <td>0.863436</td>\n",
              "      <td>0.975594</td>\n",
              "      <td>0.991287</td>\n",
              "      <td>0.927103</td>\n",
              "      <td>0.995650</td>\n",
              "      <td>0.988336</td>\n",
              "      <td>03:10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.796666</td>\n",
              "      <td>0.544046</td>\n",
              "      <td>0.920705</td>\n",
              "      <td>0.983513</td>\n",
              "      <td>0.983902</td>\n",
              "      <td>0.957129</td>\n",
              "      <td>0.999218</td>\n",
              "      <td>0.993804</td>\n",
              "      <td>03:06</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}