BPE_universal_tokenizer_pl.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.7"},"colab":{"name":"BPE_universal_tokenizer_pl.ipynb","provenance":[],"collapsed_sections":["WH5W5xUw9N5x","R1MVyt9C9N6D","NBx8vCFkBtDu"],"toc_visible":true},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"WH5W5xUw9N5x"},"source":["# Byte-level BPE, an universal tokenizer but...\n","\n","> Study about the \"universality\" of a Byte-level Byte-Pair-Encoding tokenizer (BBPE)"]},{"cell_type":"markdown","metadata":{"id":"R1MVyt9C9N6D"},"source":["## About the Byte-level BPE (BBPE) tokenizer"]},{"cell_type":"markdown","metadata":{"id":"TNGUZW6d9N6D"},"source":["From the [tutorial \"Tokenizer summary\"](https://huggingface.co/transformers/master/tokenizer_summary.html), read the paragraphs [Byte-Pair Encoding](https://huggingface.co/transformers/master/tokenizer_summary.html#byte-pair-encoding) and [Byte-level BPE](https://huggingface.co/transformers/master/tokenizer_summary.html#byte-level-bpe) to get the best overview of a Byte-level BPE (Byte-level Byte-Pair-Encoding) and read the Abstract and Conclusion paragraphs of the original paper: [Neural Machine Translation with Byte-Level Subwords](https://arxiv.org/pdf/1909.03341.pdf]) (Facebook AI, 12/05/2019).\n","\n","> **[Abstract]** Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. **In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is.** We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. **Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE.** In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.\n","\n","> **[Conclusion]** We proposed BBPE which builds a byte-level subword vocabulary for machine translation. It results in a much more compact vocabulary than character-based ones do without the loss of performance. In multilingual settings, the former often outperforms the latter. **BBPE does not have any out-of-vocabulary tokens, allowing us to transfer a model using BBPE between languages with non-overlapping vocabularies.** This transfer learning paradigm is actually very generic and can be applied to any languages and datasets for performance gain or training acceleration. With the same vocabulary size, BBPE segments sentences into shorter sequences than character-based methods do, leading to faster training and inference. Our future work includes: eliminating source-target sentence length imbalance; evaluating BBPE in one-to-many and many-to-many translation settings; exploring better segmentation algorithms for byte-level subwords."]},{"cell_type":"markdown","metadata":{"id":"WN6vRfNz9N6E"},"source":["## About the tokenizers and NLP libraries used in this study"]},{"cell_type":"markdown","metadata":{"id":"wlzkhCzi9N6E"},"source":["The tokenizers and NLP libraries used to perform this study were:\n","- English pre-trained GPT2 tokenizer ([GPT2TokenizerFast](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast)) from the [Transformers](https://github.com/huggingface/transformers) library (Hugging Face, version 3.0.0): it is a Fast GPT-2 BBPE tokenizer (backed by Hugging Face's tokenizers library)\n","\n","- Deep Learning library [fastai ](https://github.com/fastai/fastai2) (need to be careful as some librarier are changing rapidly so need some ajustment and use old version like in th elink](https://github.com/piegu/fastai-projects/blob/master/nlputils_fastai2.py)\n","\n","**Note about the Deep Learning libraries**: the [Tokenizers](https://github.com/huggingface/tokenizers) and [Transformers](https://huggingface.co/transformers/) libraries from [Hugging Face](https://huggingface.co/) are today the most up-to-date NLP libraries (Natural Language Processing) used all over the world when fastai is a great tool for training Deep Learning models, especially with powerful fastai tools like Learning rate finder, Mixed precision training, Distributed training, Gradual unfreezing, Differential learning rates and 1cycle policy.\n","\n","Pre-trained BBPE tokenizer for this study was choosen by purpose. The key point is to use BBPE tokenizers trained on huge corpus because they can then tokenize any word of any language without using the unknown token."]},{"cell_type":"markdown","metadata":{"id":"CmKm8BETBpB2"},"source":["# 1. Installing required libraries and mounting google drive"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nqp4kpUG9tsV","executionInfo":{"status":"ok","timestamp":1615916372516,"user_tz":-60,"elapsed":1845,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"80f56a6d-b65e-47d4-ae3b-db9dbaefe638"},"source":["#start by mounting google drive\r\n","from google.colab import drive, files\r\n","drive.mount('/content/gdrive', force_remount=True)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Mounted at /content/gdrive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"M2y5n_KU99lq","executionInfo":{"status":"ok","timestamp":1615922728837,"user_tz":-60,"elapsed":36688,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"a171113c-9d5a-4d9c-dff3-b9a58192d959"},"source":["# need to instal fastai 2 etc before \r\n","!pip install -q git+https://github.com/fastai/fastai\r\n","!pip install -q git+https://github.com/fastai/fastcore\r\n","!pip install -q iterative-stratification"],"execution_count":null,"outputs":[{"output_type":"stream","text":["  Building wheel for fastai (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Building wheel for fastcore (setup.py) ... \u001b[?25l\u001b[?25hdone\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"P-2JYQnD9N6F"},"source":["## Initialization"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"A7Z0xlgv-TvJ","executionInfo":{"status":"ok","timestamp":1615922729440,"user_tz":-60,"elapsed":596,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"51ea7053-99d7-4acb-d145-c50b953b9a05"},"source":["cd /content/gdrive/MyDrive/fastai"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/content/gdrive/MyDrive/fastai\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"nbpresent":{"id":"151cd18f-76e3-440f-a8c7-ffa5c6b5da01"},"id":"RsJYkiK99N6G"},"source":["# from fastai2.text.all import *\n","# from nlputils_fastai2 import * \n","\n","from fastai.text.all import *\n","from nlputilsfastai import * \n","\n","%reload_ext autoreload\n","%autoreload 2\n","%matplotlib inline"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"lHjl3W7HBdej","executionInfo":{"status":"ok","timestamp":1615916410887,"user_tz":-60,"elapsed":40180,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"45010807-3a4c-420e-ef7f-2c281851fdcf"},"source":["gpu = 0\r\n","torch.cuda.set_device(gpu)\r\n","print(f'cuda device: {torch.cuda.current_device()}')\r\n","print(f'cuda device name: {torch.cuda.get_device_name(gpu)}')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["cuda device: 0\n","cuda device name: Tesla T4\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"gK0-GZGhC4nF","executionInfo":{"status":"ok","timestamp":1615916410889,"user_tz":-60,"elapsed":40177,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"3024111b-2eb1-4fe0-e193-29d37ddcecd7"},"source":["!nvidia-smi"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Tue Mar 16 17:40:12 2021       \n","+-----------------------------------------------------------------------------+\n","| NVIDIA-SMI 460.56       Driver Version: 460.32.03    CUDA Version: 11.2     |\n","|-------------------------------+----------------------+----------------------+\n","| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n","| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n","|                               |                      |               MIG M. |\n","|===============================+======================+======================|\n","|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n","| N/A   42C    P8     9W /  70W |      3MiB / 15109MiB |      0%      Default |\n","|                               |                      |                  N/A |\n","+-------------------------------+----------------------+----------------------+\n","                                                                               \n","+-----------------------------------------------------------------------------+\n","| Processes:                                                                  |\n","|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n","|        ID   ID                                                   Usage      |\n","|=============================================================================|\n","|  No running processes found                                                 |\n","+-----------------------------------------------------------------------------+\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"pZIeiQm9Cya8"},"source":["Load standard snipet to prevent random disconnects\r\n","This cell runs JS code to automatic reconnect to runtime."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":35},"id":"T-N9y4S6C1I5","executionInfo":{"status":"ok","timestamp":1615916421133,"user_tz":-60,"elapsed":584,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"3bc33cc1-00c4-4fdc-b4ea-0f7636d09e00"},"source":["import IPython\r\n","from google.colab import output\r\n","\r\n","display(IPython.display.Javascript('''\r\n"," function ClickConnect(){\r\n","   btn = document.querySelector(\"colab-connect-button\")\r\n","   if (btn != null){\r\n","     console.log(\"Click colab-connect-button\"); \r\n","     btn.click() \r\n","     }\r\n","   \r\n","   btn = document.getElementById('ok')\r\n","   if (btn != null){\r\n","     console.log(\"Click reconnect\"); \r\n","     btn.click() \r\n","     }\r\n","  }\r\n","  \r\n","setInterval(ClickConnect,60000)\r\n","'''))\r\n","\r\n","print(\"Done.\")"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"application/javascript":["\n"," function ClickConnect(){\n","   btn = document.querySelector(\"colab-connect-button\")\n","   if (btn != null){\n","     console.log(\"Click colab-connect-button\"); \n","     btn.click() \n","     }\n","   \n","   btn = document.getElementById('ok')\n","   if (btn != null){\n","     console.log(\"Click reconnect\"); \n","     btn.click() \n","     }\n","  }\n","  \n","setInterval(ClickConnect,60000)\n"],"text/plain":["<IPython.core.display.Javascript object>"]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Done.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"o2qnh-a79N6H","executionInfo":{"status":"ok","timestamp":1615922736893,"user_tz":-60,"elapsed":8035,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"3c1fb83f-18bc-488a-84c4-53e0998ec650"},"source":["# Get config of fastai2 paths\n","config = Config()\n","config.d"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'archive_path': '/root/.fastai/archive',\n"," 'data_path': '/root/.fastai/data',\n"," 'model_path': '/root/.fastai/models',\n"," 'storage_path': '/tmp',\n"," 'version': 2}"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"markdown","metadata":{"nbpresent":{"id":"cf070ab7-babb-4cf0-a315-401f65461dc8"},"id":"5pL4tfG49N6I"},"source":["This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents (for other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias))."]},{"cell_type":"code","metadata":{"id":"2ShRUXWj_NoG"},"source":["# setup new path_data and create the corresponding folder\r\n","lang = 'pl'\r\n","name = f'{lang}wiki'\r\n","data_path = config['data_path']\r\n","path_data = data_path/name\r\n","path_data.mkdir(exist_ok=True, parents=True)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NBx8vCFkBtDu"},"source":["# 2. Loading previously prepared scraped wiki file ~1G for particular language\r\n","for that purpose another notebook was used [wiki download](https://github.com/len-sla/other/blob/main/wiki_download.ipynb)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"N-2ccH-IAuUC","executionInfo":{"status":"ok","timestamp":1615916435895,"user_tz":-60,"elapsed":1052,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"a9c296db-fd2d-485e-fe72-3530df566778"},"source":["!du -hs {'/content/gdrive/MyDrive/fastai/all_texts_plwiki.csv'}"],"execution_count":null,"outputs":[{"output_type":"stream","text":["1.1G\t/content/gdrive/MyDrive/fastai/all_texts_plwiki.csv\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":380},"id":"fdlfLPzP_3Z0","executionInfo":{"status":"ok","timestamp":1615916448401,"user_tz":-60,"elapsed":13545,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"9d59f9e9-f03f-4fcd-da71-6b2feee30427"},"source":["df = pd.read_csv('/content/gdrive/MyDrive/fastai/all_texts_plwiki.csv')\r\n","df.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Vanquish – gra konsolowa z gatunku third-person shooter, wyprodukowana przez PlatinumGames i wydana przez SEGA. Jej światowa premiera miała miejsce 19 października 2010 roku na konsolach PlayStation 3 i Xbox 360. 10 maja 2017 roku zapowiedziano wersję na komputery osobiste, której premiera odbędzie się 25 maja 2017 roku.\\n\\nAkcja gry rozgrywa się w okresie, w którym na Ziemi zaczyna brakować surowców i dwa największe mocarstwa (Stany Zjednoczone i Rosja) rozpoczynają o nie rywalizację. Pierwsi atakują Rosjanie dowodzeni przez Zajcewa. Przywódca agresorów stawia ultimatum: kapitulacja w cią...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Książka obrazkowa – specyficzna forma książki skierowana najczęściej do najmłodszych dzieci. W tego typu książkach tekstu jest niewiele lub nie ma go wcale. W publikacjach tych warstwa wizualna odgrywa nadrzędną rolę, a warstwa tekstowa jest w nich jedynie dodatkiem. Tekst powinien komponować się z grafiką, a nie odwrotnie, oderwany od ilustracji nie posiada bowiem większego znaczenia. Bez obrazu nie jest zrozumiały. Książkę obrazkową określa się mianem ewenementu natury edytorskiej. \\n\\nW polskiej bibliografii brakuje literatury przedmiotu poświęconej książce obrazkowej. W wielu źródłach ...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Gilles Simon (ur. 27 grudnia 1984 w Nicei) – francuski tenisista, reprezentant w Pucharze Davisa, olimpijczyk.\\n\\nKarierę tenisową Francuz rozpoczął w roku 2002. Pierwsze zwycięstwo w turniejach rangi \"ITF Futures\" odniósł w czerwcu 2003 roku w Lizbonie, a zawodach kategorii ATP Challenger Tour w styczniu 2005 roku w Numei.\\n\\nW turniejach rangi ATP World Tour pierwszy znaczący wynik Simon osiągnął w połowie kwietnia roku 2006, awansując do finału rozgrywek w Walencji. Finałowy pojedynek przegrał jednak w dwóch setach z Nicolásem Almagro.\\n\\nNa początku sezonu 2007 Simon wygrał po raz pier...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Wsiewołod Emilewicz Meyerhold, Mejerchold ros. Всеволод Эмильевич Мейерхольд (ur. w Penzie, zm. 2 lutego 1940 w Moskwie) – rosyjski reżyser, aktor teatralny, twórca nowatorskiego współczesnego teatru sowieckiego.\\n\\nPrzyszedł na świat jako ósme z kolei dziecko właściciela fabryki wódek i kupca, Emila Meyerholda, Niemca z Dolnego Śląska, osiadłego w Rosji. Na chrzcie (w kościele ewangelickim obrządku luterańskiego) otrzymał imiona Karl Theodor Casimir. Do 1895 roku miał obywatelstwo niemieckie. W 1895 roku ukończył II gimnazjum w Penzie. Rok wcześniej nie został dopuszczony do egzaminu dojr...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Alexandre-Édouard Kierzkowski właśc. Aleksander Edward Kierzkowski herbu Krzywda (ur. 21 listopada 1816 w Raczycach, zm. 4 sierpnia 1870) – kanadyjski żołnierz, inżynier i polityk polskiego pochodzenia.\\n\\nUrodził się w Raczycach koło Odolanowa (Wielkie Księstwo Poznańskie) jako syn Filipa Jakuba (napoleończyka, oficera wojsk polskich) i Marianny Ludwiki Liebermann. Został ochrzczony 20 października 1817 roku w Odolanowie. Wstąpił do Wojska Polskiego w czasie powstania listopadowego. Po klęsce wyemigrował do Francji, gdzie uzyskał w Paryżu w 1838 roku dyplom inżyniera na École Centrale des...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n","0  Vanquish – gra konsolowa z gatunku third-person shooter, wyprodukowana przez PlatinumGames i wydana przez SEGA. Jej światowa premiera miała miejsce 19 października 2010 roku na konsolach PlayStation 3 i Xbox 360. 10 maja 2017 roku zapowiedziano wersję na komputery osobiste, której premiera odbędzie się 25 maja 2017 roku.\\n\\nAkcja gry rozgrywa się w okresie, w którym na Ziemi zaczyna brakować surowców i dwa największe mocarstwa (Stany Zjednoczone i Rosja) rozpoczynają o nie rywalizację. Pierwsi atakują Rosjanie dowodzeni przez Zajcewa. Przywódca agresorów stawia ultimatum: kapitulacja w cią...\n","1  Książka obrazkowa – specyficzna forma książki skierowana najczęściej do najmłodszych dzieci. W tego typu książkach tekstu jest niewiele lub nie ma go wcale. W publikacjach tych warstwa wizualna odgrywa nadrzędną rolę, a warstwa tekstowa jest w nich jedynie dodatkiem. Tekst powinien komponować się z grafiką, a nie odwrotnie, oderwany od ilustracji nie posiada bowiem większego znaczenia. Bez obrazu nie jest zrozumiały. Książkę obrazkową określa się mianem ewenementu natury edytorskiej. \\n\\nW polskiej bibliografii brakuje literatury przedmiotu poświęconej książce obrazkowej. W wielu źródłach ...\n","2  Gilles Simon (ur. 27 grudnia 1984 w Nicei) – francuski tenisista, reprezentant w Pucharze Davisa, olimpijczyk.\\n\\nKarierę tenisową Francuz rozpoczął w roku 2002. Pierwsze zwycięstwo w turniejach rangi \"ITF Futures\" odniósł w czerwcu 2003 roku w Lizbonie, a zawodach kategorii ATP Challenger Tour w styczniu 2005 roku w Numei.\\n\\nW turniejach rangi ATP World Tour pierwszy znaczący wynik Simon osiągnął w połowie kwietnia roku 2006, awansując do finału rozgrywek w Walencji. Finałowy pojedynek przegrał jednak w dwóch setach z Nicolásem Almagro.\\n\\nNa początku sezonu 2007 Simon wygrał po raz pier...\n","3  Wsiewołod Emilewicz Meyerhold, Mejerchold ros. Всеволод Эмильевич Мейерхольд (ur. w Penzie, zm. 2 lutego 1940 w Moskwie) – rosyjski reżyser, aktor teatralny, twórca nowatorskiego współczesnego teatru sowieckiego.\\n\\nPrzyszedł na świat jako ósme z kolei dziecko właściciela fabryki wódek i kupca, Emila Meyerholda, Niemca z Dolnego Śląska, osiadłego w Rosji. Na chrzcie (w kościele ewangelickim obrządku luterańskiego) otrzymał imiona Karl Theodor Casimir. Do 1895 roku miał obywatelstwo niemieckie. W 1895 roku ukończył II gimnazjum w Penzie. Rok wcześniej nie został dopuszczony do egzaminu dojr...\n","4  Alexandre-Édouard Kierzkowski właśc. Aleksander Edward Kierzkowski herbu Krzywda (ur. 21 listopada 1816 w Raczycach, zm. 4 sierpnia 1870) – kanadyjski żołnierz, inżynier i polityk polskiego pochodzenia.\\n\\nUrodził się w Raczycach koło Odolanowa (Wielkie Księstwo Poznańskie) jako syn Filipa Jakuba (napoleończyka, oficera wojsk polskich) i Marianny Ludwiki Liebermann. Został ochrzczony 20 października 1817 roku w Odolanowie. Wstąpił do Wojska Polskiego w czasie powstania listopadowego. Po klęsce wyemigrował do Francji, gdzie uzyskał w Paryżu w 1838 roku dyplom inżyniera na École Centrale des..."]},"metadata":{"tags":[]},"execution_count":14}]},{"cell_type":"markdown","metadata":{"id":"YDq7PWwg9N6U"},"source":["# 3. Byte-level BPE (BBPE) tokenizers from Transformers and Tokenizers (Hugging Face libraries)"]},{"cell_type":"markdown","metadata":{"id":"qTaEcu2b9N6U"},"source":["We are following 3 steps in order to get 2 identical GPT2 tokenizers, one trained on an English corpus and the other on Wikipedia in Portuguese\n","1. **Get the pre-trained GPT2 Tokenizer (pre-training with an English corpus) from the Transformers library (Hugging Face)**: it will give us the tokenizer stucture we need and the pre-trained English tokenizer.\n","2. **Train a Byte-level BPE (BBPE) Tokenizer on the Portuguese wikipedia corpus by using the Tokenizers library (Hugging Face)**: this will give us the vocabulary files of our GPT2 tokenizer in Portuguese.\n","3. **Import the tokenizer config files in Portuguese into the pre-trained GPT2 Tokenizer**: it will give us a tokenizer structure with the vocab in Portuguese."]},{"cell_type":"markdown","metadata":{"id":"ZbrQZ9Ee9N6U"},"source":["### 1. Get the pre-trained GPT2 Tokenizer (pre-training with an English corpus)"]},{"cell_type":"markdown","metadata":{"id":"IrR3gUC89N6V"},"source":["Firstly, will need to install the transformers library."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IdgP11o39N6W","executionInfo":{"status":"ok","timestamp":1615922739832,"user_tz":-60,"elapsed":10967,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"93d6580c-a3d8-4a22-c24b-9cdd167167eb"},"source":["! pip install transformers\n","!pip freeze | grep transformers"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: transformers in /usr/local/lib/python3.7/dist-packages (4.4.0)\n","Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (20.9)\n","Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers) (3.7.2)\n","Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.41.1)\n","Requirement already satisfied: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.10.1)\n","Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n","Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n","Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.43)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n","Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n","Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.1)\n","Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.7.4.3)\n","Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n","Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2020.12.5)\n","transformers==4.4.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"2j40yfrH9N6W"},"source":["Then, we can get the English pre-trained GPT2 tokenizer ([GPT2Tokenizer](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizer) or [GPT2TokenizerFast](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast))."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"x8q1Ck6J9N6W","executionInfo":{"status":"ok","timestamp":1615922740192,"user_tz":-60,"elapsed":10300,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"55b46c7c-05e3-4e00-fcbc-c6801c006f65"},"source":["%%time\n","from transformers import GPT2TokenizerFast\n","\n","pretrained_weights = 'gpt2'\n","tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["CPU times: user 164 ms, sys: 40.9 ms, total: 205 ms\n","Wall time: 532 ms\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Z50odrrq9N6W"},"source":["# To correct the warning about token_pad (GPT2TokenizerFast), run the following code\n","# source: https://github.com/huggingface/transformers/issues/2648#issuecomment-616177044\n","tokenizer_en.pad_token = tokenizer_en.eos_token"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"amNNOcEb9N6W"},"source":["Let's check the GPT2 tokenizer vocab. "]},{"cell_type":"code","metadata":{"scrolled":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"LSV4oDRI9N6W","executionInfo":{"status":"ok","timestamp":1615922740568,"user_tz":-60,"elapsed":5409,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"b1c69bb4-527c-4aec-f37d-b776409c0220"},"source":["# source: https://huggingface.co/transformers/_modules/transformers/tokenization_utils_fast.html\n","\n","print('---------- vocab ----------')\n","print()\n","\n","print('vocab_files_names:',tokenizer_en.vocab_files_names)\n","print()\n","\n","for k,v in tokenizer_en.pretrained_vocab_files_map.items():\n","    print(k)\n","    for kk,vv in v.items():\n","        print('- ',kk,':',vv)\n","    print()\n","    \n","print('vocab_size:',tokenizer_en.vocab_size)\n","print()\n","#print(tokenizer_en.get_vocab())\n","\n","num = 50\n","print(f'First {num} items of the vocab: {dict(itertools.islice(tokenizer_en.get_vocab().items(), 20))}')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["---------- vocab ----------\n","\n","vocab_files_names: {'vocab_file': 'vocab.json', 'merges_file': 'merges.txt', 'tokenizer_file': 'tokenizer.json'}\n","\n","vocab_file\n","-  gpt2 : https://huggingface.co/gpt2/resolve/main/vocab.json\n","-  gpt2-medium : https://huggingface.co/gpt2-medium/resolve/main/vocab.json\n","-  gpt2-large : https://huggingface.co/gpt2-large/resolve/main/vocab.json\n","-  gpt2-xl : https://huggingface.co/gpt2-xl/resolve/main/vocab.json\n","-  distilgpt2 : https://huggingface.co/distilgpt2/resolve/main/vocab.json\n","\n","merges_file\n","-  gpt2 : https://huggingface.co/gpt2/resolve/main/merges.txt\n","-  gpt2-medium : https://huggingface.co/gpt2-medium/resolve/main/merges.txt\n","-  gpt2-large : https://huggingface.co/gpt2-large/resolve/main/merges.txt\n","-  gpt2-xl : https://huggingface.co/gpt2-xl/resolve/main/merges.txt\n","-  distilgpt2 : https://huggingface.co/distilgpt2/resolve/main/merges.txt\n","\n","tokenizer_file\n","-  gpt2 : https://huggingface.co/gpt2/resolve/main/tokenizer.json\n","-  gpt2-medium : https://huggingface.co/gpt2-medium/resolve/main/tokenizer.json\n","-  gpt2-large : https://huggingface.co/gpt2-large/resolve/main/tokenizer.json\n","-  gpt2-xl : https://huggingface.co/gpt2-xl/resolve/main/tokenizer.json\n","-  distilgpt2 : https://huggingface.co/distilgpt2/resolve/main/tokenizer.json\n","\n","vocab_size: 50257\n","\n","First 50 items of the vocab: {'ĠCao': 34513, 'Ġgenerals': 26770, 'Ġanecdotal': 41666, 'Ġmaize': 49235, 'ĠSal': 4849, 'Ġbribery': 37388, 'Solution': 46344, 'ãĢĤ': 16764, 'ĠMarriott': 49722, 'Ġide': 1405, 'ĠKirin': 40262, 'abc': 39305, 'rugged': 21901, 'ĠOath': 37838, 'Ġillicit': 24271, 'Ġditch': 30115, 'Ġagreeing': 24093, 'ĠMale': 12674, 'ĠWii': 16591, 'Ï': 139}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"NgPx_txH9N6W"},"source":["# 4. Train a Byte-level BPE (BBPE) tokenizer on the Polish Wikipedia"]},{"cell_type":"markdown","metadata":{"id":"Cztdiui79N6X"},"source":["We'll use the [Tokenizers](https://github.com/huggingface/tokenizers) library from Hugging Face in order to train a **Byte-level BPE (BBPE) Tokenizer** on the Portuguese Wikipedia with the objective to get the vocab files `vocab.json`, which is a list of the most frequent tokens ranked by frequency, and the `merges.txt` which is a list of merges."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"zVWvn0tb9N6X","executionInfo":{"status":"ok","timestamp":1615922745196,"user_tz":-60,"elapsed":3242,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"86e7418d-64a1-4164-95b3-274fefde1949"},"source":["!pip install tokenizers\n","!pip freeze | grep tokenizers"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: tokenizers in /usr/local/lib/python3.7/dist-packages (0.10.1)\n","tokenizers==0.10.1\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"TevENzVvERkE"},"source":["# copying previosly created wiki txt file\r\n","\r\n","\r\n","\r\n"]},{"cell_type":"code","metadata":{"id":"NnJ5kgKgEQK_"},"source":["!cp /content/gdrive/MyDrive/fastai/all_texts_plwiki.csv  /root/.fastai/data/plwiki\r\n","!cp /content/gdrive/MyDrive/fastai/all_texts_plwiki.txt  /root/.fastai/data/plwiki"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"BEHa6PoFE8QE","executionInfo":{"status":"ok","timestamp":1615921133929,"user_tz":-60,"elapsed":779,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"ad18112f-16e6-4b5f-b61e-988721887dff"},"source":["ls /root/.fastai/data/plwiki -all"],"execution_count":null,"outputs":[{"output_type":"stream","text":["total 2147980\n","drwxr-xr-x 3 root root       4096 Mar 16 18:26 \u001b[0m\u001b[01;34m.\u001b[0m/\n","drwxr-xr-x 3 root root       4096 Mar 16 17:19 \u001b[01;34m..\u001b[0m/\n","-rw------- 1 root root 1101183658 Mar 16 17:41 all_texts_plwiki.csv\n","-rw------- 1 root root 1098323868 Mar 16 17:42 all_texts_plwiki.txt\n","drwxr-xr-x 2 root root       4096 Mar 16 18:26 \u001b[01;34mByteLevelBPE_tokenizer_pl\u001b[0m/\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"YmZ9HhWB9N6X"},"source":["#### Training"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"UjrSENrEEXUM","executionInfo":{"status":"ok","timestamp":1615919133492,"user_tz":-60,"elapsed":573,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"46355a5e-d1a5-46b1-e8f9-fba9778802d0"},"source":["path_data"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["Path('/root/.fastai/data/plwiki')"]},"metadata":{"tags":[]},"execution_count":27}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"dSy-Otzk9N6X","executionInfo":{"status":"ok","timestamp":1615916535173,"user_tz":-60,"elapsed":589,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"09158175-6c28-49b6-ed78-f44899a44663"},"source":["# Get GPT2 tokenizer_en vocab size\n","ByteLevelBPE_tokenizer_pl_vocab_size = tokenizer_en.vocab_size\n","ByteLevelBPE_tokenizer_pl_vocab_size"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["50257"]},"metadata":{"tags":[]},"execution_count":23}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"p2YYmwvn9N6Y","executionInfo":{"status":"ok","timestamp":1615918526044,"user_tz":-60,"elapsed":1798039,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"b45535c0-97e0-4547-eed5-db9fee50f1c8"},"source":["%%time\n","# ByteLevelBPETokenizer Represents a Byte-level BPE as introduced by OpenAI with their GPT-2 model\n","from tokenizers import ByteLevelBPETokenizer\n","\n","ByteLevelBPE_tokenizer_pl = ByteLevelBPETokenizer()\n","\n","# Get list of paths to corpus files\n","paths = [str(path_data/'all_texts_plwiki.txt')]\n","\n","# Customize training with <|endoftext|> special GPT2 token\n","ByteLevelBPE_tokenizer_pl.train(files=paths, \n","                                vocab_size=ByteLevelBPE_tokenizer_pl_vocab_size, \n","                                min_frequency=2, \n","                                special_tokens=[\"<|endoftext|>\"],\n","                                show_progress=True) \n","\n","# Get sequence length max of 1024\n","ByteLevelBPE_tokenizer_pl.enable_truncation(max_length=1024)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["CPU times: user 58min 41s, sys: 10.3 s, total: 58min 51s\n","Wall time: 29min 57s\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9rUfqMqR9N6Z","executionInfo":{"status":"ok","timestamp":1615919175964,"user_tz":-60,"elapsed":615,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"372d389f-599b-422d-9df5-95febd71a518"},"source":["# save tokenizer pl\n","ByteLevelBPE_tokenizer_pl_rep = 'ByteLevelBPE_tokenizer_pl'\n","path_to_ByteLevelBPE_tokenizer_pl_rep = path_data/ByteLevelBPE_tokenizer_pl_rep\n","if not (path_to_ByteLevelBPE_tokenizer_pl_rep).exists():\n","    path_to_ByteLevelBPE_tokenizer_pl_rep.mkdir(exist_ok=True, parents=True)\n","ByteLevelBPE_tokenizer_pl.save_model(str(path_to_ByteLevelBPE_tokenizer_pl_rep))"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['/root/.fastai/data/plwiki/ByteLevelBPE_tokenizer_pl/vocab.json',\n"," '/root/.fastai/data/plwiki/ByteLevelBPE_tokenizer_pl/merges.txt']"]},"metadata":{"tags":[]},"execution_count":29}]},{"cell_type":"markdown","metadata":{"id":"Wa_TWRWlS6BD"},"source":["lets copy to avoid loosing 30min next time"]},{"cell_type":"code","metadata":{"id":"oXFuQjgESlQy"},"source":["!cp /root/.fastai/data/plwiki/ByteLevelBPE_tokenizer_pl/vocab.json /content/gdrive/MyDrive/fastai\r\n","!cp /root/.fastai/data/plwiki/ByteLevelBPE_tokenizer_pl/merges.txt /content/gdrive/MyDrive/fastai"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"jN4B71079N6Z"},"source":["We now have both a `vocab.json`, which is a list of the most frequent tokens ranked by frequency, and a `merges.txt` list of merges."]},{"cell_type":"markdown","metadata":{"id":"fegL3IlK9N6Z"},"source":["#### Check our tokenizer pre-trained in Polish"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"JgGoCjj79N6Z","executionInfo":{"status":"ok","timestamp":1615919323169,"user_tz":-60,"elapsed":702,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"127d0d67-f49c-4b3a-efe5-4cb99f9ab896"},"source":["# Get vocab as a list\n","ByteLevelBPE_tokenizer_pl_vocab = ByteLevelBPE_tokenizer_pl.get_vocab() \n","ByteLevelBPE_tokenizer_pl_vocab_ls = [k for k, v in sorted(ByteLevelBPE_tokenizer_pl_vocab.items(), key=lambda item: item[1])]\n","len(ByteLevelBPE_tokenizer_pl_vocab_ls),ByteLevelBPE_tokenizer_pl_vocab_ls[:5]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(50257, ['<|endoftext|>', '!', '\"', '#', '$'])"]},"metadata":{"tags":[]},"execution_count":31}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cJsdVAlP9N6a","executionInfo":{"status":"ok","timestamp":1615919401435,"user_tz":-60,"elapsed":390,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"c2c0fc02-0930-4d99-991f-01a65c415fb4"},"source":["text = \"Nic nie rozumiem, nie wiem, nic mnie nie obchodzi.\"\n","output = ByteLevelBPE_tokenizer_pl.encode(text)\n","output.ids,output.tokens,output.offsets"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["([7402, 67, 372, 22763, 77, 12, 372, 498, 77, 12, 9253, 11990, 372, 39656, 14],\n"," ['Ni',\n","  'c',\n","  'Ġnie',\n","  'Ġrozumie',\n","  'm',\n","  ',',\n","  'Ġnie',\n","  'Ġwie',\n","  'm',\n","  ',',\n","  'Ġnic',\n","  'Ġmnie',\n","  'Ġnie',\n","  'Ġobchodzi',\n","  '.'],\n"," [(0, 2),\n","  (2, 3),\n","  (3, 7),\n","  (7, 15),\n","  (15, 16),\n","  (16, 17),\n","  (17, 21),\n","  (21, 25),\n","  (25, 26),\n","  (26, 27),\n","  (27, 31),\n","  (31, 36),\n","  (36, 40),\n","  (40, 49),\n","  (49, 50)])"]},"metadata":{"tags":[]},"execution_count":34}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Mq6v3bF19N6a","executionInfo":{"status":"ok","timestamp":1615919458731,"user_tz":-60,"elapsed":847,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"c1257db7-a1b8-495a-9b7a-32edac4a8bf8"},"source":["back_to_text = ByteLevelBPE_tokenizer_pl.decode(ByteLevelBPE_tokenizer_pl.encode(text).ids)\n","\n","print('input text:', text)\n","print('tokens ids:', output.ids)\n","print('back to text:', back_to_text)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["input text: Nic nie rozumiem, nie wiem, nic mnie nie obchodzi.\n","tokens ids: [7402, 67, 372, 22763, 77, 12, 372, 498, 77, 12, 9253, 11990, 372, 39656, 14]\n","back to text: Nic nie rozumiem, nie wiem, nic mnie nie obchodzi.\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ZQl_LSxu9N6a"},"source":["### 3. Import the Polish tokenizer config files into the pre-trained GPT2 Tokenizer"]},{"cell_type":"markdown","metadata":{"id":"Q9ZfWP5T9N6a"},"source":["In order to get exactly the same tokenizer model but with different vocabs (English and Polish), we import the Polish vocab files (`vocab.json` and `merges.txt`) into the `GPT2TokenizerFast` model."]},{"cell_type":"code","metadata":{"id":"7_z14LiJ9N6b"},"source":["# Get the path to ByteLevelBPE_tokenizer_pt config files\n","ByteLevelBPE_tokenizer_pl_rep = 'ByteLevelBPE_tokenizer_pl'\n","path_to_ByteLevelBPE_tokenizer_pl_rep = path_data/ByteLevelBPE_tokenizer_pl_rep\n","\n","# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n","tokenizer_pl = GPT2TokenizerFast.from_pretrained(\n","    str(path_to_ByteLevelBPE_tokenizer_pl_rep), \n","    pad_token='<|endoftext|>')\n","\n","# Get sequence length max of 1024\n","tokenizer_pl.model_max_length = 1024"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"aqnm2Ndp9N6b","executionInfo":{"status":"ok","timestamp":1615919647502,"user_tz":-60,"elapsed":6228,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"7417e673-e421-4be5-afd9-10b0584518a0"},"source":["# Check\n","text = \"Nic nie rozumiem, nie wiem, nic mnie nie obchodzi.\"\n","tokens_ids = tokenizer_pl.encode(text)\n","back_to_text = tokenizer_pl.decode(tokenizer_pl.encode(text))\n","\n","print('input text:', text)\n","print('tokens ids:', tokens_ids)\n","print('back to text:', back_to_text)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["input text: Nic nie rozumiem, nie wiem, nic mnie nie obchodzi.\n","tokens ids: [7402, 67, 372, 22763, 77, 12, 372, 498, 77, 12, 9253, 11990, 372, 39656, 14]\n","back to text: Nic nie rozumiem, nie wiem, nic mnie nie obchodzi.\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kPc3pmjh9N6b"},"source":["## English pre-trained tokenizer on a text in 3 languages (en, pl, ge)"]},{"cell_type":"markdown","metadata":{"id":"1wMzcYzz9N6b"},"source":["Let's tokenize by an English tokenizer a text in 3 different languages ​​(English, Portuguese, French) in order to observe the number of tokens required in each case."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mhg6tLtW9N6c","executionInfo":{"status":"ok","timestamp":1615919974064,"user_tz":-60,"elapsed":584,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"4a9cc9a1-9a67-49d4-928d-8769420f9d04"},"source":["# text in 3 languages to be tokenized\n","text_en = 'More than a dozen countries have suspended the use of AstraZeneca’s vaccine, although Australia, Britain and Thailand are moving ahead with it. Moderna has begun a study that will test its vaccine in children, including babies as young as six months.'\n","text_pl = 'Kilkanaście krajów zawiesiło stosowanie szczepionki AstraZeneca, chociaż Australia, Wielka Brytania i Tajlandia posuwają się naprzód. Moderna rozpoczęła badanie, w ramach którego zostanie przetestowana szczepionka u dzieci, w tym dzieci w wieku sześciu miesięcy. '\n","text_ge = 'Mehr als ein Dutzend Länder haben die Verwendung des Impfstoffs von AstraZeneca eingestellt, obwohl Australien, Großbritannien und Thailand damit fortfahren. Moderna hat eine Studie begonnen, in der der Impfstoff bei Kindern getestet wird, einschließlich Babys im Alter von sechs Monaten.'\n","\n","langs = ['en', 'pl', 'ge']\n","texts = [text_en,text_pl,text_ge]\n","\n","for lang,text in zip(*[langs,texts]):\n","    print(f'({lang}) {TitledStr(text)}\\n')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en) More than a dozen countries have suspended the use of AstraZeneca’s vaccine, although Australia, Britain and Thailand are moving ahead with it. Moderna has begun a study that will test its vaccine in children, including babies as young as six months.\n","\n","(pl) Kilkanaście krajów zawiesiło stosowanie szczepionki AstraZeneca, chociaż Australia, Wielka Brytania i Tajlandia posuwają się naprzód. Moderna rozpoczęła badanie, w ramach którego zostanie przetestowana szczepionka u dzieci, w tym dzieci w wieku sześciu miesięcy. \n","\n","(ge) Mehr als ein Dutzend Länder haben die Verwendung des Impfstoffs von AstraZeneca eingestellt, obwohl Australien, Großbritannien und Thailand damit fortfahren. Moderna hat eine Studie begonnen, in der der Impfstoff bei Kindern getestet wird, einschließlich Babys im Alter von sechs Monaten.\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"bdK_JzG99N6c","executionInfo":{"status":"ok","timestamp":1615919996723,"user_tz":-60,"elapsed":570,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"26b0308b-6ec4-4bb0-bbd5-d50cea765b6a"},"source":["# number and list of classical tokens (ie, tokens separated by a blank)\n","for lang,text in zip(*[langs,texts]):\n","    print(f'({lang} - {len(text.split())} tokens) {TitledStr(text.split(\" \"))}\\n')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en - 41 tokens) ['More', 'than', 'a', 'dozen', 'countries', 'have', 'suspended', 'the', 'use', 'of', 'AstraZeneca’s', 'vaccine,', 'although', 'Australia,', 'Britain', 'and', 'Thailand', 'are', 'moving', 'ahead', 'with', 'it.', 'Moderna', 'has', 'begun', 'a', 'study', 'that', 'will', 'test', 'its', 'vaccine', 'in', 'children,', 'including', 'babies', 'as', 'young', 'as', 'six', 'months.']\n","\n","(pl - 33 tokens) ['Kilkanaście', 'krajów', 'zawiesiło', 'stosowanie', 'szczepionki', 'AstraZeneca,', 'chociaż', 'Australia,', 'Wielka', 'Brytania', 'i', 'Tajlandia', 'posuwają', 'się', 'naprzód.', 'Moderna', 'rozpoczęła', 'badanie,', 'w', 'ramach', 'którego', 'zostanie', 'przetestowana', 'szczepionka', 'u', 'dzieci,', 'w', 'tym', 'dzieci', 'w', 'wieku', 'sześciu', 'miesięcy.', '']\n","\n","(ge - 40 tokens) ['Mehr', 'als', 'ein', 'Dutzend', 'Länder', 'haben', 'die', 'Verwendung', 'des', 'Impfstoffs', 'von', 'AstraZeneca', 'eingestellt,', 'obwohl', 'Australien,', 'Großbritannien', 'und', 'Thailand', 'damit', 'fortfahren.', 'Moderna', 'hat', 'eine', 'Studie', 'begonnen,', 'in', 'der', 'der', 'Impfstoff', 'bei', 'Kindern', 'getestet', 'wird,', 'einschließlich', 'Babys', 'im', 'Alter', 'von', 'sechs', 'Monaten.']\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"yaawqQyf9N6d","executionInfo":{"status":"ok","timestamp":1615920037166,"user_tz":-60,"elapsed":616,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"1e2fee83-c3cf-4de9-8e09-462d8ec0c5ac"},"source":["# number and list of tokens \n","# after the text tokenization by imported BPE GPT2TokenizerFast (trained with an English corpus...)\n","for lang,text in zip(*[langs,texts]):\n","    toks = tokenizer_en.tokenize(text)\n","    print(f'({lang} - {len(toks)} tokens) {TitledStr(toks)}\\n')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en - 53 tokens) ['More', 'Ġthan', 'Ġa', 'Ġdozen', 'Ġcountries', 'Ġhave', 'Ġsuspended', 'Ġthe', 'Ġuse', 'Ġof', 'ĠAst', 'ra', 'Zen', 'eca', 'âĢ', 'Ļ', 's', 'Ġvaccine', ',', 'Ġalthough', 'ĠAustralia', ',', 'ĠBritain', 'Ġand', 'ĠThailand', 'Ġare', 'Ġmoving', 'Ġahead', 'Ġwith', 'Ġit', '.', 'ĠModern', 'a', 'Ġhas', 'Ġbegun', 'Ġa', 'Ġstudy', 'Ġthat', 'Ġwill', 'Ġtest', 'Ġits', 'Ġvaccine', 'Ġin', 'Ġchildren', ',', 'Ġincluding', 'Ġbabies', 'Ġas', 'Ġyoung', 'Ġas', 'Ġsix', 'Ġmonths', '.']\n","\n","(pl - 131 tokens) ['K', 'ilk', 'ana', 'Å', 'Ľ', 'c', 'ie', 'Ġk', 'ra', 'j', 'Ã³', 'w', 'Ġz', 'aw', 'ies', 'i', 'ÅĤ', 'o', 'Ġst', 'os', 'ow', 'anie', 'Ġs', 'z', 'cz', 'ep', 'ion', 'ki', 'ĠAst', 'ra', 'Zen', 'eca', ',', 'Ġch', 'oc', 'ia', 'Å', '¼', 'ĠAustralia', ',', 'ĠW', 'iel', 'ka', 'ĠBry', 't', 'ania', 'Ġi', 'ĠTaj', 'land', 'ia', 'Ġpos', 'u', 'w', 'aj', 'Ä', 'ħ', 'Ġsi', 'Ä', 'Ļ', 'Ġnap', 'r', 'z', 'Ã³', 'd', '.', 'ĠModern', 'a', 'Ġro', 'z', 'p', 'oc', 'z', 'Ä', 'Ļ', 'ÅĤ', 'a', 'Ġbad', 'anie', ',', 'Ġw', 'Ġram', 'ach', 'Ġk', 't', 'Ã³', 're', 'go', 'Ġz', 'ost', 'anie', 'Ġpr', 'z', 'et', 'est', 'ow', 'ana', 'Ġs', 'z', 'cz', 'ep', 'ion', 'ka', 'Ġu', 'Ġd', 'zie', 'ci', ',', 'Ġw', 'Ġty', 'm', 'Ġd', 'zie', 'ci', 'Ġw', 'Ġw', 'ie', 'ku', 'Ġs', 'ze', 'Å', 'Ľ', 'ci', 'u', 'Ġm', 'ies', 'i', 'Ä', 'Ļ', 'cy', '.', 'Ġ']\n","\n","(ge - 101 tokens) ['Me', 'hr', 'Ġal', 's', 'Ġe', 'in', 'ĠD', 'utz', 'end', 'ĠL', 'Ã¤', 'nder', 'Ġha', 'ben', 'Ġdie', 'ĠVer', 'w', 'end', 'ung', 'Ġdes', 'ĠImp', 'f', 'st', 'offs', 'Ġvon', 'ĠAst', 'ra', 'Zen', 'eca', 'Ġe', 'ing', 'est', 'ell', 't', ',', 'Ġob', 'w', 'ohl', 'ĠAustral', 'ien', ',', 'ĠGro', 'ÃŁ', 'b', 'rit', 'ann', 'ien', 'Ġund', 'ĠThailand', 'Ġdam', 'it', 'Ġfort', 'f', 'ah', 'ren', '.', 'ĠModern', 'a', 'Ġhat', 'Ġe', 'ine', 'ĠStud', 'ie', 'Ġbeg', 'on', 'nen', ',', 'Ġin', 'Ġder', 'Ġder', 'ĠImp', 'f', 'st', 'off', 'Ġbe', 'i', 'ĠKind', 'ern', 'Ġget', 'est', 'et', 'Ġw', 'ird', ',', 'Ġe', 'ins', 'ch', 'lie', 'ÃŁ', 'lich', 'ĠBab', 'ys', 'Ġim', 'ĠAlter', 'Ġvon', 'Ġse', 'ch', 's', 'ĠMon', 'aten', '.']\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"TttXfW2O9N6d","executionInfo":{"status":"ok","timestamp":1615920063035,"user_tz":-60,"elapsed":576,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"1a75fba4-77b8-44d5-a010-9611eb734ede"},"source":["# number and list of tokens ids\n","# after the text tokenization + numerization by imported BPE GPT2TokenizerFast (trained with an English corpus...)\n","for lang,text in zip(*[langs,texts]):\n","    toks_ids = tokenizer_en.encode(text)\n","    print(f'({lang} - {len(toks_ids)} tokens) {TitledStr(toks_ids)}\\n')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en - 53 tokens) [5167, 621, 257, 8667, 2678, 423, 9951, 262, 779, 286, 8304, 430, 47573, 31047, 447, 247, 82, 12319, 11, 3584, 4505, 11, 5491, 290, 16952, 389, 3867, 4058, 351, 340, 13, 12495, 64, 468, 9258, 257, 2050, 326, 481, 1332, 663, 12319, 287, 1751, 11, 1390, 11903, 355, 1862, 355, 2237, 1933, 13]\n","\n","(pl - 131 tokens) [42, 43545, 2271, 129, 249, 66, 494, 479, 430, 73, 10205, 86, 1976, 707, 444, 72, 41615, 78, 336, 418, 322, 34166, 264, 89, 26691, 538, 295, 4106, 8304, 430, 47573, 31047, 11, 442, 420, 544, 129, 120, 4505, 11, 370, 8207, 4914, 9092, 83, 5411, 1312, 32490, 1044, 544, 1426, 84, 86, 1228, 128, 227, 33721, 128, 247, 25422, 81, 89, 10205, 67, 13, 12495, 64, 686, 89, 79, 420, 89, 128, 247, 41615, 64, 2089, 34166, 11, 266, 15770, 620, 479, 83, 10205, 260, 2188, 1976, 455, 34166, 778, 89, 316, 395, 322, 2271, 264, 89, 26691, 538, 295, 4914, 334, 288, 49746, 979, 11, 266, 1259, 76, 288, 49746, 979, 266, 266, 494, 23063, 264, 2736, 129, 249, 979, 84, 285, 444, 72, 128, 247, 948, 13, 220]\n","\n","(ge - 101 tokens) [5308, 11840, 435, 82, 304, 259, 360, 27839, 437, 406, 11033, 681, 387, 11722, 4656, 4643, 86, 437, 2150, 748, 9855, 69, 301, 8210, 18042, 8304, 430, 47573, 31047, 304, 278, 395, 695, 83, 11, 909, 86, 48988, 3138, 2013, 11, 10299, 39683, 65, 799, 1236, 2013, 3318, 16952, 1801, 270, 6285, 69, 993, 918, 13, 12495, 64, 6877, 304, 500, 3604, 494, 4123, 261, 38572, 11, 287, 4587, 4587, 9855, 69, 301, 2364, 307, 72, 14927, 1142, 651, 395, 316, 266, 1447, 11, 304, 1040, 354, 14485, 39683, 33467, 12400, 893, 545, 32770, 18042, 384, 354, 82, 2892, 36686, 13]\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"zqvqye8j9N6d","executionInfo":{"status":"ok","timestamp":1615920072450,"user_tz":-60,"elapsed":567,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"d40f5d43-7558-48f0-b78d-c68fedc72f9f"},"source":["# decode (back to the text)\n","for lang,text in zip(*[langs,texts]):\n","    toks_ids = tokenizer_en.encode(text)\n","    text_decoded = tokenizer_en.decode(toks_ids)\n","    print(f'({lang}) {TitledStr(text_decoded)}\\n')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en) More than a dozen countries have suspended the use of AstraZeneca’s vaccine, although Australia, Britain and Thailand are moving ahead with it. Moderna has begun a study that will test its vaccine in children, including babies as young as six months.\n","\n","(pl) Kilkanaście krajów zawiesiło stosowanie szczepionki AstraZeneca, chociaż Australia, Wielka Brytania i Tajlandia posuwają się naprzód. Moderna rozpoczęła badanie, w ramach którego zostanie przetestowana szczepionka u dzieci, w tym dzieci w wieku sześciu miesięcy. \n","\n","(ge) Mehr als ein Dutzend Länder haben die Verwendung des Impfstoffs von AstraZeneca eingestellt, obwohl Australien, Großbritannien und Thailand damit fortfahren. Moderna hat eine Studie begonnen, in der der Impfstoff bei Kindern getestet wird, einschließlich Babys im Alter von sechs Monaten.\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":295},"id":"OK2S8EYd9N6e","executionInfo":{"status":"ok","timestamp":1615920085306,"user_tz":-60,"elapsed":635,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"5fe4b283-ea22-4f0a-b5db-0d2c995b2027"},"source":["# graph \"Number of tokens by tokenization method and lang\"\n","# source: https://matplotlib.org/3.2.1/gallery/lines_bars_and_markers/barchart.html#sphx-glr-gallery-lines-bars-and-markers-barchart-py\n","text_split = list()\n","toks_split = list()\n","\n","for text in texts:\n","    text_split.append(len(text.split()))\n","    toks_ids = tokenizer_en.encode(text)\n","    toks_split.append(len(toks_ids))\n","    \n","labels = langs\n","xy = list(np.array([1.,2.,3.]) - 0.2)\n","xz = list(np.array([1.,2.,3.]) + 0.2)\n","y = text_split\n","z = toks_split\n","\n","ax = plt.subplot(111)\n","ax.bar(xy, y, width=0.4, color='b', align='center')\n","ax.bar(xz, z, width=0.4, color='g', align='center')\n","\n","ax.set_xlabel('languages')\n","ax.set_xticks(range(1,len(labels)+1))\n","ax.set_xticklabels(labels)\n","ax.set_ylabel('number of tokens')\n","ax.legend(['split(\" \")', 'GPTTokenizerFast (en)'])\n","\n","ax.set_title('Number of tokens by tokenization method and lang')\n","\n","plt.show()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZyVZf3/8debTRARF0YCQUZzZxFhxNT4ipq5p5mZpqa5kGaSZSiZqZh9W7RS+xapaVJfNY3cvvzMNUhxSRZxQdRIMVlUREVEUcDP74/7muPNMDOcWc4cmHk/H4/zmHNf93J9zjn3nM+5r+u+70sRgZmZGUC7cgdgZmbrDicFMzMrcFIwM7MCJwUzMytwUjAzswInBTMzK3BSKDFJN0i6tEx1S9IfJL0t6Ylm2N5JkqY0R2yNrP9iSf9bprrnSvpcM29zuKQXmnObabvHSbqvubdbas25f0kaIWlec2yriLrq/B8v9/9MY7S5pJD+ud+Q1DVXdqqkyWUMq1Q+C+wP9ImIYTVnro87bHMpxZd8Q0XEwxGxQ1O2IalSUkjqkNvujRHx+aZHWDq1xW3rhjaXFJL2wLfLHURDSWrfwFX6AXMjYlkp4jGz1qetJoXLgO9J2qTmjNp+wUiaLOnU9PwkSY9I+pWkdyS9JGnPVP5qOgo5scZme0i6X9JSSf+Q1C+37R3TvLckvSDp6Ny8GySNk3S3pGXAPrXE21vSXWn9OZJOS+WnAL8H9pD0nqSxNdbbCfhdbv47qby7pD9KWiTpFUkXSKp1P5F0maQpaZ3ukq6TtFDSfEmXViex6iMSSZenpqyXJR2U285J6X1cmuYdV8fnBtBZ0i1p2RmSdknbGC3przXiu0rSlbXE/SdgK+D/0ms/N5V/QdKs9LlOTu9Rba97pxTnsWn6UEkz03qPShqUW3aupO9JelrSkhR75zSv0MQh6SsplurHh9VHr5IOkfSkpHfTPnZxLpyH0t930np71DwCTPvn1FT/VEl75uZNlvSjtE8vlXSfpB51vO4RkuZJOjft5wslHSHpYEkvpn3w/Nzy7SSNkfRvSYsl3Spps7rizq1X135S676e5nVJ/y9vS3oO2K2215Bb/sr0Xr4rabqk4bl5F6dY/5jek1mSqnLzd0373lJJtwCd66urGesdkvaDpZL+kval5m+ajog29QDmAp8DbgMuTWWnApPT80oggA65dSYDp6bnJwErga+THXFcCvwH+A2wAfB5YCmwUVr+hjT9X2n+lcCUNK8r8GraVgdgV+BNYOfcukuAvcgSeOdaXs9DwG/JdszBwCJg31ysU+p5L9aYD/wRuBPolt6LF4FT8sunWK4F7gU2TPNuB65Or2kL4AngG7n1VgCnpffsDGABoLT8u8AOadleQP864r04becooCPwPeDl9LwXsAzYJC3bAXgDGFrffpCb3j6tv3/a3rnAHKBTjf1mSPq8D03lu6Z6dk+v7cS07Aa59Z4AegObAbOB09O8EcC8WmLbOC33jdxyA9P7Pgh4HTiinv218LmmOt8GTkjvybFpevPcvv3v9Pq7pOmf1vGejSDb9y9M79FpZPvbTWT7S3/gA2DrtPy3gceBPmT7/tXAzWuJu9b9pIh9/afAw+n19gWere29zdV1PLB5ek/OAV4j/X+R7WfLgYNTHD8BHk/zOgGvAN9J78FRKeZLi/kfa4Z6v53qPRL4qK56m/Qd2ZJfyOvCg0/+uQeQfeFW0PCk8K/cvIFp+Z65ssXA4PT8BuDPuXkbAavSjvsV4OEa8V0NXJRb94/1vJa+aVvdcmU/AW6obYcsYodtn3a0nXNl38i9NycB/wRuAf7KJ1+YPYEPgS659Y4FJuXWm5Obt2F6zz5FlhTeAb6UX7+OeC+u/idJ0+2AhcDwNP034LT0/FDgubXtB7npHwK31tj2fGBEbvmxwLzqslQ+DvhRjW2/AOydW+/43LyfA79Lz0dQ44sr1TsRGFdP7FcAv6pnfy18rmTJ4Ika6z8GnJTbty/IzfsmcE8d9Y4g+9Jvn6a7pbp3zy0znU8S1mxgv9y8XmRfoB3qibuu/WRt+/pLwIG5eSNrvrdr2bfeBnbJ7WcP5ObtDHyQnv8XuUSVyh6lyKTQxHrn16h3Sl31NuXRVpuPiIhnyf75xjRi9ddzzz9I26tZtlFu+tVcve8Bb5H9cuwH7J6aHd5R1oRzHNk/wRrr1qI38FZELM2VvQJs2YDXkteD7FfIK/Vsb1vgcGBsRHyUyvql9RbmXsfVZEcM1V6rfhIR76enG0XW3/EV4PS0/v+TtGM9Mebfy4/JvqR7p6LxZL/ESH//VP/LXU1vcq87bftVVn/tpwOPRsTkXFk/4Jwan2HfXEyQe+3A+6y+b9T0Y7Iv21HVBZJ2lzRJWZPekhRHrU08a3tdSc3PtCHxLY6IVen5B+lvXft+P+D23Psym+yLvWc92691P2Ht+3pvVv9fqfmaV5Oa9GanJrV3gO6s/p7WfE86K2tS7g3Mj/StXExdJay3vu+GRmuzSSG5iOxQNf8PUt0pu2GuLP8l3Rh9q59I2ojsEHcB2Yf6j4jYJPfYKCLOyK0b1G0BsJmkbrmyrch+URSj5rbfJPsl16+e7c0ma+76m6TqM2deJTtS6JF7HRtHRP+igoi4NyL2J/sl+TxZ01Rd8u9lO7KmiQWp6A5gkKQBZEcKN9ZXbY3pBeRetySluvKv/XRgK0m/ypW9Cvy4xme4YUTcXE/dtZJ0DNkR1lERsSI36ybgLqBvRHQn6wtSHa+jptVeV9KQfaQpXgUOqvHedI6I+aw97prWtq8vJLdvpHm1Su345wJHA5tGxCZkrQaqa52chcCWaf9Ya10lrrdvXQs3RZtOChExh6wpZFSubBHZjna8pPaSTgY+3cSqDpb0WUmdgB+RNYG8Snaksr2kEyR1TI/dVEcHZy3xv0p26PoTSZ2VdXCeAhR7Lv/rQJ8UF+kX4K3AjyV1U9Yh/t2a20tfeOcDD0j6dEQsBO4DfiFp49TB+GlJe68tAEk9JR2u7BThD4H3gI/rWWWopCPTr6ez0zqPp7iWAxPIvkSfiIj/rOW1b5ObvhU4RNJ+kjqStfd+SPb+VlsKHAj8l6SfprJrgdPTr3lJ6qqsYzj/5bVWknYFfk3W9LKoxuxuZL+Sl0saBnw1N28R2fu1DbW7m2wf+6qkDpK+QtYsMbEh8TXS78j2pX4AkiokHZ7mrS3u1RSxr98KfF/SppL6AGfVs7luZH0ji4AOki4k68cpxmNp3VHp//VIYI3TvUtU7yrgW+lzPLwB9TZIm04KySVk7dp5pwGjyfoG+rP6F0Nj3ER2VPIWMJTUxJEOhT8PHEP2S+g14GdknXLFOpasfXYBWWfvRRHxQJHr/h2YBbwm6c1UdhbZ0dJLZG2WNwHX11wxIsaTvXd/l1QJfI2sM+w5snbSCWS//NemHVniWUD2/uxN1sFYlzvJmpuqO0+PrPGrejxZP8/amo5+AlyQmja+FxEvkH0uvyY7YjoMOCzXRAZARLxD1hl9kKQfRcQ0sv3lf1JMc8jakRvqcGBTYIo+OQPpb2neN4FLJC0l6+S9NRfP+2RNTo+k1/KZGvEuJjtqOodsfz6XrJP8TUrvSrIjnPtS7I+TdcivNe461LevjyVrxnmZ7AdKfZ//vcA9ZCdRvELWuVtUU0zaH44k+4zfItsXbytm3Waq9xSyPrjjyRL7h0XWXbTqXn2zVkHSVmRNUJ+KiHfLHY9ZqUj6J9lJC39ozu36SMFajdTH8F2ys72cEKxVkbS3pE+l5qMTyU5Pvqe56/El5tYqpD6J18kOyw8sczhmpbADWdNhV7Lm3aNSf16zcvORmZkVuPnIzMwK1uvmox49ekRlZWW5wzAzW69Mnz79zYioqG3eep0UKisrmTZtWrnDMDNbr0iq8ypsNx+ZmVmBk4KZmRU4KZiZWcF63adQmxUrVjBv3jyWL19e7lBsPdW5c2f69OlDx44dyx2KWYtrdUlh3rx5dOvWjcrKSla/oaDZ2kUEixcvZt68eWy99dblDsesxbW65qPly5ez+eabOyFYo0hi880395GmtVmtLikATgjWJN5/rC1rlUnBzMwap9UnBal5H83t4osv5vLLLwfgwgsv5IEHstvDX3HFFbz//vuF5SKCfffdl3ffzW7+WVlZydy5cxkxYsQa26wunzx5MieddBIAEydO5MILL2z+F2BmrUqr62hen11yySWF51dccQXHH388G26YjQp69913s8suu7DxxsUO1LS6Qw45hB/+8IeMGTOmsE0rLY1df5qh4iLfGNMyrf5IoaUtW7aMQw45hF122YUBAwZwyy23ANkv+3PPPZeBAwcybNgw5syZs8a6J510EhMmTOCqq65iwYIF7LPPPuyzzz4A3HjjjRx++OGFZSsqKmjfvj2bbbbZGtupLu/UqRPdu3cHsnbyESNGMHFiS4zCaGbrKyeFZnbPPffQu3dvnnrqKZ599lkOPPCTW/t3796dZ555hm9961ucffbZdW5j1KhR9O7dm0mTJjFp0iQAHnnkEYYOHVpYZurUqfTt25fbbltzJMDq8j333JMrr7yyUF5VVcXDDz/cHC/TzFopJ4VmNnDgQO6//37OO+88Hn744cIvdYBjjz228Pexxx5r0HbfeustunVr0Fjwa9hiiy1YsGBBk7ZhZq2bk0Iz23777ZkxYwYDBw7kggsuWK2fIH+qY0NPe+zQoQMff/xxk2Jbvnw5Xbp0adI2zKx1c1JoZgsWLGDDDTfk+OOPZ/To0cyYMaMwr7p/4ZZbbmGPPfaodzvdunVj6dKlhekddtiBl156qUmxvfjiiwwYMKBJ2zCz1q3Vn33U0qONPvPMM4wePZp27drRsWNHxo0bV5j39ttvM2jQIDbYYANuvvnmerczcuRIDjzwwELfwiGHHMLkyZPZdtttGx3bpEmT+MlPftLo9c2s9Vuvx2iuqqqKmoPszJ49m5122qlMEdWtekCgHj16NGr9hQsX8rWvfY3777+/Ueu//vrrfPWrX+XBBx9s1PptTXPsRz4l1dZVkqZHRFVt89x8tJ7o1asXp512WuHitYb6z3/+wy9+8YtmjsrMWptW33y0rpg7d26Tt3H00Uc3et3ddtutyfWbWevnIwUzMytwUjAzswInBTMzK3BSMDOzgpJ1NEu6HjgUeCMiBqSyy4DDgI+AfwNfj4h30rzvA6cAq4BREXFvs8TRzKcF+tQ9M2vNSnmkcANwYI2y+4EBETEIeBH4PoCknYFjgP5pnd9Kal/C2Eqq+pqAbbbZhqFDh7LHHntw++23AzB58mS6d+/O4MGD2WmnnRg7diz33nsvgwcPZvDgwWy00UbssMMODB48mL59+xbKO3XqxMCBAxk8eDBjxoxZo87Jkydz6KGHNjn2adOmMWrUqCZvB7K7tVbHP3jw4AafgVVzTImajjrqqEZf5e3xJcxqV7IjhYh4SFJljbL7cpOPA0el54cDf46ID4GXJc0BhgENu2vcOiAiOOKIIzjxxBO56aabAHjllVe46667CssMHz6ciRMnsmzZMgYPHsxhhx3GzJkzARgxYgSXX345VVWrX1dSWVnJpEmTGn3xW7GqqqrWqLuhVq5cSYcOHejSpUvhdTVGzTEl8mbNmsWqVavYZpttGrVtjy9hVrty9imcDPwtPd8SeDU3b14qW4OkkZKmSZq2aNGiEofYcH//+9/p1KkTp59+eqGsX79+nHXWWWss27VrV4YOHVrr2Ap1iQhGjx7NgAEDGDhwYOF+SnlTp05l11135d///jfTp09n7733ZujQoRxwwAEsXLgQyJLPeeedx7Bhw9h+++0Lt9TOH3EcfPDBhV/53bt3Z/z48axatYrRo0ez2267MWjQIK6++urCesOHD+cLX/gCO++8c62xv/fee+y3334MGTKEgQMHcueddwK1j0FR25gSeTXHl7jvvvvYY489GDJkCF/+8pd57733gCyZXnTRRYU6n3/+ecDjS5jVpSxJQdIPgJXAjQ1dNyKuiYiqiKiqqKho/uCaaNasWQwZMqSoZRcvXszjjz9O//79i97+bbfdxsyZM3nqqad44IEHGD16dOGLHuDRRx/l9NNP584772SrrbbirLPOYsKECUyfPp2TTz6ZH/zgB4VlV65cyRNPPMEVV1zB2LFj16jr7rvvZubMmVx33XX069ePI444guuuu47u3bszdepUpk6dyrXXXsvLL78MwIwZM7jyyit58cUXAfjggw8KSeWLX/winTt35vbbb2fGjBlMmjSJc845h4iodQyK2saUyMuPL/Hmm29y6aWX8sADDzBjxgyqqqr45S9/WVi2R48ezJgxgzPOOKMw9Cl4fAmz2rT4Fc2STiLrgN4vPrnx0nygb26xPqlsvXfmmWcyZcoUOnXqxNSpUwF4+OGH2XXXXWnXrh1jxoxpUFKYMmUKxx57LO3bt6dnz57svffeTJ06lY033pjZs2czcuRI7rvvPnr37s2zzz7Ls88+y/777w/AqlWr6NWrV2FbRx55JABDhw6ts73/zTff5IQTTuDWW2+le/fu3HfffTz99NNMmDABgCVLlvCvf/2LTp06MWzYMLbeeuvCujWbj1asWMH555/PQw89RLt27Zg/fz6vv/46AwcO5JxzzuG8887j0EMPZfjw4Wt9HxYuXEj1j4LHH3+c5557jr322guAjz76aLW70OZfZ35QIo8vYbamFk0Kkg4EzgX2joh8D+JdwE2Sfgn0BrYDnmjJ2JpL//79+etf/1qY/s1vfsObb765Wjt9dZ9Cc+vVqxfLly/nySefpHfv3kQE/fv3r3NAnw022ADIOoRXrly5xvxVq1ZxzDHHcOGFFxZuuR0R/PrXv+aAAw5YbdnJkyfTtWvXeuO78cYbWbRoEdOnT6djx45UVlayfPnywhgUd999NxdccAH77bffWjuBu3TpwvLlywsx7b///nXeebau1+nxJczWVMpTUm8GRgA9JM0DLiI722gD4P40yMzjEXF6RMySdCvwHFmz0pkRsao54mjpU0j33Xdfzj//fMaNG8cZZ5wBUO8ZNA01fPhwrr76ak488UTeeustHnroIS677DKef/55NtlkE6677jr2339/unbtyp577smiRYt47LHH2GOPPVixYgUvvvhi0UcmY8aMYdCgQRxzzDGFsgMOOIBx48ax77770rFjR1588UW23LLW7p81LFmyhC222IKOHTsyadIkXnnlFSAbg2KzzTbj+OOPZ5NNNuH3v/898MmYErV1ru+0007MmTOHyspKPvOZz3DmmWcyZ84ctt12W5YtW8b8+fPZfvvt643H40uYramUZx8dW0vxdfUs/2Pgx6WKp6VI4o477uA73/kOP//5z6moqKBr16787Gc/a5btf/GLX+Sxxx5jl112QRI///nP+dSnPlXoQO3ZsycTJ07koIMO4vrrr2fChAmMGjWKJUuWsHLlSs4+++yik8Lll19O//79GTx4MACXXHIJp556KnPnzmXIkCFEBBUVFdxxxx1Fbe+4447jsMMOY+DAgVRVVbHjjjsCdY9BUXNMibzq8SU+97nPUVFRwQ033MCxxx7Lhx9+CMCll1661qTg8SXM1uTxFGy99MEHH7DPPvvwyCOP0L59wy9pWdv4Eh5PwVozj6dgrU6XLl0YO3Ys8+c37nwEjy9hVrtWOZ5CRJD6LKwVq9nZ3RD1jS+xPh89mzVVqztS6Ny5M4sXL/Y/tjVKRLB48WI6d+5c7lDMyqLVHSn06dOHefPmsS5e7Wzrh86dO9OnT59yh2FWFq0uKXTs2HG1C6jMzKx4ra75yMzMGs9JwczMCpwUzMyswEnBzMwKnBTMzKzAScHMzAqcFMzMrMBJwczMCpwUzMyswEnBzMwKnBTMzKzAScHMzAqcFMzMrMBJwczMCpwUzMyswEnBzMwKnBTMzKygZElB0vWS3pD0bK5sM0n3S/pX+rtpKpekqyTNkfS0pCGlisvMzOpWyuE4bwD+B/hjrmwM8GBE/FTSmDR9HnAQsF167A6MS3/NzFqExqrcITRIXBQl2W7JjhQi4iHgrRrFhwPj0/PxwBG58j9G5nFgE0m9ShWbmZnVrqX7FHpGxML0/DWgZ3q+JfBqbrl5qWwNkkZKmiZp2qJFi0oXqZlZG1S2juaICKDBxz8RcU1EVEVEVUVFRQkiMzNru1o6Kbxe3SyU/r6RyucDfXPL9UllZmbWglo6KdwFnJienwjcmSv/WjoL6TPAklwzk5mZtZCSnX0k6WZgBNBD0jzgIuCnwK2STgFeAY5Oi98NHAzMAd4Hvl6quMzMrG4lSwoRcWwds/arZdkAzixVLGZmVhxf0WxmZgUNSgqSNpU0qFTBmJlZea01KUiaLGljSZsBM4BrJf2y9KGZmVlLK+ZIoXtEvAscSXbV8e7A50oblpmZlUMxSaFDuqbgaGBiieMxM7MyKiYpXALcC8yJiKmStgH+VdqwzMysHNZ6SmpE/AX4S276JeBLpQzKzMzKY61JQVIFcBpQmV8+Ik4uXVhmZlYOxVy8difwMPAAsKq04ZiZWTkVkxQ2jIjzSh6JmZmVXTEdzRMlHVzySMzMrOyKSQrfJksMyyW9K2mppHdLHZiZmbW8Ys4+6tYSgZiZWfkVc5sLSTpe0g/TdF9Jw0ofmpmZtbRimo9+C+wBfDVNvwf8pmQRmZlZ2RRz9tHuETFE0pMAEfG2pE4ljsvMzMqgmCOFFZLaAwGFi9k+LmlUZmZWFsUkhauA24EtJP0YmAL8d0mjMjOzsiim+WgCMJ1sGE0BRwCvlzIoMzMrj2KSwm3AERHxPEC6jfb9wNBSBmZmZi2vmOajO4BbJbWXVEl2G+3vlzIoMzMrj2IuXrs2nW10B9mdUr8REY+WOjAzM2t5dSYFSd/NTwJbATOBz0j6TEQ0epxmSd8BTiU7o+kZ4OtAL+DPwOZkfRgnRMRHja3DzMwarr7mo265x0ZkfQtzcmWNImlLYBRQFREDgPbAMcDPgF9FxLbA28Apja3DzMwap84jhYgYm5+WtFEqf6+Z6u0iaQWwIbAQ2JdPrpoeD1wMjGuGuszMrEjF3PtoQLqaeRYwS9J0Sf0bW2FEzAcuB/5DlgyWkDUXvRMRK9Ni84At64hnpKRpkqYtWrSosWGYmVktijn76BrguxHRLyL6AecA1za2QkmbAocDWwO9ga7AgcWuHxHXRERVRFRVVFQ0NgwzM6tFMUmha0RMqp6IiMlkX+SN9Tng5YhYFBEryPoq9gI2kVTdnNUHmN+EOszMrBGKSQovSfqhpMr0uAB4qQl1/ofsDKYNJYnsSunngEnAUWmZE8nGhjYzsxZUTFI4Gagg+0X/V6AH2SmkjRIR/yS7dcYMstNR25E1UZ0HfFfSHLLTUq9rbB1mZtY4xdzm4nMRMSpfIOnLwF8aW2lEXARcVKP4JcCD95iZlVExRwq13dLCt7kwM2uF6rui+SDgYGBLSVflZm0MrKx9LTMzW5/V13y0AJgGfIHsOoJqS4HvlDIoMzMrj/quaH4KeErSTenUUTMza+XW2qfghGBm1nYU09FsZmZtRJ1JQdKf0t9vt1w4ZmZWTvUdKQyV1Bs4WdKmkjbLP1oqQDMzazn1nX30O+BBYBuys4+Umxep3MzMWpE6jxQi4qqI2Am4PiK2iYitcw8nBDOzVqiYMZrPkLQLMDwVPRQRT5c2LDMzK4diBtkZBdwIbJEeN0o6q9SBmZlZyyvmhninArtHxDIAST8DHgN+XcrAzMys5RVznYKAVbnpVaze6WxmZq1EMUcKfwD+Ken2NH0EHuvAzKxVKqaj+ZeSJgOfTUVfj4gnSxqVmZmVRTFHCkTEDLKR0szMrBXzvY/MzKzAScHMzArqTQqS2kua1FLBmJlZedWbFCJiFfCxpO4tFI+ZmZVRMR3N7wHPSLofWFZdGBGjShaVmZmVRTFJ4bb0MDOzVq6Y6xTGS+oCbBURLzRHpZI2AX4PDCC7DffJwAvALUAlMBc4OiLebo76zMysOMXcEO8wYCZwT5oeLOmuJtZ7JXBPROwI7ALMBsYAD0bEdmTjOIxpYh1mZtZAxZySejEwDHgHICJm0oQBdlKn9X+RbpURER9FxDvA4cD4tNh4sttpmJlZCyomKayIiCU1yj5uQp1bA4uAP0h6UtLvJXUFekbEwrTMa0DP2laWNFLSNEnTFi1a1IQwzMyspmKSwixJXwXaS9pO0q+BR5tQZwdgCDAuInYlO6NptaaiiAiyvoY1RMQ1EVEVEVUVFRVNCMPMzGoqJimcBfQHPgRuBt4Fzm5CnfOAeRHxzzQ9gSxJvC6pF0D6+0YT6jAzs0Yo5uyj94EfpMF1IiKWNqXCiHhN0quSdkhnM+0HPJceJwI/TX/vbEo9ZmbWcGtNCpJ2A64HuqXpJcDJETG9CfWeRTasZyfgJeDrZEctt0o6BXgFOLoJ2zczs0Yo5uK164BvRsTDAJI+SzbwzqDGVprOYKqqZdZ+jd2mmZk1XTF9CquqEwJAREwBVpYuJDMzK5c6jxQkDUlP/yHparJO5gC+AkwufWhmZtbS6ms++kWN6Ytyz2s9XdTMzNZvdSaFiNinJQMxM7PyK+bso02Ar5HdqK6wvG+dbWbW+hRz9tHdwOPAMzTt9hbWBBqrcodQtLjIrYtm66tikkLniPhuySMxM7OyK+aU1D9JOk1SL0mbVT9KHpmZmbW4Yo4UPgIuA37AJ2cdBU24fbaZma2bikkK5wDbRsSbpQ7GzMzKq5jmoznA+6UOxMzMyq+YI4VlwExJk8hunw34lFQzs9aomKRwR3qYmVkrV8x4CuPXtoyZmbUOxVzR/DK13OsoInz2kZlZK1NM81F+3IPOwJcBX6dgZtYKrfXso4hYnHvMj4grgENaIDYzM2thxTQfDclNtiM7cijmCMPMzNYzxXy558dVWAnMxeMnm5m1SsWcfeRxFczM2ohimo82AL7EmuMpXFK6sMzMrByKaT66E1gCTCd3RbOZmbU+xSSFPhFxYHNXLKk9MA2YHxGHStoa+DOwOVkCOiEiPmrues3MrG7F3BDvUUkDS1D3t4HZuemfAb+KiC37Y/4AAAhfSURBVG2Bt4FTSlCnmZnVo5ik8FlguqQXJD0t6RlJTzelUkl9yK51+H2aFrAvMCEtMh44oil1rD2G9ethZtYSimk+OqgE9V4BnAt0S9ObA+9ExMo0PQ/YsgT1mplZPYo5JfWV5qxQ0qHAGxExXdKIRqw/EhgJsNVWWzVnaGbWzNaro9yLyx3AuqGY5qPmthfwBUlzyTqW9wWuBDaRVJ2k+gDza1s5Iq6JiKqIqKqoqGiJeM3M2owWTwoR8f2I6BMRlcAxwN8j4jhgEnBUWuxEslNhzcysBZXjSKEu5wHflTSHrI/hujLHY2bW5pT1xnYRMRmYnJ6/BAwrZzxmZm3dunSkYGZmZeakYGZmBU4KZmZW4KRgZmYFTgpmZlbgpGBmZgVOCrbeKPdNCX0TQ2sLnBTMzKzAScHMzAqcFMzMrMBJwczMCpwUzMyswEnBzMwKnBTMzKzAScHMzAqcFMzMrMBJwczMCpwUzMyswEnBzMwKnBTMzKzAScHMzAqcFMzMrMBJwczMClo8KUjqK2mSpOckzZL07VS+maT7Jf0r/d20pWMzM2vrynGksBI4JyJ2Bj4DnClpZ2AM8GBEbAc8mKbNzKwFtXhSiIiFETEjPV8KzAa2BA4HxqfFxgNHtHRsZmZtXVn7FCRVArsC/wR6RsTCNOs1oGcd64yUNE3StEWLFrVInGZmbUXZkoKkjYC/AmdHxLv5eRERQNS2XkRcExFVEVFVUVHRApGambUdZUkKkjqSJYQbI+K2VPy6pF5pfi/gjXLEZmbWlpXj7CMB1wGzI+KXuVl3ASem5ycCd7Z0bGZmbV2HMtS5F3AC8IykmansfOCnwK2STgFeAY4uQ2xmZm1aiyeFiJgCqI7Z+7VkLGZmtjpf0WxmZgVOCmZmVuCkYGZmBU4KZmZW4KRgZmYFTgpmZlbgpGBmZgVOCmZmVuCkYGZmBU4KZmZW4KRgZmYFTgpmZlbgpGBmZgVOCmZmVuCkYGZmBU4KZmZW4KRgZmYFTgpmZlbgpGBmZgVOCmZmVuCkYGZmBU4KZmZW4KRgZmYF61xSkHSgpBckzZE0ptzxmJm1JetUUpDUHvgNcBCwM3CspJ3LG5WZWduxTiUFYBgwJyJeioiPgD8Dh5c5JjOzNkMRUe4YCiQdBRwYEaem6ROA3SPiW7llRgIj0+QOwAstHmh59ADeLHcQ1iD+zNYvbenz6hcRFbXN6NDSkTRVRFwDXFPuOFqapGkRUVXuOKx4/szWL/68Muta89F8oG9uuk8qMzOzFrCuJYWpwHaStpbUCTgGuKvMMZmZtRnrVPNRRKyU9C3gXqA9cH1EzCpzWOuKNtdk1gr4M1u/+PNiHetoNjOz8lrXmo/MzKyMnBTMzKzAScGsxCRNltTmT3W09YOTgpmZFTgprIMkHS/pCUkzJV0tqb2k9yT9WNJTkh6X1LPccdrqJFVKel7SjZJmS5ogacNyx2W1k/TDdPPNKZJulvQ9SZ+WdI+k6ZIelrRjueNsaU4K6xhJOwFfAfaKiMHAKuA4oCvweETsAjwEnFa+KK0eOwC/jYidgHeBb5Y5HquFpN2ALwG7kN2As7p57xrgrIgYCnwP+G15Iiyfdeo6BQNgP2AoMFUSQBfgDeAjYGJaZjqwf1mis7V5NSIeSc//FxhVzmCsTnsBd0bEcmC5pP8DOgN7An9J/3sAG5QpvrJxUlj3CBgfEd9frVD6XnxyUckq/Nmtq2pe+OMLgdYf7YB30hF6m+Xmo3XPg8BRkrYAkLSZpH5ljsmKt5WkPdLzrwJTyhmM1ekR4DBJnSVtBBwKvA+8LOnLAMrsUs4gy8FJYR0TEc8BFwD3SXoauB/oVd6orAFeAM6UNBvYFBhX5nisFhExley+ak8DfwOeAZaQ9d+dIukpYBZtcDwX3+bCrJlIqgQmRsSAModiRZC0UUS8l84QewgYGREzyh1Xubld2szaqmvScL+dyfrx2nxCAB8pmJlZjvsUzMyswEnBzMwKnBTMzKzAScHaJEnvlTsGs3WRk4KZmRU4KVibJmkjSQ9KmiHpGUmHp/LKdKfTayXNknSfpC5p3m6Snk53sb1M0rOp/CRJ/5Pb9kRJI9LzcZKmpW2NzS1zcLqz6nRJV0mamMq7Sro+3S33yVxc/XN30H1a0nYt9mZZm+CkYG3dcuCLETEE2Af4hT65G9p2wG8ioj/wDtldNQH+AHwjdxfbYvwgIqqAQcDekgZJ6gxcDRyU7spZkV8e+HtEDEtxXSapK3A6cGWquwqY17iXbVY7JwVr6wT8d7qlyAPAlkD1WBUvR8TM9Hw6UClpE6BbRDyWym8qsp6jJc0AngT6AzsDOwIvRcTLaZmbc8t/HhgjaSYwmewCq62Ax4DzJZ0H9IuIDxr0as3Wwlc0W1t3HNkv9KERsULSXLIvYIAPc8utIruNeX1WsvoPrc4AkrYmuzf/bhHxtqQbcnXURcCXIuKFGuWzJf0TOAS4W9I3IuLva9mWWdF8pGBtXXfgjZQQ9gHqvSNtRLwDLJW0eyo6Jjd7LjBYUjtJfYFhqXxjYBmwJI2Yd1AqfwHYJt0zCbLBlardC5xV3ZQladf0dxuyo4urgDvJmqPMmo2PFKytuxH4P0nPANOA54tY5xTgWkkfA/8gu7smZLdjfhl4DpgNzACIiKckPZm2/Wpajoj4QNI3gXskLQOm5ur4EXAF8LSkdmm7hwJHAydIWgG8Bvx3Y1+4WW187yOzBqq+u2Z6PgboFRHfbsq20hHBb4B/RcSvmjFcswZx85FZwx2STgl9FhgOXNqEbZ2WOpNnkTVlXd0cAZo1lo8UzMyswEcKZmZW4KRgZmYFTgpmZlbgpGBmZgVOCmZmVvD/ARtS26sGJ9YhAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"jSFzyBn09N6e"},"source":["As we can see, **even if a GPT2TokenizerFast trained with an English corpus can tokenize any text in any language, it was optimized for English**( chart cleearly shows that): the number of generated tokens is lower for an English text than for the same text in another language with an equivalent number of words.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"7Mc2hrVK9N6e"},"source":["## English vs Polish tokenizer on Polish Wikipedia"]},{"cell_type":"markdown","metadata":{"id":"bH6GRvyo9N6e"},"source":["In this paragraph, we will compare the number of tokens per Portuguese text generated respectively by the English and Portuguese tokenizers."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":380},"id":"trXJDPft9N6f","executionInfo":{"status":"ok","timestamp":1615920223797,"user_tz":-60,"elapsed":38993,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"ff924d1e-68ac-4a2e-fb63-4b9bbb2894d6"},"source":["lang = 'pl'\n","fname = f'all_texts_{lang}wiki.csv'\n","df = pd.read_csv(path_data/fname)\n","df.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Vanquish – gra konsolowa z gatunku third-person shooter, wyprodukowana przez PlatinumGames i wydana przez SEGA. Jej światowa premiera miała miejsce 19 października 2010 roku na konsolach PlayStation 3 i Xbox 360. 10 maja 2017 roku zapowiedziano wersję na komputery osobiste, której premiera odbędzie się 25 maja 2017 roku.\\n\\nAkcja gry rozgrywa się w okresie, w którym na Ziemi zaczyna brakować surowców i dwa największe mocarstwa (Stany Zjednoczone i Rosja) rozpoczynają o nie rywalizację. Pierwsi atakują Rosjanie dowodzeni przez Zajcewa. Przywódca agresorów stawia ultimatum: kapitulacja w cią...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Książka obrazkowa – specyficzna forma książki skierowana najczęściej do najmłodszych dzieci. W tego typu książkach tekstu jest niewiele lub nie ma go wcale. W publikacjach tych warstwa wizualna odgrywa nadrzędną rolę, a warstwa tekstowa jest w nich jedynie dodatkiem. Tekst powinien komponować się z grafiką, a nie odwrotnie, oderwany od ilustracji nie posiada bowiem większego znaczenia. Bez obrazu nie jest zrozumiały. Książkę obrazkową określa się mianem ewenementu natury edytorskiej. \\n\\nW polskiej bibliografii brakuje literatury przedmiotu poświęconej książce obrazkowej. W wielu źródłach ...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Gilles Simon (ur. 27 grudnia 1984 w Nicei) – francuski tenisista, reprezentant w Pucharze Davisa, olimpijczyk.\\n\\nKarierę tenisową Francuz rozpoczął w roku 2002. Pierwsze zwycięstwo w turniejach rangi \"ITF Futures\" odniósł w czerwcu 2003 roku w Lizbonie, a zawodach kategorii ATP Challenger Tour w styczniu 2005 roku w Numei.\\n\\nW turniejach rangi ATP World Tour pierwszy znaczący wynik Simon osiągnął w połowie kwietnia roku 2006, awansując do finału rozgrywek w Walencji. Finałowy pojedynek przegrał jednak w dwóch setach z Nicolásem Almagro.\\n\\nNa początku sezonu 2007 Simon wygrał po raz pier...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Wsiewołod Emilewicz Meyerhold, Mejerchold ros. Всеволод Эмильевич Мейерхольд (ur. w Penzie, zm. 2 lutego 1940 w Moskwie) – rosyjski reżyser, aktor teatralny, twórca nowatorskiego współczesnego teatru sowieckiego.\\n\\nPrzyszedł na świat jako ósme z kolei dziecko właściciela fabryki wódek i kupca, Emila Meyerholda, Niemca z Dolnego Śląska, osiadłego w Rosji. Na chrzcie (w kościele ewangelickim obrządku luterańskiego) otrzymał imiona Karl Theodor Casimir. Do 1895 roku miał obywatelstwo niemieckie. W 1895 roku ukończył II gimnazjum w Penzie. Rok wcześniej nie został dopuszczony do egzaminu dojr...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Alexandre-Édouard Kierzkowski właśc. Aleksander Edward Kierzkowski herbu Krzywda (ur. 21 listopada 1816 w Raczycach, zm. 4 sierpnia 1870) – kanadyjski żołnierz, inżynier i polityk polskiego pochodzenia.\\n\\nUrodził się w Raczycach koło Odolanowa (Wielkie Księstwo Poznańskie) jako syn Filipa Jakuba (napoleończyka, oficera wojsk polskich) i Marianny Ludwiki Liebermann. Został ochrzczony 20 października 1817 roku w Odolanowie. Wstąpił do Wojska Polskiego w czasie powstania listopadowego. Po klęsce wyemigrował do Francji, gdzie uzyskał w Paryżu w 1838 roku dyplom inżyniera na École Centrale des...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n","0  Vanquish – gra konsolowa z gatunku third-person shooter, wyprodukowana przez PlatinumGames i wydana przez SEGA. Jej światowa premiera miała miejsce 19 października 2010 roku na konsolach PlayStation 3 i Xbox 360. 10 maja 2017 roku zapowiedziano wersję na komputery osobiste, której premiera odbędzie się 25 maja 2017 roku.\\n\\nAkcja gry rozgrywa się w okresie, w którym na Ziemi zaczyna brakować surowców i dwa największe mocarstwa (Stany Zjednoczone i Rosja) rozpoczynają o nie rywalizację. Pierwsi atakują Rosjanie dowodzeni przez Zajcewa. Przywódca agresorów stawia ultimatum: kapitulacja w cią...\n","1  Książka obrazkowa – specyficzna forma książki skierowana najczęściej do najmłodszych dzieci. W tego typu książkach tekstu jest niewiele lub nie ma go wcale. W publikacjach tych warstwa wizualna odgrywa nadrzędną rolę, a warstwa tekstowa jest w nich jedynie dodatkiem. Tekst powinien komponować się z grafiką, a nie odwrotnie, oderwany od ilustracji nie posiada bowiem większego znaczenia. Bez obrazu nie jest zrozumiały. Książkę obrazkową określa się mianem ewenementu natury edytorskiej. \\n\\nW polskiej bibliografii brakuje literatury przedmiotu poświęconej książce obrazkowej. W wielu źródłach ...\n","2  Gilles Simon (ur. 27 grudnia 1984 w Nicei) – francuski tenisista, reprezentant w Pucharze Davisa, olimpijczyk.\\n\\nKarierę tenisową Francuz rozpoczął w roku 2002. Pierwsze zwycięstwo w turniejach rangi \"ITF Futures\" odniósł w czerwcu 2003 roku w Lizbonie, a zawodach kategorii ATP Challenger Tour w styczniu 2005 roku w Numei.\\n\\nW turniejach rangi ATP World Tour pierwszy znaczący wynik Simon osiągnął w połowie kwietnia roku 2006, awansując do finału rozgrywek w Walencji. Finałowy pojedynek przegrał jednak w dwóch setach z Nicolásem Almagro.\\n\\nNa początku sezonu 2007 Simon wygrał po raz pier...\n","3  Wsiewołod Emilewicz Meyerhold, Mejerchold ros. Всеволод Эмильевич Мейерхольд (ur. w Penzie, zm. 2 lutego 1940 w Moskwie) – rosyjski reżyser, aktor teatralny, twórca nowatorskiego współczesnego teatru sowieckiego.\\n\\nPrzyszedł na świat jako ósme z kolei dziecko właściciela fabryki wódek i kupca, Emila Meyerholda, Niemca z Dolnego Śląska, osiadłego w Rosji. Na chrzcie (w kościele ewangelickim obrządku luterańskiego) otrzymał imiona Karl Theodor Casimir. Do 1895 roku miał obywatelstwo niemieckie. W 1895 roku ukończył II gimnazjum w Penzie. Rok wcześniej nie został dopuszczony do egzaminu dojr...\n","4  Alexandre-Édouard Kierzkowski właśc. Aleksander Edward Kierzkowski herbu Krzywda (ur. 21 listopada 1816 w Raczycach, zm. 4 sierpnia 1870) – kanadyjski żołnierz, inżynier i polityk polskiego pochodzenia.\\n\\nUrodził się w Raczycach koło Odolanowa (Wielkie Księstwo Poznańskie) jako syn Filipa Jakuba (napoleończyka, oficera wojsk polskich) i Marianny Ludwiki Liebermann. Został ochrzczony 20 października 1817 roku w Odolanowie. Wstąpił do Wojska Polskiego w czasie powstania listopadowego. Po klęsce wyemigrował do Francji, gdzie uzyskał w Paryżu w 1838 roku dyplom inżyniera na École Centrale des..."]},"metadata":{"tags":[]},"execution_count":47}]},{"cell_type":"code","metadata":{"id":"eHaZU0-qXFZe"},"source":["from tqdm import tqdm\r\n","import pandas as pd"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"taRdaWH-avJp"},"source":["### size of the file and RAM on colab are two constrains which are  cousing crashes. solution-> take  fraction of wiki text"]},{"cell_type":"code","metadata":{"id":"bP_Kjsh_cask"},"source":["lang = 'pl'\r\n","name = f'{lang}wiki'\r\n","data_path = config['data_path']\r\n","path_data = data_path/name"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":380},"id":"jF4_wcomautC","executionInfo":{"status":"ok","timestamp":1615923090787,"user_tz":-60,"elapsed":38826,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"74193547-e1ee-449d-f1e8-030317640d1a"},"source":["fname = f'all_texts_{lang}wiki.csv'\r\n","df = pd.read_csv(path_data/fname)\r\n","df.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Vanquish – gra konsolowa z gatunku third-person shooter, wyprodukowana przez PlatinumGames i wydana przez SEGA. Jej światowa premiera miała miejsce 19 października 2010 roku na konsolach PlayStation 3 i Xbox 360. 10 maja 2017 roku zapowiedziano wersję na komputery osobiste, której premiera odbędzie się 25 maja 2017 roku.\\n\\nAkcja gry rozgrywa się w okresie, w którym na Ziemi zaczyna brakować surowców i dwa największe mocarstwa (Stany Zjednoczone i Rosja) rozpoczynają o nie rywalizację. Pierwsi atakują Rosjanie dowodzeni przez Zajcewa. Przywódca agresorów stawia ultimatum: kapitulacja w cią...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Książka obrazkowa – specyficzna forma książki skierowana najczęściej do najmłodszych dzieci. W tego typu książkach tekstu jest niewiele lub nie ma go wcale. W publikacjach tych warstwa wizualna odgrywa nadrzędną rolę, a warstwa tekstowa jest w nich jedynie dodatkiem. Tekst powinien komponować się z grafiką, a nie odwrotnie, oderwany od ilustracji nie posiada bowiem większego znaczenia. Bez obrazu nie jest zrozumiały. Książkę obrazkową określa się mianem ewenementu natury edytorskiej. \\n\\nW polskiej bibliografii brakuje literatury przedmiotu poświęconej książce obrazkowej. W wielu źródłach ...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Gilles Simon (ur. 27 grudnia 1984 w Nicei) – francuski tenisista, reprezentant w Pucharze Davisa, olimpijczyk.\\n\\nKarierę tenisową Francuz rozpoczął w roku 2002. Pierwsze zwycięstwo w turniejach rangi \"ITF Futures\" odniósł w czerwcu 2003 roku w Lizbonie, a zawodach kategorii ATP Challenger Tour w styczniu 2005 roku w Numei.\\n\\nW turniejach rangi ATP World Tour pierwszy znaczący wynik Simon osiągnął w połowie kwietnia roku 2006, awansując do finału rozgrywek w Walencji. Finałowy pojedynek przegrał jednak w dwóch setach z Nicolásem Almagro.\\n\\nNa początku sezonu 2007 Simon wygrał po raz pier...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Wsiewołod Emilewicz Meyerhold, Mejerchold ros. Всеволод Эмильевич Мейерхольд (ur. w Penzie, zm. 2 lutego 1940 w Moskwie) – rosyjski reżyser, aktor teatralny, twórca nowatorskiego współczesnego teatru sowieckiego.\\n\\nPrzyszedł na świat jako ósme z kolei dziecko właściciela fabryki wódek i kupca, Emila Meyerholda, Niemca z Dolnego Śląska, osiadłego w Rosji. Na chrzcie (w kościele ewangelickim obrządku luterańskiego) otrzymał imiona Karl Theodor Casimir. Do 1895 roku miał obywatelstwo niemieckie. W 1895 roku ukończył II gimnazjum w Penzie. Rok wcześniej nie został dopuszczony do egzaminu dojr...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Alexandre-Édouard Kierzkowski właśc. Aleksander Edward Kierzkowski herbu Krzywda (ur. 21 listopada 1816 w Raczycach, zm. 4 sierpnia 1870) – kanadyjski żołnierz, inżynier i polityk polskiego pochodzenia.\\n\\nUrodził się w Raczycach koło Odolanowa (Wielkie Księstwo Poznańskie) jako syn Filipa Jakuba (napoleończyka, oficera wojsk polskich) i Marianny Ludwiki Liebermann. Został ochrzczony 20 października 1817 roku w Odolanowie. Wstąpił do Wojska Polskiego w czasie powstania listopadowego. Po klęsce wyemigrował do Francji, gdzie uzyskał w Paryżu w 1838 roku dyplom inżyniera na École Centrale des...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n","0  Vanquish – gra konsolowa z gatunku third-person shooter, wyprodukowana przez PlatinumGames i wydana przez SEGA. Jej światowa premiera miała miejsce 19 października 2010 roku na konsolach PlayStation 3 i Xbox 360. 10 maja 2017 roku zapowiedziano wersję na komputery osobiste, której premiera odbędzie się 25 maja 2017 roku.\\n\\nAkcja gry rozgrywa się w okresie, w którym na Ziemi zaczyna brakować surowców i dwa największe mocarstwa (Stany Zjednoczone i Rosja) rozpoczynają o nie rywalizację. Pierwsi atakują Rosjanie dowodzeni przez Zajcewa. Przywódca agresorów stawia ultimatum: kapitulacja w cią...\n","1  Książka obrazkowa – specyficzna forma książki skierowana najczęściej do najmłodszych dzieci. W tego typu książkach tekstu jest niewiele lub nie ma go wcale. W publikacjach tych warstwa wizualna odgrywa nadrzędną rolę, a warstwa tekstowa jest w nich jedynie dodatkiem. Tekst powinien komponować się z grafiką, a nie odwrotnie, oderwany od ilustracji nie posiada bowiem większego znaczenia. Bez obrazu nie jest zrozumiały. Książkę obrazkową określa się mianem ewenementu natury edytorskiej. \\n\\nW polskiej bibliografii brakuje literatury przedmiotu poświęconej książce obrazkowej. W wielu źródłach ...\n","2  Gilles Simon (ur. 27 grudnia 1984 w Nicei) – francuski tenisista, reprezentant w Pucharze Davisa, olimpijczyk.\\n\\nKarierę tenisową Francuz rozpoczął w roku 2002. Pierwsze zwycięstwo w turniejach rangi \"ITF Futures\" odniósł w czerwcu 2003 roku w Lizbonie, a zawodach kategorii ATP Challenger Tour w styczniu 2005 roku w Numei.\\n\\nW turniejach rangi ATP World Tour pierwszy znaczący wynik Simon osiągnął w połowie kwietnia roku 2006, awansując do finału rozgrywek w Walencji. Finałowy pojedynek przegrał jednak w dwóch setach z Nicolásem Almagro.\\n\\nNa początku sezonu 2007 Simon wygrał po raz pier...\n","3  Wsiewołod Emilewicz Meyerhold, Mejerchold ros. Всеволод Эмильевич Мейерхольд (ur. w Penzie, zm. 2 lutego 1940 w Moskwie) – rosyjski reżyser, aktor teatralny, twórca nowatorskiego współczesnego teatru sowieckiego.\\n\\nPrzyszedł na świat jako ósme z kolei dziecko właściciela fabryki wódek i kupca, Emila Meyerholda, Niemca z Dolnego Śląska, osiadłego w Rosji. Na chrzcie (w kościele ewangelickim obrządku luterańskiego) otrzymał imiona Karl Theodor Casimir. Do 1895 roku miał obywatelstwo niemieckie. W 1895 roku ukończył II gimnazjum w Penzie. Rok wcześniej nie został dopuszczony do egzaminu dojr...\n","4  Alexandre-Édouard Kierzkowski właśc. Aleksander Edward Kierzkowski herbu Krzywda (ur. 21 listopada 1816 w Raczycach, zm. 4 sierpnia 1870) – kanadyjski żołnierz, inżynier i polityk polskiego pochodzenia.\\n\\nUrodził się w Raczycach koło Odolanowa (Wielkie Księstwo Poznańskie) jako syn Filipa Jakuba (napoleończyka, oficera wojsk polskich) i Marianny Ludwiki Liebermann. Został ochrzczony 20 października 1817 roku w Odolanowie. Wstąpił do Wojska Polskiego w czasie powstania listopadowego. Po klęsce wyemigrował do Francji, gdzie uzyskał w Paryżu w 1838 roku dyplom inżyniera na École Centrale des..."]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"T-R_kZj09N6f","executionInfo":{"status":"ok","timestamp":1615924418233,"user_tz":-60,"elapsed":424832,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"0d412b43-93ea-4cd1-f170-4bba5861798a"},"source":["%%time\n","df2= df.sample(frac=0.3) #wezmy 30% moze sie nie wywali\n","# df2 = df.copy() #generally this good  but without crash\n","\n","tokens_en_list = list()\n","num_token_by_word_en_list = list()\n","tokens_pl_list = list()\n","num_token_by_word_pl_list = list()\n","\n","for index, row in tqdm(df2.iterrows()):\n","    text = row['text']\n","    \n","    tokens_en = tokenizer_en.encode(text)\n","    tokens_pl = tokenizer_pl.encode(text)\n","    \n","    tokens_en_list.append(tokens_en)\n","    tokens_pl_list.append(tokens_pl)\n","    \n","    length_text = len(text.split())\n","    tokens_by_word_en = len(tokens_en)/length_text\n","    tokens_by_word_pl = len(tokens_pl)/length_text\n","    \n","    num_token_by_word_en_list.append(tokens_by_word_en)\n","    num_token_by_word_pl_list.append(tokens_by_word_pl)\n","    \n","df2['tokens_en'] = tokens_en_list\n","df2['num_token_by_word_en'] = num_token_by_word_en_list\n","df2['tokens_pl'] = tokens_pl_list\n","df2['num_token_by_word_pl'] = num_token_by_word_pl_list"],"execution_count":null,"outputs":[{"output_type":"stream","text":["68883it [06:54, 166.01it/s]\n"],"name":"stderr"},{"output_type":"stream","text":["CPU times: user 6min 59s, sys: 6.52 s, total: 7min 5s\n","Wall time: 7min 4s\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"qO7zS5tIaZ-f","executionInfo":{"status":"ok","timestamp":1615921282929,"user_tz":-60,"elapsed":572,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"65025281-1ff9-41e1-e6f2-ee580ce7da86"},"source":["ls /root/.fastai/data/plwiki/ByteLevelBPE_tokenizer_pl/ -all"],"execution_count":null,"outputs":[{"output_type":"stream","text":["total 1412\n","drwxr-xr-x 2 root root   4096 Mar 16 18:26 \u001b[0m\u001b[01;34m.\u001b[0m/\n","drwxr-xr-x 3 root root   4096 Mar 16 18:26 \u001b[01;34m..\u001b[0m/\n","-rw-r--r-- 1 root root 545015 Mar 16 18:26 merges.txt\n","-rw-r--r-- 1 root root 886662 Mar 16 18:26 vocab.json\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"id":"lWhnYsOf9N6f","executionInfo":{"status":"ok","timestamp":1615924427377,"user_tz":-60,"elapsed":690,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"40738186-3ce6-43ac-be25-ae8c88eb7648"},"source":["df2.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>tokens_en</th>\n","      <th>num_token_by_word_en</th>\n","      <th>tokens_pl</th>\n","      <th>num_token_by_word_pl</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>183907</th>\n","      <td>Katarzyna Ida Żukrowska (ur. 12 kwietnia 1949 we Wrocławiu) – polska ekonomistka i politolog, specjalista od międzynarodowych stosunków gospodarczych i politycznych, profesor nauk ekonomicznych.\\n\\nUzyskała tytuł zawodowy magistra ekonomii w 1973 r. w Szkole Głównej Planowania i Statystyki, stopień naukowy doktora w 1984 tamże, przyznała go Rada Wydziału Handlu Zagranicznego. Dysertację habilitacyjną na temat \"Broń na sprzedaż. Kompleks militarno-przemysłowy a zbrojenia\" obroniła w 1989, stopień doktora habilitowanego nauk humanistycznych w zakresie nauk o polityce, specjalność: nauki o po...</td>\n","      <td>[42, 9459, 89, 46434, 5121, 64, 25370, 119, 2724, 8516, 4914, 357, 333, 13, 1105, 479, 86, 1155, 18142, 24977, 356, 370, 12204, 41615, 707, 16115, 8, 784, 755, 82, 4914, 304, 74, 6326, 396, 4914, 1312, 1408, 928, 11, 1020, 73, 282, 12523, 16298, 21504, 128, 247, 67, 89, 2047, 283, 375, 322, 88, 354, 336, 418, 2954, 10205, 86, 308, 2117, 375, 5605, 7357, 354, 1312, 755, 414, 26691, 3281, 354, 11, 1534, 274, 273, 299, 559, 74, 304, 74, 6326, 28051, 3281, 354, 13, 198, 198, 52, 89, 893, 4914, 41615, 64, 1259, 28047, 41615, 1976, 707, ...]</td>\n","      <td>3.803571</td>\n","      <td>[43, 6204, 264, 42734, 26703, 2757, 510, 343, 487, 14, 1104, 1314, 4643, 767, 5029, 9, 441, 5709, 4595, 4038, 286, 31631, 2743, 12, 19722, 365, 7135, 11307, 13211, 286, 5839, 12, 5379, 1299, 15910, 14, 199, 199, 53, 395, 22257, 1637, 20114, 10188, 16308, 260, 4542, 465, 14, 260, 5247, 10327, 28680, 286, 23760, 7837, 12, 4073, 9206, 4102, 260, 4472, 10629, 12, 19665, 599, 6725, 3940, 20417, 38728, 14, 1701, 1775, 7008, 25461, 3516, 289, 3398, 389, 21878, 337, 289, 10448, 14, 43454, 9612, 344, 13, 44227, 369, 334, 3173, 16056, 2, 30427, 260, 3409, 12, 4073, 4102, ...]</td>\n","      <td>1.541667</td>\n","    </tr>\n","    <tr>\n","      <th>185112</th>\n","      <td>Alberto Ascari (ur. 13 lipca 1918 w Mediolanie, zm. 26 maja 1955 na Autodromo Nazionale di Monza) – włoski kierowca wyścigowy, dwukrotny mistrz Formuły 1.\\n\\nUrodził się w Mediolanie we Włoszech, a wyścigiami zainteresował się dzięki ojcu Antonio Ascari, kierowcy wyścigów Grand Prix w latach 20. XX wieku jako kierowca Alfa Romeo. Antonio zmarł podczas Grand Prix Francji w 1925 roku.\\n\\nJako młody chłopak brał udział w wyścigach motocyklowych; było to po tym jak wstąpił do prestiżowego Mille Miglia sportów samochodowych Ferrari gdzie w końcu przeszedł do wyścigów samochodów.\\n\\nJego karierę...</td>\n","      <td>[2348, 32371, 29469, 2743, 357, 333, 13, 1511, 10645, 6888, 25859, 266, 2019, 1669, 34166, 11, 1976, 76, 13, 2608, 16486, 64, 25325, 12385, 5231, 375, 398, 78, 12819, 295, 1000, 2566, 2892, 4496, 8, 784, 266, 41615, 418, 4106, 479, 959, 322, 6888, 266, 88, 129, 249, 22683, 322, 88, 11, 43756, 2724, 10599, 3281, 4020, 81, 89, 5178, 84, 41615, 88, 352, 13, 198, 198, 52, 14892, 17027, 41615, 33721, 128, 247, 266, 2019, 1669, 34166, 356, 370, 41615, 418, 15356, 11, 257, 266, 88, 129, 249, 22683, 7871, 1976, 391, 660, 411, 8455, 41615, 33721, 128, 247, ...]</td>\n","      <td>3.097610</td>\n","      <td>[5219, 33077, 5860, 2771, 73, 343, 487, 14, 1189, 1276, 3049, 260, 21660, 12, 994, 14, 2197, 1228, 5465, 289, 11265, 3211, 525, 513, 14517, 298, 3750, 3602, 444, 9, 441, 10376, 15572, 31914, 12, 24845, 3996, 12503, 301, 14, 199, 199, 4678, 332, 260, 21660, 767, 7158, 12, 334, 3330, 3870, 308, 20109, 332, 2022, 17320, 13230, 5860, 2771, 73, 12, 13930, 16456, 5365, 6167, 260, 658, 412, 14, 1617, 868, 566, 15572, 31509, 41341, 14, 13230, 3167, 954, 5365, 6167, 2334, 260, 4866, 388, 14, 199, 199, 7206, 4499, 30790, 4230, 1065, 260, 16115, 30408, 518, 27, 939, ...]</td>\n","      <td>1.370518</td>\n","    </tr>\n","    <tr>\n","      <th>203374</th>\n","      <td>16 000 000 beli po sadze Eines Lobby\\n\\nNami na lewym ramieniu ma tatuaż przedstawiający pomarańczę oraz wiatraczek (są to symbole związane z jej przeszłością), dawniej w tym samym miejscu miała tatuaż pirackiej załogi Arlonga.\\n\\nPoczątkowo Nami nosiła krótkie skórzane botki i zwykły t-shirt, jednakże po sadze Arlong Park, zmienia nieco sposób ubierania – zaczyna nosi sandały i nieco bardziej odsłonięte i różnorodne stroje (tatuaż przestaje być tajemnicą). Od rozdziału 435 zaczyna nosić wysokie kozaki. Jej zwykły strój stanowi spódnica (często z kółkami) oraz różne t-shirty, często posiad...</td>\n","      <td>[1433, 12877, 12877, 894, 72, 745, 6507, 2736, 412, 1127, 35068, 198, 198, 45, 6277, 12385, 443, 86, 4948, 15770, 2013, 16115, 17266, 256, 265, 6413, 129, 120, 778, 8863, 301, 707, 544, 73, 128, 227, 948, 279, 296, 3301, 129, 226, 26691, 128, 247, 393, 1031, 266, 5375, 11510, 43130, 357, 82, 128, 227, 284, 5307, 2305, 1976, 37686, 128, 227, 89, 1531, 1976, 11223, 73, 778, 12271, 89, 41615, 78, 129, 249, 979, 128, 227, 828, 17577, 494, 73, 266, 1259, 76, 6072, 4948, 285, 494, 73, 1416, 84, 285, 544, 41615, 64, 256, 265, 6413, 129, 120, ...]</td>\n","      <td>3.594168</td>\n","      <td>[1890, 3889, 3889, 304, 4089, 282, 2127, 578, 18988, 2278, 432, 30669, 199, 199, 1025, 308, 289, 12841, 34169, 511, 287, 48728, 279, 17996, 282, 8262, 337, 758, 478, 45463, 3776, 343, 7517, 524, 25950, 4051, 263, 792, 7977, 268, 523, 11338, 260, 592, 1469, 1900, 1776, 287, 48728, 279, 18939, 1058, 6294, 840, 5859, 439, 14, 199, 199, 10363, 38100, 15673, 13676, 29966, 283, 304, 742, 335, 286, 46999, 287, 13, 9840, 40029, 12, 6638, 282, 2127, 578, 840, 76, 3247, 5993, 12, 5596, 4766, 1912, 45521, 316, 441, 5402, 4862, 47795, 543, 286, 4766, 2246, 10764, 394, 286, ...]</td>\n","      <td>1.611179</td>\n","    </tr>\n","    <tr>\n","      <th>182910</th>\n","      <td>Klawesyn (wł. \"cembalo\", \"clavicembalo\", skrót: \"cemb\".; fr. \"clavecin\"; ang. \"harpsichord\") – znany powszechnie w Europie od XV wieku instrument strunowy szarpany, wyposażony w klawiaturę, zaliczany do rodziny cytr.\\n\\nW instrumencie tym struny zostają wzbudzone poprzez mechanizm skoczków zaopatrzonych w tzw. „piórka”, wykonane z dutki piór ptasich, twardej skóry lub współcześnie z tworzyw sztucznych. „Piórka” wprawione w ruch za pomocą mechanizmu klawiszowego szarpią struny naciągnięte nad płytą rezonansową. W typowej formie wykształconej w XVI wieku klawesyn miał najczęściej co najmniej...</td>\n","      <td>[42, 6270, 274, 2047, 357, 86, 41615, 13, 366, 344, 2022, 7335, 1600, 366, 565, 615, 291, 368, 6893, 78, 1600, 1341, 81, 10205, 83, 25, 366, 344, 2022, 1911, 26, 1216, 13, 366, 44281, 17879, 8172, 3550, 13, 366, 9869, 862, 488, 585, 4943, 784, 1976, 77, 1092, 279, 1666, 2736, 1349, 494, 266, 1898, 21749, 16298, 29922, 266, 494, 23063, 8875, 965, 403, 322, 88, 264, 89, 5117, 1092, 11, 266, 88, 1930, 64, 129, 120, 1647, 266, 479, 6270, 72, 2541, 128, 247, 11, 1976, 282, 28051, 1092, 466, 15299, 89, 3541, 3075, 2213, 13, 198, 198, ...]</td>\n","      <td>3.802564</td>\n","      <td>[26013, 784, 12224, 343, 26484, 14, 389, 2295, 66, 2749, 682, 389, 21518, 1335, 2295, 66, 2749, 682, 24026, 26, 389, 2295, 66, 713, 27, 32934, 14, 389, 21518, 1502, 3796, 7819, 10436, 14, 389, 5007, 28258, 266, 1697, 1325, 441, 4203, 7962, 260, 3554, 365, 1507, 868, 28918, 2349, 13346, 13163, 9037, 12, 9806, 260, 943, 4832, 6810, 12, 21000, 306, 2153, 1339, 1569, 14, 199, 199, 55, 3534, 3367, 592, 2349, 281, 13435, 9500, 2666, 2477, 17676, 36270, 6324, 11414, 260, 2103, 14, 476, 430, 14217, 996, 7624, 263, 275, 562, 335, 20701, 3861, 40689, 12, 34047, 354, ...]</td>\n","      <td>1.615385</td>\n","    </tr>\n","    <tr>\n","      <th>228543</th>\n","      <td>Janusz Tymowski ps. \"Ernest, Jaworski, Skierko\" (ur. 1 września 1902 w Lublinie, zm. 23 maja 1992 w Warszawie) – porucznik czasu wojny, polski inżynier mechanik, w latach 1940–1943 szef przemysłu wojennego AK, uczestnik powstania warszawskiego (1944), od 1955 profesor Politechniki Warszawskiej.\\n\\nNajstarszy z czworga dzieci Józefa i Jadwigi z Truchlińskich Tymowskich. Jego ojciec, inżynier leśnik i geodeta, zmarł w 1911. \\n\\nW maju 1920 Janusz ukończył Gimnazjum im. Stefana Batorego w Lublinie.\\n\\nW 1920 służył w 7 pułku ułanów jako ochotnik. Dyplom inżyniera uzyskał na Wydziale Mechanicz...</td>\n","      <td>[12128, 385, 89, 309, 4948, 12079, 26692, 13, 366, 9139, 77, 395, 11, 40863, 669, 4106, 11, 3661, 959, 7204, 1, 357, 333, 13, 352, 1319, 2736, 129, 249, 18142, 45611, 266, 40753, 2815, 494, 11, 1976, 76, 13, 2242, 16486, 64, 9768, 266, 6176, 89, 707, 494, 8, 784, 16964, 1229, 89, 17187, 24785, 27345, 266, 13210, 3281, 11, 755, 20545, 287, 129, 120, 2047, 959, 3962, 1134, 11, 266, 3042, 620, 16236, 1906, 1129, 3559, 264, 89, 891, 778, 89, 368, 893, 41615, 84, 266, 13210, 29727, 2188, 15837, 11, 334, 26691, 395, 17187, 7182, 301, 5411, 9976, ...]</td>\n","      <td>3.383585</td>\n","      <td>[38217, 4389, 1615, 7184, 14, 389, 22878, 283, 262, 12, 21529, 368, 12, 21781, 327, 2, 343, 487, 14, 301, 1146, 9365, 260, 7930, 12, 994, 14, 2025, 1228, 3259, 260, 1561, 9, 441, 15198, 2260, 1096, 12, 2289, 8544, 36692, 12, 260, 658, 2986, 659, 19912, 3675, 7064, 13397, 6266, 12, 13340, 2938, 8782, 343, 13799, 523, 365, 5465, 5379, 7514, 8808, 14, 199, 199, 23811, 418, 263, 363, 3760, 439, 1871, 3681, 286, 16559, 263, 8459, 266, 40132, 4389, 5031, 14, 1638, 4373, 12, 8544, 6664, 700, 286, 24222, 360, 12, 3167, 260, 7083, 14, 800, 199, 55, ...]</td>\n","      <td>1.463987</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           text  ... num_token_by_word_pl\n","183907  Katarzyna Ida Żukrowska (ur. 12 kwietnia 1949 we Wrocławiu) – polska ekonomistka i politolog, specjalista od międzynarodowych stosunków gospodarczych i politycznych, profesor nauk ekonomicznych.\\n\\nUzyskała tytuł zawodowy magistra ekonomii w 1973 r. w Szkole Głównej Planowania i Statystyki, stopień naukowy doktora w 1984 tamże, przyznała go Rada Wydziału Handlu Zagranicznego. Dysertację habilitacyjną na temat \"Broń na sprzedaż. Kompleks militarno-przemysłowy a zbrojenia\" obroniła w 1989, stopień doktora habilitowanego nauk humanistycznych w zakresie nauk o polityce, specjalność: nauki o po...  ...             1.541667\n","185112  Alberto Ascari (ur. 13 lipca 1918 w Mediolanie, zm. 26 maja 1955 na Autodromo Nazionale di Monza) – włoski kierowca wyścigowy, dwukrotny mistrz Formuły 1.\\n\\nUrodził się w Mediolanie we Włoszech, a wyścigiami zainteresował się dzięki ojcu Antonio Ascari, kierowcy wyścigów Grand Prix w latach 20. XX wieku jako kierowca Alfa Romeo. Antonio zmarł podczas Grand Prix Francji w 1925 roku.\\n\\nJako młody chłopak brał udział w wyścigach motocyklowych; było to po tym jak wstąpił do prestiżowego Mille Miglia sportów samochodowych Ferrari gdzie w końcu przeszedł do wyścigów samochodów.\\n\\nJego karierę...  ...             1.370518\n","203374  16 000 000 beli po sadze Eines Lobby\\n\\nNami na lewym ramieniu ma tatuaż przedstawiający pomarańczę oraz wiatraczek (są to symbole związane z jej przeszłością), dawniej w tym samym miejscu miała tatuaż pirackiej załogi Arlonga.\\n\\nPoczątkowo Nami nosiła krótkie skórzane botki i zwykły t-shirt, jednakże po sadze Arlong Park, zmienia nieco sposób ubierania – zaczyna nosi sandały i nieco bardziej odsłonięte i różnorodne stroje (tatuaż przestaje być tajemnicą). Od rozdziału 435 zaczyna nosić wysokie kozaki. Jej zwykły strój stanowi spódnica (często z kółkami) oraz różne t-shirty, często posiad...  ...             1.611179\n","182910  Klawesyn (wł. \"cembalo\", \"clavicembalo\", skrót: \"cemb\".; fr. \"clavecin\"; ang. \"harpsichord\") – znany powszechnie w Europie od XV wieku instrument strunowy szarpany, wyposażony w klawiaturę, zaliczany do rodziny cytr.\\n\\nW instrumencie tym struny zostają wzbudzone poprzez mechanizm skoczków zaopatrzonych w tzw. „piórka”, wykonane z dutki piór ptasich, twardej skóry lub współcześnie z tworzyw sztucznych. „Piórka” wprawione w ruch za pomocą mechanizmu klawiszowego szarpią struny naciągnięte nad płytą rezonansową. W typowej formie wykształconej w XVI wieku klawesyn miał najczęściej co najmniej...  ...             1.615385\n","228543  Janusz Tymowski ps. \"Ernest, Jaworski, Skierko\" (ur. 1 września 1902 w Lublinie, zm. 23 maja 1992 w Warszawie) – porucznik czasu wojny, polski inżynier mechanik, w latach 1940–1943 szef przemysłu wojennego AK, uczestnik powstania warszawskiego (1944), od 1955 profesor Politechniki Warszawskiej.\\n\\nNajstarszy z czworga dzieci Józefa i Jadwigi z Truchlińskich Tymowskich. Jego ojciec, inżynier leśnik i geodeta, zmarł w 1911. \\n\\nW maju 1920 Janusz ukończył Gimnazjum im. Stefana Batorego w Lublinie.\\n\\nW 1920 służył w 7 pułku ułanów jako ochotnik. Dyplom inżyniera uzyskał na Wydziale Mechanicz...  ...             1.463987\n","\n","[5 rows x 5 columns]"]},"metadata":{"tags":[]},"execution_count":20}]},{"cell_type":"code","metadata":{"id":"VgJV93l99N6g"},"source":["# save\n","fname = \"comparaison_tokenizers_en_pl_with_corpus_pl.csv\"\n","df2.to_csv(path_data/fname, index=False)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"-NsxfQiD9N6g"},"source":["# load\n","fname = \"comparaison_tokenizers_en_pl_with_corpus_pl.csv\"\n","df2 = pd.read_csv(path_data/fname)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"U5ws4G5c9N6g","executionInfo":{"status":"ok","timestamp":1615924562562,"user_tz":-60,"elapsed":730,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"ff3a60dd-9b4d-4ab3-f589-596fb4d79fd2"},"source":["# check min\n","num_token_by_word_en_min = df2.num_token_by_word_en.min()\n","num_token_by_word_pl_min = df2.num_token_by_word_pl.min()\n","print('(en)',round(num_token_by_word_en_min,2))\n","print('(pl)',round(num_token_by_word_pl_min,2))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en) 1.6\n","(pl) 1.17\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"_HZY_drE9N6g","executionInfo":{"status":"ok","timestamp":1615924580513,"user_tz":-60,"elapsed":444,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"e7505b80-569f-4d45-b2d2-8343054f0662"},"source":["# check max\n","num_token_by_word_en_max = df2.num_token_by_word_en.max()\n","num_token_by_word_pl_max = df2.num_token_by_word_pl.max()\n","print('(en)',round(num_token_by_word_en_max,2))\n","print('(pl)',round(num_token_by_word_pl_max,2))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en) 5.59\n","(pl) 4.09\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2WmaM7JR9N6h","executionInfo":{"status":"ok","timestamp":1615924589618,"user_tz":-60,"elapsed":736,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"d5535fff-1e06-492b-b60e-ff36c7572c01"},"source":["# check mean\n","num_token_by_word_en_mean = df2.num_token_by_word_en.mean()\n","num_token_by_word_pl_mean = df2.num_token_by_word_pl.mean()\n","print('(en)',round(num_token_by_word_en_mean,2))\n","print('(pl)',round(num_token_by_word_pl_mean,2))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(en) 3.59\n","(pl) 1.55\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"NYTWBzjG9N6h","executionInfo":{"status":"ok","timestamp":1615924613401,"user_tz":-60,"elapsed":678,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"deaaa75f-7e66-43c5-ce58-b2a22f05092a"},"source":["# check increase rate and Multiplier coefficient\n","increase = 0.\n","multiplier = 0.\n","\n","for tok_en,tok_pl in zip(*(tokens_en_list,tokens_pl_list)):\n","    increase += (len(tok_en)-len(tok_pl))/len(tok_pl)\n","    multiplier += len(tok_en)/len(tok_pl)\n","    \n","# Rate of increase in % from pt to en\n","increase_pct = increase / len(tokens_en_list)\n","print('Rate of increase:',round(increase_pct*100,2),'%')\n","\n","# Multiplier coefficient = (Rate of increase in %, converted to number) + 1\n","multiplier_coef = round(increase_pct+1,2)\n","print('Multiplier coefficient:',multiplier_coef)\n","\n","# Multiplier coefficient in % = Multiplier coefficient, converted to %\n","multiplier_pct = round((multiplier/len(tokens_en_list))*100,2)\n","print('Multiplier coefficient in %:',multiplier_pct,'%')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Rate of increase: 133.69 %\n","Multiplier coefficient: 2.34\n","Multiplier coefficient in %: 233.69 %\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dwLj27DJ9N6h"},"source":["### Graph"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"HX4b8O3B9N6h","executionInfo":{"status":"ok","timestamp":1615924642418,"user_tz":-60,"elapsed":9554,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"ab7d645f-915e-4a4c-de13-c879abff18f9"},"source":["%%time\n","len_tokens_text_list = list()\n","for index, row in df2.iterrows():\n","    text = row['text']\n","    length_text = len(text.split())\n","    len_tokens_text_list.append(length_text)\n","\n","tokens_en_list = df2.tokens_en.tolist()\n","len_tokens_en_list = [len(t) for t in tokens_en_list]\n","\n","tokens_pl_list = df2.tokens_pl.tolist()\n","len_tokens_pl_list = [len(t) for t in tokens_pl_list]"],"execution_count":null,"outputs":[{"output_type":"stream","text":["CPU times: user 8.58 s, sys: 269 ms, total: 8.85 s\n","Wall time: 8.86 s\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":295},"id":"L8KSP3of9N6i","executionInfo":{"status":"ok","timestamp":1615924679842,"user_tz":-60,"elapsed":3805,"user":{"displayName":"Mark Lina","photoUrl":"","userId":"17651129667533642938"}},"outputId":"970851e0-ed32-4368-aa4b-f6ac0f94751a"},"source":["sorted_len_tokens_text_list = sorted(len_tokens_text_list)\n","y_len_tokens_en_list = (12*np.array(sorted_len_tokens_text_list)).tolist() \n","y_len_tokens_pl_list = (7*np.array(sorted_len_tokens_text_list)).tolist()                             \n","\n","ax = plt.subplot(111)\n","ax.scatter(len_tokens_text_list, len_tokens_en_list)\n","ax.plot(sorted_len_tokens_text_list, y_len_tokens_en_list)\n","ax.scatter(len_tokens_text_list, len_tokens_pl_list)\n","ax.plot(sorted_len_tokens_text_list, y_len_tokens_pl_list)\n","\n","ax.set_xlabel('length of texts')\n","ax.set_ylabel('length of en and pt tokens')\n","ax.legend(['en', 'pl'])\n","\n","ax.set_title('Number of tokens by tokenization method')\n","\n","plt.show()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEWCAYAAAC5XZqEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3yUVdbA8d8hhN6rQGgivQoRUHDFhoAFlLW7Ytd13fVdXQUU14Yrrq6+dldfCywgohRRYRUR7EgRSKgSekInEFpCypz3j3ujY0yZQCaTcr6fz3wyc592Jpnk5N7nPucRVcUYY4wpThUiHYAxxpjyx5KPMcaYYmfJxxhjTLGz5GOMMabYWfIxxhhT7Cz5GGOMKXaWfEyeROQdERkboWOLiLwtIvtFZFER7O8GEfmmKGI7zuM/IiITI3TszSJyXhHv80wRWVeU+/T7vVZEPivq/YZbUX6+RGSAiCQWxb5KMks+pYj/I7JbRKoHtd0iIgsiGFa49AfOB2JUtXfOhZFOJpEUjmRSWKr6taq2P5F9iEgrEVERqRi030mqOvDEIwyf3OI2hWfJp/SJAu6OdBCFJSJRhdykJbBZVY+EIx5jTGRZ8il9ngb+JiJ1ci7I7T8yEVkgIrf45zeIyLci8pyIHBCRjSJyhm/f5ntVI3LstoGIzBWRQyLypYi0DNp3B78sWUTWicgVQcveEZFXRWS2iBwBzs4l3qYiMstvnyAit/r2m4H/A04XkcMi8miO7ToCrwUtP+Dba4vIBBHZIyJbRGSMiOT6GReRp0XkG79NbRF5U0R2iEiSiIzNTpbZPSwRecYPAW4SkcFB+7nBfx8P+WXX5vFzA6giIu/5dX8Uke5+H/eJyLQc8b0gIs/nEvd/gBbAR/693+/bLxGRVf7nusB/j3J73x19nFf71xeJyHK/3Xci0i1o3c0i8jcRiRORFB97Fb/s56EhEbnSx5L9OJbdGxeRC0VkmYgc9J+xR4LC+cp/PeC3Oz1nj9Z/Phf74y8WkTOCli0Qkcf9Z/qQiHwmIg3yeN8DRCRRRO73n/MdIjJMRIaIyE/+M/hA0PoVRGSUiGwQkX0iMlVE6uUVd9B2eX1Ocv2s+2VV/e/LfhFZDZyW23soc1TVHqXkAWwGzgOmA2N92y3AAv+8FaBAxaBtFgC3+Oc3AJnAjbge1FhgK/AyUBkYCBwCavj13/Gvf+eXPw9845dVB7b5fVUETgX2Ap2Ctk0B+uH+yamSy/v5CngFqAL0APYA5wTF+k0+34vfLAcmAB8CNf334ifg5uD1fSxvAJ8C1fyyGcC//XtqBCwCbg/aLgO41X/P/ghsB8SvfxBo79dtAnTOI95H/H5+D0QDfwM2+edNgCNAHb9uRWA30Cu/z0HQ63Z++/P9/u4HEoBKOT43Pf3P+yLffqo/Th//3kb4dSsHbbcIaArUA9YAd/hlA4DEXGKr5de7PWi9rv773g3YBQzL5/P688/VH3M/8Af/Pbnav64f9Nne4N9/Vf96XB7fswG4z/7f/ffoVtznbTLu89IZSAVa+/XvBhYCMbjP/r+BdwuIO9fPSQif9XHA1/79NgdW5va9LWuPiAdgj0L8sH75I9IF94e9IYVPPuuDlnX16zcOatsH9PDP3wGmBC2rAWT5X5Arga9zxPdv4OGgbSfk816a+33VDGp7EngnKNaQk4//hU/HJz/fdnvQ9+YG4AfgPWAav/xhbgwcA6oGbXc1MD9ou4SgZdX89+wkXPI5AAwP3j6PeB8BFga9rgDsAM70r+cAt/rnFwGrC/ocBL1+CJiaY99JwICg9R8FErPbfPurwOM59r0OOCtou+uClv0TeM0/H0COP5D+uB8Dr+YT+/8Cz+Xzef3554pLOotybP89cEPQZ3tM0LI7gf/mcdwBuOQS5V/X9MfuE7TOUn5JjGuAc4OWNcEll4r5xJ3X56Sgz/pGYFDQsttyfm/L4sOG3UohVV2J+yUfdRyb7wp6nur3l7OtRtDrbUHHPQwk4/4Tbgn08cM1B/zQ17W4X7bfbJuLpkCyqh4KatsCNCvEewnWAPcf7ZZ89ncKMBR4VFXTfVtLv92OoPfxb1wPKNvO7CeqetQ/raHufNSVwB1++09EpEM+MQZ/LwO4ZNDUN40HrvPPrwP+k//b/ZWmBL1vv+9t/Pq93wF8p6oLgtpaAvfm+Bk2D4oJgt47cJRffzZyegL3R/0v2Q0i0kdE5vuh0BQfR65DYwW9Ly/nz7Qw8e1T1Sz/PNV/zeuz3xKYEfR9WYNLII3z2X+unxMK/qw35de/Kznfc5lkyaf0ehjXxQ/+Rcw+OV8tqC04GRyP5tlPRKQGbmhgO+6X5UtVrRP0qKGqfwzaNr+S6duBeiJSM6itBe4/9lDk3Pde3H+mLYPacu5vDW6YcI6IZM/U2obr+TQIeh+1VLVzSEGofqqq5+P+M16LG9LLS/D3sgJuSGe7b5oJdBORLriez6T8Dpvj9XaC3reIiD9W8Hu/A2ghIs8FtW0DnsjxM6ymqu/mc+xcichVuB7j71U1I2jRZGAW0FxVa+PO1Uke7yOnX70vrzCfkROxDRic43tTRVWTKDjunAr6rO8g6LPhl5V5lnxKKVVNwA0h/SWobQ/uA32diESJyE1AmxM81BAR6S8ilYDHcUNH23A9r3Yi8gcRifaP0/I60Z1L/NuA74AnRaSKP9F9MxDqtTC7gBgfF/4/2qnAEyJSU9zEiHty7s//YX0A+FxE2qjqDuAz4F8iUsufaG4jImcVFICINBaRoeKmvh8DDgOBfDbpJSKXiZsQ8j9+m4U+rjTgA9wf60WqurWA935y0OupwIUicq6IRAP3+n1/F7TOIWAQ8DsRGefb3gDu8L0TEZHqfoJA8B/JAonIqcCLuCGrPTkW18T9158mIr2Ba4KW7cF9v04md7Nxn7FrRKSiiFwJdMJ99sLtNdxnqSWAiDQUkaF+WUFx/0oIn/WpwGgRqSsiMcCfi/B9lFiWfEq3x3DnHYLdCtyHO3fTmV//AToek3G9rGSgF35oyA8hDASuwv1ntxN4CndyNlRX48bPt+NO+j+sqp+HuO0XwCpgp4js9W1/xvX+NuImF0wG3sq5oaqOx33vvhCRVsD1QCVgNe6E9ge4nkxBKuAS3Hbc9+cs3InmvHyIG6bLPol+WY5ewnjcebiChtyeBMb4IaG/qeo63M/lRVwP8GLg4qChRQBU9QBuUsJgEXlcVZfgPi8v+ZgScOcuCmsoUBf4Rn6Z8TbHL7sTeExEDuFO9k8NiucobqjuW/9e+uaIdx+uF3gv7vN8P26yxF7C73lcj+0zH/tC3MSMAuPOQ36f9UdxQ22bcP8IFWbItdTKnolhjIkwEWmBG7o7SVUPRjoeY8LJej7GlAD+HNA9uNmFlnhMmWflIYyJMH/OaBdu6GVQhMMxpljYsJsxxphiZ8Nuxhhjip0Nu3kNGjTQVq1aRToMY4wpVZYuXbpXVRsWdjtLPl6rVq1YsmRJpMMwxphSRUSOqyKDDbsZY4wpdpZ8jDHGFDtLPsYYY4qdnfPJR0ZGBomJiaSlpUU6lEKrUqUKMTExREdHRzoUY4z5DUs++UhMTKRmzZq0atUKVyi4dFBV9u3bR2JiIq1bt450OMYY8xs27JaPtLQ06tevX6oSD4CIUL9+/VLZYzPGlA9hSz6+dPgiEVkh7t7yj/r2d/z9zZf7Rw/fLuLuW58g7p7xPYP2NUJE1vvHiKD2XiIS77d5wd/HBBGpJyJz/fpzRaTuCbyP4/8mRFBpjdsYUz6Es+dzDHeP8u64e5YPCio9fp+q9vCP5b5tMNDWP27D3eIXEamHK+nfB+gNPByUTF7FlYTP3i67LtYoYJ6qtgXmcXx3/DTGmDJt/5F0Hpm1ioNpGQWvXMTClnzUOexfRvtHfoXkhgIT/HYLgToi0gS4AJirqsmquh+Yi0tkTYBaqrpQXYG6CcCwoH2N98/HB7UbY0y5p6p8tGI75z37JRMXbmHRxuRijyGs53z83TSXA7txCeQHv+gJP7T2nIhk33ysGb++j3mib8uvPTGXdoDG/g6V4G5ylut910XkNhFZIiJL9uzJeQNGY4wpe3ampHHrhKX8+d1lNKtblY/+3J/zOuX6JzKswpp8VDVLVXvg7lXf29+ffjTQATgNqAeMDHMMSh49LlV9XVVjVTW2YcNClyYqFhMnTqR379706NGD22+/naysLGrUqMGDDz5I9+7d6du3L7t27Yp0mMaYEi4QUCb/sJXzn/2SbxL28OCQjkz/4xl0bFIrIvEUy1RrVT0gIvOBQar6jG8+JiJvA3/zr5OA5kGbxfi2JGBAjvYFvj0ml/UBdolIE1Xd4Yfndp/oe3j0o1Ws3l609/jq1LQWD1/cOc/la9as4b333uPbb78lOjqaO++8k0mTJnHkyBH69u3LE088wf33388bb7zBmDFjijQ2Y0zZsWnvEUZNi+OHTcmcfnJ9xg3vSsv61SMaUzhnuzUUkTr+eVXcvePX+mSAn5k2DFjpN5kFXO9nvfUFUvzQ2afAQBGp6ycaDAQ+9csOikhfv6/rgQ+D9pU9K25EUHupMm/ePJYuXcppp51Gjx49mDdvHhs3bqRSpUpcdNFFAPTq1YvNmzdHNlBjTImUmRXgtS83MOh/v2L1joM8Nbwrk2/tE/HEA+Ht+TQBxotIFC7JTVXVj0XkCxFpCAiwHLjDrz8bGAIkAEeBGwFUNVlEHgcW+/UeU9Xss2N3Au8AVYE5/gEwDpgqIjfj7g55xYm+mfx6KOGiqowYMYInn3zyV+3PPPPMz1Opo6KiyMzMLPbYjDEl26rtKYycFsfKpIMM7NSYx4d1oXGtKpEO62dhSz6qGgecmkv7OXmsr8Cf8lj2FvBWLu1LgC65tO8Dzi1kyCXOueeey9ChQ/nrX/9Ko0aNSE5O5tChQ5EOyxhTgqVlZPHiF+t57cuN1K1WiVeu7cngLieVuGv/rLxOCdapUyfGjh3LwIEDCQQCREdH8/LLL0c6LGNMCbV4czIjp8Wxcc8Rft8rhjEXdqROtUqRDitX4jocJjY2VnPeTG7NmjV07NgxQhGduNIevzEmNIfSMvjnf9fxn4VbiKlblX9c2pXftSueGbwislRVYwu7nfV8jDGmFJu/djcPzohnx8E0burXmnsHtqN65ZL/p73kR2iMMeY3ko+k89hHq5i5fDttG9Vg2h/PoGeL4y5jWews+RhjTCmiqsxasZ1HP1rNobQM7j63LXee3YbKFaMiHVqhWPIxxphSYvuBVMbMXMkXa3fTvXkd/jm8G+1PqhnpsI6LJR9jjCnhAgFl0qKtPDVnLVkB5aGLOnHDGa2IqlCypk8XhiUfY4wpwTbuOcyoafEs2pxM/1Ma8ORlXWler1qkwzphdifTUmrAgAHknBpujCk7MrICvLIggUHPf83anQf55++78Z+be5eJxAPW8zHGmBJnZZIrjbNq+0EGdzmJRy/pTKMSVBqnKFjyKeE2b97MoEGD6NWrFz/++COdO3dmwoQJkQ7LGBMGaRlZPD9vPa9/tZF61Svx2nU9GdSlSaTDCgtLPqGaMwp2xhftPk/qCoPHFbjaunXrePPNN+nXrx833XQTr7zyStHGYYyJuB827mPU9Hg27T3CFbExPDikE7WrRUc6rLCx5FMKNG/enH79+gFw3XXX8cILL0Q4ImNMUTmUlsG4OWuZ9MNWmterysSb+9C/bYNIhxV2lnxCFUIPJVxyVqMtadVpjTHHZ96aXYyZuZJdB9O4pX9r7hnYjmqVysefZZvtVgps3bqV77//HoDJkyfTv3//CEdkjDkR+w4f4y/vLuPm8UuoVSWa6Xf2Y8xFncpN4gFLPqVC+/btefnll+nYsSP79+/nj3/8Y6RDMsYcB1Vl5rIkznv2S+as3MFfz2vHR3/uT4/mdSIdWrErP2m2FKtYsSITJ078VduCBQsiE4wx5rgkHUjlwRnxLFi3h1Nb1OGp4d1o17h0lsYpCpZ8jDEmjAIBZeIPW3hqzloCCg9f3InrTy/dpXGKgiWfEq5Vq1asXLky0mEYY45Dwu7DjJ4ex+LN+zmzbQP+cWnZKI1TFMJ2zkdEqojIIhFZISKrRORR395aRH4QkQQReU9EKvn2yv51gl/eKmhfo337OhG5IKh9kG9LEJFRQe25HuN4lNY7vZbWuI0pCzKyArw8P4Ehz3/NT7sO88zl3ZlwU9kpjVMUwjnh4Bhwjqp2B3oAg0SkL/AU8JyqngLsB272698M7Pftz/n1EJFOwFVAZ2AQ8IqIRIlIFPAyMBjoBFzt1yWfYxRKlSpV2LdvX6n7Q66q7Nu3jypVylY5DmNKg/jEFC556Vue/nQd53dqzOf3nMXve8XYJRI5hG3YTd1f7MP+ZbR/KHAOcI1vHw88ArwKDPXPAT4AXhL30xoKTFHVY8AmEUkAevv1ElR1I4CITAGGisiafI5RKDExMSQmJrJnz57CbhpxVapUISYmJtJhGFNupKZn8b+f/8QbX2+kQY3K/PsPvbig80mRDqvECus5H987WQqcguulbAAOqGqmXyURaOafNwO2AahqpoikAPV9+8Kg3QZvsy1Hex+/TV7HyBnfbcBtAC1atPjN8ujoaFq3bh3iuzXGlFffb9jH6OlxbN53lKtOa87oIR2pXbXslsYpCmFNPqqaBfQQkTrADKBDOI9XWKr6OvA6QGxsbOkaWzPGRNzBtAyenL2WdxdtpUW9aky+pQ9nnFL2S+MUhWKZ7aaqB0RkPnA6UEdEKvqeSQyQ5FdLApoDiSJSEagN7Atqzxa8TW7t+/I5hjHGFIm5q3cxZmY8ew4d47bfncxfz2tH1UpRkQ6r1AjnbLeGvseDiFQFzgfWAPOB3/vVRgAf+uez/Gv88i/8eaNZwFV+NlxroC2wCFgMtPUz2yrhJiXM8tvkdQxjjDkhew8f467JP3LrhCXUrVaJGXf244EhHS3xFFI4ez5NgPH+vE8FYKqqfiwiq4EpIjIWWAa86dd/E/iPn1CQjEsmqOoqEZkKrAYygT/54TxE5C7gUyAKeEtVV/l9jczjGMYYc1xUlRnLknjs49UcPZbFvee34/az2lCpolUpOx5S2qYRh0tsbKzabamNMblJ3H+UB2as5Kuf9tCrZV2eGt6VUxqV39I4wURkqarGFna7QvV8RKQu0FxV4wp7IGOMKW0CAWXC95v556frAHj0ks78oW9LKpTz0jhFocDkIyILgEv8ukuB3SLyrareE+bYjDEmYhJ2H2LktHiWbtnPWe0a8sSlXYipaxUKikooPZ/aqnpQRG4BJqjqwyJiPR9jTJmUnhng319u4MUvEqhWOYpnr+jOpac2swoFRSyU5FNRRJoAVwAPhjkeY4yJmBXbDjByWhxrdx7iom5NeOSSzjSoUTnSYZVJoSSfx3Azyr5R1cUicjKwPrxhGWNM8UlNz+LZuet485tNNKxZmTeuj+X8To0jHVaZVmDyUdX3gfeDXm8EhoczKGOMKS7fJexl1PR4tiYf5Zo+LRg1uAO1qlhpnHALZcJBQ+BWoFXw+qp6U/jCMsaY8EpJzeDJ2WuYsngbrepXY8ptfel7cv1Ih1VuhDLs9iHwNfA5kBXecIwxJvw+XbWTh2auZN+RdG4/y5XGqRJtFQqKUyjJp5qqjgx7JMYYE2a7D6XxyKxVzI7fSccmtXhzxGl0jakd6bDKpVCSz8ciMkRVZ4c9GmOMCQNV5YOliYz9ZA2pGVncd0F7bvvdyURHWWmcSAkl+dwNPCAi6UA6ILh7xdUKa2TGGFMEtiUf5YEZ8Xy9fi+xLesybng3TmlUI9JhlXuhzHazAkbGmFInK6CM/24zz3y2DgEeH9qZa/tYaZySIpTZbgJcC7RW1cdFpDnQRFUXhT06Y4w5Dut3HeL+aXEs23qAAe0b8sSlXWlWp2qkwzJBQhl2ewUIAOcAjwOHcbfEPi2McRljTKGlZwZ4dcEGXpq/nhqVK/K/V/ZgaI+mVhqnBAol+fRR1Z4isgxAVff7m7cZY0yJsXzbAUZ+EMe6XYe4pHtTHr64E/WtNE6JFUryyfA3hFP4+aLTQFijMsaYEB1Nz+Rfn/3E299uolHNKrw5IpZzO1ppnJIulOTzAjADaCQiT+BuTz0mrFEZY0wIvk3Yy6jpcWxLTuW6vi0YOagDNa00TqkQSvL5AHcfn3Nx06yHAbvCGZQxxuQn5WgGT8xezdQlibRuUJ33butLHyuNU6qEknymA8NUdS2Av73CXKBXOAMzxpjc/HflDh76cBXJR9L544A23H1uWyuNUwqFcnnvTGCqiESJSCvc7RVGF7SRiDQXkfkislpEVonI3b79ERFJEpHl/jEkaJvRIpIgIutE5IKg9kG+LUFERgW1txaRH3z7e9kTIUSksn+d4Je3Cu3bYYwpqXYfTOOO/yzljok/0qhmZT78Uz9GDupgiaeUCuUi0zf8H/WZuMrWt6vqdyHsOxO4V1V/FJGawFIRmeuXPaeqzwSvLCKdgKuAzkBT4HMRaecXvwycDyQCi0VklqquBp7y+5oiIq8BNwOv+q/7VfUUEbnKr3dlCDEbY0oYVeX9JYmM/WQ1aZkBRg7qwC1ntrbSOKVcnslHRO4Jfgm0AJYDfUWkr6o+m9+OVXUHsMM/PyQia4Bm+WwyFJiiqseATSKSAPT2yxL8fYQQkSnAUL+/c4Br/DrjgUdwyWeofw7unNVLIiKqqvnFbIwpWbbuc6VxvknYS+/W9Rh3WVdObmilccqC/Ho+OcvqTM+jvUB+2OtU4AegH3CXiFwPLMH1jvbjEtPCoM0S+SVZbcvR3geoDxxQ1cxc1m+WvY2qZopIil9/b464bgNuA2jRokVh35YxJkyyAsrb327iX5/9RFQFYeywLlzTu4WVxilD8kw+qvpo8GsRqeHbDxfmAH67acD/qOpBEXkVVylB/dd/ARG5MZ2qvg68DhAbG2u9ImNKgHU7XWmcFdsOcE6HRjxxaRea1LbSOGVNKLXdugD/Aer513uB61V1VQjbRuMSzyRVnQ6gqruClr8BfOxfJgHNgzaP8W3k0b4PqCMiFX3vJ3j97H0likhFoLZf3xhTQh3LzOLl+Rt4dUECNatE8/xVPbiku5XGKatCOWP3OnCPqrZU1ZbAvcAbBW3kC5K+CawJPj/kp2pnuxRY6Z/PAq7yM9VaA22BRcBioK2f2VYJNylhlj9/Mx930SvACNxdV7P3NcI//z3whZ3vMabk+nHrfi564RtemLeeC7s24fN7zmJoj2aWeMqwUK7zqa6q87NfqOoCEakewnb9gD8A8SKy3Lc9AFwtIj1ww26bgdv9fleJyFRgNW6m3J9UNQtARO7CTfGOAt4K6nWNBKaIyFhgGS7Z4b/+x09aSMYlLGNMCXPkWCbPfLaOd77bTJNaVXj7htM4u0OjSIdlioEU1CEQkRnAj7ihN4DrgF6qemmYYytWsbGxumTJkkiHYUy58fX6PYyeHk/i/lSuP70l9w/qQI3Kofw/bEoSEVmqqrGF3S6Un/RNwKO42W4KfA3cWNgDGWMMwIGj6Yz9ZA0fLE3k5IbVef+O0zmtVb1Ih2WKWSjJ5zxV/Utwg4hcDrwfnpCMMWWRqjJn5U7+/uEq9h9N509nt+HP51hpnPIqlOQzmt8mmtzajDEmV7sOpvHQzJV8tnoXXZrVYvxNp9G5ae1Ih2UiKL8KB4OBIUAzEXkhaFEt3IQAY4zJl6ry3uJtPDF7DemZAUYP7sDN/VtT0UrjlHv59Xy24yoQXIK7pUK2Q8BfwxmUMab027LvCKOnx/Pdhn30aV2PccO70bpBKBNlTXmQX4WDFcAKEZmsqhnFGJMxphTLzArw9reb+dfcdURXqMA/Lu3KVac1t9I45ldCqWpticcYE5I1Ow4ycloccYkpnNexMWOHdeGk2lUiHZYpgWxSvTHmhB3LzOKlLxJ4dcEGaleN5qVrTuXCrk2sQoHJUyi13S5X1fcLajPGlE9LtyQzclo8CbsPc1nPZjx0YSfqVq8U6bBMCWdTrY0xx+XIsUye/nQd47/fTNPaVXnnxtMY0N5K45jQ2FRrY0yhLVi3mwdnrGR7SiojTm/F3y5ob6VxTKHYVGtjTMj2H0nn8Y9XM31ZEm0aVueDO06nV0srjWMKL6Sp1rjbaHfA1XZbp6rpxRSfMaYEUFU+jtvBI7NWkZKawZ/POYU/nX2KlcYxxy2UfvL5wL+BDbgk1FpEblfVOWGNzBhTIuxMSWPMzJV8vmYX3WJqM/GWPnRsUivSYZlSLpTk8yxwtqomAIhIG+ATwJKPMWVYIKBMWbyNJ2evISMQ4MEhHbmxXysrjWOKRCjJ51B24vE24s77GGPKqM17jzBqehwLNyZz+sn1GTe8Ky3rW2kcU3RCST5LRGQ2MBV3zudyYLGIXAagqtPDGJ8xphhlZgV485tNPDv3JypVrMC4y7py5WnN7WJRU+RCST5VgF3AWf71HqAqcDEuGVnyMaYMWL3dlcaJT0phYKfGPD6sC41rWWkcEx6h1Hazu5YaU4alZWTx4hfr+feXG6lTLZpXru3J4C4nWW+nvNi/BWrHQIXinbkYtjOHItJcROaLyGoRWSUid/v2eiIyV0TW+691fbuIyAsikiAicSLSM2hfI/z660VkRFB7LxGJ99u8IP63Ja9jGGN+bfHmZIa88DUvz9/AsFOb8fk9ZzHEarKVfelHYPm78M5F8Hw32PRlsYcQzmkrmcC9qtoJ6Av8SUQ6AaOAearaFpjnXwMMBtr6x23Aq+ASCfAw0AfoDTwclExeBW4N2m6Qb8/rGMYY4PCxTP7+4Uouf+170jMDTLipN89c3p061awmW5mlClu+hw/vgmfaw8w7ICURzh4DDTsWezhhq4ehqjuAHf75IRFZAzQDhgID/GrjgQXASN8+QVUVWCgidUSkiV93rqomA4jIXGCQiCwAaqnqQt8+ARiGmwKe1zGMKffmr93NgzPi2XEwjRv7teJvA9tT3UrjlF0pibDiXVg+GZI3QnR16HwpnHottDgdItTLza+22z35baiqz4Z6EBFpBZwK/AA09okJYCfQ2D9vBmwL2izRt+XXnphLO/kcw5hyK/lIOlAo5wgAACAASURBVI99tIqZy7fTtlENPrjjDHq1tBHpMikjFdZ+AssmwsYFgELL/vC7+6DjJVC5RqQjzLfnU9N/bQ+cBszyry8GFoV6ABGpAUwD/kdVDwaPJauqiogWKuJCyu8YInIbboiPFi1ahDMMYyJGVZm1YjuPfrSaQ2kZ3H1uW+48uw2VK1ppnDJFFZKWuoSzcjocS4HaLeCs+6H71VCvdaQj/JX8ars9CiAiXwE9VfWQf/0IrsJBgUQkGpd4JgVdD7RLRJqo6g4/rLbbtycBzYM2j/FtSfwyhJbdvsC3x+Syfn7HyPkeXwdeB4iNjQ1rEjQmEnakpDJmxkrmrd1N9+Z1+OfwbrQ/qWbBG5rS4+AOiHvPDavtXQcVq0KnodDjGmh1JlQomRUpQhnobQwEFxJNJ4RhLD/z7E1gTY4hulnACGCc//phUPtdIjIFN7kgxSePT4F/BE0yGAiMVtVkETkoIn1xw3nXAy8WcAxjyoVAQJm8aCvj5qwlMxBgzIUdubFfa6Iq2Cy2MiHzGKyb7RJOwuegAWjeFy55EToNgyolv/ZeKMlnArBIRGb418NwJ/EL0g/4AxAvIst92wO4hDBVRG4GtgBX+GWzcfcPSgCOAjcC+CTzOLDYr/dY9uQD4E7gHdxFr3P4pd5cXscwpszbuOcwo6bHs2hTMme0qc+4y7rRon61SIdlTpQq7FgOyyZB/PuQdgBqNoX+f4Ue10L9NpGOsFDETS4rYCWRXkB///IrVV0W1qgiIDY2VpcsWRLpMIw5bhlZAd74eiP/+/l6KleswEMXduLy2Bi7Zqe0O7znl2G13asgqjJ0vMglnJMHFPvFoTmJyFJVjS3sdqHOr1yOmzZd0R+shapuLezBjDHhsTIphZHT4li1/SCDOp/EY0M708hK45Rememw/jNYPsl9DWRCs1i48FnoMhyq1ol0hCeswOQjIn/GXeS5C8jC3dNHgW7hDc0YU5C0jCyen7ee17/aSN1qlXj12p4M7tok0mGZ47Uz3vVw4t6Do/ugRmM4/U/Q/Rpo1CHS0RWpUHo+dwPtVXVfuIMxxoRu0aZkRk2LY+PeI1wRG8ODQzpRu1p0pMMyhXVknzuHs3wS7IyDCtHQYYgbVmtzLkSVzQuAQ3lX24CUcAdijAnNobQMnvrvWiYu3ErzelWZeHMf+rdtEOmwTGFkZbpZassnwbo5EMiAJt1h8NPQ9fdQrV6kIwy7UJLPRmCBiHwCHMtuLEyFA2NM0fhi7S4enLGSXQfTuLl/a+4d2I5qlcrmf8Zl0u61sHwirHgPjuyGag2g923umpyTukQ6umIVyqd2q39U8g9jTDHbd/gYj360mlkrttOucQ1eufYMTm1hpXFKhdT9sHKamyK9/UeoUBHaDXIJp+1AiCqfQ6Wh3M/n0eIIxBjzW6rKh8u38+hHqzh8LJO/nteOPw5oQ6WKJfOqdeMFsmDjfJdw1n4CWcegUWe44B/Q9Qqo0TDSEUZcKLPdGgL3A51xdzUFQFXPCWNcxpR72w+k8uCMeOav28OpLerw1PButGtspXFKtL0J7jzOiilwaDtUrQu9bnC9nCbdI1ZBuiQKZdhtEvAecBFwB65czZ5wBmVMeRYIKJN+2MK4OWsJKPz9ok6MOKOVlcYpqdIOwqrpbor0th9AKsAp58PgcW54rWLlSEdYIoWSfOqr6psicreqfgl8KSKLC9zKGFNoG/YcZtS0OBZv3s+ZbRvwj0u70ryelcYpcQIB2PyVG1Zb8xFkpkKD9nD+Y9DtSqh5UqQjLPFCST4Z/usOEbkQ2A6U/XmAxhSjjKwAr3+1kefnradqdBTPXN6d4T2bWWmckiZ5k+vhrHgXUrZB5drQ42rocR0062nDaoUQSvIZKyK1gXtxVaNrAX8Na1TGlCPxiSncPy2ONTsOcmHXJjx8SSca1bTSOCXGscOw+kN3LmfLt4BAm7PhvEegw4UQXTXCAZZOocx2+9g/TQHODm84xpQfaRlZPPf5T/zf15uoV70Sr13Xi0FdbLimRFB1iWb5ZFg1EzKOQL02cM5D7sZstZsVvA+TL7s6zZgI+H7DPkZPj2PzvqNcGducB4Z0tNI4JcGBrbD8XVgxGfZvhko1oetwV+qmeR8bVitClnyMKUYH0zJ4cvZa3l20lRb1qjHplj70O8VK40RU+lE3aWD5JNj0FaDQ+ncwYDR0vBgqVY90hGWSJR9jisnc1bsYMzOePYeOceuZrbnn/PZUrRTZe7GUW6qwbZErdbNyBqQfgjotXcLpfhXUbRnpCMu8UC4yrQwMB1oFr6+qj4UvLGPKjr2Hj/HIrFV8HLeDDifV5PU/xNK9eem/H0uplJIEcVPcuZx9CRBdzd12+tRrocUZUMEqRxSXUHo+H+ImGywlqLCoMSZ/qsqMZUk89vFqjh7L4p7z23HHWVYap9hlpMG6T9w1ORvngwZcoun/V+g0FCpb1YhICCX5xKjqoLBHYkwZkrj/KA/OWMmXP+2hpy+N09ZK4xQfVUj60Z3HWfkBpKVArRg48143W61+m0hHWO6Fkny+E5Guqhof9miMKeUCAeU/C7fw1H/XAvDIxZ34w+lWGqfYHNr1y7DanrVQsQp0vMTVVmt9lg2rlSChJJ/+wA0isgk37CaAqmq+t9EWkbdw9eB2q2oX3/YIcCu/1IZ7QFVn+2WjgZtxt+r+i6p+6tsHAc8DUcD/qeo4394amALUxw0J/kFV0/05qglAL2AfcKWqbg7hfRpzQhJ2H2LktHiWbtnP79o15B+XdiGmrpXGCbvMdPhpjks46+eCZkFMb7j4eeh8KVSpHekITS5CST6Dj3Pf7wAv4RJBsOdU9ZngBhHpBFyFq5zdFPhcRNr5xS8D5wOJwGIRmaWqq4Gn/L6miMhruMT1qv+6X1VPEZGr/HpXHud7MKZAGVkBXluwgRe/SKBa5SievaI7l55qpXHCStXdcnrZJHcL6tRkqNkE+v3FXZPToG2kIzQFCKXCwRYR6Q+0VdW3/S0WaoSw3Vci0irEOIYCU1T1GLBJRBKA3n5ZgqpuBBCRKcBQEVkDnANc49cZDzyCSz5D/XOAD4CXRERUVUOMxZiQxSUe4P4P4li78xAXdWvCwxd3pmFNq2IcNkf2QtxUdy5n10qIquRK3PS4zpW8qWBT10uLUKZaPwzEAu2Bt4FoYCLQ7ziPeZeIXA8sAe5V1f1AM2Bh0DqJvg1gW472PrihtgOqmpnL+s2yt1HVTBFJ8evvzeW93QbcBtCiRYvjfDumPEpNzy6Ns5GGNSvzxvWxnN+pcaTDKpuyMtxw2vJJ8NN/IZAJTXvCkGegy3CoZnWOS6NQht0uBU4FfgRQ1e0icrzTdl4FHgfUf/0XcNNx7uuEqerrwOsAsbGx1jMyIfluw15GT49ny76jXN27BaOHdKBWFSuNU+R2rXLnceLegyN7oHpD6HOHG1Zr3CnS0ZkTFErySVdVFREFEJHjrjWhqruyn4vIG0B20dIkoHnQqjG+jTza9wF1RKSi7/0Er5+9r0QRqQjU9usbc0JSUjN4cvYapizeRqv61Xj31r6c3qZ+pMMqW44mQ/wHrpezYzlUiIb2g1zCOeU8iLIkX1aEknymisi/cX/sb8X1VN44noOJSBNV3eFfXgqs9M9nAZNF5FnchIO2wCLczLq2fmZbEm5SwjU+Gc4Hfo+b8TYCdzFs9r5GAN/75V/Y+R5zoj5dtZOHZq5k7+Fj3H7Wyfz1vHZUibbzC0UiKxM2fOFK3aybA1npcFJXGPQUdL0cqluCL4tCmXDwjIicDxzEnff5u6rOLWg7EXkXGAA0EJFE4GFggIj0wA27bQZu98dYJSJTgdVAJvAnVc3y+7kL+BQ31fotVV3lDzESmCIiY4FlwJu+/U3gP37SQjIuYRlzXPYccqVxPol3pXH+b0Qs3WKsNE6R2LPO9XBWvAeHd0K1+hB7s7smp0m+V3KYMkCsU+DExsbqkiVLIh2GKSFUlWk/JvH4x6tJTc/iL+eewu1ntSE6yi5SPCGpB2DVdDdFOmkJSBS0Hehqq7W9ACpWinSEppBEZKmqxhZ2O6tqbUwO25KP8sCMeL5ev5fYlnUZN7wbpzQq8OoCk5dAFmxc4CYPrP0YMtOgYUcYOBa6XQk1GkU6QhMBlnyM8bICyvjvNvPMZ+sQ4LGhnbmuT0sqWGmc47Nvg0s4K96Fg0lQpQ6c+gc3rNb0VLsxWzmXZ/IRkXmqeq6IPKWqI4szKGOK2/pdh7h/WhzLth5gQPuGPHFpV5rVqRrpsEqfY4dg1QyXdLZ+D1IB2pwLFzwB7QZDdJVIR2hKiPx6Pk1E5AzgEl9Z4Ff/pqjqj2GNzJhikJ4Z4NUFG3hp/npqVK7Ic1d2Z1gPK41TKIEAbPnGncdZMwsyjkL9tnDeI9DtKqjVJNIRmhIov+Tzd+Ah3DU0z+ZYprjyNsaUWsu3HWDkB3Gs23WIi7s35eGLO9GghpXGCdn+zbD8XVgxGQ5shcq1oNsVrtRNTKwNq5l85Zl8VPUD4AMReUhVHy/GmIwJq6PpmTz72U+89e0mGtWswv9dH8t5VhonNOlHYPUsN0V689eAwMkD4Jy/Q8eLINqGKk1oQrnO53ERuQT4nW9aoKof57eNMSXVtwl7GTU9jm3JqVzbpwUjB1tpnAKpuvM3yyfBqpmQfhjqnQznjHHDanWaF7wPY3IIpbDok7gK05N8090icoaqPhDWyIwpQilHM3hi9mqmLkmkdYPqTLmtL31Ptivn83VgG6yY4pLO/k1QqQZ0HuZK3bQ43YbVzAkJZar1hUAPVQ0AiMh4XEUBSz6mVPjvyh089OEqko+kc8dZbfif89paaZy8ZKTCmo9dqZuNXwIKrc6Es0ZCx4uhsl3vZIpGqNf51MGVqgFXqNOYEm/3oTQe/nAVc1bupFOTWrx9w2l0aWYf399QhcQlLuGsnA7HDkLtFi7h9Lga6raKdISmDAol+TwJLPOFPAV37mdUWKMy5gSoKu8vTWTsx6tJywxw/6D23HrmyVYaJ6eDOyBuirsmZ+9PULEqdBrqSt207A8V7PtlwieUCQfvisgC4DTfNFJVd4Y1KmOO09Z9rjTONwl76d2qHk8O70qbhjZU9LOMNFg32yWcDfNAA+78zSUvQqdhUKVWpCM05URIw27+NgizwhyLMcctK6C8/e0m/vXZT0RVEB4f1oVre7ew0jjghtW2L3MJJ/59SDsAtZpB/3tcqZv6bSIdoSmHrLabKfXW7TzEyGlxLN92gHM6NGLssC40tdI4cHi3uwvo8smwezVUrAIdLnIJ5+QBUMEmXZjIseRjSq1jmVm8Mn8DryxIoGaVaJ6/qgeXdG9avkvjZKbD+k9dqZv1n4FmQbNYuOg56HwZVLV7EZmSIaTkIyJRQOPg9VV1a7iCMqYgP27dz8gP4li/+zBDezTl7xd1on55Lo2zI84Pq02Fo/ugRmM44y53TU7D9pGOzpjfCOUi0z/j7kK6Cwj4ZgXsVoOm2B05lskzn63jne82c1KtKrx1QyzndCinpXGO7HPncJZPhJ3xEFUJ2g92tdXanANRNrBhSq5QPp13A+1VdV+4gzEmP1+v38Po6fEk7k/lD31bcv+g9tQsb6VxsjIhYa6rOrDuvxDIgCY9YPDT0PX3UK1epCM0JiShJJ9tQEq4AzEmLweOpjP2kzV8sDSRkxtUZ+rtp9O7dTn7I7t7DSybCHFT4chuqNYA+tzuJg807hzp6IwptPxuJnePf7oRWCAinwDHsperas7bLOTc/i3gImC3qnbxbfWA94BWwGbgClXdL+4M8fPAEOAocEP2/YJEZAQwxu92rKqO9+29gHeAqsBs4G5V1byOUfC3wpREc+JdaZz9R9O5c0Ab/nJuOSqNk7of4j9w53K2/wgVKkK7QS7htB0IUeWs12fKlPx6PjX9163+Uck/wJ3zKcg7wEvAhKC2UcA8VR0nIqP865HAYKCtf/QBXgX6+ETyMBDrj7lURGb5ZPIqcCvwAy75DALm5HMMU4rsPpjGQx+u5NNVu+jSrBbjbzqNzk3LQWmcQBZsmO/O46z9BLLSoXEXuOBJ6Ho51GgY6QhNGTJzWRJPf7qO7QdSaVqnKvdd0J5hpzYrlmPndz+fRwFE5HJVfT94mYhcXtCOVfUrEWmVo3koMMA/Hw8swCWGocAEVVVgoYjUEZEmft25qprsjzsXGOQrLtRS1YW+fQIwDJd88jqGKQVUlalLtjH2kzWkZwYYNbgDt/RvTcWyXhpn73p3HmfFFDi0A6rWhV43ulI3J3WzCtKmyM1clsTo6fGkZmQBkHQgldHT4wGKJQGFcs5nNPB+CG2haOyrJQDsxE3fBmiGO7eULdG35deemEt7fsf4DRG5DbgNoEWLFoV9L6aIbdl3hNHT4/luwz76tK7HuOHdaN2geqTDCp+0FFg1w12Tk7gIJApOOQ8GP+WG1yqW46njJuye/nTdz4knW2pGFk9/ui6yyUdEBuPOwTQTkReCFtUCMk/0wP78TCjDd2E7hqq+DrwOEBsbG9ZYTN6yAspb32ziX3PXEV2hAk9c2oWrTyujpXECAdj0pTuPs+YjyEyFBu3h/Meg25VQ86RIR2jKie0HUgvVXtTy6/lsB5YAlwBLg9oPAX89zuPtEpEmqrrDD6vt9u1JQPDtEGN8WxK/DKFlty/w7TG5rJ/fMUwJtHbnQUZ+EMeKxBTO69iIx4d1oUntMlgaJ3mjSzjL34WDiVCltps40ONaaNbThtVMsWtapypJuSSa4ipNld85nxXAChGZrKoZRXS8WcAIYJz/+mFQ+10iMgU34SDFJ49PgX+ISF2/3kBgtKomi8hBEemLm3BwPfBiAccwJcixzCxe/iKBVxZsoHbVaF68+lQu6takbJXGOXYYVs90SWfLt4C4iz8HPgbtL4ToKpGO0JRj913Q/lfnfACqRkdx3wXFUxEjlHM+P+YydJWC6xWNzeviUxF5F9draSAiibhZa+OAqSJyM7AFuMKvPhs3xJeAm2p9I4BPMo8Di/16j2VPPgDu5Jep1nP8g3yOYUqIpVuSGTktnoTdh7ns1GY8dFEn6lavVPCGpUEgAFu/c+dxVn8IGUegXhs49+/Q7SqoXTwziYwpSPZ5nUjNdhM3wSyfFUT+CWQBk33TVUA13Mn8/qp6cVgjLCaxsbG6ZMmSSIdRph05lsnTn65j/PebaVq7Kk9c2oUB7RtFOqyisX+Lm6m2fBIc2AKVakKXS12pm+a9bVjNlFkislRVYwu7XSg9n/NUtWfQ63gR+VFVe4rIdYU9oCmfvvxpDw9Mj2d7SirX923JfYM6UKNyKa89ln4U1sxyCWfTV66t9Vlw9oPQ8WKoVC2y8RlTgoXy2x8lIr1VdRGAiJwGZF9ifsKz3kzZtv9IOo9/sprpPybRpmF13r/9dGJbleLSOKqw7QeXcFbOgPRDUKclDHgAelwNdWzKvjGhCCX53AK8JSI1AAEOAreISHXgyXAGZ0ovVeWT+B08MmsVB45mcNfZp3DXOaeU3tI4KUmw4l03eSB5A0RXh87D3Iy1FmdAhTJ+EawxRazA5KOqi4GuIlLbvw4uMjo1XIGZ0mtnShpjZq7k8zW76NqsNhNu6kOnprUiHVbhZaS6EjfLJ7mSNyi07Adn3gudhkLlGpGO0JhSK5T7+VQGhuMKdVbMngqrqo+FNTJT6gQCypTF23hy9hrSswI8MKQDN/UrZaVxVCHpR1dbLX4aHEuB2s3hd/e5YbV6J0c6wlInkvXDTMkVyrDbh7ip1UsJqmptTLDNe48wanocCzcm0/fkeoy7rButSlNpnEM7Ie49N6y2Zy1UrAIdL3G11Vr9zobVjlOk64eZkiuUqdYrs2+JUJbZVOvjk5kV4M1vNvHs3J+oFFWBBy7syFWnNS8dF4tmHoN1c1zCSfgcNAua93HncTpf6qoQmBPSb9wXuV5FX6dqNNUrV7TeUBkQzqnW34lIV1WNP464TBm2evtBRk6LIz4phfM7NebxoV04qXYJv2pfFXascOdx4t9398yp2RT63e2SToO2kY6wTMkt8QAcSM3gQGrGz+tYb6j8CSX59AduEJFNuGE3wdXs7BbWyEyJlZaRxUtfJPDalxuoUy2al6/pyZCuJ5Xs3s7hPRA/1VUe2L0KoipDhwvdsNrJZ0OFUjoLr4SLEiGrgNEVKN5qyqZkCCX5DA57FKbUWLI5mZHT4tiw5wjDe8Yw5sKOJbc0TlYGrP/MJZz1n0IgE5r2hAv/BV2Gu3vmmLAKJfFkK65qyqZkCGWq9RYR6Q+0VdW3RaQhYHNMy5nDxzJ5+r9rmbBwC01rV2X8Tb05q10JvavmzpXuPE7ce3B0L1RvBH3/6CpIN+oY6ejKlWZ5VE7OTXFVUzYlQyhTrbNvY90eeBuIBiYC/cIbmikp5q/bzYPT49lxMI0Rp7fivgvaU72klcY5muzO4Syf5M7pVIiG9oNdwjnlPIgqYfGWE7lVTo6OElDICPzSKyrOasqmZAjlN/JS4FTgRwBV3S4iNcMalSkRko+k8/jHq5mxLIlTGtXggzvOoFfLEjRUlZUJG+a5hLNuDmSlu1tOD3oKul4O1etHOsISKed1N2d3aMj8tXvCMvMsr8rJubXZ+Z7yJZSp1otUtXdQMdHqwPdlbcKBTbX+haryUdwOHp21ipTUDO48+xT+dHYbKlcsISfl96yDZRPdsNrhXVCtvrsLaI9r4KSukY6uRMt53U1uqkZH8eRlXS0ZmJCEc6r1VBH5N1BHRG4FbgLeKOyBTOmwIyWVMTNWMm/tbrrH1GbSrX3ocFIJKI2TegBWTnO9nKSlIFHQ7gI3rNZ2IFQsoZMeiklwb6ZOtWhUISU14ze9iqc/XZdv4gGbeWaKRygTDp4RkfNxBUXbA39X1blhj8wUq0BAmbxoK+PmrCUzEGDMhR25sV9roipEcPp0IAs2zneTB9Z8DFnHoFEnGPgEdLsCapSRewEFCbUUzcxlSTz60Sr2H/3tTYaD23JeQxPqjDKbeWbCLaSzsD7ZWMIpozbuOcyo6fEs2pTMGW3q8+RlXWlZP4KlcfYmwIrJ7uZsB5OgSh3oeb27JqdJjzJ7Y7bcStHc895y7pm6nIC6a2au7tOc2Jb1uO+DFWRkhTaNObgn0zTE2Wc288yEW57JR0QOAbl9urMvMi0BYzHmRGRmBXjj60089/lPVK5YgaeGd+WK2AiVxkk7CKtmuF7OtoUgFdwstQuegPZDoGLl4o/pBBxPMc3chsQC8PNvYZYqExduZfrSxJATT7bsnkxus89yc3aHEjqN3pQZeSYfVbUZbWXYyqQURk6LY9X2g1zQuTGPDe1C41rFXBonEIDNX7vzOKtnQWYqNGgH5z0C3a6CWk2KN54iklsP5r4PVrh7G6Vm/HzVf7McSSnU62GOZgQKHVN2Tybn7LMKeVQgmL92T6GPYUxhROTiBxHZDBwCsoBMVY0VkXrAe7hbN2wGrlDV/eL+DX8eGAIcBW5Q1R/9fkYAY/xux6rqeN/eC3gHqArMBu7Wgqb1lRNpGVk8P289r3+1kbrVKvHqtT0Z3LWY/8gnb/I3ZnsXUrZC5drQ/So3eSAmttQPq+XWg8nI0p9rmWX/sc95PibUUjSFlfMammGnNvs5CbUe9Umu29g5HxNukbzy7mxV3Rv0ehQwT1XHicgo/3okrrxPW//oA7wK9PHJKvsCWAWWisgsVd3v17kV+AGXfAYBc4rnbZVcizYlM2paHBv3HuHyXjE8eGFH6lQrpllixw7Dmlmu1M2WbwCBkwfAeQ+7GmvRpescw8xlSTwwPe5XvZCq0RUY3ism5B4M/Pp8TFElnuqVooiOqpDrbLec8joHZOd8TLiVpMu+hwID/PPxwAJc8hkKTPA9l4UiUkdEmvh156pqMoCIzAUGicgCoJaqLvTtE4BhlOPkcygtg6f+u5aJC7cSU7cq/7m5N2e2LYYxfVXY+r1LOKtnQvphdzO2c8a4YbU6zcMfQxjMXJb08ySAYKkZASYu3Fro/WX3MgpTiiY3dapG88glnQs1RTq3c0BWbaAciZsK8x6DlESoHQPn/t3NJC0GkUo+CnwmIgr8W1VfBxqr6g6/fCfQ2D9vBmwL2jbRt+XXnphL+2+IyG3AbQAtWrQ4kfdTYn2xdhcPzljJzoNp3NSvNX+7oB3VKoX5x35gm5uptnwS7N8ElWpA52HQ4zpo0Teiw2qFmco8enocqcdxfqWw6lSLBnJPBBXwkw5y0axOVb4ddc4JHTuvCgR2jU85EDcVPvqLu108QMo29xqKJQFFKvn0V9UkEWkEzBWRtcELVVV9Ygorn/ReB1fhINzHK077Dh/jsY9X8+Hy7bRrXIOXrz2Dni3CWBon/Sis/dglnI1fAgqtzoSzRkKnS6BS5O9qOmZmPJMWbv15CmfOcy75XTsTTofTMpm5LCnfUjTh7J0EnwMy5ci8x35JPNkyUl17WU0+qprkv+4WkRlAb2CXiDRR1R1+WG23Xz0JCB6fifFtSfwyTJfdvsC3x+SyfrmgqsxasZ1HP1rNobQM/ue8ttw54BQqVQzDbaBVIXGxK3WzagYcOwh1WriE0+NqqNuq6I8ZpDDTmWcuS/pV4smWmpHFox+tAijUtTNFKSOgP5/3yS8RWO/EFKmUxMK1F7FiTz6+NlwFVT3knw8EHgNmASOAcf7rh36TWcBdIjIFN+EgxSeoT4F/iEj2v/MDgdGqmiwiB0WkL27CwfXAi8X1/iJp+4FUxsxcyRdrd9OjeR3++ftutGschhnzB7f7YbXJsG89RFeDTkNdbbWW/aFCGBJdDrlNZx49PZ4lW5J/LpJZu2o0InDgaAYVRHK9aA1cRYD/eW952GPOT0Gzy6x3Yopc7Rg31JZbezGIRM+nMTDDX8hYEZisqv8VkcW4OnI3A1uA7H7fbNw06wTcVOsbX39RUwAAEtdJREFUAXySeRxY7Nd7LHvyAXAnv0y1nkMZn2wQCCiTFm3lqTlryQooD13UiRvOaFW0pXEy0mDdbDestuEL0MD/t3fm0VFXWR7/3KoshKWBAI3IIkjHVlBEQAHF1tYRFRVQcR1ER1vbtlt77LEVpx3EPn1G0XHj6DRi6yAzbggIro1CIygqKFsSQHYREIGwByEkVW/+eC/hl6KqklRqTe7nnF/q1ftt99Wr/L5133IfdBlol5/uORxykzctLHKHf6Bah3/l0Gao26JmqUBHlylJ58Ix1ft8wI46vXBMUm5fY1TrxkKmRrVev7OUB6cVsejb3Qz6WVseveo0Ouc3jc/FjYHvl1gPp2gqHN4LP+kIp99gvZw23eNznzowY+lW/u2t5QRClSeDEKqHDtEo0krKiMNot0RGtVbSkPJAkInzN/DsnLU0yfLxxIhejOjbKT6hcQ5st8sVLHsNdq6CrCZw8uU2tlq388CXuKUVKvtxtu49VDXpsrWL0uz1ZNKFyk87nBS2ysumrCJ4zECBq/t2TNj6OYpSJ3pdm7Sh1aGo+GQgxVv3cf/UQlZu28+Q045j7NCe/LRFPUPjVByBNX+3grP2IzAB6HQmXP409LwK8lrFx/gozFi6tVrfS2VTWbJHn9WW1k2zWTpmcNg1cvKy/Ywd2hPQgQKpnEuipC8qPhnE4fIAT89ew98+3Uh+sxwmjOzDJafWMzTOtkLbj1M4BQ7thubHwdl322a1domZaDhj6daqOGeZQOum2ZSWVVQbCZftFx6+wopLTXNlGp3YeCmcAjPugqCr632b7fvvvrQ/clSQGi3a5+NI9z6fLzfs4sHpRWwsOch1/Trz70NOoaWbnFhnDpZA0Vs28sD2IvDn2MjRZ4yEE38J/vr9Jgm3sJk3oGY6k+2XaiJT2R8D6sHExLhu9kdNTWTnwRXjVYAyEO3zaaDsP1zOYx9+w2sLv6NLflNe/VV/zvlZ27pfKFAO62bbOTlrZtlfoh16w5D/glOvhqb5MdkXOtemIhBg+4EjVfu9TWbpLjwjB3Sh3wn5jc+DSWSzWG2EB5I6uVFJD1R80pjZK7fz0Ixidhw4zK8GdeMPg2MIjbN95dFmtYM7oFk76P9r26zWvmdMds1YupU/vrWM0Mgz9YlLlmzat8ihpLScgDFVi7T9Zbj1cBqsyFTiFZu81lC2z64aC7ZZbOZvbTrZQpCkyY1KeqDik4aUlJYx9p0VvFe4jZ+3b8GEm/rSu3MdOvx/3A3F06zofL8UfFlw0iV2yYKCi8Bft+a6h2YU8frCzWnvuVTiE3jq2t5VIXO8Mdp8Ajf271IlNGlHJC8kXD5UFxGAQ3uqp0M9mdB4XuE8k8AR+PCB+IhPXn7tvZ8kTW5U0gPt83GkQ5+PMYa3l27lz++t5GBZBXdfUMCd53WvXWicQAVsmGub1VZ/YB8g7U+1gtPrWmhWc1Odd5hzptAqL5vyQJCDRwJV7+sa2TmuVInEZhC/HTXYsjMUDD7awR5JHAqnWK8jcMRzwUiDuaOFHA3B25/y9KnhZ7WHY+y+2h0XjXBl8vkB39FBCKE2KhmF9vlkOFv2/Mif3i5m3pqd9OnSinFX96KgNqFxdq5xzWpvwoFt9pdmv1tts1qH06OeGqn5LFN45rrex4pM4RSYczHMrGf/ReEU++u/8le7PweCFTayg/ihTQGUrOaoKFROHfVMITWepqyvXzp6ba8n4I0k/OEDIcID4WcQQa2FB6r3pyS7aavys4/mtelot0aJik+KCQYN/7dwE+M+/AYDPHxFD0YNrCE0zuF9UDzdis6Wr+zDsOAiuPRxOOliyMoFjh11drg8kJQlApLByAFdGO5fAE97HmAFg2H5a9FDxIeKSl4+XDqu+oMvdHgwVBcFE4CSaoHYOSoSMbQkVIpDbZunYqFSdCLF8wolL7YBKGGJNJFRxaZRo81ujlQ0u63bUcroaYV8vWkP5xa05T+vjBIaJxiAjfPsJNBV70LFYWh3smtWuw5atK869KEZRTEtapYJdGyVx+T2b9J905vE9KCPRnYzCJRZDyfphAbdiTMtO8O9xcf2+YTDlw3D/1vFQakV2uyWQZQHgrwwbz3j56wjL8fPk9eczlV9OoYPjbNrvRWc5W/A/i3QpKUVnDP+GY7vAyJc9NQnrN2RvnOUouETyM3ycag8yFDfZ9yfNYWOUkIQ26sBsE9aUNamJ+13L4LDQRt2NhGUH0zQhWtBy05w5GBivB9vsMhwzWDe/ihtAlOShHo+jmR5PoVb9nL/1EK++eEAl/XqwNgretKuRW71g8oOwIoZtlntuy9AfND9Auh9I+fOaMLm0oSbGReG+j7j4azJ5Is1eA/N+Ua6MYAivEMoTNWflC5ymkIErppok6HNfeHw5VhH6Zj+IUdOM/Dnhh/tpihxRj2fNOfQkQDPzF7Di59uoF2LXCbe1JfBPY87ekAwCJsWWMFZORPKf2Sj6cCUiuuZHhjE9uJ8KK6/HZXexfFSwvemLXOCvbnQt4zjpYS9NMcYaC2lfG/a8njFtbwTHFTtnD2mOc2kjFxqERpHjo7VAsinlLMpCndY9QMbG/1urS4OXq8k/0T49jPbzyR+6HsLXP5U5FF1KjRKhqCejyORns/n60t4cHoRm3b9yA1ndWb0pafQMs/NtdmziVV/f4Fm30yhi+zkgMnj3cAApgbOY4kpINancjiRucr3Kc2lrJp3YUxkb8P71WicHkk98OfY9Y42zot8TLjBDoqSYajnk4bsO1TOYx+u4vVFmzmhTVNeu70/Z3dvyyPTFrFv8TRG+Odztn8lPzfC58EePBm4hlnBMzmMbYYb6vuMsdmTaY1tttptmvNIxSjeCQ4Ke7+hvs94PHsiudgO80rB6CQljJLZYQUkmqio4EQhLx96XmmXDw/tp/GKynt/gMWTjvVcFKWRo56PI96ez0crfuA/Zhaz80AZxoDB0E9WM8I/n+H+BTSRcoyB/aYpHwfP4AL/8iqRKUfIdp0goQJQU3WpYNSDcKPd/Lm2b0X7ThQlLOr5pAldR79flS6QzfxP1nOc4rPzKkSObcpqKT9ytSyoJho5UYbcqrjEAe0bUZSUo+JTT7qOfp8FOXfRgb1MD55LS24iiPBE1gQG+xcTOle0rk1fSgTEZ6MNqJAoSkbSYMVHRC4BngX8wN+MMY/F+x6vPHQ1G3Nns8W05eaKBzhgmvF49gv8k28JftHmzJjRjnhFafA0SPERET/wPHARsAX4SkTeMcasjNc9vhh/C6P8s5kROIf15njGZk2mu29b1NFjjRoVFEVRPDRI8QHOAtYZYzYAiMgbwDAgbuJz5q6ZiMCpvo1c6VtAmbEfZYMXHl8WDP+rioiiKPWioYpPR8AbPXEL0D/0IBG5A7gDoEuXLnW6gd9FFe4u32MM5Eoq4oHFAfHDlRNUTBRFSSoNVXxqhTFmIjAR7FDrupwbwEcWwWMGFKSceKzBoiiKkmAaqvhsBTp73ndyeXHjqzbDGLDr7Vo3s1WbnyPVXiKT0wwuf0a9EkVRGhwNVXy+AgpEpBtWdK4HboznDQbeM4kvxsOAXW9HPMYrTCKoV6IoiuJokOJjjKkQkd8Bs7BDrV82xqyI930G3jMJmBTvyyqKojR4GqT4ABhjPgA+SLUdiqIoyrH4aj5EURRFUeKLio+iKIqSdFR8FEVRlKSj4qMoiqIkHV3PxyEiO4FNMZzaFiiJszmpRMuT3mh50p+GVqaaynOCMaZdXS+q4lNPROTrWBZSSle0POmNlif9aWhlSlR5tNlNURRFSToqPoqiKErSUfGpPxNTbUCc0fKkN1qe9KehlSkh5dE+H0VRFCXpqOejKIqiJB0VH0VRFCXpqPjEiIhcIiKrRWSdiIxOtT3REJFvRaRIRJaJyNcuL19EPhaRte61tcsXERnvylUoIn0817nZHb9WRG5Oov0vi8gOESn25MXNfhHp6z6fde7chC8RGKFMY0Vkq6unZSIyxLPvQWffahG52JMf9nsoIt1EZKHLf1NEchJYls4iMldEVorIChH5vcvP2DqKUqZMraMmIrJIRJa78jwSzQYRyXXv17n9XWMtZ0SMMbrVccMu07AeOBHIAZYDPVJtVxR7vwXahuQ9Dox26dHAOJceAnyIXetuALDQ5ecDG9xra5dunST7fwH0AYoTYT+wyB0r7txLU1SmscB9YY7t4b5juUA3993zR/seAlOA6116AvCbBJalA9DHpVsAa5zNGVtHUcqUqXUkQHOXzgYWus8zrA3AXcAEl74eeDPWckba1POJjbOAdcaYDcaYI8AbwLAU21RXhgGvuPQrwHBP/mRj+RJoJSIdgIuBj40xu40xe4CPgUuSYagxZj6wOyQ7Lva7fT8xxnxp7H/XZM+1EkaEMkViGPCGMabMGLMRWIf9Dob9Hjqv4AJgqjvf+/nEHWPMNmPMEpc+AKwCOpLBdRSlTJFI9zoyxphS9zbbbSaKDd66mwpc6GyuUzmj2aTiExsdgc2e91uI/sVMNQb4SEQWi8gdLq+9MWabS/8AtHfpSGVLtzLHy/6OLh2anyp+55qiXq5spqLuZWoD7DXGVITkJxzXPHMG9pd1g6ijkDJBhtaRiPhFZBmwAyvs66PYUGW327/P2Ry354OKT+NgkDGmD3Ap8FsR+YV3p/s1mbFj7jPdfg9/BboDvYFtwJOpNaduiEhzYBrwr8aY/d59mVpHYcqUsXVkjAkYY3oDnbCeysmptEfFJza2Ap097zu5vLTEGLPVve4A3sZ+8ba75gzc6w53eKSypVuZ42X/VpcOzU86xpjt7gERBF7E1hPUvUy7sE1ZWSH5CUNEsrEP6VeNMdNddkbXUbgyZXIdVWKM2QvMBQZGsaHKbre/pbM5bs8HFZ/Y+AoocCNFcrAdcu+k2KawiEgzEWlRmQYGA8VYeytHE90MzHTpd4BRbkTSAGCfazqZBQwWkdauqWGwy0sVcbHf7dsvIgNcm/Yoz7WSSuWD2nEltp7Alul6NwKpG1CA7YAP+z10XsZcYIQ73/v5JMJuAV4CVhljnvLsytg6ilSmDK6jdiLSyqXzgIuw/ViRbPDW3QjgH87mOpUzqlHxHlXRWDbsiJ012HbTP6Xanih2nogdebIcWFFpK7b9dg6wFpgN5Lt8AZ535SoC+nmudSu2g3Ed8C9JLMPr2CaOcmxb8m3xtB/oh32IrAeew0X+SEGZ/tfZXOj+cTt4jv+Ts281npFekb6Hrt4XubK+BeQmsCyDsE1qhcAytw3J5DqKUqZMraNewFJndzEwJpoNQBP3fp3bf2Ks5Yy0aXgdRVEUJelos5uiKIqSdFR8FEVRlKSj4qMoiqIkHRUfRVEUJemo+CiKoihJR8VHUcIgIqU1H1Xna/aW6lGQx4rIffW43jUiskpE5obkdxWRG+tx3fNF5OxYz1eU2qDioyjJozd2LkS8uA243Rjzy5D8rkDM4gOcD6j4KAlFxUdRakBE/igiX7lgkpXroHR1XseLYtdH+cjNHEdEznTHLhORJ0Sk2M36/jNwncu/zl2+h4h8IiIbROSeCPe/QexaNsUiMs7ljcFOhHxJRJ4IOeUx4Fx3n3tdQMknPGX4tbvGvSLyskuf5q7fA7gTuNedf67zsIrFrgUzP64frtJ4SfRMbt10y8QNKHWvg4GJ2Fn5PuA97Fo8XYEKoLc7bgow0qWLgYEu/RhuzR7gFuA5zz3GAp9j10Zpi42dlR1ix/HAd0A7IAv4BzDc7fsET3QAzznnA+953t8BPOTSucDX2LVYfMB8bJiYr4FzPHbd5zm/COjo0q1SXTe6NYxNPR9Fic5gty0FlmAjARe4fRuNMctcejHQ1cXPamGM+cLlv1bD9d83dm2UEmzgzfYh+88EPjHG7DQ2tP2rWPGraxlGiQ2nvxAb9qbA2OCYt2BDxswzxiyIcP4CYJKI3I5dNExR6k1WzYcoSqNGgEeNMS9Uy7RrvJR5sgJAXgzXD71GIv4nBbjbGBMuEGwBUIr1sMJijLlTRPoDlwGLRaSvMWZXAuxUGhHq+ShKdGYBt4pd1wUR6SgiP410sLHh6g+4hzXY6L6VHMAuyVwXFgHniUhbEfEDNwDzajgn9D6zgN+IXSIAETnJRTtvCYzHelJtRGREuPNFpLsxZqExZgywk+qh8xUlJtTzUZQoGGM+EpFTgC9slH1KgZFYLyUStwEvikgQKxT7XP5cYLRr/nq0lvffJiKj3bmCbaarKfR+IRAQkeXAJOBZbB/VErdUwE7scslPA88bY9aIyG3AXDeg4F1gqogMA+7GDj4ocPefg42Qrij1QqNaK0qcEZHmxphSlx6NDbv/+xSbpShphXo+ihJ/LhORB7H/X5uwnfqKonhQz0dRFEVJOjrgQFEURUk6Kj6KoihK0lHxURRFUZKOio+iKIqSdFR8FEVRlKTz/xA4q9/to3SXAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"yWPsnMRV9N6i"},"source":["As we can see, the use of a BBPE tokenizer trained with an English corpus on a corpus of another language (here, Polish) **requires on average almost 233.69 % of additional tokens**  than a BBPE tokenizer trained with the same language of the corpus of application."]}]}