Text Summarization using BERT, GPT2,XLNET.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"01_Text Summarization using BERT, GPT2,XLNET.ipynb","provenance":[{"file_id":"1pr3DQ75Mf6nJVXn9nSD3ClkTH1vtHsNn","timestamp":1620035342277}],"collapsed_sections":["-xxKB7kPuBMh","8TyvQNpsqkbm","oHCoA1mNqrq_"]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"-xxKB7kPuBMh"},"source":["# Text Summarization "]},{"cell_type":"markdown","metadata":{"id":"u97lWTzvt8Eu"},"source":["Text Summarization is the process of shortening a set of data computationally, to create a subset (a summary) that represents the most important or relevant information within the original content.\n","\n","The sentences, in summary, are generated by the model, not just extracted from the original text data."]},{"cell_type":"markdown","metadata":{"id":"Q3LzYrN1rftB"},"source":["  Notebook is based on the :  \n","https://medium.com/analytics-vidhya/text-summarization-using-bert-gpt2-xlnet-5ee80608e961\n","\n","Other common tasks where NLP is used are :\n","\n","---\n","\n","* _Sentiment Analysis_\n","\n","Sentiment analysis is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information.- Wikipedia\n","\n","* _Text Classification_\n","\n","Text Classification is the task of assigning predefined categories to free-text documents. It can provide conceptual views of document collections and has important applications in the real world.-Scholarpedia. Text classification helps to assign predefined categories to a document to help you find the information you need or simplify some activities. For example, an application of text classification is spam filtering in email.\n","\n","* _Question-Answering_\n","\n","Question answering is a computer science discipline within the fields of information retrieval and natural language processing, which is concerned with building systems that automatically answer questions posed by humans in a natural language.- Wikipedia. Question-Answering (QA) is becoming more and more popular thanks to applications such as Siri, OK Google, chat boxes, and virtual assistants.\n","\n","\n","\n","* _Machine Translation_\n","\n","\n","\n"," Machine translation, sometimes referred to by the abbreviation MT, is a sub-field of computational linguistics that investigates the use of software to translate text or speech from one language to another.- Wikipedia. Google’s language translator is a good example that uses neural machine translation to translate text in one language to every possible language!\n","\n"," ---"]},{"cell_type":"markdown","metadata":{"id":"8TyvQNpsqkbm"},"source":["# 1. Install and import libraries"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"P06w20oVdaHn","executionInfo":{"status":"ok","timestamp":1620063863510,"user_tz":-120,"elapsed":30422,"user":{"displayName":"Marco Polo","photoUrl":"","userId":"10130345471127002742"}},"outputId":"0945e1ae-28f3-4694-d039-de1e8663de2d"},"source":["!pip install bert-extractive-summarizer\n","!pip install transformers==2.2.0\n","!pip install spacy\n","from summarizer import Summarizer,TransformerSummarizer"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting bert-extractive-summarizer\n","  Downloading https://files.pythonhosted.org/packages/1a/07/fdb05f9e18b6f641499ef56737126fbd2fafe1cdc1a04ba069d5aa205901/bert_extractive_summarizer-0.7.1-py3-none-any.whl\n","Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from bert-extractive-summarizer) (0.22.2.post1)\n","Requirement already satisfied: spacy in /usr/local/lib/python3.7/dist-packages (from bert-extractive-summarizer) (2.2.4)\n","Collecting transformers\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d8/b2/57495b5309f09fa501866e225c84532d1fd89536ea62406b2181933fb418/transformers-4.5.1-py3-none-any.whl (2.1MB)\n","\u001b[K     |████████████████████████████████| 2.1MB 26.5MB/s \n","\u001b[?25hRequirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->bert-extractive-summarizer) (1.19.5)\n","Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->bert-extractive-summarizer) (1.4.1)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->bert-extractive-summarizer) (1.0.1)\n","Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (0.8.2)\n","Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (1.0.5)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (56.0.0)\n","Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (2.0.5)\n","Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (0.4.1)\n","Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (1.0.0)\n","Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (7.4.0)\n","Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (2.23.0)\n","Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (1.1.3)\n","Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (4.41.1)\n","Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (1.0.5)\n","Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy->bert-extractive-summarizer) (3.0.5)\n","Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers->bert-extractive-summarizer) (3.10.1)\n","Collecting tokenizers<0.11,>=0.10.1\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ae/04/5b870f26a858552025a62f1649c20d29d2672c02ff3c3fb4c688ca46467a/tokenizers-0.10.2-cp37-cp37m-manylinux2010_x86_64.whl (3.3MB)\n","\u001b[K     |████████████████████████████████| 3.3MB 51.8MB/s \n","\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers->bert-extractive-summarizer) (3.0.12)\n","Collecting sacremoses\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n","\u001b[K     |████████████████████████████████| 901kB 45.5MB/s \n","\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers->bert-extractive-summarizer) (20.9)\n","Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers->bert-extractive-summarizer) (2019.12.20)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2020.12.5)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2.10)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (3.0.4)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (1.24.3)\n","Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers->bert-extractive-summarizer) (3.7.4.3)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers->bert-extractive-summarizer) (3.4.1)\n","Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers->bert-extractive-summarizer) (7.1.2)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers->bert-extractive-summarizer) (1.15.0)\n","Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers->bert-extractive-summarizer) (2.4.7)\n","Installing collected packages: tokenizers, sacremoses, transformers, bert-extractive-summarizer\n","Successfully installed bert-extractive-summarizer-0.7.1 sacremoses-0.0.45 tokenizers-0.10.2 transformers-4.5.1\n","Collecting transformers==2.2.0\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ec/e7/0a1babead1b79afabb654fbec0a052e0d833ba4205a6dfd98b1aeda9c82e/transformers-2.2.0-py3-none-any.whl (360kB)\n","\u001b[K     |████████████████████████████████| 368kB 19.7MB/s \n","\u001b[?25hCollecting sentencepiece\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f5/99/e0808cb947ba10f575839c43e8fafc9cc44e4a7a2c8f79c60db48220a577/sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2MB)\n","\u001b[K     |████████████████████████████████| 1.2MB 17.8MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==2.2.0) (1.19.5)\n","Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==2.2.0) (2.23.0)\n","Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from transformers==2.2.0) (2019.12.20)\n","Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from transformers==2.2.0) (4.41.1)\n","Collecting boto3\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d2/8f/42959300c543b4d34bc9f9b54954471a33384c181084ed84f070763d7f37/boto3-1.17.62-py2.py3-none-any.whl (131kB)\n","\u001b[K     |████████████████████████████████| 133kB 58.7MB/s \n","\u001b[?25hRequirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers==2.2.0) (0.0.45)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.2.0) (1.24.3)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.2.0) (2020.12.5)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.2.0) (2.10)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.2.0) (3.0.4)\n","Collecting botocore<1.21.0,>=1.20.62\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bd/60/ba830f93176fdc23166043298173ee2aecd5cf150f1ede51d6506f021deb/botocore-1.20.62-py2.py3-none-any.whl (7.5MB)\n","\u001b[K     |████████████████████████████████| 7.5MB 36.7MB/s \n","\u001b[?25hCollecting s3transfer<0.5.0,>=0.4.0\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/63/d0/693477c688348654ddc21dcdce0817653a294aa43f41771084c25e7ff9c7/s3transfer-0.4.2-py2.py3-none-any.whl (79kB)\n","\u001b[K     |████████████████████████████████| 81kB 12.4MB/s \n","\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n","  Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n","Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==2.2.0) (1.0.1)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==2.2.0) (1.15.0)\n","Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==2.2.0) (7.1.2)\n","Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.62->boto3->transformers==2.2.0) (2.8.1)\n","\u001b[31mERROR: botocore 1.20.62 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n","Installing collected packages: sentencepiece, jmespath, botocore, s3transfer, boto3, transformers\n","  Found existing installation: transformers 4.5.1\n","    Uninstalling transformers-4.5.1:\n","      Successfully uninstalled transformers-4.5.1\n","Successfully installed boto3-1.17.62 botocore-1.20.62 jmespath-0.10.0 s3transfer-0.4.2 sentencepiece-0.1.95 transformers-2.2.0\n","Requirement already satisfied: spacy in /usr/local/lib/python3.7/dist-packages (2.2.4)\n","Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy) (2.0.5)\n","Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (2.23.0)\n","Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (7.4.0)\n","Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy) (1.1.3)\n","Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy) (1.0.5)\n","Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy) (3.0.5)\n","Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (1.0.5)\n","Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy) (1.0.0)\n","Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (4.41.1)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy) (56.0.0)\n","Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (0.4.1)\n","Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (1.19.5)\n","Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy) (0.8.2)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.10)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2020.12.5)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (1.24.3)\n","Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy) (3.10.1)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy) (3.4.1)\n","Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy) (3.7.4.3)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"oHCoA1mNqrq_"},"source":["# 2. Read the body text"]},{"cell_type":"code","metadata":{"id":"_S-l8K6kffiV","executionInfo":{"status":"ok","timestamp":1620063889060,"user_tz":-120,"elapsed":851,"user":{"displayName":"Marco Polo","photoUrl":"","userId":"10130345471127002742"}}},"source":["body = '''\n","       CLIP (Contrastive Language–Image Pre-training) builds on a large body of work on zero-shot transfer, natural language supervision, and multimodal learning. The idea of zero-data learning dates back over a decade8 but until recently was mostly studied in computer vision as a way of generalizing to unseen object categories.910 A critical insight was to leverage natural language as a flexible prediction space to enable generalization and transfer. In 2013, Richer Socher and co-authors at Stanford11 developed a proof of concept by training a model on CIFAR-10 to make predictions in a word vector embedding space and showed this model could predict two unseen classes. The same year DeVISE12 scaled this approach and demonstrated that it was possible to fine-tune an ImageNet model so that it could generalize to correctly predicting objects outside the original 1000 training set.\n","\n","Most inspirational for CLIP is the work of Ang Li and his co-authors at FAIR13 who in 2016 demonstrated using natural language supervision to enable zero-shot transfer to several existing computer vision classification datasets, such as the canonical ImageNet dataset. They achieved this by fine-tuning an ImageNet CNN to predict a much wider set of visual concepts (visual n-grams) from the text of titles, descriptions, and tags of 30 million Flickr photos and were able to reach 11.5% accuracy on ImageNet zero-shot.\n","\n","Finally, CLIP is part of a group of papers revisiting learning visual representations from natural language supervision in the past year. This line of work uses more modern architectures like the Transformer32 and includes VirTex,33 which explored autoregressive language modeling, ICMLM,34 which investigated masked language modeling, and ConVIRT,35 which studied the same contrastive objective we use for CLIP but in the field of medical imaging\n","\n","        '''"],"execution_count":2,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vj713-xIrXcE"},"source":["# 3. Prepare summaries"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Y7qZlFLqg2a7","executionInfo":{"status":"ok","timestamp":1620038718459,"user_tz":-120,"elapsed":45401,"user":{"displayName":"Marco Polo","photoUrl":"","userId":"10130345471127002742"}},"outputId":"354ae5c5-ae8a-4b85-8c70-08ea31e78958"},"source":["bert_model = Summarizer()\n","bert_summary = ''.join(bert_model(body, min_length=60))\n","print(bert_summary)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["CLIP (Contrastive Language–Image Pre-training) builds on a large body of work on zero-shot transfer, natural language supervision, and multimodal learning. The same year DeVISE12 scaled this approach and demonstrated that it was possible to fine-tune an ImageNet model so that it could generalize to correctly predicting objects outside the original 1000 training set.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"BRlVvp4Ug-PF","executionInfo":{"status":"ok","timestamp":1620038768569,"user_tz":-120,"elapsed":80029,"user":{"displayName":"Marco Polo","photoUrl":"","userId":"10130345471127002742"}},"outputId":"efe796a6-3815-4bbc-b048-cfefb365a6c8"},"source":["GPT2_model = TransformerSummarizer(transformer_type=\"GPT2\",transformer_model_key=\"gpt2-medium\")\n","full = ''.join(GPT2_model(body, min_length=60))\n","print(full)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["CLIP (Contrastive Language–Image Pre-training) builds on a large body of work on zero-shot transfer, natural language supervision, and multimodal learning. Most inspirational for CLIP is the work of Ang Li and his co-authors at FAIR13 who in 2016 demonstrated using natural language supervision to enable zero-shot transfer to several existing computer vision classification datasets, such as the canonical ImageNet dataset.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"OE0HzUQ_jtOU","executionInfo":{"status":"ok","timestamp":1620038783870,"user_tz":-120,"elapsed":92534,"user":{"displayName":"Marco Polo","photoUrl":"","userId":"10130345471127002742"}},"outputId":"144e3dc2-6e32-41da-ec7b-09d2b36a15d2"},"source":["xlnet_model = TransformerSummarizer(transformer_type=\"XLNet\",transformer_model_key=\"xlnet-base-cased\")\n","full = ''.join(xlnet_model(body, min_length=60))\n","print(full)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["CLIP (Contrastive Language–Image Pre-training) builds on a large body of work on zero-shot transfer, natural language supervision, and multimodal learning. This line of work uses more modern architectures like the Transformer32 and includes VirTex,33 which explored autoregressive language modeling, ICMLM,34 which investigated masked language modeling, and ConVIRT,35 which studied the same contrastive objective we use for CLIP but in the field of medical imaging\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"CTN_wsfY1vtG"},"source":[],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"M12Lkipx1vqh"},"source":[],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"xWcnfWYB1vni"},"source":[],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"TnXgYnVZ1vkL"},"source":[],"execution_count":null,"outputs":[]}]}