language/use-cases/document-summarization/summarization_with_documentai.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Summarization with Large Documents using Document AI and PaLM APIs\n",
        "\n",
        "> **NOTE:** This notebook uses the PaLM generative model, which will reach its [discontinuation date in October 2024](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text#model_versions). Please refer to [this updated notebook](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/document_processing.ipynb) for a version which uses the latest Gemini model.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_with_documentai.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6736e7517731"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Holt Skinner](https://github.com/holtskinner), [Mona Mona](https://github.com/Mona19) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Text summarization is the process of creating a shorter version of a text document while still preserving the most important information. This can be useful for a variety of purposes, such as quickly skimming a long document, getting the gist of an article, or sharing a summary with others.\n",
        "\n",
        "Although summarizing a short paragraph is a non-trivial task, there are a few challenges to overcome if you want to summarize a large document, such as a PDF file with multiple pages.\n",
        "\n",
        "[Document AI](https://cloud.google.com/document-ai) provides a scalable and managed way to extract data from documents using AI. In this notebook, you will use the [Document OCR processor](https://cloud.google.com/document-ai/docs/document-ocr), which is a pre-trained model that will extract text and layout information from document files. Document AI provides an API endpoint to access the models, so developers don't have to build and maintain their own models and serving infrastructure.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this notebook we will show how you can do the following:\n",
        "\n",
        "1. Extract text from PDF documents using the Document AI OCR processor.\n",
        "1. Use a MapReduce method to chunk the document text.\n",
        "1. Use PaLM `text-bison` model to generate summaries of the extracted text.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4j1gvi3jqG6U"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "- Vertex AI PaLM APIs offered by Generative AI studio\n",
        "- Document AI\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "Learn about [Document AI pricing](https://cloud.google.com/document-ai/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QDU0XJ1xRDlL"
      },
      "source": [
        "## Getting Started\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2a5AEr0lkLKD"
      },
      "source": [
        "### Install Vertex AI SDK & Other dependencies\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GYACuZHAF3DQ"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade google-cloud-aiplatform==1.35.0 google-cloud-documentai==2.20.1 google-cloud-documentai-toolbox==0.11.1a backoff==2.2.1 --user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UG3WsMySF3DQ"
      },
      "source": [
        "**Colab only**: Run the following cell to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_Hsqwn4hkLKE"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-yyDsflbCc0"
      },
      "source": [
        "### Enable the APIs in your project\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ai62J2sLPDFX"
      },
      "outputs": [],
      "source": [
        "!gcloud config set project \"YOUR_PROJECT_ID\"\n",
        "!gcloud services enable documentai.googleapis.com storage.googleapis.com aiplatform.googleapis.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UP76a2la7O-a"
      },
      "source": [
        "## Document AI\n",
        "\n",
        "The following [limits](https://cloud.google.com/document-ai/quotas) apply for online processing with the Document OCR processor.\n",
        "\n",
        "| Limit             | Value |\n",
        "| :---------------- | ----: |\n",
        "| Maximum file size | 20 MB |\n",
        "| Maximum pages     |    15 |\n",
        "\n",
        "For documents that don't meet these limits, you can use [batch processing](https://cloud.google.com/document-ai/docs/send-request#batch-process) to extract the document text. (Not covered in this notebook.)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZkLDRTjTcfm"
      },
      "source": [
        "### Preparing data files\n",
        "\n",
        "To begin, you will need to download PDFs for the summarizing tasks below.\n",
        "For this notebook, you will be using Alphabet earnings report PDFs hosted in a public Google Cloud Storage bucket.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f-IWo-lb-gbn"
      },
      "outputs": [],
      "source": [
        "# Copying the files from the GCS bucket to local storage\n",
        "!gsutil -m cp -r gs://github-repo/documents/docai ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FxmWYA05o4jj"
      },
      "source": [
        "### Create Document AI OCR Processor\n",
        "\n",
        "A [Document AI processor](https://cloud.google.com/document-ai/docs/overview#dai-processors) is an interface between a document file and a machine learning model that performs document processing actions. They can be used to classify, split, parse, or analyze a document. Each Google Cloud project needs to create its own processor instances.\n",
        "\n",
        "There are two types of Document AI processors:\n",
        "\n",
        "- Pre-trained processors: These processors are pre-trained on a large dataset of documents and can be used to perform common document processing tasks, such as Optical Character Recognition (OCR), form parsing, and entity extraction.\n",
        "- Custom processors: These processors can be trained on your own dataset of documents to perform specific tasks that are not covered by the pre-trained processors.\n",
        "\n",
        "Refer to [Full processor and detail list](https://cloud.google.com/document-ai/docs/processors-list) for all supported processors.\n",
        "\n",
        "Processors take a PDF or image file as input and output the data in the [`Document`](https://cloud.google.com/document-ai/docs/reference/rest/v1/Document) format.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t9URyhLDo4jk"
      },
      "source": [
        "### Create a processor\n",
        "\n",
        "Run this code only once to create the processor. You cannot create multiple processors with the same display name. If you receive an error, change the name of the processor and rerun.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7isig7e07O-a"
      },
      "outputs": [],
      "source": [
        "from google.api_core.client_options import ClientOptions\n",
        "from google.api_core.exceptions import AlreadyExists\n",
        "from google.cloud import documentai\n",
        "from google.cloud.documentai_toolbox.wrappers.document import Document\n",
        "\n",
        "# TODO(developer): Edit these variables before running the code.\n",
        "project_id = \"YOUR_PROJECT_ID\"\n",
        "\n",
        "# See https://cloud.google.com/document-ai/docs/regions for all options.\n",
        "location = \"us\"\n",
        "\n",
        "# Must be unique per project, e.g.: \"My Processor\"\n",
        "processor_display_name = \"YOUR_PROCESSOR_DISPLAY_NAME\"\n",
        "\n",
        "# You must set the `api_endpoint` if you use a location other than \"us\".\n",
        "client_options = ClientOptions(api_endpoint=f\"{location}-documentai.googleapis.com\")\n",
        "\n",
        "\n",
        "def create_processor(\n",
        "    project_id: str, location: str, processor_display_name: str\n",
        ") -> documentai.Processor:\n",
        "    client = documentai.DocumentProcessorServiceClient(client_options=client_options)\n",
        "\n",
        "    # The full resource name of the location\n",
        "    # e.g.: projects/project_id/locations/location\n",
        "    parent = client.common_location_path(project_id, location)\n",
        "\n",
        "    # Create a processor\n",
        "    return client.create_processor(\n",
        "        parent=parent,\n",
        "        processor=documentai.Processor(\n",
        "            display_name=processor_display_name, type_=\"OCR_PROCESSOR\"\n",
        "        ),\n",
        "    )\n",
        "\n",
        "\n",
        "try:\n",
        "    processor = create_processor(project_id, location, processor_display_name)\n",
        "    print(f\"Created Processor {processor.name}\")\n",
        "except AlreadyExists as e:\n",
        "    print(\n",
        "        f\"Processor already exits, change the processor name and rerun this code. {e.message}\"\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JkK74RRio4jk"
      },
      "source": [
        "### Process the documents\n",
        "\n",
        "Process document takes the processor name and file path of the document and extracts the text from the document.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L7bE7ZGtlcxv"
      },
      "outputs": [],
      "source": [
        "def process_document(\n",
        "    processor_name: str,\n",
        "    file_path: str,\n",
        ") -> documentai.Document:\n",
        "    client = documentai.DocumentProcessorServiceClient(client_options=client_options)\n",
        "\n",
        "    # Read the file into memory\n",
        "    with open(file_path, \"rb\") as image:\n",
        "        image_content = image.read()\n",
        "\n",
        "    # Load Binary Data into Document AI RawDocument Object\n",
        "    raw_document = documentai.RawDocument(\n",
        "        content=image_content, mime_type=\"application/pdf\"\n",
        "    )\n",
        "\n",
        "    # Configure the process request\n",
        "    request = documentai.ProcessRequest(name=processor_name, raw_document=raw_document)\n",
        "\n",
        "    result = client.process_document(request=request)\n",
        "\n",
        "    return result.document"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usKPLAna5mLd"
      },
      "source": [
        "#### Create data chunks\n",
        "\n",
        "LLMs produce the best results when summarizing documents if the input data is broken up into small \"chunks\" before being added to the prompt.\n",
        "\n",
        "The best chunk size depends on the size of the documents. It is a good idea to experiment with different chunk sizes to see what works best for your specific dataset and application.\n",
        "For the provided documents, we are using the paragraphs detected by Document AI to separate the chunks.\n",
        "You should experiment with other values as well and see how it affects your summarization.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nDf7Zw3SGuyW"
      },
      "outputs": [],
      "source": [
        "import glob\n",
        "import os\n",
        "\n",
        "# If you already have a Document AI Processor in your project, assign the full processor resource name here.\n",
        "processor_name = processor.name\n",
        "extracted_data: list[dict] = []\n",
        "\n",
        "# Loop through each PDF file in the \"docai\" directory.\n",
        "for path in glob.glob(\"docai/*.pdf\"):\n",
        "    # Extract the file name and type from the path.\n",
        "    file_name, file_type = os.path.splitext(path)\n",
        "\n",
        "    print(f\"Processing {file_name}\")\n",
        "\n",
        "    # Process the document.\n",
        "    document = process_document(processor_name, file_path=path)\n",
        "\n",
        "    if not document:\n",
        "        continue\n",
        "\n",
        "    # Using Document AI Toolbox to handle post-processing\n",
        "    wrapped_document = Document.from_documentai_document(document)\n",
        "\n",
        "    # Split the text into chunks based on paragraphs.\n",
        "    document_chunks = [\n",
        "        paragraph.text\n",
        "        for page in wrapped_document.pages\n",
        "        for paragraph in page.paragraphs\n",
        "    ]\n",
        "\n",
        "    # Can also split into chunks by page or blocks.\n",
        "    # document_chunks = [page.text for page in wrapped_document.pages]\n",
        "    # document_chunks = [block.text for page in wrapped_document.pages for block in page.blocks]\n",
        "\n",
        "    # Loop through each chunk and create a dictionary with metadata and content.\n",
        "    for chunk_number, chunk_content in enumerate(document_chunks, start=1):\n",
        "        # Append the chunk information to the extracted_data list.\n",
        "        extracted_data.append(\n",
        "            {\n",
        "                \"file_name\": file_name,\n",
        "                \"file_type\": file_type,\n",
        "                \"chunk_number\": chunk_number,\n",
        "                \"content\": chunk_content,\n",
        "            }\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jj7KMupOPUX-"
      },
      "source": [
        "## Summarization using the [PaLM](https://ai.google/discover/palm2/) Model\n",
        "\n",
        "You have just used Document AI to extract text from PDF files.\n",
        "\n",
        "In the next section, you will summarize the extracted text using the PaLM model with Vertex AI.\n",
        "In order to summarize the text, You can use MapReduce to chunk the text to fit the prompt size.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xe7OuYuGkLKF"
      },
      "source": [
        "### Authenticating your notebook environment\n",
        "\n",
        "- If you are using **Colab** to run this notebook, run the cell below and continue.\n",
        "- If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8YRKSFYOqSH4"
      },
      "outputs": [],
      "source": [
        "from google.colab import auth\n",
        "import vertexai\n",
        "\n",
        "auth.authenticate_user()\n",
        "\n",
        "PROJECT_ID = \"YOUR_PROJECT_ID\"  # @param {type:\"string\"}\n",
        "vertexai.init(project=PROJECT_ID, location=\"us-central1\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H3edk4BiDkQ4"
      },
      "source": [
        "### Import models\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CKzpNLuzDeC4"
      },
      "outputs": [],
      "source": [
        "from __future__ import annotations\n",
        "\n",
        "import backoff\n",
        "from google.api_core.exceptions import ResourceExhausted\n",
        "from vertexai.preview.language_models import TextGenerationModel\n",
        "\n",
        "generation_model = TextGenerationModel.from_pretrained(\"text-bison\")\n",
        "\n",
        "\n",
        "# This decorator is used to handle exceptions and apply exponential backoff in case of ResourceExhausted errors.\n",
        "# It means the function will be retried with increasing time intervals in case of this specific exception.\n",
        "@backoff.on_exception(backoff.expo, ResourceExhausted, max_time=10)\n",
        "def text_generation_model_with_backoff(**kwargs):\n",
        "    \"\"\"\n",
        "    :param **kwargs: Keyword arguments for the prediction.\n",
        "    :return: The generated text.\n",
        "    \"\"\"\n",
        "    # Calls the generation_model's 'predict' method with the provided keyword arguments (**kwargs)\n",
        "    # and then accesses the 'text' attribute to get the generated text.\n",
        "    return generation_model.predict(**kwargs).text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RM3V1JARZ9-k"
      },
      "source": [
        "## MapReduce\n",
        "\n",
        "MapReduce is a very effective approach for processing large datasets because it is scalable and efficient. It can be used to process datasets that are too large to be processed on a single machine.\n",
        "\n",
        "Using this approach we will first split the large data into chunks, then running a prompt on each chunk of text. For summarization tasks, the output from the initial prompt would be a summary of that chunk. Once all the initial outputs have been generated, a different prompt is run to combine them. This can be more effective for large datasets.\n",
        "\n",
        "This consists of two main steps, map and reduce:\n",
        "\n",
        "- The map step will split the dataset into chunks and run a prompt on each chunk of text. The output from the prompt is a summary of that chunk.\n",
        "\n",
        "- The reduce step combines the summaries from all the chunks into a single summary.\n",
        "\n",
        "Here are some pros and cons of using MapReduce method for summarization:\n",
        "\n",
        "Pros:\n",
        "\n",
        "- Can summarize a large document\n",
        "- Can work well with parallel processing as the processes to summarize pages are independent to each other.\n",
        "\n",
        "Cons:\n",
        "\n",
        "- Multiple calls to the model is needed\n",
        "- As the pages are summarized individually, the context between the pages could be lost.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eo5NkotOJs3Y"
      },
      "source": [
        "#### Map step\n",
        "\n",
        "In this section, you will use the model to summarize each chunk of text individually using the initial prompt template.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oT6brl-VCd8l"
      },
      "outputs": [],
      "source": [
        "prompt_template = \"\"\"\n",
        "    Write a concise summary of the following text delimited by triple backquotes.\n",
        "\n",
        "    ```{text}```\n",
        "\n",
        "    CONCISE SUMMARY:\n",
        "\"\"\"\n",
        "\n",
        "# Create an empty list to store the summaries\n",
        "initial_summary: list[str] = []\n",
        "\n",
        "# Iterate over the pages and generate a summary for each page\n",
        "for individual_chunk in extracted_data:\n",
        "    # Create a prompt for the model using the extracted text and a prompt template\n",
        "    prompt = prompt_template.format(text=individual_chunk[\"content\"].strip())\n",
        "\n",
        "    # Generate a summary using the model and the prompt\n",
        "    summary = text_generation_model_with_backoff(prompt=prompt, max_output_tokens=1024)\n",
        "\n",
        "    # Append the summary to the list of summaries\n",
        "    initial_summary.append(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NJ3cOxWOJs3d"
      },
      "source": [
        "Take a look at the first few summaries of from the initial Map phase. These are the summaries of each individual chunk of text.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "z-YHPaQbDH5t"
      },
      "outputs": [],
      "source": [
        "print(\"\\n\\n\".join(initial_summary[:5]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9HczirNnJs3d"
      },
      "source": [
        "#### Reduce step\n",
        "\n",
        "Here you will create a reduce function that concatenates the summaries from the inital summarization step (Map step) and use the final prompt template to create a summary of the initial summaries.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4MF4JryiDTIK"
      },
      "outputs": [],
      "source": [
        "# Concatenate the summaries from the inital step\n",
        "concat_summary = \"\\n\".join(initial_summary)\n",
        "\n",
        "# Create a prompt for the model using the concatenated text and a prompt template\n",
        "prompt = prompt_template.format(text=concat_summary)\n",
        "\n",
        "# Generate a summary using the model and the prompt\n",
        "summary = text_generation_model_with_backoff(prompt=prompt, max_output_tokens=34)\n",
        "print(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WLFuRGrtN-9l"
      },
      "source": [
        "# Conclusion\n",
        "\n",
        "In this notebook, you learned:\n",
        "\n",
        "1. How to use Document AI to extract text from these PDFs.\n",
        "2. How to use MapReduce to efficiently process large amounts of text data.\n",
        "3. How to summarize the text extracted from the PDFs using the PaLM `text-bison` model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhR1vTtpXj1q"
      },
      "source": [
        "## Clean Up\n",
        "\n",
        "If you no longer need the Document AI processor, you can delete it using the following code.\n",
        "\n",
        "Alternatively, you can use the Cloud Console to delete the processor as outlined in [Creating and managing processors > Delete a processor](https://cloud.google.com/document-ai/docs/create-processor#documentai_delete_processor-web).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EuTdzGhbe9C3"
      },
      "outputs": [],
      "source": [
        "def delete_processor(processor_name: str) -> None:\n",
        "    client = documentai.DocumentProcessorServiceClient(client_options=client_options)\n",
        "\n",
        "    # Delete a processor\n",
        "    operation = client.delete_processor(name=processor_name)\n",
        "    # Print operation details\n",
        "    print(operation.operation.name)\n",
        "    # Wait for operation to complete\n",
        "    operation.result()\n",
        "\n",
        "\n",
        "delete_processor(delete_processor, processor_name)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "summarization_with_documentai.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}