NeMo Forced Aligner for general karaoke use v0.01.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "> *Use this Colab notebook to auto-generate sub-word timestamps for a vocals-only audio file, for your karaoke-making purposes.*\n",
        ">\n",
        ">*It's very loosely based and heavily modified off of [this official tutorial notebook](https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/tools/NeMo_Forced_Aligner_Tutorial.ipynb).*\n",
        ">\n",
        ">*It doesn't require a GPU and should be able to run on any linux-based runtime, including the default free ones. 👍*\n",
        ">\n",
        ">*Enjoy!*\n",
        "\n",
        "-- *Kadda OK*"
      ],
      "metadata": {
        "id": "CrWQwJLv3mIP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1. Configuration"
      ],
      "metadata": {
        "id": "eGT7UyADpo-f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Vocal Audio Source\n",
        "\n",
        "# @markdown  There are a few options of how to get the vocal wave file into this colab instance.\n",
        "# @markdown\n",
        "# @markdown  - **Google Drive**: Upload the wave file to your drive, then select it in the drive\n",
        "# @markdown  and go to `Share`. \\\n",
        "# @markdown  In the Share popup, under General Access, select \"`Anyone with the link`\", and then\n",
        "# @markdown  Copy link. \\\n",
        "# @markdown Paste it in the field below. \\\n",
        "#@markdown *(For example, \"`https://drive.google.com/file/d/SoMe_iDeNtIfIer/view?usp-drive_link`\")*\n",
        "google_drive_url = \"\" # @param {type:\"string\"}\n",
        "# @markdown **OR**\n",
        "\n",
        "# @markdown  - **Direct Download**: If you have somewhere that's open to the internet that you can\n",
        "# @markdown  put a wave file, and get a url to it that, when visited in a browser, will either\n",
        "# @markdown  start downloading it immediately as a file, or will show play/pause/seek/time controls\n",
        "# @markdown  and nothing else, no site branding or other navigation, then you can use that URL here.\n",
        "# @markdown  (If it doesn't behave that way, this won't work.)\\\n",
        "# @markdown  You can paste such a link in this field.\\\n",
        "# @markdown Note: you may have to include quotation marks if `wget` yells at you about extra parameters. \\\n",
        "# @markdown *(For example, \"`\"https://some.site/your_stuff/songToLoad.wav\"`\")*\n",
        "direct_url = \"\" # @param {type:\"string\"}\n",
        "# @markdown **OR**\n",
        "\n",
        "# @markdown  - **Local File Path**: If you are running this colab connected to a ***LOCAL*** runtime,\n",
        "# @markdown  you can just use the path of the file directly.\\\n",
        "# @markdown Note: you will have to include quotation marks if the path contains spaces. \\\n",
        "# @markdown *(For example, \"`\"/mnt/c/audio/song To Load.wav\"`\"\")*\n",
        "local_file_path = \"\" # @param {type:\"string\"}\n",
        "# @markdown\n",
        "\n",
        "# @markdown **SO**, which of the above do you want me to use? \\\n",
        "# @markdown *(YES you still have to tell me, even if you only filled in one of the inputs above, because I'm lazy.)*\n",
        "get_audio_from = \"Google Drive\" # @param [\"Google Drive\", \"Direct Download\", \"Local File\"]"
      ],
      "metadata": {
        "id": "URqOA-SGPYT1",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Lyrics\n",
        "\n",
        "Input between the `\"\"\"` lines the text of all the lyrics.\n",
        "\n",
        "This should include all repeated lines, as this exact text in order is what will be aligned.\n",
        "\n",
        "For best results you probably want to *remove* any `.`, `,`, `!`, and `?` characters.\n",
        "\n",
        "To control line breaks in the .ASS output, end each line with a `|` character."
      ],
      "metadata": {
        "id": "qiGJAn-wqVQU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IS-ef4o5s_tv"
      },
      "outputs": [],
      "source": [
        "text = \"\"\"\n",
        "Twinkle twinkle little star |\n",
        "How I wonder what you are |\n",
        "Up above the world so high |\n",
        "Like a diamond in this guy\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## .ASS generation settings"
      ],
      "metadata": {
        "id": "f8RUj6eP-s-4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "# @markdown To be honest, I don't know much about .ASS files, I'm just exposing\n",
        "# @markdown what the NeMo Forced Aligner script provides here.\n",
        "\n",
        "# @markdown ---\n",
        "fontsize = 20 # @param {type:\"number\"}\n",
        "verticalalign = \"center\" # @param [\"center\", \"bottom\"]\n",
        "# @markdown ---\n",
        "# @markdown > If `resegment` is checked, the ASS file will use new segments\n",
        "# @markdown such that each segment will not take up more than (approximately)\n",
        "# @markdown `maxlines` when the ASS file is applied to a video.\n",
        "resegment = False # @param {type:\"boolean\"}\n",
        "maxlines = 2 # @param {type:\"number\"}\n",
        "# @markdown ---\n",
        "# @markdown Color of text already sung:\n",
        "sung_R = 49 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "sung_G = 46 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "sung_B = 61 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "# @markdown ---\n",
        "# @markdown Color of text while it is sung:\n",
        "singing_R = 57 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "singing_G = 171 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "singing_B = 9 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "# @markdown ---\n",
        "# @markdown Color of text not yet sung:\n",
        "unsung_R = 194 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "unsung_G = 193 # @param {type:\"slider\", min:0, max:255, step:1}\n",
        "unsung_B = 199 # @param {type:\"slider\", min:0, max:255, step:1}"
      ],
      "metadata": {
        "id": "c0ev-uatuLpV",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "Below this line, you shouldn't have to alter anything unless you run into problems.\n",
        "\n",
        "---"
      ],
      "metadata": {
        "id": "eM9E0oj1ssxv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Installation\n",
        "This takes about 5 min to run, but you should only have to run it once per session.  Once it has run, to work on other files or tweak settings aand try again, you can just re-run the Configuration section and then the Execution section, skipping this section entirely."
      ],
      "metadata": {
        "id": "7q1PIMgWpgfc"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JDk9zxC6pdod"
      },
      "outputs": [],
      "source": [
        "# cython is required to install nemo toolkit\n",
        "!pip install cython\n",
        "\n",
        "# current nemo_toolkit version as of this notebook is 1.22, but it has a bug:\n",
        "# https://github.com/NVIDIA/NeMo/issues/8179\n",
        "Get_NeMo_Version = \"1.21\"\n",
        "!pip install nemo_toolkit[all]==$Get_NeMo_Version\n",
        "\n",
        "# still need the source though apparently\n",
        "BRANCH = f\"v{Get_NeMo_Version}.0\"\n",
        "!git clone -b $BRANCH https://github.com/NVIDIA/NeMo\n",
        "\n",
        "# need these items in order to downmix and resample to mono 16KHz\n",
        "!pip install ffmpeg pydub soundfile python-magic"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Execution"
      ],
      "metadata": {
        "id": "4JLDYupP2KiD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download Audio"
      ],
      "metadata": {
        "id": "9mRVS_Iap1-8"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sl34bUsUp_EV"
      },
      "outputs": [],
      "source": [
        "# first make a directory WORK_DIR that we will save all our new files in\n",
        "WORK_DIR=\"WORK_DIR\"\n",
        "!mkdir $WORK_DIR\n",
        "\n",
        "# name for the output file\n",
        "INPUT_FILE_NAME=f\"{WORK_DIR}/downloadedInput.unknown\" # it's easier this way because of gdown\n",
        "\n",
        "if get_audio_from == \"Google Drive\":\n",
        "  !pip install gdown\n",
        "  !gdown --fuzzy $google_drive_url --output $INPUT_FILE_NAME\n",
        "\n",
        "if get_audio_from == \"Direct Download\":\n",
        "  !wget --user-agent=\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36\" --no-check-certificate $direct_url --output-document=$INPUT_FILE_NAME\n",
        "\n",
        "if get_audio_from == \"Local File\":\n",
        "  !cp -v -T $local_file_path $INPUT_FILE_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prepare Audio for NFA\n",
        "\n",
        "It seems like this only works with mono 16KHz PCM, so we'll convert the input audio into that form."
      ],
      "metadata": {
        "id": "BP_q_Dgqp_iv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pydub import AudioSegment\n",
        "import magic\n",
        "import soundfile as sf\n",
        "\n",
        "def detect_audio_format(file_path):\n",
        "    mime_type = magic.Magic(mime=True).from_file(file_path)\n",
        "    if mime_type == 'audio/x-wav':\n",
        "        return 'wav'\n",
        "    elif mime_type == 'audio/flac':\n",
        "        return 'flac'\n",
        "    elif mime_type == 'audio/mpeg':\n",
        "        return 'mp3'\n",
        "    # Add more mime types as needed\n",
        "    else:\n",
        "        raise ValueError(\"Unsupported audio format\")\n",
        "\n",
        "format = detect_audio_format(INPUT_FILE_NAME)\n",
        "\n",
        "# Load the audio file\n",
        "sound = AudioSegment.from_file(INPUT_FILE_NAME, format=format)\n",
        "\n",
        "RESAMPLED_FILE_NAME = f\"{WORK_DIR}/resampledInput.wav\"\n",
        "# Downmix to mono if stereo\n",
        "if sound.channels == 2:\n",
        "    sound = sound.set_channels(1)\n",
        "\n",
        "# Resample to 16000 Hz if higher\n",
        "if sound.frame_rate > 16000:\n",
        "    sound = sound.set_frame_rate(16000)\n",
        "\n",
        "# Export the processed audio\n",
        "sound.export(RESAMPLED_FILE_NAME, format=\"wav\")"
      ],
      "metadata": {
        "id": "W4D0vi3S8L0M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGsMdjOsdBn6"
      },
      "source": [
        "## Prepare NFA Manifest"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OHq0trf2h24g"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "manifest_filepath = f\"{WORK_DIR}/manifest.json\"\n",
        "manifest_data = {\n",
        "    \"audio_filepath\": f\"{RESAMPLED_FILE_NAME}\",\n",
        "    \"text\": text\n",
        "}\n",
        "with open(manifest_filepath, 'w') as f:\n",
        "  line = json.dumps(manifest_data)\n",
        "  f.write(line + \"\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RvTrg8C3wHYw"
      },
      "outputs": [],
      "source": [
        "!cat $manifest_filepath"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKnTJG5Ig4lP"
      },
      "source": [
        "## Run NFA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QDcyfW_RWrqB"
      },
      "outputs": [],
      "source": [
        "!python NeMo/tools/nemo_forced_aligner/align.py \\\n",
        "  pretrained_name=\"stt_en_fastconformer_hybrid_large_pc\" \\\n",
        "  manifest_filepath=$manifest_filepath \\\n",
        "  output_dir=$WORK_DIR/nfa_output/ \\\n",
        "  additional_segment_grouping_separator=\"|\" \\\n",
        "  ass_file_config.fontsize=$fontsize \\\n",
        "  ass_file_config.vertical_alignment=$verticalalign \\\n",
        "  ass_file_config.resegment_text_to_fill_space=$resegment \\\n",
        "  ass_file_config.max_lines_per_segment=$maxlines \\\n",
        "  ass_file_config.text_already_spoken_rgb=[$sung_R,$sung_G,$sung_B] \\\n",
        "  ass_file_config.text_being_spoken_rgb=[$singing_R,$singing_G,$singing_B] \\\n",
        "  ass_file_config.text_not_yet_spoken_rgb=[$unsung_R,$unsung_G,$unsung_B]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Finished\n",
        "\n",
        "If this succeeded, you should now be able to go to the Files section on the left and find the appropriate files in `WORK_DIR/nfa_output`:\n",
        "\n",
        "* `ass/tokens/resampledInput.ass` should be a usable ASS subtitle file with sub-word-level timing\n",
        "\n",
        "* `ctm/tokens/resampledInput.ctm` can be imported into the `Recognize` tab [Kadda OK Tools](https://github.com/KaddaOK/KaddaOKTools/) to target Karaoke Builder Studio or YouTube Movie Maker ([coming soon](https://github.com/KaddaOK/KaddaOKTools/issues/4))\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "WBIwf7DVlJl3"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Av7L4Qzojykv"
      },
      "outputs": [],
      "source": [
        "!ls -R $WORK_DIR/nfa_output"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "7q1PIMgWpgfc",
        "9mRVS_Iap1-8",
        "BP_q_Dgqp_iv",
        "VGsMdjOsdBn6"
      ],
      "toc_visible": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}