2022_03_18-Baltruschat-Optuna_with_MLflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d58d114a-078e-46da-a337-4e8c727fdef2",
   "metadata": {},
   "source": [
    "# Track Optuna Hyperparameter Tuning with MLflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca7ce4b4-782e-447f-bf53-7065192f74b2",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Author:** Marcel Baltruschat (<a href=\"https://github.com/mrcblt\">@GitHub</a>)<br>\n",
    "**Date:** 18.03.2022<br>\n",
    "**License:** MIT\n",
    "<br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a312aa95-11a2-4fc8-b2ae-534a9189c8bb",
   "metadata": {},
   "source": [
    "### Installation with Conda"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69d9be1-5ebc-4332-af61-3fabdd9c6fe9",
   "metadata": {},
   "source": [
    "```\n",
    "conda create -n opt_ml -c conda-forge python=3.9 jupyterlab rdkit scikit-learn optuna mlflow ipywidgets multiprocess pytorch\n",
    "```\n",
    "Remark 1: `ipywidgets` is not directly used, but some imports trigger a warning if it is not installed<br>\n",
    "Remark 2: `multiprocess` is a fork of Python's `multiprocessing` module, that supports interactive usage of `Pool` on macOS.<br>\n",
    "Remark 3: On Windows, Pool multiprocessing in Jupyter Notebooks seems not to work and is disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a0a5b6e-1a00-4588-8c8a-b1b2156514e4",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "\n",
    "### Imports and Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "addd9c18-8890-454f-9283-4ee2917a5974",
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform\n",
    "OS = platform.system()\n",
    "\n",
    "if OS == 'Linux':\n",
    "    from multiprocessing import Pool\n",
    "elif OS == 'Darwin':\n",
    "    from multiprocess import Pool\n",
    "\n",
    "import sys\n",
    "import warnings\n",
    "from subprocess import Popen\n",
    "\n",
    "import mlflow\n",
    "import numpy as np\n",
    "import optuna\n",
    "import pandas as pd\n",
    "import rdkit\n",
    "import torch\n",
    "from rdkit.Chem import AllChem as Chem, Descriptors, Crippen\n",
    "from rdkit.DataStructs import ConvertToNumpyArray\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import cohen_kappa_score, ConfusionMatrixDisplay\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from torch import nn, optim\n",
    "from torch.utils.data import DataLoader, Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "429de7b0-b916-4d74-866e-edc9fef5cad0",
   "metadata": {},
   "outputs": [],
   "source": [
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "random_seed = 42\n",
    "num_cores = 12"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e1f993-d11f-4cf8-967e-af69472762d0",
   "metadata": {},
   "source": [
    "#### Used Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e5cc7891-cc7a-4fd0-b0fa-fa6b5ea3b040",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python: 3.9.10 \n",
      "MLflow: 1.24.0\n",
      "Optuna: 2.10.0\n",
      "RDKit: 2021.09.5\n",
      "PyTorch: 1.10.2\n"
     ]
    }
   ],
   "source": [
    "print(f'Python: {sys.version.split(\"|\")[0]}\\nMLflow: {mlflow.__version__}\\nOptuna: {optuna.__version__}\\nRDKit: {rdkit.__version__}\\nPyTorch: {torch.__version__}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01d45c7f-a37e-415d-a115-1b9726df3918",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "#### Start MLflow Server\n",
    "It get's terminated by stopping / restarting / interrupting the kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "37bc4eec-d4e7-4d48-8c4a-7792f8f2be48",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2022-03-18 09:02:29 +0100] [48103] [INFO] Starting gunicorn 20.1.0\n",
      "[2022-03-18 09:02:29 +0100] [48103] [INFO] Listening at: http://127.0.0.1:5000 (48103)\n",
      "[2022-03-18 09:02:29 +0100] [48103] [INFO] Using worker: sync\n",
      "[2022-03-18 09:02:29 +0100] [48104] [INFO] Booting worker with pid: 48104\n"
     ]
    }
   ],
   "source": [
    "# Adjust host and port as necessary\n",
    "mlflow_proc = Popen(['mlflow', 'ui'])  # '-h', '0.0.0.0', '-p', '8891'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61c69d8c-29b6-448a-b45c-27aa2dd1abcb",
   "metadata": {},
   "source": [
    "You can already visit http://localhost:5000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612015ef-5a57-4ed4-9469-4a09e5093da0",
   "metadata": {},
   "source": [
    "---\n",
    "### Loading Example Dataset\n",
    "\n",
    "The original dataset was published by **Ogura _et al._ (2019)** [1].<br>\n",
    "The only changes made were the conversion from XLSX to CSV and the filtering out of all molecules with invalid valences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e229ede-35c9-4dba-9561-e28efda715df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "190464\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smi</th>\n",
       "      <th>act</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CCOC(=O)C1CCN(CC1)C(C)C(=O)c2c(C)[nH]c3cc(C)ccc23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>COc1ccc(NC(=O)N2CCC3(CCN(C)CC3)CC2)cc1F</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 smi  act\n",
       "0  CCOC(=O)C1CCN(CC1)C(C)C(=O)c2c(C)[nH]c3cc(C)ccc23    1\n",
       "1            COc1ccc(NC(=O)N2CCC3(CCN(C)CC3)CC2)cc1F    1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('datasets/example_dataset_hERG.csv', names=['smi', 'act'], header=0)\n",
    "print(len(df))\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e2abd231-9725-4094-a7a9-6c7dbe1b546e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    184044\n",
       "1      6420\n",
       "Name: act, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.act.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142dc9ec-a563-4c8a-a389-cacd79741909",
   "metadata": {},
   "source": [
    "=> *Very unbalanced dataset*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5453bc41-de63-4b3b-bafa-5b2eee37b477",
   "metadata": {},
   "source": [
    "### Perform Undersampling Based on MolWt and LogP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "20802e61-15ef-4bb6-934e-9552cf9ba5f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_from_smi(smi):\n",
    "    mol = Chem.MolFromSmiles(smi)\n",
    "    mw_logp = Descriptors.MolWt(mol) / 100 * Crippen.MolLogP(mol)\n",
    "    return mol, mw_logp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3f96396a-edf5-48fe-9bee-14e1c78b8ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "if OS == 'Windows':\n",
    "    res = np.array(list(map(data_from_smi, df.smi)))\n",
    "else:\n",
    "    with Pool(num_cores) as p:\n",
    "        res = np.array(p.map(data_from_smi, df.smi))\n",
    "df['ROMol'], df['mw_logp'] = res[:, 0], res[:, 1]\n",
    "df.sort_values('mw_logp', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "58cf85aa-9239-4098-b7ea-f829ed8a5b80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    6420\n",
       "0    6420\n",
       "Name: act, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "act0 = df.query('act == 0').reset_index(drop=True)\n",
    "ix = np.linspace(0, len(act0) - 1, num=len(df) - len(act0), dtype=np.int32)\n",
    "df = pd.concat([df.query('act == 1'), act0.loc[ix]])\n",
    "df.act.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33f4e34b-a1ae-4698-9415-7b11a5aca650",
   "metadata": {},
   "source": [
    "### Calculating Morgan Fingerprints (FCFP6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bd2c9650-f520-4dc5-80e4-606ca78524d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mol_to_FCFP6(mol):\n",
    "    fp = Chem.GetMorganFingerprintAsBitVect(mol, radius=3, nBits=2048, useFeatures=True)\n",
    "    ar = np.empty(2048, dtype=np.uint8)\n",
    "    ConvertToNumpyArray(fp, ar)\n",
    "    return ar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1aec2ee9-92d9-4a13-8bd1-b903e719c7ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12840, 2048)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_data = np.array(list(map(mol_to_FCFP6, df.ROMol)))\n",
    "x_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9854e7b-cb95-454a-a680-be718724b2b3",
   "metadata": {},
   "source": [
    "### Split Into Training and Test Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "797ef067-e16c-4696-a38f-44516e26ad0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_data = np.array(df.act)\n",
    "x_train, x_test, y_train, y_test = train_test_split(x_data, y_data,\n",
    "                                                    test_size=0.1,\n",
    "                                                    stratify=y_data,\n",
    "                                                    random_state=random_seed,\n",
    "                                                    shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60aa8ae5-7961-4517-8ebc-000f80e84a37",
   "metadata": {},
   "source": [
    "---\n",
    "### Optimize models with Optuna and Track Results with MLflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "20c66ba7-fb89-4477-9d3e-b499be07c5e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Activate autologging for Scikit-learn\n",
    "mlflow.sklearn.autolog()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ffc7e85-813a-4920-9228-dc1d7c7842f7",
   "metadata": {},
   "source": [
    "#### Scikit-Learn Random Forest (with auto logging)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9a0e42b-1ff7-49a3-ab58-237ec2605442",
   "metadata": {},
   "source": [
    "**If you want to avoid having duplicated parameter sets you need to uncomment the commented code lines**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "06b18992-01e6-49a8-8834-e25739fc8e39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# seen_param = []\n",
    "\n",
    "def rf_obj(trial):\n",
    "    param = {\n",
    "        'n_estimators': trial.suggest_int('n_estimators', 50, 1000, step=50),\n",
    "        'max_depth':    trial.suggest_int('max_depth', 1, 20),\n",
    "        'criterion':    trial.suggest_categorical('criterion', ['gini', 'entropy']),\n",
    "        'max_features': trial.suggest_categorical('max_features', ['auto', 'sqrt', 'log2', None]),\n",
    "        'random_state': random_seed,\n",
    "        'n_jobs':       num_cores,\n",
    "    }\n",
    "    # if param in seen_param:\n",
    "    #     raise optuna.exceptions.TrialPruned()\n",
    "    # else:\n",
    "    #     seen_param.append(param)\n",
    "    metrics = {}\n",
    "    model = RandomForestClassifier(**param)\n",
    "    with mlflow.start_run():\n",
    "        model.fit(x_train, y_train)\n",
    "        metrics['test_kappa'] = cohen_kappa_score(y_test, model.predict(x_test))\n",
    "        metrics['training_kappa'] = cohen_kappa_score(y_train, model.predict(x_train))\n",
    "        mlflow.log_metrics(metrics)\n",
    "    return metrics['test_kappa']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "548dd9cf-4738-4922-bd07-05f8821e8471",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022/03/18 09:03:12 INFO mlflow.tracking.fluent: Experiment with name 'hERG Random Forest' does not exist. Creating a new experiment.\n"
     ]
    }
   ],
   "source": [
    "# Creates a new MLflow experiment and set it as active\n",
    "mlflow.set_experiment('hERG Random Forest')\n",
    "\n",
    "# Creates a new Optuna study for maximizing an outcome\n",
    "study = optuna.create_study(direction='maximize')\n",
    "\n",
    "# MLflow currently uses scikit-learn functions for metric calculation that were deprecated with version 1.0\n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    # Optimize the objective function\n",
    "    study.optimize(rf_obj, n_trials=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92de596c-6e07-40b3-9b2c-b2642bb90039",
   "metadata": {},
   "source": [
    "#### Scikit-Learn MLP (with auto logging)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bbb3a7c9-bc04-4208-8bf9-3315d5d5ef5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# seen_param = []\n",
    "\n",
    "def nn_obj(trial):\n",
    "    n_hidden_layers = trial.suggest_int('n_hidden_layers', 0, 4)\n",
    "    n_neurons = trial.suggest_int('n_neurons', 16, 128) if n_hidden_layers > 0 else 0\n",
    "    param = {\n",
    "        'hidden_layer_sizes': [n_neurons] * n_hidden_layers,\n",
    "        'activation':         trial.suggest_categorical('activation', ['identity', 'logistic', 'tanh', 'relu']),\n",
    "        'solver':             trial.suggest_categorical('solver', ['lbfgs', 'sgd', 'adam']),\n",
    "        'alpha':              trial.suggest_float('alpha', 0.00001, 0.1, log=True),\n",
    "        'learning_rate_init': trial.suggest_float('learning_rate', 0.00001, 0.1, log=True),\n",
    "        'max_iter':           trial.suggest_int('epochs', 20, 300),\n",
    "        'random_state':       random_seed,\n",
    "    }\n",
    "    # if param in seen_param:\n",
    "    #     raise optuna.exceptions.TrialPruned()\n",
    "    # else:\n",
    "    #     seen_param.append(param)\n",
    "    metrics = {}\n",
    "    model = MLPClassifier(**param)\n",
    "    with mlflow.start_run():\n",
    "        mlflow.log_params(dict(n_hidden_layers=n_hidden_layers, n_neurons=n_neurons))\n",
    "        model.fit(x_train, y_train)\n",
    "        metrics['test_kappa'] = cohen_kappa_score(y_test, model.predict(x_test))\n",
    "        metrics['training_kappa'] = cohen_kappa_score(y_train, model.predict(x_train))\n",
    "        mlflow.log_metrics(metrics)\n",
    "    return metrics['test_kappa']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b39ce272-3501-436e-9f21-607986a867ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022/03/18 19:38:15 INFO mlflow.tracking.fluent: Experiment with name 'hERG MLP' does not exist. Creating a new experiment.\n"
     ]
    }
   ],
   "source": [
    "# Creates a new MLflow experiment and set it as active\n",
    "mlflow.set_experiment('hERG MLP')\n",
    "\n",
    "# Creates a new Optuna study for maximizing an outcome\n",
    "study = optuna.create_study(direction='maximize')\n",
    "\n",
    "# MLflow currently uses scikit-learn functions for metric calculation that were deprecated with version 1.0\n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    # Optimize the objective function\n",
    "    study.optimize(nn_obj, n_trials=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9f6d84d-fd77-4a7f-8e90-6f2f504b7c8e",
   "metadata": {},
   "source": [
    "#### PyTorch MLP (manual logging)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "790ee5a6-898f-43c5-b742-0dbb600bddc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "class hERGDataset(Dataset):\n",
    "    def __init__(self, x_data, y_data):\n",
    "        self.x_data = x_data.astype(np.float32)\n",
    "        self.y_data = y_data.astype(np.float32).reshape(-1, 1)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.y_data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.x_data[idx], self.y_data[idx]\n",
    "\n",
    "\n",
    "train_ds = hERGDataset(x_train, y_train)\n",
    "test_ds = hERGDataset(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bdf73da4-0e49-4fcc-8f9b-f64e82160063",
   "metadata": {},
   "outputs": [],
   "source": [
    "# seen_param = []\n",
    "\n",
    "def pt_nn_obj(trial):\n",
    "    param = dict(\n",
    "        n_hidden_layers = trial.suggest_int('n_hidden_layers', 0, 3),\n",
    "        n_neurons       = trial.suggest_int('n_neurons', 16, 128),\n",
    "        act             = trial.suggest_categorical('activation', ['Sigmoid', 'Tanh', 'ReLU']),\n",
    "        lr              = trial.suggest_float('learning_rate', 0.00001, 0.1, log=True),\n",
    "        epochs          = trial.suggest_int('epochs', 20, 300),\n",
    "        batch_size      = trial.suggest_int('batch_size', 1, 128),\n",
    "        random_seed     = random_seed,\n",
    "        optimizer       = 'Adam',\n",
    "        criterion       = 'binary_crossentropy',\n",
    "    )\n",
    "    if param['n_hidden_layers'] == 0:\n",
    "        param['n_neurons'] = 0\n",
    "    # if param in seen_param:\n",
    "    #     raise optuna.exceptions.TrialPruned()\n",
    "    # else:\n",
    "    #     seen_param.append(param)\n",
    "\n",
    "    torch.manual_seed(param['random_seed'])\n",
    "    np.random.seed(param['random_seed'])\n",
    "\n",
    "    act_func = eval(f'nn.{param[\"act\"]}')\n",
    "\n",
    "    layers = [nn.Linear(2048, param['n_neurons']), act_func()]  # input layer\n",
    "    for i in range(param['n_hidden_layers']):\n",
    "        layers.append(nn.Linear(param['n_neurons'], param['n_neurons']))\n",
    "        layers.append(act_func())\n",
    "    layers.append(nn.Linear(param['n_neurons'], 1))  # output layer\n",
    "    layers.append(nn.Sigmoid())\n",
    "    model = nn.Sequential(*layers)\n",
    "\n",
    "    opt = optim.Adam(model.parameters(), lr=param['lr'])\n",
    "    crit = nn.BCELoss()\n",
    "\n",
    "    train_loader = DataLoader(train_ds, batch_size=param['batch_size'], shuffle=True)\n",
    "    test_loader = DataLoader(test_ds, batch_size=param['batch_size'], shuffle=False)\n",
    "\n",
    "    with mlflow.start_run():\n",
    "        mlflow.log_params(param)\n",
    "        for i in range(param['epochs']):\n",
    "            model.train()\n",
    "            train_loss = 0\n",
    "            train_kappa = 0\n",
    "            for data, labels in train_loader:\n",
    "                opt.zero_grad()\n",
    "                out = model(data)\n",
    "                loss = crit(out, labels)\n",
    "                loss.backward()\n",
    "                opt.step()\n",
    "                train_loss += loss.item()\n",
    "                train_kappa += cohen_kappa_score(labels.data.numpy(), out[:,-1].detach().numpy().round())\n",
    "\n",
    "            model.eval()\n",
    "            metrics = dict(train_loss=train_loss / len(train_loader), train_kappa=train_kappa / len(train_loader))\n",
    "            with torch.no_grad():\n",
    "                test_loss = 0\n",
    "                test_kappa = 0\n",
    "                for data, labels in test_loader:\n",
    "                    out = model(data)\n",
    "                    test_loss += crit(out, labels).item()\n",
    "                    test_kappa += cohen_kappa_score(labels.data.numpy(), out[:,-1].numpy().round())\n",
    "            metrics['test_loss'] = test_loss / len(test_loader)\n",
    "            metrics['test_kappa'] = test_kappa / len(test_loader)\n",
    "            mlflow.log_metrics(metrics, step=i + 1)\n",
    "        mlflow.pytorch.log_model(model, 'model')\n",
    "    return metrics['test_kappa']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1a376a70-ef6f-4b8a-af4e-82cc1469f4a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022/03/18 20:17:35 INFO mlflow.tracking.fluent: Experiment with name 'hERG PyTorch NN' does not exist. Creating a new experiment.\n",
      "\u001b[33m[W 2022-03-18 20:47:08,936]\u001b[0m Trial 20 failed, because the objective function returned nan.\u001b[0m\n",
      "\u001b[33m[W 2022-03-18 21:12:11,468]\u001b[0m Trial 28 failed, because the objective function returned nan.\u001b[0m\n",
      "\u001b[33m[W 2022-03-18 21:22:34,855]\u001b[0m Trial 35 failed, because the objective function returned nan.\u001b[0m\n",
      "\u001b[33m[W 2022-03-18 21:34:25,375]\u001b[0m Trial 38 failed, because the objective function returned nan.\u001b[0m\n",
      "\u001b[33m[W 2022-03-18 21:41:10,054]\u001b[0m Trial 42 failed, because the objective function returned nan.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Creates a new MLflow experiment and set it as active\n",
    "mlflow.set_experiment('hERG PyTorch NN')\n",
    "\n",
    "# Creates a new Optuna study for maximizing an outcome\n",
    "study = optuna.create_study(direction='maximize')\n",
    "\n",
    "# MLflow currently uses scikit-learn functions for metric calculation that were deprecated with version 1.0\n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    # Optimize the objective function\n",
    "    study.optimize(pt_nn_obj, n_trials=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1d7e3cd-f2ac-47ed-805a-c63bc8b57396",
   "metadata": {},
   "source": [
    "### While Optuna is \"studying\" you can already investigate finished results on the MLflow server webpage...\n",
    "http://localhost:5000\n",
    "\n",
    "All results that are shown on the MLflow page are retrieved from the local folder `mlruns` in the current directory."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cca1bcb-6277-405e-9b94-613c43403b1e",
   "metadata": {},
   "source": [
    "---\n",
    "### References\n",
    "[1] Ogura, K., Sato, T., Yuki, H. et al. Support Vector Machine model for hERG inhibitory activities based on the integrated hERG database using descriptor selection by NSGA-II. Sci Rep 9, 12220 (2019). https://doi.org/10.1038/s41598-019-47536-3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e70dce-4b14-4d7a-b4e2-a02f91fc3d3e",
   "metadata": {},
   "source": [
    "### Useful Resources\n",
    "- https://optuna.org/\n",
    "- https://mlflow.org/docs/latest/tracking.html\n",
    "- https://pytorch.org\n",
    "- [Optuna Introduction Notebook by Aishvarya Tandon](https://github.com/czodrowskilab/5minfame/blob/main/2022-02-04-Tandon-QuickIntroOptuna.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65b2a6c0-443c-4da6-815c-6a92a8a961aa",
   "metadata": {},
   "source": [
    "### Disclaimer\n",
    "The configurations for modelling and hyperparamter optimization might be suboptimal for the specific task and dataset. Since the key point of this notebook is to show the usage of the MLflow tracking feature together with Optuna, configuration optimisation was neglected."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "4526764c2cf04ff1a6185f79ba71bc3b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "59ce0b799ce146d08954d62f49cb7f55": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "7f1195620f954dcd9e91cbe70801035b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DropdownModel",
      "state": {
       "_options_labels": [
        "0",
        "1",
        "2",
        "3",
        "4",
        "5",
        "6",
        "7",
        "8",
        "9",
        "10",
        "11",
        "12",
        "13",
        "14",
        "15",
        "16",
        "17",
        "18",
        "19",
        "20",
        "21",
        "22",
        "23",
        "24",
        "25",
        "26",
        "27",
        "28",
        "29",
        "30",
        "31",
        "32",
        "33",
        "34",
        "35",
        "36",
        "37"
       ],
       "description": "test_data_id",
       "index": 5,
       "layout": "IPY_MODEL_e5d917d6e4f24f72a014d6ff649ae3f7",
       "style": "IPY_MODEL_59ce0b799ce146d08954d62f49cb7f55"
      }
     },
     "81ed02b2b9d14752987f816c2b06b09c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b5504e3eeff3468387c876e2823834c9": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_4526764c2cf04ff1a6185f79ba71bc3b",
       "outputs": [
        {
         "name": "stdout",
         "output_type": "stream",
         "text": "\nLeft: GNNExplainer (Threshold: 0.7)\nRight: PyTorch Captum (Threshold: 0.6)\nTrue Class: 0\n\n"
        },
        {
         "data": {
          "text/html": "<div style=\"white-space: nowrap\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:rdkit=\"http://www.rdkit.org/xml\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" version=\"1.1\" baseProfile=\"full\" xml:space=\"preserve\" width=\"500px\" height=\"300px\" viewBox=\"0 0 500 300\">\n<!-- END OF HEADER -->\n<rect style=\"opacity:1.0;fill:#FFFFFF;stroke:none\" width=\"500.0\" height=\"300.0\" x=\"0.0\" y=\"0.0\"> </rect>\n<path class=\"bond-0 atom-0 atom-1\" d=\"M 293.0,252.5 L 206.7,265.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-0 atom-0 atom-1\" d=\"M 277.5,237.2 L 217.1,246.0\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-1 atom-0 atom-5\" d=\"M 293.0,252.5 L 325.2,171.5\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-1 atom-0 atom-5\" d=\"M 281.6,233.9 L 304.2,177.2\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-2 atom-1 atom-2\" d=\"M 206.7,265.1 L 152.7,196.7\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-2 atom-1 atom-2\" d=\"M 212.3,244.1 L 174.5,196.2\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-3 atom-2 atom-3\" d=\"M 152.7,196.7 L 184.9,115.7\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-3 atom-2 atom-3\" d=\"M 173.7,191.0 L 196.3,134.3\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-4 atom-2 atom-10\" d=\"M 152.7,196.7 L 115.3,202.2\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-4 atom-2 atom-10\" d=\"M 115.3,202.2 L 78.0,207.7\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-5 atom-3 atom-4\" d=\"M 184.9,115.7 L 271.2,103.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-5 atom-3 atom-4\" d=\"M 200.4,131.1 L 260.7,122.3\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-6 atom-3 atom-9\" d=\"M 184.9,115.7 L 163.8,89.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-6 atom-3 atom-9\" d=\"M 163.8,89.1 L 142.7,62.4\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-7 atom-4 atom-5\" d=\"M 271.2,103.1 L 325.2,171.5\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-7 atom-4 atom-5\" d=\"M 265.6,124.2 L 303.4,172.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-8 atom-5 atom-6\" d=\"M 325.2,171.5 L 362.5,166.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-8 atom-5 atom-6\" d=\"M 362.5,166.1 L 399.9,160.6\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 416.1,178.9 L 431.8,198.7\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 431.8,198.7 L 447.4,218.5\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 429.8,168.0 L 445.4,187.9\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 445.4,187.9 L 461.1,207.7\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-10 atom-6 atom-8\" d=\"M 417.2,144.4 L 427.5,118.5\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-10 atom-6 atom-8\" d=\"M 427.5,118.5 L 437.8,92.6\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"atom-6\" d=\"M 406.0 146.6 L 414.1 159.7 Q 414.9 161.0, 416.2 163.3 Q 417.5 165.6, 417.5 165.8 L 417.5 146.6 L 420.8 146.6 L 420.8 171.3 L 417.4 171.3 L 408.7 157.0 Q 407.7 155.3, 406.7 153.4 Q 405.6 151.5, 405.3 150.9 L 405.3 171.3 L 402.1 171.3 L 402.1 146.6 L 406.0 146.6 \" fill=\"#0000FF\"/>\n<path class=\"atom-7\" d=\"M 454.2 227.4 Q 454.2 221.5, 457.1 218.2 Q 460.0 214.8, 465.5 214.8 Q 471.0 214.8, 473.9 218.2 Q 476.8 221.5, 476.8 227.4 Q 476.8 233.4, 473.9 236.8 Q 470.9 240.2, 465.5 240.2 Q 460.1 240.2, 457.1 236.8 Q 454.2 233.4, 454.2 227.4 M 465.5 237.4 Q 469.3 237.4, 471.3 234.9 Q 473.3 232.3, 473.3 227.4 Q 473.3 222.5, 471.3 220.1 Q 469.3 217.6, 465.5 217.6 Q 461.7 217.6, 459.7 220.1 Q 457.6 222.5, 457.6 227.4 Q 457.6 232.4, 459.7 234.9 Q 461.7 237.4, 465.5 237.4 \" fill=\"#FF0000\"/>\n<path class=\"atom-8\" d=\"M 432.3 78.0 Q 432.3 72.1, 435.3 68.8 Q 438.2 65.4, 443.7 65.4 Q 449.1 65.4, 452.1 68.8 Q 455.0 72.1, 455.0 78.0 Q 455.0 84.0, 452.0 87.4 Q 449.1 90.8, 443.7 90.8 Q 438.2 90.8, 435.3 87.4 Q 432.3 84.0, 432.3 78.0 M 443.7 88.0 Q 447.4 88.0, 449.4 85.5 Q 451.5 82.9, 451.5 78.0 Q 451.5 73.2, 449.4 70.7 Q 447.4 68.2, 443.7 68.2 Q 439.9 68.2, 437.8 70.7 Q 435.8 73.1, 435.8 78.0 Q 435.8 83.0, 437.8 85.5 Q 439.9 88.0, 443.7 88.0 \" fill=\"#FF0000\"/>\n<path class=\"atom-8\" d=\"M 458.0 65.7 L 461.3 65.7 L 461.3 76.2 L 473.9 76.2 L 473.9 65.7 L 477.3 65.7 L 477.3 90.4 L 473.9 90.4 L 473.9 79.0 L 461.3 79.0 L 461.3 90.4 L 458.0 90.4 L 458.0 65.7 \" fill=\"#FF0000\"/>\n<path class=\"atom-9\" d=\"M 98.5 35.1 L 101.8 35.1 L 101.8 45.6 L 114.4 45.6 L 114.4 35.1 L 117.8 35.1 L 117.8 59.8 L 114.4 59.8 L 114.4 48.4 L 101.8 48.4 L 101.8 59.8 L 98.5 59.8 L 98.5 35.1 \" fill=\"#FF0000\"/>\n<path class=\"atom-9\" d=\"M 119.5 47.4 Q 119.5 41.5, 122.4 38.2 Q 125.4 34.9, 130.9 34.9 Q 136.3 34.9, 139.3 38.2 Q 142.2 41.5, 142.2 47.4 Q 142.2 53.4, 139.2 56.8 Q 136.3 60.2, 130.9 60.2 Q 125.4 60.2, 122.4 56.8 Q 119.5 53.4, 119.5 47.4 M 130.9 57.4 Q 134.6 57.4, 136.6 54.9 Q 138.7 52.4, 138.7 47.4 Q 138.7 42.6, 136.6 40.1 Q 134.6 37.7, 130.9 37.7 Q 127.1 37.7, 125.0 40.1 Q 123.0 42.5, 123.0 47.4 Q 123.0 52.4, 125.0 54.9 Q 127.1 57.4, 130.9 57.4 \" fill=\"#FF0000\"/>\n<path class=\"atom-10\" d=\"M 22.7 197.0 L 26.1 197.0 L 26.1 207.5 L 38.7 207.5 L 38.7 197.0 L 42.0 197.0 L 42.0 221.7 L 38.7 221.7 L 38.7 210.3 L 26.1 210.3 L 26.1 221.7 L 22.7 221.7 L 22.7 197.0 \" fill=\"#0000FF\"/>\n<path class=\"atom-10\" d=\"M 43.2 220.8 Q 43.8 219.3, 45.3 218.4 Q 46.7 217.6, 48.7 217.6 Q 51.1 217.6, 52.5 218.9 Q 53.9 220.2, 53.9 222.6 Q 53.9 225.0, 52.1 227.3 Q 50.3 229.5, 46.6 232.2 L 54.2 232.2 L 54.2 234.0 L 43.2 234.0 L 43.2 232.5 Q 46.2 230.3, 48.0 228.7 Q 49.8 227.1, 50.7 225.7 Q 51.6 224.2, 51.6 222.7 Q 51.6 221.1, 50.8 220.3 Q 50.0 219.4, 48.7 219.4 Q 47.4 219.4, 46.5 219.9 Q 45.6 220.5, 45.0 221.6 L 43.2 220.8 \" fill=\"#0000FF\"/>\n<path class=\"atom-10\" d=\"M 61.0 197.0 L 69.1 210.1 Q 69.9 211.4, 71.2 213.7 Q 72.4 216.0, 72.5 216.2 L 72.5 197.0 L 75.8 197.0 L 75.8 221.7 L 72.4 221.7 L 63.7 207.4 Q 62.7 205.7, 61.6 203.8 Q 60.6 201.9, 60.3 201.3 L 60.3 221.7 L 57.1 221.7 L 57.1 197.0 L 61.0 197.0 \" fill=\"#0000FF\"/>\n</svg><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:rdkit=\"http://www.rdkit.org/xml\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" version=\"1.1\" baseProfile=\"full\" xml:space=\"preserve\" width=\"500px\" height=\"300px\" viewBox=\"0 0 500 300\">\n<!-- END OF HEADER -->\n<rect style=\"opacity:1.0;fill:#FFFFFF;stroke:none\" width=\"500.0\" height=\"300.0\" x=\"0.0\" y=\"0.0\"> </rect>\n<path d=\"M 293.0,252.5 L 206.7,265.1\" style=\"fill:none;fill-rule:evenodd;stroke:#FF000080;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 293.0,252.5 L 325.2,171.5\" style=\"fill:none;fill-rule:evenodd;stroke:#FF00006A;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 206.7,265.1 L 152.7,196.7\" style=\"fill:none;fill-rule:evenodd;stroke:#FF00007B;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 152.7,196.7 L 184.9,115.7\" style=\"fill:none;fill-rule:evenodd;stroke:#FF000069;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 152.7,196.7 L 66.4,209.3\" style=\"fill:none;fill-rule:evenodd;stroke:#FF00006D;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 184.9,115.7 L 271.2,103.1\" style=\"fill:none;fill-rule:evenodd;stroke:#FF00006E;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 271.2,103.1 L 325.2,171.5\" style=\"fill:none;fill-rule:evenodd;stroke:#FF00006B;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 411.4,158.9 L 465.5,227.3\" style=\"fill:none;fill-rule:evenodd;stroke:#FF000069;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path d=\"M 411.4,158.9 L 443.7,77.9\" style=\"fill:none;fill-rule:evenodd;stroke:#FF000069;stroke-width:18.6px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-0 atom-0 atom-1\" d=\"M 293.0,252.5 L 206.7,265.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-0 atom-0 atom-1\" d=\"M 277.5,237.2 L 217.1,246.0\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-1 atom-0 atom-5\" d=\"M 293.0,252.5 L 325.2,171.5\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-1 atom-0 atom-5\" d=\"M 281.6,233.9 L 304.2,177.2\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-2 atom-1 atom-2\" d=\"M 206.7,265.1 L 152.7,196.7\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-2 atom-1 atom-2\" d=\"M 212.3,244.1 L 174.5,196.2\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-3 atom-2 atom-3\" d=\"M 152.7,196.7 L 184.9,115.7\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-3 atom-2 atom-3\" d=\"M 173.7,191.0 L 196.3,134.3\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-4 atom-2 atom-10\" d=\"M 152.7,196.7 L 115.3,202.2\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-4 atom-2 atom-10\" d=\"M 115.3,202.2 L 78.0,207.7\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-5 atom-3 atom-4\" d=\"M 184.9,115.7 L 271.2,103.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-5 atom-3 atom-4\" d=\"M 200.4,131.1 L 260.7,122.3\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-6 atom-3 atom-9\" d=\"M 184.9,115.7 L 163.8,89.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-6 atom-3 atom-9\" d=\"M 163.8,89.1 L 142.7,62.4\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-7 atom-4 atom-5\" d=\"M 271.2,103.1 L 325.2,171.5\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-7 atom-4 atom-5\" d=\"M 265.6,124.2 L 303.4,172.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:6,6\"/>\n<path class=\"bond-8 atom-5 atom-6\" d=\"M 325.2,171.5 L 362.5,166.1\" style=\"fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-8 atom-5 atom-6\" d=\"M 362.5,166.1 L 399.9,160.6\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 416.1,178.9 L 431.8,198.7\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 431.8,198.7 L 447.4,218.5\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 429.8,168.0 L 445.4,187.9\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-9 atom-6 atom-7\" d=\"M 445.4,187.9 L 461.1,207.7\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-10 atom-6 atom-8\" d=\"M 417.2,144.4 L 427.5,118.5\" style=\"fill:none;fill-rule:evenodd;stroke:#0000FF;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"bond-10 atom-6 atom-8\" d=\"M 427.5,118.5 L 437.8,92.6\" style=\"fill:none;fill-rule:evenodd;stroke:#FF0000;stroke-width:2.0px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1\"/>\n<path class=\"atom-6\" d=\"M 406.0 146.6 L 414.1 159.7 Q 414.9 161.0, 416.2 163.3 Q 417.5 165.6, 417.5 165.8 L 417.5 146.6 L 420.8 146.6 L 420.8 171.3 L 417.4 171.3 L 408.7 157.0 Q 407.7 155.3, 406.7 153.4 Q 405.6 151.5, 405.3 150.9 L 405.3 171.3 L 402.1 171.3 L 402.1 146.6 L 406.0 146.6 \" fill=\"#0000FF\"/>\n<path class=\"atom-7\" d=\"M 454.2 227.4 Q 454.2 221.5, 457.1 218.2 Q 460.0 214.8, 465.5 214.8 Q 471.0 214.8, 473.9 218.2 Q 476.8 221.5, 476.8 227.4 Q 476.8 233.4, 473.9 236.8 Q 470.9 240.2, 465.5 240.2 Q 460.1 240.2, 457.1 236.8 Q 454.2 233.4, 454.2 227.4 M 465.5 237.4 Q 469.3 237.4, 471.3 234.9 Q 473.3 232.3, 473.3 227.4 Q 473.3 222.5, 471.3 220.1 Q 469.3 217.6, 465.5 217.6 Q 461.7 217.6, 459.7 220.1 Q 457.6 222.5, 457.6 227.4 Q 457.6 232.4, 459.7 234.9 Q 461.7 237.4, 465.5 237.4 \" fill=\"#FF0000\"/>\n<path class=\"atom-8\" d=\"M 432.3 78.0 Q 432.3 72.1, 435.3 68.8 Q 438.2 65.4, 443.7 65.4 Q 449.1 65.4, 452.1 68.8 Q 455.0 72.1, 455.0 78.0 Q 455.0 84.0, 452.0 87.4 Q 449.1 90.8, 443.7 90.8 Q 438.2 90.8, 435.3 87.4 Q 432.3 84.0, 432.3 78.0 M 443.7 88.0 Q 447.4 88.0, 449.4 85.5 Q 451.5 82.9, 451.5 78.0 Q 451.5 73.2, 449.4 70.7 Q 447.4 68.2, 443.7 68.2 Q 439.9 68.2, 437.8 70.7 Q 435.8 73.1, 435.8 78.0 Q 435.8 83.0, 437.8 85.5 Q 439.9 88.0, 443.7 88.0 \" fill=\"#FF0000\"/>\n<path class=\"atom-8\" d=\"M 458.0 65.7 L 461.3 65.7 L 461.3 76.2 L 473.9 76.2 L 473.9 65.7 L 477.3 65.7 L 477.3 90.4 L 473.9 90.4 L 473.9 79.0 L 461.3 79.0 L 461.3 90.4 L 458.0 90.4 L 458.0 65.7 \" fill=\"#FF0000\"/>\n<path class=\"atom-9\" d=\"M 98.5 35.1 L 101.8 35.1 L 101.8 45.6 L 114.4 45.6 L 114.4 35.1 L 117.8 35.1 L 117.8 59.8 L 114.4 59.8 L 114.4 48.4 L 101.8 48.4 L 101.8 59.8 L 98.5 59.8 L 98.5 35.1 \" fill=\"#FF0000\"/>\n<path class=\"atom-9\" d=\"M 119.5 47.4 Q 119.5 41.5, 122.4 38.2 Q 125.4 34.9, 130.9 34.9 Q 136.3 34.9, 139.3 38.2 Q 142.2 41.5, 142.2 47.4 Q 142.2 53.4, 139.2 56.8 Q 136.3 60.2, 130.9 60.2 Q 125.4 60.2, 122.4 56.8 Q 119.5 53.4, 119.5 47.4 M 130.9 57.4 Q 134.6 57.4, 136.6 54.9 Q 138.7 52.4, 138.7 47.4 Q 138.7 42.6, 136.6 40.1 Q 134.6 37.7, 130.9 37.7 Q 127.1 37.7, 125.0 40.1 Q 123.0 42.5, 123.0 47.4 Q 123.0 52.4, 125.0 54.9 Q 127.1 57.4, 130.9 57.4 \" fill=\"#FF0000\"/>\n<path class=\"atom-10\" d=\"M 22.7 197.0 L 26.1 197.0 L 26.1 207.5 L 38.7 207.5 L 38.7 197.0 L 42.0 197.0 L 42.0 221.7 L 38.7 221.7 L 38.7 210.3 L 26.1 210.3 L 26.1 221.7 L 22.7 221.7 L 22.7 197.0 \" fill=\"#0000FF\"/>\n<path class=\"atom-10\" d=\"M 43.2 220.8 Q 43.8 219.3, 45.3 218.4 Q 46.7 217.6, 48.7 217.6 Q 51.1 217.6, 52.5 218.9 Q 53.9 220.2, 53.9 222.6 Q 53.9 225.0, 52.1 227.3 Q 50.3 229.5, 46.6 232.2 L 54.2 232.2 L 54.2 234.0 L 43.2 234.0 L 43.2 232.5 Q 46.2 230.3, 48.0 228.7 Q 49.8 227.1, 50.7 225.7 Q 51.6 224.2, 51.6 222.7 Q 51.6 221.1, 50.8 220.3 Q 50.0 219.4, 48.7 219.4 Q 47.4 219.4, 46.5 219.9 Q 45.6 220.5, 45.0 221.6 L 43.2 220.8 \" fill=\"#0000FF\"/>\n<path class=\"atom-10\" d=\"M 61.0 197.0 L 69.1 210.1 Q 69.9 211.4, 71.2 213.7 Q 72.4 216.0, 72.5 216.2 L 72.5 197.0 L 75.8 197.0 L 75.8 221.7 L 72.4 221.7 L 63.7 207.4 Q 62.7 205.7, 61.6 203.8 Q 60.6 201.9, 60.3 201.3 L 60.3 221.7 L 57.1 221.7 L 57.1 197.0 L 61.0 197.0 \" fill=\"#0000FF\"/>\n</svg></div>",
          "text/plain": "<IPython.core.display.HTML object>"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ]
      }
     },
     "ce4f0873f0854320a426fc62ca12b6e3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "VBoxModel",
      "state": {
       "_dom_classes": [
        "widget-interact"
       ],
       "children": [
        "IPY_MODEL_7f1195620f954dcd9e91cbe70801035b",
        "IPY_MODEL_b5504e3eeff3468387c876e2823834c9"
       ],
       "layout": "IPY_MODEL_81ed02b2b9d14752987f816c2b06b09c"
      }
     },
     "e5d917d6e4f24f72a014d6ff649ae3f7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}