ensemble-methods-notebooks-master/Ch5.5-case-study-document-retrieval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains code and comments from Section 5.4 of the book [Ensemble Methods for Machine Learning](https://www.manning.com/books/ensemble-methods-for-machine-learning). Please see the book for additional details on this topic. This notebook and code are released under the [MIT license](https://github.com/gkunapuli/ensemble-methods-notebooks/blob/master/LICENSE)._\n",
    "\n",
    "## 5.5 Case Study: Document Retrieval\n",
    "Document retrieval is the task of retrieving documents from a database to match a user’s query. For example, a paralegal at a law firm might need to search for information about previous cases from legal archives in order to establish precedent and research case law. Or perhaps a graduate student might need to search for articles from a journal’s database during the course of a literature survey of work in a specific area. \n",
    "\n",
    "In this case study, the problem is set up as a 3-class classification problem of identifying the relevance rank/class (least, moderately or highly relevant) given a query-document pair. We explore the performance of different LightGBM classifiers for this task. \n",
    "\n",
    "### 5.5.1\tThe LETOR Data Set\n",
    "The data set we will use for this case study is called the LEarning TO Rank (LETOR) ver. 4.0, which was itself created from a large corpus of webpages called GOV2. The [GOV2 data set](http://ir.dcs.gla.ac.uk/test_collections/access_to_data.html) is a collection of about 25 million webpages extracted from the .gov domain.\n",
    "\n",
    "The LETOR 4.0  data collection is derived from the GOV2 corpus and is made freely available by Microsoft Research. The collection contains several data sets, and we will use the data set that was originally developed for the Million Query track of the 2008 Text Retrieval Conference (TREC), specifically, [MQ2008.rar](https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/#!letor-4-0). \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_svmlight_file\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Load the data\n",
    "query_data_file = './data/ch05/MQ2008/Querylevelnorm.txt'\n",
    "X, y = load_svmlight_file(query_data_file)\n",
    "\n",
    "# Split into train and test sets\n",
    "Xtrn, Xtst, ytrn, ytst = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(12168, 46) (3043, 46)\n"
     ]
    }
   ],
   "source": [
    "print(Xtrn.shape, Xtst.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LightGBM has numerous parameters that we want to optimize over\n",
    "import numpy as np\n",
    "from scipy.stats import randint, uniform\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.metrics import accuracy_score\n",
    "import lightgbm as lgb\n",
    "import time\n",
    "\n",
    "fixed_params = {'early_stopping_rounds': 25, \n",
    "                'eval_metric' : 'multi_logloss', \n",
    "                'eval_set' : [(Xtst, ytst)],\n",
    "                'eval_names': ['test set'],\n",
    "                'verbose': 100}\n",
    "\n",
    "num_random_iters = 20\n",
    "num_cv_folds = 5\n",
    "\n",
    "cv_scores = {}\n",
    "tst_scores = {}\n",
    "run_time = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5.2\tDocument Retrieval with LightGBM\n",
    "We will learn four models using LightGBM. Each of these models represents a tradeoff between speed and accuracy:\n",
    "* random forest: our now familiar parallel homogeneous ensemble of randomized decision trees; this method will serve as a baseline approach;\n",
    "* gradient boosted decision trees (GBDT): this is the standard approach to gradient boosting and represents a balance between models with good generalization performance and training speed;\n",
    "* gradient boosting with gradient one-side sampling (GOSS): this variant of gradient boosting downsamples the training data and is ideally suited for large data sets; due to downsampling, it may lose out on generalization, but is typically very fast to train;\n",
    "* Dropout meets Multiple Additive Regression Trees (DART): this variant incorporates the notion of dropout from deep learning, where neural units are randomly and temporarily dropped during backpropagation iterations to mitigate overfitting. Similarly, DART randomly and temporarily drops base estimators from the overall ensemble during gradient fitting iterations to mitigate overfitting. DART is often the slowest of all the gradient boosting options available in LightGBM.\n",
    "\n",
    "\n",
    "---\n",
    "#### Learning with Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
      "Training until validation scores don't improve for 25 rounds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttest set's multi_logloss: 0.53411\n",
      "Early stopping, best iteration is:\n",
      "[155]\ttest set's multi_logloss: 0.531288\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.534096\n",
      "Early stopping, best iteration is:\n",
      "[115]\ttest set's multi_logloss: 0.533459\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.534166\n",
      "Early stopping, best iteration is:\n",
      "[90]\ttest set's multi_logloss: 0.533725\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532142\n",
      "Early stopping, best iteration is:\n",
      "[76]\ttest set's multi_logloss: 0.531735\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531562\n",
      "Early stopping, best iteration is:\n",
      "[105]\ttest set's multi_logloss: 0.531365\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.527426\n",
      "Early stopping, best iteration is:\n",
      "[110]\ttest set's multi_logloss: 0.527111\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.53171\n",
      "[200]\ttest set's multi_logloss: 0.529151\n",
      "Early stopping, best iteration is:\n",
      "[199]\ttest set's multi_logloss: 0.52915\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533273\n",
      "[200]\ttest set's multi_logloss: 0.529756\n",
      "Early stopping, best iteration is:\n",
      "[215]\ttest set's multi_logloss: 0.528866\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[74]\ttest set's multi_logloss: 0.532995\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.53092\n",
      "Early stopping, best iteration is:\n",
      "[96]\ttest set's multi_logloss: 0.530825\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.535904\n",
      "Early stopping, best iteration is:\n",
      "[161]\ttest set's multi_logloss: 0.531423\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.535433\n",
      "Early stopping, best iteration is:\n",
      "[75]\ttest set's multi_logloss: 0.533991\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[47]\ttest set's multi_logloss: 0.53083\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531193\n",
      "Early stopping, best iteration is:\n",
      "[105]\ttest set's multi_logloss: 0.531018\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532225\n",
      "Early stopping, best iteration is:\n",
      "[145]\ttest set's multi_logloss: 0.529348\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.528565\n",
      "Early stopping, best iteration is:\n",
      "[76]\ttest set's multi_logloss: 0.527621\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531447\n",
      "[200]\ttest set's multi_logloss: 0.529621\n",
      "Early stopping, best iteration is:\n",
      "[188]\ttest set's multi_logloss: 0.52902\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531595\n",
      "[200]\ttest set's multi_logloss: 0.528084\n",
      "Early stopping, best iteration is:\n",
      "[197]\ttest set's multi_logloss: 0.528066\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532885\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttest set's multi_logloss: 0.531759\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.528881\n",
      "[200]\ttest set's multi_logloss: 0.526222\n",
      "Early stopping, best iteration is:\n",
      "[189]\ttest set's multi_logloss: 0.52584\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.528257\n",
      "Early stopping, best iteration is:\n",
      "[144]\ttest set's multi_logloss: 0.526194\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.53137\n",
      "Early stopping, best iteration is:\n",
      "[136]\ttest set's multi_logloss: 0.530001\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533139\n",
      "Early stopping, best iteration is:\n",
      "[104]\ttest set's multi_logloss: 0.532622\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531382\n",
      "Early stopping, best iteration is:\n",
      "[138]\ttest set's multi_logloss: 0.528697\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532316\n",
      "Early stopping, best iteration is:\n",
      "[112]\ttest set's multi_logloss: 0.531402\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.538584\n",
      "Early stopping, best iteration is:\n",
      "[102]\ttest set's multi_logloss: 0.538524\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.542843\n",
      "Early stopping, best iteration is:\n",
      "[78]\ttest set's multi_logloss: 0.542518\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.539296\n",
      "Early stopping, best iteration is:\n",
      "[108]\ttest set's multi_logloss: 0.538478\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[60]\ttest set's multi_logloss: 0.537093\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.538113\n",
      "Early stopping, best iteration is:\n",
      "[96]\ttest set's multi_logloss: 0.537931\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.530568\n",
      "Early stopping, best iteration is:\n",
      "[138]\ttest set's multi_logloss: 0.526944\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531855\n",
      "Early stopping, best iteration is:\n",
      "[82]\ttest set's multi_logloss: 0.531266\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[58]\ttest set's multi_logloss: 0.533127\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.52943\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttest set's multi_logloss: 0.528535\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.52784\n",
      "Early stopping, best iteration is:\n",
      "[138]\ttest set's multi_logloss: 0.526722\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.529953\n",
      "Early stopping, best iteration is:\n",
      "[133]\ttest set's multi_logloss: 0.527209\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.530011\n",
      "Early stopping, best iteration is:\n",
      "[100]\ttest set's multi_logloss: 0.530011\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532917\n",
      "Early stopping, best iteration is:\n",
      "[96]\ttest set's multi_logloss: 0.532479\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[61]\ttest set's multi_logloss: 0.531655\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.529157\n",
      "Early stopping, best iteration is:\n",
      "[94]\ttest set's multi_logloss: 0.528716\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531233\n",
      "Early stopping, best iteration is:\n",
      "[173]\ttest set's multi_logloss: 0.528202\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[71]\ttest set's multi_logloss: 0.531164\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532746\n",
      "[200]\ttest set's multi_logloss: 0.528595\n",
      "Early stopping, best iteration is:\n",
      "[232]\ttest set's multi_logloss: 0.52809\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[74]\ttest set's multi_logloss: 0.530414\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.526664\n",
      "Early stopping, best iteration is:\n",
      "[136]\ttest set's multi_logloss: 0.525469\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.530412\n",
      "Early stopping, best iteration is:\n",
      "[144]\ttest set's multi_logloss: 0.52961\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533787\n",
      "Early stopping, best iteration is:\n",
      "[140]\ttest set's multi_logloss: 0.532312\n",
      "Training until validation scores don't improve for 25 rounds\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttest set's multi_logloss: 0.53429\n",
      "Early stopping, best iteration is:\n",
      "[104]\ttest set's multi_logloss: 0.533863\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533675\n",
      "Early stopping, best iteration is:\n",
      "[144]\ttest set's multi_logloss: 0.53189\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531601\n",
      "Early stopping, best iteration is:\n",
      "[120]\ttest set's multi_logloss: 0.531227\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.530301\n",
      "[200]\ttest set's multi_logloss: 0.525107\n",
      "Early stopping, best iteration is:\n",
      "[211]\ttest set's multi_logloss: 0.524051\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[68]\ttest set's multi_logloss: 0.53282\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.527964\n",
      "[200]\ttest set's multi_logloss: 0.525553\n",
      "Early stopping, best iteration is:\n",
      "[175]\ttest set's multi_logloss: 0.525157\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.530345\n",
      "Early stopping, best iteration is:\n",
      "[105]\ttest set's multi_logloss: 0.529459\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.524573\n",
      "[200]\ttest set's multi_logloss: 0.523335\n",
      "Early stopping, best iteration is:\n",
      "[231]\ttest set's multi_logloss: 0.522963\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[38]\ttest set's multi_logloss: 0.535008\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533908\n",
      "Early stopping, best iteration is:\n",
      "[149]\ttest set's multi_logloss: 0.532572\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.538086\n",
      "Early stopping, best iteration is:\n",
      "[104]\ttest set's multi_logloss: 0.537813\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.535341\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttest set's multi_logloss: 0.535037\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531834\n",
      "Early stopping, best iteration is:\n",
      "[83]\ttest set's multi_logloss: 0.531733\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.537583\n",
      "Early stopping, best iteration is:\n",
      "[88]\ttest set's multi_logloss: 0.537437\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.540081\n",
      "Early stopping, best iteration is:\n",
      "[160]\ttest set's multi_logloss: 0.538745\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.538658\n",
      "[200]\ttest set's multi_logloss: 0.537117\n",
      "Early stopping, best iteration is:\n",
      "[232]\ttest set's multi_logloss: 0.536755\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[71]\ttest set's multi_logloss: 0.537173\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.537428\n",
      "Early stopping, best iteration is:\n",
      "[96]\ttest set's multi_logloss: 0.537369\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.538033\n",
      "Early stopping, best iteration is:\n",
      "[161]\ttest set's multi_logloss: 0.536583\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.539544\n",
      "Early stopping, best iteration is:\n",
      "[140]\ttest set's multi_logloss: 0.538646\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.536513\n",
      "Early stopping, best iteration is:\n",
      "[112]\ttest set's multi_logloss: 0.535791\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.536448\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttest set's multi_logloss: 0.536154\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533293\n",
      "Early stopping, best iteration is:\n",
      "[90]\ttest set's multi_logloss: 0.533136\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531343\n",
      "Early stopping, best iteration is:\n",
      "[135]\ttest set's multi_logloss: 0.529061\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533736\n",
      "[200]\ttest set's multi_logloss: 0.531111\n",
      "Early stopping, best iteration is:\n",
      "[191]\ttest set's multi_logloss: 0.53084\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[68]\ttest set's multi_logloss: 0.533099\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.528351\n",
      "Early stopping, best iteration is:\n",
      "[160]\ttest set's multi_logloss: 0.52653\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.529258\n",
      "[200]\ttest set's multi_logloss: 0.527535\n",
      "Early stopping, best iteration is:\n",
      "[225]\ttest set's multi_logloss: 0.527534\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.530367\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttest set's multi_logloss: 0.529721\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.529643\n",
      "Early stopping, best iteration is:\n",
      "[101]\ttest set's multi_logloss: 0.529639\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.531115\n",
      "Early stopping, best iteration is:\n",
      "[96]\ttest set's multi_logloss: 0.53072\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[61]\ttest set's multi_logloss: 0.531514\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.528834\n",
      "Early stopping, best iteration is:\n",
      "[95]\ttest set's multi_logloss: 0.528507\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[48]\ttest set's multi_logloss: 0.538801\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[54]\ttest set's multi_logloss: 0.536344\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[54]\ttest set's multi_logloss: 0.53903\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.535127\n",
      "Early stopping, best iteration is:\n",
      "[84]\ttest set's multi_logloss: 0.534913\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.533736\n",
      "Early stopping, best iteration is:\n",
      "[91]\ttest set's multi_logloss: 0.533551\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[22]\ttest set's multi_logloss: 0.542062\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[45]\ttest set's multi_logloss: 0.540346\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.539714\n",
      "Early stopping, best iteration is:\n",
      "[135]\ttest set's multi_logloss: 0.538855\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[39]\ttest set's multi_logloss: 0.538599\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[50]\ttest set's multi_logloss: 0.537983\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.539334\n",
      "Early stopping, best iteration is:\n",
      "[136]\ttest set's multi_logloss: 0.536933\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.538048\n",
      "Early stopping, best iteration is:\n",
      "[96]\ttest set's multi_logloss: 0.53801\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[72]\ttest set's multi_logloss: 0.540373\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.536739\n",
      "Early stopping, best iteration is:\n",
      "[80]\ttest set's multi_logloss: 0.536072\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.534146\n",
      "Early stopping, best iteration is:\n",
      "[79]\ttest set's multi_logloss: 0.533605\n",
      "Training until validation scores don't improve for 25 rounds\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttest set's multi_logloss: 0.537375\n",
      "Early stopping, best iteration is:\n",
      "[135]\ttest set's multi_logloss: 0.535379\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.536482\n",
      "Early stopping, best iteration is:\n",
      "[160]\ttest set's multi_logloss: 0.535669\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.537186\n",
      "Early stopping, best iteration is:\n",
      "[80]\ttest set's multi_logloss: 0.536902\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.534946\n",
      "Early stopping, best iteration is:\n",
      "[78]\ttest set's multi_logloss: 0.534467\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.532706\n",
      "Early stopping, best iteration is:\n",
      "[105]\ttest set's multi_logloss: 0.532474\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:  2.0min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.536153\n",
      "[200]\ttest set's multi_logloss: 0.536062\n",
      "Early stopping, best iteration is:\n",
      "[176]\ttest set's multi_logloss: 0.53523\n"
     ]
    }
   ],
   "source": [
    "rf_params = {'bagging_fraction': [0.4, 0.5, 0.6, 0.7, 0.8],\n",
    "             'bagging_freq': [5, 6, 7, 8],\n",
    "             'num_leaves': randint(5, 50)}\n",
    "\n",
    "start = time.time()\n",
    "ens = lgb.LGBMClassifier(boosting='rf', n_estimators=1000, max_depth=-1, metric='multi_logloss', random_state=42)\n",
    "cv = RandomizedSearchCV(estimator=ens, \n",
    "                        param_distributions=rf_params, \n",
    "                        n_iter=num_random_iters, \n",
    "                        cv=num_cv_folds, \n",
    "                        refit=True,\n",
    "                        random_state=42, verbose=True)\n",
    "\n",
    "cv.fit(Xtrn, ytrn, **fixed_params)\n",
    "run_time['rf'] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RANDOM FORESTS: Best score: 0.8051450187962624, best params: {'bagging_fraction': 0.6, 'bagging_freq': 8, 'num_leaves': 7} \n"
     ]
    }
   ],
   "source": [
    "print('RANDOM FORESTS: Best score: {0}, best params: {1} '.format(cv.best_score_, cv.best_params_))\n",
    "cv_scores['rf'] = cv.best_score_\n",
    "\n",
    "ypred = cv.best_estimator_.predict(Xtst)\n",
    "tst_scores['rf'] = accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### GBDT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.692218\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.66966\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.667761\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.68124\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.688833\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[47]\ttest set's multi_logloss: 0.512479\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[31]\ttest set's multi_logloss: 0.511625\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[29]\ttest set's multi_logloss: 0.508891\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[19]\ttest set's multi_logloss: 0.505264\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[28]\ttest set's multi_logloss: 0.508864\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.04422\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.14058\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.11439\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.13322\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.07973\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.499998\n",
      "Early stopping, best iteration is:\n",
      "[128]\ttest set's multi_logloss: 0.499198\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.500354\n",
      "Early stopping, best iteration is:\n",
      "[82]\ttest set's multi_logloss: 0.500052\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[71]\ttest set's multi_logloss: 0.501745\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.497604\n",
      "Early stopping, best iteration is:\n",
      "[77]\ttest set's multi_logloss: 0.497395\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[58]\ttest set's multi_logloss: 0.501381\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.96236\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.91542\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.76615\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.90103\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.81103\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[23]\ttest set's multi_logloss: 0.502565\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[22]\ttest set's multi_logloss: 0.509302\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[17]\ttest set's multi_logloss: 0.510826\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[37]\ttest set's multi_logloss: 0.507698\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[21]\ttest set's multi_logloss: 0.506613\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[19]\ttest set's multi_logloss: 0.524033\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[30]\ttest set's multi_logloss: 0.518884\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[22]\ttest set's multi_logloss: 0.520025\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[9]\ttest set's multi_logloss: 0.523773\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[6]\ttest set's multi_logloss: 0.528524\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.506203\n",
      "Early stopping, best iteration is:\n",
      "[85]\ttest set's multi_logloss: 0.505461\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.507876\n",
      "Early stopping, best iteration is:\n",
      "[158]\ttest set's multi_logloss: 0.506356\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.505724\n",
      "Early stopping, best iteration is:\n",
      "[126]\ttest set's multi_logloss: 0.504546\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.502399\n",
      "[200]\ttest set's multi_logloss: 0.498113\n",
      "Early stopping, best iteration is:\n",
      "[209]\ttest set's multi_logloss: 0.497829\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.503509\n",
      "Early stopping, best iteration is:\n",
      "[172]\ttest set's multi_logloss: 0.500436\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[58]\ttest set's multi_logloss: 0.505335\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[67]\ttest set's multi_logloss: 0.502418\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[56]\ttest set's multi_logloss: 0.503191\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[44]\ttest set's multi_logloss: 0.500396\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[58]\ttest set's multi_logloss: 0.500521\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.03883\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.02155\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.07789\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.04064\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.02792\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.50811\n",
      "Early stopping, best iteration is:\n",
      "[130]\ttest set's multi_logloss: 0.506845\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.507999\n",
      "Early stopping, best iteration is:\n",
      "[109]\ttest set's multi_logloss: 0.507713\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.50616\n",
      "Early stopping, best iteration is:\n",
      "[135]\ttest set's multi_logloss: 0.503764\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.505058\n",
      "Early stopping, best iteration is:\n",
      "[142]\ttest set's multi_logloss: 0.503689\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.508784\n",
      "Early stopping, best iteration is:\n",
      "[77]\ttest set's multi_logloss: 0.508226\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.31309\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.68451\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.62631\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.45729\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttest set's multi_logloss: 4.52148\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 3.98366\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.21882\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.42541\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.33247\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.32844\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 3.96925\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.03533\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.23798\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.13925\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 3.89316\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 3.84331\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.51997\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.33351\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.41572\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.40656\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[60]\ttest set's multi_logloss: 0.506367\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[39]\ttest set's multi_logloss: 0.507409\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[51]\ttest set's multi_logloss: 0.50337\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[44]\ttest set's multi_logloss: 0.502047\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[34]\ttest set's multi_logloss: 0.503645\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.784127\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.842515\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.804595\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.785896\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.801803\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.514355\n",
      "Early stopping, best iteration is:\n",
      "[80]\ttest set's multi_logloss: 0.511852\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[20]\ttest set's multi_logloss: 0.511687\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[57]\ttest set's multi_logloss: 0.508662\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[46]\ttest set's multi_logloss: 0.504017\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[72]\ttest set's multi_logloss: 0.507013\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[2]\ttest set's multi_logloss: 0.607542\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.612549\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.622699\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[10]\ttest set's multi_logloss: 0.563903\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.59339\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.507942\n",
      "Early stopping, best iteration is:\n",
      "[139]\ttest set's multi_logloss: 0.505194\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[58]\ttest set's multi_logloss: 0.510915\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[61]\ttest set's multi_logloss: 0.510221\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.502457\n",
      "[200]\ttest set's multi_logloss: 0.499769\n",
      "Early stopping, best iteration is:\n",
      "[191]\ttest set's multi_logloss: 0.499192\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.505284\n",
      "Early stopping, best iteration is:\n",
      "[144]\ttest set's multi_logloss: 0.504398\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:   22.3s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[74]\ttest set's multi_logloss: 0.493537\n"
     ]
    }
   ],
   "source": [
    "gbdt_params = {'num_leaves': randint(5, 50), \n",
    "               'learning_rate': [0.25, 0.5, 1, 2, 4, 8, 16],\n",
    "               'min_child_samples': randint(100, 500), \n",
    "               'min_child_weight': [1e-2, 1e-1, 1, 1e1, 1e2],\n",
    "               'subsample': uniform(loc=0.2, scale=0.8), \n",
    "               'colsample_bytree': uniform(loc=0.4, scale=0.6),\n",
    "               'reg_alpha': [0, 1e-1, 1, 10, 100],\n",
    "               'reg_lambda': [0, 1e-1, 1, 10, 100]}\n",
    "\n",
    "start = time.time()\n",
    "ens = lgb.LGBMClassifier(boosting='gbdt', n_estimators=1000, max_depth=-1, metric='multi_logloss', random_state=42)\n",
    "cv = RandomizedSearchCV(estimator=ens, \n",
    "                        param_distributions=gbdt_params, \n",
    "                        n_iter=num_random_iters, \n",
    "                        cv=num_cv_folds, \n",
    "                        refit=True,\n",
    "                        random_state=42, verbose=True)\n",
    "\n",
    "cv.fit(Xtrn, ytrn, **fixed_params)\n",
    "run_time['gbdt'] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GBDT: Best score: 0.8122129268166652, best params: {'colsample_bytree': 0.40423978313183045, 'learning_rate': 0.25, 'min_child_samples': 148, 'min_child_weight': 1, 'num_leaves': 46, 'reg_alpha': 10, 'reg_lambda': 10, 'subsample': 0.4930894746349534} \n"
     ]
    }
   ],
   "source": [
    "print('GBDT: Best score: {0}, best params: {1} '.format(cv.best_score_, cv.best_params_))\n",
    "cv_scores['gbdt'] = cv.best_score_\n",
    "\n",
    "ypred = cv.best_estimator_.predict(Xtst)\n",
    "tst_scores['gbdt'] = accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### GOSS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.2144\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.82534\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.37402\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.40047\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.56711\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[37]\ttest set's multi_logloss: 0.515495\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[19]\ttest set's multi_logloss: 0.514802\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[23]\ttest set's multi_logloss: 0.506899\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[22]\ttest set's multi_logloss: 0.506286\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[21]\ttest set's multi_logloss: 0.508444\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[3]\ttest set's multi_logloss: 0.533471\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[5]\ttest set's multi_logloss: 0.533182\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[3]\ttest set's multi_logloss: 0.527426\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[4]\ttest set's multi_logloss: 0.53085\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[3]\ttest set's multi_logloss: 0.530055\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[39]\ttest set's multi_logloss: 0.49369\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[50]\ttest set's multi_logloss: 0.499662\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[34]\ttest set's multi_logloss: 0.497904\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[40]\ttest set's multi_logloss: 0.497175\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[40]\ttest set's multi_logloss: 0.500159\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[52]\ttest set's multi_logloss: 0.50417\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.505753\n",
      "Early stopping, best iteration is:\n",
      "[76]\ttest set's multi_logloss: 0.503628\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.503428\n",
      "Early stopping, best iteration is:\n",
      "[89]\ttest set's multi_logloss: 0.502645\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.501227\n",
      "Early stopping, best iteration is:\n",
      "[93]\ttest set's multi_logloss: 0.50087\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.501215\n",
      "Early stopping, best iteration is:\n",
      "[82]\ttest set's multi_logloss: 0.50072\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.00499\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.10022\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.43147\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.38134\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.56408\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.620585\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.631345\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.619392\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[2]\ttest set's multi_logloss: 0.60637\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.606254\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.07045\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.25763\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.20163\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.22161\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.23747\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[15]\ttest set's multi_logloss: 0.533841\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[16]\ttest set's multi_logloss: 0.531807\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[33]\ttest set's multi_logloss: 0.530338\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[19]\ttest set's multi_logloss: 0.528941\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[20]\ttest set's multi_logloss: 0.530262\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.522765\n",
      "Early stopping, best iteration is:\n",
      "[170]\ttest set's multi_logloss: 0.51966\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.520943\n",
      "Early stopping, best iteration is:\n",
      "[103]\ttest set's multi_logloss: 0.520767\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[70]\ttest set's multi_logloss: 0.521024\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[69]\ttest set's multi_logloss: 0.521555\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[67]\ttest set's multi_logloss: 0.519527\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.508548\n",
      "Early stopping, best iteration is:\n",
      "[114]\ttest set's multi_logloss: 0.507087\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.505949\n",
      "Early stopping, best iteration is:\n",
      "[83]\ttest set's multi_logloss: 0.505931\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.503177\n",
      "Early stopping, best iteration is:\n",
      "[115]\ttest set's multi_logloss: 0.502999\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.502156\n",
      "Early stopping, best iteration is:\n",
      "[142]\ttest set's multi_logloss: 0.499018\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "[100]\ttest set's multi_logloss: 0.506359\n",
      "Early stopping, best iteration is:\n",
      "[91]\ttest set's multi_logloss: 0.504812\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[23]\ttest set's multi_logloss: 0.520573\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[14]\ttest set's multi_logloss: 0.515926\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[4]\ttest set's multi_logloss: 0.51874\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[9]\ttest set's multi_logloss: 0.520267\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[7]\ttest set's multi_logloss: 0.519068\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[51]\ttest set's multi_logloss: 0.531401\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[45]\ttest set's multi_logloss: 0.53057\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[47]\ttest set's multi_logloss: 0.528413\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[51]\ttest set's multi_logloss: 0.527034\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[64]\ttest set's multi_logloss: 0.528625\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.42024\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.60167\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.44576\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.57699\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.54936\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.51892\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.75525\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.84855\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.79356\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 4.86997\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[12]\ttest set's multi_logloss: 0.531937\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[15]\ttest set's multi_logloss: 0.531709\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[12]\ttest set's multi_logloss: 0.530976\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[12]\ttest set's multi_logloss: 0.529537\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[9]\ttest set's multi_logloss: 0.534345\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.974054\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.0382\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.00218\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.02773\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 1.00197\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.943566\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.999373\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.962311\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.962811\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 0.951981\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.04849\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.18528\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.07985\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.16885\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.14947\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.22012\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.35554\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.3623\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.37183\n",
      "Training until validation scores don't improve for 25 rounds\n",
      "Early stopping, best iteration is:\n",
      "[1]\ttest set's multi_logloss: 2.40285\n",
      "Training until validation scores don't improve for 25 rounds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:   18.6s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Early stopping, best iteration is:\n",
      "[47]\ttest set's multi_logloss: 0.489955\n"
     ]
    }
   ],
   "source": [
    "goss_params = {'num_leaves': randint(5, 50), \n",
    "               'learning_rate': [0.25, 0.5, 1, 2, 4, 8, 16],\n",
    "               'min_child_samples': randint(100, 500), \n",
    "               'min_child_weight': [1e-2, 1e-1, 1, 1e1, 1e2],\n",
    "               'top_rate': uniform(loc=0.05, scale=0.45), \n",
    "               'other_rate': uniform(loc=0.05, scale=0.45),\n",
    "               'reg_alpha': [0, 1e-1, 1, 10, 100],\n",
    "               'reg_lambda': [0, 1e-1, 1, 10, 100]}\n",
    "\n",
    "start = time.time()\n",
    "ens = lgb.LGBMClassifier(boosting='goss', n_estimators=1000, max_depth=-1, metric='multi_logloss', random_state=42)\n",
    "cv = RandomizedSearchCV(estimator=ens, \n",
    "                        param_distributions=goss_params, \n",
    "                        n_iter=num_random_iters, \n",
    "                        cv=num_cv_folds, \n",
    "                        refit=True,\n",
    "                        random_state=42, verbose=True)\n",
    "\n",
    "cv.fit(Xtrn, ytrn, **fixed_params)\n",
    "run_time['goss'] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GOSS: Best score: 0.8131167212266558, best params: {'learning_rate': 0.25, 'min_child_samples': 158, 'min_child_weight': 0.1, 'num_leaves': 32, 'other_rate': 0.4881899834786566, 'reg_alpha': 1, 'reg_lambda': 10, 'top_rate': 0.28140549728612524} \n"
     ]
    }
   ],
   "source": [
    "print('GOSS: Best score: {0}, best params: {1} '.format(cv.best_score_, cv.best_params_))\n",
    "cv_scores['goss'] = cv.best_score_\n",
    "\n",
    "ypred = cv.best_estimator_.predict(Xtst)\n",
    "tst_scores['goss'] = accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### DART"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\callback.py:188: UserWarning: Early stopping is not available in dart mode\n",
      "  warnings.warn('Early stopping is not available in dart mode')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttest set's multi_logloss: 0.689428\n",
      "[200]\ttest set's multi_logloss: 0.68362\n",
      "[300]\ttest set's multi_logloss: 0.681356\n",
      "[400]\ttest set's multi_logloss: 0.681477\n",
      "[500]\ttest set's multi_logloss: 0.678007\n",
      "[600]\ttest set's multi_logloss: 0.679833\n",
      "[700]\ttest set's multi_logloss: 0.677925\n",
      "[800]\ttest set's multi_logloss: 0.676639\n",
      "[900]\ttest set's multi_logloss: 0.674501\n",
      "[1000]\ttest set's multi_logloss: 0.673694\n",
      "[100]\ttest set's multi_logloss: 0.55347\n",
      "[200]\ttest set's multi_logloss: 0.579132\n",
      "[300]\ttest set's multi_logloss: 0.616199\n",
      "[400]\ttest set's multi_logloss: 0.610611\n",
      "[500]\ttest set's multi_logloss: 0.610739\n",
      "[600]\ttest set's multi_logloss: 0.610023\n",
      "[700]\ttest set's multi_logloss: 0.609824\n",
      "[800]\ttest set's multi_logloss: 0.610063\n",
      "[900]\ttest set's multi_logloss: 0.609894\n",
      "[1000]\ttest set's multi_logloss: 0.609327\n",
      "[100]\ttest set's multi_logloss: 0.593722\n",
      "[200]\ttest set's multi_logloss: 0.602097\n",
      "[300]\ttest set's multi_logloss: 0.683435\n",
      "[400]\ttest set's multi_logloss: 0.679543\n",
      "[500]\ttest set's multi_logloss: 0.680587\n",
      "[600]\ttest set's multi_logloss: 0.677344\n",
      "[700]\ttest set's multi_logloss: 0.683037\n",
      "[800]\ttest set's multi_logloss: 0.683465\n",
      "[900]\ttest set's multi_logloss: 0.681398\n",
      "[1000]\ttest set's multi_logloss: 0.681224\n",
      "[100]\ttest set's multi_logloss: 0.554503\n",
      "[200]\ttest set's multi_logloss: 0.653364\n",
      "[300]\ttest set's multi_logloss: 0.650944\n",
      "[400]\ttest set's multi_logloss: 0.64736\n",
      "[500]\ttest set's multi_logloss: 0.646845\n",
      "[600]\ttest set's multi_logloss: 0.645328\n",
      "[700]\ttest set's multi_logloss: 0.644986\n",
      "[800]\ttest set's multi_logloss: 0.645092\n",
      "[900]\ttest set's multi_logloss: 0.644727\n",
      "[1000]\ttest set's multi_logloss: 0.644751\n",
      "[100]\ttest set's multi_logloss: 0.596853\n",
      "[200]\ttest set's multi_logloss: 0.73245\n",
      "[300]\ttest set's multi_logloss: 0.732798\n",
      "[400]\ttest set's multi_logloss: 0.736107\n",
      "[500]\ttest set's multi_logloss: 0.740028\n",
      "[600]\ttest set's multi_logloss: 0.74119\n",
      "[700]\ttest set's multi_logloss: 0.742074\n",
      "[800]\ttest set's multi_logloss: 0.742483\n",
      "[900]\ttest set's multi_logloss: 0.743244\n",
      "[1000]\ttest set's multi_logloss: 0.743494\n",
      "[100]\ttest set's multi_logloss: 0.531531\n",
      "[200]\ttest set's multi_logloss: 0.543477\n",
      "[300]\ttest set's multi_logloss: 0.567759\n",
      "[400]\ttest set's multi_logloss: 0.564\n",
      "[500]\ttest set's multi_logloss: 0.561275\n",
      "[600]\ttest set's multi_logloss: 0.559752\n",
      "[700]\ttest set's multi_logloss: 0.561161\n",
      "[800]\ttest set's multi_logloss: 0.58117\n",
      "[900]\ttest set's multi_logloss: 0.579529\n",
      "[1000]\ttest set's multi_logloss: 0.57892\n",
      "[100]\ttest set's multi_logloss: 0.651576\n",
      "[200]\ttest set's multi_logloss: 0.64723\n",
      "[300]\ttest set's multi_logloss: 0.645803\n",
      "[400]\ttest set's multi_logloss: 0.64669\n",
      "[500]\ttest set's multi_logloss: 0.646074\n",
      "[600]\ttest set's multi_logloss: 0.645874\n",
      "[700]\ttest set's multi_logloss: 0.646917\n",
      "[800]\ttest set's multi_logloss: 0.64715\n",
      "[900]\ttest set's multi_logloss: 0.64734\n",
      "[1000]\ttest set's multi_logloss: 0.646832\n",
      "[100]\ttest set's multi_logloss: 0.60144\n",
      "[200]\ttest set's multi_logloss: 0.603494\n",
      "[300]\ttest set's multi_logloss: 0.604105\n",
      "[400]\ttest set's multi_logloss: 0.603544\n",
      "[500]\ttest set's multi_logloss: 0.603001\n",
      "[600]\ttest set's multi_logloss: 0.603228\n",
      "[700]\ttest set's multi_logloss: 0.603096\n",
      "[800]\ttest set's multi_logloss: 0.60289\n",
      "[900]\ttest set's multi_logloss: 0.602567\n",
      "[1000]\ttest set's multi_logloss: 0.602325\n",
      "[100]\ttest set's multi_logloss: 0.553987\n",
      "[200]\ttest set's multi_logloss: 0.604569\n",
      "[300]\ttest set's multi_logloss: 0.600926\n",
      "[400]\ttest set's multi_logloss: 0.600385\n",
      "[500]\ttest set's multi_logloss: 0.59948\n",
      "[600]\ttest set's multi_logloss: 0.599432\n",
      "[700]\ttest set's multi_logloss: 0.599011\n",
      "[800]\ttest set's multi_logloss: 0.598078\n",
      "[900]\ttest set's multi_logloss: 0.597899\n",
      "[1000]\ttest set's multi_logloss: 0.597961\n",
      "[100]\ttest set's multi_logloss: 0.658866\n",
      "[200]\ttest set's multi_logloss: 0.655974\n",
      "[300]\ttest set's multi_logloss: 0.655967\n",
      "[400]\ttest set's multi_logloss: 0.655628\n",
      "[500]\ttest set's multi_logloss: 0.654053\n",
      "[600]\ttest set's multi_logloss: 0.653866\n",
      "[700]\ttest set's multi_logloss: 0.654511\n",
      "[800]\ttest set's multi_logloss: 0.654267\n",
      "[900]\ttest set's multi_logloss: 0.655193\n",
      "[1000]\ttest set's multi_logloss: 0.655693\n",
      "[100]\ttest set's multi_logloss: 0.500598\n",
      "[200]\ttest set's multi_logloss: 0.499965\n",
      "[300]\ttest set's multi_logloss: 0.501425\n",
      "[400]\ttest set's multi_logloss: 0.501904\n",
      "[500]\ttest set's multi_logloss: 0.503089\n",
      "[600]\ttest set's multi_logloss: 0.503399\n",
      "[700]\ttest set's multi_logloss: 0.503237\n",
      "[800]\ttest set's multi_logloss: 0.503624\n",
      "[900]\ttest set's multi_logloss: 0.50329\n",
      "[1000]\ttest set's multi_logloss: 0.504218\n",
      "[100]\ttest set's multi_logloss: 0.500312\n",
      "[200]\ttest set's multi_logloss: 0.500508\n",
      "[300]\ttest set's multi_logloss: 0.501726\n",
      "[400]\ttest set's multi_logloss: 0.504629\n",
      "[500]\ttest set's multi_logloss: 0.503606\n",
      "[600]\ttest set's multi_logloss: 0.503133\n",
      "[700]\ttest set's multi_logloss: 0.503558\n",
      "[800]\ttest set's multi_logloss: 0.502966\n",
      "[900]\ttest set's multi_logloss: 0.503527\n",
      "[1000]\ttest set's multi_logloss: 0.503052\n",
      "[100]\ttest set's multi_logloss: 0.498884\n",
      "[200]\ttest set's multi_logloss: 0.497622\n",
      "[300]\ttest set's multi_logloss: 0.496868\n",
      "[400]\ttest set's multi_logloss: 0.497842\n",
      "[500]\ttest set's multi_logloss: 0.496628\n",
      "[600]\ttest set's multi_logloss: 0.499968\n",
      "[700]\ttest set's multi_logloss: 0.500863\n",
      "[800]\ttest set's multi_logloss: 0.501709\n",
      "[900]\ttest set's multi_logloss: 0.500735\n",
      "[1000]\ttest set's multi_logloss: 0.500313\n",
      "[100]\ttest set's multi_logloss: 0.495196\n",
      "[200]\ttest set's multi_logloss: 0.494608\n",
      "[300]\ttest set's multi_logloss: 0.499323\n",
      "[400]\ttest set's multi_logloss: 0.500589\n",
      "[500]\ttest set's multi_logloss: 0.500651\n",
      "[600]\ttest set's multi_logloss: 0.501664\n",
      "[700]\ttest set's multi_logloss: 0.50174\n",
      "[800]\ttest set's multi_logloss: 0.502893\n",
      "[900]\ttest set's multi_logloss: 0.503191\n",
      "[1000]\ttest set's multi_logloss: 0.503791\n",
      "[100]\ttest set's multi_logloss: 0.49694\n",
      "[200]\ttest set's multi_logloss: 0.495674\n",
      "[300]\ttest set's multi_logloss: 0.496119\n",
      "[400]\ttest set's multi_logloss: 0.496234\n",
      "[500]\ttest set's multi_logloss: 0.496836\n",
      "[600]\ttest set's multi_logloss: 0.498461\n",
      "[700]\ttest set's multi_logloss: 0.499385\n",
      "[800]\ttest set's multi_logloss: 0.499358\n",
      "[900]\ttest set's multi_logloss: 0.498917\n",
      "[1000]\ttest set's multi_logloss: 0.498293\n",
      "[100]\ttest set's multi_logloss: 0.672735\n",
      "[200]\ttest set's multi_logloss: 0.672662\n",
      "[300]\ttest set's multi_logloss: 0.6744\n",
      "[400]\ttest set's multi_logloss: 0.675056\n",
      "[500]\ttest set's multi_logloss: 0.677732\n",
      "[600]\ttest set's multi_logloss: 0.67687\n",
      "[700]\ttest set's multi_logloss: 0.676316\n",
      "[800]\ttest set's multi_logloss: 0.677425\n",
      "[900]\ttest set's multi_logloss: 0.676229\n",
      "[1000]\ttest set's multi_logloss: 0.676571\n",
      "[100]\ttest set's multi_logloss: 0.653227\n",
      "[200]\ttest set's multi_logloss: 0.658503\n",
      "[300]\ttest set's multi_logloss: 0.659185\n",
      "[400]\ttest set's multi_logloss: 0.661498\n",
      "[500]\ttest set's multi_logloss: 0.661216\n",
      "[600]\ttest set's multi_logloss: 0.661363\n",
      "[700]\ttest set's multi_logloss: 0.664006\n",
      "[800]\ttest set's multi_logloss: 0.662076\n",
      "[900]\ttest set's multi_logloss: 0.661984\n",
      "[1000]\ttest set's multi_logloss: 0.661521\n",
      "[100]\ttest set's multi_logloss: 0.657435\n",
      "[200]\ttest set's multi_logloss: 0.660343\n",
      "[300]\ttest set's multi_logloss: 0.662094\n",
      "[400]\ttest set's multi_logloss: 0.661624\n",
      "[500]\ttest set's multi_logloss: 0.661394\n",
      "[600]\ttest set's multi_logloss: 0.661717\n",
      "[700]\ttest set's multi_logloss: 0.661829\n",
      "[800]\ttest set's multi_logloss: 0.661484\n",
      "[900]\ttest set's multi_logloss: 0.661813\n",
      "[1000]\ttest set's multi_logloss: 0.662167\n",
      "[100]\ttest set's multi_logloss: 0.657414\n",
      "[200]\ttest set's multi_logloss: 0.658942\n",
      "[300]\ttest set's multi_logloss: 0.659634\n",
      "[400]\ttest set's multi_logloss: 0.659467\n",
      "[500]\ttest set's multi_logloss: 0.659309\n",
      "[600]\ttest set's multi_logloss: 0.660264\n",
      "[700]\ttest set's multi_logloss: 0.661081\n",
      "[800]\ttest set's multi_logloss: 0.66134\n",
      "[900]\ttest set's multi_logloss: 0.660127\n",
      "[1000]\ttest set's multi_logloss: 0.660167\n",
      "[100]\ttest set's multi_logloss: 0.662115\n",
      "[200]\ttest set's multi_logloss: 0.66496\n",
      "[300]\ttest set's multi_logloss: 0.666114\n",
      "[400]\ttest set's multi_logloss: 0.666412\n",
      "[500]\ttest set's multi_logloss: 0.666752\n",
      "[600]\ttest set's multi_logloss: 0.66615\n",
      "[700]\ttest set's multi_logloss: 0.666323\n",
      "[800]\ttest set's multi_logloss: 0.666122\n",
      "[900]\ttest set's multi_logloss: 0.666373\n",
      "[1000]\ttest set's multi_logloss: 0.66684\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttest set's multi_logloss: 0.49978\n",
      "[200]\ttest set's multi_logloss: 0.49841\n",
      "[300]\ttest set's multi_logloss: 0.499873\n",
      "[400]\ttest set's multi_logloss: 0.499029\n",
      "[500]\ttest set's multi_logloss: 0.500585\n",
      "[600]\ttest set's multi_logloss: 0.502022\n",
      "[700]\ttest set's multi_logloss: 0.5028\n",
      "[800]\ttest set's multi_logloss: 0.502663\n",
      "[900]\ttest set's multi_logloss: 0.501919\n",
      "[1000]\ttest set's multi_logloss: 0.501623\n",
      "[100]\ttest set's multi_logloss: 0.50048\n",
      "[200]\ttest set's multi_logloss: 0.501181\n",
      "[300]\ttest set's multi_logloss: 0.501026\n",
      "[400]\ttest set's multi_logloss: 0.499301\n",
      "[500]\ttest set's multi_logloss: 0.502053\n",
      "[600]\ttest set's multi_logloss: 0.502425\n",
      "[700]\ttest set's multi_logloss: 0.501855\n",
      "[800]\ttest set's multi_logloss: 0.502062\n",
      "[900]\ttest set's multi_logloss: 0.503149\n",
      "[1000]\ttest set's multi_logloss: 0.503056\n",
      "[100]\ttest set's multi_logloss: 0.497446\n",
      "[200]\ttest set's multi_logloss: 0.499998\n",
      "[300]\ttest set's multi_logloss: 0.501656\n",
      "[400]\ttest set's multi_logloss: 0.501157\n",
      "[500]\ttest set's multi_logloss: 0.501184\n",
      "[600]\ttest set's multi_logloss: 0.50232\n",
      "[700]\ttest set's multi_logloss: 0.503769\n",
      "[800]\ttest set's multi_logloss: 0.505828\n",
      "[900]\ttest set's multi_logloss: 0.505979\n",
      "[1000]\ttest set's multi_logloss: 0.504836\n",
      "[100]\ttest set's multi_logloss: 0.499991\n",
      "[200]\ttest set's multi_logloss: 0.497469\n",
      "[300]\ttest set's multi_logloss: 0.50027\n",
      "[400]\ttest set's multi_logloss: 0.499004\n",
      "[500]\ttest set's multi_logloss: 0.498181\n",
      "[600]\ttest set's multi_logloss: 0.499878\n",
      "[700]\ttest set's multi_logloss: 0.501443\n",
      "[800]\ttest set's multi_logloss: 0.501121\n",
      "[900]\ttest set's multi_logloss: 0.502635\n",
      "[1000]\ttest set's multi_logloss: 0.504424\n",
      "[100]\ttest set's multi_logloss: 0.501019\n",
      "[200]\ttest set's multi_logloss: 0.498944\n",
      "[300]\ttest set's multi_logloss: 0.497775\n",
      "[400]\ttest set's multi_logloss: 0.499655\n",
      "[500]\ttest set's multi_logloss: 0.500507\n",
      "[600]\ttest set's multi_logloss: 0.501098\n",
      "[700]\ttest set's multi_logloss: 0.498924\n",
      "[800]\ttest set's multi_logloss: 0.499302\n",
      "[900]\ttest set's multi_logloss: 0.500549\n",
      "[1000]\ttest set's multi_logloss: 0.500447\n",
      "[100]\ttest set's multi_logloss: 0.507345\n",
      "[200]\ttest set's multi_logloss: 0.510265\n",
      "[300]\ttest set's multi_logloss: 0.52629\n",
      "[400]\ttest set's multi_logloss: 0.532559\n",
      "[500]\ttest set's multi_logloss: 0.538848\n",
      "[600]\ttest set's multi_logloss: 0.551728\n",
      "[700]\ttest set's multi_logloss: 0.553363\n",
      "[800]\ttest set's multi_logloss: 0.569471\n",
      "[900]\ttest set's multi_logloss: 0.574567\n",
      "[1000]\ttest set's multi_logloss: 0.581055\n",
      "[100]\ttest set's multi_logloss: 0.511682\n",
      "[200]\ttest set's multi_logloss: 0.512572\n",
      "[300]\ttest set's multi_logloss: 0.527582\n",
      "[400]\ttest set's multi_logloss: 0.535383\n",
      "[500]\ttest set's multi_logloss: 0.541453\n",
      "[600]\ttest set's multi_logloss: 0.553702\n",
      "[700]\ttest set's multi_logloss: 0.561298\n",
      "[800]\ttest set's multi_logloss: 0.574092\n",
      "[900]\ttest set's multi_logloss: 0.577726\n",
      "[1000]\ttest set's multi_logloss: 0.584518\n",
      "[100]\ttest set's multi_logloss: 0.509285\n",
      "[200]\ttest set's multi_logloss: 0.513085\n",
      "[300]\ttest set's multi_logloss: 0.529191\n",
      "[400]\ttest set's multi_logloss: 0.537137\n",
      "[500]\ttest set's multi_logloss: 0.546411\n",
      "[600]\ttest set's multi_logloss: 0.558445\n",
      "[700]\ttest set's multi_logloss: 0.564401\n",
      "[800]\ttest set's multi_logloss: 0.580753\n",
      "[900]\ttest set's multi_logloss: 0.586257\n",
      "[1000]\ttest set's multi_logloss: 0.593752\n",
      "[100]\ttest set's multi_logloss: 0.505059\n",
      "[200]\ttest set's multi_logloss: 0.507367\n",
      "[300]\ttest set's multi_logloss: 0.519187\n",
      "[400]\ttest set's multi_logloss: 0.527911\n",
      "[500]\ttest set's multi_logloss: 0.535129\n",
      "[600]\ttest set's multi_logloss: 0.548691\n",
      "[700]\ttest set's multi_logloss: 0.555283\n",
      "[800]\ttest set's multi_logloss: 0.568637\n",
      "[900]\ttest set's multi_logloss: 0.575073\n",
      "[1000]\ttest set's multi_logloss: 0.581339\n",
      "[100]\ttest set's multi_logloss: 0.511358\n",
      "[200]\ttest set's multi_logloss: 0.510744\n",
      "[300]\ttest set's multi_logloss: 0.525477\n",
      "[400]\ttest set's multi_logloss: 0.532246\n",
      "[500]\ttest set's multi_logloss: 0.538146\n",
      "[600]\ttest set's multi_logloss: 0.549845\n",
      "[700]\ttest set's multi_logloss: 0.559735\n",
      "[800]\ttest set's multi_logloss: 0.572458\n",
      "[900]\ttest set's multi_logloss: 0.576462\n",
      "[1000]\ttest set's multi_logloss: 0.581274\n",
      "[100]\ttest set's multi_logloss: 5.78235\n",
      "[200]\ttest set's multi_logloss: 6.82149\n",
      "[300]\ttest set's multi_logloss: 12.8395\n",
      "[400]\ttest set's multi_logloss: 15.6293\n",
      "[500]\ttest set's multi_logloss: 19.6246\n",
      "[600]\ttest set's multi_logloss: 7.83166\n",
      "[700]\ttest set's multi_logloss: 6.9577\n",
      "[800]\ttest set's multi_logloss: 6.94635\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 7.21875\n",
      "[200]\ttest set's multi_logloss: 6.84419\n",
      "[300]\ttest set's multi_logloss: 6.75339\n",
      "[400]\ttest set's multi_logloss: 6.79879\n",
      "[500]\ttest set's multi_logloss: 6.82149\n",
      "[600]\ttest set's multi_logloss: 6.74204\n",
      "[700]\ttest set's multi_logloss: 6.73069\n",
      "[800]\ttest set's multi_logloss: 6.74204\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 9.21296\n",
      "[200]\ttest set's multi_logloss: 7.2331\n",
      "[300]\ttest set's multi_logloss: 8.10407\n",
      "[400]\ttest set's multi_logloss: 12.4626\n",
      "[500]\ttest set's multi_logloss: 9.58885\n",
      "[600]\ttest set's multi_logloss: 22.3355\n",
      "[700]\ttest set's multi_logloss: 7.0485\n",
      "[800]\ttest set's multi_logloss: 6.82149\n",
      "[900]\ttest set's multi_logloss: 6.82149\n",
      "[1000]\ttest set's multi_logloss: 6.79879\n",
      "[100]\ttest set's multi_logloss: 6.53183\n",
      "[200]\ttest set's multi_logloss: 10.9984\n",
      "[300]\ttest set's multi_logloss: 14.2528\n",
      "[400]\ttest set's multi_logloss: 7.35495\n",
      "[500]\ttest set's multi_logloss: 7.61601\n",
      "[600]\ttest set's multi_logloss: 6.74204\n",
      "[700]\ttest set's multi_logloss: 6.76474\n",
      "[800]\ttest set's multi_logloss: 6.74204\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 6.25341\n",
      "[200]\ttest set's multi_logloss: 9.65273\n",
      "[300]\ttest set's multi_logloss: 11.3048\n",
      "[400]\ttest set's multi_logloss: 18.291\n",
      "[500]\ttest set's multi_logloss: 7.0031\n",
      "[600]\ttest set's multi_logloss: 7.0258\n",
      "[700]\ttest set's multi_logloss: 7.08255\n",
      "[800]\ttest set's multi_logloss: 7.1393\n",
      "[900]\ttest set's multi_logloss: 7.0939\n",
      "[1000]\ttest set's multi_logloss: 6.73069\n",
      "[100]\ttest set's multi_logloss: 0.504384\n",
      "[200]\ttest set's multi_logloss: 0.501379\n",
      "[300]\ttest set's multi_logloss: 0.502335\n",
      "[400]\ttest set's multi_logloss: 0.502911\n",
      "[500]\ttest set's multi_logloss: 0.508163\n",
      "[600]\ttest set's multi_logloss: 0.512226\n",
      "[700]\ttest set's multi_logloss: 0.513743\n",
      "[800]\ttest set's multi_logloss: 0.511357\n",
      "[900]\ttest set's multi_logloss: 0.512686\n",
      "[1000]\ttest set's multi_logloss: 0.510217\n",
      "[100]\ttest set's multi_logloss: 0.511665\n",
      "[200]\ttest set's multi_logloss: 0.507232\n",
      "[300]\ttest set's multi_logloss: 0.509226\n",
      "[400]\ttest set's multi_logloss: 0.510428\n",
      "[500]\ttest set's multi_logloss: 0.513744\n",
      "[600]\ttest set's multi_logloss: 0.519035\n",
      "[700]\ttest set's multi_logloss: 0.522173\n",
      "[800]\ttest set's multi_logloss: 0.520262\n",
      "[900]\ttest set's multi_logloss: 0.517916\n",
      "[1000]\ttest set's multi_logloss: 0.516646\n",
      "[100]\ttest set's multi_logloss: 0.509887\n",
      "[200]\ttest set's multi_logloss: 0.503481\n",
      "[300]\ttest set's multi_logloss: 0.505861\n",
      "[400]\ttest set's multi_logloss: 0.511167\n",
      "[500]\ttest set's multi_logloss: 0.513314\n",
      "[600]\ttest set's multi_logloss: 0.521465\n",
      "[700]\ttest set's multi_logloss: 0.527484\n",
      "[800]\ttest set's multi_logloss: 0.523593\n",
      "[900]\ttest set's multi_logloss: 0.528296\n",
      "[1000]\ttest set's multi_logloss: 0.526547\n",
      "[100]\ttest set's multi_logloss: 0.504046\n",
      "[200]\ttest set's multi_logloss: 0.499315\n",
      "[300]\ttest set's multi_logloss: 0.503301\n",
      "[400]\ttest set's multi_logloss: 0.510742\n",
      "[500]\ttest set's multi_logloss: 0.513568\n",
      "[600]\ttest set's multi_logloss: 0.519225\n",
      "[700]\ttest set's multi_logloss: 0.522473\n",
      "[800]\ttest set's multi_logloss: 0.520869\n",
      "[900]\ttest set's multi_logloss: 0.519094\n",
      "[1000]\ttest set's multi_logloss: 0.520699\n",
      "[100]\ttest set's multi_logloss: 0.508798\n",
      "[200]\ttest set's multi_logloss: 0.508129\n",
      "[300]\ttest set's multi_logloss: 0.510226\n",
      "[400]\ttest set's multi_logloss: 0.513483\n",
      "[500]\ttest set's multi_logloss: 0.517271\n",
      "[600]\ttest set's multi_logloss: 0.518238\n",
      "[700]\ttest set's multi_logloss: 0.516286\n",
      "[800]\ttest set's multi_logloss: 0.518178\n",
      "[900]\ttest set's multi_logloss: 0.515675\n",
      "[1000]\ttest set's multi_logloss: 0.518564\n",
      "[100]\ttest set's multi_logloss: 3.89964\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[200]\ttest set's multi_logloss: 5.67251\n",
      "[300]\ttest set's multi_logloss: 9.80333\n",
      "[400]\ttest set's multi_logloss: 8.19888\n",
      "[500]\ttest set's multi_logloss: 7.00055\n",
      "[600]\ttest set's multi_logloss: 11.7702\n",
      "[700]\ttest set's multi_logloss: 10.9727\n",
      "[800]\ttest set's multi_logloss: 10.9604\n",
      "[900]\ttest set's multi_logloss: 7.28685\n",
      "[1000]\ttest set's multi_logloss: 10.1076\n",
      "[100]\ttest set's multi_logloss: 4.72089\n",
      "[200]\ttest set's multi_logloss: 9.10376\n",
      "[300]\ttest set's multi_logloss: 9.68413\n",
      "[400]\ttest set's multi_logloss: 8.24027\n",
      "[500]\ttest set's multi_logloss: 16.1776\n",
      "[600]\ttest set's multi_logloss: 7.162\n",
      "[700]\ttest set's multi_logloss: 7.05985\n",
      "[800]\ttest set's multi_logloss: 6.76474\n",
      "[900]\ttest set's multi_logloss: 6.75339\n",
      "[1000]\ttest set's multi_logloss: 6.75339\n",
      "[100]\ttest set's multi_logloss: 4.94597\n",
      "[200]\ttest set's multi_logloss: 5.27581\n",
      "[300]\ttest set's multi_logloss: 8.4393\n",
      "[400]\ttest set's multi_logloss: 8.02173\n",
      "[500]\ttest set's multi_logloss: 6.74204\n",
      "[600]\ttest set's multi_logloss: 6.74204\n",
      "[700]\ttest set's multi_logloss: 6.74204\n",
      "[800]\ttest set's multi_logloss: 6.74204\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 2.25659\n",
      "[200]\ttest set's multi_logloss: 2.28611\n",
      "[300]\ttest set's multi_logloss: 2.20027\n",
      "[400]\ttest set's multi_logloss: 2.13022\n",
      "[500]\ttest set's multi_logloss: 2.06388\n",
      "[600]\ttest set's multi_logloss: 2.04867\n",
      "[700]\ttest set's multi_logloss: 2.00455\n",
      "[800]\ttest set's multi_logloss: 1.99064\n",
      "[900]\ttest set's multi_logloss: 1.96812\n",
      "[1000]\ttest set's multi_logloss: 1.94246\n",
      "[100]\ttest set's multi_logloss: 2.41123\n",
      "[200]\ttest set's multi_logloss: 2.58107\n",
      "[300]\ttest set's multi_logloss: 3.6525\n",
      "[400]\ttest set's multi_logloss: 4.18917\n",
      "[500]\ttest set's multi_logloss: 10.7825\n",
      "[600]\ttest set's multi_logloss: 12.3718\n",
      "[700]\ttest set's multi_logloss: 7.94517\n",
      "[800]\ttest set's multi_logloss: 6.86689\n",
      "[900]\ttest set's multi_logloss: 6.86689\n",
      "[1000]\ttest set's multi_logloss: 6.87824\n",
      "[100]\ttest set's multi_logloss: 6.935\n",
      "[200]\ttest set's multi_logloss: 6.9577\n",
      "[300]\ttest set's multi_logloss: 6.9577\n",
      "[400]\ttest set's multi_logloss: 6.74204\n",
      "[500]\ttest set's multi_logloss: 6.74204\n",
      "[600]\ttest set's multi_logloss: 6.74204\n",
      "[700]\ttest set's multi_logloss: 6.74204\n",
      "[800]\ttest set's multi_logloss: 6.74204\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 7.2074\n",
      "[200]\ttest set's multi_logloss: 8.29702\n",
      "[300]\ttest set's multi_logloss: 8.41053\n",
      "[400]\ttest set's multi_logloss: 13.3819\n",
      "[500]\ttest set's multi_logloss: 9.69053\n",
      "[600]\ttest set's multi_logloss: 8.63419\n",
      "[700]\ttest set's multi_logloss: 8.37648\n",
      "[800]\ttest set's multi_logloss: 10.3957\n",
      "[900]\ttest set's multi_logloss: 13.6203\n",
      "[1000]\ttest set's multi_logloss: 11.9972\n",
      "[100]\ttest set's multi_logloss: 7.31087\n",
      "[200]\ttest set's multi_logloss: 7.75221\n",
      "[300]\ttest set's multi_logloss: 7.24145\n",
      "[400]\ttest set's multi_logloss: 7.162\n",
      "[500]\ttest set's multi_logloss: 6.75339\n",
      "[600]\ttest set's multi_logloss: 6.74204\n",
      "[700]\ttest set's multi_logloss: 6.74204\n",
      "[800]\ttest set's multi_logloss: 6.74204\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 6.75339\n",
      "[200]\ttest set's multi_logloss: 6.75339\n",
      "[300]\ttest set's multi_logloss: 6.75339\n",
      "[400]\ttest set's multi_logloss: 6.74204\n",
      "[500]\ttest set's multi_logloss: 6.74204\n",
      "[600]\ttest set's multi_logloss: 6.75339\n",
      "[700]\ttest set's multi_logloss: 6.74204\n",
      "[800]\ttest set's multi_logloss: 6.74204\n",
      "[900]\ttest set's multi_logloss: 6.74204\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 16.7189\n",
      "[200]\ttest set's multi_logloss: 6.78744\n",
      "[300]\ttest set's multi_logloss: 8.61483\n",
      "[400]\ttest set's multi_logloss: 8.61483\n",
      "[500]\ttest set's multi_logloss: 7.08255\n",
      "[600]\ttest set's multi_logloss: 6.74204\n",
      "[700]\ttest set's multi_logloss: 6.74204\n",
      "[800]\ttest set's multi_logloss: 7.1847\n",
      "[900]\ttest set's multi_logloss: 7.33225\n",
      "[1000]\ttest set's multi_logloss: 7.08255\n",
      "[100]\ttest set's multi_logloss: 12.3604\n",
      "[200]\ttest set's multi_logloss: 27.093\n",
      "[300]\ttest set's multi_logloss: 29.0226\n",
      "[400]\ttest set's multi_logloss: 29.5333\n",
      "[500]\ttest set's multi_logloss: 22.3373\n",
      "[600]\ttest set's multi_logloss: 22.3373\n",
      "[700]\ttest set's multi_logloss: 22.3373\n",
      "[800]\ttest set's multi_logloss: 22.3373\n",
      "[900]\ttest set's multi_logloss: 22.3373\n",
      "[1000]\ttest set's multi_logloss: 22.3373\n",
      "[100]\ttest set's multi_logloss: 8.25162\n",
      "[200]\ttest set's multi_logloss: 7.91112\n",
      "[300]\ttest set's multi_logloss: 7.44576\n",
      "[400]\ttest set's multi_logloss: 7.44576\n",
      "[500]\ttest set's multi_logloss: 7.44576\n",
      "[600]\ttest set's multi_logloss: 7.44576\n",
      "[700]\ttest set's multi_logloss: 7.44576\n",
      "[800]\ttest set's multi_logloss: 7.74086\n",
      "[900]\ttest set's multi_logloss: 7.74086\n",
      "[1000]\ttest set's multi_logloss: 7.86572\n",
      "[100]\ttest set's multi_logloss: 7.2074\n",
      "[200]\ttest set's multi_logloss: 7.1847\n",
      "[300]\ttest set's multi_logloss: 7.1847\n",
      "[400]\ttest set's multi_logloss: 7.3663\n",
      "[500]\ttest set's multi_logloss: 7.1847\n",
      "[600]\ttest set's multi_logloss: 7.1847\n",
      "[700]\ttest set's multi_logloss: 7.1847\n",
      "[800]\ttest set's multi_logloss: 7.15065\n",
      "[900]\ttest set's multi_logloss: 7.15065\n",
      "[1000]\ttest set's multi_logloss: 7.15065\n",
      "[100]\ttest set's multi_logloss: 9.52285\n",
      "[200]\ttest set's multi_logloss: 9.4661\n",
      "[300]\ttest set's multi_logloss: 9.4661\n",
      "[400]\ttest set's multi_logloss: 11.0097\n",
      "[500]\ttest set's multi_logloss: 11.0097\n",
      "[600]\ttest set's multi_logloss: 11.0097\n",
      "[700]\ttest set's multi_logloss: 11.0097\n",
      "[800]\ttest set's multi_logloss: 11.0097\n",
      "[900]\ttest set's multi_logloss: 11.0097\n",
      "[1000]\ttest set's multi_logloss: 11.0097\n",
      "[100]\ttest set's multi_logloss: 6.94635\n",
      "[200]\ttest set's multi_logloss: 6.94635\n",
      "[300]\ttest set's multi_logloss: 8.53538\n",
      "[400]\ttest set's multi_logloss: 6.83284\n",
      "[500]\ttest set's multi_logloss: 6.83284\n",
      "[600]\ttest set's multi_logloss: 7.0712\n",
      "[700]\ttest set's multi_logloss: 7.0712\n",
      "[800]\ttest set's multi_logloss: 6.9804\n",
      "[900]\ttest set's multi_logloss: 6.9804\n",
      "[1000]\ttest set's multi_logloss: 6.9804\n",
      "[100]\ttest set's multi_logloss: 0.518305\n",
      "[200]\ttest set's multi_logloss: 0.52395\n",
      "[300]\ttest set's multi_logloss: 0.550612\n",
      "[400]\ttest set's multi_logloss: 0.57153\n",
      "[500]\ttest set's multi_logloss: 0.603966\n",
      "[600]\ttest set's multi_logloss: 0.632923\n",
      "[700]\ttest set's multi_logloss: 0.657157\n",
      "[800]\ttest set's multi_logloss: 0.697218\n",
      "[900]\ttest set's multi_logloss: 0.712484\n",
      "[1000]\ttest set's multi_logloss: 0.729907\n",
      "[100]\ttest set's multi_logloss: 0.513586\n",
      "[200]\ttest set's multi_logloss: 0.523168\n",
      "[300]\ttest set's multi_logloss: 0.551873\n",
      "[400]\ttest set's multi_logloss: 0.575728\n",
      "[500]\ttest set's multi_logloss: 0.601726\n",
      "[600]\ttest set's multi_logloss: 0.634202\n",
      "[700]\ttest set's multi_logloss: 0.656055\n",
      "[800]\ttest set's multi_logloss: 0.699309\n",
      "[900]\ttest set's multi_logloss: 0.716988\n",
      "[1000]\ttest set's multi_logloss: 0.733868\n",
      "[100]\ttest set's multi_logloss: 0.517338\n",
      "[200]\ttest set's multi_logloss: 0.523553\n",
      "[300]\ttest set's multi_logloss: 0.553848\n",
      "[400]\ttest set's multi_logloss: 0.575435\n",
      "[500]\ttest set's multi_logloss: 0.593066\n",
      "[600]\ttest set's multi_logloss: 0.624847\n",
      "[700]\ttest set's multi_logloss: 0.651003\n",
      "[800]\ttest set's multi_logloss: 0.695841\n",
      "[900]\ttest set's multi_logloss: 0.712228\n",
      "[1000]\ttest set's multi_logloss: 0.730772\n",
      "[100]\ttest set's multi_logloss: 0.509994\n",
      "[200]\ttest set's multi_logloss: 0.515078\n",
      "[300]\ttest set's multi_logloss: 0.538566\n",
      "[400]\ttest set's multi_logloss: 0.560239\n",
      "[500]\ttest set's multi_logloss: 0.57741\n",
      "[600]\ttest set's multi_logloss: 0.605672\n",
      "[700]\ttest set's multi_logloss: 0.633806\n",
      "[800]\ttest set's multi_logloss: 0.679482\n",
      "[900]\ttest set's multi_logloss: 0.692757\n",
      "[1000]\ttest set's multi_logloss: 0.71628\n",
      "[100]\ttest set's multi_logloss: 0.516786\n",
      "[200]\ttest set's multi_logloss: 0.526903\n",
      "[300]\ttest set's multi_logloss: 0.555083\n",
      "[400]\ttest set's multi_logloss: 0.58199\n",
      "[500]\ttest set's multi_logloss: 0.604696\n",
      "[600]\ttest set's multi_logloss: 0.641582\n",
      "[700]\ttest set's multi_logloss: 0.65892\n",
      "[800]\ttest set's multi_logloss: 0.698318\n",
      "[900]\ttest set's multi_logloss: 0.71503\n",
      "[1000]\ttest set's multi_logloss: 0.733179\n",
      "[100]\ttest set's multi_logloss: 6.82149\n",
      "[200]\ttest set's multi_logloss: 6.82149\n",
      "[300]\ttest set's multi_logloss: 6.81014\n",
      "[400]\ttest set's multi_logloss: 6.77609\n",
      "[500]\ttest set's multi_logloss: 6.77609\n",
      "[600]\ttest set's multi_logloss: 6.77609\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[700]\ttest set's multi_logloss: 6.77609\n",
      "[800]\ttest set's multi_logloss: 6.77609\n",
      "[900]\ttest set's multi_logloss: 6.77609\n",
      "[1000]\ttest set's multi_logloss: 6.74204\n",
      "[100]\ttest set's multi_logloss: 7.84301\n",
      "[200]\ttest set's multi_logloss: 7.26415\n",
      "[300]\ttest set's multi_logloss: 6.9123\n",
      "[400]\ttest set's multi_logloss: 6.9123\n",
      "[500]\ttest set's multi_logloss: 6.9123\n",
      "[600]\ttest set's multi_logloss: 9.25044\n",
      "[700]\ttest set's multi_logloss: 9.25044\n",
      "[800]\ttest set's multi_logloss: 9.25044\n",
      "[900]\ttest set's multi_logloss: 9.25044\n",
      "[1000]\ttest set's multi_logloss: 9.25044\n",
      "[100]\ttest set's multi_logloss: 13.2798\n",
      "[200]\ttest set's multi_logloss: 13.2798\n",
      "[300]\ttest set's multi_logloss: 13.2798\n",
      "[400]\ttest set's multi_logloss: 11.3389\n",
      "[500]\ttest set's multi_logloss: 9.25044\n",
      "[600]\ttest set's multi_logloss: 7.17335\n",
      "[700]\ttest set's multi_logloss: 7.17335\n",
      "[800]\ttest set's multi_logloss: 6.78744\n",
      "[900]\ttest set's multi_logloss: 6.78744\n",
      "[1000]\ttest set's multi_logloss: 6.84419\n",
      "[100]\ttest set's multi_logloss: 20.8163\n",
      "[200]\ttest set's multi_logloss: 19.3408\n",
      "[300]\ttest set's multi_logloss: 19.3408\n",
      "[400]\ttest set's multi_logloss: 18.7733\n",
      "[500]\ttest set's multi_logloss: 18.7733\n",
      "[600]\ttest set's multi_logloss: 18.7733\n",
      "[700]\ttest set's multi_logloss: 16.5713\n",
      "[800]\ttest set's multi_logloss: 18.1036\n",
      "[900]\ttest set's multi_logloss: 18.1036\n",
      "[1000]\ttest set's multi_logloss: 18.1036\n",
      "[100]\ttest set's multi_logloss: 12.1561\n",
      "[200]\ttest set's multi_logloss: 7.55926\n",
      "[300]\ttest set's multi_logloss: 7.55926\n",
      "[400]\ttest set's multi_logloss: 9.20504\n",
      "[500]\ttest set's multi_logloss: 9.25044\n",
      "[600]\ttest set's multi_logloss: 9.25044\n",
      "[700]\ttest set's multi_logloss: 9.25044\n",
      "[800]\ttest set's multi_logloss: 9.25044\n",
      "[900]\ttest set's multi_logloss: 9.25044\n",
      "[1000]\ttest set's multi_logloss: 9.25044\n",
      "[100]\ttest set's multi_logloss: 2.99874\n",
      "[200]\ttest set's multi_logloss: 2.94407\n",
      "[300]\ttest set's multi_logloss: 2.94476\n",
      "[400]\ttest set's multi_logloss: 2.81956\n",
      "[500]\ttest set's multi_logloss: 2.79206\n",
      "[600]\ttest set's multi_logloss: 2.69702\n",
      "[700]\ttest set's multi_logloss: 2.65302\n",
      "[800]\ttest set's multi_logloss: 2.63234\n",
      "[900]\ttest set's multi_logloss: 2.56867\n",
      "[1000]\ttest set's multi_logloss: 2.52358\n",
      "[100]\ttest set's multi_logloss: 4.91685\n",
      "[200]\ttest set's multi_logloss: 5.23904\n",
      "[300]\ttest set's multi_logloss: 5.14804\n",
      "[400]\ttest set's multi_logloss: 4.94017\n",
      "[500]\ttest set's multi_logloss: 4.99971\n",
      "[600]\ttest set's multi_logloss: 4.96714\n",
      "[700]\ttest set's multi_logloss: 4.83338\n",
      "[800]\ttest set's multi_logloss: 4.87979\n",
      "[900]\ttest set's multi_logloss: 5.0613\n",
      "[1000]\ttest set's multi_logloss: 4.99224\n",
      "[100]\ttest set's multi_logloss: 3.96695\n",
      "[200]\ttest set's multi_logloss: 3.74812\n",
      "[300]\ttest set's multi_logloss: 3.57391\n",
      "[400]\ttest set's multi_logloss: 3.49245\n",
      "[500]\ttest set's multi_logloss: 3.48407\n",
      "[600]\ttest set's multi_logloss: 3.50848\n",
      "[700]\ttest set's multi_logloss: 3.45773\n",
      "[800]\ttest set's multi_logloss: 3.37581\n",
      "[900]\ttest set's multi_logloss: 3.34605\n",
      "[1000]\ttest set's multi_logloss: 3.34912\n",
      "[100]\ttest set's multi_logloss: 4.58718\n",
      "[200]\ttest set's multi_logloss: 4.50615\n",
      "[300]\ttest set's multi_logloss: 4.54534\n",
      "[400]\ttest set's multi_logloss: 4.48186\n",
      "[500]\ttest set's multi_logloss: 4.35475\n",
      "[600]\ttest set's multi_logloss: 4.47369\n",
      "[700]\ttest set's multi_logloss: 4.33373\n",
      "[800]\ttest set's multi_logloss: 4.33107\n",
      "[900]\ttest set's multi_logloss: 4.32153\n",
      "[1000]\ttest set's multi_logloss: 4.31494\n",
      "[100]\ttest set's multi_logloss: 4.35661\n",
      "[200]\ttest set's multi_logloss: 4.43558\n",
      "[300]\ttest set's multi_logloss: 4.241\n",
      "[400]\ttest set's multi_logloss: 4.52142\n",
      "[500]\ttest set's multi_logloss: 4.61289\n",
      "[600]\ttest set's multi_logloss: 4.73957\n",
      "[700]\ttest set's multi_logloss: 4.80008\n",
      "[800]\ttest set's multi_logloss: 4.66167\n",
      "[900]\ttest set's multi_logloss: 4.56991\n",
      "[1000]\ttest set's multi_logloss: 4.56904\n",
      "[100]\ttest set's multi_logloss: 0.543166\n",
      "[200]\ttest set's multi_logloss: 0.533238\n",
      "[300]\ttest set's multi_logloss: 0.547146\n",
      "[400]\ttest set's multi_logloss: 0.555749\n",
      "[500]\ttest set's multi_logloss: 0.557899\n",
      "[600]\ttest set's multi_logloss: 0.583489\n",
      "[700]\ttest set's multi_logloss: 0.595695\n",
      "[800]\ttest set's multi_logloss: 0.629277\n",
      "[900]\ttest set's multi_logloss: 0.652932\n",
      "[1000]\ttest set's multi_logloss: 0.676953\n",
      "[100]\ttest set's multi_logloss: 0.590817\n",
      "[200]\ttest set's multi_logloss: 0.588654\n",
      "[300]\ttest set's multi_logloss: 0.594738\n",
      "[400]\ttest set's multi_logloss: 0.594683\n",
      "[500]\ttest set's multi_logloss: 0.599666\n",
      "[600]\ttest set's multi_logloss: 0.613695\n",
      "[700]\ttest set's multi_logloss: 0.623612\n",
      "[800]\ttest set's multi_logloss: 0.66394\n",
      "[900]\ttest set's multi_logloss: 0.676307\n",
      "[1000]\ttest set's multi_logloss: 0.699575\n",
      "[100]\ttest set's multi_logloss: 0.562654\n",
      "[200]\ttest set's multi_logloss: 0.551529\n",
      "[300]\ttest set's multi_logloss: 0.555225\n",
      "[400]\ttest set's multi_logloss: 0.567226\n",
      "[500]\ttest set's multi_logloss: 0.575245\n",
      "[600]\ttest set's multi_logloss: 0.600459\n",
      "[700]\ttest set's multi_logloss: 0.609207\n",
      "[800]\ttest set's multi_logloss: 0.638208\n",
      "[900]\ttest set's multi_logloss: 0.65763\n",
      "[1000]\ttest set's multi_logloss: 0.686388\n",
      "[100]\ttest set's multi_logloss: 0.542113\n",
      "[200]\ttest set's multi_logloss: 0.528228\n",
      "[300]\ttest set's multi_logloss: 0.541184\n",
      "[400]\ttest set's multi_logloss: 0.548161\n",
      "[500]\ttest set's multi_logloss: 0.558294\n",
      "[600]\ttest set's multi_logloss: 0.579029\n",
      "[700]\ttest set's multi_logloss: 0.587107\n",
      "[800]\ttest set's multi_logloss: 0.617756\n",
      "[900]\ttest set's multi_logloss: 0.633221\n",
      "[1000]\ttest set's multi_logloss: 0.656063\n",
      "[100]\ttest set's multi_logloss: 0.533459\n",
      "[200]\ttest set's multi_logloss: 0.526592\n",
      "[300]\ttest set's multi_logloss: 0.536667\n",
      "[400]\ttest set's multi_logloss: 0.550677\n",
      "[500]\ttest set's multi_logloss: 0.555585\n",
      "[600]\ttest set's multi_logloss: 0.578666\n",
      "[700]\ttest set's multi_logloss: 0.603047\n",
      "[800]\ttest set's multi_logloss: 0.636159\n",
      "[900]\ttest set's multi_logloss: 0.650074\n",
      "[1000]\ttest set's multi_logloss: 0.670532\n",
      "[100]\ttest set's multi_logloss: 0.531156\n",
      "[200]\ttest set's multi_logloss: 0.523264\n",
      "[300]\ttest set's multi_logloss: 0.555639\n",
      "[400]\ttest set's multi_logloss: 0.574667\n",
      "[500]\ttest set's multi_logloss: 0.594499\n",
      "[600]\ttest set's multi_logloss: 0.609195\n",
      "[700]\ttest set's multi_logloss: 0.629118\n",
      "[800]\ttest set's multi_logloss: 0.640496\n",
      "[900]\ttest set's multi_logloss: 0.6444\n",
      "[1000]\ttest set's multi_logloss: 0.649826\n",
      "[100]\ttest set's multi_logloss: 0.536922\n",
      "[200]\ttest set's multi_logloss: 0.537759\n",
      "[300]\ttest set's multi_logloss: 0.560906\n",
      "[400]\ttest set's multi_logloss: 0.584164\n",
      "[500]\ttest set's multi_logloss: 0.605367\n",
      "[600]\ttest set's multi_logloss: 0.617213\n",
      "[700]\ttest set's multi_logloss: 0.638866\n",
      "[800]\ttest set's multi_logloss: 0.651215\n",
      "[900]\ttest set's multi_logloss: 0.655289\n",
      "[1000]\ttest set's multi_logloss: 0.662273\n",
      "[100]\ttest set's multi_logloss: 0.531121\n",
      "[200]\ttest set's multi_logloss: 0.536591\n",
      "[300]\ttest set's multi_logloss: 0.566772\n",
      "[400]\ttest set's multi_logloss: 0.586273\n",
      "[500]\ttest set's multi_logloss: 0.607386\n",
      "[600]\ttest set's multi_logloss: 0.62097\n",
      "[700]\ttest set's multi_logloss: 0.637325\n",
      "[800]\ttest set's multi_logloss: 0.650515\n",
      "[900]\ttest set's multi_logloss: 0.657233\n",
      "[1000]\ttest set's multi_logloss: 0.661441\n",
      "[100]\ttest set's multi_logloss: 0.524678\n",
      "[200]\ttest set's multi_logloss: 0.526245\n",
      "[300]\ttest set's multi_logloss: 0.550346\n",
      "[400]\ttest set's multi_logloss: 0.57017\n",
      "[500]\ttest set's multi_logloss: 0.596017\n",
      "[600]\ttest set's multi_logloss: 0.612999\n",
      "[700]\ttest set's multi_logloss: 0.631451\n",
      "[800]\ttest set's multi_logloss: 0.645899\n",
      "[900]\ttest set's multi_logloss: 0.650833\n",
      "[1000]\ttest set's multi_logloss: 0.655153\n",
      "[100]\ttest set's multi_logloss: 0.5324\n",
      "[200]\ttest set's multi_logloss: 0.532292\n",
      "[300]\ttest set's multi_logloss: 0.557994\n",
      "[400]\ttest set's multi_logloss: 0.578126\n",
      "[500]\ttest set's multi_logloss: 0.596824\n",
      "[600]\ttest set's multi_logloss: 0.613036\n",
      "[700]\ttest set's multi_logloss: 0.6323\n",
      "[800]\ttest set's multi_logloss: 0.645699\n",
      "[900]\ttest set's multi_logloss: 0.648224\n",
      "[1000]\ttest set's multi_logloss: 0.65399\n",
      "[100]\ttest set's multi_logloss: 0.507439\n",
      "[200]\ttest set's multi_logloss: 0.506469\n",
      "[300]\ttest set's multi_logloss: 0.50738\n",
      "[400]\ttest set's multi_logloss: 0.504915\n",
      "[500]\ttest set's multi_logloss: 0.507492\n",
      "[600]\ttest set's multi_logloss: 0.506856\n",
      "[700]\ttest set's multi_logloss: 0.506519\n",
      "[800]\ttest set's multi_logloss: 0.509236\n",
      "[900]\ttest set's multi_logloss: 0.512197\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1000]\ttest set's multi_logloss: 0.513916\n",
      "[100]\ttest set's multi_logloss: 0.508581\n",
      "[200]\ttest set's multi_logloss: 0.507406\n",
      "[300]\ttest set's multi_logloss: 0.505363\n",
      "[400]\ttest set's multi_logloss: 0.504794\n",
      "[500]\ttest set's multi_logloss: 0.506412\n",
      "[600]\ttest set's multi_logloss: 0.504288\n",
      "[700]\ttest set's multi_logloss: 0.50336\n",
      "[800]\ttest set's multi_logloss: 0.506148\n",
      "[900]\ttest set's multi_logloss: 0.50987\n",
      "[1000]\ttest set's multi_logloss: 0.509836\n",
      "[100]\ttest set's multi_logloss: 0.506834\n",
      "[200]\ttest set's multi_logloss: 0.504142\n",
      "[300]\ttest set's multi_logloss: 0.500984\n",
      "[400]\ttest set's multi_logloss: 0.503884\n",
      "[500]\ttest set's multi_logloss: 0.504228\n",
      "[600]\ttest set's multi_logloss: 0.504247\n",
      "[700]\ttest set's multi_logloss: 0.506116\n",
      "[800]\ttest set's multi_logloss: 0.506846\n",
      "[900]\ttest set's multi_logloss: 0.506633\n",
      "[1000]\ttest set's multi_logloss: 0.507928\n",
      "[100]\ttest set's multi_logloss: 0.50562\n",
      "[200]\ttest set's multi_logloss: 0.501403\n",
      "[300]\ttest set's multi_logloss: 0.500096\n",
      "[400]\ttest set's multi_logloss: 0.498676\n",
      "[500]\ttest set's multi_logloss: 0.502137\n",
      "[600]\ttest set's multi_logloss: 0.503107\n",
      "[700]\ttest set's multi_logloss: 0.503608\n",
      "[800]\ttest set's multi_logloss: 0.503547\n",
      "[900]\ttest set's multi_logloss: 0.504255\n",
      "[1000]\ttest set's multi_logloss: 0.505527\n",
      "[100]\ttest set's multi_logloss: 0.507996\n",
      "[200]\ttest set's multi_logloss: 0.504339\n",
      "[300]\ttest set's multi_logloss: 0.504983\n",
      "[400]\ttest set's multi_logloss: 0.505123\n",
      "[500]\ttest set's multi_logloss: 0.504251\n",
      "[600]\ttest set's multi_logloss: 0.507265\n",
      "[700]\ttest set's multi_logloss: 0.508708\n",
      "[800]\ttest set's multi_logloss: 0.509431\n",
      "[900]\ttest set's multi_logloss: 0.51062\n",
      "[1000]\ttest set's multi_logloss: 0.512596\n",
      "[100]\ttest set's multi_logloss: 0.501416\n",
      "[200]\ttest set's multi_logloss: 0.49877\n",
      "[300]\ttest set's multi_logloss: 0.506143\n",
      "[400]\ttest set's multi_logloss: 0.513561\n",
      "[500]\ttest set's multi_logloss: 0.519824\n",
      "[600]\ttest set's multi_logloss: 0.536101\n",
      "[700]\ttest set's multi_logloss: 0.542278\n",
      "[800]\ttest set's multi_logloss: 0.561061\n",
      "[900]\ttest set's multi_logloss: 0.568096\n",
      "[1000]\ttest set's multi_logloss: 0.575491\n",
      "[100]\ttest set's multi_logloss: 0.504602\n",
      "[200]\ttest set's multi_logloss: 0.505087\n",
      "[300]\ttest set's multi_logloss: 0.514495\n",
      "[400]\ttest set's multi_logloss: 0.522123\n",
      "[500]\ttest set's multi_logloss: 0.527822\n",
      "[600]\ttest set's multi_logloss: 0.537696\n",
      "[700]\ttest set's multi_logloss: 0.548455\n",
      "[800]\ttest set's multi_logloss: 0.56448\n",
      "[900]\ttest set's multi_logloss: 0.569885\n",
      "[1000]\ttest set's multi_logloss: 0.577651\n",
      "[100]\ttest set's multi_logloss: 0.503783\n",
      "[200]\ttest set's multi_logloss: 0.503674\n",
      "[300]\ttest set's multi_logloss: 0.513012\n",
      "[400]\ttest set's multi_logloss: 0.5168\n",
      "[500]\ttest set's multi_logloss: 0.52657\n",
      "[600]\ttest set's multi_logloss: 0.537998\n",
      "[700]\ttest set's multi_logloss: 0.547947\n",
      "[800]\ttest set's multi_logloss: 0.564384\n",
      "[900]\ttest set's multi_logloss: 0.570622\n",
      "[1000]\ttest set's multi_logloss: 0.578049\n",
      "[100]\ttest set's multi_logloss: 0.499548\n",
      "[200]\ttest set's multi_logloss: 0.498964\n",
      "[300]\ttest set's multi_logloss: 0.506834\n",
      "[400]\ttest set's multi_logloss: 0.514821\n",
      "[500]\ttest set's multi_logloss: 0.520006\n",
      "[600]\ttest set's multi_logloss: 0.534878\n",
      "[700]\ttest set's multi_logloss: 0.54696\n",
      "[800]\ttest set's multi_logloss: 0.564258\n",
      "[900]\ttest set's multi_logloss: 0.571378\n",
      "[1000]\ttest set's multi_logloss: 0.577614\n",
      "[100]\ttest set's multi_logloss: 0.501432\n",
      "[200]\ttest set's multi_logloss: 0.500619\n",
      "[300]\ttest set's multi_logloss: 0.505256\n",
      "[400]\ttest set's multi_logloss: 0.514357\n",
      "[500]\ttest set's multi_logloss: 0.517691\n",
      "[600]\ttest set's multi_logloss: 0.530433\n",
      "[700]\ttest set's multi_logloss: 0.539225\n",
      "[800]\ttest set's multi_logloss: 0.560151\n",
      "[900]\ttest set's multi_logloss: 0.567357\n",
      "[1000]\ttest set's multi_logloss: 0.577343\n",
      "[100]\ttest set's multi_logloss: 29.3966\n",
      "[200]\ttest set's multi_logloss: 29.3773\n",
      "[300]\ttest set's multi_logloss: 27.5257\n",
      "[400]\ttest set's multi_logloss: 28.8164\n",
      "[500]\ttest set's multi_logloss: 29.3548\n",
      "[600]\ttest set's multi_logloss: 29.4622\n",
      "[700]\ttest set's multi_logloss: 29.5779\n",
      "[800]\ttest set's multi_logloss: 29.6695\n",
      "[900]\ttest set's multi_logloss: 29.8174\n",
      "[1000]\ttest set's multi_logloss: 29.5081\n",
      "[100]\ttest set's multi_logloss: 27.9385\n",
      "[200]\ttest set's multi_logloss: 27.6214\n",
      "[300]\ttest set's multi_logloss: 28.8542\n",
      "[400]\ttest set's multi_logloss: 29.1857\n",
      "[500]\ttest set's multi_logloss: 29.4719\n",
      "[600]\ttest set's multi_logloss: 29.4643\n",
      "[700]\ttest set's multi_logloss: 29.699\n",
      "[800]\ttest set's multi_logloss: 26.2028\n",
      "[900]\ttest set's multi_logloss: 28.221\n",
      "[1000]\ttest set's multi_logloss: 29.1642\n",
      "[100]\ttest set's multi_logloss: 28.6364\n",
      "[200]\ttest set's multi_logloss: 28.7012\n",
      "[300]\ttest set's multi_logloss: 28.8856\n",
      "[400]\ttest set's multi_logloss: 28.353\n",
      "[500]\ttest set's multi_logloss: 28.7453\n",
      "[600]\ttest set's multi_logloss: 29.0888\n",
      "[700]\ttest set's multi_logloss: 29.3243\n",
      "[800]\ttest set's multi_logloss: 29.2755\n",
      "[900]\ttest set's multi_logloss: 28.0897\n",
      "[1000]\ttest set's multi_logloss: 26.6562\n",
      "[100]\ttest set's multi_logloss: 13.6793\n",
      "[200]\ttest set's multi_logloss: 29.1204\n",
      "[300]\ttest set's multi_logloss: 29.6738\n",
      "[400]\ttest set's multi_logloss: 29.4287\n",
      "[500]\ttest set's multi_logloss: 29.0522\n",
      "[600]\ttest set's multi_logloss: 29.2367\n",
      "[700]\ttest set's multi_logloss: 29.1016\n",
      "[800]\ttest set's multi_logloss: 29.5119\n",
      "[900]\ttest set's multi_logloss: 29.7493\n",
      "[1000]\ttest set's multi_logloss: 29.8625\n",
      "[100]\ttest set's multi_logloss: 17.4357\n",
      "[200]\ttest set's multi_logloss: 29.6904\n",
      "[300]\ttest set's multi_logloss: 27.3885\n",
      "[400]\ttest set's multi_logloss: 25.3119\n",
      "[500]\ttest set's multi_logloss: 28.3809\n",
      "[600]\ttest set's multi_logloss: 28.6955\n",
      "[700]\ttest set's multi_logloss: 29.3636\n",
      "[800]\ttest set's multi_logloss: 29.9339\n",
      "[900]\ttest set's multi_logloss: 29.8776\n",
      "[1000]\ttest set's multi_logloss: 29.8852\n",
      "[100]\ttest set's multi_logloss: 5.18704\n",
      "[200]\ttest set's multi_logloss: 4.22883\n",
      "[300]\ttest set's multi_logloss: 6.38055\n",
      "[400]\ttest set's multi_logloss: 6.03104\n",
      "[500]\ttest set's multi_logloss: 5.12092\n",
      "[600]\ttest set's multi_logloss: 22.2906\n",
      "[700]\ttest set's multi_logloss: 22.4735\n",
      "[800]\ttest set's multi_logloss: 28.4447\n",
      "[900]\ttest set's multi_logloss: 29.2819\n",
      "[1000]\ttest set's multi_logloss: 28.7655\n",
      "[100]\ttest set's multi_logloss: 5.65458\n",
      "[200]\ttest set's multi_logloss: 4.99767\n",
      "[300]\ttest set's multi_logloss: 6.75205\n",
      "[400]\ttest set's multi_logloss: 6.50463\n",
      "[500]\ttest set's multi_logloss: 5.82685\n",
      "[600]\ttest set's multi_logloss: 27.4647\n",
      "[700]\ttest set's multi_logloss: 27.2375\n",
      "[800]\ttest set's multi_logloss: 27.5229\n",
      "[900]\ttest set's multi_logloss: 28.5032\n",
      "[1000]\ttest set's multi_logloss: 29.1235\n",
      "[100]\ttest set's multi_logloss: 5.12442\n",
      "[200]\ttest set's multi_logloss: 4.888\n",
      "[300]\ttest set's multi_logloss: 29.6714\n",
      "[400]\ttest set's multi_logloss: 29.5144\n",
      "[500]\ttest set's multi_logloss: 29.5662\n",
      "[600]\ttest set's multi_logloss: 29.8298\n",
      "[700]\ttest set's multi_logloss: 30.1636\n",
      "[800]\ttest set's multi_logloss: 31.6316\n",
      "[900]\ttest set's multi_logloss: 30.8302\n",
      "[1000]\ttest set's multi_logloss: 29.0425\n",
      "[100]\ttest set's multi_logloss: 7.56195\n",
      "[200]\ttest set's multi_logloss: 24.4418\n",
      "[300]\ttest set's multi_logloss: 28.9364\n",
      "[400]\ttest set's multi_logloss: 29.0584\n",
      "[500]\ttest set's multi_logloss: 28.9029\n",
      "[600]\ttest set's multi_logloss: 29.0614\n",
      "[700]\ttest set's multi_logloss: 29.586\n",
      "[800]\ttest set's multi_logloss: 29.581\n",
      "[900]\ttest set's multi_logloss: 29.643\n",
      "[1000]\ttest set's multi_logloss: 29.6002\n",
      "[100]\ttest set's multi_logloss: 5.41118\n",
      "[200]\ttest set's multi_logloss: 28.0305\n",
      "[300]\ttest set's multi_logloss: 28.7219\n",
      "[400]\ttest set's multi_logloss: 29.2583\n",
      "[500]\ttest set's multi_logloss: 29.437\n",
      "[600]\ttest set's multi_logloss: 29.3628\n",
      "[700]\ttest set's multi_logloss: 29.8511\n",
      "[800]\ttest set's multi_logloss: 28.2856\n",
      "[900]\ttest set's multi_logloss: 28.4852\n",
      "[1000]\ttest set's multi_logloss: 29.7853\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed: 18.8min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttest set's multi_logloss: 0.503604\n",
      "[200]\ttest set's multi_logloss: 0.503323\n",
      "[300]\ttest set's multi_logloss: 0.520501\n",
      "[400]\ttest set's multi_logloss: 0.536871\n",
      "[500]\ttest set's multi_logloss: 0.552135\n",
      "[600]\ttest set's multi_logloss: 0.578353\n",
      "[700]\ttest set's multi_logloss: 0.59892\n",
      "[800]\ttest set's multi_logloss: 0.635033\n",
      "[900]\ttest set's multi_logloss: 0.655433\n",
      "[1000]\ttest set's multi_logloss: 0.675652\n"
     ]
    }
   ],
   "source": [
    "dart_params = {'num_leaves': randint(5, 50), \n",
    "               'learning_rate': [0.25, 0.5, 1, 2, 4, 8],\n",
    "               'drop_rate': [0.05, 0.1, 0.15, 0.2],\n",
    "               'subsample': uniform(loc=0.2, scale=0.8), \n",
    "               'colsample_bytree': uniform(loc=0.4, scale=0.6),\n",
    "               'reg_alpha': [0, 1e-1, 1, 10, 100],\n",
    "               'reg_lambda': [0, 1e-1, 1, 10, 100]}\n",
    "\n",
    "start = time.time()\n",
    "ens = lgb.LGBMClassifier(boosting='dart', n_estimators=1000, max_depth=-1, metric='multi_logloss', random_state=42)\n",
    "cv = RandomizedSearchCV(estimator=ens, \n",
    "                        param_distributions=dart_params, \n",
    "                        n_iter=num_random_iters, \n",
    "                        cv=num_cv_folds, \n",
    "                        refit=True,\n",
    "                        random_state=42, verbose=True)\n",
    "\n",
    "cv.fit(Xtrn, ytrn, **fixed_params)\n",
    "run_time['dart'] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DART: Best score: 0.8155001366110529, best params: {'colsample_bytree': 0.9067203092068908, 'drop_rate': 0.05, 'learning_rate': 0.5, 'num_leaves': 28, 'reg_alpha': 0.1, 'reg_lambda': 0, 'subsample': 0.4247476077499046} \n"
     ]
    }
   ],
   "source": [
    "print('DART: Best score: {0}, best params: {1} '.format(cv.best_score_, cv.best_params_))\n",
    "cv_scores['dart'] = cv.best_score_\n",
    "\n",
    "ypred = cv.best_estimator_.predict(Xtst)\n",
    "tst_scores['dart'] = accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'rf': 0.8051450187962624, 'gbdt': 0.8122129268166652, 'goss': 0.8131167212266558, 'dart': 0.8155001366110529}\n",
      "{'rf': 0.8034834045349983, 'gbdt': 0.8120276043378245, 'goss': 0.8139993427538613, 'dart': 0.8245152809727243}\n",
      "{'rf': 122.31625509262085, 'gbdt': 23.24663543701172, 'goss': 19.403377532958984, 'dart': 1144.2085926532745}\n"
     ]
    }
   ],
   "source": [
    "print(cv_scores)\n",
    "print(tst_scores)\n",
    "print(run_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Visualizing the results\n",
    "\n",
    "For each of these approaches, we are interested in looking at two performance measures: the test set accuracy and overall model development time, which includes parameter selection and training time. These are shown in the figure below. The key takeaways are:\n",
    "* GOSS and GBDT perform similarly. However, GOSS runs faster than GBDT. This will be much more pronounced for increasingly larger data sets, especially those with hundreds of thousands of training examples.\n",
    "* DART achieves best training performance. However, this comes at a cost: significantly increased training time. Here, for instance, DART has a running time of close to 20 minutes, compared to random forest (2 min.), GBDT and GOSS (under half a minute).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = np.c_[np.array(list(cv_scores.values())), np.array(list(tst_scores.values()))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAn4AAAEXCAYAAAAzw0lvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5i0lEQVR4nO3de7xWY/7/8ddb6aAiKU1TVNKPiq+wh/mG0TCI4ZvDjNn5GmmQKBnGqGbGyGCUJmZyajDFfJHxHafCaDCU0yBEJ+k4OqEU49DhW31+f6y1d3e7vXd32vfe1Xo/H4/7cd/rOqx9Xffm6rPXtdZ1KSIwMzMzsx3fTjXdADMzMzOrHg78zMzMzDLCgZ+ZmZlZRjjwMzMzM8sIB35mZmZmGeHAz8zMzCwjHPiZmZlZZkiaLylyXpPT9J+measlzZN0SSXn+IWkhZK+lPQXSbum6YPLnDskRZp3oKQZkj6VdHnOuUZIGlTgbm9ou9fxMzMzs6yQNB/4F3BHmrQCmAu8D8wDhgODgJbA3hGxoEz9M4C/Ao8Bk4DrgFsior+kA4AD0qJ7ALcCb0fEIZIeAP4f8ApwEbAr0AZ4HDgwIlYXoLub8BU/MzMzy5p5wJMR8WBEjGdDPLQIeBb4EFgNrCqnbtf0/XcRcX1a9lyAiJianvNBoH5abmT63gCYTxL41QbqATcBA6sr6AMHfmZmZpY95wD/lvSxpPMiYiYwEDgCeA84GOgdEUvLqftx+t5V0reApkAjSXuUFJAkoDfwb+CBNPle4FRgDMnVwi5AvYh4pIr7VilP9ZqZmVlmSPolMJPkitsQ4BvAfwKPkgR11wBXA/sCHSNiYZn6TYEXgf3TpC+AhkDDiPgyLXMM8Bxwe0T0zanbGmgGTAXeBHoAPwLOBmYDZ0fEkqrv9QYFveInqZukmZJmSxpYTv5uksZJekfSNEm90vR6kl7PSb8mp04TSc9ImpW+717IPphZtkkalV4VmJqTNkzSe5LelfSopMY5eYPSMW+mpBNy0g+VNCXNG5FeETCzahYR10fEXyPiPuAvQC1gH5J7+h6JiMeBR4BGJAFhSVxSJ62/DDgI+BbJPXuLgQ9Kgr5Un/R9ZE4aEfGviJhEco/fi8Aa4BfAd9Ii/au4u5soWOAnqRZwG3Ai0BHoIaljmWJ9gekRcRDJnPnw9ItdDRyTpncGukn6dlpnIPBcRLQniaY3CSjNzKrQPUC3MmnPAAdExH+Q3BA+CCAd44qBTmmd29OxEJIbyXsD7dNX2XOaWYGlT9aOk3SxpP4kU74rgZKrbGdLOg/47/T4/fR9JfBWeo5vAoNJHuK4liT4+13Oz9iTZEr35YiYUk4bmpIEeFeRBJ0APwHakdz7V1CFvOJ3GDA7IuZGxBrgQaB7mTJBMi8uksuky4G1kfgiLbNz+iqZk+5OMk8OG+bLzcwKIiImkoxNuWl/j4i16eE/gVbp5+7AgxGxOiLmkUzdHCapBbBrRLwayf01f8Zjl1lNWEYSbP2GZJr3X8Bp6f/nPwPqkly0qgv0i4h3yjnHeuA04I8kV+quIXl6t8RPSOKWkZtWBZJgcURELI2IacDt6c9eXuY8BVGwe/wk/QDoFhHnp8c/Bg6PiH45ZRoBY0nmyRsBP4qIJ9O8WiTz3/sCt0XEgDT904honHOOFRGxyXSvpN4kf13ToEGDQ/fff/+yRcxsO/bmm28ui4hm1fGzJLUBnoiIA8rJGwf8JSLuk3Qr8M90CglJfwL+RvIk35CI+F6afhQwICJO3tzPbtq0abRp06aqumJm24DqHL/KKuQlxfLuXykbZZ4ATAaOIbnE+YykFyPi3xGxDuic3jvzqKQDImIqeYqIO4E7AYqKimLSpElfowtmtq2S9K9toA2/BNYC95cklVMsKkmv6Lylf7juvffeePwy27HU5PhVyKnehcBeOcetSG6AzNWL5EbKiIjZJOvqbHRpLiI+BV5gw/0wH6XTJqTvH2NmVs0k9QROBv47NkydVDTuLWTDdHBuerki4s6IKIqIombNauSigJntoAoZ+L0BtJfUNn1go5hkWjfXB8CxAJKaA/sBcyU1K3lKTlJ94Hsk6+qQnqNn+rknyYrXZmbVRlI3YADwXxHxVU7WWKBYUl1JbUke4ng9XZ7hc0nfTu9pPgePXWZWAwoW+KU3PvcDxgMzgIciYpqkPpJKHnO+FugiaQrJE7oD0sekWwDPS3qXJIB8JiKeSOsMAY6TNAs4Lj02MysISWOAV4H9lOzNeR7JDdiNSG5PmSxpJEB6o/ZDwHTgaaBvetsKJMs33E3ywMccknv/zKwa9e/fn+bNmyOJk08+Oe88gFWrVrHffvshiX79+uWdV+LVV1+lS5cuNG7cGJJb2R6WVHpJX9JfJa1Qsr9vwR7yKOhjwxHxFPBUmbSROZ8XA8eXU+9dklWzyzvnJ6RXCc3MCi0iepST/KdKyl8PXF9O+iQ27OFpZjWkuLiYESNGbHHeb37zGxYuXLjFeSXef/99mjZtytChQ+nTp89nwOkkO3v0SousJllEuldF56gK3rLNzMzMMmHEiBFcdtllW5z37rvvcvPNNzN48OAtysvVo0cPxo4dy4UXXgjJMjKQrPkJQET8N8lST5WSNFjSR5JWpQvCn7W5OrkKvlCgmZmZ2fZq/fr1nH/++fTt25dvfetbeeeVVadOndzD3dL3iVvSlnS3squBCSSLy7dhCy/i+YqfmZmZWQVGjx7N/PnzOeecc1i0aBEAn332GUuXLq00ryIvv/wyJAHbmyQ7gGyJL4APSR4c60KyYsAjW3ICX/EzMzMzq8CCBQtYunQpBx10UGnafffdR926dWnVqlWFeXfffTerVq1ip512Kr3aN3HiRL7//e9Dcj/fCTm7lOUlIv5P0kHAGSTPQowk2fL27HQFlZ0iYlVl53DgZ2ZmZpnw5JNPMnVqshfEggULuPvuuzn66KNp3759hXlnnnkmBxyQPJc1bdo0Bg8eTLdu3bjooouoX79+hXkA9evXp1OnTkydOpW33nqLE088kXTZz6UkK5R8GRHjACT9CChKm9pR0vnAk+lyUKRlGgE3kqw0MAk4C/hmmv134GhJzdIVUsrlwM/MzMwyYdiwYUyYMAFIHsq44IILGD16NO3bt68w79xzz6Vjx44ANG3aFIB27dpx6KGHAlSal+vdd9/lq69Kl/3cGxhD8pDHuDRtKNA6/fzdnNeSnNOsBdqS7Aten2S5vF9tyXdQsL16tyXess1sxyPpzYgo2nzJ7ZvHL7Oq12bgk1V+zvlDvp932Zocv/xwh5mZmVlGOPAzMzMzywgHfmZmZmYZ4cDPzMzMLCMc+JmZmZllhAM/MzMzs4xw4GdmZmaWEQ78zMzMzDLCgZ+ZmZlZRjjwMzMzM8sIB35mZmZmGeHAz8zMzCwjHPiZmZmZZYQDPzMzM7OMcOBnZmZmlhEO/MzMzMwywoGfmZmZWUY48DMzMzPLCAd+ZmZmZhnhwM/MzMwsIxz4mZmZmWWEAz8zMzOzjHDgZ2ZmZpYRDvzMzMzMMqKggZ+kbpJmSpotaWA5+btJGifpHUnTJPVK0/eS9LykGWn6pTl1BktaJGly+jqpkH0ws2yTNErSx5Km5qQ1kfSMpFnp++45eYPSMW+mpBNy0g+VNCXNGyFJ1d0XM7OCBX6SagG3AScCHYEekjqWKdYXmB4RBwFdgeGS6gBrgZ9FRAfg20DfMnVvjojO6eupQvXBzAy4B+hWJm0g8FxEtAeeS49Jx6lioFNa5/Z0LAS4A+gNtE9fZc9pZlZwhbzidxgwOyLmRsQa4EGge5kyATRK//JtCCwH1kbEkoh4CyAiPgdmAC0L2FYzs3JFxESSsSlXd+De9PO9wKk56Q9GxOqImAfMBg6T1ALYNSJejYgA/pxTx8ys2hQy8GsJLMg5XsimwdutQAdgMTAFuDQi1ucWkNQGOBh4LSe5n6R30ymY3SmHpN6SJkmatHTp0q3riZnZxppHxBKA9H3PNL2ica9l+rlserk8fplZoRQy8Cvv/pUoc3wCMBn4JtAZuFXSrqUnkBoCDwM/jYh/p8l3AO3S8kuA4eX98Ii4MyKKIqKoWbNmX78XZmb5q2jcy2c83JDh8cvMCqSQgd9CYK+c41YkV/Zy9QIeicRsYB6wP4CknUmCvvsj4pGSChHxUUSsS68M3kUypWxmVp0+SqdvSd8/TtMrGvcWpp/LppuZVatCBn5vAO0ltU0f2CgGxpYp8wFwLICk5sB+wNz0nr8/ATMi4qbcCiWDbeo0YCpmZtVrLNAz/dwTeDwnvVhSXUltSR7ieD2dDv5c0rfT8e2cnDpmZtWmdqFOHBFrJfUDxgO1gFERMU1SnzR/JHAtcI+kKSRTIQMiYpmkI4EfA1MkTU5P+Yv0Cd4bJXUmmSaZD1xYqD6Y2WYM3q0KzvHZ1p+jgCSNIVl1oKmkhcDVwBDgIUnnkfwB+0OAdIx7CJhOsjpB34hYl57qIpInhOsDf0tfZmbVqmCBH0AaqD1VJm1kzufFwPHl1HuJ8u+JISJ+XMXNNDOrUET0qCDr2ArKXw9cX076JOCAKmyamdkW884dZmZmZhnhwM/MzMwsIxz4mZmZmWWEAz8zMzOzjHDgZ2ZmZpYRDvzMzMzMMsKBn5mZmVlGOPAzMzMzywgHfmZmZmYZ4cDPzMzMLCMc+JmZmZllhAM/MzMzs4xw4GdmZmaWEQ78zMzMzDLCgZ+ZmZlZRjjwMzMzM8sIB35mZmZmGeHAz8zMzCwjHPiZmZmZZYQDPzMzM7OMcOBnZmZmlhEO/MzMzMwywoGfmZmZWUY48DMzMzPLCAd+ZmZmZhnhwM/MzMwsIxz4mZmZmWWEAz8zMzOzjHDgZ2ZmZpYRDvzMzMzMMsKBn5mZmVlGFDTwk9RN0kxJsyUNLCd/N0njJL0jaZqkXmn6XpKelzQjTb80p04TSc9ImpW+717IPpiZVUTSZekYNVXSGEn1KhujJA1Kx8OZkk6oybabWTYVLPCTVAu4DTgR6Aj0kNSxTLG+wPSIOAjoCgyXVAdYC/wsIjoA3wb65tQdCDwXEe2B59JjM7NqJakl0B8oiogDgFpAMRWMUekYVgx0AroBt6fjpJlZtSnkFb/DgNkRMTci1gAPAt3LlAmgkSQBDYHlwNqIWBIRbwFExOfADKBlWqc7cG/6+V7g1AL2wcysMrWB+pJqA7sAi6l4jOoOPBgRqyNiHjCbZJw0M6s2hQz8WgILco4XsiF4K3Er0IFksJwCXBoR63MLSGoDHAy8liY1j4glAOn7nuX9cEm9JU2SNGnp0qVb2RUzs41FxCLgd8AHwBLgs4j4OxWPUfmMiYDHLzMrnEIGfionLcocnwBMBr4JdAZulbRr6QmkhsDDwE8j4t9b8sMj4s6IKIqIombNmm1JVTOzzUrv3esOtCUZwxpIOruyKuWklR0Tk0SPX2ZWIIUM/BYCe+UctyK5sperF/BIJGYD84D9ASTtTBL03R8Rj+TU+UhSi7RMC+DjArXfzKwy3wPmRcTSiPg/4BGgCxWPUfmMiWZmBVXIwO8NoL2ktukDG8XA2DJlPgCOBZDUHNgPmJve8/cnYEZE3FSmzligZ/q5J/B4gdpvZlaZD4BvS9olHbOOJbkfuaIxaixQLKmupLZAe+D1am6zmWVc7UKdOCLWSuoHjCd52m1UREyT1CfNHwlcC9wjaQrJNMiAiFgm6Ujgx8AUSZPTU/4iIp4ChgAPSTqPZOD9YaH6YGZWkYh4TdJfgbdIViJ4G7iT5EG1TcaodPx7CJielu8bEetqpPFmllkFC/wA0kDtqTJpI3M+LwaOL6feS5R/PwwR8QnpVUIzs5oUEVcDV5dJXk0FY1REXA9cX+h2mZlVxDt3mJmZmWWEAz8zMzOzjHDgZ2aZI6mBd80wsyzKK/CTtLukTpL2keRg0cy2K5J2knSWpCclfQy8ByxJ99kdJql9TbfRzKw6VPhwh6TdSPbS7QHUAZYC9YDmkv4J3B4Rz1dLK83Mts7zwLPAIGBqyQ5BkpoA3wWGSHo0Iu6rwTaamRVcZU/1/hX4M3BURHyamyHpUODHkvaJiD8VsH1mZlXhe+kiyxuJiOUkC8U/nC4ab2a2Q6sw8IuI4yrJexN4syAtMjOrYmWDPkn1gLOB+sADEfFJeYGhmdmOJu/79SQ1k3SdpOGS9i1ko8zMCuwPJAvLrwIeq9mmmJlVny15UGM4MBF4GhhTmOaYmVU9SQ9IapeT1AS4n2Qs271mWmVmVv0qDPwkPS3pqJykOsD89FW3sM0yM6tSvwKulfS79MG135Hsnft3YHBNNszMrDpV9nDHj4CrJF0EXJW+ria5J+biamibmVmViIi5wFnpPuB/AZ4EjvNeuWaWNZU93PEZcIWkfUj2llxEsqn4Z9XVODOzqiBpd+As4P+AM4FTgfGSfh8RT9Rk28zMqlNlU737SBoGnA/8DHgceEjSJV7x3sy2M48Bq0nWIv2fiPgzcApwqKSxNdkwM7PqVNnDHWNIHuT4J8lA+WJEnAD8m+S+GDOz7cUewAMkD3S0BIiIlRFxDXBhTTbMzKw6VXaPXz1gHtAA2KUkMSLulfRQoRtmZlaFrgaeAdYBA3MzImJJjbTIzKwGVBb4XQwMA9YAfXIzImJlIRtlZlaVIuJhkh06zMwyrbKHO14GXq7GtpiZFYSkO4ERETG1nLwGJKsYrI6I+6u9cWZm1ajCwE/SOOCPwPhytjvaBzgXmB8RowraQjOzrXc78GtJBwJTgaUkt7O0B3YFRpHc/2dmtkOrbKr3AuBy4A+SlrNhoGwDzAFujYjHC95CM7OtFBGTgTMlNQSKgBbASmBGRMysybaZmVWnyqZ6PwSuBK6U1IYNA+X7EfFV9TTPzKzqRMQXwAs13Q4zs5pS2RW/UhExn2SrNjMzMzPbTlW2jp+ZmZmZ7UAc+JlZ5qRP8pqZZc5mAz9JJ0tygGhm2z1JXSRNB2akxwdJur2Gm2VmVm3yCeiKgVmSbpTUodANMjMroJuBE4BPACLiHeA7NdoiM7NqtNnALyLOBg4mWcJltKRXJfWW1KjgrTMzq2IRsaBM0roaaYiZWQ3Iawo3Iv5Nst3RgyTLupwGvCXpkgK2zcysqi2Q1AUISXUkXUE67WtmlgX53ON3iqRHgX8AOwOHRcSJwEHAFQVun5lZVeoD9AVaAguBzumxmVkm5LOO3w+BmyNiYm5iRHwl6SeFaZaZWdWLiGXAf9d0O8zMako+gd/VwJKSA0n1geYRMT8initYy8zMqpiktsAlJFtPlo5/EfFfNdUmM7PqlE/g979Al5zjdWnatwrSIjOrUm0GPrnV55g/5PtV0JJtwmPAn4BxwPqabYqZWfXL5+GO2hGxpuQg/Vwnn5NL6iZppqTZkgaWk7+bpHGS3pE0TVKvnLxRkj6WNLVMncGSFkmanL5OyqctZmbAqogYERHPR8SEktfXPZmkxpL+Kuk9STMk/aekJpKekTQrfd89p/ygdDycKemEqumSmVn+8gn8lkoqnQaR1B1YtrlKkmoBtwEnAh2BHpI6linWF5geEQcBXYHhkkqCynuAbhWc/uaI6Jy+nsqjD2ZmAH+QdHUaoB1S8tqa8wFPR8T+JA+8zQAGAs9FRHvgufSYdPwrBjqRjG23p+OkmVm1yWeqtw9wv6RbAQELgHPyqHcYMDsi5gJIehDoDkzPKRNAI0kCGgLLgbUAETFRUps8+2Fmlo8DgR8Dx7BhqjfS4y0iaVeSxZ/PhdLZkDXpH8dd02L3Ai8AA0jGvwcjYjUwT9JsknHy1a/XFTOzLbfZwC8i5gDfltQQUER8nue5W5IEiSUWAoeXKXMrMBZYDDQCfhQR+dx300/SOcAk4GcRsaJsAUm9gd4Ae++9d55NNrMd3GnAPrm3r2yFfYClJAvbHwS8CVxK8vDbEoCIWCJpz7R8S+CfOfUXpmmb8PhlZoWS1wLOkr4PXAxcJunXkn6dT7Vy0qLM8QnAZOCbJOtp3Zr+FV2ZO4B2afklwPDyCkXEnRFRFBFFzZo1y6O5ZpYB7wCNq+hctYFDgDsi4mDgS9Jp3QrkMyYmiR6/zKxA8lnAeSTwI5IlEESyrl/rPM69ENgr57gVyZW9XL2ARyIxG5gH7F/ZSSPio4hYl14ZvItkqsTMLB/NgfckjZc0tuT1Nc+1EFgYEa+lx38lCQQ/ktQCIH3/OKf85sZEM7OCyucevy4R8R+S3o2IayQNBx7Jo94bQPt03axFJDc1n1WmzAfAscCLkpoD+wFzKzuppBYl0ygk0zZTKytvZpbj6qo6UUR8KGmBpP0iYibJWDY9ffUEhqTvj6dVxgIPSLqJZJajPfB6VbXHzCwf+QR+q9L3ryR9E/gEaLu5ShGxVlI/YDxQCxgVEdMk9UnzRwLXAvdImkJyNXFAurI+ksaQ3CDdVNJC4OqI+BNwo6TOJFMk84EL8+yrmWXc1izdUoFLSB5+q0PyR2svkpmUhySdR/LH7Q/Tnz1N0kMkgeFaoG9ErKvi9piZVSqfwG+cpMbAMOAtkoDrrnxOni618lSZtJE5nxcDx1dQt0cF6T/O52ebmZWQ9FJEHCnpcza+r05ARMTm7i0uV0RMBorKyTq2gvLXA9d/nZ9lZlYVKg38JO1Esh7Vp8DDkp4A6kXEZ9XRODOzKvJzgIhoVNMNMTOrSZU+3JE+QDE853i1gz4z2w7dVtMNMDPbFuQz1ft3SWeQPn1b6AaZ7RAG71ZF5/HfWVWkvKVUzMwyJ5/A73KgAbBW0iq28p4YM7Ma0LayZVsi4r8qyjMz25Hks3OH74kxs+3dUipY7N3MLEs2G/hJ+k556RExseqbY2ZWEJ8XYCkXM7PtTj5TvT/P+VyPZKeMN/kam5qbmdWQ+TXdADOzbUE+U72n5B5L2gu4sWAtMjOrYhFxek23wcxsW7DZvXrLsRA4oKobYmZmZmaFlc89frewYaX7nYDOwDsFbJNZtWkz8MmtPsf8Id+vgpaYmZkVXj73+E3K+bwWGBMRLxeoPWZmBSWpJdCanPHPD6uZWVbkE/j9FVhVspm4pFqSdomIrwrbNDOzqiVpKPAjYDqwLk0OwIGfmWVCPoHfc8D3gC/S4/rA34EuhWqUmVmBnArsFxGra7ohZmY1IZ+HO+pFREnQR/p5l8I1ycysYOYCO9d0I8zMako+V/y+lHRIRLwFIOlQYGVhm2VmVhBfAZMlPQeUXvWLiP411yQzs+qTT+D3U+B/JS1Oj1uQ3CNjZra9GZu+zMwyKZ8FnN+QtD+wHyDgvYj4v4K3zGqUlzmxHVFE3FvTbTAzq0n5rOPXF7g/Iqamx7tL6hERtxe8dWZmVUjSPDasS1oqIvapgeaYmVW7fKZ6L4iI20oOImKFpAsAB3729QzerQrO8dnWn8OyqCjncz3gh0CTGmqLmVm1y+ep3p0kqeRAUi2gTuGaZGZWGBHxSc5rUUT8HjimpttlZlZd8rniNx54SNJIkimSPsDTBW2VmVkBSDok53AnkiuAjWqoOWZm1S6fwG8A0Bu4iOThjr8DdxWyUWZmBTI85/NaYD7JdK+ZWSbk81TvemBk+kLSkcAtQN/CNs3MrGpFxHdzjyXVJlme6v2aaZGZWfXK5x4/JHWWNFTSfOBa4L2CtsrMrApJ2lXSIEm3SjpOiX7AbODMmm6fmVl1qfCKn6T/BxQDPYBPgL8AKvsXs5nZduB/gBXAq8AFwJUkD6mdGhGTa7BdZmbVqrKp3veAF4FTImI2gKTLqqVVZmZVa5+IOBBA0t3AMmDviPi8ZptlZla9KpvqPQP4EHhe0l2SjiV5uMPMbHtTuttQRKwD5jnoM7MsqvCKX0Q8CjwqqQFwKnAZ0FzSHcCjEfH36mmimdlWO0jSv9PPAuqnxwIiInatuaaZmVWfzT7cERFfRsT9EXEy0AqYDAwsdMPMzKpKRNSKiF3TV6OIqJ3z2UGfmWVGXk/1loiI5RHxx4jwSvdmZmZm25ktCvzMzMzMbPtV0MBPUjdJMyXNlrTJ9LCk3SSNk/SOpGmSeuXkjZL0saSpZeo0kfSMpFnp++6F7IOZWWUk1ZL0tqQn0uMKx6h0LcHZ6bh4Qs212syyqmCBn6RawG3AiUBHoIekjmWK9QWmR8RBQFdguKQ6ad49QLdyTj0QeC4i2gPP4fsNzaxmXQrMyDkud4xKx79ioBPJ2HZ7Ok6amVWbQl7xOwyYHRFzI2IN8CDQvUyZABpJEtAQWE6yfyYRMTE9Lqs7cG/6+V6SJ47NzKqdpFbA94G7c5IrGqO6Aw9GxOqImEeya8hh1dRUMzOgsIFfS2BBzvHCNC3XrUAHYDEwBbg03Ru4Ms0jYglA+r5neYUk9ZY0SdKkpUuXfp32m5ltzu9JdgHJHbcqGqPyGRMBj19mVjiFDPzKW+w5yhyfQLI8zDeBzsCtkqpkaYWIuDMiiiKiqFmzZlVxSjOzUpJOBj6OiDfzrVJOWtkxMUn0+GVmBVLIwG8hsFfOcSuSK3u5egGPRGI2MA/YfzPn/UhSC4D0/eMqaq+Z2ZY4AvgvSfNJbmU5RtJ9VDxG5TMmmpkVVCEDvzeA9pLapg9sFANjy5T5ADgWQFJzYD9g7mbOOxbomX7uCTxeZS02M8tTRAyKiFYR0YZkfPtHRJxNxWPUWKBYUl1JbYH2wOvV3Gwzy7iCBX4RsRboB4wneeLtoYiYJqmPpD5psWuBLpKmkDz9NiAilgFIGgO8CuwnaaGk89I6Q4DjJM0CjkuPzcy2FeWOURExDXgImA48DfRN9w02M6s2Fe7VWxUi4ingqTJpI3M+LwaOr6BujwrSPyG9Smhmti2IiBeAF9LPFY5REXE9cH21NczMrAzv3GFmZmaWEQ78zMzMzDLCgZ+ZmZlZRjjwMzMzM8sIB35mZmZmGeHAz8zMzCwjHPiZmZmZZYQDPzMzM7OMcOBnZmZmlhEO/MzMzMwywoGfmZmZWUY48DMzMzPLCAd+ZmZmZhnhwM/MzMwsIxz4mZmZmWWEAz8zMzOzjHDgZ2ZmZpYRDvzMzMzMMsKBn5mZmVlGOPAzMzMzywgHfmZmZmYZ4cDPzMzMLCMc+JmZmZllhAM/MzMzs4xw4GdmZmaWEQ78zMzMzDLCgZ+ZmZlZRjjwMzMzM8sIB35mZmZmGeHAz8zMzCwjHPiZmW3nPv30U8455xwaN25Mw4YN+c53vrNJmaVLl9K5c2caNGhAo0aNOProo5k6dWpp/mOPPca+++5LvXr16Nq1K/PmzQNgypQpdOjQgcaNG3PTTTeVlu/fvz833HBD4TtnZlWqoIGfpG6SZkqaLWlgOfm7SRon6R1J0yT12lxdSYMlLZI0OX2dVMg+mJmVR9Jekp6XNCMdvy5N05tIekbSrPR995w6g9IxbaakE6qqLT/5yU+4//77Oe+88/j973/PvvvuW265E088kdtvv52LLrqIiRMncvnllwPw4YcfUlxczK677sqwYcN488036dmzJwA33HADDRo04JxzzmHAgAGsXLmSGTNm8PTTT5fWN7PtR8ECP0m1gNuAE4GOQA9JHcsU6wtMj4iDgK7AcEl18qh7c0R0Tl9PFaoPZmaVWAv8LCI6AN8G+qbj1EDguYhoDzyXHpPmFQOdgG7A7elYt1Xmzp3Lo48+So8ePbjhhhvo1asXo0aN2qRcs2bNuO666zjppJM45phjANhpp+SfgDFjxrB69WoGDRrEJZdcwmmnncaLL77InDlz+PLLL2nTpg1dunRh7dq1rFq1issvv5whQ4ZQt27drW2+mVWz2gU892HA7IiYCyDpQaA7MD2nTACNJAloCCwnGUwPz6NujWkz8MmtPsf8Id+vgpaYWU2JiCXAkvTz55JmAC1JxqquabF7gReAAWn6gxGxGpgnaTbJOPnq1rRj+vRkWHzjjTdo0KABtWrV4tJLL2Xo0KGblJ0yZQoHH3wwAC1btuT3v/89QOm0bsuWLQFo1aoVkASVPXv25Mwzz+Thhx/m1FNP5ZVXXmHVqlWcfvrpW9NsM6shhZzqbQksyDlemKbluhXoACwGpgCXRsT6POr2k/SupFG50yi5JPWWNEnSpKVLl25lV8zMKiapDXAw8BrQPA0KS4LDPdNi+YyJJefLe/xavXo1AF9++SV/+ctfOOKII7jxxht59tlnNym77777Mn78eK699loWL17MjTfeWO45I6KkHZx++unMmTOHN954gzFjxnDllVfyhz/8gV/+8pe0bt2aY489liVLllTaRjPbdhQy8FM5aVHm+ARgMvBNoDNwq6RdN1P3DqBdWn4JMLy8Hx4Rd0ZEUUQUNWvWbEvbbmaWF0kNgYeBn0bEvysrWk5a2TExSdyC8atNmzYAHHXUUZx++umceeaZAMyZM4dVq1axZs2a0rINGzbk+OOP51e/+hV77bUXDz30EABt27YFYOHChQAsWrRoo/TWrVtTVFTEHXfcwVFHHUWdOnX47W9/y8SJEwEYMWJEpW00s21HIad6FwJ75Ry3Irmyl6sXMCSSPy9nS5oH7F9Z3Yj4qCRR0l3AE1XfdDOzzZO0M0nQd39EPJImfySpRUQskdQC+DhNz2dM3GKHHHIIBx54IM899xx33XUXo0ePplatWhxxxBHUr1+fTp06MXXqVEaPHs3kyZPp3Lkz7777Lh988AHf+ta3ACguLmbgwIEMHTqUjz76iEcffZQjjzySdu3alf6cZcuWMWLECF5//XU+/jjp0qhRo5gzZw6HHHLI1nbDzKpJIa/4vQG0l9RWUh2Sm5rHlinzAXAsgKTmwH7A3MrqpgNpidOAqZiZVbP03uQ/ATMi4qacrLFAz/RzT+DxnPRiSXUltQXaA69XQTsYM2YM7dq145JLLmH58uX8+c9/5oADDtioXLNmzXjqqafo06cPf/7znzn55JO5//77AWjRogVjxozh008/5YorruDggw/mnnvu2aj+VVddRf/+/WnWrBmdOnXi4osvZvjw4TRp0oR+/fptbTfMrJoU7IpfRKyV1A8YD9QCRkXENEl90vyRwLXAPZKmkEyDDIiIZQDl1U1PfaOkziRTJPOBCwvVBzOzShwB/BiYImlymvYLYAjwkKTzSP64/SFAOv49RPKQ2lqgb0Ssq4qGdOrUiVdf3fQZkZJ79QBOPvlkTj755ArPcfrpp1f6wMYdd9yx0fFtt93Gbbfd9jVaa2Y1qZBTvaRLrTxVJm1kzufFwPH51k3Tf1zFzTQz22IR8RLl37cH6UxGOXWuB64vWKPMzDbDO3eYmZmZZYQDPzMzM7OMcOBnZrYd69+/P82bN0fSRvfwzZo1i+9+97vsscceNGrUiOOOO445c+ZsNq+syvb4ffXVV+nSpQuNGzemcePGnHHGGXjdVLNtmwM/M7PtXHFx8SZpixYtYv369VxzzTX06tWLZ599lvPPP3+zeeWpaI/f999/n6ZNmzJ06FBOOukkHnnkEa688srCdNLMqkRBH+4wM7PCGjFiBPPnz99kEeUuXbowYcKE0uP777+fadOmbTavrJI9fpcvX07z5s0ZNmxY6R6/PXr0oGfPZOWas846izFjxlR4nsGDB3PHHXfw2Wef0apVK37zm99w1llnff2Om9nX4sDPzGwHVKdOndLPkyZNYvny5ZxxxhmbzStPRXv85p5n/PjxAHznO9/ZpP6KFSu45pprOProozn33HOZP38+69ev//qdM7OvzYHftmbwblVwjs+2/hxmtkOYOXMm3bt3p02bNtxyyy155+Uq2eP39ddf59e//jU33ngjo0aNKs1/+eWX+clPfsKhhx7K4MGDN6nfsGFDvvGNbzBr1ixeeeUVDjvssErXDDSzwvE9fmZmO6jp06dz9NFHU7t2bf7xj3/QokWLvPLy3eMXYOLEiXTr1o127doxfvx4GjZsuEk7dt55Z9555x1+9atfAdCnTx969+4NwJo1a1i1alWV993MyufAz8xsO/bkk0/yl7/8BYAFCxZw9913M2vWLBYsWEDXrl1ZtmwZF110Ea+99hoPPvhgabmK8gDq169fuv/u6NGjufTSSxk9ejSXXXYZH3zwAR07dgTgrbfe4sQTT2TdunVccMEFPPPMM4wbN26TNn7++edceeWV7LTTThQVFVGvXj0WL062KT7++OOpX78+y5YtK+j3ZGYJT/WamW3Hhg0bVvqgxrvvvssFF1zAHif9lNq77Vm6tMqgQYNKyw+c3IhVH7xbYV6J9z/6HNiwx+/IkSNp2LAhJ598MjfddFPpz/vqq68A6Nu3LwCtW7fmlFNO2aiNtWvXZt68eTz++OOsXLmSDh06cN1111Xp92Bm+fEVPzOz7dgLL7xARBARtB7wBK0HPEHDA79Hvb3/o/Q49wVUmgfQesATfPO824Fkj99Zs2axevVqPvnkE8aNG0f79u0BOPfcc0t/dslr/vz5m7Sxfv36TJgwgRUrVrBq1SrefvttunTpslH7mzZtWiXfR0XrGuba3PqDjz32GPvuuy/16tWja9euzJs3D0gecunQoQONGzcuDX5LfuYNN9xQJe2vavl8H1BxnyvL2x6/D3PgZ2ZmO5jy1jXMVdn6gx9++CHFxcXsuuuuDBs2jDfffLN0yZobbriBBg0acM455zBgwABWrlzJjBkzePrpp0vXNtwWbe77qKzPO+L3kXUO/MzMbIcxYsQILrvsskrL9OjRg7Fjx3LhhRfyxz/+EaB0/cExY8awevVqBg0axCWXXMJpp53Giy++yJw5c/jyyy9p06YNXbp0Ye3ataxatYrLL7+cIUOGULdu3YL37evI5/uorM872vdhDvzMzCxjKlt/sGQas2XLlgC0atUKgLlz59KzZ08ee+wxevTowamnnsorr7zCqlWrtvulaSrrcxa/jx2dH+4wM7NM2tz6gwARAYAkTj/9dObMmcPSpUs54IADOPTQQxkzZgy//OUvue+++9h333257777NloaZ3uU2+fK8rLyfexofMXPzMx2eGXXJqxo/cG2bdsCsHDhQiDZ1zg3vXXr1hQVFXHHHXdw1FFHUadOHX77298yceJEgE22zttW5X4flfU5K99HljjwMzOzHUZF6xrmrk1Y2fqDxcXF1KlTh6FDh3LLLbfw6KOPcuSRR9KuXbvSn7Fs2TJGjBjBtddey7p16wAYNWoUc+bMYe3atdXc48rl831U1ucd7fswB35mZrYDGTZsGAMHDgQ2rGv48ssvb1SmZP3BlStX0rdvX3r06MEll1wCQIsWLRgzZgyffvopV1xxBQcffDD33HPPRvWvuuoq+vfvT7NmzejUqRMXX3wxw4cPp0mTJvTr169a+pmvfL6Pyvq8o30fBiqZr9+RFRUVxaRJk6rsfG0GPrnV55g/5PvlZ2wje/VmoY9QwH5WRR/Bv8tKSHozIoq2vgHbti0Zv6rid52rwt+72Xauqv9fgS37/6Umxy9f8TMzMzPLCAd+ZmZmZhnhwM/MzHYIs2bN4rvf/S577LEHjRo14rjjjmPOnDml+YcffjiNGjVil112oaioqPTJ07I2t6WbpI1ep556aqG7ZlZlHPiZmdkOYdGiRaxfv55rrrmGXr168eyzz3L++eeX5nfp0oURI0Zw1VVXMXny5I3yclW2pVuJM844gzFjxjBmzBiuuOKKgvZra1S2V++oUaNo164d9evX54QTTihdqqUiS5cupWnTpkjid7/7XWl6Zfv82rbHgZ+Zme0QunTpwoQJE+jXrx8jRoygSZMmpVuxAdx0002ccsopHHvssdStW5eddir/n8DKtnQr0bFjR0455RSKi4s58sgjyz3P4MGDad68OfXq1WPfffflgQceqKKebpny9uqdNGkS559/Pi1btmTo0KG88MILXHTRRZWe59JLL2XlypUbpVW2l69tmxz4mZnZDiF3K7ZJkyaxfPny0q3YAD777DOaNWvG4YcfTp06dbj77rs3e56yW7qVuO6662jYsCGtW7fmiSee2OQcK1as4JprrqFDhw6MHDmSs88+m/Xr129V/76OivbqnTBhAhHBhRdeSP/+/TnkkEN44okn+OSTT8o9z9/+9jfGjRvHgAEDNkqvbC/fsraVQDjrHPiZmdkOZebMmXTv3p02bdpwyy23lKY3bNiQv//974wYMYJVq1bx61//utLzVLSl24ABA3jkkUe48847WbFiBT169OCrr77aqG7Dhg35xje+waxZs3jllVdo1arVNrWH7Z577gnASy+9xHvvvcesWbOICObPn79J2S+++II+ffpwww03sPfee2+UV9levrm2lUDYHPiZmdkOZPr06Rx99NHUrl2bf/zjHxvtE1u7dm2OO+44LrnkEg477DCef/55li1bBuS/pRvAkCFDOPXUU7ngggs47rjj+OKLL1iwYMFG7dh555155513+NWvfgVAnz596N27NwBr1qxh1apVBfsO8nHmmWdyxBFHMHLkSDp06FDa93r16m1SdujQoeyyyy4cf/zxfPzxxwB88sknrFixYpOyFe3zu60HwllSu6YbYGZmVhUWLFhA165dWb58Oddddx2vvfYar732GsXFxYwfP56HHnqILl26sGDBAl555RWaN2/OHnvsAUD9+vXp1KkTU6dOLd3SLSJKt3Rr0KABp5xyCk899RT33XcfXbt2ZcWKFfztb3+jWbNmpXvXlvj888+58sor+c///E+Kiop44IEHWLx4MQDHH388EyZMKH1YoibUrVuXiRMnMmXKFGrXrs1Pf/pTXnrpJfbZZx8gCYR32mkn6tSpw4IFC3jvvffYb7/9SusPGTKEBg0abHYv3xIlgfDDDz/M22+/TZ8+fXjhhRe47777WLNmDevXry836LSq58DPzMx2CHPmzClddmXQoEGl6cXFxTRp0oTXXnuNBx54gLp163LkkUdy4403bnJlCjZs6QbQt29fAFq3bs0pp5xC69atWbJkCVdeeSXr1q2jqKiI4cOHb3RfICRXF+fNm8fjjz/OypUr6dChA9ddd12hul6hJ598kqlTpwIb9uo9+uij2Weffbj88ss5+OCDeeONN3j22We5/PLLqV+/PrBxINyvX7/SJ4JfeOEFbrvtNs455xx+8IMfsNtuuzFw4ECGDh3KRx99VO5evrDtB8JZ4sDPzMx2CF27diV3G9KSbblKt+c6eSjN07x5wA8f/hgeTvJaD3iCL0rLNqP1gE0f2ADo1KkTzz///GbbUr9+fSZMmFBu3gsvvJBHb6rGsGHDSttRslfv6NGjadeuHRMmTOCPf/wjDRo0oF+/fvz2t78t9xxFRUUUFSW7i33xxRcAHHjggey///5A8oDHz3/+c6644goOP/xwRo8evck5tpVA2Bz4mZmZ7bBKgszcvWkHvweDf/E36HY93+iWpI0Dxl39bGmZjQPhXElQfOsyKFm98PTTT9/s/XrbSiBsBX64Q1I3STMlzZY0sJz83SSNk/SOpGmSem2urqQmkp6RNCt9372QfTAzq0qbGxfNzAqpYIGfpFrAbcCJQEegh6SOZYr1BaZHxEFAV2C4pDqbqTsQeC4i2gPPpcdmZtu8PMdFM7OCKeQVv8OA2RExNyLWAA8C3cuUCaCRkrtrGwLLgbWbqdsduDf9fC9wagH7YGZWlfIZF83MCka5N8JW6YmlHwDdIuL89PjHwOER0S+nTCNgLLA/0Aj4UUQ8WVldSZ9GROOcc6yIiE2meyX1Bnqnh/sBMwvRzyrSFFhW040osCz0EbLRz22lj60jollNN2JL5DMupumFHr+2ld/htsLfx8b8fWxQqO+ixsavQj7csekz8skVvlwnAJOBY4B2wDOSXsyzbqUi4k7gzi2pU1MkTYqIoppuRyFloY+QjX5moY8FlNfYVujxy7/Djfn72Ji/jw12xO+ikFO9C4G9co5bAYvLlOkFPBKJ2SRP2O+/mbofSWoBkL5/XIC2m5kVQj7joplZwRQy8HsDaC+praQ6QDHJtG6uD4BjASQ1J5nSmLuZumOBnunnnsDjBeyDmVlVymdcNDMrmIJN9UbEWkn9gPFALWBUREyT1CfNHwlcC9wjaQrJFMiAiFgGUF7d9NRDgIcknUcSOP6wUH2oRtvFlPRWykIfIRv9zEIfC6KicbEGmuLf4cb8fWzM38cGO9x3UbCHO8zMzMxs21LQBZzNzMzMbNvhwM/MzMwsIxz4VTNJ6yRNljQ13a6ucZreRtLKNK/kVaeGm1shSc0lPSBprqQ3Jb0q6TRJXSV9lrb/XUnPStozrXOupKWS3k633BsvqUuad1taZ3qZ7+EH21of07wjJb0u6b301Tun3n6SXkjbP0PSnWn6LpLulzQl/f2/JKlhTfUvp70l/01OS7dPvFzSTmXKPC7p1TJpgyUtyvm99ZDUK+d3tybt62RJQ6q3V1ZiRxlz8pWFsWlLZGUcy5fHOyAi/KrGF/BFzud7gV+mn9sAU2u6fXn2QcCrQJ+ctNbAJSRb7z2Rk34DcE36+Vzg1py87wIfAh1y0raJ72EzffwGyYNFh6TpTYE3ge+nx+OB7jn1DkzfBwE35aTvB9TdBvqa+9/knsCzJb+zNK0xsACYAbTNSR8MXJF+bg/8G9g5J38+0LSm+5f1144w5mxBX3f4sakKv48dahzbgu8k8+Odr/jVrFeBljXdiK/hGGBNJE9mAxAR/4qIW3ILSRLJjiwryjtJRDxP8sRU7/Lya1hlfewL3BMRb6Xpy4Ar2bBvdAuS9dpK6k3JSV+Ukz4zIlYXtBdbKCI+Jvl99Et/fwBnAONIthcrrqDeLOArYJNddGybsr2OOfnKwti0JTI5juUrq+OdA78aomSz9mPZeA2vdjmXjW+roabloxPwViX5R0maTPLX5PeAUZWUfYtk0e5tTWV97ETyl3GuSWk6wM3APyT9TdJlJVNrJN/DgHSq5TpJ7au60VUhIuaSjA17pkk9gDHpq0d5dSQdAsxKB1LbBm3nY06+sjA2bYnMjmP5yuJ458Cv+tVPB55PgCbAMzl5cyKic/rqWyOt+xrSe2DekfRGmvRi2oe9gNHAjZVVL3wLt16ZPorytxAMgIgYDXQA/pdkeumfkupGxGRgH2AYye/+DUkdqqH5X4egdGH1fYGXIuJ9YK2kA3LKXSZpJvAayVSIbXt2uDEnX1kYm7ZEBsexfGVqvHPgV/1WRkRnkvss6pBcbt/eTAMOKTlI/8E4Fihvw+mxwHcqOdfBJPdSbGsq6+M0oOzejYcC03PKL46IURHRHVgLHJCmfxERj0TExcB9wEkF7cXXIGkfYB3Jdog/IpnOmCdpPsl9TrnTHzdHxH5puT9Lqle9rbU87AhjTr6yMDZticyOY/nK4njnwK+GRMRnQH/gCkk713R7ttA/gHqSLspJ26WCskcCc8rLkHQ0yf0Vd1Vt86pEZX28DThXUmcASXsAQ0mvHkjqVvI7lfQNYA9gkaQjJO2eptcBOgL/qoa+5E1SM2AkyY3uQTLV0S0i2kREG5J/GDa57yUiHiGZJupZNs+2Ddv5mJOvLIxNWyKT41i+sjreFWzLNtu8iHhb0jsk/2G9WNPtyVdEhKRTgZslXQksBb4EBqRFSu6jEfAZcH5O9R9JOpJk8JkHnBER29xf1ZX1MSKWSDobuEtSI5J+/j4ixqXVjwf+IGlVevzziPhQ0vHAHelNxDsBTwIPV2O3KlIyFbgzyV/1/wPcJKkNsDfwz5KCETFP0r8lHV7OeX4DPCDprohYX/hm25baXsecfGVhbNoSGRvH8pX58c5btpmZmZllhKd6zczMzDLCgZ+ZmZlZRjjwMzMzM8sIB35mZmZmGeHAz8zMzCwjHPiZmZmZZYQDPzMzM7OM+P/c/wMIDBslFAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 648x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(9, 4))\n",
    "\n",
    "scores = np.array(list(cv_scores.values()))\n",
    "ax[0].bar(np.arange(1, 5)-0.2, scores, width=0.3)\n",
    "scores = np.array(list(tst_scores.values()))\n",
    "ax[0].bar(np.arange(1, 5)+0.2, scores, width=0.3)\n",
    "ax[0].axis([0.3, 4.6, 0.8, 0.83])\n",
    "ax[0].set_ylabel('Accuracy (%)')\n",
    "ax[0].set_xticks([1, 2, 3, 4]);\n",
    "ax[0].set_xticklabels(['RF', 'GBDT', 'GOSS', 'DART']);\n",
    "\n",
    "times = np.array(list(run_time.values()))\n",
    "times_scaled = times / np.min(times)\n",
    "ax[1].bar(np.arange(1, 5), times, width=0.3)\n",
    "for i, v in enumerate(times_scaled):\n",
    "    ax[1].text(i + 0.8, times[i] + 3.5, '{0:3.2f}%\\n{1:4.2f} s.'.format(v, times[i]), fontweight='bold')\n",
    "ax[1].set_ylabel('Run Time (%)')\n",
    "ax[1].set_xticks([1, 2, 3, 4]);\n",
    "ax[1].set_xticklabels(['RF', 'GBDT', 'GOSS', 'DART']);\n",
    "\n",
    "fig.tight_layout()\n",
    "pngFile = './figures/CH05_F22_Kunapuli.png'\n",
    "plt.savefig(pngFile, dpi=300, pad_inches=0)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}