03_CrossValidation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CrossValidation\n",
    "[公式ドキュメント](http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "下ごしらえ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from ipywidgets import interact\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris=load_iris()\n",
    "data=iris.data\n",
    "target=iris.target\n",
    "from sklearn.utils import shuffle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "clf=SVC(probability=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 今回のお題"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score, train_test_split, StratifiedKFold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- モデルのインスタンス、記述子、ラベル、分割数、ジョブ数(-1でコア数分並列)を指定してあげると一行でクロスバリデーションできる\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "score=cross_val_score(clf,data,target,cv=5,n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- scoreのデフォルトは正答率（accurary）\n",
    "\n",
    "- 他にもあるけどマルチクラス分類では使えないのが多い。\n",
    "\n",
    "- [参照](http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all:  [0.96666667 1.         0.96666667 0.96666667 1.        ]\n",
      "Accuracy: 0.9800 (+/- 0.0163)\n"
     ]
    }
   ],
   "source": [
    "print(\"all: \",score)\n",
    "print(\"Accuracy: {0:04.4f} (+/- {1:04.4f})\".format(score.mean(),score.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "単純に入力順にn分割している様子  \n",
    "入力をシャッフルすればOK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 1.        , 0.93333333, 0.96666667, 1.        ])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_shuffle,target_shuffle=shuffle(data,target)\n",
    "cross_val_score(clf,data_shuffle,target_shuffle,n_jobs=-1,cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### SVMのコストパラメータを探索する場合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- まずざっくりとlogscaleでコストを振ってみる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "C_params=np.logspace(-2, 2, 9)\n",
    "scores=list()\n",
    "for C in C_params:\n",
    "    clf.C=C\n",
    "    tmp_score=cv.cross_val_score(clf,data_shuffle,target_shuffle,cv=5,n_jobs=-1)\n",
    "    scores.append(np.mean(tmp_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "コスト探索の可視化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x115fa0ba8>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEACAYAAABRQBpkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEv9JREFUeJzt3WFsHGd+3/HvL7J1URunOlisG5M6S8kpxhGwYSWs0sa4yjF6sYwWtiwFjd00SIogRpA472xAgvtKhaFL7QLFoUYLBRV8enOGcXBVpc1BLhTrkqKXQqvoZEcW6KoCUpE8JEwBKbmcWlvGvy84vKzXlLkkl9xd8vsBCM0888zsfx4u96eZ5fJJVSFJ2th+oN8FSJL6zzCQJBkGkiTDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRJwR78L6LRt27basWNHv8uQpKFy/vz5P6+qkeXuP3BhsGPHDlqtVr/LkKShkuRPVrK/t4kkSYaBJMkwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEl2GQZJ9SSaTXElyaIHt9yU5k+SdJGeTjLVt+60kf9x8/Xwvi5ck9caiYZBkE/Aq8DgwDjyTZLyj2yvAiap6EDgCHG32/UfATwAPAT8FvJDkh3tXviSpF7q5MtgDXKmqq1X1AfA68GRHn3HgTLP8dtv2ceCbVXWrqv4KuAjsW3nZkqRe6iYMRoFrbetTTVu7i8DBZvkp4K4kdzftjyf5G0m2AT8DbF9ZyZKkXusmDLJAW3WsPw/sTXIB2AtMA7eq6i3gd4H/DnwN+BZw6xMPkDybpJWkNTs7u5T6JUk90E0YTPHx/82PATPtHapqpqoOVNVu4MWm7Ubz70tV9VBVfYm5YPmfnQ9QVceqaqKqJkZGRpZ5KpKk5eomDM4Bu5LsTLIZeBo41d4hybYk88c6DBxv2jc1t4tI8iDwIPBWr4qXJPXGHYt1qKpbSZ4DTgObgONVdSnJEaBVVaeAR4CjSQr4feA3mt3vBP4gCcBfAP+sqj5xm0iS1F+p6rz9318TExPVarX6XYYkDZUk56tqYrn7+wlkSZJhIEkyDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkugyDJPuSTCa5kuTQAtvvS3ImyTtJziYZa9v2r5JcSnI5yVeSpJcnIElauTsW65BkE/Aq8CVgCjiX5FRVvdfW7RXgRFV9NcmjwFHgF5P8NPAw8GDT778Be4GzvTsFafWdvDDNy6cnmbl+k3u3buGFx+5n/+7Rfpf1CcNSpwZPN1cGe4ArVXW1qj4AXgee7OgzDpxplt9u217ADwKbgc8AdwJ/utKipbV08sI0h998l+nrNylg+vpNDr/5LicvTPe7tI8Zljo1mLoJg1HgWtv6VNPW7iJwsFl+Crgryd1V9S3mwuE7zdfpqrq8spKltfXy6UlufvjRx9pufvgRL5+e7FNFCxuWOjWYugmDhe7xV8f688DeJBeYuw00DdxK8nngC8AYcwHyaJJ/8IkHSJ5N0krSmp2dXdIJSKtt5vrNJbX3y7DUqcHUTRhMAdvb1seAmfYOVTVTVQeqajfwYtN2g7mrhD+squ9W1XeBbwB/r/MBqupYVU1U1cTIyMgyT0VaHfdu3bKk9n4Zljo1mLoJg3PAriQ7k2wGngZOtXdIsi3J/LEOA8eb5f/N3BXDHUnuZO6qwdtEGiovPHY/W+7c9LG2LXdu4oXH7u9TRQsbljo1mBYNg6q6BTwHnGbuhfyNqrqU5EiSJ5pujwCTSd4H7gFeatq/Dvwv4F3m3le4WFW/09tTkFbX/t2jHD3wAKNbtxBgdOsWjh54YOB+S2dY6tRgSlXn7f/+mpiYqFar1e8yJGmoJDlfVRPL3d9PIEuSDANJkmEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEmii2kvJWkj2mhTiBoGktRhfgrR+Znj5qcQBdZtIHibSJI6bMQpRA0DSeqwEacQNQwkqcNGnELUMJCkDhtxClHfQJakDvNvEvvbRJK0we3fPbquX/w7eZtIktRdGCTZl2QyyZUkhxbYfl+SM0neSXI2yVjT/jNJvt329X+T7O/1SUiSVmbRMEiyCXgVeBwYB55JMt7R7RXgRFU9CBwBjgJU1dtV9VBVPQQ8CnwPeKuH9UuSeqCbK4M9wJWqulpVHwCvA0929BkHzjTLby+wHeDngG9U1feWW6wkaXV0EwajwLW29ammrd1F4GCz/BRwV5K7O/o8DXxtOUVKklZXN2GQBdqqY/15YG+SC8BeYBq49f0DJD8CPACcXvABkmeTtJK0ZmdnuypcktQ73YTBFLC9bX0MmGnvUFUzVXWgqnYDLzZtN9q6/BPgP1bVhws9QFUdq6qJqpoYGRlZ0glIklaumzA4B+xKsjPJZuZu95xq75BkW5L5Yx0Gjncc4xm8RSRJA2vRMKiqW8BzzN3iuQy8UVWXkhxJ8kTT7RFgMsn7wD3AS/P7J9nB3JXFN3tauSSpZ1LVefu/vyYmJqrVavW7DEkaKknOV9XEcvf3E8iSJMNAkuQfqlOfbbR5ZjXH7/vgMQzUNxtxnln5fR9U3iZS32zEeWbl931QGQbqm404z6z8vg8qw0B9sxHnmZXf90FlGKhvNuI8s/L7Pqh8A1l9sxHnmZXf90HlJ5AlaR3wE8iSpBUzDCRJhoEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CSRJdhkGRfkskkV5IcWmD7fUnOJHknydkkY23bPpfkrSSXk7yXZEfvypck9cKiYZBkE/Aq8DgwDjyTZLyj2yvAiap6EDgCHG3bdgJ4uaq+AOwB/qwXhUuSeqebK4M9wJWqulpVHwCvA0929BkHzjTLb89vb0Ljjqr6rwBV9d2q+l5PKpck9Uw3YTAKXGtbn2ra2l0EDjbLTwF3Jbkb+HHgepI3k1xI8nJzpSFJGiDdhEEWaOv8u9fPA3uTXAD2AtPALebmS/his/3vAj8K/PInHiB5NkkrSWt2drb76iVJPdFNGEwB29vWx4CZ9g5VNVNVB6pqN/Bi03aj2fdCc4vpFnAS+InOB6iqY1U1UVUTIyMjyzwVSdJydRMG54BdSXYm2Qw8DZxq75BkW5L5Yx0Gjrft+9kk86/wjwLvrbxsSVIvLRoGzf/onwNOA5eBN6rqUpIjSZ5ouj0CTCZ5H7gHeKnZ9yPmbhGdSfIuc7ecfrvnZyFJWhGnvZSkdcBpLyVJK2YYSJIMA0mSYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgSWJu8hlJ0pA6eWGal09PsvnvfP4nV3Icw0CShtTJC9McfvNdbn740YqP5W0iSRpSL5+e7EkQgGEgSUNr5vrNnh3LMJCkIXXv1i09O5ZhIElD6oXH7mfLnZt6cizfQJakIbV/9ygw997Bd1Z4LOdAlqR1wDmQJUkr1lUYJNmXZDLJlSSHFth+X5IzSd5JcjbJWNu2j5J8u/k61cviJUm9seh7Bkk2Aa8CXwKmgHNJTlXVe23dXgFOVNVXkzwKHAV+sdl2s6oe6nHdkqQe6ubKYA9wpaquVtUHwOvAkx19xoEzzfLbC2yXJA2wbsJgFLjWtj7VtLW7CBxslp8C7kpyd7P+g0laSf4wyf4VVStJWhXdhEEWaOv8FaTngb1JLgB7gWngVrPtc8073P8U+DdJfuwTD5A82wRGa3Z2tvvqJUk90U0YTAHb29bHgJn2DlU1U1UHqmo38GLTdmN+W/PvVeAssLvzAarqWFVNVNXEyMjIcs5DkrQC3YTBOWBXkp1JNgNPAx/7raAk25LMH+swcLxp/2ySz8z3AR4G2t94liQNgEXDoKpuAc8Bp4HLwBtVdSnJkSRPNN0eASaTvA/cA7zUtH8BaCW5yNwby1/u+C0kSdIA8BPIkrQO+AlkSdKK+Yfq1qn5qfBmrt/k3q1beOGx+7//R60kqZNhsA51ToU3ff0mh998F8BAkLQgbxOtQwtNhXfzw494+fRknyqSNOgMg3XodlPh9XKKPEnri2GwDt1uKrxeTpEnaX0xDNahhabC23LnJl547P4+VSRp0PkG8jrUPhWev00kqRuGwTq1f/eoL/6SuuZtIkmSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEl0GQZJ9iWZTHIlyaEFtt+X5EySd5KcTTLWsf2Hk0wn+be9KlyS1DuLhkGSTcCrwOPAOPBMkvGObq8AJ6rqQeAIcLRj+78EvtlNQe9O3+DhL/8eJy9Md9N9zZ28MM3DX/49dh76LwNdpyQtRTdXBnuAK1V1tao+AF4HnuzoMw6caZbfbt+e5CeBe4C3ui1qfs7eQXuhnZ9bePr6TYrBrVOSlqqbMBgFrrWtTzVt7S4CB5vlp4C7ktyd5AeAfw28sNTCBnHOXucWlrRedRMGWaCtOtafB/YmuQDsBaaBW8CvA79bVdf4FEmeTdJK0vroeze+3z5oc/Y6t7Ck9aqbyW2mgO1t62PATHuHqpoBDgAk+SHgYFXdSPL3gS8m+XXgh4DNSb5bVYc69j8GHAP4zI/s+n7QDNqcvfdu3cL0Ai/8g1anJC1VN1cG54BdSXYm2Qw8DZxq75BkW3NLCOAwcBygqn6hqj5XVTuYu3o40RkEtzOIc/Y6t7Ck9WrRMKiqW8BzwGngMvBGVV1KciTJE023R4DJJO8z92bxSyspanTrFo4eeGDgpm3cv3uUowceYHTrFsLg1ilJS5Wqztv//TUxMVGtVqvfZUjSUElyvqomlru/n0CWJBkGkiTDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEmiyzBIsi/JZJIrSQ4tsP2+JGeSvJPkbJKxtvbzSb6d5FKSX+v1CUiSVm7RMEiyCXgVeBwYB55JMt7R7RXgRFU9CBwBjjbt3wF+uqoeAn4KOJTk3l4VL0nqjW6uDPYAV6rqalV9ALwOPNnRZxw40yy/Pb+9qj6oqv/XtH+my8eTJK2xbl6cR4FrbetTTVu7i8DBZvkp4K4kdwMk2Z7kneYYv1VVMysrWZLUa92EQRZoq47154G9SS4Ae4Fp4BZAVV1rbh99HvilJPd84gGSZ5O0krRmZ2eXdAKSpJXrJgymgO1t62PAx/53X1UzVXWgqnYDLzZtNzr7AJeAL3Y+QFUdq6qJqpoYGRlZ4ilIklaqmzA4B+xKsjPJZuBp4FR7hyTbkswf6zBwvGkfS7KlWf4s8DAw2aviJUm9sWgYVNUt4DngNHAZeKOqLiU5kuSJptsjwGSS94F7gJea9i8A/yPJReCbwCtV9W6Pz0GStEKp6rz9318TExPVarX6XYYkDZUk56tqYrn7+6uekiTDQJJkGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJLEAE5uk+QvWZupMf8WcGPRXivbb7G+n7Z9oW3dtHWubwP+fNFKV2a5Y7nUfZc7nktp3yjj2evn5kLt3YzvII/nsP2s319Vdy1e6m1U1UB9Aa01epxjq73fYn0/bftC27ppW2B91cdzuWO5VuO5lPaNMp69fm52M3bDNp4b7Wd9I98m+p012G+xvp+2faFt3bQt97xWYiWPuRbjuZT2jTKevX5uLtTe7XN4tfmz3oVBvE3UqhXM46mPczx7y/HsLcezd1Y6loN4ZXCs3wWsM45nbzmeveV49s6KxnLgrgwkSWtvEK8MJElrzDCQJBkGkqQhC4Mk+5P8dpL/lORn+13PsEvyo0n+Q5Kv97uWYZTkbyb5avOc/IV+1zPsfD721lJfL9csDJIcT/JnSf64o31fkskkV5Ic+rRjVNXJqvpV4JeBn1/Fcgdej8bzalX9yupWOlyWOK4HgK83z8kn1rzYIbCU8fT5uLgljueSXi/X8srgNWBfe0OSTcCrwOPAOPBMkvEkDyT5zx1ff7tt13/R7LeRvUbvxlN/7TW6HFdgDLjWdPtoDWscJq/R/Xhqca+x9PHs6vXyjt7V+Omq6veT7Oho3gNcqaqrAEleB56sqqPAP+48RpIAXwa+UVV/tLoVD7ZejKc+aSnjCkwxFwjfZshuua6VJY7ne2tb3fBZyngmucwSXi/7/QQe5a//ZwVzP1yjn9L/N4F/CPxckl9bzcKG1JLGM8ndSf49sDvJ4dUubojdblzfBA4m+Xf0588sDKsFx9Pn47Ld7vm5pNfLNbsyuI0s0HbbT8FV1VeAr6xeOUNvqeP5fwBDdXELjmtV/RXwz9e6mHXgduPp83F5bjeeS3q97PeVwRSwvW19DJjpUy3rgeO5OhzX3nI8e6sn49nvMDgH7EqyM8lm4GngVJ9rGmaO5+pwXHvL8eytnoznWv5q6deAbwH3J5lK8itVdQt4DjgNXAbeqKpLa1XTMHM8V4fj2luOZ2+t5nj6h+okSX2/TSRJGgCGgSTJMJAkGQaSJAwDSRKGgSQJw0CShGEgScIwkCQB/x+LceEdWp9sEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_xscale(\"log\")\n",
    "ax.set_xlim(0.01,100)\n",
    "ax.scatter(C_params,scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 桁を抑えたところで0.01-10まで探索してみる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "C_params2=np.linspace(0.001,10,21)\n",
    "scores2=list()\n",
    "for C in C_params2:\n",
    "    clf.C=C\n",
    "    tmp_score=cross_val_score(clf,data_shuffle,target_shuffle,cv=5,n_jobs=-1)\n",
    "    scores2.append(np.mean(tmp_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1a1a6f57f0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAE6RJREFUeJzt3XGMXWV+3vHvExtnvYGVt9hFwePFJKFkrULxdko2izYmpAijVBjsJoU2aFtVQlWWNo0KlV3+qOQKOSm0SquiSjR1C2q0CG0odatNzMqB3VZNVh7Ha1jjDnHdZj0zNDtRZdLNugWTX/+Y43AZT3bu2Hfmzsz7/Ugjn/Oe95z3d2TrmXPfe6/fVBWSpDZ8z7ALkCQtHUNfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1JC1wy5gto0bN9bWrVuHXYYkrSjHjh37/araNF+/ZRf6W7duZWxsbNhlSNKKkuR3++nn9I4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSF9hX6SnUnGk5xOsneO4zckOZLk9SSvJRnpOfaLSb7R/fyVQRYvSVqYeUM/yRrgGeBeYBvwUJJts7o9DTxfVbcC+4ED3bk/CXwKuA34EeDxJB8bXPmSpIXo50n/duB0VZ2pqneBF4Bds/psA45026/2HN8GfKWqLlTVHwIngJ1XXrYk6XL0E/qbgbM9+xNdW68TwJ5u+wHgmiTXdu33Jvloko3AjwNbZg+Q5JEkY0nGpqenF3oPkqQ+9RP6maOtZu0/BuxIchzYAUwCF6rqFeBLwH8FvgD8JnDhkotVPVtVo1U1umnTvIu5S5IuUz+hP8GHn85HgKneDlU1VVW7q2o78ETX9k7355NVdVtV3c3ML5DfGUjlkqQF6yf0jwI3JbkxyTrgQeBQb4ckG5NcvNY+4GDXvqab5iHJrcCtwCuDKl6StDBr5+tQVReSPAocBtYAB6vqZJL9wFhVHQLuBA4kKeCrwOe7068C/nMSgD8AfqaqLpnekSQtjVTNnp4frtHR0RobGxt2GZK0oiQ5VlWj8/XzG7mS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWpIX6GfZGeS8SSnk+yd4/gNSY4keT3Ja0lGeo794yQnk5xK8s+TZJA3IEnq39r5OiRZAzwD3A1MAEeTHKqqN3u6PQ08X1XPJbkLOAA8nOQzwB3ArV2//wLsAF4b3C0M18vHJ3nq8DhT585z/Yb1PH7Pzdy/ffOyH7u1cyXNmDf0gduB01V1BiDJC8AuoDf0twE/322/CrzcbRfwEWAdEOAq4PeuvOzl4eXjk+x76Q3Ov/c+AJPnzrPvpTcAFj2MrmTs1s6V9IF+pnc2A2d79ie6tl4ngD3d9gPANUmurarfZOaXwNvdz+GqOnVlJS8fTx0e/+MQuuj8e+/z1OHxZT12a+dK+kA/oT/XHHzN2n8M2JHkODPTN5PAhSQ/BHwSGGHmF8VdSX7skgGSR5KMJRmbnp5e0A0M09S58wtqXy5jt3aupA/0E/oTwJae/RFgqrdDVU1V1e6q2g480bW9w8xT/29V1ber6tvArwGfnj1AVT1bVaNVNbpp06bLvJWld/2G9QtqXy5jt3aupA/0E/pHgZuS3JhkHfAgcKi3Q5KNSS5eax9wsNv+JjOvANYmuYqZVwGrZnrn8XtuZv1Vaz7Utv6qNTx+z83LeuzWzpX0gXnfyK2qC0keBQ4Da4CDVXUyyX5grKoOAXcCB5IU8FXg893pXwTuAt5gZkro16vqPw7+Nobj4huIw/hEyZWM3dq5kj6QqtnT88M1OjpaY2Njwy5DklaUJMeqanS+fn4jV5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1Jakg/K2dpEbkE4NJZics8DnN5SZe2XJ0M/SFyCcClsxKXeRzm8pIubbl6Ob0zRC4BuHRW4jKPw1xe0qUtVy9Df4hcAnDprMRlHoe5vKRLW65ehv4QuQTg0lmJyzwOc3lJl7ZcvQz9IXIJwKWzEpd5HObyki5tuXr5Ru4QuQTg0lmJyzwOc3lJl7ZcvVwuUZJWAZdLlCRdoq/QT7IzyXiS00n2znH8hiRHkrye5LUkI137jyf5es/P/01y/6BvQpLUn3lDP8ka4BngXmAb8FCSbbO6PQ08X1W3AvuBAwBV9WpV3VZVtwF3Ad8BXhlg/ZKkBejnSf924HRVnamqd4EXgF2z+mwDjnTbr85xHOAvA79WVd+53GIlSVemn9DfDJzt2Z/o2nqdAPZ02w8A1yS5dlafB4EvXE6RkqTB6Cf0M0fb7I/8PAbsSHIc2AFMAhf++ALJ9wO3AIfnHCB5JMlYkrHp6em+CpckLVw/oT8BbOnZHwGmejtU1VRV7a6q7cATXds7PV1+Gvj3VfXeXANU1bNVNVpVo5s2bVrQDUiS+tdP6B8FbkpyY5J1zEzTHOrtkGRjkovX2gccnHWNh3BqR5KGbt7Qr6oLwKPMTM2cAl6sqpNJ9ie5r+t2JzCe5C3gOuDJi+cn2crMK4WvDLRySdKC+Y1cSVoF/EauJOkShr4kNcT/ZVPSsuKawov7v5Ia+pKWDdcUXvw1hZ3ekbRsuKbwwsdeKENf0rLhmsILH3uhDH1Jy4ZrCi987IUy9CUtG64pvPCxF8o3ciUtG64pvPif3vEbuZK0CviNXEnSJQx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqSF+hn2RnkvEkp5PsneP4DUmOJHk9yWtJRnqOfSLJK0lOJXkzydbBlS9JWoh5/8O1JGuAZ4C7gQngaJJDVfVmT7engeer6rkkdwEHgIe7Y88DT1bVl5NcDfzRQO9gAJZyqTJJGqZ+nvRvB05X1Zmqehd4Adg1q8824Ei3/erF40m2AWur6ssAVfXtqvrOQCofkItLlU2eO0/xwVJlLx+fHHZpkjRw/YT+ZuBsz/5E19brBLCn234AuCbJtcCfAc4leSnJ8SRPda8clo2lXqpMkoapn9DPHG2z/z/mx4AdSY4DO4BJ4AIz00ef7Y7/BeAHgL9+yQDJI0nGkoxNT0/3X/0ALPVSZZI0TP2E/gSwpWd/BJjq7VBVU1W1u6q2A090be905x7vpoYuAC8Dn5o9QFU9W1WjVTW6adOmy7yVy7PUS5VJ0jD1E/pHgZuS3JhkHfAgcKi3Q5KNSS5eax9wsOfcjye5mOR3Ab1vAA/dUi9VJknDNG/od0/ojwKHgVPAi1V1Msn+JPd13e4ExpO8BVwHPNmd+z4zUztHkrzBzFTRvxr4XVyB+7dv5sDuW9i8YT0BNm9Yz4Hdt/jpHUmrksslStIq4HKJkqRLGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWpIX6GfZGeS8SSnk+yd4/gNSY4keT3Ja0lGeo69n+Tr3c+hQRYvSVqYtfN1SLIGeAa4G5gAjiY5VFVv9nR7Gni+qp5LchdwAHi4O3a+qm4bcN2SpMvQz5P+7cDpqjpTVe8CLwC7ZvXZBhzptl+d47gkaRnoJ/Q3A2d79ie6tl4ngD3d9gPANUmu7fY/kmQsyW8luX+uAZI80vUZm56eXkD5kqSF6Cf0M0dbzdp/DNiR5DiwA5gELnTHPlFVo8BfBX4pyQ9ecrGqZ6tqtKpGN23a1H/1kqQFmXdOn5kn+y09+yPAVG+HqpoCdgMkuRrYU1Xv9Byjqs4keQ3YDvz3K65ckrRg/TzpHwVuSnJjknXAg8CHPoWTZGOSi9faBxzs2j+e5Hsv9gHuAHrfAJYkLaF5Q7+qLgCPAoeBU8CLVXUyyf4k93Xd7gTGk7wFXAc82bV/EhhLcoKZN3h/YdanfiRJSyhVs6fnh2t0dLTGxsaGXYYkrShJjnXvn35XfiNXkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kN6Sv0k+xMMp7kdJK9cxy/IcmRJK8neS3JyKzjH0symeRfDKpwSdLCrZ2vQ5I1wDPA3cAEcDTJoap6s6fb08DzVfVckruAA8DDPcf/EfCVwZV9qZePT/LU4XGmzp3n+g3refyem7l/++bFHFKSVpx+nvRvB05X1Zmqehd4Adg1q8824Ei3/Wrv8SR/HrgOeOXKy53by8cn2ffSG0yeO08Bk+fOs++lN3j5+ORiDSlJK1I/ob8ZONuzP9G19ToB7Om2HwCuSXJtku8B/gnw+JUW+t08dXic8++9/6G28++9z1OHxxdzWElacfoJ/czRVrP2HwN2JDkO7AAmgQvAzwJfqqqzfBdJHkkylmRsenq6j5I+bOrc+QW1S1Kr5p3TZ+bJfkvP/ggw1duhqqaA3QBJrgb2VNU7SX4U+GySnwWuBtYl+XZV7Z11/rPAswCjo6Ozf6HM6/oN65mcI+Cv37B+oZeSpFWtnyf9o8BNSW5Msg54EDjU2yHJxm4qB2AfcBCgqv5aVX2iqrYy82rg+dmBPwiP33Mz669a86G29Vet4fF7bh70UJK0os0b+lV1AXgUOAycAl6sqpNJ9ie5r+t2JzCe5C1m3rR9cpHqndP92zdzYPctbN6wngCbN6znwO5b/PSOJM2SqgXPpiyq0dHRGhsbG3YZkrSiJDlWVaPz9fMbuZLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JakhfoZ9kZ5LxJKeT7J3j+A1JjiR5PclrSUZ62o8l+XqSk0n+1qBvQJLUv3lDP8ka4BngXmAb8FCSbbO6PQ08X1W3AvuBA13728Bnquo24EeAvUmuH1TxkqSF6edJ/3bgdFWdqap3gReAXbP6bAOOdNuvXjxeVe9W1f/r2r+3z/EkSYuknxDeDJzt2Z/o2nqdAPZ02w8A1yS5FiDJliSvd9f4xaqaurKSJUmXq5/QzxxtNWv/MWBHkuPADmASuABQVWe7aZ8fAj6X5LpLBkgeSTKWZGx6enpBNyBJ6l8/oT8BbOnZHwE+9LReVVNVtbuqtgNPdG3vzO4DnAQ+O3uAqnq2qkaranTTpk0LvAVJUr/6Cf2jwE1JbkyyDngQONTbIcnGJBevtQ842LWPJFnfbX8cuAMYH1TxkqSFmTf0q+oC8ChwGDgFvFhVJ5PsT3Jf1+1OYDzJW8B1wJNd+yeBryU5AXwFeLqq3hjwPUiS+pSq2dPzwzU6OlpjY2PDLkOSVpQkx6pqdL5+foRSkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktSQZbeISpJp4Hev4BIbgd8fUDkrRWv33Nr9gvfciiu55xuqat5Fxpdd6F+pJGP9rB6zmrR2z63dL3jPrViKe3Z6R5IaYuhLUkNWY+g/O+wChqC1e27tfsF7bsWi3/Oqm9OXJP3JVuOTviTpT7BqQj/JziTjSU4n2TvsehZbki1JXk1yKsnJJD837JqWSpI1SY4n+U/DrmUpJNmQ5ItJ/lv39/2jw65pMSX5+e7f9DeSfCHJR4Zd02JIcjDJt5J8o6ftTyX5cpLf6f78+KDHXRWhn2QN8AxwL7ANeCjJtuFWteguAH+vqj4JfBr4fAP3fNHPAaeGXcQS+mfAr1fVDwN/jlV870k2A38HGK2qPwusAR4cblWL5t8CO2e17QWOVNVNwJFuf6BWRegDtwOnq+pMVb0LvADsGnJNi6qq3q6q3+62/w8zQbB5uFUtviQjwE8CvzzsWpZCko8BPwb8a4Cqereqzg23qkW3FlifZC3wUWBqyPUsiqr6KvC/ZzXvAp7rtp8D7h/0uKsl9DcDZ3v2J2ggAC9KshXYDnxtuJUsiV8C/j7wR8MuZIn8ADAN/JtuSuuXk3zfsItaLFU1CTwNfBN4G3inql4ZblVL6rqqehtmHuyAPz3oAVZL6GeOtiY+lpTkauBXgb9bVX8w7HoWU5K/BHyrqo4Nu5YltBb4FPAvq2o78Icswkv+5aKbw94F3AhcD3xfkp8ZblWry2oJ/QlgS8/+CKv0JWGvJFcxE/i/UlUvDbueJXAHcF+S/8nMFN5dSf7dcEtadBPARFVdfBX3RWZ+CaxWfxH4H1U1XVXvAS8BnxlyTUvp95J8P0D357cGPcBqCf2jwE1Jbkyyjpk3fg4NuaZFlSTMzPOeqqp/Oux6lkJV7auqkarayszf8W9U1ap+Cqyq/wWcTXJz1/QTwJtDLGmxfRP4dJKPdv/Gf4JV/Mb1HA4Bn+u2Pwf8h0EPsHbQFxyGqrqQ5FHgMDPv9h+sqpNDLmux3QE8DLyR5Otd2z+oqi8NsSYtjr8N/Er3QHMG+BtDrmfRVNXXknwR+G1mPqF2nFX6zdwkXwDuBDYmmQD+IfALwItJ/iYzvwB/auDj+o1cSWrHapnekST1wdCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakh/x8J95+/UvNRFgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(C_params2,scores2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 細かく探索して終了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1a1a664390>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFoZJREFUeJzt3X+MXeWd3/H3p7bJOgopCZ5GBBNMu14St3FMciFot4kJVRaTVMH86C5sfjWKam0Lq1QpFriojeTUYlnYktKSRDRxHKQUNk1Z4q3CmsiFkCqQZVxj88MycUEbxkaboQQ2BDZg8u0f95nsZRg8d354xjN+v6Sruef5ceZ57Kv7uec5555JVSFJ0t+Z7QFIko4MBoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDULZ3sAE7FkyZJatmzZbA9DkuaUHTt2PFVVA+O1m1OBsGzZMgYHB2d7GJI0pyT5y37auWQkSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCegzEJJsTvKTJA+9Rn2S3JBkX5LdSd7dU/fJJD9qj0/2lL8nyYOtzw1JMvXpSJImq98jhC3AmkPUnwssb491wJcAkrwZ+BzwXuAM4HNJ3tT6fKm1Hel3qP1Lkg6zvgKhqu4Bnj5Ek/OAm6vrPuC4JCcA5wDfraqnq+qnwHeBNa3ujVV1b1UVcDOwdkozkSRNyXSdQzgReKJne6iVHap8aIxySdIsma5AGGv9vyZR/uodJ+uSDCYZHB4ensIQJUmHMl2BMASc1LO9FDgwTvnSMcpfpapuqqpOVXUGBsa9WZ8kaZKmKxC2Ap9oVxudCTxbVU8C24DfTvKmdjL5t4Ftre5nSc5sVxd9Avj2NI1FkjQJfd3+OsktwFnAkiRDdK8cWgRQVV8GvgN8CNgHPA98qtU9neTzwP1tVxurauTk9L+ke/XSYuCO9pAkzZJ0L/KZGzqdTvn3ECRpYpLsqKrOeO38prIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElS01cgJFmTZG+SfUmuHKP+5CTbk+xOcneSpT111yR5qD1+t6d8S5LHkzzQHqumZ0qSpMkYNxCSLABuBM4FVgCXJFkxqtl1wM1VtRLYCFzd+n4YeDewCngvsD7JG3v6ra+qVe3xwJRnI0matH6OEM4A9lXVY1X1InArcN6oNiuA7e35XT31K4DvVdXBqvo5sAtYM/VhS5KmWz+BcCLwRM/2UCvrtQu4sD0/Hzg2yfGt/Nwkr0+yBPgAcFJPv01tmen6JK+b1AwkSdOin0DIGGU1avtyYHWSncBqYD9wsKruBL4D/AC4BbgXONj6bADeDpwOvBm4YsxfnqxLMphkcHh4uI/hSpImo59AGOKVn+qXAgd6G1TVgaq6oKpOA65qZc+2n5vaOYIP0g2XH7XyJ6vrF8DX6C5NvUpV3VRVnarqDAwMTHB6kqR+9RMI9wPLk5yS5BjgYmBrb4MkS5KM7GsDsLmVL2hLRyRZCawE7mzbJ7SfAdYCD019OpKkyVo4XoOqOpjkMmAbsADYXFUPJ9kIDFbVVuAs4OokBdwDXNq6LwK+333P56+Bj1XVyJLRN5IM0D1qeAD4/embliRpolI1+nTAkavT6dTg4OBsD0OS5pQkO6qqM147v6ksSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1PQVCEnWJNmbZF+SK8eoPznJ9iS7k9ydZGlP3TVJHmqP3+0pPyXJD5P8KMmfJDlmeqYkSZqMcQMhyQLgRuBcYAVwSZIVo5pdB9xcVSuBjcDVre+HgXcDq4D3AuuTvLH1uQa4vqqWAz8FPj316UiSJqufI4QzgH1V9VhVvQjcCpw3qs0KYHt7fldP/Qrge1V1sKp+DuwC1iQJcDbwrdbu68DayU9DkjRV/QTCicATPdtDrazXLuDC9vx84Ngkx7fyc5O8PskS4APAScDxwDNVdfAQ+5QkzaB+AiFjlNWo7cuB1Ul2AquB/cDBqroT+A7wA+AW4F7gYJ/77P7yZF2SwSSDw8PDfQxXkjQZ/QTCEN1P9SOWAgd6G1TVgaq6oKpOA65qZc+2n5uqalVVfZBuEPwIeAo4LsnC19pnz75vqqpOVXUGBgYmMDVJ0kT0Ewj3A8vbVUHHABcDW3sbJFmSZGRfG4DNrXxBWzoiyUpgJXBnVRXdcw0XtT6fBL491clIkiZv3EBo6/yXAduAPcA3q+rhJBuTfKQ1OwvYm+RR4C3Apla+CPh+kkeAm4CP9Zw3uAL4bJJ9dM8pfHWa5iRJmoR0P6zPDZ1OpwYHB2d7GJI0pyTZUVWd8dr5TWVJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEwMLxm0h6Lbfv3M+12/Zy4JkXeOtxi1l/zqmsPa2/v/Vk35npq/4ZCNIk3b5zPxtue5AXXnoZgP3PvMCG2x4EGPfNyr4z01cT45KRNEnXbtv7qzepES+89DLXbttr3yOkrybGQJAm6cAzL0yo3L4z31cTYyBIk/TW4xZPqNy+M99XE2MgSJO0/pxTWbxowSvKFi9awPpzTrXvEdJXE+NJZWmSRk5oTubqF/vOTF9NjH9CU5LmOf+EpiRpQvoKhCRrkuxNsi/JlWPUn5xke5LdSe5OsrSn7o+SPJxkT5IbkqSV3932+UB7/L3pm5YkaaLGDYQkC4AbgXOBFcAlSVaManYdcHNVrQQ2Ale3vr8J/BawEvhHwOnA6p5+H62qVe3xk6lORpI0ef0cIZwB7Kuqx6rqReBW4LxRbVYA29vzu3rqC/g14BjgdcAi4K+mOmhJ0vTrJxBOBJ7o2R5qZb12ARe25+cDxyY5vqrupRsQT7bHtqra09Pva2256N+NLCWNlmRdksEkg8PDw30MV5I0Gf0Ewlhv1KMvTbocWJ1kJ90lof3AwSS/DrwDWEo3RM5O8v7W56NV9U7gfe3x8bF+eVXdVFWdquoMDAz0MVxJ0mT0EwhDwEk920uBA70NqupAVV1QVacBV7WyZ+keLdxXVc9V1XPAHcCZrX5/+/kz4L/RXZqSJM2SfgLhfmB5klOSHANcDGztbZBkSZKRfW0ANrfnP6Z75LAwySK6Rw972vaS1ncR8E+Bh6Y+HUnSZI0bCFV1ELgM2AbsAb5ZVQ8n2ZjkI63ZWcDeJI8CbwE2tfJvAf8XeJDueYZdVfVndE8wb0uyG3iA7hLTf522WUmSJsxvKkvSPOc3lSVJE2IgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkABb20yjJGuA/AQuAr1TVH46qPxnYDAwATwMfq6qhVvdHwIfphs93gc9UVSV5D7AFWAx8Z6R8Oib1Wm7fuZ9rt+3lwDMv8NbjFrP+nFNZe9qJ9j0Cfudc7asj31x8Xc3WazLjvQcnWQA8CnwQGALuBy6pqkd62vx34H9W1deTnA18qqo+nuQ3gWuB97em/xvYUFV3J/kL4DPAfXQD4YaquuNQY+l0OjU4ODiZeXL7zv1suO1BXnjp5V+VLV60gKsveOe4/9BHU9+5Nt7Z7Ksj31x8XR2O12SSHVXVGa9dP0tGZwD7quqxqnoRuBU4b1SbFcD29vyunvoCfg04BngdsAj4qyQnAG+sqnvbUcHNwNo+xjJp127b+4p/YIAXXnqZa7ftte8cHu9s9tWRby6+rmbzNdlPIJwIPNGzPdTKeu0CLmzPzweOTXJ8Vd1LNyCebI9tVbWn9R8aZ58AJFmXZDDJ4PDwcB/DHduBZ16YUPnR2neujXc2++rINxdfV7P5muwnEDJG2eh1psuB1Ul2AquB/cDBJL8OvANYSvcN/+wk7+9zn93CqpuqqlNVnYGBgT6GO7a3Hrd4QuVHa9+5Nt7Z7Ksj31x8Xc3ma7KfQBgCTurZXgoc6G1QVQeq6oKqOg24qpU9S/do4b6qeq6qngPuAM5s+1x6qH1Ot/XnnMriRQteUbZ40QLWn3OqfefweGezr458c/F1NZuvyX6uMrofWJ7kFLqf/C8Gfq+3QZIlwNNV9UtgA90rjgB+DPyLJFfTPSpYDXyhqp5M8rMkZwI/BD4B/OfpmNBrGTkZM5kz90dT37k23tnsqyPfXHxdzeZrctyrjACSfAj4At3LTjdX1aYkG4HBqtqa5CLgarrLPvcAl1bVL9oVSl+ke5VRAX9eVZ9t++zwt5ed3gH8wXiXnU7lKiNJOlr1e5VRX4FwpDAQJGnipvOyU0nSUcBAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJQJ+BkGRNkr1J9iW5coz6k5NsT7I7yd1JlrbyDyR5oOfxN0nWtrotSR7vqVs1vVOTJE3EwvEaJFkA3Ah8EBgC7k+ytaoe6Wl2HXBzVX09ydnA1cDHq+ouYFXbz5uBfcCdPf3WV9W3pmcqkqSp6OcI4QxgX1U9VlUvArcC541qswLY3p7fNUY9wEXAHVX1/GQHK0k6fPoJhBOBJ3q2h1pZr13Ahe35+cCxSY4f1eZi4JZRZZvaMtP1SV431i9Psi7JYJLB4eHhPoYrSZqMfgIhY5TVqO3LgdVJdgKrgf3AwV/tIDkBeCewrafPBuDtwOnAm4ErxvrlVXVTVXWqqjMwMNDHcCVJkzHuOQS6RwQn9WwvBQ70NqiqA8AFAEneAFxYVc/2NPkd4E+r6qWePk+2p79I8jW6oSJJmiX9HCHcDyxPckqSY+gu/WztbZBkSZKRfW0ANo/axyWMWi5qRw0kCbAWeGjiw5ckTZdxA6GqDgKX0V3u2QN8s6oeTrIxyUdas7OAvUkeBd4CbBrpn2QZ3SOM743a9TeSPAg8CCwB/sOUZiJJmpJUjT4dcOTqdDo1ODg428OQpDklyY6q6ozXzm8qS5IAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkoM9ASLImyd4k+5JcOUb9yUm2J9md5O4kS1v5B5I80PP4myRrW90pSX6Y5EdJ/iTJMdM7NUnSRIwbCEkWADcC5wIrgEuSrBjV7Drg5qpaCWwErgaoqruqalVVrQLOBp4H7mx9rgGur6rlwE+BT0/DfCRJk9TPEcIZwL6qeqyqXgRuBc4b1WYFsL09v2uMeoCLgDuq6vkkoRsQ32p1XwfWTnTwkqTp008gnAg80bM91Mp67QIubM/PB45NcvyoNhcDt7TnxwPPVNXBQ+xTkjSD+gmEjFFWo7YvB1Yn2QmsBvYDI2/2JDkBeCewbQL7HOm7LslgksHh4eE+hitJmox+AmEIOKlneylwoLdBVR2oqguq6jTgqlb2bE+T3wH+tKpeattPAcclWfha++zZ901V1amqzsDAQB/DlSRNRj+BcD+wvF0VdAzdpZ+tvQ2SLEkysq8NwOZR+7iEv10uoqqK7rmGi1rRJ4FvT3z4kqTpMm4gtHX+y+gu9+wBvllVDyfZmOQjrdlZwN4kjwJvATaN9E+yjO4RxvdG7foK4LNJ9tE9p/DVKc1EkjQl6X5Ynxs6nU4NDg7O9jAkaU5JsqOqOuO185vKkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSU1fgZBkTZK9SfYluXKM+pOTbE+yO8ndSZb21L0tyZ1J9iR5JMmyVr4lyeNJHmiPVdM1KUnSxI0bCEkWADcC5wIrgEuSrBjV7Drg5qpaCWwEru6puxm4tqreAZwB/KSnbn1VrWqPB6YwD0nSFPVzhHAGsK+qHquqF4FbgfNGtVkBbG/P7xqpb8GxsKq+C1BVz1XV89MycknStOonEE4EnujZHmplvXYBF7bn5wPHJjke+A3gmSS3JdmZ5Np2xDFiU1tmuj7J6yY5B0nSNOgnEDJGWY3avhxYnWQnsBrYDxwEFgLva/WnA38f+Oetzwbg7a38zcAVY/7yZF2SwSSDw8PDfQxXkjQZ/QTCEHBSz/ZS4EBvg6o6UFUXVNVpwFWt7NnWd2dbbjoI3A68u9U/WV2/AL5Gd2nqVarqpqrqVFVnYGBggtOTJPWrn0C4H1ie5JQkxwAXA1t7GyRZkmRkXxuAzT1935Rk5J38bOCR1ueE9jPAWuChqUxEkjQ1qRq9+jNGo+RDwBeABcDmqtqUZCMwWFVbk1xE98qiAu4BLm2f/EnyQeCP6S497QDWVdWLSf4XMNDKHwB+v6qeG2ccw8BfTm6qr7AEeGoa9jNXHE3zPZrmCs53vpuu+Z5cVeMusfQVCPNNksGq6sz2OGbK0TTfo2mu4Hznu5mer99UliQBBoIkqTlaA+Gm2R7ADDua5ns0zRWc73w3o/M9Ks8hSJJe7Wg9QpAkjTJvAyHJ5iQ/STLm9xvSdUO7g+vuJO+e6TFOpz7m+9E2z91JfpDkXTM9xuk03nx72p2e5OV2afSc1M9ck5zV7hr8cJLvzeT4plsfr+W/m+TPkuxq8/3UTI9xOiU5Kcld7Y7QDyf5zBhtZuT9at4GArAFWHOI+nOB5e2xDvjSDIzpcNrCoef7OLC63ZH288z9tdgtHHq+I3fqvQbYNhMDOoy2cIi5JjkO+CLwkar6h8A/m6FxHS5bOPT/7aXAI1X1LuAs4I/bl2bnqoPAv2l3hD4TuHSMO0rPyPvVvA2EqroHePoQTc6je8vuqqr7gONGvj09F40336r6QVX9tG3eR/cWJHNWH/+/AH8A/A9eecv1OaePuf4ecFtV/bi1n+/zLbo30Azwhtb24EyM7XBot/H5P+35z4A9vPoGojPyfjVvA6EP/dzFdb76NHDHbA/icEpyIt077355tscyA36D7i1i7k6yI8knZntAh9l/Ad5B955qDwKfqapfzu6Qpkf7A2KnAT8cVTUj71cLp3uHc0g/d3Gdd5J8gG4g/OPZHsth9gXgiqp6uftBcl5bCLwH+CfAYuDeJPdV1aOzO6zD5hy6t7s5G/gHwHeTfL+q/np2hzU1Sd5A94j2X48xlxl5vzqaA2Hcu7jON0lWAl8Bzq2q/zfb4znMOsCtLQyWAB9KcrCqbp/dYR0WQ8BTVfVz4OdJ7gHeBczXQPgU8IfVvWZ+X5LH6d5K/y9md1iTl2QR3TD4RlXdNkaTGXm/OpqXjLYCn2hn788Enq2qJ2d7UIdLkrcBtwEfn8efHH+lqk6pqmVVtQz4FvCv5mkYAHwbeF+ShUleD7yX7jr0fPVjukdDJHkLcCrw2KyOaArauZCvAnuq6j++RrMZeb+at0cISW6hewXCkiRDwOeARQBV9WXgO8CHgH3A83Q/dcxZfcz33wPHA19sn5oPzuWbhPUx33ljvLlW1Z4kfw7sBn4JfKWq5uzt5Pv4v/08sCXJg3SXUq6oqrl8B9TfAj4OPJhk5G/L/1vgbTCz71d+U1mSBBzdS0aSpB4GgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiQA/j+wUECaI8qFawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "C_params3=np.linspace(1,2,21)\n",
    "scores3=list()\n",
    "for C in C_params3:\n",
    "    clf.C=C\n",
    "    tmp_score=cross_val_score(clf,data,target,cv=5,n_jobs=-1)\n",
    "    scores3.append(np.mean(tmp_score))\n",
    "plt.scatter(C_params3,scores3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- ランダムシャッフリングして確認してみる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9667 (+/- 0.0211)\n"
     ]
    }
   ],
   "source": [
    "clf.C=1.6\n",
    "data_shuffle,target_shuffle=shuffle(data,target)\n",
    "score=cross_val_score(clf,data_shuffle,target_shuffle,cv=5)\n",
    "print(\"Accuracy: {0:04.4f} (+/- {1:04.4f})\".format(score.mean(),score.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "あまりチューニングできてない..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bfb16a8f24f499e9bbb29519110d7e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=1, description='min_C', max=10, min=1), IntSlider(value=10, description=…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(min_C=(1,10,1),max_C=(1,10,1),step=(5,20,1))\n",
    "def cost_search(min_C=1,max_C=10,step=5):\n",
    "    params=np.linspace(min_C,max_C,step)\n",
    "    rslt=list()\n",
    "    for C in params:\n",
    "        clf.C=C\n",
    "        rslt.append(np.mean(cross_val_score(clf,data,target,cv=5,n_jobs=-1)))\n",
    "    plt.scatter(params,rslt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# おまけ\n",
    "> cross_val_scoreを使わず実装するなら"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "skf=StratifiedKFold(n_splits=5, shuffle=True) # generaterを作成する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all:  [0.93333333 0.96666667 0.96666667 1.         1.        ]\n",
      "Accuracy: 0.9733 (+/- 0.0249)\n"
     ]
    }
   ],
   "source": [
    "score=list()\n",
    "for train_index,test_index in skf.split(data, target):\n",
    "    data_train=data[train_index]\n",
    "    data_test=data[test_index]\n",
    "    target_train=target[train_index]\n",
    "    target_test=target[test_index]\n",
    "    clf.fit(data_train,target_train)\n",
    "    rslt=clf.predict(data_test)\n",
    "    tmp_score=0\n",
    "    for i,j in enumerate(rslt):\n",
    "        if j==target_test[i]:tmp_score+=1\n",
    "    score.append(tmp_score/len(test_index))\n",
    "score=np.array(score)\n",
    "print(\"all: \",score)\n",
    "print(\"Accuracy: {0:04.4f} (+/- {1:04.4f})\".format(score.mean(),score.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# トレーニング/検証用データの分割\n",
    "\n",
    "> train_test_split\n",
    "\n",
    "- マニュアルで二分割したい時"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "data_train,data_test,target_train,target_test=cv.train_test_split(\n",
    "                                                data, target, train_size=0.95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "142\n",
      "8\n",
      "142\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "print(len(data_train))\n",
    "print(len(data_test))\n",
    "print(len(target_train))\n",
    "print(len(target_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- train_sizeで0~1の範囲で分割する\n",
    "- 最初に出てくるものがtrain用データ\n",
    "- random_stateでseed固定も可能\n",
    "- 実は分割したい配列は１つや２つだけでなく、いくつでも渡せる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "clf.fit(data_train,target_train)\n",
    "target_pred=clf.predict(data_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>prediction</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fact</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0  1  2  3  4  5  6  7\n",
       "prediction  1  1  0  0  2  0  1  0\n",
       "fact        1  1  0  0  2  0  1  0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(np.c_[target_pred,target_test],columns=[\"prediction\",\"fact\"]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "shuffleと同様に、random_stateを指定せずに分割すると、１行で実行しないとばらばらなseedでランダム分割されるので注意"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train_NG,data_test_NG=cv.train_test_split(data,train_size=0.95)\n",
    "target_train_NG,target_test_NG=cv.train_test_split(target,train_size=0.95)#上とseedが違う可能性あり!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>prediction</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fact</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0  1  2  3  4  5  6  7\n",
       "prediction  2  2  1  0  2  0  1  2\n",
       "fact        2  0  1  0  2  2  2  0"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(data_train_NG,target_train_NG)\n",
    "target_pred_NG=clf.predict(data_test_NG)\n",
    "pd.DataFrame(np.c_[target_pred_NG,target_test_NG],columns=[\"prediction\",\"fact\"]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ここまでやってなんですが、CVを単独で使うケースは少ないです。\n",
    "次で紹介するGridSearchはGridSearchCVでまとめてやるほうが多い気がします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 終わり\n",
    "@y__sama"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}