-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
426 lines (225 loc) · 305 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>简单点</title>
<link href="/atom.xml" rel="self"/>
<link href="http://czx.im/"/>
<updated>2018-04-18T15:56:14.015Z</updated>
<id>http://czx.im/</id>
<author>
<name>Zixiang CAI</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>Tensorflow学习笔记7(tf.slim）</title>
<link href="http://czx.im/2018/04/18/Tensorflow7/"/>
<id>http://czx.im/2018/04/18/Tensorflow7/</id>
<published>2018-04-18T15:55:06.000Z</published>
<updated>2018-04-18T15:56:14.015Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="What’s-tf-slim"><a href="#What’s-tf-slim" class="headerlink" title="What’s tf.slim?"></a>What’s tf.slim?</h1><p># </p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="What’s-tf-slim"><a href="#What’s-tf-slim" class="headerlink" title="What’s tf.slim
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>木心语录</title>
<link href="http://czx.im/2018/04/18/words_of_muxin/"/>
<id>http://czx.im/2018/04/18/words_of_muxin/</id>
<published>2018-04-18T09:01:05.000Z</published>
<updated>2018-04-18T12:38:00.259Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><ul><li><p>往过去看，一代比一代多情，往未来看，一代比一代无情。多情可以多到没际涯，无情则有限，无情而已。</p></li><li><p>好看的人，咬指甲时尤其好看。</p></li><li><p>那口唇美得已是一个吻。</p></li><li><p>雨后，总像有谁离去了。</p></li><li><p>容易钟情的人，是无酒量的贪杯者。</p></li><li><p>岁月不饶人，我亦未曾饶过岁月。</p></li><li><p>我追索人心的深度，却看到人心的浅薄。</p></li><li><p>麻木的人都爱说，跟着感觉走。</p></li><li><p>玩物丧志，其志小，志大者玩物养志。</p></li><li><p>美无性别，若有性别则是性不是美。</p></li><li><p>你常常笑得使我看不清。</p></li><li><p>给他们面子，是我自己要面子。</p></li><li><p>我的存在，已是礼节性的存在。</p></li><li><p>我是一个在黑暗中大雪纷飞的人哪，你再不来，我要下雪了。</p></li><li><p>从前的那个我，如果来找现在的我，会得到很好的款待。</p></li></ul>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><ul>
<li><p>往过去看，一代比一代多情，往未来看，一代比一代无情。多情可以多到没际涯，无情则有限，无情而已。</p>
</li>
<li><p>好看的人，咬指甲时尤其好看
</summary>
<category term="语录" scheme="http://czx.im/categories/%E8%AF%AD%E5%BD%95/"/>
<category term="无病呻吟" scheme="http://czx.im/tags/%E6%97%A0%E7%97%85%E5%91%BB%E5%90%9F/"/>
</entry>
<entry>
<title>Tensorflow学习笔记6(Transfer Learning迁移学习）</title>
<link href="http://czx.im/2018/04/18/Tensorflow6/"/>
<id>http://czx.im/2018/04/18/Tensorflow6/</id>
<published>2018-04-18T07:32:55.000Z</published>
<updated>2018-04-19T19:08:09.370Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="Dogs-vs-Cats"><a href="#Dogs-vs-Cats" class="headerlink" title="Dogs vs. Cats"></a>Dogs vs. Cats</h1><p>这次用的数据集来自 kaggle 上的一个竞赛：Dogs vs. Cats，训练集有25000张，猫狗各占一半。测试集12500张，没有标定是猫还是狗。<br>首先用一个3层conv+pooling和3层fc的网络来学习，这个模型在batch=32的情况下，收敛比较不稳定。但是经过7000个epoch，测试集准确率也达到70%。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">import</span> input</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="comment">#Hyperparameters</span></span><br><span class="line">learning_rate = <span class="number">0.0001</span></span><br><span class="line">batch_size = <span class="number">32</span></span><br><span class="line">dropout = <span class="number">0.5</span></span><br><span class="line">training_iters = <span class="number">100000</span></span><br><span class="line">display_step = <span class="number">30</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#params</span></span><br><span class="line">length = <span class="number">224</span></span><br><span class="line">width = <span class="number">256</span></span><br><span class="line">dimension = <span class="number">3</span></span><br><span class="line">n_classes = <span class="number">2</span></span><br><span class="line"></span><br><span class="line">os.environ[<span class="string">"CUDA_DEVICE_ORDER"</span>]=<span class="string">"PCI_BUS_ID"</span> <span class="comment"># see issue #152</span></span><br><span class="line">os.environ[<span class="string">"CUDA_VISIBLE_DEVICES"</span>]=<span class="string">"0"</span></span><br><span class="line"><span class="comment">#create data generator</span></span><br><span class="line">train_generator = input.trainGenerator(int(batch_size/<span class="number">2</span>), length, width)</span><br><span class="line">val_generator = input.valGenerator(int(batch_size/<span class="number">2</span>), length, width)</span><br><span class="line"></span><br><span class="line"><span class="comment">#io</span></span><br><span class="line">x = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, length, width, dimension])</span><br><span class="line">y = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, n_classes])</span><br><span class="line">keep_prob = tf.placeholder(tf.float32)</span><br><span class="line"></span><br><span class="line"><span class="comment">#kernel size</span></span><br><span class="line">weights = {</span><br><span class="line"> <span class="string">'c1'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">3</span>,<span class="number">32</span>], stddev=<span class="number">0.001</span>, name=<span class="string">'wc1'</span>)),</span><br><span class="line"> <span class="string">'c2'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">32</span>,<span class="number">32</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc2'</span>)),</span><br><span class="line"> <span class="string">'c3'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">32</span>,<span class="number">64</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc3'</span>)),</span><br><span class="line"> <span class="string">'f1'</span>: tf.Variable(tf.truncated_normal([<span class="number">32</span>*<span class="number">28</span>*<span class="number">64</span>, <span class="number">512</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f2'</span>: tf.Variable(tf.truncated_normal([<span class="number">512</span>, <span class="number">128</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'fc2'</span>)),</span><br><span class="line"> <span class="string">'f3'</span>: tf.Variable(tf.truncated_normal([<span class="number">128</span>, <span class="number">2</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'fc3'</span>)),</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">biases = {</span><br><span class="line"> <span class="string">'c1'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">32</span>], name=<span class="string">'bc1'</span>)),</span><br><span class="line"> <span class="string">'c2'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">32</span>], name=<span class="string">'bc2'</span>)),</span><br><span class="line"> <span class="string">'c3'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">64</span>], name=<span class="string">'bc3'</span>)),</span><br><span class="line"> <span class="string">'f1'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">512</span>], name=<span class="string">'fb1'</span>)),</span><br><span class="line"> <span class="string">'f2'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">128</span>], name=<span class="string">'fb2'</span>)),</span><br><span class="line"> <span class="string">'f3'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">2</span>], name=<span class="string">'fb3'</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">#funcs</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">conv2d</span><span class="params">(input, kernel, bias, name)</span>:</span></span><br><span class="line"> wx_plus_b = tf.nn.bias_add(tf.nn.conv2d(input, kernel, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>), bias)</span><br><span class="line"> batch_norm = bn_layer(wx_plus_b, <span class="keyword">True</span>, name)</span><br><span class="line"> <span class="keyword">return</span> tf.nn.sigmoid(batch_norm, name=name)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">pooling</span><span class="params">(input, k, name)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.max_pool(input, ksize=[<span class="number">1</span>,k,k,<span class="number">1</span>], strides=[<span class="number">1</span>,k,k,<span class="number">1</span>], padding=<span class="string">'SAME'</span>, name=name)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实现Batch Normalization</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bn_layer</span><span class="params">(x,is_training,name,moving_decay=<span class="number">0.9</span>,eps=<span class="number">1e-5</span>)</span>:</span></span><br><span class="line"> <span class="comment"># 获取输入维度并判断是否匹配卷积层(4)或者全连接层(2)</span></span><br><span class="line"> shape = x.shape</span><br><span class="line"> <span class="keyword">assert</span> len(shape) <span class="keyword">in</span> [<span class="number">2</span>,<span class="number">4</span>]</span><br><span class="line"></span><br><span class="line"> param_shape = shape[<span class="number">-1</span>]</span><br><span class="line"> <span class="keyword">with</span> tf.variable_scope(name):</span><br><span class="line"> <span class="comment"># 声明BN中唯一需要学习的两个参数，y=gamma*x+beta</span></span><br><span class="line"> gamma = tf.get_variable(<span class="string">'gamma'</span>,param_shape,initializer=tf.constant_initializer(<span class="number">1</span>))</span><br><span class="line"> beta = tf.get_variable(<span class="string">'beat'</span>, param_shape,initializer=tf.constant_initializer(<span class="number">0</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 计算当前整个batch的均值与方差</span></span><br><span class="line"> axes = list(range(len(shape)<span class="number">-1</span>))</span><br><span class="line"> batch_mean, batch_var = tf.nn.moments(x,axes,name=<span class="string">'moments'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 采用滑动平均更新均值与方差</span></span><br><span class="line"> ema = tf.train.ExponentialMovingAverage(moving_decay)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">mean_var_with_update</span><span class="params">()</span>:</span></span><br><span class="line"> ema_apply_op = ema.apply([batch_mean,batch_var])</span><br><span class="line"> <span class="keyword">with</span> tf.control_dependencies([ema_apply_op]):</span><br><span class="line"> <span class="keyword">return</span> tf.identity(batch_mean), tf.identity(batch_var)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 训练时，更新均值与方差，测试时使用之前最后一次保存的均值与方差</span></span><br><span class="line"> mean, var = tf.cond(tf.equal(is_training,<span class="keyword">True</span>),mean_var_with_update,</span><br><span class="line"> <span class="keyword">lambda</span>:(ema.average(batch_mean),ema.average(batch_var)))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 最后执行batch normalization</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.batch_normalization(x,mean,var,beta,gamma,eps)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fc_layer</span><span class="params">(input, weight, bias, name, batch_norm = True)</span>:</span></span><br><span class="line"> wx_plus_b = tf.matmul(input, weight) + bias</span><br><span class="line"> <span class="keyword">if</span> batch_norm:</span><br><span class="line"> wx_plus_b = bn_layer(wx_plus_b, <span class="keyword">True</span>, name)</span><br><span class="line"> <span class="keyword">return</span> tf.nn.sigmoid(wx_plus_b, name=name)</span><br><span class="line"></span><br><span class="line"><span class="comment">#model</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alexnet</span><span class="params">(X, weights, biases, dropout)</span>:</span></span><br><span class="line"> <span class="comment"># X = tf.reshape(X, [-1, 28, 28, 1])</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">#conv1</span></span><br><span class="line"> conv1 = conv2d(X, weights[<span class="string">'c1'</span>], biases[<span class="string">'c1'</span>], <span class="string">'conv1'</span>)</span><br><span class="line"> pool1 = pooling(conv1, <span class="number">2</span>, <span class="string">'pool1'</span>)</span><br><span class="line"> norm1 = tf.nn.dropout(pool1, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv2</span></span><br><span class="line"> conv2 = conv2d(norm1, weights[<span class="string">'c2'</span>], biases[<span class="string">'c2'</span>], <span class="string">'conv2'</span>)</span><br><span class="line"> pool2 = pooling(conv2, <span class="number">2</span>, <span class="string">'pool2'</span>)</span><br><span class="line"> norm2 = tf.nn.dropout(pool2, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv3</span></span><br><span class="line"> conv3 = conv2d(norm2, weights[<span class="string">'c3'</span>], biases[<span class="string">'c3'</span>], <span class="string">'conv3'</span>)</span><br><span class="line"> pool3 = pooling(conv3, <span class="number">2</span>, <span class="string">'pool3'</span>)</span><br><span class="line"> norm3 = tf.nn.dropout(pool3, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment">#fc</span></span><br><span class="line"> fc1 = tf.reshape(norm3, [<span class="number">-1</span>, weights[<span class="string">'f1'</span>].get_shape().as_list()[<span class="number">0</span>]])</span><br><span class="line"> fc1 = fc_layer(fc1, weights[<span class="string">'f1'</span>], biases[<span class="string">'f1'</span>], <span class="string">'fc1'</span>)</span><br><span class="line"> fc2 = fc_layer(fc1, weights[<span class="string">'f2'</span>], biases[<span class="string">'f2'</span>], <span class="string">'fc2'</span>)</span><br><span class="line"> fc2 = tf.nn.dropout(fc2, dropout)</span><br><span class="line"> fc3 = fc_layer(fc2, weights[<span class="string">'f3'</span>], biases[<span class="string">'f3'</span>], <span class="string">'fc3'</span>, batch_norm=<span class="keyword">False</span>)</span><br><span class="line"> <span class="keyword">return</span> fc3</span><br><span class="line"></span><br><span class="line"><span class="comment">#construct model</span></span><br><span class="line">model = alexnet(x, weights, biases, keep_prob)</span><br><span class="line"></span><br><span class="line"><span class="comment">#learn</span></span><br><span class="line">loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=model))</span><br><span class="line">optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)</span><br><span class="line">correct_prediction = tf.equal(tf.argmax(model,<span class="number">1</span>), tf.argmax(y,<span class="number">1</span>))</span><br><span class="line">accuracy = tf.reduce_mean(tf.cast(correct_prediction, <span class="string">"float"</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment">#init</span></span><br><span class="line">init = tf.initialize_all_variables()</span><br><span class="line"></span><br><span class="line"><span class="comment">#run!</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"> step = <span class="number">0</span></span><br><span class="line"> <span class="comment"># Keep training until reach max iterations</span></span><br><span class="line"> <span class="keyword">while</span> step < training_iters:</span><br><span class="line"> batch_xs, batch_ys = next(train_generator)</span><br><span class="line"> <span class="comment"># get batch data</span></span><br><span class="line"> sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys, keep_prob: dropout})</span><br><span class="line"> <span class="keyword">if</span> step % display_step == <span class="number">0</span>:</span><br><span class="line"> <span class="comment"># res = sess.run(model, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})</span></span><br><span class="line"> <span class="comment"># print(res)</span></span><br><span class="line"> <span class="comment"># Train acc</span></span><br><span class="line"> acc = sess.run(accuracy, feed_dict={x: batch_xs, y: batch_ys, keep_prob: <span class="number">1.</span>})</span><br><span class="line"> <span class="comment"># train loss</span></span><br><span class="line"> loss_val = sess.run(loss, feed_dict={x: batch_xs, y: batch_ys, keep_prob: <span class="number">1.</span>})</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Iter "</span> + str(step) + <span class="string">", Minibatch Loss= "</span> + <span class="string">"{:.6f}"</span>.format(loss_val) + <span class="string">", Training Accuracy= "</span> + <span class="string">"{:.5f}"</span>.format(acc))</span><br><span class="line"> <span class="comment"># Test acc</span></span><br><span class="line"> <span class="keyword">if</span> step % <span class="number">100</span> == <span class="number">0</span>:</span><br><span class="line"> batch_x, batch_y = next(val_generator)</span><br><span class="line"> print(<span class="string">"Validation Accuracy:"</span>, sess.run(accuracy, feed_dict={x: batch_x, y: batch_y, keep_prob: <span class="number">1.</span>}),<span class="string">"loss:"</span>, sess.run(loss, feed_dict={x: batch_xs, y: batch_ys, keep_prob: <span class="number">1.</span>}))</span><br><span class="line"> step += <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Optimization Finished!"</span>)</span><br></pre></td></tr></table></figure></p><h1 id="What’s-transfer-learning"><a href="#What’s-transfer-learning" class="headerlink" title="What’s transfer learning"></a>What’s transfer learning</h1><p>通过使用之前在大数据集上经过训练的预训练模型，我们可以直接使用相应的结构和权重，将它们应用到我们正在面对的问题上。这被称作是“迁移学习”，即将预训练的模型“迁移”到我们正在应对的特定问题中。<br>严格来说，这种迁移的方式是一种模型迁移。在具体操作的时候，将一个预先训练好的神经网络，用新数据集重新训练网络中的一小部分，通常是最后一层或者几层全连接层，卷积层一般不动。从而完成在新数据集上建立的算法任务，即完成了神经网络的迁移学习。<br>我们可以理解成将前面固定不动的部分作为一个特征提取器，而后面训练的层作为分类器。<br>具体训练哪些层，也取决于使用的预训练模型和现有的标注数据的规模：<br><img src="http://p59louklr.bkt.clouddn.com/18-4-19/86822586.jpg" alt=""></p><h1 id="Why-we-need-transfer-learning"><a href="#Why-we-need-transfer-learning" class="headerlink" title="Why we need transfer learning"></a>Why we need transfer learning</h1><p>之后想着用一个复杂一点的模型来拟合，Alexnet和VGG16之类的，但是试了一下，发现都没法收敛。刚开始还想是不是模型或者超参数哪里出了问题，调了半天。后来觉得可能是因为数据量小而模型过于复杂导致的。<br>所以这时候如果还希望用这些复杂网络，最好的办法就是到网上去下载网络的预训练模型(pretrained model)，然后对最后几层进行fine tuning。</p><h1 id="Inception-V3"><a href="#Inception-V3" class="headerlink" title="Inception_V3"></a>Inception_V3</h1><p><img src="http://p59louklr.bkt.clouddn.com/18-4-19/69872972.jpg" alt=""><br>首先用tensorflow实现一个Inception_V3的迁移学习。在这个模型中只取到bottleneck，就是卷积层的末尾，抛掉预训练的所有全连接层。预训练部分的参数在迁移学习中不会被改变，这里可以把这部分网络当做特征提取器。之后再接上一个全连接层用来分类，这一层是需要训练的。<br>预训练模型的graph和参数是一个pb文件，可以在网上下载。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> glob</span><br><span class="line"><span class="keyword">import</span> os.path</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">from</span> tensorflow.python.platform <span class="keyword">import</span> gfile</span><br><span class="line"></span><br><span class="line"><span class="comment"># Inception-v3模型瓶颈层的节点个数</span></span><br><span class="line">BOTTLENECK_TENSOR_SIZE = <span class="number">2048</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Inception-v3模型中代表瓶颈层结果的张量名称。</span></span><br><span class="line"><span class="comment"># 在谷歌提出的Inception-v3模型中，这个张量名称就是'pool_3/_reshape:0'。</span></span><br><span class="line"><span class="comment"># 在训练模型时，可以通过tensor.name来获取张量的名称。</span></span><br><span class="line">BOTTLENECK_TENSOR_NAME = <span class="string">'pool_3/_reshape:0'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 图像输入张量所对应的名称。</span></span><br><span class="line">JPEG_DATA_TENSOR_NAME = <span class="string">'DecodeJpeg/contents:0'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 下载的谷歌训练好的Inception-v3模型文件目录</span></span><br><span class="line">MODEL_DIR = <span class="string">'model/'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 下载的谷歌训练好的Inception-v3模型文件名</span></span><br><span class="line">MODEL_FILE = <span class="string">'tensorflow_inception_graph.pb'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 因为一个训练数据会被使用多次，所以可以将原始图像通过Inception-v3模型计算得到的特征向量保存在文件中，免去重复的计算。</span></span><br><span class="line"><span class="comment"># 下面的变量定义了这些文件的存放地址。</span></span><br><span class="line">CACHE_DIR = <span class="string">'tmp/bottleneck/'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 图片数据文件夹。</span></span><br><span class="line"><span class="comment"># 在这个文件夹中每一个子文件夹代表一个需要区分的类别，每个子文件夹中存放了对应类别的图片。</span></span><br><span class="line">INPUT_DATA = <span class="string">'train/'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 验证的数据百分比</span></span><br><span class="line">VALIDATION_PERCENTAGE = <span class="number">10</span></span><br><span class="line"><span class="comment"># 测试的数据百分比</span></span><br><span class="line">TEST_PERCENTAGE = <span class="number">10</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义神经网络的设置</span></span><br><span class="line">LEARNING_RATE = <span class="number">0.01</span></span><br><span class="line">STEPS = <span class="number">4000</span></span><br><span class="line">BATCH = <span class="number">100</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数从数据文件夹中读取所有的图片列表并按训练、验证、测试数据分开。</span></span><br><span class="line"><span class="comment"># testing_percentage和validation_percentage参数指定了测试数据集和验证数据集的大小。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">create_image_lists</span><span class="params">(testing_percentage, validation_percentage)</span>:</span></span><br><span class="line"> <span class="comment"># 得到的所有图片都存在result这个字典(dictionary)里。</span></span><br><span class="line"> <span class="comment"># 这个字典的key为类别的名称，value也是一个字典，字典里存储了所有的图片名称。</span></span><br><span class="line"> result = {}</span><br><span class="line"> <span class="comment"># 获取当前目录下所有的子目录</span></span><br><span class="line"> sub_dirs = [x[<span class="number">0</span>] <span class="keyword">for</span> x <span class="keyword">in</span> os.walk(INPUT_DATA)]</span><br><span class="line"> <span class="comment"># 得到的第一个目录是当前目录，不需要考虑</span></span><br><span class="line"> is_root_dir = <span class="keyword">True</span></span><br><span class="line"> <span class="keyword">for</span> sub_dir <span class="keyword">in</span> sub_dirs:</span><br><span class="line"> <span class="keyword">if</span> is_root_dir:</span><br><span class="line"> is_root_dir = <span class="keyword">False</span></span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 获取当前目录下所有的有效图片文件。</span></span><br><span class="line"> extensions = [<span class="string">'jpg'</span>, <span class="string">'jpeg'</span>, <span class="string">'JPG'</span>, <span class="string">'JPEG'</span>]</span><br><span class="line"> file_list = []</span><br><span class="line"> dir_name = os.path.basename(sub_dir)</span><br><span class="line"> <span class="keyword">for</span> extension <span class="keyword">in</span> extensions:</span><br><span class="line"> file_glob = os.path.join(INPUT_DATA, dir_name, <span class="string">'*.'</span>+extension)</span><br><span class="line"> file_list.extend(glob.glob(file_glob))</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> file_list:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 通过目录名获取类别的名称。</span></span><br><span class="line"> label_name = dir_name.lower()</span><br><span class="line"> <span class="comment"># 初始化当前类别的训练数据集、测试数据集和验证数据集</span></span><br><span class="line"> training_images = []</span><br><span class="line"> testing_images = []</span><br><span class="line"> validation_images = []</span><br><span class="line"> <span class="keyword">for</span> file_name <span class="keyword">in</span> file_list:</span><br><span class="line"> base_name = os.path.basename(file_name)</span><br><span class="line"> <span class="comment"># 随机将数据分到训练数据集、测试数据集和验证数据集。</span></span><br><span class="line"> chance = np.random.randint(<span class="number">100</span>)</span><br><span class="line"> <span class="keyword">if</span> chance < validation_percentage:</span><br><span class="line"> validation_images.append(base_name)</span><br><span class="line"> <span class="keyword">elif</span> chance < (testing_percentage + validation_percentage):</span><br><span class="line"> testing_images.append(base_name)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> training_images.append(base_name)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 将当前类别的数据放入结果字典。</span></span><br><span class="line"> result[label_name] = {</span><br><span class="line"> <span class="string">'dir'</span>: dir_name,</span><br><span class="line"> <span class="string">'training'</span>: training_images,</span><br><span class="line"> <span class="string">'testing'</span>: testing_images,</span><br><span class="line"> <span class="string">'validation'</span>: validation_images</span><br><span class="line"> }</span><br><span class="line"> <span class="comment"># 返回整理好的所有数据</span></span><br><span class="line"> <span class="keyword">return</span> result</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数通过类别名称、所属数据集和图片编号获取一张图片的地址。</span></span><br><span class="line"><span class="comment"># image_lists参数给出了所有图片信息。</span></span><br><span class="line"><span class="comment"># image_dir参数给出了根目录。存放图片数据的根目录和存放图片特征向量的根目录地址不同。</span></span><br><span class="line"><span class="comment"># label_name参数给定了类别的名称。</span></span><br><span class="line"><span class="comment"># index参数给定了需要获取的图片的编号。</span></span><br><span class="line"><span class="comment"># category参数指定了需要获取的图片是在训练数据集、测试数据集还是验证数据集。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_image_path</span><span class="params">(image_lists, image_dir, label_name, index, category)</span>:</span></span><br><span class="line"> <span class="comment"># 获取给定类别中所有图片的信息。</span></span><br><span class="line"> label_lists = image_lists[label_name]</span><br><span class="line"> <span class="comment"># 根据所属数据集的名称获取集合中的全部图片信息。</span></span><br><span class="line"> category_list = label_lists[category]</span><br><span class="line"> mod_index = index % len(category_list)</span><br><span class="line"> <span class="comment"># 获取图片的文件名。</span></span><br><span class="line"> base_name = category_list[mod_index]</span><br><span class="line"> sub_dir = label_lists[<span class="string">'dir'</span>]</span><br><span class="line"> <span class="comment"># 最终的地址为数据根目录的地址 + 类别的文件夹 + 图片的名称</span></span><br><span class="line"> full_path = os.path.join(image_dir, sub_dir, base_name)</span><br><span class="line"> <span class="keyword">return</span> full_path</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数通过类别名称、所属数据集和图片编号获取经过Inception-v3模型处理之后的特征向量文件地址。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_bottlenect_path</span><span class="params">(image_lists, label_name, index, category)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> get_image_path(image_lists, CACHE_DIR, label_name, index, category) + <span class="string">'.txt'</span>;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数使用加载的训练好的Inception-v3模型处理一张图片，得到这个图片的特征向量。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_bottleneck_on_image</span><span class="params">(sess, image_data, image_data_tensor, bottleneck_tensor)</span>:</span></span><br><span class="line"> <span class="comment"># 这个过程实际上就是将当前图片作为输入计算瓶颈张量的值。这个瓶颈张量的值就是这张图片的新的特征向量。</span></span><br><span class="line"> bottleneck_values = sess.run(bottleneck_tensor, {image_data_tensor: image_data})</span><br><span class="line"> <span class="comment"># 经过卷积神经网络处理的结果是一个四维数组，需要将这个结果压缩成一个特征向量（一维数组）</span></span><br><span class="line"> bottleneck_values = np.squeeze(bottleneck_values)</span><br><span class="line"> <span class="keyword">return</span> bottleneck_values</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数获取一张图片经过Inception-v3模型处理之后的特征向量。</span></span><br><span class="line"><span class="comment"># 这个函数会先试图寻找已经计算且保存下来的特征向量，如果找不到则先计算这个特征向量，然后保存到文件。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_or_create_bottleneck</span><span class="params">(sess, image_lists, label_name, index, category, jpeg_data_tensor, bottleneck_tensor)</span>:</span></span><br><span class="line"> <span class="comment"># 获取一张图片对应的特征向量文件的路径。</span></span><br><span class="line"> label_lists = image_lists[label_name]</span><br><span class="line"> sub_dir = label_lists[<span class="string">'dir'</span>]</span><br><span class="line"> sub_dir_path = os.path.join(CACHE_DIR, sub_dir)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> os.path.exists(sub_dir_path):</span><br><span class="line"> os.makedirs(sub_dir_path)</span><br><span class="line"> bottleneck_path = get_bottlenect_path(image_lists, label_name, index, category)</span><br><span class="line"> <span class="comment"># 如果这个特征向量文件不存在，则通过Inception-v3模型来计算特征向量，并将计算的结果存入文件。</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> os.path.exists(bottleneck_path):</span><br><span class="line"> <span class="comment"># 获取原始的图片路径</span></span><br><span class="line"> image_path = get_image_path(image_lists, INPUT_DATA, label_name, index, category)</span><br><span class="line"> <span class="comment"># 获取图片内容。</span></span><br><span class="line"> image_data = gfile.FastGFile(image_path, <span class="string">'rb'</span>).read()</span><br><span class="line"> <span class="comment"># print(len(image_data))</span></span><br><span class="line"> <span class="comment"># 由于输入的图片大小不一致，此处得到的image_data大小也不一致（已验证），但却都能通过加载的inception-v3模型生成一个2048的特征向量。具体原理不详。</span></span><br><span class="line"> <span class="comment"># 通过Inception-v3模型计算特征向量</span></span><br><span class="line"> bottleneck_values = run_bottleneck_on_image(sess, image_data, jpeg_data_tensor, bottleneck_tensor)</span><br><span class="line"> <span class="comment"># 将计算得到的特征向量存入文件</span></span><br><span class="line"> bottleneck_string = <span class="string">','</span>.join(str(x) <span class="keyword">for</span> x <span class="keyword">in</span> bottleneck_values)</span><br><span class="line"> <span class="keyword">with</span> open(bottleneck_path, <span class="string">'w'</span>) <span class="keyword">as</span> bottleneck_file:</span><br><span class="line"> bottleneck_file.write(bottleneck_string)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="comment"># 直接从文件中获取图片相应的特征向量。</span></span><br><span class="line"> <span class="keyword">with</span> open(bottleneck_path, <span class="string">'r'</span>) <span class="keyword">as</span> bottleneck_file:</span><br><span class="line"> bottleneck_string = bottleneck_file.read()</span><br><span class="line"> bottleneck_values = [float(x) <span class="keyword">for</span> x <span class="keyword">in</span> bottleneck_string.split(<span class="string">','</span>)]</span><br><span class="line"> <span class="comment"># 返回得到的特征向量</span></span><br><span class="line"> <span class="keyword">return</span> bottleneck_values</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数随机获取一个batch的图片作为训练数据。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_random_cached_bottlenecks</span><span class="params">(sess, n_classes, image_lists, how_many, category,</span></span></span><br><span class="line"><span class="function"><span class="params"> jpeg_data_tensor, bottleneck_tensor)</span>:</span></span><br><span class="line"> bottlenecks = []</span><br><span class="line"> ground_truths = []</span><br><span class="line"> <span class="keyword">for</span> _ <span class="keyword">in</span> range(how_many):</span><br><span class="line"> <span class="comment"># 随机一个类别和图片的编号加入当前的训练数据。</span></span><br><span class="line"> label_index = random.randrange(n_classes)</span><br><span class="line"> label_name = list(image_lists.keys())[label_index]</span><br><span class="line"> image_index = random.randrange(<span class="number">65536</span>)</span><br><span class="line"> bottleneck = get_or_create_bottleneck(sess, image_lists, label_name, image_index, category,</span><br><span class="line"> jpeg_data_tensor, bottleneck_tensor)</span><br><span class="line"> ground_truth = np.zeros(n_classes, dtype=np.float32)</span><br><span class="line"> ground_truth[label_index] = <span class="number">1.0</span></span><br><span class="line"> bottlenecks.append(bottleneck)</span><br><span class="line"> ground_truths.append(ground_truth)</span><br><span class="line"> <span class="keyword">return</span> bottlenecks, ground_truths</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 这个函数获取全部的测试数据。在最终测试的时候需要在所有的测试数据上计算正确率。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_test_bottlenecks</span><span class="params">(sess, image_lists, n_classes, jpeg_data_tensor, bottleneck_tensor)</span>:</span></span><br><span class="line"> bottlenecks = []</span><br><span class="line"> ground_truths = []</span><br><span class="line"> label_name_list = list(image_lists.keys())</span><br><span class="line"> <span class="comment"># 枚举所有的类别和每个类别中的测试图片。</span></span><br><span class="line"> <span class="keyword">for</span> label_index, label_name <span class="keyword">in</span> enumerate(label_name_list):</span><br><span class="line"> category = <span class="string">'testing'</span></span><br><span class="line"> <span class="keyword">for</span> index, unused_base_name <span class="keyword">in</span> enumerate(image_lists[label_name][category]):</span><br><span class="line"> <span class="comment"># 通过Inception-v3模型计算图片对应的特征向量，并将其加入最终数据的列表。</span></span><br><span class="line"> bottleneck = get_or_create_bottleneck(sess, image_lists, label_name, index, category,</span><br><span class="line"> jpeg_data_tensor, bottleneck_tensor)</span><br><span class="line"> ground_truth = np.zeros(n_classes, dtype = np.float32)</span><br><span class="line"> ground_truth[label_index] = <span class="number">1.0</span></span><br><span class="line"> bottlenecks.append(bottleneck)</span><br><span class="line"> ground_truths.append(ground_truth)</span><br><span class="line"> <span class="keyword">return</span> bottlenecks, ground_truths</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(_)</span>:</span></span><br><span class="line"> <span class="comment"># 读取所有图片。</span></span><br><span class="line"> image_lists = create_image_lists(TEST_PERCENTAGE, VALIDATION_PERCENTAGE)</span><br><span class="line"> n_classes = len(image_lists.keys())</span><br><span class="line"> <span class="comment"># 读取已经训练好的Inception-v3模型。</span></span><br><span class="line"> <span class="comment"># 谷歌训练好的模型保存在了GraphDef Protocol Buffer中，里面保存了每一个节点取值的计算方法以及变量的取值。</span></span><br><span class="line"> <span class="comment"># TensorFlow模型持久化的问题在第5章中有详细的介绍。</span></span><br><span class="line"> <span class="keyword">with</span> gfile.FastGFile(os.path.join(MODEL_DIR, MODEL_FILE), <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> graph_def = tf.GraphDef()</span><br><span class="line"> graph_def.ParseFromString(f.read())</span><br><span class="line"> <span class="comment"># 加载读取的Inception-v3模型，并返回数据输入所对应的张量以及计算瓶颈层结果所对应的张量。</span></span><br><span class="line"> bottleneck_tensor, jpeg_data_tensor = tf.import_graph_def(graph_def, return_elements=[BOTTLENECK_TENSOR_NAME, JPEG_DATA_TENSOR_NAME])</span><br><span class="line"> <span class="comment"># 定义新的神经网络输入，这个输入就是新的图片经过Inception-v3模型前向传播到达瓶颈层时的结点取值。</span></span><br><span class="line"> <span class="comment"># 可以将这个过程类似的理解为一种特征提取。</span></span><br><span class="line"> bottleneck_input = tf.placeholder(tf.float32, [<span class="keyword">None</span>, BOTTLENECK_TENSOR_SIZE], name=<span class="string">'BottleneckInputPlaceholder'</span>)</span><br><span class="line"> <span class="comment"># 定义新的标准答案输入</span></span><br><span class="line"> ground_truth_input = tf.placeholder(tf.float32, [<span class="keyword">None</span>, n_classes], name=<span class="string">'GroundTruthInput'</span>)</span><br><span class="line"> <span class="comment"># 定义一层全连接层来解决新的图片分类问题。</span></span><br><span class="line"> <span class="comment"># 因为训练好的Inception-v3模型已经将原始的图片抽象为了更加容易分类的特征向量了，所以不需要再训练那么复杂的神经网络来完成这个新的分类任务。</span></span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'final_training_ops'</span>):</span><br><span class="line"> weights = tf.Variable(tf.truncated_normal([BOTTLENECK_TENSOR_SIZE, n_classes], stddev=<span class="number">0.001</span>))</span><br><span class="line"> biases = tf.Variable(tf.zeros([n_classes]))</span><br><span class="line"> logits = tf.matmul(bottleneck_input, weights) + biases</span><br><span class="line"> final_tensor = tf.nn.softmax(logits)</span><br><span class="line"> <span class="comment"># 定义交叉熵损失函数</span></span><br><span class="line"> cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=ground_truth_input)</span><br><span class="line"> cross_entropy_mean = tf.reduce_mean(cross_entropy)</span><br><span class="line"> train_step = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(cross_entropy_mean)</span><br><span class="line"> <span class="comment"># 计算正确率</span></span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'evaluation'</span>):</span><br><span class="line"> correct_prediction = tf.equal(tf.argmax(final_tensor, <span class="number">1</span>), tf.argmax(ground_truth_input, <span class="number">1</span>))</span><br><span class="line"> evaluation_step = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> tf.global_variables_initializer().run()</span><br><span class="line"> <span class="comment"># 训练过程</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(STEPS):</span><br><span class="line"> <span class="comment"># 每次获取一个batch的训练数据</span></span><br><span class="line"> train_bottlenecks, train_ground_truth = get_random_cached_bottlenecks(</span><br><span class="line"> sess, n_classes, image_lists, BATCH, <span class="string">'training'</span>, jpeg_data_tensor, bottleneck_tensor)</span><br><span class="line"> sess.run(train_step, feed_dict={bottleneck_input: train_bottlenecks, ground_truth_input: train_ground_truth})</span><br><span class="line"> <span class="comment"># 在验证集上测试正确率。</span></span><br><span class="line"> <span class="keyword">if</span> i%<span class="number">100</span> == <span class="number">0</span> <span class="keyword">or</span> i+<span class="number">1</span> == STEPS:</span><br><span class="line"> validation_bottlenecks, validation_ground_truth = get_random_cached_bottlenecks(</span><br><span class="line"> sess, n_classes, image_lists, BATCH, <span class="string">'validation'</span>, jpeg_data_tensor, bottleneck_tensor)</span><br><span class="line"> validation_accuracy = sess.run(evaluation_step, feed_dict={</span><br><span class="line"> bottleneck_input:validation_bottlenecks, ground_truth_input: validation_ground_truth})</span><br><span class="line"> print(<span class="string">'Step %d: Validation accuracy on random sampled %d examples = %.1f%%'</span></span><br><span class="line"> % (i, BATCH, validation_accuracy*<span class="number">100</span>))</span><br><span class="line"> <span class="comment"># 在最后的测试数据上测试正确率</span></span><br><span class="line"> test_bottlenecks, test_ground_truth = get_test_bottlenecks(sess, image_lists, n_classes,</span><br><span class="line"> jpeg_data_tensor, bottleneck_tensor)</span><br><span class="line"> test_accuracy = sess.run(evaluation_step, feed_dict={bottleneck_input: test_bottlenecks,</span><br><span class="line"> ground_truth_input: test_ground_truth})</span><br><span class="line"> print(<span class="string">'Final test accuracy = %.1f%%'</span> % (test_accuracy * <span class="number">100</span>))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> tf.app.run()</span><br></pre></td></tr></table></figure></p><p>然后用keras做一个类似的版本，最后改成接两层可训练的Dense。<br>keras的预训练模型下载都在<a href="https://github.com/fchollet/deep-learning-models/releases" target="_blank" rel="noopener">这里</a>。要换其他的模型来迁移学习只需要换一下h5文件，然后在代码里面改两三个地方。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br></pre></td><td class="code"><pre><span class="line">WEIGHTS_PATH_NO_TOP = <span class="string">'inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> keras.preprocessing.image <span class="keyword">import</span> ImageDataGenerator</span><br><span class="line"><span class="keyword">from</span> keras <span class="keyword">import</span> __version__</span><br><span class="line"><span class="keyword">from</span> keras.applications.inception_v3 <span class="keyword">import</span> InceptionV3, preprocess_input</span><br><span class="line"><span class="keyword">from</span> keras.models <span class="keyword">import</span> Model</span><br><span class="line"><span class="keyword">from</span> keras.layers <span class="keyword">import</span> Dense, GlobalAveragePooling2D</span><br><span class="line"><span class="keyword">from</span> keras.preprocessing.image <span class="keyword">import</span> ImageDataGenerator</span><br><span class="line"><span class="keyword">from</span> keras.optimizers <span class="keyword">import</span> SGD</span><br><span class="line"><span class="keyword">from</span> keras.callbacks <span class="keyword">import</span> EarlyStopping, ModelCheckpoint</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> glob</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line">IM_WIDTH = <span class="number">256</span></span><br><span class="line">IM_HEIGHT = <span class="number">256</span></span><br><span class="line">batch_size = <span class="number">128</span></span><br><span class="line">FC_SIZE = <span class="number">1024</span></span><br><span class="line">epochs = <span class="number">50</span></span><br><span class="line">NB_IV3_LAYERS_TO_FREEZE = <span class="number">-2</span></span><br><span class="line"></span><br><span class="line">TRAIN_DIR = <span class="string">'./catdog/train/'</span></span><br><span class="line">VAL_DIR = <span class="string">'./catdog/validation/'</span></span><br><span class="line">output = <span class="string">'incv3.h5'</span></span><br><span class="line"></span><br><span class="line">train_datagen = ImageDataGenerator(</span><br><span class="line"> rotation_range=<span class="number">30</span>,</span><br><span class="line"> width_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> height_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> shear_range=<span class="number">0.2</span>,</span><br><span class="line"> zoom_range=<span class="number">0.2</span>,</span><br><span class="line"> horizontal_flip=<span class="keyword">True</span></span><br><span class="line">)</span><br><span class="line">val_datagen = ImageDataGenerator(</span><br><span class="line"> rotation_range=<span class="number">30</span>,</span><br><span class="line"> width_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> height_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> shear_range=<span class="number">0.2</span>,</span><br><span class="line"> zoom_range=<span class="number">0.2</span>,</span><br><span class="line"> horizontal_flip=<span class="keyword">True</span></span><br><span class="line">)</span><br><span class="line">test_datagen = ImageDataGenerator(</span><br><span class="line">)</span><br><span class="line">train_generator = train_datagen.flow_from_directory(</span><br><span class="line"> <span class="string">'./catdog/train'</span>,</span><br><span class="line"> target_size=(IM_WIDTH, IM_HEIGHT),</span><br><span class="line"> batch_size=batch_size,</span><br><span class="line">)</span><br><span class="line">validation_generator = val_datagen.flow_from_directory(</span><br><span class="line"> <span class="string">'./catdog/validation'</span>,</span><br><span class="line"> target_size=(IM_WIDTH, IM_HEIGHT),</span><br><span class="line"> batch_size=batch_size,</span><br><span class="line">)</span><br><span class="line">test_generator = test_datagen.flow_from_directory(</span><br><span class="line"> <span class="string">'./catdog/test'</span>,</span><br><span class="line"> target_size=(IM_WIDTH, IM_HEIGHT),</span><br><span class="line"> batch_size=<span class="number">512</span>,</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">setup_to_finetune</span><span class="params">(model)</span>:</span></span><br><span class="line"> <span class="string">"""Freeze the bottom NB_IV3_LAYERS and retrain the remaining top layers.</span></span><br><span class="line"><span class="string"> note: NB_IV3_LAYERS corresponds to the top 2 inception blocks in the inceptionv3 arch</span></span><br><span class="line"><span class="string"> Args:</span></span><br><span class="line"><span class="string"> model: keras model</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">for</span> layer <span class="keyword">in</span> model.layers[:NB_IV3_LAYERS_TO_FREEZE]:</span><br><span class="line"> layer.trainable = <span class="keyword">False</span></span><br><span class="line"> <span class="keyword">for</span> layer <span class="keyword">in</span> model.layers[NB_IV3_LAYERS_TO_FREEZE:]:</span><br><span class="line"> layer.trainable = <span class="keyword">True</span></span><br><span class="line"> model.compile(optimizer=SGD(lr=<span class="number">0.001</span>, momentum=<span class="number">0.5</span>), loss=<span class="string">'categorical_crossentropy'</span>, metrics=[<span class="string">'accuracy'</span>])</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">add_new_last_layer</span><span class="params">(base_model, nb_classes)</span>:</span></span><br><span class="line"> <span class="string">"""Add last layer to the convnet</span></span><br><span class="line"><span class="string"> Args:</span></span><br><span class="line"><span class="string"> base_model: keras model excluding top</span></span><br><span class="line"><span class="string"> nb_classes: # of classes</span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> new keras model with last layer</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> x = base_model.output</span><br><span class="line"> x = GlobalAveragePooling2D()(x)</span><br><span class="line"> x = Dense(FC_SIZE, activation=<span class="string">'relu'</span>)(x)</span><br><span class="line"> predictions = Dense(nb_classes, activation=<span class="string">'softmax'</span>)(x)</span><br><span class="line"> model = Model(input=base_model.input, output=predictions)</span><br><span class="line"> <span class="keyword">return</span> model</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">setup_to_transfer_learn</span><span class="params">(model, base_model)</span>:</span></span><br><span class="line"> <span class="string">"""Freeze all layers and compile the model"""</span></span><br><span class="line"> <span class="comment"># for layer in base_model.layers:</span></span><br><span class="line"> <span class="comment"># layer.trainable = False</span></span><br><span class="line"> <span class="keyword">for</span> layer <span class="keyword">in</span> model.layers[:<span class="number">-2</span>]:</span><br><span class="line"> layer.trainable = <span class="keyword">False</span></span><br><span class="line"> <span class="keyword">for</span> layer <span class="keyword">in</span> model.layers[<span class="number">-2</span>:]:</span><br><span class="line"> layer.trainable = <span class="keyword">True</span></span><br><span class="line"> model.compile(optimizer=<span class="string">'adam'</span>,</span><br><span class="line"> loss=<span class="string">'categorical_crossentropy'</span>,</span><br><span class="line"> metrics=[<span class="string">'accuracy'</span>])</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_nb_files</span><span class="params">(directory)</span>:</span></span><br><span class="line"> <span class="string">"""Get number of files by searching directory recursively"""</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> os.path.exists(directory):</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span></span><br><span class="line"> cnt = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span> r, dirs, files <span class="keyword">in</span> os.walk(directory):</span><br><span class="line"> <span class="keyword">for</span> dr <span class="keyword">in</span> dirs:</span><br><span class="line"> cnt += len(glob.glob(os.path.join(r, dr + <span class="string">"/*"</span>)))</span><br><span class="line"> <span class="keyword">return</span> cnt</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">train</span><span class="params">(train_dir, val_dir, epochs, output, batch_size)</span>:</span></span><br><span class="line"> <span class="string">"""Use transfer learning and fine-tuning to train a network on a new dataset"""</span></span><br><span class="line"> nb_train_samples = get_nb_files(train_dir)</span><br><span class="line"> nb_classes = len(glob.glob(train_dir + <span class="string">"/*"</span>))</span><br><span class="line"> nb_val_samples = get_nb_files(val_dir)</span><br><span class="line"> nb_epoch = int(epochs)</span><br><span class="line"> batch_size = int(batch_size)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># data prep</span></span><br><span class="line"> train_datagen = ImageDataGenerator(</span><br><span class="line"> preprocessing_function=preprocess_input,</span><br><span class="line"> rotation_range=<span class="number">30</span>,</span><br><span class="line"> width_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> height_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> shear_range=<span class="number">0.2</span>,</span><br><span class="line"> zoom_range=<span class="number">0.2</span>,</span><br><span class="line"> horizontal_flip=<span class="keyword">True</span></span><br><span class="line"> )</span><br><span class="line"> test_datagen = ImageDataGenerator(</span><br><span class="line"> preprocessing_function=preprocess_input,</span><br><span class="line"> rotation_range=<span class="number">30</span>,</span><br><span class="line"> width_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> height_shift_range=<span class="number">0.2</span>,</span><br><span class="line"> shear_range=<span class="number">0.2</span>,</span><br><span class="line"> zoom_range=<span class="number">0.2</span>,</span><br><span class="line"> horizontal_flip=<span class="keyword">True</span></span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> train_generator = train_datagen.flow_from_directory(</span><br><span class="line"> train_dir,</span><br><span class="line"> target_size=(IM_WIDTH, IM_HEIGHT),</span><br><span class="line"> batch_size=batch_size,</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> validation_generator = test_datagen.flow_from_directory(</span><br><span class="line"> val_dir,</span><br><span class="line"> target_size=(IM_WIDTH, IM_HEIGHT),</span><br><span class="line"> batch_size=batch_size,</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> <span class="comment"># setup model</span></span><br><span class="line"> base_model = InceptionV3(weights=<span class="string">'imagenet'</span>, include_top=<span class="keyword">False</span>) <span class="comment">#include_top=False excludes final FC layer</span></span><br><span class="line"> base_model.load_weights(WEIGHTS_PATH_NO_TOP)</span><br><span class="line"> model = add_new_last_layer(base_model, nb_classes)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># transfer learning</span></span><br><span class="line"> setup_to_transfer_learn(model, base_model)</span><br><span class="line"></span><br><span class="line"> earlystopper = EarlyStopping(patience=<span class="number">10</span>, verbose=<span class="number">1</span>)</span><br><span class="line"> checkpointer = ModelCheckpoint(<span class="string">'inceptionv3.h5'</span>, verbose=<span class="number">1</span>, save_best_only=<span class="keyword">True</span>)</span><br><span class="line"> history_tl = model.fit_generator(</span><br><span class="line"> train_generator,</span><br><span class="line"> nb_epoch=nb_epoch,</span><br><span class="line"> steps_per_epoch=<span class="number">100</span>,</span><br><span class="line"> validation_data=validation_generator,</span><br><span class="line"> validation_steps=<span class="number">10</span>,</span><br><span class="line"> callbacks=[earlystopper, checkpointer])</span><br><span class="line"> <span class="comment"># # fine-tuning</span></span><br><span class="line"> <span class="comment"># setup_to_finetune(model)</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># history_ft = model.fit_generator(</span></span><br><span class="line"> <span class="comment"># train_generator,</span></span><br><span class="line"> <span class="comment"># samples_per_epoch=nb_train_samples,</span></span><br><span class="line"> <span class="comment"># nb_epoch=nb_epoch,</span></span><br><span class="line"> <span class="comment"># validation_data=validation_generator,</span></span><br><span class="line"> <span class="comment"># nb_val_samples=nb_val_samples,</span></span><br><span class="line"> <span class="comment"># class_weight='auto')</span></span><br><span class="line"> print(model.evaluate_generator(test_generator))</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__==<span class="string">"__main__"</span>:</span><br><span class="line"> train(TRAIN_DIR, VAL_DIR, epochs, output, batch_size)</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="Dogs-vs-Cats"><a href="#Dogs-vs-Cats" class="headerlink" title="Dogs vs. Cats"></a
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>现代诗</title>
<link href="http://czx.im/2018/04/18/modern_poem/"/>
<id>http://czx.im/2018/04/18/modern_poem/</id>
<published>2018-04-18T02:53:03.000Z</published>
<updated>2018-05-02T06:28:17.814Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="从前慢-木心"><a href="#从前慢-木心" class="headerlink" title="从前慢 木心"></a>从前慢 木心</h1><p>记得早先少年时<br>大家诚诚恳恳<br>说一句 是一句</p><p>清早上火车站<br>长街黑暗无行人<br>卖豆浆的小店冒着热气</p><p>从前的日色变得慢<br>车，马，邮件都慢<br>一生只够爱一个人</p><p>从前的锁也好看<br>钥匙精美有样子<br>你锁了 人家就懂了</p><h1 id="波兰来客-北岛"><a href="#波兰来客-北岛" class="headerlink" title="波兰来客 北岛"></a>波兰来客 北岛</h1><p>那时我们有梦想，<br>关于文学，<br>关于爱情，<br>关于穿越世界的旅行。<br>如今我们深夜饮酒，<br>杯子碰到一起，<br>都是梦破碎的声音。</p><h1 id="Every-Day-You-Play-Pablo-Neruda"><a href="#Every-Day-You-Play-Pablo-Neruda" class="headerlink" title="Every Day You Play Pablo Neruda"></a>Every Day You Play Pablo Neruda</h1><p>Every day you play with the light of the universe.<br>Subtle visitor, you arrive in the flower and the water,<br>You are more than this white head that I hold tightly<br>as a bunch of flowers, every day, between my hands.</p><p>You are like nobody since I love you.<br>Let me spread you out among yellow garlands.<br>Who writes your name in letters of smoke among the stars of the south?<br>Oh let me remember you as you were before you existed.</p><p>Suddenly the wind howls and bangs at my shut window.<br>The sky is a net crammed with shadowy fish.<br>Here all the winds let go sooner or later, all of them.<br>The rain takes off her clothes.</p><p>The birds go by, fleeing.<br>The wind. The wind.<br>I alone can contend against the power of men.<br>The storm whirls dark leaves<br>and turns loose all the boats that were moored last night to the sky.</p><p>You are here. Oh, you do not run away.<br>You will answer me to the last cry.<br>Curl round me as though you were frightened.<br>Even so, a strange shadow once ran through your eyes.</p><p>Now, now too, little one, you bring me honeysuckle,<br>and even your breasts smell of it.<br>While the sad wind goes slaughtering butterflies<br>I love you, and my happiness bites the plum of your mouth.</p><p>How you must have suffered getting accustomed to me,<br>my savage, solitary soul, my name that sends them all running.<br>So many times we have seen the morning star burn, kissing our eyes,<br>and over our heads the grey light unwinds in turning fans.</p><p>My words rained over you, stroking you.<br>A long time I have loved the sunned mother-of-pearl of your body.<br>Until I even believe that you own the universe.<br>I will bring you happy flowers from the mountains, bluebells, dark hazels, and rustic baskets of kisses.<br>I want to do with you what spring does with the cherry trees.</p><h1 id="过程-林白"><a href="#过程-林白" class="headerlink" title="过程 林白"></a>过程 林白</h1><p>一月你还没有出现<br>二月你睡在隔壁<br>三月下起了大雨<br>四月里遍地蔷薇<br>五月我们对面坐着 犹如梦中 就这样六月到了 <br>六月里青草盛开 处处芬芳<br>七月，悲喜交加 麦浪翻滚连同草地 直到天涯<br>八月就是八月<br>八月我守口如瓶 八月里我是瓶中的水 你是青天的云<br>九月和十月 是两只眼睛，装满了大海<br>你在海上 我在海下<br>十一月尚未到来<br>透过它的窗口 我望见了十二月 十二月大雪弥漫</p><h1 id="门前-顾城"><a href="#门前-顾城" class="headerlink" title="门前 顾城"></a>门前 顾城</h1><p>我多么希望，有一个门口 <br>早晨，阳光照在草上 </p><p>我们站着 <br>扶着自己的门扇 <br>门很低，但太阳是明亮的 </p><p>草在结它的种子 <br>风在摇它的叶子 <br>我们站着，不说话 <br>就十分美好 </p><p>有门，不用开开 <br>是我们的，就十分美好 </p><p>早晨，黑夜还要流浪 <br>我们把六弦琴交给他 <br>我们不走了 </p><p>我们需要土地 <br>需要永不毁灭的土地 <br>我们要乘着它 <br>度过一生 </p><p>土地是粗糙的，有时狭隘 <br>然而，它有历史 <br>有一份天空，一份月亮 <br>一份露水和早晨 </p><p>我们爱土地 <br>我们站着 <br>用木鞋挖着泥土 <br>门也晒热了 <br>我们轻轻靠着，十分美好 </p><p>墙后的草 <br>不会再长大了<br>它只用指尖，触了触阳光</p><h1 id="我曾经爱过你-普希金"><a href="#我曾经爱过你-普希金" class="headerlink" title="我曾经爱过你 普希金"></a>我曾经爱过你 普希金</h1><p>我曾经爱过你：爱情，也许<br>在我的心灵里还没有完全消亡，<br>但愿它不会再打扰你，<br>我也不想再使你难过悲伤。<br>我曾经默默无语、毫无指望地爱过你，<br>我既忍受着羞怯，又忍受着嫉妒的折磨，<br>我曾经那样真诚、那样温柔地爱过你，<br>但愿上帝保佑你，<br>另一个人也会像我一样地爱你。</p><h1 id="借我-樊小纯（一说是木心先生本人）"><a href="#借我-樊小纯（一说是木心先生本人）" class="headerlink" title="借我 樊小纯（一说是木心先生本人）"></a>借我 樊小纯（一说是木心先生本人）</h1><p>借我一个暮年，<br>借我碎片，<br>借我瞻前与顾后，<br>借我执拗如少年。<br>借我后天长成的先天，<br>借我变如不曾改变。<br>借我素淡的世故和明白的愚，<br>借我可预知的脸。<br>借我悲怆的磊落，<br>借我温软的鲁莽和玩笑的庄严。<br>借我最初与最终的不敢，借我不言而喻的不见。<br>借我一场秋啊，可你说这已是冬天。</p><h1 id="When-You-Are-Old-William-Butler-Yeats"><a href="#When-You-Are-Old-William-Butler-Yeats" class="headerlink" title="When You Are Old William Butler Yeats"></a>When You Are Old William Butler Yeats</h1><p>When you are old and grey and full of sleep,<br>And nodding by the fire, take down this book,<br>And slowly read, and dream of the soft look<br>Your eyes had once, and of their shadows deep;</p><p>How many loved your moments of glad grace,<br>And loved your beauty with love false or true,<br>But one man loved the pilgrim soul in you,<br>And loved the sorrows of your changing face;</p><p>And bending down beside the glowing bars,<br>Murmur, a little sadly, how Love fled<br>And paced upon the mountains overhead<br>And hid his face amid a crowd of stars.</p><h1 id="夏天的太阳-海子"><a href="#夏天的太阳-海子" class="headerlink" title="夏天的太阳 海子"></a>夏天的太阳 海子</h1><p>夏天<br>如果这条街没有鞋匠</p><p>我就打赤脚<br>站到太阳下看太阳</p><p>我想到在白天出生的孩子<br>一定是出于故意</p><p>你来人间一趟<br>你要看看太阳</p><p>和你的心上人<br>一起走在街上</p><p>了解她<br>也要了解太阳</p><p>（一组健康的工人<br>正午抽着纸烟）</p><p>夏天的太阳<br>太阳</p><p>当年基督入世<br>他也在这阳光下长大</p><p>1985.1</p><h1 id="答案-顾城"><a href="#答案-顾城" class="headerlink" title="答案 顾城"></a>答案 顾城</h1><p>这是最美的季节<br>可以忘记梦想<br>到处都是花朵<br>满山阴影飘荡</p><p>这是最美的阴影<br>可以摇动阳光<br>轻轻走下山去<br>酒杯叮当作响</p><p>这是最美的酒杯<br>可以发出歌唱<br>放上花香捡回<br>四边都是太阳</p><p>这是最美的太阳<br>把花印在地上<br>谁要拾走影子<br>谁就拾走光芒</p><h1 id="Gift-Czeslaw-Milosz"><a href="#Gift-Czeslaw-Milosz" class="headerlink" title="Gift Czesław Miłosz"></a>Gift Czesław Miłosz</h1><p>A day so happy.<br>Fog lifted early. I worked in the garden.<br>Hummingbirds were stopping over the honeysuckle flowers.<br>There was no thing on earth I wanted to possess.<br>I knew no one worth my envying him.<br>Whatever evil I had suffered, I forgot.<br>To think that once I was the same man did not embarrass me.<br>In my body I felt no pain.<br>When straightening up, I saw blue sea and sails.</p><p>日子过得多么舒畅。<br>晨雾早早消散，我在院中劳动。<br>成群蜂鸟流连在金银花丛。<br>人世间我再也不需要别的事物。<br>没有任何人值得我羡慕。<br>遇到什么逆运，我都把它忘在一边。<br>想到往昔的日子，也不觉得羞惭。<br>我一身轻快，毫无痛苦。<br>昂首远望，唯见湛蓝大海上点点白帆。</p><h1 id="身体清单-李庄"><a href="#身体清单-李庄" class="headerlink" title="身体清单 李庄"></a>身体清单 李庄</h1><p>身体里的碳<br>可以制成九千支铅笔<br>赠给诗人<br>但每根铅笔必须配一块橡皮</p><p>身体里的磷<br>要制成两千根火柴<br>全部给盲者<br>让他点燃血中的火焰</p><p>身体里的脂肪<br>还能做八块肥皂<br>送给妓女<br>请她洗净骨头去做母亲</p><p>身体里的铁<br>只够打一枚钢钉<br>留给我飘泊一世的灵魂<br>就钉在爱人的心上 </p><h1 id="The-Sweetest-Little-Song-Leonard-Cohen"><a href="#The-Sweetest-Little-Song-Leonard-Cohen" class="headerlink" title="The Sweetest Little Song Leonard Cohen"></a>The Sweetest Little Song Leonard Cohen</h1><p>You go your way<br>I’ll go your way too</p><h1 id="我想和你虚度时光-李元胜"><a href="#我想和你虚度时光-李元胜" class="headerlink" title="我想和你虚度时光 李元胜"></a>我想和你虚度时光 李元胜</h1><p>我想和你虚度时光，比如低头看鱼<br>比如把茶杯留在桌子上，离开<br>浪费它们好看的阴影<br>我还想连落日一起浪费，比如散步<br>一直消磨到星光满天<br>我还要浪费风起的时候<br>坐在走廊发呆，直到你眼中乌云<br>全部被吹到窗外<br>我已经虚度了世界，它经过我<br>疲倦，又像从未被爱过<br>但是明天我还要这样，虚度<br>满目的花草，生活应该像它们一样美好<br>一样无意义，像被虚度的电影<br>那些绝望的爱和赴死<br>为我们带来短暂的沉默<br>我想和你互相浪费<br>一起虚度短的沉默，长的无意义<br>一起消磨精致而苍老的宇宙<br>比如靠在栏杆上，低头看水的镜子<br>直到所有被虚度的事物<br>在我们身后，长出薄薄的翅膀 </p><h1 id="青春-席慕容"><a href="#青春-席慕容" class="headerlink" title="青春 席慕容"></a>青春 席慕容</h1><p>所有的结局都已写好<br>所有的泪水都已启程<br>却忽然忘了是怎么样的一个开始<br>在那个古老的不再回来的夏日<br>无论我如何的去追索<br>年轻的你只如云影掠过<br>而你微笑的面容极浅极淡<br>逐渐隐没在日后的群岚<br>遂翻开那发黄的扉页<br>命运将它装订得极其拙劣<br>含着泪我一读再读<br>却不得不承认<br>青春是一本太仓促的书</p><h1 id="我是一个任性的孩子-顾城"><a href="#我是一个任性的孩子-顾城" class="headerlink" title="我是一个任性的孩子 顾城"></a>我是一个任性的孩子 顾城</h1><p>我希望<br>能在心爱的白纸上画画<br>画出笨拙的自由<br>画下一只永远不会<br>流泪的眼睛</p><p>我是一个任性的孩子<br>我想涂去一切不幸<br>我想在大地上<br>画满窗子<br>让所有习惯黑暗的眼睛<br>都习惯光明</p><h1 id="我和这个世界不熟-北岛"><a href="#我和这个世界不熟-北岛" class="headerlink" title="我和这个世界不熟 北岛"></a>我和这个世界不熟 北岛</h1><p>我和这个世界不熟。<br>这并非是我安静的原因。<br>我依旧有很多问题，<br>问南方，问故里，问希望，问距离。</p><p>我和这个世界不熟。<br>这并非是我绝望的原因。<br>我依旧有很多热情，<br>给分开，给死亡，给昨天，给安寂。</p><p>我和这个世界不熟。<br>这并非是我虚假的原因。<br>我依旧有很多真诚，<br>离不开，放不下，活下去，爱得起。</p><p>我和这个世界不熟。<br>这并非是我孤寂的原因。<br>我依旧有很多诉求，<br>需慰藉，待分享，惹心烦，告诉你。</p><p>我和这个世界不熟。<br>这并非是我冷漠的原因。<br>我依旧有很多动情，<br>为时间，为白云，为天黑，畏天命。</p><p>我和这个世界不熟。<br>这并非是我逃避的原因。<br>我依旧有很多憧憬，<br>对梦想，对记忆，对失败，对希冀。</p><p>我和这个世界不熟。<br>这并非是我卑微的原因。<br>我依旧有很多勇敢，<br>不信输，不信神，不信天，不信地。</p><p>我和这个世界不熟。<br>这并非是我失落的原因。<br>我依旧有很多高昂，<br>有存在，有价值，有独特，有意义。</p><p>我和这个世界不熟。<br>这并非是我迷茫的原因。<br>我依旧有很多方向，<br>往前走，回头望，会跳跃，会停息。</p><p>我和这个世界不熟。<br>这并非是我撕裂的原因。<br>我依旧有很多完整，<br>至少我要成全我自己。</p><p>我和这个世界不熟。<br>这并非是我选择死亡或者生存的原因。<br>我依旧有很多选择，<br>刚巧，这两者，都不是选项之一。</p><h1 id="眉目-木心"><a href="#眉目-木心" class="headerlink" title="眉目 木心"></a>眉目 木心</h1><p>你的眉目笑语使我病了一场<br>热势退尽，还我寂寞的健康<br>如若再晤见，感觉是远远的<br>像有人在地平线上走，走过<br>只剩地平线，早春的雾迷濛了<br>所幸的是你毕竟算不得美<br>美，我就病重，就难痊愈<br>你这点儿才貌只够我病十九天<br>第二十天你就粗糙难看起来<br>你一生的华彩乐段也就完了<br>别人怎会当你是什么宝贝呢<br>蔓草丛生，细雨如粉，鹧鸪幽啼<br>我将迁徙，卜居森林小丘之陬<br>静等那足够我爱的人物的到来</p><h1 id="今夜我可以写出最悲哀的诗行-聂鲁达"><a href="#今夜我可以写出最悲哀的诗行-聂鲁达" class="headerlink" title="今夜我可以写出最悲哀的诗行 聂鲁达"></a>今夜我可以写出最悲哀的诗行 聂鲁达</h1><p>今夜我可以写出最悲哀的诗行。</p><p>可以写出：比如：“夜繁星满布，<br>天蓝色，星星们，在远处，颤抖着。”</p><p>夜风在空中旋转并歌唱。</p><p>今夜我可以写出最悲哀的诗行。<br>我爱过她，有时她也爱过我。</p><p>在那些如今夜一般的夜里我拥过她入怀。<br>在无尽的天空下我吻过她那么多遍。</p><p>她爱过我，有时我也爱过她。<br>怎会不爱上她直勾勾的大眼睛。</p><p>今夜我可以写出最悲哀的诗行。<br>可以想到不是正拥着她。可以感到失去了她。</p><p>听到浩渺的夜，没有她更加浩渺。<br>而诗句落向灵魂如同向着牧草露水落下。</p><p>我的爱没能留住她又有什么关系。<br>夜繁星满布而她没和我一起。</p><p>这便是一切。远处有人在歌唱。在远处。<br>我的灵魂不甘于失去了她。</p><p>如同为了靠近她我的目光搜寻着她。<br>我的心搜寻着她，而她没和我一起。</p><p>把那些同样的树涂白的同样的夜。<br>而我们，那时的我们，现在不再是同样的了。</p><p>我不再爱她，肯定是，可我多么深地爱过她。<br>我的声音曾一直搜寻风去触她的耳。</p><p>属于别人。要属于别人了。就像之前属于我的吻。<br>她的声音，她明亮的身体。她无尽的眼睛。</p><p>我不再爱她，肯定是，可也许我仍爱着她。<br>爱情太短，而遗忘太过漫长。</p><p>因为在那些如今夜一般的夜里我拥过她入怀，<br>我的灵魂不甘于失去了她。</p><p>虽然这是她带给我的最后的痛苦，<br>而这些是我正为她写出的最后的诗行。</p><h1 id="在此我爱你-聂鲁达"><a href="#在此我爱你-聂鲁达" class="headerlink" title="在此我爱你 聂鲁达"></a>在此我爱你 聂鲁达</h1><p>在此我爱你。<br>风在幽暗的松林里解开自己。<br>月亮在游荡的水上发出磷光。<br>同样的日子相互追逐纠缠。</p><p>雾气散开成舞蹈的形体。<br>一只银色的海鸥从西天滑落。<br>有时一片帆。高高，高高在上的星星。</p><p>或者一条船的黑色十字。<br>孤独的。<br>有时清晨醒来，连我的灵魂也是湿的。<br>海远远地发声，又发声。<br>这是港口。<br>在此我爱你。</p><p>在此我爱你，而地平线徒劳地将你遮掩。<br>置身这些冰冷的东西中我依然爱你。<br>有时我的吻登上那些沉重的船只<br>由海上驶向无法到达的地方。</p><p>我看见自己如那些旧锚般被遗忘。<br>当黄昏靠岸，码头格外悲伤。<br>我的生命已倦，毫无成效的饥饿。<br>我爱我没有的东西。你如此遥远。<br>我的厌烦与缓慢的暮色搏斗着。<br>但夜来临，并开始对我歌唱。</p><p>月亮转动它梦的圆盘。<br>最大的那些星星藉你的眼睛望着我。<br>而因为我爱你，风中的松树<br>要用它们的针叶歌唱你的名。</p><h1 id="中药-冯唐"><a href="#中药-冯唐" class="headerlink" title="中药 冯唐"></a>中药 冯唐</h1><p>世间草木都美<br>人不是</p><p>中药很苦<br>你也是。</p><h1 id="印记-席慕蓉"><a href="#印记-席慕蓉" class="headerlink" title="印记 席慕蓉"></a>印记 席慕蓉</h1><p>不要因为也许会改变<br>就不肯说那句美丽的誓言<br>不要因为也许会分离<br>就不敢求一次倾心的相遇<br>总有一些什么<br>会留下来的吧<br>留下来作一件不灭的印记<br>好让那些<br>不相识的人也能知道<br>我曾经怎样深深地爱过你</p><h1 id="绝色-余光中"><a href="#绝色-余光中" class="headerlink" title="绝色 余光中"></a>绝色 余光中</h1><p>若逢新雪初霁，满月当空<br>下面平铺着皓影<br>上面流转着亮银<br>而你带笑地向我步来<br>月色与雪色之间<br>你是第三种绝色</p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="从前慢-木心"><a href="#从前慢-木心" class="headerlink" title="从前慢 木心"></a>从前慢 木心</h1><p>记得
</summary>
<category term="现代诗" scheme="http://czx.im/categories/%E7%8E%B0%E4%BB%A3%E8%AF%97/"/>
<category term="无病呻吟" scheme="http://czx.im/tags/%E6%97%A0%E7%97%85%E5%91%BB%E5%90%9F/"/>
</entry>
<entry>
<title>边缘检测</title>
<link href="http://czx.im/2018/03/23/edge-detection/"/>
<id>http://czx.im/2018/03/23/edge-detection/</id>
<published>2018-03-23T14:40:55.000Z</published>
<updated>2018-03-23T19:04:44.489Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script>
</summary>
</entry>
<entry>
<title>局部特征描述和匹配</title>
<link href="http://czx.im/2018/03/15/local-feature-description-and-matching/"/>
<id>http://czx.im/2018/03/15/local-feature-description-and-matching/</id>
<published>2018-03-15T09:46:56.000Z</published>
<updated>2018-03-25T18:31:33.466Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="SIFT"><a href="#SIFT" class="headerlink" title="SIFT"></a>SIFT</h1><p>这篇文章应该会很长。</p><h2 id="Introduction"><a href="#Introduction" class="headerlink" title="Introduction"></a>Introduction</h2><p>SIFT是由UBC（university of British Column）的教授David Lowe于1999年提出，并在2004年得以完善的一种检测图像关键点（key points , 或者称为图像的interest points(兴趣点) )，并对关键点提取其局部尺度不变特征的描绘子，可以采用这个描绘子进行用于对两幅相关的图像进行匹配（matching）。<br>算法目标：将图像内容用一组局部特征表示，该特征对于平移，旋转，缩放和其他成像参数不变。<br>SIFT特征的优点由如下几种：</p><ul><li>Locality：特征是局部的， 所以对于遮挡（occusion）和复杂的背景（clutter）的鲁棒性非常好，very robust.</li><li>Distinctiveness: 特征具有独一无二性。也就是说 individual features can be matched to a large database of objects.</li><li>Quantity: 即使是一个很小的物体（objects）, 也能够产生大量的SIFT features.。</li><li>Efficiency： SIFT接近于实时的性能， 效率很高。</li></ul><h2 id="Procedure"><a href="#Procedure" class="headerlink" title="Procedure"></a>Procedure</h2><ul><li>Scale-space extrema detection<br>这一步的目的是在尺度空间中选择选择潜在的满足尺度不变性和旋转不变性的关键点（或者称为兴趣点)。<br>和之前DoG生成尺度空间的过程相似，需要在不同尺寸下用不同的σ的DoG去生成尺度空间。<br>对于尺度空间大小的选择，也就是取多少个σ，实验证明3个的时候稳定的极值点最多。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-26/51678471.jpg" alt=""><br>接下来就可以求极值了，SIFT中的极值不是单一尺度下的极值，而是要求这个点是尺度空间中的极值。所以，如下图，为了利用这个三维空间中找到图像的中的兴趣点，我们需要找到这个尺度的3x3邻域的局部极值（由于这个尺度共有3幅图像，所以共需要比较绿色区域中心点附近3x3x3-1（即26个像素），看看这个这个中心点像素的的值是否大于周围的26个像素。如果是，我们就说这个中心点是局部极值。这个点就是我们的兴趣点。以此类推，找到所有的极值点（兴趣点）。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-26/90846760.jpg" alt=""></li><li>Keypoint localization<br>这一步的目的就是精确定位出特征关键点的位置, 涉及到剔除伪关键点。<br>并不是所有的极值点都是关键点。而且由于对于一幅图像，我们可能找到很多的极值点，所以我们需要剔除掉一些不是关键点的点（可能是边缘的点，因为DoG对于边缘有很强的响应）。<br>剔去的是低对比度的候选关键点</li><li>Orientation assignment<br>这一步的目的就是基于关键点的局部梯度方向，给每个相应的关键点分配方向。</li><li>Keypoint description<br>此步骤的目的就是对于每个关键点，用一个向量去描述每个关键点。</li></ul><h1 id="引用"><a href="#引用" class="headerlink" title="引用"></a>引用</h1><p><a href="https://www.jianshu.com/p/a2366be565cb" target="_blank" rel="noopener">https://www.jianshu.com/p/a2366be565cb</a></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="SIFT"><a href="#SIFT" class="headerlink" title="SIFT"></a>SIFT</h1><p>这篇文章应该会很长。</
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Computer Vision" scheme="http://czx.im/tags/Computer-Vision/"/>
<category term="Point matching" scheme="http://czx.im/tags/Point-matching/"/>
<category term="SIFT" scheme="http://czx.im/tags/SIFT/"/>
</entry>
<entry>
<title>特征点检测</title>
<link href="http://czx.im/2018/03/13/Interest-Point-Detection/"/>
<id>http://czx.im/2018/03/13/Interest-Point-Detection/</id>
<published>2018-03-13T15:44:57.000Z</published>
<updated>2018-03-16T22:18:47.550Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="特征点检测有啥用？"><a href="#特征点检测有啥用？" class="headerlink" title="特征点检测有啥用？"></a>特征点检测有啥用？</h1><p>特征点检测广泛应用到目标匹配、目标跟踪、三维重建等应用中，在进行目标建模时会对图像进行目标特征的提取，常用的有颜色、角点、特征点、轮廓、纹理等特征。<br>我们有这两张图片，想要把他们拼成一张，比如常用的全景照片。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/48393057.jpg" alt=""><br>这个过程一共有三步，首先检测出两张图片特征点，然后再把对应的特征点配对，最后进行图像拼接。因为我们不可能把两张图所有的对应点都找出来，这在复杂度上很不划算，而且也不好实现。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/78904585.jpg" alt=""><br>图像拼接后的结果：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/8374961.jpg" alt=""><br>此外特征点检测还可以用来进行视频中物体的跟踪，比如Lucas-Kanade光流法实现目标跟踪。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/10133593.jpg" alt=""></p><h1 id="Harris角点检测"><a href="#Harris角点检测" class="headerlink" title="Harris角点检测"></a>Harris角点检测</h1><p>首先，为什么选择角点进行检测？当然是因为它们有比较容易进行判断的特征。角点在至少两个方向上会有很大的梯度，在物体内部的点没有，而边缘的点一般在某个方向梯度为0。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/31193356.jpg" alt=""><br>Harris角点检测是特征点检测的基础，提出了应用邻近像素点灰度差值概念，从而进行判断是否为角点、边缘、平滑区域。Harris角点检测原理是利用移动的窗口在图像中计算灰度变化值，其中关键流程包括转化为灰度图像、计算差分图像、高斯平滑、计算局部极值、确认角点。<br>算法的核心是利用局部窗口在图像上进行移动判断灰度发生较大的变化。<br>了解了算法的思想就可以构建数学模型了。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/85797736.jpg" alt=""><br>这里的w是窗口函数，一般有两种选择，括号里第一项是平移后图像灰度，第二项是原图像灰度。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/58350395.jpg" alt=""><br>这里我们选取的窗口移动是非常小的变化，所以u，v都很小。这时候就对括号内第一项泰勒展开化简原式。结果如下：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/27468753.jpg" alt=""><br>这是一个二阶近似值，其中M可由原图上的点求梯度得到。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/9517756.jpg" alt=""><br>原图求梯度后得到的图像长这个模样：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/48282009.jpg" alt=""><br>从上面得到的式子可以看出，通过E来判断点的特征等价于用M来判断，而对于一个矩阵，首先想到求特征值。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/58418164.jpg" alt=""><br>然后有人聪明地发现了这样的关系：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/77263100.jpg" alt=""><br>于是聪明地提出了这样一个容易编程实现的函数：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/76949109.jpg" alt=""><br>R取决于M的特征值，对于角点|R|很大，平坦的区域|R|很小，边缘的R为负值。<br>算出R之后还需要进行非极大值抑制（NMS），主要是为了更精确的定位特征。<br>上代码，不自己实现了，直接用opencv。opencv中的Harris检测方法使用函数 cv2.cornerHarris()<br>参数如下：<br> img 输入图像，数据类型为float32<br> blockSize 角点检测当中的邻域值。<br> ksize 使用Sobel函数求偏导的窗口大小<br> k 角点检测参数，取值为0.04到0.06<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> cv2</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line">filename = <span class="string">'5.jpeg'</span></span><br><span class="line"></span><br><span class="line">img = cv2.imread(filename)</span><br><span class="line">gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)</span><br><span class="line">gray = np.float32(gray)</span><br><span class="line"><span class="comment">#图像转换为float32</span></span><br><span class="line">dst = cv2.cornerHarris(gray,<span class="number">5</span>,<span class="number">5</span>,<span class="number">0.06</span>)</span><br><span class="line"><span class="comment">#图像膨胀, 让角点看起来明显点</span></span><br><span class="line">dst = cv2.dilate(dst,<span class="keyword">None</span>)</span><br><span class="line"><span class="comment">#这里的打分值以大于0.01×dst中最大值为边界， 理论上阈值越低越精确</span></span><br><span class="line">img[dst><span class="number">0.01</span>*dst.max()]=[<span class="number">0</span>,<span class="number">0</span>,<span class="number">255</span>]<span class="comment">#角点位置用红色标记</span></span><br><span class="line"></span><br><span class="line">cv2.imwrite(<span class="string">'dst2.jpeg'</span>,img)</span><br></pre></td></tr></table></figure></p><p>看起来效果还不错。还试了像素更高的图片，会有很多冗余，猜测是因为噪点的缘故。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/992741.jpg" alt=""><br>opencv还提供了一个函数cv2.cornerSubPix()进行亚像素级别的角点检测，就是说精确度会更高。感觉不太好用。</p><h1 id="尺度空间"><a href="#尺度空间" class="headerlink" title="尺度空间"></a>尺度空间</h1><h2 id="What"><a href="#What" class="headerlink" title="What"></a>What</h2><p>自然界中进行度量或观测时有许多不同的表现形态。比如度量身高用“米”，观测微粒用“纳米”或者“微米”。更形象的例子比如Google地图，滑动鼠标轮可以改变观测地图的尺度，看到的地图绘制也不同；还有电影中的拉伸镜头等等……<br>简而言之，从人眼的角度，尺度越大图像越模糊。<br>关于尺度的意义，有一张好玩的图，在不同尺度下能看出不同的结果。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-17/81466677.jpg" alt=""><br><img src="http://p59louklr.bkt.clouddn.com/18-3-17/40117740.jpg" alt=""></p><h2 id="Why"><a href="#Why" class="headerlink" title="Why"></a>Why</h2><p>对尺度空间研究的意义在于用机器视觉系统分析未知场景时，计算机并不预先知道图像中物体的尺度。我们需要同时考虑图像在多尺度下的描述，获知感兴趣物体的最佳尺度。另外如果不同的尺度下都有同样的关键点，那么在不同的尺度的输入图像下就都可以检测出来关键点匹配，也就是尺度不变性。<br>图像的尺度空间表达就是图像在所有尺度下的描述。</p><h2 id="尺度空间与图像金字塔区别"><a href="#尺度空间与图像金字塔区别" class="headerlink" title="尺度空间与图像金字塔区别"></a>尺度空间与图像金字塔区别</h2><p>尺度空间是多尺度，图像金字塔是多分辨率。<br>尺度空间表达和图像金字塔表达之间的不同具体说来是：<br> （1）尺度空间表达是由不同高斯核平滑卷积得到，在所有尺度上有相同的分辨率；<br> （2）金字塔多分辨率表达每层分辨率减少固定比率。</p><h1 id="LoG-Laplacian-of-Gaussian-和DoG-Difference-of-Gaussian-算子"><a href="#LoG-Laplacian-of-Gaussian-和DoG-Difference-of-Gaussian-算子" class="headerlink" title="LoG(Laplacian of Gaussian)和DoG(Difference of Gaussian)算子"></a>LoG(Laplacian of Gaussian)和DoG(Difference of Gaussian)算子</h1><h2 id="高斯平滑滤波"><a href="#高斯平滑滤波" class="headerlink" title="高斯平滑滤波"></a>高斯平滑滤波</h2><p>高斯核是唯一可以产生多尺度空间的核（《Scale-space theory: A basic tool for analysing structures at different scales》）。一个图像的尺度空间L(x,y,σ) ,定义为原始图像I(x,y)与一个可变尺度的2维高斯函数G(x,y,σ)卷积运算。<br>这个算子用来去噪效果很不错的，当然要选择合适尺度的高斯滤波器，否则整个图像会变模糊。<br>说到去噪滤波器的话还有好几个别的：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-17/56023727.jpg" alt=""><br>这里面应该要属中值滤波器鲁棒性最好，效果也最好。<br>扯远了，二维空间高斯函数：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/68573368.jpg" alt=""><br>很显然这个函数是圆对称的，且卷积的结果使原始像素值有最大的权重，距离中心越远的相邻像素值权重也越小。高斯模糊另一个重要的性质就是线性可分，使用二维矩阵变换的高斯模糊可以通过在水平和竖直方向各进行一维高斯矩阵变换相加得到。</p><h2 id="LoG"><a href="#LoG" class="headerlink" title="LoG"></a>LoG</h2><p>Laplace算子通过对图像求取二阶导数的零交叉点(zero-cross)来进行边缘检测，因为在边缘处一阶导取到一个极大值，而二阶导则会与横轴相交。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/15861353.jpg" alt=""><br>由于微分运算对噪声比较敏感，可以先对图像进行高斯平滑滤波，再使用Laplace算子进行边缘检测，以降低噪声的影响。由此便形成了用于极值点检测的LoG算子。顾名思义，LoG就是对高斯函数作用一个拉普拉斯算子得到的算子：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/63233749.jpg" alt=""><br>草稿纸推了一下：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-15/17794003.jpg" alt=""><br>在OpenCV-Python中，Laplace算子的函数原型如下：<br>dst = cv2.Laplacian(src, ddepth[, dst[, ksize[, scale[, delta[, borderType]]]]])<br>参数作用分别如下：</p><ul><li>dst是滤波之后的图像。</li><li>第一个参数是需要处理的图像。</li><li>第二个参数是图像的深度，-1表示采用的是与原图像相同的深度。目标图像的深度必须大于等于原图像的深度。</li><li>ksize是算子的大小，必须为1、3、5、7。默认为1。</li><li>scale是缩放导数的比例常数，默认情况下没有伸缩系数。</li><li>delta是一个可选的增量，将会加到最终的dst中，同样，默认情况下没有额外的值加到dst中。</li><li>borderType是判断图像边界的模式。这个参数默认值为cv2.BORDER_DEFAULT。<br>Code:<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> cv2</span><br><span class="line"></span><br><span class="line">img = cv2.imread(<span class="string">"5.jpeg"</span>, <span class="number">0</span>)</span><br><span class="line"></span><br><span class="line">gray_lap = cv2.Laplacian(img,cv2.CV_16S,ksize = <span class="number">3</span>)</span><br><span class="line">dst = cv2.convertScaleAbs(gray_lap)</span><br><span class="line"></span><br><span class="line">cv2.imwrite(<span class="string">'laplacian.jpeg'</span>,dst)</span><br></pre></td></tr></table></figure></li></ul><p>效果如下：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-15/54197302.jpg" alt=""></p><h2 id="DoG"><a href="#DoG" class="headerlink" title="DoG"></a>DoG</h2><p>下面是LoG和DoG的公式：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/43747394.jpg" alt=""><br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/39765063.jpg" alt=""><br>LoG计算可以利用DoG来近似，其中差分是由两个高斯滤波与不同变量的卷积结果逼近求得。证明如下：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-15/49585039.jpg" alt=""><br>看到另一个好玩的近似：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-17/67261430.jpg" alt=""><br>从滤波器的角度来理解，高斯滤波器是低通滤波器，两个低通滤波器的差可以构成一个带通滤波器（LoG是带通滤波器）。<br>从函数图像上来看两种方法也是非常接近了。但是用差分的方法明显在计算上要简单很多。<br><img src="https://i.loli.net/2017/12/18/5a3741bdd1528.jpg" alt=""><br>DoG的效果如下：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-14/90871784.jpg" alt=""><br>当然DoG并不止是简单的一个近似，作为一个增强算法，DOG可以被用来增加边缘和其他细节的可见性，大部分的边缘算子使用增强高频信号的方法，但是因为随机噪声也是高频信号，很多锐化算子也增强了噪声。DOG算法去除的高频信号中通常包含了随机噪声，所以这种方法是最适合处理那些有高频噪声的图像。这个算法的一个主要缺点就是在调整图像对比度的过程中信息量会减少。<br>当它被用于图像增强时，DoG算法中两个高斯核的半径之比通常为4:1或5:1。当k设为1.6时，即为高斯拉普拉斯算子的近似。高斯拉普拉斯算子在多尺度多分辨率像片中，用于近似高斯拉普拉斯算子两个高斯核的确切大小决定了两个高斯模糊后的影像间的尺度。<br>DoG也被用于尺度不变特征变换中的斑点检测。事实上，DoG算法作为两个多元正态分布的差通常总额为零，把它和一个恒定信号进行卷积没有意义。当K约等于1.6时它很好的近似了高斯拉普拉斯变换，当K约等于5时又很好的近似了视网膜上神经节细胞的视野。它可以很好的作为一个实时斑点检测算子和尺度选择算子应用于递归程序。</p><h1 id="引用"><a href="#引用" class="headerlink" title="引用"></a>引用</h1><p><a href="http://www.cnblogs.com/polly333/p/5416172.html" target="_blank" rel="noopener">http://www.cnblogs.com/polly333/p/5416172.html</a><br><a href="http://blog.csdn.net/xiaowei_cqu/article/details/8067881" target="_blank" rel="noopener">http://blog.csdn.net/xiaowei_cqu/article/details/8067881</a><br><a href="https://senitco.github.io/2017/06/20/image-feature-LoG-DoG/" target="_blank" rel="noopener">https://senitco.github.io/2017/06/20/image-feature-LoG-DoG/</a><br><a href="http://skystarry.cn/index.php/2017/04/20/dog/" target="_blank" rel="noopener">http://skystarry.cn/index.php/2017/04/20/dog/</a></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="特征点检测有啥用？"><a href="#特征点检测有啥用？" class="headerlink" title="特征点检测有啥用？"></a>特征点检测有啥用？
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Computer Vision" scheme="http://czx.im/tags/Computer-Vision/"/>
<category term="Point detection" scheme="http://czx.im/tags/Point-detection/"/>
</entry>
<entry>
<title>Tensorflow学习笔记5(DNN与CIFAR10）</title>
<link href="http://czx.im/2018/03/10/Tensorflow5/"/>
<id>http://czx.im/2018/03/10/Tensorflow5/</id>
<published>2018-03-10T08:32:55.000Z</published>
<updated>2018-04-18T21:02:07.426Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="Alexnet"><a href="#Alexnet" class="headerlink" title="Alexnet"></a>Alexnet</h1><p>说好了不说神经网络的内容，但是Alexnet确实太经典太重要了，可以说这一波AI的崛起就是从这个模型开始的，还是得说两句。<br><a href="http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf" target="_blank" rel="noopener">ImageNet Classification with Deep Convolutional Neural Networks</a>是Hinton和他的学生Alex Krizhevsky在12年ImageNet Challenge使用的模型结构，刷新了Image Classification的几率，从此deep learning在Image这块开始一次次超过state-of-art，甚至于搭到打败人类的地步。<br>先看看Alexnet的创新点：</p><ul><li>Data Augmentation<br>最简单有效的提高结果，防止过拟合的方法。常用的方法有，几何特征方面：水平翻转、随机剪裁、平移变换；颜色纹理方面：颜色光照变换，如增加减少亮度，一些滤光算法等等之类的。</li><li>Dropout<br>Dropout是一种防止神经网络过拟合的方法。在每次训练中，按照一定比例抛弃一些神经元直到本次训练结束。这可以理解为每次训练一些不同的神经网络（因为不同的神经元组合会构成不同的网络）。最后通过多个网络结合的方式达到类似集成学习的效果。</li><li>LRN<br>使用ReLU f(x)=max(0,x)后，你会发现激活函数之后的值没有了tanh、sigmoid函数那样有一个值域区间，所以一般在ReLU之后会做一个normalization，LRU就是文中提出（这里不确定，应该是提出？）一种方法，在神经科学中有个概念叫“Lateral inhibition”，讲的是活跃的神经元对它周边神经元的影响。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-17/21670298.jpg" alt=""><br>不过后来有人说这种方法其实没有用，反正现在都用Batch normalisation来取代他了。</li><li><p>ReLU<br>之前的激活函数如sigmoid，tanh，都会在取值过大或者过小的时候梯度开始趋于0，这非常不利于网络的训练，会加剧梯度消失，而且使得网络的训练速度变慢。用RELU以后偏离的信息能更快更完整地从后向前传播到整个网络。简单说收敛速度变快了。不过这种方法也有一个缺点就是，Dead ReLU Problem（神经元坏死现象），就是那些在负数部分梯度为0的神经元不会被用到。<br>再看看结构图：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/38547884.jpg" alt=""><br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/51077672.jpg" alt=""><br>alexnet总共包括8层，其中前5层convolutional，后面3层是full-connected，文章里面说的是减少任何一个卷积结果会变得很差，下面我来具体讲讲每一层的构成：</p><p> 第一层卷积层 输入图像为227<em>227</em>3(paper上貌似有点问题224<em>224</em>3)的图像，使用了96个kernels（96,11,11,3），以4个pixel为一个单位来右移或者下移，能够产生5555个卷积后的矩形框值，然后进行response-normalized（其实是Local Response Normalized，后面我会讲下这里）和pooled之后，pool这一层好像caffe里面的alexnet和paper里面不太一样，alexnet里面采样了两个GPU，所以从图上面看第一层卷积层厚度有两部分，池化pool_size=(3,3),滑动步长为2个pixels，得到96个2727个feature。<br> 第二层卷积层使用256个（同样，分布在两个GPU上，每个128kernels（5<em>5</em>48）），做pad_size(2,2)的处理，以1个pixel为单位移动（感谢网友指出），能够产生27<em>27个卷积后的矩阵框，做LRN处理，然后pooled，池化以3</em>3矩形框，2个pixel为步长，得到256个13<em>13个features。<br> 第三层、第四层都没有LRN和pool，第五层只有pool，其中第三层使用384个kernels（3</em>3<em>384，pad_size=(1,1),得到384</em>15<em>15，kernel_size为（3，3),以1个pixel为步长，得到384</em>13<em>13）；第四层使用384个kernels（pad_size(1,1)得到384</em>15<em>15，核大小为（3，3）步长为1个pixel，得到384</em>13<em>13）；第五层使用256个kernels（pad_size(1,1)得到384</em>15<em>15，kernel_size(3,3)，得到256</em>13<em>13，pool_size(3，3）步长2个pixels，得到256</em>6*6）。<br> 全连接层： 前两层分别有4096个神经元，最后输出softmax为1000个（ImageNet），注意caffe图中全连接层中有relu、dropout、innerProduct。</p></li></ul><h1 id="Tensorflow简单实现"><a href="#Tensorflow简单实现" class="headerlink" title="Tensorflow简单实现"></a>Tensorflow简单实现</h1><p>用了一个简化版的Alexnet，也不太能叫Alexnet其实，没有用数据增强，去掉了LRN改用BN，六层卷积加上三层全连接。最后测试集上最好成绩是84%，不算理想。因为用keras有个教程能跑到94%，他用了一种新型初始化权重的方法：<a href="https://arxiv.org/abs/1511.06422" target="_blank" rel="noopener">All you need is a good init</a>。<br>在搭结构的时候也是遇到了一些坑，最大的坑在于LRN。感觉它好像根本不work，导致死掉了一大片神经元，之前LRN用的少没想到那么不靠谱，果断换成BN，立马解决问题。<br>测试这个网络用了cifar10的数据，老用mnist没什么意思。Cifar-10由60000张32*32的RGB彩色图片构成，共10个分类。50000张训练，10000张测试（交叉验证）。这个数据集最大的特点在于将识别迁移到了普适物体，而且应用于多分类（姊妹数据集Cifar-100达到100类，ILSVRC比赛则是1000类）。</p><h2 id="数据输入"><a href="#数据输入" class="headerlink" title="数据输入"></a>数据输入</h2><p>首先是数据读取部分。要先到<a href="https://www.cs.toronto.edu/~kriz/cifar.html" target="_blank" rel="noopener">cifar10官网</a>下载数据集，然后具体的读取方法网站上也说了，我这里又多写了一个generator，batch training的时候用。有一个要注意的地方是cifar10的label都是int，需要做一个one hot编码。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="keyword">import</span> pickle <span class="keyword">as</span> p</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">from</span> numpy.random <span class="keyword">import</span> randint</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">load_CIFAR_batch</span><span class="params">(filename)</span>:</span></span><br><span class="line"> <span class="string">""" 载入cifar数据集的一个batch """</span></span><br><span class="line"> <span class="keyword">with</span> open(filename, <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> datadict = p.load(f, encoding=<span class="string">'latin1'</span>)</span><br><span class="line"> X = datadict[<span class="string">'data'</span>]</span><br><span class="line"> Y = datadict[<span class="string">'labels'</span>]</span><br><span class="line"> X = X.reshape(<span class="number">10000</span>, <span class="number">3</span>, <span class="number">32</span>, <span class="number">32</span>).transpose(<span class="number">0</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">1</span>).astype(<span class="string">"float"</span>)</span><br><span class="line"> Y = np.array(Y)</span><br><span class="line"> <span class="keyword">return</span> X, Y</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">load_CIFAR10</span><span class="params">(ROOT)</span>:</span></span><br><span class="line"> <span class="string">""" 载入cifar全部数据 """</span></span><br><span class="line"> xs = []</span><br><span class="line"> ys = []</span><br><span class="line"> <span class="keyword">for</span> b <span class="keyword">in</span> range(<span class="number">1</span>, <span class="number">6</span>):</span><br><span class="line"> f = os.path.join(ROOT, <span class="string">'data_batch_%d'</span> % (b,))</span><br><span class="line"> X, Y = load_CIFAR_batch(f)</span><br><span class="line"> xs.append(X) <span class="comment">#将所有batch整合起来</span></span><br><span class="line"> ys.append(Y)</span><br><span class="line"> Xtr = np.concatenate(xs) <span class="comment">#使变成行向量,最终Xtr的尺寸为(50000,32,32,3)</span></span><br><span class="line"> Ytr = np.concatenate(ys)</span><br><span class="line"> <span class="keyword">del</span> X, Y</span><br><span class="line"> Xte, label = load_CIFAR_batch(os.path.join(ROOT, <span class="string">'test_batch'</span>))</span><br><span class="line"> test_label = np.zeros((len(label), <span class="number">10</span>))</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(len(label)):</span><br><span class="line"> test_label[i][label[i]] = <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> Xtr, Ytr, Xte, test_label</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">data_generator</span><span class="params">(x, y, batch_size)</span>:</span></span><br><span class="line"> count = <span class="number">0</span></span><br><span class="line"> loopcount = len(y) // batch_size</span><br><span class="line"> <span class="keyword">while</span> <span class="number">1</span>:</span><br><span class="line"> j = randint(<span class="number">0</span>, loopcount)</span><br><span class="line"> train_data = x[j*batch_size: (j+<span class="number">1</span>)*batch_size]</span><br><span class="line"> label = y[j*batch_size: (j+<span class="number">1</span>)*batch_size]</span><br><span class="line"> train_label = np.zeros((batch_size, <span class="number">10</span>))</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(batch_size):</span><br><span class="line"> train_label[i][label[i]] = <span class="number">1</span></span><br><span class="line"> <span class="keyword">yield</span> train_data, train_label</span><br><span class="line"> count += batch_size</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> <span class="comment"># 载入CIFAR-10数据集</span></span><br><span class="line"> cifar10_dir = <span class="string">'./cifar-10-batches-py/'</span></span><br><span class="line"> X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)</span><br><span class="line"> data_gen = data_generator(X_train,y_train,<span class="number">32</span>)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">30</span>):</span><br><span class="line"> a, b = next(data_gen)</span><br></pre></td></tr></table></figure></p><h2 id="模型"><a href="#模型" class="headerlink" title="模型"></a>模型</h2><p>接下来是训练的部分：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">import</span> cifar10</span><br><span class="line"></span><br><span class="line"><span class="comment"># # #读入mnist数据集</span></span><br><span class="line"><span class="comment"># mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)</span></span><br><span class="line"><span class="comment"># 载入CIFAR-10数据集</span></span><br><span class="line"><span class="comment">#label are one_hot encoded</span></span><br><span class="line">cifar10_dir = <span class="string">'./cifar-10-batches-py/'</span></span><br><span class="line">X_train, y_train, X_test, y_test = cifar10.load_CIFAR10(cifar10_dir)</span><br><span class="line"><span class="comment">#create data generator</span></span><br><span class="line">train_generator = cifar10.data_generator(X_train, y_train, <span class="number">256</span>)</span><br><span class="line">test_generator = cifar10.data_generator(X_test, y_test, <span class="number">1024</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">#Hyperparameters</span></span><br><span class="line">learning_rate = <span class="number">0.001</span></span><br><span class="line">batch_size = <span class="number">256</span></span><br><span class="line">dropout = <span class="number">0.5</span></span><br><span class="line">training_iters = <span class="number">100000</span></span><br><span class="line">display_step = <span class="number">50</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#params</span></span><br><span class="line">length = <span class="number">32</span></span><br><span class="line">width = <span class="number">32</span></span><br><span class="line">dimension = <span class="number">3</span></span><br><span class="line">n_classes = <span class="number">10</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#io</span></span><br><span class="line">x = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, length, width, dimension])</span><br><span class="line">y = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, n_classes])</span><br><span class="line">keep_prob = tf.placeholder(tf.float32)</span><br><span class="line"></span><br><span class="line"><span class="comment">#kernel size</span></span><br><span class="line">weights = {</span><br><span class="line"> <span class="string">'c1'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">3</span>,<span class="number">64</span>], stddev=<span class="number">0.001</span>, name=<span class="string">'wc1'</span>)),</span><br><span class="line"> <span class="string">'c2'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">64</span>,<span class="number">64</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc2'</span>)),</span><br><span class="line"> <span class="string">'c3'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">64</span>,<span class="number">128</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc3'</span>)),</span><br><span class="line"> <span class="string">'c4'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">128</span>,<span class="number">128</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'wc4'</span>)),</span><br><span class="line"> <span class="string">'c5'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">128</span>,<span class="number">256</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'wc5'</span>)),</span><br><span class="line"> <span class="string">'c6'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">256</span>,<span class="number">256</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'wc6'</span>)),</span><br><span class="line"> <span class="string">'f1'</span>: tf.Variable(tf.truncated_normal([<span class="number">4</span>*<span class="number">4</span>*<span class="number">256</span>, <span class="number">1024</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f2'</span>: tf.Variable(tf.truncated_normal([<span class="number">1024</span>, <span class="number">128</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'fc2'</span>)),</span><br><span class="line"> <span class="string">'f3'</span>: tf.Variable(tf.truncated_normal([<span class="number">128</span>, <span class="number">10</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'fc3'</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">biases = {</span><br><span class="line"> <span class="string">'c1'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">64</span>], name=<span class="string">'bc1'</span>)),</span><br><span class="line"> <span class="string">'c2'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">64</span>], name=<span class="string">'bc2'</span>)),</span><br><span class="line"> <span class="string">'c3'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">128</span>], name=<span class="string">'bc3'</span>)),</span><br><span class="line"> <span class="string">'c4'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">128</span>], name=<span class="string">'bc4'</span>)),</span><br><span class="line"> <span class="string">'c5'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">256</span>], name=<span class="string">'bc5'</span>)),</span><br><span class="line"> <span class="string">'c6'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">256</span>], name=<span class="string">'bc6'</span>)),</span><br><span class="line"> <span class="string">'f1'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">1024</span>], name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f2'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">128</span>], name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f3'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">10</span>], name=<span class="string">'fc1'</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">#funcs</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">conv2d</span><span class="params">(input, kernel, bias, name)</span>:</span></span><br><span class="line"> wx_plus_b = tf.nn.bias_add(tf.nn.conv2d(input, kernel, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>), bias)</span><br><span class="line"> batch_norm = bn_layer(wx_plus_b, <span class="keyword">True</span>, name)</span><br><span class="line"> <span class="keyword">return</span> tf.nn.relu(batch_norm, name=name)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">pooling</span><span class="params">(input, k, name)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.max_pool(input, ksize=[<span class="number">1</span>,k,k,<span class="number">1</span>], strides=[<span class="number">1</span>,k,k,<span class="number">1</span>], padding=<span class="string">'SAME'</span>, name=name)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">lrn</span><span class="params">(input, name, lsize=<span class="number">5</span>)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.lrn(input, lsize, bias=<span class="number">1.0</span>, alpha=<span class="number">0.001</span> / <span class="number">9.0</span>, beta=<span class="number">0.75</span>, name=name)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实现Batch Normalization</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bn_layer</span><span class="params">(x,is_training,name,moving_decay=<span class="number">0.9</span>,eps=<span class="number">1e-5</span>)</span>:</span></span><br><span class="line"> <span class="comment"># 获取输入维度并判断是否匹配卷积层(4)或者全连接层(2)</span></span><br><span class="line"> shape = x.shape</span><br><span class="line"> <span class="keyword">assert</span> len(shape) <span class="keyword">in</span> [<span class="number">2</span>,<span class="number">4</span>]</span><br><span class="line"></span><br><span class="line"> param_shape = shape[<span class="number">-1</span>]</span><br><span class="line"> <span class="keyword">with</span> tf.variable_scope(name):</span><br><span class="line"> <span class="comment"># 声明BN中唯一需要学习的两个参数，y=gamma*x+beta</span></span><br><span class="line"> gamma = tf.get_variable(<span class="string">'gamma'</span>,param_shape,initializer=tf.constant_initializer(<span class="number">1</span>))</span><br><span class="line"> beta = tf.get_variable(<span class="string">'beat'</span>, param_shape,initializer=tf.constant_initializer(<span class="number">0</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 计算当前整个batch的均值与方差</span></span><br><span class="line"> axes = list(range(len(shape)<span class="number">-1</span>))</span><br><span class="line"> batch_mean, batch_var = tf.nn.moments(x,axes,name=<span class="string">'moments'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 采用滑动平均更新均值与方差</span></span><br><span class="line"> ema = tf.train.ExponentialMovingAverage(moving_decay)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">mean_var_with_update</span><span class="params">()</span>:</span></span><br><span class="line"> ema_apply_op = ema.apply([batch_mean,batch_var])</span><br><span class="line"> <span class="keyword">with</span> tf.control_dependencies([ema_apply_op]):</span><br><span class="line"> <span class="keyword">return</span> tf.identity(batch_mean), tf.identity(batch_var)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 训练时，更新均值与方差，测试时使用之前最后一次保存的均值与方差</span></span><br><span class="line"> mean, var = tf.cond(tf.equal(is_training,<span class="keyword">True</span>),mean_var_with_update,</span><br><span class="line"> <span class="keyword">lambda</span>:(ema.average(batch_mean),ema.average(batch_var)))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 最后执行batch normalization</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.batch_normalization(x,mean,var,beta,gamma,eps)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fc_layer</span><span class="params">(input, weight, bias, name, batch_norm = True)</span>:</span></span><br><span class="line"> wx_plus_b = tf.matmul(input, weight) + bias</span><br><span class="line"> <span class="keyword">if</span> batch_norm:</span><br><span class="line"> wx_plus_b = bn_layer(wx_plus_b, <span class="keyword">True</span>, name)</span><br><span class="line"> <span class="keyword">return</span> tf.nn.relu(wx_plus_b, name=name)</span><br><span class="line"></span><br><span class="line"><span class="comment">#model</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alexnet</span><span class="params">(X, weights, biases, dropout)</span>:</span></span><br><span class="line"> <span class="comment"># X = tf.reshape(X, [-1, 28, 28, 1])</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">#conv1</span></span><br><span class="line"> conv1 = conv2d(X, weights[<span class="string">'c1'</span>], biases[<span class="string">'c1'</span>], <span class="string">'conv1'</span>)</span><br><span class="line"> <span class="comment"># pool1 = pooling(conv1, 2, 'pool1')</span></span><br><span class="line"> norm1 = tf.nn.dropout(conv1, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv2</span></span><br><span class="line"> conv2 = conv2d(norm1, weights[<span class="string">'c2'</span>], biases[<span class="string">'c2'</span>], <span class="string">'conv2'</span>)</span><br><span class="line"> pool2 = pooling(conv2, <span class="number">2</span>, <span class="string">'pool2'</span>)</span><br><span class="line"> norm2 = tf.nn.dropout(pool2, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv3</span></span><br><span class="line"> conv3 = conv2d(norm2, weights[<span class="string">'c3'</span>], biases[<span class="string">'c3'</span>], <span class="string">'conv3'</span>)</span><br><span class="line"> <span class="comment"># pool3 = pooling(conv3, 2, 'pool3')</span></span><br><span class="line"> norm3 = tf.nn.dropout(conv3, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv4</span></span><br><span class="line"> conv4 = conv2d(norm3, weights[<span class="string">'c4'</span>], biases[<span class="string">'c4'</span>], <span class="string">'conv4'</span>)</span><br><span class="line"> pool4 = pooling(conv4, <span class="number">2</span>, <span class="string">'pool4'</span>)</span><br><span class="line"> norm4 = tf.nn.dropout(pool4, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv5</span></span><br><span class="line"> conv5 = conv2d(norm4, weights[<span class="string">'c5'</span>], biases[<span class="string">'c5'</span>], <span class="string">'conv5'</span>)</span><br><span class="line"> <span class="comment"># pool5 = pooling(conv5, 2, 'pool5')</span></span><br><span class="line"> norm5 = tf.nn.dropout(conv5, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># conv6</span></span><br><span class="line"> conv6 = conv2d(norm5, weights[<span class="string">'c6'</span>], biases[<span class="string">'c6'</span>], <span class="string">'conv6'</span>)</span><br><span class="line"> pool6 = pooling(conv6, <span class="number">2</span>, <span class="string">'pool6'</span>)</span><br><span class="line"> norm6 = tf.nn.dropout(pool6, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment">#fc</span></span><br><span class="line"> fc1 = tf.reshape(norm6, [<span class="number">-1</span>, weights[<span class="string">'f1'</span>].get_shape().as_list()[<span class="number">0</span>]])</span><br><span class="line"> fc1 = fc_layer(fc1, weights[<span class="string">'f1'</span>], biases[<span class="string">'f1'</span>], <span class="string">'fc1'</span>)</span><br><span class="line"> fc2 = fc_layer(fc1, weights[<span class="string">'f2'</span>], biases[<span class="string">'f2'</span>], <span class="string">'fc2'</span>)</span><br><span class="line"> fc3 = fc_layer(fc2, weights[<span class="string">'f3'</span>], biases[<span class="string">'f3'</span>], <span class="string">'fc3'</span>, batch_norm=<span class="keyword">False</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> fc3</span><br><span class="line"></span><br><span class="line"><span class="comment">#construct model</span></span><br><span class="line">model = alexnet(x, weights, biases, keep_prob)</span><br><span class="line"></span><br><span class="line"><span class="comment">#learn</span></span><br><span class="line">loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=model))</span><br><span class="line">optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)</span><br><span class="line">correct_prediction = tf.equal(tf.argmax(model,<span class="number">1</span>), tf.argmax(y,<span class="number">1</span>))</span><br><span class="line">accuracy = tf.reduce_mean(tf.cast(correct_prediction, <span class="string">"float"</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment">#init</span></span><br><span class="line">init = tf.initialize_all_variables()</span><br><span class="line"></span><br><span class="line"><span class="comment">#run!</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"> step = <span class="number">0</span></span><br><span class="line"> <span class="comment"># Keep training until reach max iterations</span></span><br><span class="line"> <span class="keyword">while</span> step < training_iters:</span><br><span class="line"> batch_xs, batch_ys = next(train_generator)</span><br><span class="line"> <span class="comment"># get batch data</span></span><br><span class="line"> sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys, keep_prob: dropout})</span><br><span class="line"> <span class="keyword">if</span> step % display_step == <span class="number">0</span>:</span><br><span class="line"> <span class="comment"># Train acc</span></span><br><span class="line"> acc = sess.run(accuracy, feed_dict={x: batch_xs, y: batch_ys, keep_prob: <span class="number">1.</span>})</span><br><span class="line"> <span class="comment"># train loss</span></span><br><span class="line"> loss_val = sess.run(loss, feed_dict={x: batch_xs, y: batch_ys, keep_prob: <span class="number">1.</span>})</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Iter "</span> + str(step) + <span class="string">", Minibatch Loss= "</span> + <span class="string">"{:.6f}"</span>.format(loss_val) + <span class="string">", Training Accuracy= "</span> + <span class="string">"{:.5f}"</span>.format(acc))</span><br><span class="line"> <span class="comment">#Test acc</span></span><br><span class="line"> <span class="keyword">if</span> step % <span class="number">1000</span> == <span class="number">0</span>:</span><br><span class="line"> batch_x, batch_y = next(test_generator)</span><br><span class="line"> print(<span class="string">"Testing Accuracy:"</span>, sess.run(accuracy, feed_dict={x: batch_x, y: batch_y, keep_prob: <span class="number">1.</span>}))</span><br><span class="line"> step += <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Optimization Finished!"</span>)</span><br></pre></td></tr></table></figure></p><h1 id="VGG16-and-so-on…"><a href="#VGG16-and-so-on…" class="headerlink" title="VGG16 and so on…"></a>VGG16 and so on…</h1><p>至于VGG16以及inception，Googlenet，其实就是不断加深网络的层数，用了一些小trick，实现起来大同小异，就不多写了。<br>VGG16的实现如下：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#kernel size</span></span><br><span class="line">weights = {</span><br><span class="line"> <span class="string">'c11'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">3</span>,<span class="number">64</span>], stddev=<span class="number">0.001</span>, name=<span class="string">'wc11'</span>)),</span><br><span class="line"> <span class="string">'c12'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">64</span>,<span class="number">64</span>], stddev=<span class="number">0.001</span>, name=<span class="string">'wc12'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c21'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">64</span>,<span class="number">128</span>], stddev=<span class="number">0.001</span>, name=<span class="string">'wc21'</span>)),</span><br><span class="line"> <span class="string">'c22'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">128</span>,<span class="number">128</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc22'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c31'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">128</span>,<span class="number">256</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc31'</span>)),</span><br><span class="line"> <span class="string">'c32'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">256</span>,<span class="number">256</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc32'</span>)),</span><br><span class="line"> <span class="string">'c33'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">256</span>,<span class="number">256</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc33'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c41'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">256</span>,<span class="number">512</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc41'</span>)),</span><br><span class="line"> <span class="string">'c42'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">512</span>,<span class="number">512</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc42'</span>)),</span><br><span class="line"> <span class="string">'c43'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">512</span>,<span class="number">512</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'wc43'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c51'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">512</span>,<span class="number">512</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'wc51'</span>)),</span><br><span class="line"> <span class="string">'c52'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">512</span>,<span class="number">512</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'wc52'</span>)),</span><br><span class="line"> <span class="string">'c53'</span>: tf.Variable(tf.truncated_normal([<span class="number">3</span>,<span class="number">3</span>,<span class="number">512</span>,<span class="number">512</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'wc53'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'f1'</span>: tf.Variable(tf.truncated_normal([<span class="number">8</span>*<span class="number">7</span>*<span class="number">512</span>, <span class="number">2048</span>], stddev=<span class="number">0.01</span>, name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f2'</span>: tf.Variable(tf.truncated_normal([<span class="number">2048</span>, <span class="number">1024</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'fc2'</span>)),</span><br><span class="line"> <span class="string">'f3'</span>: tf.Variable(tf.truncated_normal([<span class="number">1024</span>, <span class="number">2</span>], stddev=<span class="number">0.1</span>, name=<span class="string">'fc3'</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">biases = {</span><br><span class="line"> <span class="string">'c11'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">64</span>], name=<span class="string">'bc11'</span>)),</span><br><span class="line"> <span class="string">'c12'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">64</span>], name=<span class="string">'bc12'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c21'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">128</span>], name=<span class="string">'bc21'</span>)),</span><br><span class="line"> <span class="string">'c22'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">128</span>], name=<span class="string">'bc22'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c31'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">256</span>], name=<span class="string">'bc31'</span>)),</span><br><span class="line"> <span class="string">'c32'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">256</span>], name=<span class="string">'bc32'</span>)),</span><br><span class="line"> <span class="string">'c33'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">256</span>], name=<span class="string">'bc32'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c41'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">512</span>], name=<span class="string">'bc41'</span>)),</span><br><span class="line"> <span class="string">'c42'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">512</span>], name=<span class="string">'bc42'</span>)),</span><br><span class="line"> <span class="string">'c43'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">512</span>], name=<span class="string">'bc42'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'c51'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">512</span>], name=<span class="string">'bc51'</span>)),</span><br><span class="line"> <span class="string">'c52'</span>: tf.Variable(tf.constant(<span class="number">0.01</span>, shape=[<span class="number">512</span>], name=<span class="string">'bc52'</span>)),</span><br><span class="line"> <span class="string">'c53'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">512</span>], name=<span class="string">'bc52'</span>)),</span><br><span class="line"></span><br><span class="line"> <span class="string">'f1'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">2048</span>], name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f2'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">1024</span>], name=<span class="string">'fc1'</span>)),</span><br><span class="line"> <span class="string">'f3'</span>: tf.Variable(tf.constant(<span class="number">0.1</span>, shape=[<span class="number">2</span>], name=<span class="string">'fc1'</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">#funcs</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">conv2d</span><span class="params">(input, kernel, bias, name)</span>:</span></span><br><span class="line"> wx_plus_b = tf.nn.bias_add(tf.nn.conv2d(input, kernel, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>), bias)</span><br><span class="line"> batch_norm = bn_layer(wx_plus_b, <span class="keyword">True</span>, name)</span><br><span class="line"> <span class="keyword">return</span> tf.nn.relu(batch_norm, name=name)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">pooling</span><span class="params">(input, k, name)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.max_pool(input, ksize=[<span class="number">1</span>,k,k,<span class="number">1</span>], strides=[<span class="number">1</span>,k,k,<span class="number">1</span>], padding=<span class="string">'SAME'</span>, name=name)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实现Batch Normalization</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bn_layer</span><span class="params">(x,is_training,name,moving_decay=<span class="number">0.9</span>,eps=<span class="number">1e-3</span>)</span>:</span></span><br><span class="line"> <span class="comment"># 获取输入维度并判断是否匹配卷积层(4)或者全连接层(2)</span></span><br><span class="line"> shape = x.shape</span><br><span class="line"> <span class="keyword">assert</span> len(shape) <span class="keyword">in</span> [<span class="number">2</span>,<span class="number">4</span>]</span><br><span class="line"></span><br><span class="line"> param_shape = shape[<span class="number">-1</span>]</span><br><span class="line"> <span class="keyword">with</span> tf.variable_scope(name):</span><br><span class="line"> <span class="comment"># 声明BN中唯一需要学习的两个参数，y=gamma*x+beta</span></span><br><span class="line"> gamma = tf.get_variable(<span class="string">'gamma'</span>,param_shape,initializer=tf.constant_initializer(<span class="number">1</span>))</span><br><span class="line"> beta = tf.get_variable(<span class="string">'beat'</span>, param_shape,initializer=tf.constant_initializer(<span class="number">0</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 计算当前整个batch的均值与方差</span></span><br><span class="line"> axes = list(range(len(shape)<span class="number">-1</span>))</span><br><span class="line"> batch_mean, batch_var = tf.nn.moments(x,axes,name=<span class="string">'moments'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 采用滑动平均更新均值与方差</span></span><br><span class="line"> ema = tf.train.ExponentialMovingAverage(moving_decay)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">mean_var_with_update</span><span class="params">()</span>:</span></span><br><span class="line"> ema_apply_op = ema.apply([batch_mean,batch_var])</span><br><span class="line"> <span class="keyword">with</span> tf.control_dependencies([ema_apply_op]):</span><br><span class="line"> <span class="keyword">return</span> tf.identity(batch_mean), tf.identity(batch_var)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 训练时，更新均值与方差，测试时使用之前最后一次保存的均值与方差</span></span><br><span class="line"> mean, var = tf.cond(tf.equal(is_training,<span class="keyword">True</span>),mean_var_with_update,</span><br><span class="line"> <span class="keyword">lambda</span>:(ema.average(batch_mean),ema.average(batch_var)))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 最后执行batch normalization</span></span><br><span class="line"> <span class="keyword">return</span> tf.nn.batch_normalization(x,mean,var,beta,gamma,eps)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fc_layer</span><span class="params">(input, weight, bias, name, batch_norm = True)</span>:</span></span><br><span class="line"> wx_plus_b = tf.matmul(input, weight) + bias</span><br><span class="line"> <span class="keyword">if</span> batch_norm:</span><br><span class="line"> wx_plus_b = bn_layer(wx_plus_b, <span class="keyword">True</span>, name)</span><br><span class="line"> <span class="keyword">return</span> tf.nn.sigmoid(wx_plus_b, name=name)</span><br><span class="line"></span><br><span class="line"><span class="comment">#model</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">vgg16</span><span class="params">(X, weights, biases, dropout)</span>:</span></span><br><span class="line"> <span class="comment"># block 1</span></span><br><span class="line"> conv1_1 = conv2d(X, weights[<span class="string">'c11'</span>], biases[<span class="string">'c11'</span>], <span class="string">'conv11'</span>)</span><br><span class="line"> conv1_2 = conv2d(conv1_1, weights[<span class="string">'c12'</span>], biases[<span class="string">'c12'</span>], <span class="string">'conv12'</span>)</span><br><span class="line"> pool1 = pooling(conv1_2, <span class="number">2</span>, <span class="string">'pool1'</span>)</span><br><span class="line"> drop1 = tf.nn.dropout(pool1, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># block 2</span></span><br><span class="line"> conv2_1 = conv2d(drop1, weights[<span class="string">'c21'</span>], biases[<span class="string">'c21'</span>], <span class="string">'conv21'</span>)</span><br><span class="line"> conv2_2 = conv2d(conv2_1, weights[<span class="string">'c22'</span>], biases[<span class="string">'c22'</span>], <span class="string">'conv22'</span>)</span><br><span class="line"> pool2 = pooling(conv2_2, <span class="number">2</span>, <span class="string">'pool2'</span>)</span><br><span class="line"> drop2 = tf.nn.dropout(pool2, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># block 3</span></span><br><span class="line"> conv3_1 = conv2d(drop2, weights[<span class="string">'c31'</span>], biases[<span class="string">'c31'</span>], <span class="string">'conv31'</span>)</span><br><span class="line"> conv3_2 = conv2d(conv3_1, weights[<span class="string">'c32'</span>], biases[<span class="string">'c32'</span>], <span class="string">'conv32'</span>)</span><br><span class="line"> conv3_3 = conv2d(conv3_2, weights[<span class="string">'c33'</span>], biases[<span class="string">'c33'</span>], <span class="string">'conv33'</span>)</span><br><span class="line"> pool3 = pooling(conv3_3, <span class="number">2</span>, <span class="string">'pool3'</span>)</span><br><span class="line"> drop3 = tf.nn.dropout(pool3, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># block 4</span></span><br><span class="line"> conv4_1 = conv2d(drop3, weights[<span class="string">'c41'</span>], biases[<span class="string">'c41'</span>], <span class="string">'conv41'</span>)</span><br><span class="line"> conv4_2 = conv2d(conv4_1, weights[<span class="string">'c42'</span>], biases[<span class="string">'c42'</span>], <span class="string">'conv42'</span>)</span><br><span class="line"> conv4_3 = conv2d(conv4_2, weights[<span class="string">'c43'</span>], biases[<span class="string">'c43'</span>], <span class="string">'conv43'</span>)</span><br><span class="line"> pool4 = pooling(conv4_3, <span class="number">2</span>, <span class="string">'pool4'</span>)</span><br><span class="line"> drop4 = tf.nn.dropout(pool4, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># block 5</span></span><br><span class="line"> conv5_1 = conv2d(drop4, weights[<span class="string">'c51'</span>], biases[<span class="string">'c51'</span>], <span class="string">'conv51'</span>)</span><br><span class="line"> conv5_2 = conv2d(conv5_1, weights[<span class="string">'c52'</span>], biases[<span class="string">'c52'</span>], <span class="string">'conv52'</span>)</span><br><span class="line"> conv5_3 = conv2d(conv5_2, weights[<span class="string">'c53'</span>], biases[<span class="string">'c53'</span>], <span class="string">'conv53'</span>)</span><br><span class="line"> pool5 = pooling(conv5_3, <span class="number">2</span>, <span class="string">'pool5'</span>)</span><br><span class="line"> drop5 = tf.nn.dropout(pool5, dropout)</span><br><span class="line"></span><br><span class="line"> <span class="comment">#fc</span></span><br><span class="line"> fc1 = tf.reshape(drop5, [<span class="number">-1</span>, weights[<span class="string">'f1'</span>].get_shape().as_list()[<span class="number">0</span>]])</span><br><span class="line"> fc1 = fc_layer(fc1, weights[<span class="string">'f1'</span>], biases[<span class="string">'f1'</span>], <span class="string">'fc1'</span>)</span><br><span class="line"> fc2 = fc_layer(fc1, weights[<span class="string">'f2'</span>], biases[<span class="string">'f2'</span>], <span class="string">'fc2'</span>)</span><br><span class="line"> fc3 = fc_layer(fc2, weights[<span class="string">'f3'</span>], biases[<span class="string">'f3'</span>], <span class="string">'fc3'</span>, batch_norm=<span class="keyword">False</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> fc3</span><br></pre></td></tr></table></figure></p><p>基本上和Alexnet一样的。</p><p>#Next<br>接下来写个fine tuning的。</p><h2 id="引用"><a href="#引用" class="headerlink" title="引用"></a>引用</h2><p><a href="https://zhuanlan.zhihu.com/p/30486789" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/30486789</a><br><a href="http://www.cnblogs.com/52machinelearning/p/5821591.html" target="_blank" rel="noopener">http://www.cnblogs.com/52machinelearning/p/5821591.html</a></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="Alexnet"><a href="#Alexnet" class="headerlink" title="Alexnet"></a>Alexnet</h1><p>
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>一些动人的诗词</title>
<link href="http://czx.im/2018/03/09/poem/"/>
<id>http://czx.im/2018/03/09/poem/</id>
<published>2018-03-09T10:26:10.000Z</published>
<updated>2018-04-22T13:24:35.327Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h1 id="江夏别宋之悌-李白"><a href="#江夏别宋之悌-李白" class="headerlink" title="江夏别宋之悌.李白"></a>江夏别宋之悌.李白</h1><p>平生不下泪，于此泣无穷。 </p><h1 id="浪淘沙-欧阳修"><a href="#浪淘沙-欧阳修" class="headerlink" title="浪淘沙.欧阳修"></a>浪淘沙.欧阳修</h1><p>把酒祝东风，且共从容，垂杨紫陌洛城东。总是当时携手处，游遍芳丛。<br>聚散苦匆匆，此恨无穷。今年花胜去年红。可惜明年花更好，知与谁同？</p><h1 id="无门关-慧开禅师"><a href="#无门关-慧开禅师" class="headerlink" title="无门关.慧开禅师"></a>无门关.慧开禅师</h1><p>春有百花秋有月，夏有凉风冬有雪。</p><h1 id="秋夜寄僧-欧阳詹"><a href="#秋夜寄僧-欧阳詹" class="headerlink" title="秋夜寄僧.欧阳詹"></a>秋夜寄僧.欧阳詹</h1><p>遥知是夜檀溪上，月照千峰为一人。</p><h1 id="浣溪沙·一向年光有限身-晏殊"><a href="#浣溪沙·一向年光有限身-晏殊" class="headerlink" title="浣溪沙·一向年光有限身.晏殊"></a>浣溪沙·一向年光有限身.晏殊</h1><p>一向年光有限身，等闲离别易销魂，酒筵歌席莫辞频。<br>满目山河空念远，落花风雨更伤春，不如怜取眼前人。</p><h1 id="云南某烟盒"><a href="#云南某烟盒" class="headerlink" title="云南某烟盒"></a>云南某烟盒</h1><p>与君初相见，犹如故人归。</p><h1 id="长恨歌-白居易"><a href="#长恨歌-白居易" class="headerlink" title="长恨歌.白居易"></a>长恨歌.白居易</h1><p>玉容寂寞泪阑干，梨花一枝春带雨。</p><h1 id="北大女生节"><a href="#北大女生节" class="headerlink" title="北大女生节"></a>北大女生节</h1><p>幸得识卿桃花面，从此阡陌多暖春。</p><h1 id="菩萨蛮其三-韦庄"><a href="#菩萨蛮其三-韦庄" class="headerlink" title="菩萨蛮其三.韦庄"></a>菩萨蛮其三.韦庄</h1><p>如今却忆江南乐，当时年少春衫薄。骑马倚斜桥，满楼红袖招。</p><h1 id="题龙阳县青草湖-唐温如"><a href="#题龙阳县青草湖-唐温如" class="headerlink" title="题龙阳县青草湖.唐温如"></a>题龙阳县青草湖.唐温如</h1><p>西风吹老洞庭波，一夜湘君白发多。<br>醉后不知天在水，满船清梦压星河。</p><h1 id="留侯论-苏轼"><a href="#留侯论-苏轼" class="headerlink" title="留侯论.苏轼"></a>留侯论.苏轼</h1><p>古人所谓豪杰之士者，必有过人之节。人情有所不能忍者，匹夫见辱，拔剑而起，挺身而斗，此不足为勇也。天下有大勇者，卒然临之而不惊。无故加之而不怒；此其所挟持者甚大，而其志甚远也。</p><h1 id="石家小鬼"><a href="#石家小鬼" class="headerlink" title="石家小鬼"></a>石家小鬼</h1><p>严冬思暖夏，梦回满繁花。</p><h1 id="青杏儿-赵秉文"><a href="#青杏儿-赵秉文" class="headerlink" title="青杏儿.赵秉文"></a>青杏儿.赵秉文</h1><p>风雨替花愁。风雨罢，花也应休。劝君莫惜花前醉，今年花谢，明年花谢，白了人头。<br>乘兴两三瓯。拣溪山好处追游。但教有酒身无事，有花也好，无花也好，选甚春秋。</p><h1 id="吴越王妃每岁春必归临安，王以书遗妃曰："><a href="#吴越王妃每岁春必归临安，王以书遗妃曰：" class="headerlink" title="吴越王妃每岁春必归临安，王以书遗妃曰："></a>吴越王妃每岁春必归临安，王以书遗妃曰：</h1><p>陌上花开，可缓缓归矣。</p><h1 id="鹧鸪天·-纳兰性德"><a href="#鹧鸪天·-纳兰性德" class="headerlink" title="鹧鸪天· 纳兰性德"></a>鹧鸪天· 纳兰性德</h1><p>别绪如丝睡不成，那堪孤枕梦边城。因听紫塞三更雨，却忆红楼半夜灯。<br>书郑重，恨分明，天将愁味酿多情。起来呵手封题处，偏到鸳鸯两字冰。</p><h1 id="鹧鸪天-晏几道"><a href="#鹧鸪天-晏几道" class="headerlink" title="鹧鸪天.晏几道"></a>鹧鸪天.晏几道</h1><p>彩袖殷勤捧玉钟，当年拚却醉颜红。舞低杨柳楼心月，歌尽桃花扇底风。<br>从别后，忆相逢，几回魂梦与君同。今宵剩把银釭照，犹恐相逢是梦中。</p><h1 id="国风·唐风·绸缪"><a href="#国风·唐风·绸缪" class="headerlink" title="国风·唐风·绸缪"></a>国风·唐风·绸缪</h1><p>绸缪束薪，三星在天。今夕何夕，见此良人？子兮子兮，如此良人何。</p><h1 id="唐多令·芦叶满汀洲-辛弃疾"><a href="#唐多令·芦叶满汀洲-辛弃疾" class="headerlink" title="唐多令·芦叶满汀洲.辛弃疾"></a>唐多令·芦叶满汀洲.辛弃疾</h1><p>欲买桂花同载酒，终不似，少年游。</p><h1 id="春山夜月-于良史"><a href="#春山夜月-于良史" class="headerlink" title="春山夜月.于良史"></a>春山夜月.于良史</h1><p>掬水月在手，弄花香满衣。</p><h1 id="菩萨蛮-温庭筠"><a href="#菩萨蛮-温庭筠" class="headerlink" title="菩萨蛮.温庭筠"></a>菩萨蛮.温庭筠</h1><p>小山重叠金明灭，鬓云欲度香腮雪。懒起画蛾眉，弄妆梳洗迟。<br>照花前后镜，花面交相映。新帖绣罗襦，双双金鹧鸪。</p><h1 id="汉江临眺-王维"><a href="#汉江临眺-王维" class="headerlink" title="汉江临眺.王维"></a>汉江临眺.王维</h1><p>江流天地外，山色有无中。</p><h1 id="春怨-刘方平"><a href="#春怨-刘方平" class="headerlink" title="春怨.刘方平"></a>春怨.刘方平</h1><p>寂寞空庭春欲晚，梨花满地不开门。</p><h1 id="鹧鸪天·元夕有所梦-姜夔"><a href="#鹧鸪天·元夕有所梦-姜夔" class="headerlink" title="鹧鸪天·元夕有所梦.姜夔"></a>鹧鸪天·元夕有所梦.姜夔</h1><p>肥水东流无尽期，当初不合种相思。梦中未比丹青见，暗里忽惊山鸟啼。<br>春未绿，鬓先丝，人间别久不成悲。谁教岁岁红莲夜，两处沉吟各自知。</p><h1 id="三五七言-李白"><a href="#三五七言-李白" class="headerlink" title="三五七言.李白"></a>三五七言.李白</h1><p>入我相思门，知我相思苦，长相思兮长相忆，短相思兮无穷极。</p><h1 id="采莲子-皇甫松"><a href="#采莲子-皇甫松" class="headerlink" title="采莲子.皇甫松"></a>采莲子.皇甫松</h1><p>无端隔水抛莲子，遥被人知半日羞。</p><h1 id="小重山-岳飞"><a href="#小重山-岳飞" class="headerlink" title="小重山.岳飞"></a>小重山.岳飞</h1><p>欲将心事付瑶琴。知音少，弦断有谁听？</p><h1 id="寄黄几复-黄庭坚"><a href="#寄黄几复-黄庭坚" class="headerlink" title="寄黄几复 黄庭坚"></a>寄黄几复 黄庭坚</h1><p>桃李春风一杯酒，江湖夜雨十年灯。</p><h1 id="浣溪沙-王国维"><a href="#浣溪沙-王国维" class="headerlink" title="浣溪沙.王国维"></a>浣溪沙.王国维</h1><p>试上高峰窥皓月，偶开天眼觑红尘。可怜身是眼中人。</p><h1 id="浣溪沙·苏轼"><a href="#浣溪沙·苏轼" class="headerlink" title="浣溪沙·苏轼"></a>浣溪沙·苏轼</h1><p>元丰七年十二月二十四日，从泗州刘倩叔游南山。<br>细雨斜风作晓寒，淡烟疏柳媚晴滩。入淮清洛渐漫漫。<br>雪沫乳花浮午盏，蓼茸蒿笋试春盘。人间有味是清欢。</p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h1 id="江夏别宋之悌-李白"><a href="#江夏别宋之悌-李白" class="headerlink" title="江夏别宋之悌.李白"></a>江夏别宋之悌.李白
</summary>
<category term="古诗词" scheme="http://czx.im/categories/%E5%8F%A4%E8%AF%97%E8%AF%8D/"/>
<category term="无病呻吟" scheme="http://czx.im/tags/%E6%97%A0%E7%97%85%E5%91%BB%E5%90%9F/"/>
</entry>
<entry>
<title>Tensorflow学习笔记4(Tensorboard与可视化）</title>
<link href="http://czx.im/2018/03/09/Tensorflow4/"/>
<id>http://czx.im/2018/03/09/Tensorflow4/</id>
<published>2018-03-09T08:32:55.000Z</published>
<updated>2018-03-09T23:17:41.903Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><p><div class="aplayer" data-id="3767733839" data-server="tencent" data-type="playlist" data-mode="random"></div></p><h2 id="Tensorboard的可视化过程"><a href="#Tensorboard的可视化过程" class="headerlink" title="Tensorboard的可视化过程"></a>Tensorboard的可视化过程</h2><p>（1）首先肯定是先建立一个graph,你想从这个graph中获取某些数据的信息<br>（2）确定要在graph中的哪些节点放置summary operations以记录信息<br>使用tf.summary.scalar记录标量<br>使用tf.summary.histogram记录数据的直方图<br>使用tf.summary.distribution记录数据的分布图<br>使用tf.summary.image记录图像数据<br>（3）operations并不会去真的执行计算，除非你告诉他们需要去run,或者它被其他的需要run的operation所依赖。而我们上一步创建的这些summary operations其实并不被其他节点依赖，因此，我们需要特地去运行所有的summary节点。但是呢，一份程序下来可能有超多这样的summary 节点，要手动一个一个去启动自然是及其繁琐的，因此我们可以使用tf.summary.merge_all去将所有summary节点合并成一个节点，只要运行这个节点，就能产生所有我们之前设置的summary data。<br>（4）使用tf.summary.FileWriter将运行后输出的数据都保存到本地磁盘中<br>（5）运行整个程序，并在终端输入运行tensorboard的指令（tensorboard –logdir logs所在文件夹）。执行成功应该能看到TensorBoard 1.6.0 at http://你的电脑名字:6006 (Press CTRL+C to quit)。之后打开web端可查看可视化的结果。</p><h2 id="举个栗子"><a href="#举个栗子" class="headerlink" title="举个栗子"></a>举个栗子</h2><p>用的还是上一次分类mnist的cnn，但是注意在变量的声明阶段要加上name_scope。声明变量时加的tf.summary.histogram，可以让我们在最后看到训练过程中变量统计直方图的变化。<br>加了一层卷积的目的是为了把特征减少到4个，用tf.summary.image看看卷积网络提取的特征长什么样子（结果也看不出来什么）。因为tf.summary.image只能存储1,3,4维特征的tensor，就是说tensor的第四个数必须是1,3,4之一。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> tensorflow.examples.tutorials.mnist <span class="keyword">import</span> input_data</span><br><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">weight_variable</span><span class="params">(shape, name)</span>:</span></span><br><span class="line"> initial = tf.truncated_normal(shape, stddev=<span class="number">0.1</span>, name=name)</span><br><span class="line"> <span class="keyword">return</span> tf.Variable(initial)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bias_variable</span><span class="params">(shape, name)</span>:</span></span><br><span class="line"> initial = tf.constant(<span class="number">0.1</span>, shape=shape, name=name)</span><br><span class="line"> <span class="keyword">return</span> tf.Variable(initial)</span><br><span class="line"><span class="comment"># 读入mnist数据集</span></span><br><span class="line">mnist = input_data.read_data_sets(<span class="string">"MNIST_data/"</span>, one_hot=<span class="keyword">True</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 变量声明</span></span><br><span class="line"><span class="comment"># define placeholder for inputs to network</span></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'inputs'</span>):</span><br><span class="line"> x = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, <span class="number">784</span>], name=<span class="string">'x_input'</span>) <span class="comment"># None 为了留给后面的batch_size</span></span><br><span class="line"> y_ = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, <span class="number">10</span>], name=<span class="string">'y_input'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'conv1'</span>):</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'weights'</span>):</span><br><span class="line"> W_conv1 = weight_variable([<span class="number">5</span>, <span class="number">5</span>, <span class="number">1</span>, <span class="number">32</span>], name=<span class="string">'W1'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'conv1/weights'</span>, W_conv1)</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'bias'</span>):</span><br><span class="line"> b_conv1 = bias_variable([<span class="number">32</span>], name=<span class="string">'B1'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'conv1/bias'</span>, b_conv1)</span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'conv2'</span>):</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'weights'</span>):</span><br><span class="line"> W_conv2 = weight_variable([<span class="number">5</span>, <span class="number">5</span>, <span class="number">32</span>, <span class="number">64</span>], name=<span class="string">'W2'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'conv2/weights'</span>, W_conv2)</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'bias'</span>):</span><br><span class="line"> b_conv2 = bias_variable([<span class="number">64</span>], name=<span class="string">'B2'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'conv2/bias'</span>, b_conv2)</span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'conv3'</span>):</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'weights'</span>):</span><br><span class="line"> W_conv3 = weight_variable([<span class="number">5</span>, <span class="number">5</span>, <span class="number">64</span>, <span class="number">4</span>], name=<span class="string">'W3'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'conv2/weights'</span>, W_conv3)</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'bias'</span>):</span><br><span class="line"> b_conv3 = bias_variable([<span class="number">4</span>], name=<span class="string">'B3'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'conv3/bias'</span>, b_conv3)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'fc1'</span>):</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'weights'</span>):</span><br><span class="line"> W_fc1 = weight_variable([<span class="number">7</span> * <span class="number">7</span> * <span class="number">4</span>, <span class="number">1024</span>], name=<span class="string">'w1'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'fc1/weights'</span>, W_fc1)</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'bias'</span>):</span><br><span class="line"> b_fc1 = bias_variable([<span class="number">1024</span>], name=<span class="string">'b1'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'fc1/weights'</span>, b_fc1)</span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'fc2'</span>):</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'weights'</span>):</span><br><span class="line"> W_fc2 = weight_variable([<span class="number">1024</span>, <span class="number">10</span>], name=<span class="string">'w2'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'fc2/weights'</span>, W_fc2)</span><br><span class="line"> <span class="keyword">with</span> tf.name_scope(<span class="string">'bias'</span>):</span><br><span class="line"> b_fc2 = bias_variable([<span class="number">10</span>], name=<span class="string">'b2'</span>)</span><br><span class="line"> tf.summary.histogram(<span class="string">'fc2/weights'</span>, b_fc2)</span><br></pre></td></tr></table></figure></p><p>接下来是计算图的部分。这里存的变量是两个image和两个scalar，其他的name_scope是为了之后在tensorboard可视化计算图用的。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 计算图</span></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'input_reshape'</span>):</span><br><span class="line"> x_image = tf.reshape(x, [<span class="number">-1</span>, <span class="number">28</span>, <span class="number">28</span>, <span class="number">1</span>])</span><br><span class="line"> tf.summary.image(<span class="string">'input'</span>, x_image, <span class="number">10</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'c1'</span>):</span><br><span class="line"> conv1 = tf.nn.relu(tf.nn.conv2d(x_image, W_conv1, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>) + b_conv1)</span><br><span class="line"> pool1 = tf.nn.max_pool(conv1, ksize=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], strides=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'c2'</span>):</span><br><span class="line"> conv2 = tf.nn.relu(tf.nn.conv2d(pool1, W_conv2, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>) + b_conv2)</span><br><span class="line"> pool2 = tf.nn.max_pool(conv2, ksize=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], strides=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'c3'</span>):</span><br><span class="line"> conv3 = tf.nn.relu(tf.nn.conv2d(pool2, W_conv3, strides=[<span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>], padding=<span class="string">'SAME'</span>) + b_conv3)</span><br><span class="line"> tf.summary.image(<span class="string">'third_conv'</span>, conv3, <span class="number">10</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'f1'</span>):</span><br><span class="line"> pool2_flat = tf.reshape(conv3, [<span class="number">-1</span>, <span class="number">7</span>*<span class="number">7</span>*<span class="number">4</span>])</span><br><span class="line"> fc1 = tf.nn.relu(tf.matmul(pool2_flat,W_fc1) + b_fc1)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'f2'</span>):</span><br><span class="line"> y = tf.nn.softmax(tf.matmul(fc1, W_fc2) + b_fc2)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'loss'</span>):</span><br><span class="line"> loss = -tf.reduce_sum(y_ * tf.log(y))</span><br><span class="line"> tf.summary.scalar(<span class="string">'loss'</span>, loss)</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'train'</span>):</span><br><span class="line"> train_step = tf.train.AdamOptimizer(<span class="number">1e-4</span>).minimize(loss)</span><br><span class="line"></span><br><span class="line">correct_prediction = tf.equal(tf.argmax(y, <span class="number">1</span>), tf.argmax(y_, <span class="number">1</span>))</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.name_scope(<span class="string">'accuracy'</span>):</span><br><span class="line"> accuracy = tf.reduce_mean(tf.cast(correct_prediction, <span class="string">'float'</span>))</span><br><span class="line"> tf.summary.scalar(<span class="string">'accuracy'</span>, accuracy)</span><br></pre></td></tr></table></figure></p><p>在训练过程中同时记录模型在训练集和测试集上的表现，所以用了两个tf.summary.FileWriter。其他的训练步骤和之前一样。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 初始化变量</span></span><br><span class="line">init = tf.global_variables_initializer()</span><br><span class="line"><span class="comment">#logs</span></span><br><span class="line">merged_summary_op = tf.summary.merge_all()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 开始训练</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"></span><br><span class="line"> train_summary_writer = tf.summary.FileWriter(<span class="string">'./logs'</span>, sess.graph)</span><br><span class="line"> test_summary_writer = tf.summary.FileWriter(<span class="string">'./logs'</span>, sess.graph)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1000</span>):</span><br><span class="line"> x_batch, y_batch = mnist.train.next_batch(<span class="number">50</span>)</span><br><span class="line"> sess.run(train_step, feed_dict={x: x_batch, y_: y_batch})</span><br><span class="line"> <span class="keyword">if</span> i % <span class="number">100</span> == <span class="number">0</span>:</span><br><span class="line"> print(sess.run(accuracy, feed_dict={x: x_batch, y_: y_batch}))</span><br><span class="line"> <span class="comment">#eval on train set</span></span><br><span class="line"> summary_str = sess.run(merged_summary_op, feed_dict={x: x_batch, y_: y_batch})</span><br><span class="line"> train_summary_writer.add_summary(summary_str, i)</span><br><span class="line"> <span class="comment">#eval on test set</span></span><br><span class="line"> summary_str = sess.run(merged_summary_op, feed_dict={x: mnist.test.images, y_: mnist.test.labels})</span><br><span class="line"> test_summary_writer.add_summary(summary_str, i)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"test accuracy %g"</span>%accuracy.eval(feed_dict={</span><br><span class="line"> x: mnist.test.images, y_: mnist.test.labels}))</span><br></pre></td></tr></table></figure></p><h2 id="Tensorboard-Web端"><a href="#Tensorboard-Web端" class="headerlink" title="Tensorboard Web端"></a>Tensorboard Web端</h2><ul><li><p>SCALAR<br>展示的是标量的信息，我程序中用tf.summary.scalars()定义的信息都会在这个窗口。<br>回顾本文程序中定义的标量有：准确率accuracy,dropout的保留率，隐藏层中的参数信息，已经交叉熵损失。这些都在SCLARS窗口下显示出来了。展示的是标量的信息，我程序中用tf.summary.scalars()定义的信息都会在这个窗口。<br>程序中定义的标量有：accuracy, loss。这些都在SCALARS窗口下显示出来了。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/16803152.jpg" alt=""><br>老实说要看这两个用tensorboard有点大材小用，有非常方便的解决方案。</p></li><li><p>IMAGES<br>在程序中我们设置了一处保存了图像信息，就是在转变了输入特征的shape，然后记录到了image中，于是在tensorflow中就会还原出原始的图片了：<br>原始图片28×28×1：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/10140397.jpg" alt=""><br>经过三层卷积后7<em>7</em>4, 这个已经看不出特征了，也许该想个更好的方法。或者这时候特征已经比较抽象：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/88391373.jpg" alt=""></p></li><li><p>GRAPHS<br>这里展示的是整个训练过程的计算图graph,从中我们可以清洗地看到整个程序的逻辑与过程。对于比较深或者结构比较复杂的神经网络，这个工具用来做可视化还是很方便的。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/79191402.jpg" alt=""><br>单击某个节点，可以查看属性，输入，输出等信息<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/21852739.jpg" alt=""></p></li><li><p>DISTRIBUTIONS<br>这里查看的是神经元输出的分布，有激活函数之前的分布，激活函数之后的分布等。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/33995949.jpg" alt=""></p></li><li><p>HISTOGRAMS<br>也可以看以上保存的变量的直方图。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-10/40620232.jpg" alt=""></p></li></ul><h2 id="引用"><a href="#引用" class="headerlink" title="引用"></a>引用</h2><p><a href="http://blog.csdn.net/u012052268/article/details/75394077" target="_blank" rel="noopener">http://blog.csdn.net/u012052268/article/details/75394077</a><br><a href="http://blog.csdn.net/sinat_33761963/article/details/62433234" target="_blank" rel="noopener">http://blog.csdn.net/sinat_33761963/article/details/62433234</a></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><p><div class="aplayer" data-id="3767733839" data-server="tencent" data-type="playlist" da
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>Tensorflow学习笔记3(mnist和cnn)</title>
<link href="http://czx.im/2018/03/09/Tensorflow3/"/>
<id>http://czx.im/2018/03/09/Tensorflow3/</id>
<published>2018-03-08T21:24:10.000Z</published>
<updated>2018-03-13T14:54:02.890Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h2 id="MNIST数据集"><a href="#MNIST数据集" class="headerlink" title="MNIST数据集"></a>MNIST数据集</h2><p>简而言之就是手写数字识别，在所有教程里面都会把这个拿来入门用。这里记录一下自己用fc和CNN在tensorflow上的代码。结果就不重要了，主要为了练手。</p><h2 id="读入数据"><a href="#读入数据" class="headerlink" title="读入数据"></a>读入数据</h2><p>直接从tf模块里读入mnist<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> tensorflow.examples.tutorials.mnist <span class="keyword">import</span> input_data</span><br><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="comment">#读入mnist数据集</span></span><br><span class="line">mnist = input_data.read_data_sets(<span class="string">"MNIST_data/"</span>, one_hot=<span class="keyword">True</span>)</span><br></pre></td></tr></table></figure></p><h2 id="一层全连接网络"><a href="#一层全连接网络" class="headerlink" title="一层全连接网络"></a>一层全连接网络</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#变量声明</span></span><br><span class="line">x = tf.placeholder(<span class="string">'float'</span>,[<span class="keyword">None</span>, <span class="number">784</span>]) <span class="comment">#None 为了留给后面的batch_size</span></span><br><span class="line">y = tf.placeholder(<span class="string">'float'</span>,[<span class="keyword">None</span>, <span class="number">10</span>])</span><br><span class="line">W = tf.Variable(tf.zeros([<span class="number">784</span>,<span class="number">10</span>]))</span><br><span class="line">b = tf.Variable(tf.zeros([<span class="number">10</span>]))</span><br><span class="line"><span class="comment">#计算图</span></span><br><span class="line">y = tf.nn.softmax(tf.matmul(x, W)+b)</span><br><span class="line">loss = -tf.reduce_sum(y_*tf.log(y))</span><br><span class="line">train_step = tf.train.AdamOptimizer(<span class="number">0.01</span>).minimize(loss)</span><br><span class="line">correct_prediction = tf.equal(tf.argmax(y,<span class="number">1</span>), tf.argmax(y_,<span class="number">1</span>))</span><br><span class="line">accuracy = tf.reduce_mean(correct_prediction, <span class="string">'float'</span>)</span><br><span class="line"><span class="comment">#初始化变量</span></span><br><span class="line">init = tf.global_variables_initializer()</span><br><span class="line"><span class="comment"># 开始训练</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1000</span>):</span><br><span class="line"> x_batch, y_batch = mnist.train.next_batch(<span class="number">50</span>)</span><br><span class="line"> sess.run(train_step, feed_dict={x: x_batch, y_: y_batch})</span><br><span class="line"> <span class="keyword">if</span> i % <span class="number">100</span> == <span class="number">0</span>:</span><br><span class="line"> print(sess.run(accuracy, feed_dict={x: x_batch, y_: y_batch}))</span><br><span class="line"><span class="comment">#测试</span></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"test accuracy %g"</span>%accuracy.eval(feed_dict={</span><br><span class="line"> x: mnist.test.images, y_: mnist.test.labels}))</span><br></pre></td></tr></table></figure><h2 id="卷积网络"><a href="#卷积网络" class="headerlink" title="卷积网络"></a>卷积网络</h2><p>原理和结构很简单，但是自己写还是容易错，写一遍就当复习了。也是有段时间没有写tf底层的结构了。<br>初始化变量，这里用了两层卷积加两层fc，最后softmax分类。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 变量声明</span></span><br><span class="line"><span class="comment">#输入输出占位变量</span></span><br><span class="line">x = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, <span class="number">784</span>]) <span class="comment"># None 为了留给后面的batch_size</span></span><br><span class="line">y_ = tf.placeholder(<span class="string">'float'</span>, [<span class="keyword">None</span>, <span class="number">10</span>])</span><br><span class="line"><span class="comment">#这里是两层卷积，都用5*5卷积核</span></span><br><span class="line">w_conv1 = tf.Variable(tf.truncated_normal([<span class="number">5</span>,<span class="number">5</span>,<span class="number">1</span>,<span class="number">32</span>], stddev=<span class="number">0.1</span>))</span><br><span class="line">b1 = tf.Variable(tf.constant(<span class="number">0.1</span>, shape = [<span class="number">32</span>]))</span><br><span class="line">w_conv2 = tf.Variable(tf.truncated_normal([<span class="number">5</span>,<span class="number">5</span>,<span class="number">32</span>,<span class="number">64</span>], stddev=<span class="number">0.1</span>))</span><br><span class="line">b2 = tf.Variable(tf.constant(<span class="number">0.1</span>, shape = [<span class="number">64</span>]))</span><br><span class="line"><span class="comment">#接两层全连接</span></span><br><span class="line">w_fc1 = tf.Variable(tf.truncated_normal([<span class="number">7</span>*<span class="number">7</span>*<span class="number">64</span>, <span class="number">1024</span>], stddev=<span class="number">0.1</span>))</span><br><span class="line">b_fc1 = tf.Variable(tf.constant(<span class="number">0.1</span>, shape = [<span class="number">1024</span>]))</span><br><span class="line">W_fc2 = tf.Variable(tf.truncated_normal([<span class="number">1024</span>, <span class="number">10</span>], stddev=<span class="number">0.1</span>))</span><br><span class="line">b_fc2 = tf.Variable(tf.constant(<span class="number">0.1</span>, shape = [<span class="number">10</span>]))</span><br><span class="line"><span class="comment">#引入dropout</span></span><br><span class="line">keep_prob = tf.placeholder(<span class="string">"float"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">#计算图</span></span><br><span class="line"><span class="comment">#先把输入reshape成一个二维变量</span></span><br><span class="line">x_image = tf.reshape(x, [<span class="number">-1</span>, <span class="number">28</span>, <span class="number">28</span>, <span class="number">1</span>])</span><br><span class="line"><span class="comment">#第一层卷积，用relu作activation</span></span><br><span class="line">conv1 = tf.nn.relu(tf.nn.conv2d(x_image, w_conv1, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>))</span><br><span class="line">pool1 = tf.nn.max_pool(conv1, ksize=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], strides=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>)</span><br><span class="line"><span class="comment">#第二层卷积</span></span><br><span class="line">conv2 = tf.nn.relu(tf.nn.conv2d(pool1, w_conv2, strides=[<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>))</span><br><span class="line">pool2 = tf.nn.max_pool(conv2, ksize=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], strides=[<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>], padding=<span class="string">'SAME'</span>)</span><br><span class="line"><span class="comment">#flatten后接到全连接层</span></span><br><span class="line">pool2_flat = tf.reshape(pool2, [<span class="number">-1</span>, <span class="number">7</span>*<span class="number">7</span>*<span class="number">64</span>])</span><br><span class="line">fc1 = tf.nn.relu(tf.matmul(pool2_flat,w_fc1) + b_fc1)</span><br><span class="line"><span class="comment">#最后用softmax分类</span></span><br><span class="line">y = tf.nn.softmax(tf.matmul(fc1, W_fc2) + b_fc2)</span><br><span class="line"><span class="comment">#同之前的网络</span></span><br><span class="line">loss = -tf.reduce_sum(y_ * tf.log(y))</span><br><span class="line">train_step = tf.train.AdamOptimizer(<span class="number">1e-4</span>).minimize(loss)</span><br><span class="line">correct_prediction = tf.equal(tf.argmax(y, <span class="number">1</span>), tf.argmax(y_, <span class="number">1</span>))</span><br><span class="line">accuracy = tf.reduce_mean(tf.cast(correct_prediction, <span class="string">'float'</span>))</span><br><span class="line"><span class="comment"># 初始化变量</span></span><br><span class="line">init = tf.global_variables_initializer()</span><br><span class="line"><span class="comment"># 开始训练</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">10000</span>):</span><br><span class="line"> x_batch, y_batch = mnist.train.next_batch(<span class="number">50</span>)</span><br><span class="line"> sess.run(train_step, feed_dict={x: x_batch, y_: y_batch})</span><br><span class="line"> <span class="keyword">if</span> i % <span class="number">100</span> == <span class="number">0</span>:</span><br><span class="line"> print(sess.run(accuracy, feed_dict={x: x_batch, y_: y_batch}))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"test accuracy %g"</span>%accuracy.eval(feed_dict={</span><br><span class="line"> x: mnist.test.images, y_: mnist.test.labels}))</span><br></pre></td></tr></table></figure></p><p>大功告成，tf还是要多写，不然容易漏东西。</p><div id="music163player"><br> <iframe frameborder="no" border="0" marginwidth="0" marginheight="0" width="280" height="86" src="//music.163.com/outchain/player?type=2&id=38358214&auto=0&height=66"><br> </iframe><br></div>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h2 id="MNIST数据集"><a href="#MNIST数据集" class="headerlink" title="MNIST数据集"></a>MNIST数据集</h2
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>Tensorflow学习笔记2(变量与计算图搭建)</title>
<link href="http://czx.im/2018/03/09/Tensorflow2/"/>
<id>http://czx.im/2018/03/09/Tensorflow2/</id>
<published>2018-03-08T16:50:02.000Z</published>
<updated>2018-03-19T07:51:32.822Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h2 id="常量、变量和占位符"><a href="#常量、变量和占位符" class="headerlink" title="常量、变量和占位符"></a>常量、变量和占位符</h2><p>在定义计算图之前要了解一下tf中变量保存的形式。TensorFlow 中最基本的单位是常量（Constant）、变量（Variable）和占位符（Placeholder）。<br>常量定义后值和维度不可变，变量定义后值可变而维度不可变。在神经网络中，变量一般可作为储存权重和其他信息的矩阵，而常量可作为储存超参数或其他结构信息的变量。在上面的计算图中，结点 1 和结点 2 都是定义的常量 tf.constant()。我们可以分别声明不同的常量（tf.constant()）和变量（tf.Variable()），其中 tf.float和tf.int分别声明了不同的浮点型和整数型数据。<br>通常常量用来保存计算图中可训练的数据，常量则是不可训练数据。有一点非常重要，如果一个变量没有声明是tf.Variable()，他很有可能被当成是一个常量来处理，就是说在训练过程中始终得不到优化。<br>占位符的特点是没有初始值，它只会分配必要的内存。在会话中，占位符可以使用 feed_dict 馈送数据。占位符一般用来读取训练或者预测所用的数据，并送入图中，以及接受最后计算得到的结果。feed_dict 是一个字典，在字典中需要给出每一个用到的占位符的取值。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line">a=tf.placeholder(<span class="string">'float32'</span>, [<span class="number">3</span>])</span><br><span class="line">b=tf.placeholder(<span class="string">'float32'</span>, [<span class="number">3</span>])</span><br><span class="line"></span><br><span class="line">c=tf.multiply(a, b)</span><br><span class="line"></span><br><span class="line">x1 = [np.random.random() <span class="keyword">for</span> _ <span class="keyword">in</span> range(<span class="number">3</span>)]</span><br><span class="line">x2 = [np.random.random() <span class="keyword">for</span> _ <span class="keyword">in</span> range(<span class="number">3</span>)]</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line">print(sess.run(c, feed_dict={a:x1,b:x2}))</span><br></pre></td></tr></table></figure></p><p>在初始化这些张量的时候一定要注意shape，整个计算图中只要有一个地方对不上，就会报错。</p><h2 id="计算图"><a href="#计算图" class="headerlink" title="计算图"></a>计算图</h2><p>TensorFlow 是一种采用数据流图（data flow graphs），用于数值计算的开源软件库。其中 Tensor 代表传递的数据为张量（多维数组），Flow 代表使用计算图进行运算。数据流图用「结点」（nodes）和「边」（edges）组成的有向图来描述数学运算。「结点」一般用来表示施加的数学操作，但也可以表示数据输入的起点和输出的终点，或者是读取/写入持久变量（persistent variable）的终点。边表示结点之间的输入/输出关系。这些数据边可以传送维度可动态调整的多维数据数组，即张量（tensor）。<br>简单来说，计算图定义了从input到output过程中张量所经过的计算和变化。在tensorflow中用的是静态图。静态的意思就是计算图的声明和执行是分开的，我们接下来可以看到使用tf之前，整个计算的流程就已经被设计好了。这份计算图一般会包含计算执行顺序和内存空间分配的策略，这些策略的制定一般是这个过程中最消耗时间的部分；执行阶段构建的图叫实体计算图，这个过程包括为参数和中间结果实际分配内存空间，并按照当前需求进行计算等，数据就在这张实体计算图中计算和传递。<br>而PyTorch用的则是动态图，声明和执行一起进行的。<br>上代码：<br>先声明需要拟合的数据，看一下他们的分布<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">import</span> numpy</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line">rng = numpy.random</span><br><span class="line"></span><br><span class="line"><span class="comment"># Parameters</span></span><br><span class="line">learning_rate = <span class="number">0.01</span></span><br><span class="line">training_epochs = <span class="number">2000</span></span><br><span class="line">display_step = <span class="number">50</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Training Data</span></span><br><span class="line">train_X = numpy.asarray([<span class="number">3.3</span>,<span class="number">4.4</span>,<span class="number">5.5</span>,<span class="number">6.71</span>,<span class="number">6.93</span>,<span class="number">4.168</span>,<span class="number">9.779</span>,<span class="number">6.182</span>,<span class="number">7.59</span>,<span class="number">2.167</span>,<span class="number">7.042</span>,<span class="number">10.791</span>,<span class="number">5.313</span>,<span class="number">7.997</span>,<span class="number">5.654</span>,<span class="number">9.27</span>,<span class="number">3.1</span>])</span><br><span class="line">train_Y = numpy.asarray([<span class="number">1.7</span>,<span class="number">2.76</span>,<span class="number">2.09</span>,<span class="number">3.19</span>,<span class="number">1.694</span>,<span class="number">1.573</span>,<span class="number">3.366</span>,<span class="number">2.596</span>,<span class="number">2.53</span>,<span class="number">1.221</span>,<span class="number">2.827</span>,<span class="number">3.465</span>,<span class="number">1.65</span>,<span class="number">2.904</span>,<span class="number">2.42</span>,<span class="number">2.94</span>,<span class="number">1.3</span>])</span><br><span class="line">n_samples = train_X.shape[<span class="number">0</span>]</span><br><span class="line"></span><br><span class="line">plt.plot(train_X, train_Y, <span class="string">'.'</span>)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure></p><p>这么一看大概就能画出来了。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-9/35305347.jpg" alt=""></p><p>接下来定义各种变量。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># tf Graph Input</span></span><br><span class="line">X = tf.placeholder(<span class="string">"float"</span>)</span><br><span class="line">Y = tf.placeholder(<span class="string">"float"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create Model</span></span><br><span class="line"><span class="comment"># Set model weights</span></span><br><span class="line">W = tf.Variable(rng.randn(), name=<span class="string">"weight"</span>)</span><br><span class="line">b = tf.Variable(rng.randn(), name=<span class="string">"bias"</span>)</span><br></pre></td></tr></table></figure></p><p>然后定义计算图和优化对象，这是最重要的一部分，定义张量计算的方式和传递的路径。这个模型很简单，只有一步，Y=X×W+b。<br>定义完计算图就能通过得到结果了，这时候可以定义损失函数了，因为是线性回归，用的就是均方差。优化器是普通的梯度下降。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Construct a linear model</span></span><br><span class="line">activation = tf.add(tf.multiply(X, W), b)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Minimize the squared errors</span></span><br><span class="line">cost = tf.reduce_sum(tf.pow(activation-Y, <span class="number">2</span>))/(<span class="number">2</span>*n_samples) <span class="comment">#L2 loss</span></span><br><span class="line">optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) <span class="comment">#Gradient descent</span></span><br></pre></td></tr></table></figure></p><p>在正式训练开始之前需要初始化所有变量，其实就是分配内存的过程。这在上面静态图的部分说过了。喂数据用的是feed_dict上面也写过了。就可以开始训练了。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Initializing the variables</span></span><br><span class="line">init = tf.global_variables_initializer()</span><br><span class="line"></span><br><span class="line"><span class="comment"># Launch the graph</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># Fit all training data</span></span><br><span class="line"> <span class="keyword">for</span> epoch <span class="keyword">in</span> range(training_epochs):</span><br><span class="line"> <span class="keyword">for</span> (x, y) <span class="keyword">in</span> zip(train_X, train_Y):</span><br><span class="line"> sess.run(optimizer, feed_dict={X: x, Y: y})</span><br><span class="line"></span><br><span class="line"> <span class="comment">#Display logs per epoch step</span></span><br><span class="line"> <span class="keyword">if</span> epoch % display_step == <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Epoch:"</span>, <span class="string">'%04d'</span> % (epoch+<span class="number">1</span>), <span class="string">"cost="</span>, \</span><br><span class="line"> <span class="string">"{:.9f}"</span>.format(sess.run(cost, feed_dict={X: train_X, Y:train_Y})), \</span><br><span class="line"> <span class="string">"W="</span>, sess.run(W), <span class="string">"b="</span>, sess.run(b))</span><br><span class="line"></span><br><span class="line"> print(<span class="string">"Optimization Finished!"</span>)</span><br><span class="line"> print(<span class="string">"cost="</span>, sess.run(cost, feed_dict={X: train_X, Y: train_Y}), \</span><br><span class="line"> <span class="string">"W="</span>, sess.run(W), <span class="string">"b="</span>, sess.run(b))</span><br><span class="line"></span><br><span class="line"> <span class="comment">#Graphic display</span></span><br><span class="line"> plt.plot(train_X, train_Y, <span class="string">'ro'</span>, label=<span class="string">'Original data'</span>)</span><br><span class="line"> plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label=<span class="string">'Fitted line'</span>)</span><br><span class="line"> plt.legend()</span><br><span class="line"> plt.show()</span><br></pre></td></tr></table></figure></p><p>最后的结果：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-9/15956860.jpg" alt=""></p><h2 id="模型的保存和读取"><a href="#模型的保存和读取" class="headerlink" title="模型的保存和读取"></a>模型的保存和读取</h2><p>用tf.train.Saver()创建一个Saver来管理模型中的所有变量，这样的话方便下一次继续使用训练好的模型。在训练阶段最后加上一句把模型存下来。存下来的模型是一个ckpt文件，里面包含了上面定义的所有变量，所以如果重新读取使用的话要定义好计算图。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Launch the graph</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> sess.run(init)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># Fit all training data</span></span><br><span class="line"> <span class="keyword">for</span> epoch <span class="keyword">in</span> range(training_epochs):</span><br><span class="line"> <span class="keyword">for</span> (x, y) <span class="keyword">in</span> zip(train_X, train_Y):</span><br><span class="line"> sess.run(optimizer, feed_dict={X: x, Y: y})</span><br><span class="line"></span><br><span class="line"> <span class="comment">#Display logs per epoch step</span></span><br><span class="line"> <span class="keyword">if</span> epoch % display_step == <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">print</span> (<span class="string">"Epoch:"</span>, <span class="string">'%04d'</span> % (epoch+<span class="number">1</span>), <span class="string">"cost="</span>, \</span><br><span class="line"> <span class="string">"{:.9f}"</span>.format(sess.run(cost, feed_dict={X: train_X, Y:train_Y})), \</span><br><span class="line"> <span class="string">"W="</span>, sess.run(W), <span class="string">"b="</span>, sess.run(b))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># Save the variables to disk.</span></span><br><span class="line"> save_path = saver.save(sess, <span class="string">"./tmp/model.ckpt"</span>)</span><br><span class="line"> print(<span class="string">"Model saved in file: "</span>, save_path)</span><br><span class="line"> print(<span class="string">"Optimization Finished!"</span>)</span><br></pre></td></tr></table></figure></p><p>然后可以直接读取训练好的模型进行预测，读取网上一些训练好的模型来做迁移学习也可以这么用。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#restore model</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line"> <span class="comment"># Restore variables from disk.</span></span><br><span class="line"> saver.restore(sess, <span class="string">"./tmp/model.ckpt"</span>)</span><br><span class="line"> print(<span class="string">"Model restored."</span>)</span><br><span class="line"> print(<span class="string">"cost="</span>, sess.run(cost, feed_dict={X: train_X, Y: train_Y}), \</span><br><span class="line"> <span class="string">"W="</span>, sess.run(W), <span class="string">"b="</span>, sess.run(b))</span><br><span class="line"></span><br><span class="line"> <span class="comment">#Graphic display</span></span><br><span class="line"> plt.plot(train_X, train_Y, <span class="string">'ro'</span>, label=<span class="string">'Original data'</span>)</span><br><span class="line"> plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label=<span class="string">'Fitted line'</span>)</span><br><span class="line"> plt.legend()</span><br><span class="line"> plt.show()</span><br></pre></td></tr></table></figure></p><h2 id="引用"><a href="#引用" class="headerlink" title="引用"></a>引用</h2><p><a href="https://zhuanlan.zhihu.com/p/29936078" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/29936078</a><br><a href="https://zhuanlan.zhihu.com/p/25216368" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/25216368</a><br><a href="https://github.com/aymericdamien/TensorFlow-Examples/" target="_blank" rel="noopener">https://github.com/aymericdamien/TensorFlow-Examples/</a><br><a href="http://www.tensorfly.cn/tfdoc/how_tos/variables.html" target="_blank" rel="noopener">http://www.tensorfly.cn/tfdoc/how_tos/variables.html</a></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h2 id="常量、变量和占位符"><a href="#常量、变量和占位符" class="headerlink" title="常量、变量和占位符"></a>常量、变量和占位符
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>Tensorflow学习笔记1(Tensor和基本计算）</title>
<link href="http://czx.im/2018/03/08/Tensorflow1/"/>
<id>http://czx.im/2018/03/08/Tensorflow1/</id>
<published>2018-03-08T08:24:12.000Z</published>
<updated>2018-03-09T09:03:45.936Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h2 id="Hello-World"><a href="#Hello-World" class="headerlink" title="Hello World"></a>Hello World</h2><p>创建一个session，它是tf对所有对象操作所需的基本单元。<br>Session 是 Tensorflow 为了控制，和输出文件的执行的语句。 运行 session.run() 可以获得你要得知的运算结果, 或者是你所要运算的部分。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line">hello = tf.constant(<span class="string">'Hello, TensorFlow!'</span>)</span><br><span class="line">sess = tf.Session()</span><br><span class="line"><span class="keyword">print</span> sess.run(hello)</span><br></pre></td></tr></table></figure></p><h2 id="Tensor"><a href="#Tensor" class="headerlink" title="Tensor"></a>Tensor</h2><p>Tensor实际上就是一个多维数组（multidimensional array），是TF的主要数据结构。它们在一个或多个由节点（nodes）和边（edges）组成的图（graphs）中流动。边代表的是tensors，节点代表的是对tensors的操作（operations，or Ops for short）。tensors在图中从一个节点流向另一个节点，每次经过一个节点都会接受一次操作。<br>在计算机视觉的应用中，tensor一般有(dimmension, height, width)三个维度，比如一张三维256*256的图片，tensor的shape就是（3,256,256)。有时候还包含第四维batch size，用于批量学习。</p><p>创建tensor的方法可以分为2种，一种是用TF自带的一些函数直接创建，例如：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf </span><br><span class="line"><span class="comment"># create a zero filled tensor </span></span><br><span class="line">tf.zeros([row_dim, col_dim]) </span><br><span class="line"><span class="comment"># create a one filled tensor </span></span><br><span class="line">tf.ones([row_dim, col_dim]) </span><br><span class="line"><span class="comment"># create a constant filled tensor </span></span><br><span class="line">tf.fill([row_dim, col_dim], <span class="number">42</span>) </span><br><span class="line"><span class="comment"># create a tensor out of an existing constant </span></span><br><span class="line">tf.constant([<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]) </span><br><span class="line"><span class="comment"># generate random numbers from a uniform distribution </span></span><br><span class="line">tf.random_uniform([row_dim, col_dim], minval=<span class="number">0</span>, maxval=<span class="number">1</span>) </span><br><span class="line"><span class="comment"># generate random numbers from a normal distribution </span></span><br><span class="line">tf.random_normal([row_dim, col_dim], mean=<span class="number">0.0</span>, stddev=<span class="number">1.0</span>)</span><br></pre></td></tr></table></figure></p><p>另一种是将Python对象（Numpy arrays， Python lists，Python scalars）转成tensor，例如：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">x_data = np.array([[<span class="number">1.</span>, <span class="number">2.</span>, <span class="number">3.</span>], [<span class="number">3.</span>, <span class="number">2.</span>, <span class="number">6.</span>]])</span><br><span class="line">tf.convert_to_tensor(x_data, dtype=tf.float32)</span><br></pre></td></tr></table></figure></p><p>Tensor对象有3个属性：<br> 一个名字，它用于键值对的存储，用于后续的检索：Const: 0<br> 一个形状描述， 描述数据的每一维度的元素个数：（6，3，7）<br> 数据类型，比如 float32<br>下面举一个简单的例子来说明tensor是如何在graph中流动的：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/39315037.jpg" alt=""><br>节点a接收了一个1-D tensor，该tensor从节点a流出后，分别流向了节点b和c，节点b执行的是prod操作（5*3），节点c执行的是sum操作（5+3）。当tensor从节点b流出时变成了15，从节点c流出时变成了8。此时，2个tensor又同时流入节点d，接受的是add操作（15+8），最后从节点d流出的tensor就是23。<br>用TF代码来创建上面的graph：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf </span><br><span class="line">a = tf.constant([<span class="number">5</span>, <span class="number">3</span>], name=<span class="string">'input_a'</span>) </span><br><span class="line">b = tf.reduce_prod(a, name=<span class="string">'prod_b'</span>) </span><br><span class="line">c = tf.reduce_sum(a, name=<span class="string">'sum_c'</span>) </span><br><span class="line">d = tf.add(b, c, name=<span class="string">'add_d'</span>)</span><br></pre></td></tr></table></figure></p><p>在上面的代码中，我们用不同的构造函数（constructor）定义了四个操作（对应图上4个节点）。例如，tf.constant()创建的操作实际上就是一个“二传手”：接受一个tensor（或者一个list对象，自动将其转换成tensor对象），然后传给与它直接相连的下一个node。tf.reduce_prod()和tf.reduce_sum()操作可以把input tensor对象中的所有值相乘或相加，然后传递给下一个直接相连的node。</p><h2 id="基本运算"><a href="#基本运算" class="headerlink" title="基本运算"></a>基本运算</h2><p>首先是int加减乘除。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line">a = tf.constant(<span class="number">2</span>)</span><br><span class="line">b = tf.constant(<span class="number">3</span>)</span><br><span class="line"><span class="comment">#一定要在一个session下进行tf的所有计算, 或者说所有动态操作。计算图搭建的过程倒是不需要</span></span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line">print(<span class="string">'a=%i,b=%i)%(sess.run(a),sess.run(b)))</span></span><br><span class="line"><span class="string">print('</span>Addtition:%i<span class="string">'%sess.run(a+b))</span></span><br><span class="line"><span class="string">print('</span>Addtition:%i<span class="string">'%sess.run(a+b))</span></span><br></pre></td></tr></table></figure></p><p>然后进行tensor的计算，tensor可以说是tf计算的最基本操作对象了，毕竟名字就叫tensorflow，对tensor的运算有多重要不言而喻。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line">row_dim = <span class="number">3</span></span><br><span class="line">col_dim=<span class="number">3</span></span><br><span class="line"><span class="comment"># generate random numbers from a uniform distribution</span></span><br><span class="line">a=tf.random_uniform([row_dim, col_dim], minval=<span class="number">0</span>, maxval=<span class="number">1</span>)</span><br><span class="line"><span class="comment"># generate random numbers from a normal distribution</span></span><br><span class="line">b=tf.random_normal([row_dim, col_dim], mean=<span class="number">0.0</span>, stddev=<span class="number">1.0</span>)</span><br><span class="line"><span class="keyword">with</span> tf.Session() <span class="keyword">as</span> sess:</span><br><span class="line">print((sess.run(a), sess.run(b)))</span><br><span class="line">print(sess.run(a*b))</span><br><span class="line">print(sess.run(a+b))</span><br></pre></td></tr></table></figure></p><p>数学运算可能会用到的一些函数：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">tf.add</span><br><span class="line">tf.sub</span><br><span class="line">tf.multiply <span class="comment">#这个和*一样的。如果shape一样对应元素相乘。不一样的时候broadcast。</span></span><br><span class="line">tf.div</span><br><span class="line">tf.mod</span><br><span class="line">tf.abs tf.neg</span><br><span class="line">tf.sign</span><br><span class="line">tf.inv</span><br><span class="line">tf.square</span><br><span class="line">tf.round</span><br><span class="line">tf.sqrt tf.pow tf.exp tf.log tf.maximum</span><br><span class="line">tf.minimum</span><br><span class="line">tf.cos tf.sin</span><br></pre></td></tr></table></figure></p><p>对二维张量的一些计算：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">tf.diag生成对角阵</span><br><span class="line">tf.transpose转置</span><br><span class="line">tf.matmul矩阵乘法 <span class="comment">#（2,3）×（3,2）->(2,2)</span></span><br><span class="line">tf.matrix_determinant计算行列式的值</span><br><span class="line">tf.matrix_inverse计算矩阵的逆</span><br></pre></td></tr></table></figure></p><h2 id="引用"><a href="#引用" class="headerlink" title="引用"></a>引用</h2><p><a href="https://zhuanlan.zhihu.com/p/30487008" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/30487008</a><br><a href="https://www.jianshu.com/p/6fec37e6ccc1" target="_blank" rel="noopener">https://www.jianshu.com/p/6fec37e6ccc1</a></p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h2 id="Hello-World"><a href="#Hello-World" class="headerlink" title="Hello World"></a>Hel
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
</entry>
<entry>
<title>Tensorflow学习笔记0（简介）</title>
<link href="http://czx.im/2018/03/07/Tensorflow0/"/>
<id>http://czx.im/2018/03/07/Tensorflow0/</id>
<published>2018-03-07T09:02:23.000Z</published>
<updated>2018-03-13T15:46:59.652Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><h2 id="Preface"><a href="#Preface" class="headerlink" title="Preface"></a>Preface</h2><p>之前其实也用了快一年Tensorflow, 但是实际上很多具体的实现并没有很清楚, 所以决定从头学一遍，边学变记吧。会参考一些别的教程，结合自己的理解，把学习的内容整理下来和大家分享。从基本的变量，计算图的搭建开始，后面实现一些常用的网络结构，Alexnet， VGG， Resnet之类。争取一个月之内全部更完。<br>这一系列笔记主要针对的是tf模型和api的复习，所以关于神经网络方面的内容就不深入了写了。</p><h2 id="笔记目录"><a href="#笔记目录" class="headerlink" title="笔记目录"></a>笔记目录</h2><ul><li>tf基本元素</li><li>变量与计算图搭建</li><li>mnist和cnn</li><li>tensorboard可视化</li><li>Alexnet</li><li>tf.slim</li><li>VGG</li><li>Resnet<div class="aplayer" data-id="864701453" data-server="netease" data-type="playlist" data-mode="random"></div></li></ul>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><h2 id="Preface"><a href="#Preface" class="headerlink" title="Preface"></a>Preface</h2><p>
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="Tensorflow" scheme="http://czx.im/tags/Tensorflow/"/>
<category term="Deeplearning" scheme="http://czx.im/tags/Deeplearning/"/>
</entry>
<entry>
<title>2018机器学习实习面试知识点整理</title>
<link href="http://czx.im/2018/03/06/2018intern_interview/"/>
<id>http://czx.im/2018/03/06/2018intern_interview/</id>
<published>2018-03-06T09:20:07.000Z</published>
<updated>2018-03-08T09:40:27.701Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><p>算是第一次正式的跟职场接触, 刚开始的时候也是一头雾水的状态。面得多了也慢慢有经验, 除了第一个头条挂了, 嗯挂的很惨。也是很想去的一家公司, 但是第一次哎, 多半会这样, 有种似曾相识的感觉。后面的除了阿里还没有回复, 一共拿了5个offer, 还算满意了吧。有些遗憾的是因为时间不合适, 没有去尝试一些大厂, 下次一定要补上。</p><h2 id="这里总结的问题一部分是我自己遇到的-还有一大部分是网上看了别人的面经摘抄或者总结出来的。还有一些有意思的问题-因为懒也没有来得及写上-以后会慢慢补充。很多地方可能考虑不周-或者有别的角度-欢迎大家补充或者指出。"><a href="#这里总结的问题一部分是我自己遇到的-还有一大部分是网上看了别人的面经摘抄或者总结出来的。还有一些有意思的问题-因为懒也没有来得及写上-以后会慢慢补充。很多地方可能考虑不周-或者有别的角度-欢迎大家补充或者指出。" class="headerlink" title="这里总结的问题一部分是我自己遇到的, 还有一大部分是网上看了别人的面经摘抄或者总结出来的。还有一些有意思的问题, 因为懒也没有来得及写上, 以后会慢慢补充。很多地方可能考虑不周, 或者有别的角度, 欢迎大家补充或者指出。"></a>这里总结的问题一部分是我自己遇到的, 还有一大部分是网上看了别人的面经摘抄或者总结出来的。还有一些有意思的问题, 因为懒也没有来得及写上, 以后会慢慢补充。很多地方可能考虑不周, 或者有别的角度, 欢迎大家补充或者指出。</h2><h1 id="面试准备"><a href="#面试准备" class="headerlink" title="面试准备:"></a>面试准备:</h1><h2 id="针对简历问题"><a href="#针对简历问题" class="headerlink" title="针对简历问题"></a>针对简历问题</h2><p>项目用到的特征</p><ul><li>Auto Color Correlogram (acc)<br>颜色自动相关图（color auto-correlogram），它仅仅考察具有相同颜色的像素间的空间关系</li><li>Color and Edge Directivity Descriptor (cedd)<br>CEDD结合了图像的颜色和纹理信息，生成一个144位的直方图。<br>1.1.RGB模型转换为HSV模型<br>H （Hue）代表色调，指通过物体传播或从物体射出的颜色，一般在使用中是由颜色名称来标识的。S （Saturation）代表饱和度，表示色调中灰色成分的比例，指颜色的纯度或强度。V （Value）代表亮度.</li><li>Color Layout (cl)<br>Color Layout Descriptor是mpeg-7多媒体内容标准描述中一种高效的局部颜色特征描述，在基于内容的图像检索(Content Based Image Retrieval (CBIR) ) 中表现出很好性能，拥有计算成本低，匹配计算速度快，识别准确率高等优点。<br>可以用来做以图搜图</li><li><p>Edge Histogram (eh)</p></li><li><p>Fuzzy Color and Texture Histogram (fcth)<br>模糊颜色和纹理直方图</p></li><li>Gabor (gabor)<br>Gabor 特征是一种可以用来描述图像纹理信息的特征，Gabor 滤波器的频率和方向与人类的视觉系统类似，特别适合于纹理表示与判别。<br>从图像处理的角度来看，Gabor特征有如下好处：<br>（1）Gabor核函数由于去掉了直流分量，因此对局部光照的变化不敏感，常常被用在要求对光照有适应性的场合；<br>（2）Gabor滤波结果可以反映图像不同尺度、不同方向上的灰度分布信息。一般说来，大尺度滤波可以反映全局性较强的信息，同时可以掩盖图像中噪声的影响；小尺度可以反映比较精细的局部结构，但容易受到噪声影响。</li><li><p>Joint descriptor joining CEDD and FCTH in one histogram (jcd)</p></li><li><p>Scalable Color (sc)<br>基于MPEG-7推荐的可伸缩颜色描述符</p></li><li>Tamura (tamura)<br>Tamura 纹理特征基于人类对纹理的视觉感知心理学研究，提出6种属性，即：粗糙度、对比度、方向度、线像度、规整度和粗略度。</li><li>Local Binary Patterns (lbp)</li></ul><p>LBP特征(从灰度图提取)<br>下图反应了某个像素的具体的LBP特征值的计算过程，需要注意的是，LBP值是按照顺时针方向组成的二进制数。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/65401032.jpg" alt=""><br>用公式来定义的话，如下所示：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/57695817.jpg" alt=""><br>其中代表3x3邻域的中心元素，它的像素值为ic，ip代表邻域内其他像素的值。s(x)是符号函数，定义如下：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/9308923.jpg" alt=""></p><p>HOG特征(从灰度图提取)<br>方向梯度直方图（Histogram of Oriented Gradient, HOG）<br>计算图像横坐标和纵坐标方向的梯度，并据此计算每个像素位置的梯度方向值；求导操作不仅能够捕获轮廓，人影和一些纹理信息，还能进一步弱化光照的影响。<br>图像中像素点(x,y)的梯度为：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/12921290.jpg" alt=""><br> 最常用的方法是：首先用[-1,0,1]梯度算子对原图像做卷积运算，得到x方向（水平方向，以向右为正方向）的梯度分量gradscalx，然后用[1,0,-1]T梯度算子对原图像做卷积运算，得到y方向（竖直方向，以向上为正方向）的梯度分量gradscaly。然后再用以上公式计算该像素点的梯度大小和方向。<br>（3）为每个细胞单元构建梯度方向直方图<br>（4）把细胞单元组合成大的块（block），块内归一化梯度直方图</p><p>ORB特征(提取特征点)<br>ORB：An Efficient Alternative to SIFT or SURF</p><h2 id="调参"><a href="#调参" class="headerlink" title="调参"></a>调参</h2><p>目的：偏差和方差的协调！！！bias，variance<br>原则：模型从简单到复杂，避免过拟合<br>传统方法：<br>1）树类分类器<br>n_estimators:子模型数量<br>max_features:每个子树能用到的最大特征。一般去总特征数开方或者log或者全部<br>max_depth:最大深度<br>min_sample_split:<br>min_sample_leaf:<br>神经网络：<br>learning rate: 1 0.1 0.01 0.001, 一般从1开始尝试。很少见learning rate大于10的。学习率一般要随着训练进行衰减。衰减系数一般是0.5。 衰减时机，可以是验证集准确率不再上升时，或固定训练多少个周期以后。<br>不过更建议使用自适应梯度的办法，例如adam,adadelta,rmsprop等，这些一般使用相关论文提供的默认值即可，可以避免再费劲调节学习率。对RNN来说，有个经验，如果RNN要处理的序列比较长，或者RNN层数比较多，那么learning rate一般小一些比较好，否则有可能出现结果不收敛，甚至Nan等问题。<br>网络层数： 先从1层开始。<br>每层结点数： 16 32 128，超过1000的情况比较少见。超过1W的从来没有见过。<br>batch size: 128上下开始。batch size值增加，的确能提高训练速度。但是有可能收敛结果变差。如果显存大小允许，可以考虑从一个比较大的值开始尝试。因为batch size太大，一般不会对结果有太大的影响，而batch size太小的话，结果有可能很差。<br>clip c(梯度裁剪): 限制最大梯度,其实是value = sqrt(w1^2+w2^2….),如果value超过了阈值，就算一个衰减系系数,让value的值等于阈值: 5,10,15<br>dropout： 0.5<br>L2正则：1.0，超过10的很少见。<br>正负样本比例： 这个是非常忽视，但是在很多分类问题上，又非常重要的参数。很多人往往习惯使用训练数据中默认的正负类别比例，当训练数据非常不平衡的时候，模型很有可能会偏向数目较大的类别，从而影响最终训练结果。除了尝试训练数据默认的正负类别比例之外，建议对数目较小的样本做过采样，例如进行复制。提高他们的比例，看看效果如何，这个对多分类问题同样适用。<br>在使用mini-batch方法进行训练的时候，尽量让一个batch内，各类别的比例平衡，这个在图像识别等多分类任务上非常重要。</p><h2 id="什么是boosting-tree（加法模型-前向分布）"><a href="#什么是boosting-tree（加法模型-前向分布）" class="headerlink" title="什么是boosting tree（加法模型+前向分布）"></a>什么是boosting tree（加法模型+前向分布）</h2><p><img src="http://p59louklr.bkt.clouddn.com/18-3-8/14429617.jpg" alt=""><br>Boosting方法： Boosting这其实思想相当的简单，大概是，对一份数据，建立M个模型（比如分类），一般这种模型比较简单，称为弱分类器(weak learner)每次分类都将上一次分错的数据权重提高一点再进行分类，这样最终得到的分类器在测试数据与训练数据上都可以得到比较好的成绩。<br>前向分布算法 实际上是一个贪心的算法，也就是在每一步求解弱分类器Φ(m)和其参数w(m)的时候不去修改之前已经求好的分类器和参数。<br>用当前模型的残差，即r=y-fm(x), 来计算下一颗树的参数。</p><h2 id="GBDT"><a href="#GBDT" class="headerlink" title="GBDT"></a>GBDT</h2><p>梯度提升<br>与普通boosting的区别是，利用的是损失函数的负梯度在当前函数的值来拟合回归树。<br>算法：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/37604864.jpg" alt=""></p><h2 id="XGB"><a href="#XGB" class="headerlink" title="XGB"></a>XGB</h2><p>传统GBDT在优化时只用到一阶导数信息，xgboost则对代价函数进行了二阶泰勒展开，同时用到了一阶和二阶导数。顺便提一下，xgboost工具支持自定义代价函数，只要函数可一阶和二阶求导。<br>xgboost在代价函数里加入了正则项，用于控制模型的复杂度。正则项里包含了树的叶子节点个数、每个叶子节点上输出的score的L2模的平方和。从Bias-variance tradeoff角度来讲，正则项降低了模型的variance，使学习出来的模型更加简单，防止过拟合，这也是xgboost优于传统GBDT的一个特性。<br>xgboost工具支持并行。xgboost的并行是在特征粒度上的。我们知道，决策树的学习最耗时的一个步骤就是对特征的值进行排序（因为要确定最佳分割点），xgboost在训练之前，预先对数据进行了排序，然后保存为block结构，后面的迭代中重复地使用这个结构，大大减小计算量。这个block结构也使得并行成为了可能，在进行节点的分裂时，需要计算每个特征的增益，最终选增益最大的那个特征去做分裂，那么各个特征的增益计算就可以开多线程进行。</p><h2 id="L1和L2正则为什么可以减弱overfitting"><a href="#L1和L2正则为什么可以减弱overfitting" class="headerlink" title="L1和L2正则为什么可以减弱overfitting"></a>L1和L2正则为什么可以减弱overfitting</h2><p>欠拟合(underfitting)，或者叫作叫做高偏差(bias).过拟合(overfitting)，也叫高方差(variance).<br>越简单的模型泛化能力越强。<br>对高阶项进行一定的惩罚，避免模型在数据量不够的时候过于复杂。<br>Ps：防止过拟合的其他方法，early stopping、数据集扩增（Data augmentation），dropout<br>L1和L2正则有什么区别<br>L1范数和L0范数可以实现稀疏（使没用的特征为0），L1因具有比L0更好的优化求解特性而被广泛应用。L1使损失函数可导。<br>L2范数不但可以防止过拟合，还可以让我们的优化求解变得稳定和快速。<br>L1会趋向于产生少量的特征，而其他的特征都是0，而L2会选择更多的特征，这些特征都会接近于0。Lasso在特征选择时候非常有用，而Ridge就只是一种规则化而已。<br>为什么L1正则可以实现参数稀疏，而L2正则不可以？答了：L1正则因为是绝对值形式，很多系数被压缩为0,。而L2正则是很多系数被压迫到接近于0，而不是0<br>为什么L1很多系数可以被压缩为0，L2是被压缩至接近于0？答了：图像上，L1正则是正方形，L2正则是圆形。L1正则的往往取到正方形顶点，即有很多参数为0.L2正则往往去不到圆形和参数线的交点，即很多分量被压缩到接近于0</p><h2 id="怎么理解dropout"><a href="#怎么理解dropout" class="headerlink" title="怎么理解dropout"></a>怎么理解dropout</h2><p>在每次训练的时候使用dropout，每个神经元有一定的概率被移除，这样可以使得一个神经元的训练不依赖于另外一个神经元，同样也就使得特征之间的协同作用被减弱。Hinton认为，过拟合可以通过阻止某些特征的协同作用来缓解。增加鲁棒性。<br>也可以理解为相当于在训练不同的网络，最后投票来决定结果。</p><h2 id="KNN和Lr有什么本质区别"><a href="#KNN和Lr有什么本质区别" class="headerlink" title="KNN和Lr有什么本质区别"></a>KNN和Lr有什么本质区别</h2><p>LR属于线性模型。<br>因为 logistic 回归的决策边界（decision boundary）是线性的。<br>KNN属于非线性模型</p><h2 id="学习率如何影响训练？"><a href="#学习率如何影响训练？" class="headerlink" title="学习率如何影响训练？"></a>学习率如何影响训练？</h2><p>如果学习率很低，训练会变得更加可靠，但是优化会耗费较长的时间，因为朝向损失函数最小值的每个步长很小。<br>如果学习率很高，训练可能根本不会收敛，甚至会发散。权重的改变量可能非常大，使得优化越过最小值，使得损失函数变得更糟。</p><h2 id="如何解决样本不均衡的问题"><a href="#如何解决样本不均衡的问题" class="headerlink" title="如何解决样本不均衡的问题?"></a>如何解决样本不均衡的问题?</h2><p>从数据集外补充, 我的项目里直接从网上爬图片, 加上SMOTE等过采样的算法(有好多种, 各有优劣, SMOTE应该是最好的), 以及对图片进行一些几何上的处理。</p><h2 id="线性分类器和非线性分类器"><a href="#线性分类器和非线性分类器" class="headerlink" title="线性分类器和非线性分类器"></a>线性分类器和非线性分类器</h2><p>线性分类器：模型是参数的线性函数，分类平面是（超）平面；<br>非线性分类器：模型分界面可以是曲面或者超平面的组合。<br>典型的线性分类器有感知机，LDA，逻辑斯特回归，SVM（线性核；<br>典型的非线性分类器有朴素贝叶斯（有文章说这个本质是线性的，<a href="http://dataunion.org/12344.html），kNN，决策树，SVM（非线性核）" target="_blank" rel="noopener">http://dataunion.org/12344.html），kNN，决策树，SVM（非线性核）</a></p><h2 id="为什么random-forest具有特征选择的功能"><a href="#为什么random-forest具有特征选择的功能" class="headerlink" title="为什么random forest具有特征选择的功能"></a>为什么random forest具有特征选择的功能</h2><p>树类分类器其实都可以, 因为可以判断每个特征对应的信息增益, 信息增益越大的特征分类效果理论上来说就越好。在非结构话数据的项目中常常会用这种方法来选取有用的特征, 从而降低特征维度。在特特征维度高的时候还是特别好用的。</p><h2 id="random-forest有哪些重要的参数？"><a href="#random-forest有哪些重要的参数？" class="headerlink" title="random forest有哪些重要的参数？"></a>random forest有哪些重要的参数？</h2><p>A. max_features：<br>随机森林允许单个决策树使用特征的最大数量。 Python为最大特征数提供了多个可选项。 下面是其中的几个：<br>Auto/None ：简单地选取所有特征，每颗树都可以利用他们。这种情况下，每颗树都没有任何的限制。<br>sqrt ：此选项是每颗子树可以利用总特征数的平方根个。 例如，如果变量（特征）的总数是100，所以每颗子树只能取其中的10个。“log2”是另一种相似类型的选项。<br>0.2：此选项允许每个随机森林的子树可以利用变量（特征）数的20％。如果想考察的特征x％的作用， 我们可以使用“0.X”的格式。<br>max_features如何影响性能和速度？<br>增加max_features一般能提高模型的性能，因为在每个节点上，我们有更多的选择可以考虑。 然而，这未必完全是对的，因为它降低了单个树的多样性，而这正是随机森林独特的优点。 但是，可以肯定，你通过增加max_features会降低算法的速度。 因此，你需要适当的平衡和选择最佳max_features。<br>B. n_estimators：<br>在利用最大投票数或平均值来预测之前，你想要建立子树的数量。 较多的子树可以让模型有更好的性能，但同时让你的代码变慢。 你应该选择尽可能高的值，只要你的处理器能够承受的住，因为这使你的预测更好更稳定。<br>C. min_sample_leaf：<br>如果您以前编写过一个决策树，你能体会到最小样本叶片大小的重要性。 叶是决策树的末端节点。 较小的叶子使模型更容易捕捉训练数据中的噪声。 一般来说，我更偏向于将最小叶子节点数目设置为大于50。在你自己的情况中，你应该尽量尝试多种叶子大小种类，以找到最优的那个。</p><h2 id="为什么梯度反方向是函数值局部下降最快的方向？"><a href="#为什么梯度反方向是函数值局部下降最快的方向？" class="headerlink" title="为什么梯度反方向是函数值局部下降最快的方向？"></a>为什么梯度反方向是函数值局部下降最快的方向？</h2><p><img src="http://p59louklr.bkt.clouddn.com/18-3-8/62464763.jpg" alt=""><br>那么此时如果要取得最大值，也就是当为0度的时候，也就是向量(这个方向是一直在变，在寻找一个函数变化最快的方向）与向量（这个方向当点固定下来的时候，它就是固定的）平行的时候，方向导数最大.方向导数最大，也就是单位步伐，函数值朝这个反向变化最快.</p><h2 id="DNN为什么功能强大，说说你的理解"><a href="#DNN为什么功能强大，说说你的理解" class="headerlink" title="DNN为什么功能强大，说说你的理解"></a>DNN为什么功能强大，说说你的理解</h2><p>深度学习的深度一方面增加了大量的参数，增加的参数意味着这个网络的表达能力更强大了。可以学习和区分的特征更多了。而一旦学习到的特征变多的话，我们在分类和识别的能力也就变好了。<br>从简单特征到抽象特征。随着网络深度增加，提取的特征不断复杂化。更能理解复杂概念。</p><h2 id="SVM的损失函数是什么？怎么理解"><a href="#SVM的损失函数是什么？怎么理解" class="headerlink" title="SVM的损失函数是什么？怎么理解"></a>SVM的损失函数是什么？怎么理解</h2><p>2分类SVM等于Hinge损失 + L2正则化。SVM最大化分类间距的目标函数等价于最小化Hinge损失 + L2正则化。推导并不复杂, 详见《统计学习方法》。</p><h2 id="介绍下Maxout"><a href="#介绍下Maxout" class="headerlink" title="介绍下Maxout"></a>介绍下Maxout</h2><p>maxout激活函数，它具有如下性质：<br>1、maxout激活函数并不是一个固定的函数，不像Sigmod、Relu、Tanh等函数，是一个固定的函数方程<br>2、它是一个可学习的激活函数，因为我们W参数是学习变化的。<br>3、它是一个分段线性函数：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/76764892.jpg" alt=""><br>maxout的拟合能力是非常强的，它可以拟合任意的的凸函数。最直观的解释就是任意的凸函数都可以由分段线性函数以任意精度拟合（学过高等数学应该能明白），而maxout又是取k个隐隐含层节点的最大值，这些”隐隐含层”节点也是线性的，所以在不同的取值范围下，最大值也可以看做是分段线性的（分段的个数与k值有关）</p><h2 id="根据混淆矩阵可以得到评价分类模型的指标有以下几种。"><a href="#根据混淆矩阵可以得到评价分类模型的指标有以下几种。" class="headerlink" title="根据混淆矩阵可以得到评价分类模型的指标有以下几种。"></a>根据混淆矩阵可以得到评价分类模型的指标有以下几种。</h2><p>分类准确度，就是正负样本分别被正确分类的概率，计算公式为：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/60634589.jpg" alt=""><br>召回率，就是正样本被识别出的概率，计算公式为：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/25932847.jpg" alt=""><br>虚警率，就是负样本被错误分为正样本的概率，计算公式为：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/4082036.jpg" alt=""><br>精确度，就是分类结果为正样本的情况真实性程度，计算公式为：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/52471608.jpg" alt=""></p><h2 id="优化器类型"><a href="#优化器类型" class="headerlink" title="优化器类型"></a>优化器类型</h2><ol><li>Batch gradient descent<br>梯度更新规则:<br>BGD 采用整个训练集的数据来计算 cost function 对参数的梯度：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/35803593.jpg" alt=""><br>Batch gradient descent 对于凸函数可以收敛到全局极小值，对于非凸函数可以收敛到局部极小值。</li><li>Stochastic gradient descent<br>梯度更新规则:<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/16824201.jpg" alt=""><br>和 BGD 的一次用所有数据计算梯度相比，SGD 每次更新时对每个样本进行梯度更新，<br>对于很大的数据集来说，可能会有相似的样本，这样 BGD 在计算梯度时会出现冗余，<br>而 SGD 一次只进行一次更新，就没有冗余，而且比较快，并且可以新增样本。<br>缺点:<br>但是 SGD 因为更新比较频繁，会造成 cost function 有严重的震荡。<br>BGD 可以收敛到局部极小值，当然 SGD 的震荡可能会跳到更好的局部极小值处。</li></ol><p>3.Adam<br>自适应优化器，能够自发地改变学习率。效果最好。<br>存储了过去梯度的平方 vt 的指数衰减平均值 ，也像 momentum 一样保持了过去梯度 mt 的指数衰减平均值：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/42015632.jpg" alt=""><br>梯度更新规则:<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/55724328.jpg" alt=""><br>超参数设定值:<br>建议 β1 ＝ 0.9，β2 ＝ 0.999，ϵ ＝ 10e−8<br>实践表明，Adam 比其他适应性学习方法效果要好</p><h2 id="什么是梯度消失？怎么解决"><a href="#什么是梯度消失？怎么解决" class="headerlink" title="什么是梯度消失？怎么解决"></a>什么是梯度消失？怎么解决</h2><p>梯度消失问题发生时，接近于输出层的hidden layer 3等的权值更新相对正常，但前面的hidden layer 1的权值更新会变得很慢，导致前面的层权值几乎不变，仍接近于初始化的权值，这就导致hidden layer 1相当于只是一个映射层，对所有的输入做了一个同一映射，这是此深层网络的学习就等价于只有后几层的浅层网络的学习了。<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/42823180.jpg" alt=""><br>其实梯度爆炸和梯度消失问题都是因为网络太深，链式求导，网络权值更新不稳定造成的，本质上是因为梯度反向传播中的连乘效应。对于更普遍的梯度消失问题，可以考虑用ReLU激活函数取代sigmoid激活函数。</p><h2 id="batch-normalisation作用"><a href="#batch-normalisation作用" class="headerlink" title="batch normalisation作用"></a>batch normalisation作用</h2><p>那BN到底是什么原理呢？说到底还是为了防止“梯度弥散”。关于梯度弥散，大家都知道一个简单的栗子：<br><img src="http://p59louklr.bkt.clouddn.com/18-3-8/83379060.jpg" alt=""><br>在BN中，是通过将activation规范为均值和方差一致的手段使得原本会减小的activation的scale变大。可以说是一种更有效的local response normalization方法。</p><h2 id="LSTM模型介绍和BPTT推导"><a href="#LSTM模型介绍和BPTT推导" class="headerlink" title="LSTM模型介绍和BPTT推导"></a>LSTM模型介绍和BPTT推导</h2><p><img src="http://p59louklr.bkt.clouddn.com/18-3-8/38457430.jpg" alt=""></p><h2 id="LR-和-linear-SVM区别"><a href="#LR-和-linear-SVM区别" class="headerlink" title="LR 和 linear SVM区别"></a>LR 和 linear SVM区别</h2><p>相同点：<br>1，LR和SVM都是分类算法。<br>2，如果不考虑核函数，LR和SVM都是线性分类算法，即分类决策面都是线性的。<br>3，LR和SVM都是监督学习算法。<br>不同点：<br>1，本质上是其loss function不同。<br>2，支持向量机只考虑局部的边界线附近的点，而逻辑回归考虑全局（远离的点对边界线的确定也起作用）。<br>线性SVM不直接依赖于数据分布，分类平面不受一类点影响；LR则受所有数据点的影响，如果数据不同类别strongly unbalance，一般需要先对数据做balancing。<br>3，在解决非线性问题时，支持向量机采用核函数的机制，而LR通常不采用核函数的方法。<br>这个问题理解起来非常简单。分类模型的结果就是计算决策面，模型训练的过程就是决策面的计算过程。通过上面的第二点不同点可以了解，在计算决策面时，SVM算法里只有少数几个代表支持向量的样本参与了计算，也就是只有少数几个样本需要参与核计算（即kernal machine解的系数是稀疏的）。然而，LR算法里，每个样本点都必须参与决策面的计算过程，也就是说，假设我们在LR里也运用核函数的原理，那么每个样本点都必须参与核计算，这带来的计算复杂度是相当高的。所以，在具体应用时，LR很少运用核函数机制。<br>4，线性SVM依赖数据表达的距离测度，所以需要对数据先做normalization，LR不受其影响。<br>5，SVM的损失函数就自带正则！！！（损失函数中的1/2||w||^2项），这就是为什么SVM是结构风险最小化算法的原因！！！而LR必须另外在损失函数上添加正则项！！！<br>在Andrew NG的课里讲到过：</p><ol><li>如果Feature的数量很大，跟样本数量差不多，这时候选用LR或者是Linear Kernel的SVM</li><li>如果Feature的数量比较小，样本数量一般，不算大也不算小，选用SVM+Gaussian Kernel</li><li>如果Feature的数量比较小，而样本数量很多，需要手工添加一些feature变成第一种情况</li></ol><h2 id="SVM面试问题"><a href="#SVM面试问题" class="headerlink" title="SVM面试问题"></a>SVM面试问题</h2><p>为什么要把原问题转换为对偶问题？因为原问题是凸二次规划问题，转换为对偶问题更加高效。<br>为什么求解对偶问题更加高效？因为只用求解alpha系数，而alpha系数只有支持向量才非0，其他全部为0.<br>alpha系数有多少个？样本点的个数</p><h2 id="线性回归基本假设"><a href="#线性回归基本假设" class="headerlink" title="线性回归基本假设?"></a>线性回归基本假设?</h2><p>线性回归需要满足四个前提假设.LINE!!!</p><ol><li>Linearity 线性. 应变量和每个自变量都是线性关系。 </li><li>Indpendence 独立性. 对于所有的观测值，它们的误差项相互之间是独立的。 </li><li>Normality 正态性. 误差项服从正态分布。 </li><li>Equal-variance 等方差. 所有的误差项具有同样方差。</li></ol><h2 id="用cos做激活函数行不行"><a href="#用cos做激活函数行不行" class="headerlink" title="用cos做激活函数行不行?"></a>用cos做激活函数行不行?</h2><p>答案是可以的, 面试的时候想太多答得不好。当时觉得cos容易使得模型出现梯度消失的情况, 但是其实要看具体的问题, cos在某些问题上表现得很好。其实激活函数的目的是为了使模型具有非线性, 否则再深的神经网络到最后也只是一个线性分类器。<br>一个好的激活函数应该有如下几个标准(摘自知乎):<br>作者：Hengkai Guo<br>链接：<a href="https://www.zhihu.com/question/67366051/answer/262087707" target="_blank" rel="noopener">https://www.zhihu.com/question/67366051/answer/262087707</a><br>来源：知乎<br>著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。</p><ol><li>非线性：即导数不是常数。这个条件前面很多答主都提到了，是多层神经网络的基础，保证多层网络不退化成单层线性网络。这也是激活函数的意义所在。</li><li>几乎处处可微：可微性保证了在优化中梯度的可计算性。传统的激活函数如sigmoid等满足处处可微。对于分段线性函数比如ReLU，只满足几乎处处可微（即仅在有限个点处不可微）。对于SGD算法来说，由于几乎不可能收敛到梯度接近零的位置，有限的不可微点对于优化结果不会有很大影响[1]。</li><li>计算简单：正如题主所说，非线性函数有很多。极端的说，一个多层神经网络也可以作为一个非线性函数，类似于Network In Network[2]中把它当做卷积操作的做法。但激活函数在神经网络前向的计算次数与神经元的个数成正比，因此简单的非线性函数自然更适合用作激活函数。这也是ReLU之流比其它使用Exp等操作的激活函数更受欢迎的其中一个原因。</li><li>非饱和性（saturation）：饱和指的是在某些区间梯度接近于零（即梯度消失），使得参数无法继续更新的问题。最经典的例子是Sigmoid，它的导数在x为比较大的正值和比较小的负值时都会接近于0。更极端的例子是阶跃函数，由于它在几乎所有位置的梯度都为0，因此处处饱和，无法作为激活函数。ReLU在x>0时导数恒为1，因此对于再大的正值也不会饱和。但同时对于x<0，其梯度恒为0，这时候它也会出现饱和的现象（在这种情况下通常称为dying ReLU）。Leaky ReLU[3]和PReLU[4]的提出正是为了解决这一问题。</li><li>单调性（monotonic）：即导数符号不变。这个性质大部分激活函数都有，除了诸如sin、cos等。个人理解，单调性使得在激活函数处的梯度方向不会经常改变，从而让训练更容易收敛。</li><li>输出范围有限：有限的输出范围使得网络对于一些比较大的输入也会比较稳定，这也是为什么早期的激活函数都以此类函数为主，如Sigmoid、TanH。但这导致了前面提到的梯度消失问题，而且强行让每一层的输出限制到固定范围会限制其表达能力。因此现在这类函数仅用于某些需要特定输出范围的场合，比如概率输出（此时loss函数中的log操作能够抵消其梯度消失的影响[1]）、LSTM里的gate函数。7. 接近恒等变换（identity）：即约等于x。这样的好处是使得输出的幅值不会随着深度的增加而发生显著的增加，从而使网络更为稳定，同时梯度也能够更容易地回传。这个与非线性是有点矛盾的，因此激活函数基本只是部分满足这个条件，比如TanH只在原点附近有线性区（在原点为0且在原点的导数为1），而ReLU只在x>0时为线性。这个性质也让初始化参数范围的推导更为简单[5][4]。额外提一句，这种恒等变换的性质也被其他一些网络结构设计所借鉴，比如CNN中的ResNet[6]和RNN中的LSTM。</li><li>参数少：大部分激活函数都是没有参数的。像PReLU带单个参数会略微增加网络的大小。还有一个例外是Maxout[7]，尽管本身没有参数，但在同样输出通道数下k路Maxout需要的输入通道数是其它函数的k倍，这意味着神经元数目也需要变为k倍；但如果不考虑维持输出通道数的情况下，该激活函数又能将参数个数减少为原来的k倍。</li><li>归一化（normalization）：这个是最近才出来的概念，对应的激活函数是SELU[8]，主要思想是使样本分布自动归一化到零均值、单位方差的分布，从而稳定训练。在这之前，这种归一化的思想也被用于网络结构的设计，比如Batch Normalization[9]。</li></ol><h2 id="To-be-solved…"><a href="#To-be-solved…" class="headerlink" title="To be solved…"></a>To be solved…</h2><p>项目中over-fitting了，你怎么办<br>详细说一个你知道的优化算法(Adam等)<br>项目(比赛）怎么做的模型的ensemble<br>stacking是什么？需要注意哪些问题<br>了解哪些online learning的算法<br>如何解决样本不均衡的问题<br>fasterRCNN中的ROIPooling是如何实现的<br>如何进行特征的选择<br>如何进行模型的选择<br>常用的有哪些损失函数<br>XX用户画像挖掘怎么做的feature engineering?<br>假设一个5*5的filter与图像卷积，如何降低计算量？<br>做过模型压缩吗？介绍下<br>什么是residual learning？说说你的理解<br>residual learning所说的residual和GBDT中的residual有什么区别？<br>FFM和FTRL有过了解吗？<br>你对现在Deep Learning的发展和遇到的问题有什么看法？</p>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><p>算是第一次正式的跟职场接触, 刚开始的时候也是一头雾水的状态。面得多了也慢慢有经验, 除了第一个头条挂了, 嗯挂的很惨。也是很想去的一家公司, 但是第一次哎, 多半会这样,
</summary>
<category term="笔记" scheme="http://czx.im/categories/%E7%AC%94%E8%AE%B0/"/>
<category term="机器学习" scheme="http://czx.im/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"/>
</entry>
<entry>
<title>写在开始</title>
<link href="http://czx.im/2018/03/06/%E5%86%99%E5%9C%A8%E5%BC%80%E5%A7%8B/"/>
<id>http://czx.im/2018/03/06/写在开始/</id>
<published>2018-03-05T21:38:12.000Z</published>
<updated>2018-03-12T08:46:59.699Z</updated>
<content type="html"><![CDATA[<script src="/assets/js/APlayer.min.js"> </script><p><script type="text/javascript" src="//rf.revolvermaps.com/0/0/6.js?i=5puhtu8vfhh&m=7&c=e63100&cr1=ffffff&f=arial&l=0&bv=90&lx=-420&ly=420&hi=20&he=7&hc=a8ddff&rs=80" async="async"></script><br>其实写一个个人网站是一个已经酝酿许久的想法, 但是一直太懒, 也没有时间;), 所以就耽搁下了。这次趁着在法国生活即将结束之际, 偷懒地用github把这个小计划也实现了。以后会在这里写些技术文章, 机器学习, 算法, AI估计都会有, 希望能和大家共同进步吧, 和一些无病呻吟的文章, 毕竟生活还需要记录和一些仪式感。<br>马上就要回国了, 总算可以专心写代码学算法了, 之前学的时候都太仓促, 以后要多总结下。<br>最后, 要把写博客的习惯坚持下去。<br>以上</p><iframe frameborder="no" border="0" marginwidth="0" marginheight="0" width="330" height="86" src="http://music.163.com/outchain/player?type=2&id=28285910&auto=1&height=66"></iframe>]]></content>
<summary type="html">
<script src="/assets/js/APlayer.min.js"> </script><p><script type="text/javascript" src="//rf.revolvermaps.com/0/0/6.js?i=5puhtu8vfhh&amp;m=
</summary>
<category term="杂记" scheme="http://czx.im/categories/%E6%9D%82%E8%AE%B0/"/>
<category term="无病呻吟" scheme="http://czx.im/tags/%E6%97%A0%E7%97%85%E5%91%BB%E5%90%9F/"/>
</entry>
</feed>