“““
backprop_magnitude_nabla
~~~~~~~~~~~~~~~~~~~~~~~~

Using?backprop2?I?constructed?a?784-30-30-30-30-30-10?network?to?classify
MNIST?data.??I?ran?ten?mini-batches?of?size?100?with?eta?=?0.01?and
lambda?=?0.05?using:

net.SGD（otd[:1000]?1?100?0.01?0.05

I?obtained?the?following?norms?for?the?（unregularized）?nabla_w?for?the
respective?mini-batches:

[0.90845722175923671?2.8852730656073566?10.696793986223632?37.75701921183488?157.7365422527995?304.43990075227839]
[0.22493835119537842?0.6555126517964851?2.6036801277234076?11.408825365731225?46.882319190445472?70.499637502698221]
[0.11935180022357521?0.19756069137133489?0.8152794148335869?3.4590802543293977?15.470507965493903?31.032396017142556]
[0.15130005837653659?0.39687135985664701?1.4810006139254532?4.392519005642268?16.831939776937311?34.082104455938733]
[0.11594085276308999?0.17177668061395848?0.72204558746599512?3.05062409378366?14.133001132214286?29.776204839994385]
[0.10790389807606221?0.20707152756018626?0.96348134037828603?3.9043824079499561?15.986873430586924?39.195258080490895]
[0.088613291101645356?0.129173436407863?0.4242933114455002?1.6154682713449411?7.5451567587160069?20.180545544006566]
[0.086175380639289575?0.12571016850457151?0.44231149185805047?1.8435833504677326?7.61973813981073?19.474539356281781]
[0.095372080184163904?0.15854489503205446?0.70244235144444678?2.6294803575724157?10.427062019753425?24.309420272033819]
[0.096453131000155692?0.13574642196947601?0.53551377709415471?2.0247466793066895?9.4503978546018068?21.73772148470092]

Note?that?results?are?listed?in?order?of?layer.??They?clearly?show?how
the?magnitude?of?nabla_w?decreases?as?we?go?back?through?layers.

In?this?program?I?take?min-batches?7?8?9?as?representative?and?plot
them.??I?omit?the?results?from?the?first?and?final?layers?since?they
correspond?to?784?input?neurons?and?10?output?neurons?not?30?as?in
the?other?layers?making?it?difficult?to?compare?results.

Note?that?I?haven‘t?attempted?to?preserve?the?whole?workflow?here.?It
involved?some?minor?hacking?around?with?backprop2?which?messed?up
that?code.??That‘s?why?I‘ve?simply?put?the?results?in?by?hand?below.
“““

#?Third-party?libraries
import?matplotlib.pyplot?as?plt

nw1?=?[0.129173436407863?0.4242933114455002?
???????1.6154682713449411?7.5451567587160069]
nw2?=?[0.12571016850457151?0.44231149185805047?
???????1.8435833504677326?7.61973813981073]
nw3?=?[0.15854489503205446?0.70244235144444678?
???????2.6294803575724157?10.427062019753425]
plt.plot（range（1?5）?nw1?“ro-“?range（1?5）?nw2?“go-“?
?????????range（1?5）?nw3?“bo-“）
plt.xlabel（‘layer?$l$‘）
plt.ylabel（r“$\Vert\nabla?C^l_w\Vert$“）
plt.xticks（[1?2?3?4]）
plt.show（）

?屬性????????????大小?????日期????時間???名稱
-----------?---------??----------?-----??----

?????文件?????????52??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\.gitignore

?????文件???17051982??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\data\mnist.pkl.gz

?????文件??????29523??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\backprop_magnitude_nabla.png

?????文件???????2790??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\backprop_magnitude_nabla.py

?????文件????5375943??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\data_1000.json

?????文件???????8414??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\digits.png

?????文件???????8218??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\digits_separate.png

?????文件?????150522??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\false_minima.png

?????文件???????1066??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\false_minima.py

?????文件???????3848??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\generate_gradient.py

?????文件????????272??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\initial_gradient.json

?????文件?????190268??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\misleading_gradient.png

?????文件???????1207??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\misleading_gradient.py

?????文件??????59286??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\misleading_gradient_contours.png

?????文件????????514??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\misleading_gradient_contours.py

?????文件??????12449??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist.py

?????文件??????58028??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_100_digits.png

?????文件???????5499??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_2_and_1.png

?????文件???????4934??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_complete_zero.png

?????文件???????4904??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_first_digit.png

?????文件???????4715??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_other_features.png

?????文件??????11964??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_really_bad_images.png

?????文件???????3940??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\mnist_top_left_feature.png

?????文件?????????63??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data.json

?????文件??????33106??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data.png

?????文件???????3821??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data.py

?????文件???????4832??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data_5.png

?????文件??????43656??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data_comparison.png

?????文件??????34589??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data_log.png

?????文件???????5308??2018-03-12?11:02??neural-networks-and-deep-learning-master（python3）\fig\more_data_rotated_5.png

............此處省略68個文件信息