The Star Also Rises

Tuesday, December 24, 2019

ShakeDrop

ShakeDrop

2019/12/09

-----

# ShakeDrop

-----

References

# ShakeDrop

Yamada, Yoshihiro, et al. "Shakedrop regularization for deep residual learning." arXiv preprint arXiv:1802.02375 (2018).

https://arxiv.org/pdf/1802.02375.pdf

Shake-Shake

Shake-Shake

2019/12/09

-----

# Shake-Shake

-----

References

# Shake-Shake

Gastaldi, Xavier. "Shake-shake regularization." arXiv preprint arXiv:1705.07485 (2017).

https://arxiv.org/pdf/1705.07485.pdf

大白话《Shake-Shake regularization》 - 知乎
https://zhuanlan.zhihu.com/p/33751164

NASNet（Scheduled DropPath）

NASNet（Scheduled DropPath）

2019/12/09

-----

# NASNet（Scheduled DropPath）

-----

# NASNet（Scheduled DropPath）

-----

References

# NASNet（Scheduled DropPath）

Zoph, Barret, et al. "Learning transferable architectures for scalable image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

http://openaccess.thecvf.com/content_cvpr_2018/papers/Zoph_Learning_Transferable_Architectures_CVPR_2018_paper.pdf

机器学习论文笔记-Learning Transferable Architectures for Scalable Image Recognition - 知乎
https://zhuanlan.zhihu.com/p/31655995

ResNet-D

ResNet-D

2019/11/12

前言：

論文《Deep networks with stochastic depth》，我將其命名為 ResNet-D，取其名稱中的 depth。此外，論文裡的實驗隨機丟棄一些殘差層來訓練網路，使用的技巧是 dropout。經由這樣的設計，也可將原始的殘差網路從一百層提升到一千層。

-----

// Review Stochastic Depth (Image Classification) - Towards Data Science

-----

# ResNet-D

-----

# ResNet-D

-----

References

◎ 論文

# ResNet-D

Huang, Gao, et al. "Deep networks with stochastic depth." European conference on computer vision. Springer, Cham, 2016.

https://arxiv.org/pdf/1603.09382.pdf

-----

◎ 英文參考資料

Review Stochastic Depth (Image Classification) - Towards Data Science
https://towardsdatascience.com/review-stochastic-depth-image-classification-a4e225807f4a

-----

◎ 簡體中文參考資料

Deep Networks with Stochastic Depth
https://bingning.wang/research/Article/?id=59

DropPath

DropPath

2019/12/09

-----

# FractalNet（DropPath）

-----

# FractalNet（DropPath）

-----

References

# FractalNet（DropPath）

Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. "Fractalnet: Ultra-deep neural networks without residuals." arXiv preprint arXiv:1605.07648 (2016).

https://arxiv.org/pdf/1605.07648.pdf

论文笔记：分形网络（FractalNet Ultra-Deep Neural Networks without Residuals） - PilgrimHui - 博客园
https://www.cnblogs.com/liaohuiqiang/p/9218445.html

Dropconnect

Dropconnect

2019/12/09

-----

# DropConnect

-----

// Dropout, DropConnect, and Maxout Mechanism Network. _ Download Scientific Diagram

-----

References

# DropConnect

Wan, Li, et al. "Regularization of neural networks using dropconnect." International conference on machine learning. 2013.

http://proceedings.mlr.press/v28/wan13.pdf

Deep learning：四十六(DropConnect简单理解) - tornadomeet - 博客园
https://www.cnblogs.com/tornadomeet/p/3430312.html

Dropout

Dropout

2019/12/09

-----

// DNN tip

-----

# Dropout

-----

# Dropout

-----

References

# Dropout

Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." The Journal of Machine Learning Research 15.1 (2014): 1929-1958.

http://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf

深度学习中Dropout原理解析 - 知乎
https://zhuanlan.zhihu.com/p/38200980

DNN tip
http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/DNN%20tip.pdf

Early Stopping

Early Stopping

2019/12/20

-----

// Improving Deep Neural Networks Hyperparameter tuning, Regularization and Optimization - week 1

-----

// regularization

-----

References

# Early Stopping

Li, Mingchen, Mahdi Soltanolkotabi, and Samet Oymak. "Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks." arXiv preprint arXiv:1903.11680 (2019).

https://arxiv.org/pdf/1903.11680.pdf

regularization
http://www.deeplearningbook.org/contents/regularization.html

深度学习技巧之Early Stopping（早停法） - df19900725的博客
https://blog.csdn.net/df19900725/article/details/82973049

Improving Deep Neural Networks Hyperparameter tuning, Regularization and Optimization - week 1
https://medium.com/chiukevin0321/improving-deep-neural-networks-hyperparameter-tuning-regularization-and-optimization-week-1-59e873f40e66

L0

L0

2019/12/09

-----

// Regularization in Machine Learning Connect the dots

-----

// Foundations of Machine Learning Part 4 - DZone AI

-----

// Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]

-----

// Lp space - Wikipedia

-----

// Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]

-----

// Machine Learning & Data Mining CS_CNS_EE 155 Lecture 3 Regularization, Sparsity & Lasso ppt download

-----

// Regularization in Machine Learning Connect the dots

-----

References

# L0

Louizos, Christos, Max Welling, and Diederik P. Kingma. "Learning Sparse Neural Networks through $ L_0 $ Regularization." arXiv preprint arXiv:1712.01312 (2017).

https://arxiv.org/pdf/1712.01312.pdf

Foundations of Machine Learning Part 4 - DZone AI
https://dzone.com/articles/foundations-of-machine-learning-part-4

Machine Learning & Data Mining CS_CNS_EE 155 Lecture 3 Regularization, Sparsity & Lasso ppt download
https://slideplayer.com/slide/3345346/

L0 Norm, L1 Norm, L2 Norm & L-Infinity Norm - Sara Iris Garcia - Medium
https://medium.com/@montjoile/l0-norm-l1-norm-l2-norm-l-infinity-norm-7a7d18a4f40c

Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]
https://towardsdatascience.com/topographic-regularized-feature-learning-in-tensorflow-manual-backprop-in-tf-f50507e69472

Norm (mathematics) - Wikipedia
https://en.m.wikipedia.org/wiki/Norm_(mathematics)

Lp space - Wikipedia
https://en.m.wikipedia.org/wiki/Lp_space

Regularization in Machine Learning Connect the dots
https://towardsdatascience.com/regularization-in-machine-learning-connecting-the-dots-c6e030bfaddd

笔记︱范数正则化L0、L1、L2-岭回归&Lasso回归（稀疏与特征工程） - 云+社区 - 腾讯云
https://cloud.tencent.com/developer/article/1436207

L1

L1

2019/12/09

-----

// Lasso Regression - R Statistics Blog

-----

// Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]

-----

// regression - Why L1 norm for sparse models - Cross Validated

-----

References

# L1
# Lasso Regression

Tibshirani, Robert. "Regression shrinkage and selection via the lasso." Journal of the Royal Statistical Society: Series B (Methodological) 58.1 (1996): 267-288.

http://www.stat.ucla.edu/~sczhu/courses/ucla/stat_232b/chapters/LASSO.pdf

Lasso Regression - R Statistics Blog
https://www.rstatisticsblog.com/data-science-in-action/lasso-regression/

Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]
https://towardsdatascience.com/topographic-regularized-feature-learning-in-tensorflow-manual-backprop-in-tf-f50507e69472

regression - Why L1 norm for sparse models - Cross Validated
https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models

l1正则与l2正则的特点是什么，各有什么优势？ - 知乎
https://www.zhihu.com/question/26485586

L2

L2

2019/12/09

-----

// Ridge Regression - R Statistics Blog

-----

// Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]

-----

References

# L2
# Ridge Regression

Hoerl, Arthur E., and Robert W. Kennard. "Ridge regression: Biased estimation for nonorthogonal problems." Technometrics 12.1 (1970): 55-67.

https://amstat.tandfonline.com/doi/pdf/10.1080/00401706.1970.10488634

Ridge Regression - R Statistics Blog
https://www.rstatisticsblog.com/data-science-in-action/ridge-regression/

Topographic _ Regularized Feature Learning in Tensorflow [ Manual Backprop in TF ]
https://towardsdatascience.com/topographic-regularized-feature-learning-in-tensorflow-manual-backprop-in-tf-f50507e69472

深度学习（Deep Learning）基础概念8：L2正则化（L2 Regularization）、Dropout原理及其python实现 - 知乎
https://zhuanlan.zhihu.com/p/29592806

ResNet-U

ResNet-U

2019/12/24

-----

References

# ResNet-U

Liu, Tianyi, et al. "Towards Understanding the Importance of Shortcut Connections in Residual Networks." Advances in Neural Information Processing Systems. 2019.

http://papers.nips.cc/paper/9003-towards-understanding-the-importance-of-shortcut-connections-in-residual-networks.pdf

ResNet-V

ResNet-V

2019/12/20

-----

# ResNet-V

-----

References

# ResNet-V

Li, Hao, et al. "Visualizing the loss landscape of neural nets." Advances in Neural Information Processing Systems. 2018.

https://papers.nips.cc/paper/7875-visualizing-the-loss-landscape-of-neural-nets.pdf

[论文阅读]损失函数可视化及其对神经网络的指导作用 - 知乎
https://zhuanlan.zhihu.com/p/5231427

ResNet-F

ResNet-F

2019/12/24

-----

References

# ResNet-F

Zhang, Hongyi, Yann N. Dauphin, and Tengyu Ma. "Fixup initialization: Residual learning without normalization." arXiv preprint arXiv:1901.09321 (2019).

https://arxiv.org/pdf/1901.09321.pdf

IterNorm

IterNorm

2019/12/24

-----

References

# IterNorm

Huang, Lei, et al. "Iterative Normalization: Beyond Standardization towards Efficient Whitening." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

http://openaccess.thecvf.com/content_CVPR_2019/papers/Huang_Iterative_Normalization_Beyond_Standardization_Towards_Efficient_Whitening_CVPR_2019_paper.pdf

KN

KN

2019/12/24

-----

References

# KN

Wang, Guangrun, et al. "Kalman normalization: Normalizing internal representations across network layers." Advances in Neural Information Processing Systems. 2018.

https://papers.nips.cc/paper/7288-kalman-normalization-normalizing-internal-representations-across-network-layers.pdf

DBN

DBN

2019/12/24

-----

References

# DBN

Huang, Lei, et al. "Decorrelated batch normalization." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

http://openaccess.thecvf.com/content_cvpr_2018/papers/Huang_Decorrelated_Batch_Normalization_CVPR_2018_paper.pdf

GWNN

GWNN

2019/12/24

-----

References

# GWNN

Luo, Ping. "Learning deep architectures via generalized whitened neural networks." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.

http://proceedings.mlr.press/v70/luo17a/luo17a.pdf