Regularization

Regularization

2020/10/19

-----

https://pixabay.com/zh/photos/health-fitness-health-is-wealth-4861815/

----- 

Regularization
https://hemingwang.blogspot.com/2019/10/an-overview-of-regularization.html 
https://hemingwang.blogspot.com/2019/10/regularization.html

-----

Regularization 一般被翻譯成正則化。正則化比較難以理解，所以讀者只要想成是避免過擬合即可。

那什麼叫過擬合呢？簡單說，機器學習，或者說統計學，是要從有限的樣本裡面，歸納出一個模型，用來預測真實世界裡面的無窮資料。如果我們過份考慮資料集的資料，則訓練出來的模型，就很可能無法預測資料集以外的資料。正則化，就是避免過擬合的一些方法的總稱。

-----

◎ Weight Decay、L2、L1、L0、

◎ Early Stopping、

◎ Fully Connected Networks（Dropout、DropConnect、）

CNN（DropPath、Scheduled DropPath、
Shake-Shake、ShakeDrop、
Spatial Dropout、Cutout、DropBlock、）

RNN（Fast Dropout、RNN Regularization、
Variational Dropout、Information Dropout、
rnnDrop、DropEmbedding、Recurrent Dropout、Zoneout、AWD-LSTM、）

Attention（DropAttention、）

◎ Label Smoothing、

◎ Pairing Samples、Mixup。 

----- 

Weight Decay
https://hemingwang.blogspot.com/2019/12/weight-decay.html

L2
https://hemingwang.blogspot.com/2019/12/l2.html

L1
https://hemingwang.blogspot.com/2019/12/l1.html

L0
https://hemingwang.blogspot.com/2019/12/l0.html

-----

Early Stopping 
https://hemingwang.blogspot.com/2019/12/early-stopping.html 

----- 

// FNN 

Dropout
https://hemingwang.blogspot.com/2020/09/dropout.html
https://hemingwang.blogspot.com/2019/12/dropout.html 

Dropconnect
https://hemingwang.blogspot.com/2019/12/dropconnect.html

----- 

// CNN

DropPath（FractalNet）
https://hemingwang.blogspot.com/2019/12/droppath.html

ResNet-D
https://hemingwang.blogspot.com/2019/11/resnet-d.html 

NASNet（Scheduled DropPath）
https://hemingwang.blogspot.com/2019/12/scheduled-droppath.html

Shake-Shake
https://hemingwang.blogspot.com/2019/12/shake-shake.html

ShakeDrop
https://hemingwang.blogspot.com/2019/12/shakedrop.html

Spatial Dropout
https://hemingwang.blogspot.com/2019/12/spatial-dropout.html 

Cutout
https://hemingwang.blogspot.com/2019/12/cutout.html

DropBlock
https://hemingwang.blogspot.com/2019/12/dropblock.html

-----

// RNN

Fast Dropout 
https://hemingwang.blogspot.com/2019/12/fast-dropout.html 

RNN Regularization 
https://hemingwang.blogspot.com/2019/12/rnn-regularization.html 

Variational Dropout
https://hemingwang.blogspot.com/2019/12/variational-dropout.html

Information Dropout
https://hemingwang.blogspot.com/2019/12/information-dropout.html

rnnDrop
https://hemingwang.blogspot.com/2019/12/rnndrop.html

DropEmbbeding
https://hemingwang.blogspot.com/2019/12/dropembbeding.html 

Recurrent Dropout
https://hemingwang.blogspot.com/2019/12/recurrent-dropout.html

Zoneout
https://hemingwang.blogspot.com/2019/12/zoneout.html

AWD-LSTM
https://hemingwang.blogspot.com/2019/12/awd-lstm.html 

-----

// Self Attention

DropAttention 
https://hemingwang.blogspot.com/2019/12/dropattention.html

-----

// Data Augmentation

Pairing Samples
http://hemingwang.blogspot.com/2019/12/pairing-samples.html

Mixup
http://hemingwang.blogspot.com/2019/12/mixup.html

-----