AI 從頭學（2021 年版）

AI 從頭學（2021 年版）

2020/01/01

全方位 AI 課程（精華篇）
http://hemingwang.blogspot.com/2020/01/all-round-ai-lectures-highlight.html

-----

Fig. 2021（圖片來源：Pixabay）。

-----

From Statistics to Deep Learning

-----

https://hemingwang.blogspot.com/2020/10/from-statistics-to-deep-learning.html

-----

What's the Main Points of Deep Learning?

-----

https://hemingwang.blogspot.com/2020/10/whats-main-points-of-deep-learning.html

-----

一一、Regularization


一二、Normalization

一三、Optimization


一四、Activation Function


一五、Loss Function

-----

全方位 AI 課程（六十小時搞定深度學習）
https://hemingwang.blogspot.com/2020/01/all-round-ai-lectures.html

全方位 AI 課程（介紹篇）
https://hemingwang.blogspot.com/2020/01/all-round-ai-lectures-introduction.html

AI Seminar 2020 Taipei
https://hemingwang.blogspot.com/2019/12/ai-seminar-2020-taipei.html

-----

Python
https://hemingwang.blogspot.com/2019/02/python.html

-----

Part I：Computer Vision

◎ Image Classification

Stage 01：LeNet、（AlexNet、ZFNet）

Stage 02：NIN、（SENet、GoogLeNet、VGGNet、PreVGGNet、Highway v1 v2）、（Inception v3 v4、PolyNet）

Stage 03：ResNet v1 v2、（ResNet-D、ResNet-E、ResNet-I、ResNet-Q、ResNet-S、ResNet-W、WRN、ResNeXt、DenseNet、DPN、DLA、Res2Net）

◎ Semantic Segmentation

Stage 04：FCN、（DeconvNet、SegNet、U-Net、U-Net++、DilatedNet、ENet、DRN、FC-CRF、DeepLab v1 v2 v3 v3+、ResNet-38、RefineNet、RefineNet-LW、RefineNet-AA、PSPNet、ICNet、BiSeNet、Fast-SCNN、BlitzNet）

◎ Object Detection

Stage 05：（DPM、SS、R-CNN、SPPNet、Fast R-CNN、OHEM、Faster R-CNN、OverFeat）、YOLOv1、（SSD、DSSD、YOLOv2、ION、R-FCN、SATO、DCNv1、DCNv2、Cascade R-CNN、FPN、STDN、YOLOv3、RON、RefineDet、M2Det、DetNet、TridentNet、OHEM、Focal Loss、GHM、Libra R-CNN、DCRv1、DCRv2、PISA）

// ◎ Dataset

-----

Part II：Natural Language Processing

◎ LSTM
Stage 06：LSTM、（NNLM、Word2vec）

◎ Seq2seq
Stage 07：Seq2seq、（GloVe、fastText）

◎ Attention
Stage 08：Attention、（NTM、KVMN）

◎ ConvS2S
Stage 09：ConvS2S、（ELMo、ULMFiT）

◎ Transformer
Stage 10：Transformer、（GPT-1、BERT、GPT-2）

----- 

Part III：Fundamental Topics

◎ Regularization
Stage 11：（Weight Decay、L2、L1、L0、Dropout、DropConnect、DropPath、Scheduled DropPath、Shake-Shake、ShakeDrop、Spatial Dropout、Cutout、DropBlock、Fast Dropout、RNN Regularization、Variational Dropout、Information Dropout、DropEmbedding、Recurrent Dropout、Zoneout、AWD-LSTM、DropAttention、Mixup、Pairing Samples、AutoAugment）

◎ Normalization
Stage 12：（Batch、Weight、Layer、Instance、Group、Positional）

◎ Optimization
Stage 13：（SGD、Momentum、NAG、AdaGrad、AdaDelta、RMSProp、Adam、AdaMax、Nadam、AMSGrad、Lookahead、RAdam、LAMB、CLR、SGDR、AdamW、Super-Convergence、ADMM、ADMM-S、dlADMM）

◎ Activation Function
Stage 14：（sigmoid、tanh、ReLU、Softplus、LReLU、PReLU、ELU、SELU、GELU、Swish）

◎ Loss Function
Stage 15：

// ◎ Pooling
// ◎ Convolution
// ◎ Automatic Differentiation
// ◎ Back Propagation
// ◎ Computational Graph

-----

Part IV：Advanced Topics

◎ Instance Segmentation
Stage16：（Hypercolumn、MNC、DeepMask、SharpMask、MultiPathNet、InstanceFCN、FCIS）、Mask R-CNN、（MaskX R-CNN、MaskLab、PANet、HTC、RetinaMask、MS R-CNN、YOLACT）

◎ Mobile
Stage17：SqueezeNet、（MobileNet v1 v2 v3、ShuffleNet v1 v2、Xception）

◎ NAS
Stage18：NAS-RL、NASNet（Scheduled DropPath）、EfficientNet、Auto-DeepLab、NAS-FPN、 AutoAugment。

◎ GAN
Stage19：

◎ BERT
Stage20：

-----

Intelligence Science

-----

Intelligence Science
http://hemingwang.blogspot.com/2019/09/intelligence-science.html

-----


Part I：Computer Vision

-----

Computer Vision
https://hemingwang.blogspot.com/2019/10/computer-vision.html

https://hemingwang.blogspot.com/2019/10/gaussiansmooth.html

https://hemingwang.blogspot.com/2019/10/sobeledgedetection.html

https://hemingwang.blogspot.com/2019/10/structuretensor.html

https://hemingwang.blogspot.com/2019/10/nms.html

-----

◎ Image Classification

-----

Image Classification
https://hemingwang.blogspot.com/2019/10/image-classification.html

-----

◎ 1. LeNet（Image Classification）

-----

LeNet
https://hemingwang.blogspot.com/2019/05/trilogy.html
http://hemingwang.blogspot.com/2018/02/deep-learninglenet-bp.html
http://hemingwang.blogspot.com/2017/03/ailenet.html
http://hemingwang.blogspot.com/2017/03/ailenet-f6.html

AlexNet
http://hemingwang.blogspot.com/2017/05/aialexnet.html

PreAlexNet

ZFNet
http://hemingwang.blogspot.com/2017/05/aikernel-visualizing.html

PreZFNet

Deconv

-----

◎ 2. NIN（Image Classification）

-----

NIN
http://hemingwang.blogspot.com/2017/06/ainetwork-in-network.html

SENet
https://hemingwang.blogspot.com/2019/10/senet.html

SKNet
https://hemingwang.blogspot.com/2020/08/sknet.html

STNet
http://hemingwang.blogspot.com/2020/04/stnet.html

RANet

BAM

CBAM

RASNet

GoogLeNet
http://hemingwang.blogspot.com/2017/06/aigooglenet.html
http://hemingwang.blogspot.com/2017/06/aiconv1.html
http://hemingwang.blogspot.com/2017/08/aiinception.html

VGGNet
http://hemingwang.blogspot.com/2018/09/aivggnet.html 

PreVGGNet
https://hemingwang.blogspot.com/2019/11/prevggnet.html

Highway v1
http://hemingwang.blogspot.com/2019/11/highway.html

Highway v2

Inception v3
https://hemingwang.blogspot.com/2019/11/inception-v3.html

Inception v4
https://hemingwang.blogspot.com/2019/11/inception-v4.html

CapsNet v0

CapsNet v1
https://hemingwang.blogspot.com/2019/12/capsnet.html

CapsNet v2

CapsNet v3

-----

◎ 3. ResNet（Image Classification）

-----

ResNet
https://hemingwang.blogspot.com/2019/05/vanishing-gradient.html
https://hemingwang.blogspot.com/2019/05/exploding-gradient.html
http://hemingwang.blogspot.com/2019/10/an-overview-of-resnet-and-its-variants.html
https://hemingwang.blogspot.com/2019/10/universal-approximation-theorem.html 
https://hemingwang.blogspot.com/2019/10/understanding-boxplots.html
https://hemingwang.blogspot.com/2019/10/ensemble-learning.html
http://hemingwang.blogspot.com/2018/09/airesnet.html



ResNet v1

ResNet-D
https://hemingwang.blogspot.com/2019/11/resnet-d.html

ResNet v2

ResNet-E
https://hemingwang.blogspot.com/2019/11/resnet-e.html

ResNet-V
https://hemingwang.blogspot.com/2019/12/resnet-v.html

-----

ResNet-F
https://hemingwang.blogspot.com/2019/12/resnet-f.html

ResNet-I
https://hemingwang.blogspot.com/2019/12/resnet-i.html

ResNet-Q
https://hemingwang.blogspot.com/2019/12/resnet-q.html

ResNet-S
https://hemingwang.blogspot.com/2019/11/resnet-s.html

ResNet-U
https://hemingwang.blogspot.com/2019/12/resnet-u.html 

ResNet-W
https://hemingwang.blogspot.com/2019/12/resnet-w.html

WRN
https://hemingwang.blogspot.com/2019/11/wrn.html

ResNeXt
https://hemingwang.blogspot.com/2019/10/resnext.html

DenseNet
https://hemingwang.blogspot.com/2019/11/densenet.html

DPN
https://hemingwang.blogspot.com/2019/11/dpn.html

DLA
https://hemingwang.blogspot.com/2019/11/dla.html

Res2Net
https://hemingwang.blogspot.com/2019/11/res2net.html

-----

PolyNet
https://hemingwang.blogspot.com/2019/11/polynet.html

FractalNet
https://hemingwang.blogspot.com/2019/12/fractalnet.html

RevNet

----- 

◎ 3.1. SqueezeNet（Image Classification - Mobile）

-----

Mobile
https://hemingwang.blogspot.com/2019/10/mobile.html

SqueezeNet
https://hemingwang.blogspot.com/2019/10/squeezenet.html

MobileNet v1
https://hemingwang.blogspot.com/2019/10/mobilenet-v1.html

ShuffleNet

Xception
https://hemingwang.blogspot.com/2019/10/xception.html

-----

NAS-RL
https://hemingwang.blogspot.com/2019/12/nas-rl.html 

NASNet（Scheduled DropPath）
https://hemingwang.blogspot.com/2019/12/scheduled-droppath.html

EfficientNet 
https://hemingwang.blogspot.com/2019/12/efficientnet.html

Auto-DeepLab
https://hemingwang.blogspot.com/2019/12/auto-deeplab.html

NAS-FPN
https://hemingwang.blogspot.com/2019/12/nas-fpn.html 

AutoAugment
http://hemingwang.blogspot.com/2019/12/autoaugment.html

-----

◎ 4.1. FCN（Semantic  Segmentation）

-----

https://hemingwang.blogspot.com/2020/08/semantic-segmentation.html

-----

◎ 4.2. Instance Segmentation

-----

https://hemingwang.blogspot.com/2020/08/instance-segmentation.html

-----

◎ 4.3. Panoptic Segmentation

-----

https://hemingwang.blogspot.com/2020/08/panoptic-segmentation.html

-----

◎ 5. YOLOv1（Object Detection）

-----
 
Object Detection
https://hemingwang.blogspot.com/2019/10/object-detection.html 

-----

一、求好

DPM、SS、R-CNN、
SPPNet、Fast R-CNN、
Faster R-CNN、

-----

二、求快

OverFeat、YOLOv1、SSD、DSSD、YOLOv2、

-----

三、求好求快

ION、R-FCN、SATO、DCNv1、DCNv2、Cascade R-CNN、
FPN、STDN、YOLOv3、RON、RefineDet、M2Det、
SNIP、SNIPER、AutoFocus、
DetNet、TridentNet、

四、HARD

OHEM、Focal Loss、GHM、
Libra R-CNN、DCRv1、DCRv2、
PISA。

-----

DPM
https://hemingwang.blogspot.com/2019/11/dpm.html

SS
https://hemingwang.blogspot.com/2019/11/ss.html

R-CNN
https://hemingwang.blogspot.com/2019/11/r-cnn.html

SPPNet
https://hemingwang.blogspot.com/2019/11/sppnet.html

Fast R-CNN
https://hemingwang.blogspot.com/2019/11/fast-r-cnn.html

Faster R-CNN
https://hemingwang.blogspot.com/2019/09/faster-r-cnn.html 

-----

OverFeat
https://hemingwang.blogspot.com/2019/11/overfeat.html

YOLOv1
http://hemingwang.blogspot.com/2018/04/deep-learningyolo-v1.html
http://hemingwang.blogspot.com/2018/04/machine-learning-conceptmean-average.html
http://hemingwang.blogspot.com/2018/04/machine-learning-conceptnon-maximum.html
https://hemingwang.blogspot.com/2019/11/yolo-v1.html

SSD
https://hemingwang.blogspot.com/2019/09/ssd.html 

DSSD
https://hemingwang.blogspot.com/2019/11/dssd.html 

YOLOv2
https://hemingwang.blogspot.com/2019/11/yolo-v2.html

-----

ION
https://hemingwang.blogspot.com/2019/11/ion.html

R-FCN
https://hemingwang.blogspot.com/2019/11/r-fcn.html

SATO
https://hemingwang.blogspot.com/2019/10/sato.html 

DCNv1
https://hemingwang.blogspot.com/2019/12/dcn-v1.html

DCNv2
https://hemingwang.blogspot.com/2019/12/dcn-v2.html

Cascade R-CNN
https://hemingwang.blogspot.com/2019/12/cascade-r-cnn.html

FPN
https://hemingwang.blogspot.com/2019/11/fpn.html

STDN
https://hemingwang.blogspot.com/2019/12/stdn.html

YOLOv3
https://hemingwang.blogspot.com/2019/11/yolo-v3.html

RON
https://hemingwang.blogspot.com/2019/12/ron.html

RefineDet
https://hemingwang.blogspot.com/2019/11/refinedet.html

M2Det
https://hemingwang.blogspot.com/2019/10/m2det.html

SNIP
https://hemingwang.blogspot.com/2019/12/snip.html 

SNIPER
https://hemingwang.blogspot.com/2019/12/sniper.html

AutoFocus
https://hemingwang.blogspot.com/2019/12/autofocus.html

DetNet
https://hemingwang.blogspot.com/2019/12/detnet.html

TridentNet
https://hemingwang.blogspot.com/2019/12/tridentnet.html

-----

OHEM
https://hemingwang.blogspot.com/2019/11/ohem.html

Focal Loss
https://hemingwang.blogspot.com/2019/10/retinanet.html

GHM
https://hemingwang.blogspot.com/2019/12/ghm.html

Libra R-CNN
https://hemingwang.blogspot.com/2019/12/libra-r-cnn.html

DCRv1
https://hemingwang.blogspot.com/2019/12/dcr-v1.html

DCRv2
https://hemingwang.blogspot.com/2019/12/dcr-v2.html

PISA
https://hemingwang.blogspot.com/2019/12/pisa.html

-----
 
Part II：Natural Language Processing
 
-----

◎ 6. LSTM（NLP）

-----

LSTM
http://hemingwang.blogspot.com/2019/09/understanding-lstm-networks.html
https://hemingwang.blogspot.com/2019/09/lstm.html

-----

◎ NNLM

-----

MLE
EM
STM
n-gram

NNLM
https://hemingwang.blogspot.com/2019/04/nnlm.html

C&W
https://hemingwang.blogspot.com/2020/07/c.html

RNNLM
https://hemingwang.blogspot.com/2020/07/rnnlm.html

-----

◎ Word2vec 

-----

Word2vec
https://hemingwang.blogspot.com/2019/04/word2vec.html

Word2vec v1：CBOW and Skip-gram
https://hemingwang.blogspot.com/2020/07/word2vec-v1.html

Word2vec v2：Hierarchical Softmax and Negative Sampling
https://hemingwang.blogspot.com/2020/07/word2vec-v2.html

Word2vec v3：Simplified Word2vec v1 and v2
https://hemingwang.blogspot.com/2020/08/word2vec-v3.html

LSA
https://hemingwang.blogspot.com/2020/07/lsa.html

GloVe
https://hemingwang.blogspot.com/2020/07/glove.html

fastText v1
https://hemingwang.blogspot.com/2020/07/fasttext-v1.html

fastText v2
https://hemingwang.blogspot.com/2020/07/fasttext-v2.html

WordRank
https://hemingwang.blogspot.com/2020/07/wordrank.html

-----

◎ 7. Seq2seq（NLP）

-----

Seq2seq
http://hemingwang.blogspot.com/2019/10/word-level-english-to-marathi-neural.html
https://hemingwang.blogspot.com/2019/09/seq2seq.html

RNN Encoder-Decoder 1
http://hemingwang.blogspot.com/2020/08/rnn-encoder-decoder-1.html

RNN Encoder-Decoder 2
http://hemingwang.blogspot.com/2020/08/rnn-encoder-decoder-2.html

Teacher Forcing 1
http://hemingwang.blogspot.com/2020/08/teacher-forcing.html

Beam Search
http://hemingwang.blogspot.com/2020/08/beam-search.html

Curriculum Learning
http://hemingwang.blogspot.com/2020/08/curriculum-learning.html

-----

BoW 1-gram

Paragraph2vec
https://hemingwang.blogspot.com/2020/08/paragraph2vec.html

B4SE

PSE

-----

Skip-Thought

Quick-Thought

InferSent

MILA SE

Google SE

-----

S3E

SASE

SBERT

RRSE

-----

◎ 8. Attention（NLP）

-----

Attention
http://hemingwang.blogspot.com/2019/10/attention-in-nlp.html
http://hemingwang.blogspot.com/2019/01/attention.html

Attention 1
https://hemingwang.blogspot.com/2020/08/attention-1.html

Visual Attention
https://hemingwang.blogspot.com/2020/08/visual-attention.html

Grad-CAM
https://hemingwang.blogspot.com/2020/08/grad-cam.html

Attention 2
https://hemingwang.blogspot.com/2020/08/attention-2.html

GNMT
https://hemingwang.blogspot.com/2020/08/gnmt.html

-----

NTM

DNC

One Shot MANN

SMA

INTM

-----

MN

DMN

EEMN

KVMN

PN

Set2set

One Shot MN

FSA

-----

◎ 9. ConvS2S（NLP）

-----

ConvS2S
http://hemingwang.blogspot.com/2019/10/understanding-incremental-decoding-in.html 
https://hemingwang.blogspot.com/2019/04/convs2s.html

Key-Value
http://hemingwang.blogspot.com/2019/09/key-value.html

-----

S3L

Context2vec
https://hemingwang.blogspot.com/2020/08/context2vec.html

CoVe

ELLM

ELMo
https://hemingwang.blogspot.com/2019/04/elmo.html

ULMFiT

MultiFiT

-----

◎ 10. Transformer（NLP）

-----

Transformer
http://hemingwang.blogspot.com/2019/10/the-illustrated-transformer.html 
http://hemingwang.blogspot.com/2019/01/transformer.html

GPT-1
https://hemingwang.blogspot.com/2020/01/gpt-1.html

GPT-2

GPT-3

Grover

BERT
https://hemingwang.blogspot.com/2019/01/bert.html

-----

Reformer

LSH

RevNet

Adafactor

-----

Longformer

Synthesizer

Linformer

-----

Part III：Fundamental Topics

-----

Hyper-parameters
https://hemingwang.blogspot.com/2020/09/hyper-parameters.html


-----

◎ 11. Regularization

-----

Regularization

https://hemingwang.blogspot.com/2020/10/regularization.html

-----

◎ 12. Normalization

-----

PCA

https://hemingwang.blogspot.com/2020/10/understanding-principal-components.html

Normalization
http://hemingwang.blogspot.com/2019/10/an-overview-of-normalization-methods-in.html
https://hemingwang.blogspot.com/2019/10/normalization.html

BN
http://hemingwang.blogspot.com/2019/12/bn.html 

WN
http://hemingwang.blogspot.com/2019/12/wn.html 

LN
http://hemingwang.blogspot.com/2019/12/ln.html 

IN
http://hemingwang.blogspot.com/2019/12/in.html 

AIN
https://hemingwang.blogspot.com/2019/12/ain.html

GN
http://hemingwang.blogspot.com/2019/12/gn.html

PN
http://hemingwang.blogspot.com/2019/12/pn.html

UBN
http://hemingwang.blogspot.com/2019/12/ubn.html 

TUBN
https://hemingwang.blogspot.com/2019/12/tubn.html

ResNet-V
https://hemingwang.blogspot.com/2019/12/resnet-v.html

BNHO
http://hemingwang.blogspot.com/2019/12/bnho.html

URBN
http://hemingwang.blogspot.com/2019/12/urbn.html

NormProp
https://hemingwang.blogspot.com/2019/12/normprop.html 

Efficient Backprop
https://hemingwang.blogspot.com/2019/12/efficient-backprop.html

Whitening
https://hemingwang.blogspot.com/2019/12/whitening.html

GWNN
https://hemingwang.blogspot.com/2019/12/gwnn.html

DBN
https://hemingwang.blogspot.com/2019/12/dbn.html

KN
https://hemingwang.blogspot.com/2019/12/kn.html

IterNorm
https://hemingwang.blogspot.com/2019/12/iternorm.html

-----

-----

◎ 13. Optimization

-----

Optimization（一階）
http://hemingwang.blogspot.com/2019/10/an-overview-of-gradient-descent.html
https://hemingwang.blogspot.com/2018/03/deep-learningoptimization.html
https://hemingwang.blogspot.com/2019/01/optimization.html


Optimization（一階與二階）

https://hemingwang.blogspot.com/2020/10/5-algorithms-to-train-neural-network.html

https://hemingwang.blogspot.com/2020/10/optimization.html

-----

一階：

SGD、Momentum、NAG、
AdaGrad、AdaDelta、RMSProp、
Adam、AdaMax、 Nadam、AMSGrad、
RAdam、SMA、Lookahead、EMA、

AdaBound、SWATS、

LAMB、
CLR、SGDR、AdamW、Super-Convergence、

二階：

1. Gradient Descent, Jacobian, and Hessian、Taylor series and Maclaurin series、

2. Newton's Method、Gauss-Newton Method（Gauss-Newton Matrix）、

3. Conjugate Gradient（Gradient Descent + Newton's Method）、

4. Quasi Newton（Template）、SR1、Broyden（Family）、DFA、BFGS、L-BFGS、

5. Levenberg-Marquardt Algorithm（Gradient Descent + Gauss-Newton Method）

6. Natural Gradient Method（Fisher Information Matrix）、

7. K-FAC-G（Gauss-Newton Matrix）、K-FAC-F（Fisher Information Matrix）、

8. Shampoo v1、Shampoo v2、

三：

ADMM、ADMM-S、dlADMM。

-----

SGD
http://hemingwang.blogspot.com/2019/12/sgd.html 

Momentum
https://hemingwang.blogspot.com/2019/12/momentum.html 

NAG
https://hemingwang.blogspot.com/2019/12/nag.html

-----

AdaGrad
https://hemingwang.blogspot.com/2019/12/adagrad.html

AdaDelta
https://hemingwang.blogspot.com/2019/12/adadelta.html

RMSProp
https://hemingwang.blogspot.com/2019/12/rmsprop.html

-----

Adam
https://hemingwang.blogspot.com/2019/12/adam.html

AdaMax
https://hemingwang.blogspot.com/2019/12/adamax.html

Nadam
https://hemingwang.blogspot.com/2019/12/nadam.html

AMSGrad
https://hemingwang.blogspot.com/2019/12/amsgrad.html

-----