Thursday, February 06, 2020

Policy Gradient

Policy Gradient

2020/02/06

-----


-----


-----


-----


 -----


 -----

 https://blog.csdn.net/qq_25037903/article/details/84573048

Wednesday, February 05, 2020

Reinforcement Learning

Reinforcement Learning

2020/02/05

-----


https://mp.weixin.qq.com/s?__biz=MzAwNTAyMDY0MQ==&mid=2652548584&idx=1&sn=358648e930920650e280f0cf3a6b7ad9&chksm=80cd0366b7ba8a70f7d5df4465a9eb3afaacb81c688cce7eec5a0a0ee4d8d22b4372984f9159#rd

-----


https://zhuanlan.zhihu.com/p/21262246

-----


https://sherlockbear.github.io/2019/12/03/Neural-Architecture-Search-with-Reinforcement-Learning/

-----


https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html

-----


https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html

-----


https://zhuanlan.zhihu.com/p/49429128

-----


https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html

Saturday, February 01, 2020

跑步(二六二):22 圈

跑步(二六二):22 圈

2020/01/31

熱身2,跑(5*4)。

-----

22 圈適應中。

Thursday, January 30, 2020

AI 從頭學(2021 年版)

AI 從頭學(2021 年版)

2020/01/01

全方位 AI 課程(精華篇)
http://hemingwang.blogspot.com/2020/01/all-round-ai-lectures-highlight.html

-----


Fig. 2021(圖片來源:Pixabay)。

-----

全方位 AI 課程(六十小時搞定深度學習)
https://hemingwang.blogspot.com/2020/01/all-round-ai-lectures.html

全方位 AI 課程(介紹篇)
https://hemingwang.blogspot.com/2020/01/all-round-ai-lectures-introduction.html

AI Seminar 2020 Taipei
https://hemingwang.blogspot.com/2019/12/ai-seminar-2020-taipei.html

-----

Python
https://hemingwang.blogspot.com/2019/02/python.html

-----

Part I:Computer Vision

◎ Image Classification

Stage 01:LeNet、(AlexNet、ZFNet)

Stage 02:NIN、(SENet、GoogLeNet、VGGNet、PreVGGNet、Highway v1 v2)、(Inception v3 v4、PolyNet)

Stage 03:ResNet v1 v2、(ResNet-D、ResNet-E、ResNet-I、ResNet-Q、ResNet-S、ResNet-W、WRN、ResNeXt、DenseNet、DPN、DLA、Res2Net)

◎ Semantic Segmentation

Stage 04:FCN、(DeconvNet、SegNet、U-Net、U-Net++、DilatedNet、ENet、DRN、FC-CRF、DeepLab v1 v2 v3 v3+、ResNet-38、RefineNet、RefineNet-LW、RefineNet-AA、PSPNet、ICNet、BiSeNet、Fast-SCNN、BlitzNet)

◎ Object Detection

Stage 05:(DPM、SS、R-CNN、SPPNet、Fast R-CNN、OHEM、Faster R-CNN、OverFeat)、YOLOv1、(SSD、DSSD、YOLOv2、ION、R-FCN、SATO、DCNv1、DCNv2、Cascade R-CNN、FPN、STDN、YOLOv3、RON、RefineDet、M2Det、DetNet、TridentNet、OHEM、Focal Loss、GHM、Libra R-CNN、DCRv1、DCRv2、PISA)

// ◎ Dataset

-----

Part II:Natural Language Processing

◎ LSTM
Stage 06:LSTM、(NNLM、Word2vec)

◎ Seq2seq
Stage 07:Seq2seq、(GloVe、fastText)

◎ Attention
Stage 08:Attention、(NTM、KVMN)

◎ ConvS2S
Stage 09:ConvS2S、(ELMo、ULMFiT)

◎ Transformer
Stage 10:Transformer、(GPT-1、BERT、GPT-2)

----- 

Part III:Fundamental Topics

◎ Regularization
Stage 11:(Weight Decay、L2、L1、L0、Dropout、DropConnect、DropPath、Scheduled DropPath、Shake-Shake、ShakeDrop、Spatial Dropout、Cutout、DropBlock、Fast Dropout、RNN Regularization、Variational Dropout、Information Dropout、DropEmbedding、Recurrent Dropout、Zoneout、AWD-LSTM、DropAttention、Mixup、Pairing Samples、AutoAugment)

◎ Normalization
Stage 12:(Batch、Weight、Layer、Instance、Group、Positional)

◎ Optimization
Stage 13:(SGD、Momentum、NAG、AdaGrad、AdaDelta、RMSProp、Adam、AdaMax、Nadam、AMSGrad、Lookahead、RAdam、LAMB、CLR、SGDR、AdamW、Super-Convergence、ADMM、ADMM-S、dlADMM)

◎ Activation Function
Stage 14:(sigmoid、tanh、ReLU、Softplus、LReLU、PReLU、ELU、SELU、GELU、Swish)

◎ Loss Function
Stage 15:

// ◎ Pooling
// ◎ Convolution
// ◎ Automatic Differentiation
// ◎ Back Propagation
// ◎ Computational Graph

-----

Part IV:Advanced Topics

◎ Instance Segmentation
Stage16:(Hypercolumn、MNC、DeepMask、SharpMask、MultiPathNet、InstanceFCN、FCIS)、Mask R-CNN、(MaskX R-CNN、MaskLab、PANet、HTC、RetinaMask、MS R-CNN、YOLACT)

◎ Mobile
Stage17:SqueezeNet、(MobileNet v1 v2 v3、ShuffleNet v1 v2、Xception)

◎ NAS
Stage18:NAS-RL、NASNet(Scheduled DropPath)、EfficientNet、Auto-DeepLab、NAS-FPN、 AutoAugment。

◎ GAN
Stage19:

◎ BERT
Stage20:

-----

Intelligence Science

-----

Intelligence Science
http://hemingwang.blogspot.com/2019/09/intelligence-science.html

-----


Part I:Computer Vision

-----

Computer Vision
https://hemingwang.blogspot.com/2019/10/computer-vision.html

https://hemingwang.blogspot.com/2019/10/gaussiansmooth.html

https://hemingwang.blogspot.com/2019/10/sobeledgedetection.html

https://hemingwang.blogspot.com/2019/10/structuretensor.html

https://hemingwang.blogspot.com/2019/10/nms.html

-----

◎ Image Classification

-----

Image Classification
https://hemingwang.blogspot.com/2019/10/image-classification.html

-----

◎ 1. LeNet(Image Classification)

-----

LeNet
https://hemingwang.blogspot.com/2019/05/trilogy.html
http://hemingwang.blogspot.com/2018/02/deep-learninglenet-bp.html
http://hemingwang.blogspot.com/2017/03/ailenet.html
http://hemingwang.blogspot.com/2017/03/ailenet-f6.html

AlexNet
http://hemingwang.blogspot.com/2017/05/aialexnet.html

ZFNet
http://hemingwang.blogspot.com/2017/05/aikernel-visualizing.html

-----

◎ 2. NINImage Classification)

-----

NIN
http://hemingwang.blogspot.com/2017/06/ainetwork-in-network.html

SENet
https://hemingwang.blogspot.com/2019/10/senet.html

GoogLeNet
http://hemingwang.blogspot.com/2017/06/aigooglenet.html
http://hemingwang.blogspot.com/2017/06/aiconv1.html
http://hemingwang.blogspot.com/2017/08/aiinception.html

VGGNet
http://hemingwang.blogspot.com/2018/09/aivggnet.html 

PreVGGNet
https://hemingwang.blogspot.com/2019/11/prevggnet.html

Highway
http://hemingwang.blogspot.com/2019/11/highway.html

Inception v3
https://hemingwang.blogspot.com/2019/11/inception-v3.html

Inception v4
https://hemingwang.blogspot.com/2019/11/inception-v4.html

FractalNet
https://hemingwang.blogspot.com/2019/12/fractalnet.html

CapsNet
https://hemingwang.blogspot.com/2019/12/capsnet.html

PolyNet
https://hemingwang.blogspot.com/2019/11/polynet.html

-----

◎ 3. ResNetImage Classification)

-----

ResNet
https://hemingwang.blogspot.com/2019/05/vanishing-gradient.html
https://hemingwang.blogspot.com/2019/05/exploding-gradient.html
http://hemingwang.blogspot.com/2019/10/an-overview-of-resnet-and-its-variants.html
https://hemingwang.blogspot.com/2019/10/universal-approximation-theorem.html 
https://hemingwang.blogspot.com/2019/10/understanding-boxplots.html
https://hemingwang.blogspot.com/2019/10/ensemble-learning.html
http://hemingwang.blogspot.com/2018/09/airesnet.html

ResNet-D
https://hemingwang.blogspot.com/2019/11/resnet-d.html

ResNet-E
https://hemingwang.blogspot.com/2019/11/resnet-e.html

ResNet-F
https://hemingwang.blogspot.com/2019/12/resnet-f.html

ResNet-I
https://hemingwang.blogspot.com/2019/12/resnet-i.html

ResNet-Q
https://hemingwang.blogspot.com/2019/12/resnet-q.html

ResNet-S
https://hemingwang.blogspot.com/2019/11/resnet-s.html

ResNet-U
https://hemingwang.blogspot.com/2019/12/resnet-u.html 

ResNet-V
https://hemingwang.blogspot.com/2019/12/resnet-v.html

ResNet-W
https://hemingwang.blogspot.com/2019/12/resnet-w.html

WRN
https://hemingwang.blogspot.com/2019/11/wrn.html

ResNeXt
https://hemingwang.blogspot.com/2019/10/resnext.html

DenseNet
https://hemingwang.blogspot.com/2019/11/densenet.html

DPN
https://hemingwang.blogspot.com/2019/11/dpn.html

DLA
https://hemingwang.blogspot.com/2019/11/dla.html

Res2Net
https://hemingwang.blogspot.com/2019/11/res2net.html

----- 

◎ 3.1. SqueezeNetImage Classification - Mobile)

-----

Mobile
https://hemingwang.blogspot.com/2019/10/mobile.html

SqueezeNet
https://hemingwang.blogspot.com/2019/10/squeezenet.html

MobileNet v1
https://hemingwang.blogspot.com/2019/10/mobilenet-v1.html

ShuffleNet

Xception
https://hemingwang.blogspot.com/2019/10/xception.html

-----

NAS-RL
https://hemingwang.blogspot.com/2019/12/nas-rl.html 

NASNet(Scheduled DropPath)
https://hemingwang.blogspot.com/2019/12/scheduled-droppath.html

EfficientNet 
https://hemingwang.blogspot.com/2019/12/efficientnet.html

Auto-DeepLab
https://hemingwang.blogspot.com/2019/12/auto-deeplab.html

NAS-FPN
https://hemingwang.blogspot.com/2019/12/nas-fpn.html 

AutoAugment
http://hemingwang.blogspot.com/2019/12/autoaugment.html

-----

◎ 4. FCN(Semantic  Segmentation)

-----

一、經典

FCN、DeconvNet、SegNet、U-Net、U-Net++、

-----

二、擴張卷積

DilatedNet、ENet、DRN、Fast FCN、
FC-CRF、
DeepLabv1、DeepLabv2、DeepLabv3、DeepLabv3+、Gated-SCNN、

-----

三、較新

ResNet-38、Tiramisu、
RefineNet、RefineNet-LW、RefineNet-AA、VPLR、
PSPNet、ICNet、BiSeNet、Fast-SCNN、
BlitzNet。

-----

Semantic Segmentation
https://hemingwang.blogspot.com/2019/01/semantic-segmentation.html

-----

FCN
http://hemingwang.blogspot.com/2018/02/deep-learningfcn.html
https://hemingwang.blogspot.com/2019/11/fcn.html

DeconvNet
https://hemingwang.blogspot.com/2019/11/deconvnet.html 

SegNet
https://hemingwang.blogspot.com/2019/11/segnet.html

U-Net
https://hemingwang.blogspot.com/2019/10/u-net.html

U-Net++
https://hemingwang.blogspot.com/2019/11/u-net.html

-----

DilatedNet
https://hemingwang.blogspot.com/2019/11/dilatednet.html

ENet
https://hemingwang.blogspot.com/2019/11/enet.html

DRN
https://hemingwang.blogspot.com/2019/11/drn.html 

FastFCN
https://hemingwang.blogspot.com/2019/12/fastfcn.html

-----

FC-CRF
https://hemingwang.blogspot.com/2019/11/fc-crf.html

DeepLab
https://hemingwang.blogspot.com/2019/10/deeplab.html

DeepLab v1
https://hemingwang.blogspot.com/2019/11/deeplab-v1.html

DeepLab v2
https://hemingwang.blogspot.com/2019/11/deeplab-v2.html

DeepLab v3
https://hemingwang.blogspot.com/2019/11/deeplab-v3.html

DeepLab v3+
https://hemingwang.blogspot.com/2019/11/deeplab-v3-plus.html

Gated-SCNN
https://hemingwang.blogspot.com/2019/12/gated-scnn.html

-----

ResNet-38
https://hemingwang.blogspot.com/2019/11/resnet-38.html

Tiramisu
https://hemingwang.blogspot.com/2019/12/tiramisu.html

RefineNet
https://hemingwang.blogspot.com/2019/11/refinenet.html

RefineNet-LW
https://hemingwang.blogspot.com/2019/11/refinenet-lw.html

RefineNet-AA
https://hemingwang.blogspot.com/2019/11/refinenet-aa.html

VPLR
https://hemingwang.blogspot.com/2019/12/vplr.html

PSPNet
https://hemingwang.blogspot.com/2019/10/pspnet.html

ICNet
https://hemingwang.blogspot.com/2019/11/icnet.html

BiSeNet
https://hemingwang.blogspot.com/2019/11/bisenet.html

Fast-SCNN
https://hemingwang.blogspot.com/2019/11/fast-scnn.html

BlitzNet
https://hemingwang.blogspot.com/2019/11/blitznet.html

-----

◎ 4.1. Instance Segmentation

-----

Hypercolumn

MNC

DeepMask

SharpMask

MultiPathNet

InstanceFCN

FCIS
https://hemingwang.blogspot.com/2019/10/fcis.html

Mask R-CNN
https://hemingwang.blogspot.com/2019/10/mask-r-cnn.html

MaskX R-CNN

MaskLab

PANet

HTC

RetinaMask

MS R-CNN

YOLACT

-----

◎ 5. YOLOv1(Object Detection)

-----
 
Object Detection
https://hemingwang.blogspot.com/2019/10/object-detection.html 

-----

一、求好

DPM、SS、R-CNN、
SPPNet、Fast R-CNN、
Faster R-CNN、

-----

二、求快

OverFeat、YOLOv1、SSD、DSSD、YOLOv2、

-----

三、求好求快

ION、R-FCN、SATO、DCNv1、DCNv2、Cascade R-CNN、
FPN、STDN、YOLOv3、RON、RefineDet、M2Det、
SNIP、SNIPER、AutoFocus、
DetNet、TridentNet、

四、HARD

OHEM、Focal Loss、GHM、
Libra R-CNN、DCRv1、DCRv2、
PISA。

-----

DPM
https://hemingwang.blogspot.com/2019/11/dpm.html

SS
https://hemingwang.blogspot.com/2019/11/ss.html

R-CNN
https://hemingwang.blogspot.com/2019/11/r-cnn.html

SPPNet
https://hemingwang.blogspot.com/2019/11/sppnet.html

Fast R-CNN
https://hemingwang.blogspot.com/2019/11/fast-r-cnn.html

Faster R-CNN
https://hemingwang.blogspot.com/2019/09/faster-r-cnn.html 

-----

OverFeat
https://hemingwang.blogspot.com/2019/11/overfeat.html

YOLOv1
http://hemingwang.blogspot.com/2018/04/deep-learningyolo-v1.html
http://hemingwang.blogspot.com/2018/04/machine-learning-conceptmean-average.html
http://hemingwang.blogspot.com/2018/04/machine-learning-conceptnon-maximum.html
https://hemingwang.blogspot.com/2019/11/yolo-v1.html

SSD
https://hemingwang.blogspot.com/2019/09/ssd.html 

DSSD
https://hemingwang.blogspot.com/2019/11/dssd.html 

YOLOv2
https://hemingwang.blogspot.com/2019/11/yolo-v2.html

-----

ION
https://hemingwang.blogspot.com/2019/11/ion.html

R-FCN
https://hemingwang.blogspot.com/2019/11/r-fcn.html

SATO
https://hemingwang.blogspot.com/2019/10/sato.html 

DCNv1
https://hemingwang.blogspot.com/2019/12/dcn-v1.html

DCNv2
https://hemingwang.blogspot.com/2019/12/dcn-v2.html

Cascade R-CNN
https://hemingwang.blogspot.com/2019/12/cascade-r-cnn.html

FPN
https://hemingwang.blogspot.com/2019/11/fpn.html

STDN
https://hemingwang.blogspot.com/2019/12/stdn.html

YOLOv3
https://hemingwang.blogspot.com/2019/11/yolo-v3.html

RON
https://hemingwang.blogspot.com/2019/12/ron.html

RefineDet
https://hemingwang.blogspot.com/2019/11/refinedet.html

M2Det
https://hemingwang.blogspot.com/2019/10/m2det.html

SNIP
https://hemingwang.blogspot.com/2019/12/snip.html 

SNIPER
https://hemingwang.blogspot.com/2019/12/sniper.html

AutoFocus
https://hemingwang.blogspot.com/2019/12/autofocus.html

DetNet
https://hemingwang.blogspot.com/2019/12/detnet.html

TridentNet
https://hemingwang.blogspot.com/2019/12/tridentnet.html

-----

OHEM
https://hemingwang.blogspot.com/2019/11/ohem.html

Focal Loss
https://hemingwang.blogspot.com/2019/10/retinanet.html

GHM
https://hemingwang.blogspot.com/2019/12/ghm.html

Libra R-CNN
https://hemingwang.blogspot.com/2019/12/libra-r-cnn.html

DCRv1
https://hemingwang.blogspot.com/2019/12/dcr-v1.html

DCRv2
https://hemingwang.blogspot.com/2019/12/dcr-v2.html

PISA
https://hemingwang.blogspot.com/2019/12/pisa.html

-----

◎ 5.1. Dataset

-----

Dataset
https://hemingwang.blogspot.com/2019/10/dataset.html

CALTECH
CIFAR-10
PASCAL VO
COCO
MNIST
ILSVRC 14
Cityspace

-----

◎ One Shot

-----




-----

◎ Face Detection

-----



-----

◎ Face Recognition

-----

DeepFace

DeepID

MobileID

VGGFace

FaceNet

MobileFace

Center Loss

Sphere Face(A-softmax)

CosFace(AM-softmax)

ArcFace

OpenFace

SeetaFace

----- 

◎ Visual Tracking

-----



-----
 
Part II:Natural Language Processing
 
-----

◎ 6. LSTM(NLP)

-----

LSTM
http://hemingwang.blogspot.com/2019/09/understanding-lstm-networks.html
https://hemingwang.blogspot.com/2019/09/lstm.html

-----

◎ NNLM

-----



-----

◎ Word2vec 

-----




-----

◎ 7. Seq2seq(NLP)

-----

Seq2seq
http://hemingwang.blogspot.com/2019/10/word-level-english-to-marathi-neural.html
https://hemingwang.blogspot.com/2019/09/seq2seq.html

-----

◎ 8. Attention(NLP)

-----

Attention
http://hemingwang.blogspot.com/2019/10/attention-in-nlp.html
http://hemingwang.blogspot.com/2019/01/attention.html

-----

◎ 9. ConvS2S(NLP)

-----

ConvS2S
http://hemingwang.blogspot.com/2019/10/understanding-incremental-decoding-in.html 
https://hemingwang.blogspot.com/2019/04/convs2s.html

NNLM

Word2vec

Key-Value
http://hemingwang.blogspot.com/2019/09/key-value.html

ULMFiT


ELMo
https://hemingwang.blogspot.com/2019/04/elmo.html

-----

◎ 10. Transformer(NLP)

-----

Transformer
http://hemingwang.blogspot.com/2019/10/the-illustrated-transformer.html 
http://hemingwang.blogspot.com/2019/01/transformer.html

GPT-1
https://hemingwang.blogspot.com/2020/01/gpt-1.html

BERT
https://hemingwang.blogspot.com/2019/01/bert.html

-----

Part III:Fundamental Topics

-----

◎ 11. Regularization

-----

Regularization
https://hemingwang.blogspot.com/2019/10/an-overview-of-regularization.html
https://hemingwang.blogspot.com/2019/10/regularization.html

-----

Weight Decay、L2、L1、L0、

Early Stopping

Dropout、DropConnect、

DropPath、Scheduled DropPath、
Shake-Shake、ShakeDrop
Spatial Dropout、Cutout、DropBlock

Fast Dropout、RNN Regularization
Variational Dropout、Information Dropout、
rnnDrop、DropEmbedding、Recurrent Dropout、Zoneout、AWD-LSTM、

DropAttention、

Pairing Samples、Mixup。

-----

Weight Decay
https://hemingwang.blogspot.com/2019/12/weight-decay.html

L2
https://hemingwang.blogspot.com/2019/12/l2.html

L1
https://hemingwang.blogspot.com/2019/12/l1.html

L0
https://hemingwang.blogspot.com/2019/12/l0.html

-----

Early Stopping
https://hemingwang.blogspot.com/2019/12/early-stopping.html

-----

// FNN

Dropout
https://hemingwang.blogspot.com/2019/12/dropout.html

Dropconnect
https://hemingwang.blogspot.com/2019/12/dropconnect.html

-----

// CNN

DropPath(FractalNet)
https://hemingwang.blogspot.com/2019/12/droppath.html

ResNet-D
https://hemingwang.blogspot.com/2019/11/resnet-d.html

NASNet(Scheduled DropPath)
https://hemingwang.blogspot.com/2019/12/scheduled-droppath.html

Shake-Shake
https://hemingwang.blogspot.com/2019/12/shake-shake.html

ShakeDrop
https://hemingwang.blogspot.com/2019/12/shakedrop.html

Spatial Dropout
https://hemingwang.blogspot.com/2019/12/spatial-dropout.html

Cutout
https://hemingwang.blogspot.com/2019/12/cutout.html

DropBlock
https://hemingwang.blogspot.com/2019/12/dropblock.html

-----

// RNN

Fast Dropout
https://hemingwang.blogspot.com/2019/12/fast-dropout.html 

RNN Regularization
https://hemingwang.blogspot.com/2019/12/rnn-regularization.html

Variational Dropout
https://hemingwang.blogspot.com/2019/12/variational-dropout.html

Information Dropout
https://hemingwang.blogspot.com/2019/12/information-dropout.html

rnnDrop
https://hemingwang.blogspot.com/2019/12/rnndrop.html

DropEmbbeding
https://hemingwang.blogspot.com/2019/12/dropembbeding.html

Recurrent Dropout
https://hemingwang.blogspot.com/2019/12/recurrent-dropout.html

Zoneout
https://hemingwang.blogspot.com/2019/12/zoneout.html

AWD-LSTM
https://hemingwang.blogspot.com/2019/12/awd-lstm.html

-----

// Self Attention

DropAttention
https://hemingwang.blogspot.com/2019/12/dropattention.html

-----

// Data Augmentation

Pairing Samples
http://hemingwang.blogspot.com/2019/12/pairing-samples.html

Mixup
http://hemingwang.blogspot.com/2019/12/mixup.html

-----

◎ 12. Normalization

-----

Normalization
http://hemingwang.blogspot.com/2019/10/an-overview-of-normalization-methods-in.html
https://hemingwang.blogspot.com/2019/10/normalization.html

BN
http://hemingwang.blogspot.com/2019/12/bn.html 

WN
http://hemingwang.blogspot.com/2019/12/wn.html 

LN
http://hemingwang.blogspot.com/2019/12/ln.html 

IN
http://hemingwang.blogspot.com/2019/12/in.html 

AIN
https://hemingwang.blogspot.com/2019/12/ain.html

GN
http://hemingwang.blogspot.com/2019/12/gn.html

PN
http://hemingwang.blogspot.com/2019/12/pn.html

UBN
http://hemingwang.blogspot.com/2019/12/ubn.html 

TUBN
https://hemingwang.blogspot.com/2019/12/tubn.html

ResNet-V
https://hemingwang.blogspot.com/2019/12/resnet-v.html

BNHO
http://hemingwang.blogspot.com/2019/12/bnho.html

URBN
http://hemingwang.blogspot.com/2019/12/urbn.html

NormProp
https://hemingwang.blogspot.com/2019/12/normprop.html 

Efficient Backprop
https://hemingwang.blogspot.com/2019/12/efficient-backprop.html

Whitening
https://hemingwang.blogspot.com/2019/12/whitening.html

GWNN
https://hemingwang.blogspot.com/2019/12/gwnn.html

DBN
https://hemingwang.blogspot.com/2019/12/dbn.html

KN
https://hemingwang.blogspot.com/2019/12/kn.html

IterNorm
https://hemingwang.blogspot.com/2019/12/iternorm.html

-----

-----

◎ 13. Optimization

-----

Optimization
http://hemingwang.blogspot.com/2019/10/an-overview-of-gradient-descent.html
https://hemingwang.blogspot.com/2018/03/deep-learningoptimization.html
https://hemingwang.blogspot.com/2019/01/optimization.html

-----

SGD、Momentum、NAG、
AdaGrad、AdaDelta、RMSProp、
Adam、AdaMax、 Nadam、AMSGrad、
RAdam、SMA、Lookahead、EMA、
LAMB、
CLR、SGDR、AdamW、Super-Convergence、

ADMM、ADMM-S、dlADMM。

-----

SGD
http://hemingwang.blogspot.com/2019/12/sgd.html 

Momentum
https://hemingwang.blogspot.com/2019/12/momentum.html 

NAG
https://hemingwang.blogspot.com/2019/12/nag.html

-----

AdaGrad
https://hemingwang.blogspot.com/2019/12/adagrad.html

AdaDelta
https://hemingwang.blogspot.com/2019/12/adadelta.html

RMSProp
https://hemingwang.blogspot.com/2019/12/rmsprop.html

-----

Adam
https://hemingwang.blogspot.com/2019/12/adam.html

AdaMax
https://hemingwang.blogspot.com/2019/12/adamax.html

Nadam
https://hemingwang.blogspot.com/2019/12/nadam.html

AMSGrad
https://hemingwang.blogspot.com/2019/12/amsgrad.html

-----

RAdam
https://hemingwang.blogspot.com/2019/12/radam.html

SMA
https://hemingwang.blogspot.com/2019/12/sma.html

Lookahead
https://hemingwang.blogspot.com/2019/12/lookahead.html

EMA
http://hemingwang.blogspot.com/2019/12/ema.html

-----

LAMB
https://hemingwang.blogspot.com/2019/12/lamb.html

-----

CLR
https://hemingwang.blogspot.com/2019/12/clr.html

SGDR
https://hemingwang.blogspot.com/2019/12/sgdr.html

AdamW
https://hemingwang.blogspot.com/2019/12/adamw.html

Super-Convergence
https://hemingwang.blogspot.com/2019/12/super-convergence.html

-----

ADMM
https://hemingwang.blogspot.com/2019/12/admm.html

ADMM-S
https://hemingwang.blogspot.com/2019/12/admm-s.html

dlADMM
https://hemingwang.blogspot.com/2019/12/dladmm.html

-----

◎ 14. Activation Function

-----

Activation Function
https://hemingwang.blogspot.com/2019/10/understanding-activation-functions-in.html

Maxout
https://hemingwang.blogspot.com/2019/12/maxout.html

-----

◎ 15. Loss Function

-----

Loss Function
https://hemingwang.blogspot.com/2019/10/a-brief-overview-of-loss-functions-in.html
http://hemingwang.blogspot.com/2019/05/loss-function.html

-----

◎ Pooling

-----


◎ Convolution

-----
 
Convolution
https://hemingwang.blogspot.com/2019/11/convolution.html

-----

◎ Automatic Differentiation

-----

自動微分

-----

◎ Back Propagation

-----

反向傳播

-----

◎ Computational Graph

-----

計算圖

-----


跑步(二六一):22 圈

跑步(二六一):22 圈

2020/01/29

熱身2,跑(5*4)。

-----

有跑起來。

跑步(二六0):22 圈

跑步(二六0):22 圈

2020/01/23

熱身2,跑(5*4)。

-----

有跑起來。

跑步(二五九):22 圈

跑步(二五九):22 圈

2020/01/21

熱身2,跑(5*4)。

-----

有跑起來。

Tuesday, January 21, 2020

全方位 AI 課程(精華篇)

全方位 AI 課程(精華篇)

2020/01/01

-----


Fig. Start(圖片來源:Pixabay)。

-----

Outline

一、LeNet
二、LeNet Python Lab
三、NIN
四、ResNet
五、FCN
六、YOLOv1

七、LSTM
八、Seq2seq
九、Attention
一0、ConvS2S
一一、Transformer
一二、BERT

一三、Weight Decay
一四、Dropout
一五、Batch Normalization
一六、Layer Nirmalization
一七、Adam
一八、Lookahead

-----


// Amazon.com:《Python Programming  An Introduction to Computer Science》第三版。 (9781590282755)  John Zelle  Books

-----


// Amazon.com:Advanced Engineering Mathematics, 10Th Ed, Isv (9788126554232)  Erwin Kreyszig  Books

-----


// Amazon.com:Discrete - Time Signal Processing (9789332535039)  Oppenheim Schafer  Books

-----


// History of Deep Learning

-----


// History of Deep Learning

-----


// Deep Learning Paper

-----


// Deep Learning Paper

-----


// Deep Learning Paper

-----


// Recent Advances in CNN

-----


-----


-----

◎ LeNet

-----


-----


-----


-----


-----


-----


// 奇異值分解 (SVD) _ 線代啟示錄

-----


// Activation function 到底怎麼影響模型? - Dream Maker

-----


// Activation function 到底怎麼影響模型? - Dream Maker

-----


-----


-----


-----


-----


-----

◎ NIN

-----


-----


-----

◎ SENet

-----


# SENet

-----

◎ ResNet

-----


// DNN tip

-----


# ResNet v1

-----


-----


# ResNet-D

-----


# ResNet v2

-----


# ResNet-E

-----


# ResNet-V

-----

◎ FCN

-----


# FCN

-----


# FCN

-----


-----

◎ YOLOv1

-----


# YOLO v1

-----


# YOLO v1

-----


# YOLO v1

-----

◎ YOLOv3

-----


// Sensors _ Free Full-Text _ Improved UAV Opium Poppy Detection Using an Updated YOLOv3 Model _ HTML

-----

◎ LSTM

-----


-----


-----

◎ Seq2seq

-----



-----

◎ Attention

-----


-----


// Attention and Memory in Deep Learning and NLP – WildML

-----

◎ ConvS2S

-----


-----


-----

◎ Transformer

-----


// Attention  Attention!

-----


-----


-----


-----


-----


-----

◎ BERT

-----


# BERT

-----

ELMo

-----


// Learn how to build powerful contextual word embeddings with ELMo

-----

GPT-1

-----


// LeeMeng - 直觀理解 GPT-2 語言模型並生成金庸武俠小說

-----


# GPT-1

-----

BERT

-----


// LeeMeng - 進擊的 BERT:NLP 界的巨人之力與遷移學習

-----


// LeeMeng - 進擊的 BERT:NLP 界的巨人之力與遷移學習

-----


// LeeMeng - 進擊的 BERT:NLP 界的巨人之力與遷移學習

-----

◎ Weight Decay

-----


// DNN tip

-----


# AdamW

-----

◎ Dropout

-----


// Dropout, DropConnect, and Maxout Mechanism Network. _ Download Scientific Diagram

-----

◎ Batch Normalization

-----


# PN

-----


// [ML筆記] Batch Normalization

-----


// An Intuitive Explanation of Why Batch Normalization Really Works (Normalization in Deep Learning Part 1) _ Machine Learning Explained

-----


# BN

-----

◎ Layer Normalization

-----


// 你是怎样看待刚刚出炉的 Layer Normalisation 的? - 知乎

-----


// Weight Normalization and Layer Normalization Explained (Normalization in Deep Learning Part 2) _ Machine Learning Explained

-----

◎ Adam

-----


// SGD算法比较 – Slinuxer

-----

◎ Lookahead

-----


// Lookahead Optimizer  k steps forward, 1 step back - YouTube

-----



-----



-----

References

AI 三部曲(深度學習:從入門到精通)
https://hemingwang.blogspot.com/2019/05/trilogy.html

-----

Amazon.com:《Python Programming  An Introduction to Computer Science》第三版。 (9781590282755)  John Zelle  Books
https://www.amazon.com/-/zh_TW/dp/1590282752/ 

Amazon.com:Advanced Engineering Mathematics, 10Th Ed, Isv (9788126554232)  Erwin Kreyszig  Books
https://www.amazon.com/-/zh_TW/dp/8126554231/  

Amazon.com:Discrete - Time Signal Processing (9789332535039)  Oppenheim Schafer  Books
https://www.amazon.com/-/zh_TW/dp/9332535035/ 

-----

History of Deep Learning

Alom, Md Zahangir, et al. "The history began from alexnet: A comprehensive survey on deep learning approaches." arXiv preprint arXiv:1803.01164 (2018).
https://arxiv.org/ftp/arxiv/papers/1803/1803.01164.pdf
Deep Learning Paper

LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436.
https://creativecoding.soe.ucsc.edu/courses/cs523/slides/week3/DeepLearning_LeCun.pdf 

Recent Advances in CNN
Gu, Jiuxiang, et al. "Recent advances in convolutional neural networks." Pattern Recognition 77 (2018): 354-377.
https://arxiv.org/pdf/1512.07108.pdf

-----

奇異值分解 (SVD) _ 線代啟示錄
https://ccjou.wordpress.com/2009/09/01/%E5%A5%87%E7%95%B0%E5%80%BC%E5%88%86%E8%A7%A3-svd/

Activation function 到底怎麼影響模型? - Dream Maker
https://yuehhua.github.io/2018/07/27/activation-function/ 

DNN tip
http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/DNN%20tip.pdf

Sensors _ Free Full-Text _ Improved UAV Opium Poppy Detection Using an Updated YOLOv3 Model _ HTML
https://www.mdpi.com/1424-8220/19/22/4851/htm

Attention and Memory in Deep Learning and NLP – WildML
http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/

Attention  Attention!
https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html

[ML筆記] Batch Normalization
http://violin-tao.blogspot.com/2018/02/ml-batch-normalization.html 

An Intuitive Explanation of Why Batch Normalization Really Works (Normalization in Deep Learning Part 1) _ Machine Learning Explained
https://mlexplained.com/2018/01/10/an-intuitive-explanation-of-why-batch-normalization-really-works-normalization-in-deep-learning-part-1/  

你是怎样看待刚刚出炉的 Layer Normalisation 的? - 知乎
https://www.zhihu.com/question/48820040 

Weight Normalization and Layer Normalization Explained (Normalization in Deep Learning Part 2) _ Machine Learning Explained
https://mlexplained.com/2018/01/13/weight-normalization-and-layer-normalization-explained-normalization-in-deep-learning-part-2/  

-----

全方位 AI 課程(六十小時搞定深度學習)
http://hemingwang.blogspot.com/2020/01/all-round-ai-lectures.html 

全方位 AI 課程報名處
https://www.facebook.com/permalink.php?story_fbid=113391586856343&id=104808127714689