FT（四）：Activation Function

FT（四）：Activation Function

2019/01/17

-----

Fig. Activation Function（圖片來源）。

-----

Sigmoid
tanh

ReLU
LReLU
RReLU
Softplus

ELU
SELU
GELU

Swish
Swish-1

-----

-----

----- 

// 神经网络激励函数的作用是什么？有没有形象的解释？ - 知乎

-----

Fig. Derivative of Activation Functions
// Activation Functions in Neural Networks – Towards Data Science

-----

導數小於 1，梯度消失。
初始值過大，梯度爆炸。

-----

References

[1] Activation functions

Ramachandran, Prajit, Barret Zoph, and Quoc V. Le. "Searching for activation functions." (2018).

https://openreview.net/pdf?id=SkBYYyZRZ

[2] Activation functions and others

Gu, Jiuxiang, et al. "Recent advances in convolutional neural networks." arXiv preprint arXiv:1512.07108 (2015).

https://arxiv.org/pdf/1512.07108.pdf

Understanding Activation Functions in Neural Networks
https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

Activation Functions in Neural Networks – Towards Data Science
https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6

Activation functions and it’s types-Which is better
https://towardsdatascience.com/activation-functions-and-its-types-which-is-better-a9a5310cc8f

6 种激活函数核心知识点，请务必掌握！
https://mp.weixin.qq.com/s/Cvf3ReuCKSugp9lrZqlaqg

神经网络激励函数的作用是什么？有没有形象的解释？ - 知乎
https://www.zhihu.com/question/22334626

深度學習：使用激勵函數的目的、如何選擇激勵函數 Deep Learning   the role of the activation function _ Mr. Opengate
https://mropengate.blogspot.com/2017/02/deep-learning-role-of-activation.html

Activation function 到底怎麼影響模型？ - Dream Maker
https://yuehhua.github.io/2018/07/27/activation-function/