NLP（三）：Attention

NLP（三）：Attention

2019/01/18

-----

https://zhuanlan.zhihu.com/p/37601161

# Seq2seq

-----

https://zhuanlan.zhihu.com/p/37601161

# Attention

-----

Fig. 2. An illustration of the attention mechanism (RNNSearch) proposed by [Bahdanau, 2014]. Instead of converting the entire input sequence into a single context vector, we create a separate context vector for each output (target) word. These vectors consist of the weighted sums of encoder’s hidden states.

-----

https://zhuanlan.zhihu.com/p/37601161

-----

https://zhuanlan.zhihu.com/p/37601161

----- 

# Global Attention。[1]。

-----

https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html

# Global Attention and Local Attention。

-----

https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html

-----

https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html

-----

References

◎ 論文

[1] Attention - using GRU

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473 (2014).

https://arxiv.org/pdf/1409.0473.pdf

[2] Global Attention - using LSTM

Luong, Minh-Thang, Hieu Pham, and Christopher D. Manning. "Effective approaches to attention-based neural machine translation." arXiv preprint arXiv:1508.04025 (2015).

https://arxiv.org/pdf/1508.04025.pdf 

[3] Visual Attention

Xu, Kelvin, et al. "Show, attend and tell: Neural image caption generation with visual attention." International conference on machine learning. 2015.

http://proceedings.mlr.press/v37/xuc15.pdf

-----




-----

◎ 英文參考資料

# 綜述
[1] Attention  Attention!
https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html 

# 綜述
# 680 claps
[2] Attention in NLP – Kate Loginova – Medium
https://medium.com/@joealato/attention-in-nlp-734c6fa9d983

# 1.4K claps
Attn  Illustrated Attention - Towards Data Science
https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3

# 1.3K claps
A Brief Overview of Attention Mechanism - SyncedReview - Medium
https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129

# 799 claps
Intuitive Understanding of Attention Mechanism in Deep Learning
https://towardsdatascience.com/intuitive-understanding-of-attention-mechanism-in-deep-learning-6c9482aecf4f

# 126 claps
Understanding Attention Mechanism - Shashank Yadav - Medium
https://medium.com/@shashank7.iitd/understanding-attention-mechanism-35ff53fc328e

Attention and Memory in Deep Learning and NLP – WildML
http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/

-----

◎ 日文參考資料

深層学習による自然言語処理 - RNN, LSTM, ニューラル機械翻訳の理論 - ディープラーニングブログ
http://deeplearning.hatenablog.com/entry/neural_machine_translation_theory

-----

◎ 簡體中文參考資料

# 綜述
Attention-Mechanisms-paper_Attention-mechanisms-paper.md at master · yuquanle_Attention-Mechanisms-paper · GitHub
https://github.com/yuquanle/Attention-Mechanisms-paper/blob/master/Attention-mechanisms-paper.md

深度学习中的注意力模型（2017版） - 知乎
https://zhuanlan.zhihu.com/p/37601161

自然语言处理中的Attention Model：是什么及为什么 - 张俊林的博客 - CSDN博客
https://blog.csdn.net/malefactor/article/details/50550211



目前主流的attention方法都有哪些？ - 知乎
https://www.zhihu.com/question/68482809/answer/264632289

# 110 claps
自然语言处理中注意力机制综述 - 知乎
https://zhuanlan.zhihu.com/p/54491016 

# 14 claps
NLP硬核入门-Seq2Seq和Attention机制 - 知乎
https://zhuanlan.zhihu.com/p/73589030

注意力机制（Attention Mechanism）在自然语言处理中的应用 - Soul Joy Hub - CSDN博客
https://blog.csdn.net/u011239443/article/details/80418489

【NLP】Attention Model（注意力模型）学习总结 - 郭耀华 - 博客园
https://www.cnblogs.com/guoyaohua/p/9429924.html
 
-----

◎ 繁體中文參考資料

# 486 claps
[1] Seq2seq pay Attention to Self Attention  Part 1(中文版)
https://medium.com/@bgg/seq2seq-pay-attention-to-self-attention-part-1-%E4%B8%AD%E6%96%87%E7%89%88-2714bbd92727

-----

◎ 代碼實作

Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.0.0.dev20181228 documentation
https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

PyTorch 1.1 Tutorials   テキスト   Sequence to Sequence ネットワークと Attention で翻訳 – PyTorch
http://torch.classcat.com/2019/07/20/pytorch-1-1-tutorials-text-seq2seq-translation/

# 832 claps
Attention in Deep Networks with Keras - Towards Data Science
https://towardsdatascience.com/light-on-math-ml-attention-with-keras-dc8dbc1fad39