AlexNet（一）：Paper Translation

 AlexNet（一）：Paper Translation

2021/03/06

-----

ImageNet Classification with Deep Convolutional Neural Networks

深度卷積神經網路的 ImageNet 分類

-----

https://pixabay.com/zh/photos/berlin-alexanderplatz-world-clock-4759124/

-----

Abstract

摘要

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

我們訓練了一個大型的深度卷積神經網路，將 ImageNet LSVRC-2010 競賽中的 120 萬個高解析圖像分成 1000 個不同的類別。在測試數據上，我們的 top-1 和 top-5 錯誤率分別為 37.5％ 和 17.0％，比之前的最佳技術要好得多。此神經網路有 6,000 萬個參數和 650,000 個神經元，它由五個卷積層組成，其中一些後面跟著 max-pooling 層，之後再到三個全連接層，最後是輸出為 1000 類別的 softmax。為了使訓練更快，我們使用了非飽和神經元和卷積運算非常高效的 GPU 實現。為了減少全連接層的過度擬合，我們採用了一種新近開發的正則化方法，稱為“丟包”，這種方法被證明很有效。我們還在 ILSVRC-2012 競賽中使用了該模型的一種變體，並獲得了最高的 top-5 測試錯誤率 15.3％，而第二名只有 26.2％。

-----

1 Introduction

1 引言

Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the currentbest error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.

當前的物件辨識主要使用機器學習。為了提高性能，我們可以收集更大的數據集，學習更強大的模型，並使用更好的技術來防止過度擬合。直到最近，帶標籤圖像的數據集還相對較小-數以萬計的圖像（例如，NORB [16]，Caltech-101 / 256 [8、9] 和 CIFAR-10 / 100 [12]）。使用這種大小的數據集可以有效解決簡單的辨識任務，特別是如果使用保留標籤的轉換來增強它們的話。例如，MNIST 數字識別任務上的當前最佳錯誤率（<0.3％）接近人類的表現 [4]。但是現實環境中的物體表現出相當大的可變性，因此要學會識別它們，有必要使用更大的訓練集。確實，小型圖像數據集的缺點已得到廣泛認可（例如 Pinto 等人 [21]），但直到最近才有可能收集帶有數百萬個圖像的標記數據集。新的更大的數據集包括 LabelMe [23] 和 ImageNet [6]，LabelMe [23] 由成千上萬個完全分割的圖像組成，ImageNet 由超過 1500 萬個帶標籤的高解析圖像組成，於 22,000 多個類別中。

To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

要從數百萬個圖像中認出數千個對象，我們需要一個具有較大學習能力的模型。 但是，物件辨識任務的巨大複雜性意味著，即使像 ImageNet 這樣龐大的數據集也無法解決此問題，因此我們的模型還應該具有很多先驗知識，以彌補我們所沒有的數據。 卷積神經網絡（CNN）是了這類模型之一 [16、11、13、18、15、22、26]。 可以通過改變其深度和寬度來控制它們的容量，並且它們還對圖像的性質（即統計的平穩性和像素依存性的局部性）做出強有力且幾乎正確的假設。 因此，與具有類似大小的層的標準前饋神經網路相比，CNN 的連接和參數要少得多，因此更易於訓練，而其理論上最好的性能可能只會稍差一些。

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.

儘管 CNN 具有吸引人的品質，且儘管其局部架構相對有效，但將它們大規模應用於高解析圖像仍然非常昂貴。 幸運的是，當前的 GPU 與 2D 卷積的高度優化實現相結合，具有足夠的功能來促進訓練大到夠吸引人的 CNN，而最近的數據集（如 ImageNet）包含足夠的帶標籤範例，可以在不嚴重擬合的情況下訓練此類模型。

In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

最後，網路的大小主要受到當前 GPU 可用的內存量以及我們願意接受的訓練時間的限制。 我們的網路需要五到六天的時間來訓練兩個 GTX 580 3GB GPU。 我們所有的實驗都表明，只要有更快的 GPU 和更大的數據集就可以改善我們的結果。

-----

2 The Dataset

2 資料集

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.

ImageNet 是超過 1500 萬張帶標籤的高解析圖像的資料集，這些圖像大約有 22,000 個類別。 圖像從網上收集，並由人工貼標籤者使用亞馬遜的 Mechanical Turk 眾包工具貼上標籤。 從 2010 年開始，作為 Pascal 視覺物件挑戰賽的一部分，每年舉行一次名為 ImageNet 大規模視覺識別挑戰賽（ILSVRC）的競賽。 ILSVRC 使用 ImageNet 的子集，在 1000 個類別中的每個類別中大約包含 1000 張圖像。 總共大約有 120 萬張訓練圖像，50,000 張驗證圖像和 150,000 張測試圖像。

ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.

ILSVRC-2010 是 ILSVRC 唯一具有測試集標籤的版本，因此這是我們執行大部分實驗的版本。 由於我們也在 ILSVRC-2012 競賽中輸入了我們的模型，因此在第 6 節中，我們還將報告關於此版本資料集的結果，而該資料集的測試集標籤不可用。 在 ImageNet 上，通常會報告兩個錯誤率：top-1 和t op-5，其中 top-5 錯誤率是測試圖像中未正確標註的部分，該正確標註不在模型認為最可能的五個標註中 。

ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256  256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

ImageNet 由可變解析度的圖像組成，而我們的系統需要恆定的輸入維數。 因此，我們將圖像下採樣到 256X256的 固定解析度。給定矩形圖像，我們首先對圖像進行縮放，以使較短的一邊的長度為 256，然後從結果的圖像中裁剪出中心 256X256 色塊。 除了從每個像素中減去訓練集上的平均值以外，我們沒有以其他任何方式對圖像進行預處理。 因此，我們在像素的（居中）原始 RGB 值上訓練了我們的網路。

-----

3 The Architecture

3 架構

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

圖2 總結了我們網路的架構。它包含 8 個學習層 - 5 個卷積層和 3 個全連接層。 下面，我們描述網路架構的一些新穎或少見的功能。 根據我們對它們的重要性的評估，對第 3.1 - 3.4 節進行了排序，最重要的是第一個。

-----

3.1 ReLU Nonlinearity

3.1 ReLU 非線性

The standard way to model a neuron’s output f as a function of its input x is with f(x) = tanh(x) or f(x) = (1 + e^-x)^-1. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0; x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.

根據輸入 x 來模擬神經元輸出 f 的標準方法是f（x）= tanh（x）或 f（x）=（1 + e ^ -x）^-1。 就梯度下降的訓練時間而言，這些飽和非線性要比非飽和非線性f（x）= max（0; x）慢得多。 繼 Nair 和 Hinton  [20] 之後，我們將具有這種非線性的神經元稱為整流線性單位（ReLUs）。 帶有 ReLU的 深度卷積神經網路的訓練速度比同等的 tanh 單元快幾倍。 這在圖1 中得到了證明，該圖顯示了對於特定的四層卷積網路，在 CIFAR-10 資料集上達到 25％ 訓練誤差所需的迭代次數。 該圖表明，如果使用傳統的飽和神經元模型，我們將無法使用如此大型的神經網路進行這項工作。

We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity f(x) = |tanh(x)| works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.

我們不是第一個在 CNN 中考慮替代傳統神經元模型的人。 例如，Jarrett 等 [11] 聲稱非線性 f（x）= | tanh（x）| 在對比技術歸一化類型上非常有效，然後在 Caltech-101 資料集上進行局部平均池化。 但是，在此資料集上，主要的問題是防止過度擬合，因此他們觀察到的效果與使用 ReLU 時擬合報告的訓練集的加速能力有所不同。 更快的學習對在大型資料集上訓練的大型模型的性能有很大的影響。

-----

3.2 Training on Multiple GPUs

3.2 在多個 GPU 上進行訓練

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.

單個 GTX 580 GPU 僅具有 3GB 內存，這限制了可以在其上訓練的網路的最大大小。事實證明，有 120 萬個訓練樣本足以訓練太大而無法安裝在一個 GPU 上的網絡。因此，我們將網路配置在兩個 GPU上。當前的 GPU 特別適合跨 GPU 平行化，因為它們能夠直接讀取和寫入彼此的內存，而無需通過主機內存。實際上，我們採用的平行化方案將每個內核的一半（或神經元）放在每個 GPU 上，還有一個額外的技巧：GPU 僅在某些層進行通信。這意味著，例如，第 3 層的內核從第 2 層的所有內核映射中獲取輸入。但是，第 4 層的內核僅從第 3 層中駐留在同一 GPU 上的那些內核映射中獲取輸入。選擇連接模式是交叉驗證的一個問題，但這使我們可以精確地調整通信量，直到它是計算量的可接受的一部分為止。

The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Ciresan et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net 2.

最終的架構與 Ciresan 等人使用的 “ columnar” CNN 的架構有些相似  [5]，只是我們的列不是獨立的（請參見圖2）。 與在一個 GPU 上訓練的每個卷積層中內核數量減少一半的網路相比，該方案分別將我們的 top-1 和 top-5 錯誤率降低了 1.7％ 和 1.2％。 兩 GPU 的網路的訓練時間比單 GPU 網路少一點點 2。

2 The one-GPU net actually has the same number of kernels as the two-GPU net in the final convolutional layer. This is because most of the net’s parameters are in the first fully-connected layer, which takes the last convolutional layer as input. So to make the two nets have approximately the same number of parameters, we did not halve the size of the final convolutional layer (nor the fully-conneced layers which follow). Therefore this comparison is biased in favor of the one-GPU net, since it is bigger than “half the size” of the two-GPU net.

2 在最後的卷積層中，一個 GPU 網路實際上具有與兩個 GPU 網路相同的內核數量。 這是因為大多數網路參數位於第一個完全連接的層中，該層將最後一個卷積層作為輸入。 因此，為了使兩個網路具有大約相同數量的參數，我們沒有將最終卷積層的大小減半（也沒有將隨後的全連接層的大小減半）。 因此，此比較偏向於一個 GPU 網路，因為它大於兩個 GPU 網路的“一半大小”。

-----

3.3 Local Response Normalization

3.3 局部響應規範化

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by ai x;y the activity of a neuron computed by applying kernel i at position (x; y) and then applying the ReLU nonlinearity, the response-normalized activity bi x;y is given by the expression

ReLU 有理想的特性，即它們不需要將輸入規範化即可防止飽和。 如果至少一些訓練樣本對 ReLU 產生了正值的輸入，那麼該神經元就會進行學習。 但是，我們仍然發現以下局部歸一化方案有助於泛化。 用ai x; y表示神經元的活動，該活動是通過在位置（x; y）處應用核 i，然後應用 ReLU 非線性計算得到的，響應歸一化活動 bi x; y由以下表達式給出

where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k; n; , and  are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5,  α = 10^(-4), and β = 0:75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).

其中，總和在相同的空間位置上遍歷 n 個“相鄰”內核映射，而 N 是該層中內核的總數。 內核映射的順序當然是任意的，並且在訓練開始之前確定。 這種響應歸一化實現了一種由實際神經元中發現的類型激發的橫向抑制形式，從而在使用不同核計算的神經元輸出中競爭大型活動。 常數 k; n; ，並且是使用驗證集確定其值的超參數； 我們使用 k = 2，n = 5，α = 10 ^（-4），β = 0:75。 在某些層中應用 ReLU 非線性之後，我們應用了此歸一化（請參見第 3.5 節）。

This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization 3.

該方案與 Jarrett 等人的局部對比度歸一化方案有些相似，[11]，但是我們將其更正確地稱為“亮度歸一化”，因為我們沒有減去平均活性。 響應標準化使我們的 top-1 和t op-5 錯誤率分別降低了 1.4％ 和 1.2％。 我們還在 CIFAR-10 數據集上驗證了該方案的有效性：四層 CNN 在未進行標準化的情況下實現了 13％ 的測試錯誤率，在進行標準化的情況下實現了 11％ 的測試錯誤率 3。

3 We cannot describe this network in detail due to space constraints, but it is specified precisely by the code and parameter files provided here: http://code.google.com/p/cuda-convnet/.

3 由於篇幅所限，我們無法詳細描述此網路，但是它是由此處提供的代碼和參數文件精確指定的：http：//code.google.com/p/cuda-convnet/。

-----

3.4 Overlapping Pooling

3.4 重疊池化

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z x z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2; z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

CNN 中的池化層匯總了同一核圖中的神經元相鄰組的輸出。 傳統上，由相鄰池化單元匯總的鄰域不重疊（例如[17、11、4]）。 更準確地說，可以將池化層視為由間隔為 s個 像素的池化單元的網格組成，每個網格都匯總了以池化單元的位置為中心的大小為 z x z 的鄰域。如果設置 s = z，我們將獲得 CNN 中常用的傳統局部池化。 如果設置 s  < z，則獲得重疊池化。 這就是我們在整個網路中使用的數據，其中 s = 2 和 z = 3。與非重疊方案 s = 2; z = 2 相比，該方案分別將 top-1 和 top-5 錯誤率降低了 0.4％ 和 0.3％。這將產生等效尺寸的輸出。我們通常會在訓練過程中觀察到，具有重疊池化的模型比較不會過擬合。

-----

3.5 Overall Architecture

3.5 總體架構

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.

現在，我們準備描述 CNN 的總體架構。 如圖2 所示，該網包含 8 個帶權重的層；每個層都包含權重。 前五個是卷積的，其餘三個是全連接的。 最後一個完全連接層的輸出被饋送到 1000 路 softmax，後者在 1000 類標籤上產生分佈。 我們的網路將多項式邏輯回歸目標最大化，這等效於在預測分佈下，將正確標籤的對數概率的訓練案例之間的平均值最大化。

The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully-connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.

第二，第四和第五卷積層的內核僅連接到駐留在同一 GPU 上的上一層中的那些內核映射（請參見圖2）。 第三卷積層的內核連接到第二層中的所有內核映射。 全連接層中的神經元連接到上一層中的所有神經元。 響應歸一化層跟隨第一和第二卷積層。 在第 3.4 節中描述的那種最大池化層，緊隨響應歸一化層以及第五個卷積層。 ReLU 非線性應用於每個卷積和全連接層的輸出。

The first convolutional layer filters the 224 x 224 x 3 input image with 96 kernels of size 11 x 11 x 3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 x 5 x 48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 x 3 x 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 x 3 x 192 , and the fifth convolutional layer has 256 kernels of size 3 x 3 x 192. The fully-connected layers have 4096 neurons each.

第一卷積層過濾 224 x 224 x 3 輸入圖像，具有 96 個大小為 11 x 11 x 3 的內核，步幅為 4 像素（這是內核圖中相鄰神經元的感受野中心之間的距離 ）。 第二個卷積層將第一個卷積層的（響應歸一化和合併的）輸出作為輸入，並使用大小為 5 x 5 x 48 的 256 個內核對其進行過濾。第三，第四和第五個卷積層連接到一個 另一個沒有任何中間池或規範化層。 第三卷積層具有 384 個大小為 3 x 3 x 256 的內核，這些內核連接到第二卷積層的（標準化，合併的）輸出。 第四卷積層具有 384 個 3 x 3 x 192 大小的內核，第五卷積層具有 256 個 3 x 3 x 192 大小的內核。全連接層每個都有 4096 個神經元。

-----

4 Reducing Overfitting

4 減少過度擬合

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.

我們的神經網路架構具有 6000 萬個參數。 儘管 ILSVRC 的 1000 個類別使每個訓練樣本在從圖像到標籤的映射上施加 10 位元約束，但事實證明這不足以學習太多參數而又不會出現過度擬合的情況。 下面，我們描述了克服過度擬合的兩種主要方法。

4.1 Data Augmentation

4.1 資料擴充

The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.

減少圖像資料過擬合的最簡單，最常見的方法是使用保留標籤的變換（例如[ 25、4、5]）人為地放大資料集。 我們採用兩種不同的資料擴充形式，這兩種形式都允許通過很少的計算就可以從原始圖像生成轉換後的圖像，因此不需要將轉換後的圖像存儲在磁碟上。 在我們的實現中，當 GPU 訓練上一批圖像時，轉換後的圖像在 CPU 上以 Python 代碼生成。 因此，這些資料擴充方案實際上在計算上是免費的。

The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random 224 x 224 patches (and their horizontal reflections) from the 256 x 256 images and training our network on these extracted patches 4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 x 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.

資料擴充的第一種形式包括生成圖像平移和水平反射。 為此，我們從 256 x 256 張圖像中提取了 224 x 224 個隨機色塊（及其水平反射），並用這些提取的色塊訓練我們的網路 4。這當然使我們的訓練集大小增加了 2048 倍，儘管生成的訓練樣本當然是高度相關的。 沒有這種方案，我們的網路會嚴重過度擬合，這將迫使我們使用較小的網路。 在測試時，網路通過提取五個 224 x 224 色塊（四個角點和中央色塊）及其水平反射（因此總共有十個色塊）進行預測，並平均網路的 softmax 做出的預測十個補丁上的圖層。

4 這就是為什麼圖2 中的輸入圖像為 224 x 224 x 3維的原因。

4 This is the reason why the input images in Figure 2 are 224 x 224 x 3-dimensional.

The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy = [IR xy; IG xy; IB xy]T we add the following quantity:

資料增強的第二種形式包括更改訓練圖像中 RGB 通道的強度。 具體來說，我們在整個 ImageNet 訓練集中對 RGB 像素值集執行 PCA。 對於每個訓練圖像，我們添加找到的主成分的倍數，其大小與相應的特徵值乘以從均值為零且標準偏差為 0.1 的高斯得出的隨機變量成比例。 因此，對於每個 RGB 圖像像素，Ixy = [IR xy; IG xy; IB xy] T，我們添加以下數量：

where pi and λi are ith eigenvector and eigenvalue of the 3 x 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over 1%.

其中，pi 和 λi 分別是 RGB 像素值的 3 x 3 協方差矩陣的第 i 個特徵向量和特徵值，而 αi 是上述隨機變量。 對於特定訓練圖像的所有像素，每個 αi 僅繪製一次，直到再次使用該圖像進行訓練為止，此時將其重新繪製。 該方案近似地捕獲了自然圖像的重要性質，即，對象身份對於明暗的強度和顏色的變化是不變的。 此方案將 top-1 錯誤率降低了 1％ 以上。

-----

4.2 Dropout

4.2 Dropout

Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights.  This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.

結合許多不同模型的預測是減少測試錯誤的非常成功的方法[ 1，3]，但是對於已經花了幾天時間進行訓練的大型神經網路來說，這似乎太昂貴了。 但是，有一個非常有效的模型組合版本，在訓練過程中僅花費約 2 倍的成本。 最近引入的技術稱為“dropout” [10]，包括將每個隱藏神經元的輸出以 0.5 的概率設置為零。 以這種方式“退出”的神經元不會對正向做出貢獻，也不會參與反向傳播。 因此，每次出現輸入時，神經網路都會對不同的體系結構進行採樣，但是所有這些體系結構都會共享權重。由於神經元不能依賴於特定其他神經元的存在，因此該技術減少了神經元的複雜共適應。 因此，被迫學習更穩健的功能，這些功能可與其他神經元的許多不同隨機子集結合使用。 在測試時，我們使用所有神經元，但將它們的輸出乘以 0.5，這是採用指數上的多個 dropout 網路產生的預測分佈的幾何平均值的合理近似值。

We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.

我們在圖2 的前兩個全連接層中使用了dropout。如果沒有 dropout，我們的網路將表現出過大的過擬合。 Dropout 大約會使收斂所需的迭代次數增加一倍。

-----

5 Details of learning

5 學習的細節

We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model’s training error. The update rule for weight w was

我們使用隨機梯度下降訓練了模型，批次大小為 128 個樣本，動量為0.9，權重衰減為 0.0005。 我們發現，少量的權重衰減對於模型的學習很重要。 換句話說，這裡的權重衰減不僅僅是一個正則化器：它可以減少模型的訓練誤差。 權重 w 的更新規則為

We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0.

我們從零均值高斯分佈（標準差為 0.01）初始化每一層的權重。 我們使用常數 1 初始化了第二，第四和第五卷積層以及完全連接的隱藏層中的神經元偏差。此初始化通過為 ReLU 提供正輸入來加速學習的早期階段。 我們用常數 0 初始化其餘層中的神經元偏差。

We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.

我們對所有層使用了相等的學習率，我們在整個訓練過程中手動對其進行了調整。 我們遵循的啟發式方法是，當驗證錯誤率不再隨著當前學習率提高而提高時，將學習率除以 10。 將學習率初始化為 0.01，並在終止之前降低三倍。 我們通過 120 萬張圖像的訓練集對網路進行了大約 90 個週期的訓練，在兩個 NVIDIA GTX 580 3GB GPU上花了五到六天的時間。

-----

6 Results

6 結果

Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%5. The best performance achieved during the ILSVRC-2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are 45.7% and 25.7% with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].

表1 總結了我們在 ILSVRC-2010 上的結果。我們的網路實現了 top-1 和 top-5 測試集錯誤率分別為 37.5％ 和 17.0％ 5。 在 ILSVRC-2010 競賽中獲得的最佳性能分別為 47.1％ 和 28.2％，其方法是對根據不同特徵訓練的六個稀疏編碼模型產生的預測取平均值 [2]，此後，最佳的公開結果是 45.7％ 和 25.7％ 的方法，對兩個分類器的預測值進行平均，該分類器是根據兩種類型的密集採樣特徵計算的，在 Fisher 向量（FV）上訓練的 [24]。

5 The error rates without averaging predictions over ten patches as described in Section 4.1 are 39.0% and 18.3%.

5 如第 4.1 節所述，未對十個補丁進行平均預測的錯誤率分別為 39.0％ 和 18.3％。

We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1% (see Table 2). The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of 15.3%. The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7].

我們也將模型輸入了 ILSVRC-2012 競賽，並在表2 中報告了結果。由於 ILSVRC-2012 測試集標籤不是公開可用的，因此我們無法報告所有嘗試過的模型的測試錯誤率。 在本段的其餘部分，我們將驗證和測試錯誤率互換使用，因為根據我們的經驗，它們的相差不超過 0.1％（請參見表2）。 本文所述的 CNN 的 top-5 錯誤率達到了 18.2％。 平均五個相似 CNN 的預測得出的錯誤率為 16.4％。訓練一個 CNN，在最後一個池化層上再加上一個第六卷積層，以對整個 ImageNet Fall 2011版本（1500萬張圖像，22K 個類別）進行分類，然後在 ILSVRC-2012 上對其進行“微調”，得出的錯誤率為 16.6 ％。 將在整個 2011 年秋季版本中經過預訓練的兩個 CNN 與上述五個 CNN 的預測平均得出錯誤率 15.3％。 次優競賽條目的錯誤率達到 26.2％，該方法對從不同類型的密集採樣特徵計算出的 FV 訓練的幾個分類器的預測取平均值 [7]。

Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are 67.4% and 40.9%, attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].

最後，我們還報告了 ImageNet 2009 年秋季版本的錯誤率，其中包含 10,184 個類別和 890 萬張圖像。 在此資料集上，我們遵循文獻中的慣例，即使用一半的圖像進行訓練，一半使用圖像進行測試。 由於沒有已建立的測試集，因此我們的劃分必定與以前的作者使用的劃分不同，但這不會對結果產生明顯影響。 通過上述網路，我們在此資料集上的 top-1 和 top-5 錯誤率分別為 67.4％ 和 40.9％，但在最後一個池化層上還有一個第六卷積層。 在該資料集上發表的最佳結果分別為 78.1 ％和 60.9％ [19]。

-----

6.1 Qualitative Evaluations

6.1 定性評估

Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).

圖3 顯示了網路的兩個資料連接層所學習的捲積內核。 網路已經學會了各種頻率和方向選擇內核，以及各種有色斑點。 請注意，這是兩個 GPU 所展現出的特性，這是第 3.5 節中描述的受限連接性的結果。 GPU 1 上的內核在很大程度上與顏色無關，而 GPU 2 上的內核在很大程度上是特定於顏色的。 這種特殊化發生在每次運行期間，並且與任何特定的隨機權重初始化（對GPU進行重新編號的模數）無關。

In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph.

在圖4 的左面板中，我們通過在八幅測試圖像上計算其 top-5 的預測，定性地評估網路學到的知識。 請注意，即使偏心的對象（例如左上角的蟎）也可以被網路識別。 大多數 top-5 標籤看起來都是合理的。 例如，只有其他類型的貓才被視為豹子的合理標籤。 在某些情況下（穀物，櫻桃），對於照片的預期焦點有真正的歧義。

Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material.

探查網路視覺知識的另一種方法是考慮由最後 4096 維隱藏層中的圖像引起的特徵激活。 如果兩個圖像產生的特徵激活向量具有較小的歐幾里得距離，則可以說神經網路的較高級別認為它們是相似的。 圖4 顯示了根據此度量，來自測試集的五幅圖像和來自訓練集的六幅圖像，它們與每個圖像最為相似。 請注意，在像素級別，檢索到的訓練圖像在 L2 中通常不接近第一列中的查詢圖像。 例如，檢索的狗和大象以各種姿勢出現。 我們在補充材料中提供了更多測試圖像的結果。

Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying autoencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.

通過使用兩個 4096 維實值向量之間的歐幾里德距離來計算相似度效率不高，但是可以通過訓練自動編碼器將這些向量壓縮為短二進制代碼來提高相似度。 與將自動編碼器應用於原始像素相比，這應該產生一種更好的圖像檢索方法 [14]，該方法不使用圖像標籤，因此有檢索具有相似邊緣模式的圖像的趨勢，而無論它們在語義上是否相似。

-----

7 Discussion

7 討論

Our results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results.

我們的結果表明，大型的深層卷積神經網路能夠使用純監督學習在具有高度挑戰性的資料集上實現創紀錄的結果。 值得注意的是，如果移除某個卷積層，我們的網路性能就會下降。 例如，刪除任何中間層都會導致網路的 top-1 性能損失約 2％。 因此深度對於實現我們的結果確實很重要。

To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.

為了簡化我們的實驗，我們沒有使用任何無監督的預訓練，即使我們希望這會有所幫助，尤其是如果我們獲得足夠的計算能力來顯著增加網路的大小而又未獲得標記資料量的相應增加時。 到目前為止，由於我們擴大了網路並訓練了更久，我們的結果有所改善，但為了匹配人類視覺系統的下側顳葉路徑，我們還有很長的路要走。 最終，我們希望在視頻序列上使用非常大和較深的卷積網路，其中時間結構提供了非常有用的信息，這些信息在靜態圖像中丟失或不那麼明顯。

-----

# AlexNet。被引用 74398 次。較早使用 GPU 的大型卷積神經網路之一，效能比之前有飛躍的提升，成功使用 dropout 避免 overfitting。

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems 25 (2012): 1097-1105.

https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf

-----