Self Attention for Convolutional Neuronal Networks

Self Attention can help Neuronal Networks to understand long-term dependencies. With normal convolutions, this can be difficult, as the receptive field is small.

Self-Attention GAN (SAGAN) by Zhang et al. (2019) introduces a self-attention mechanism to convolutional GANs leading to clearer features. The issue is that convolutions are good at local information, but the receptive fields are not large enough to include larger structures. Instead of increasing filter sizes, we calculate self-attention.

Structure of Self-Attention in CNN-Networks [Adapted from: Zhang et al. (2019)]

$f$, also called the Query Layer (produced by a size 1 conv) is transposed and then combined with $g$ the Key layer via Dot Product to form the attention map. The Attention map defines the attention score between two pixels. This means how much attention do we pay to region $i$ when synthesizing $j$. By combining $h$ and the Attention map (via dot product), one gets the Attention Feature Maps (or Attention Weights).

To calculate the output we add the attention weight (scaled by a factor $\gamma$) to the input feature $\boldsymbol{y}_{\boldsymbol{i}}=\gamma \boldsymbol{o}_{\boldsymbol{i}}+\boldsymbol{x}_{\boldsymbol{i}}$. By starting with $\gamma = 0$, we focus on local neighorhoods and then increase it to shift attention to the bigger contexts.

Check out the fastai-Code to get an idea for the implementation.

Bibliography

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, 7354–7363. PMLR, 2019. ↩ ^{1 2}