Twelve days ago, on June 12, a paper with an unusually provocative title landed on arXiv: "Attention Is All You Need", authored by eight researchers from Google Brain and Google Research (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin). I printed it out at work the following morning and have spent the last two weeks taking apart its equations over multiple cups of coffee.
For anyone tracking natural language processing (NLP), the proposition feels jarring. Late last year, Google made headlines by migrating Google Translate to GNMT (Google Neural Machine Translation), a formidable engineering feat comprising eight stacked LSTM layers in the encoder and eight in the decoder, packed with residual connections and additive attention mechanisms. Just weeks ago, in May, Facebook AI Research introduced ConvS2S, attempting to replace recurrence with convolutional layers to gain parallel execution speed on GPUs.
Right when the field was fiercely debating whether the future belonged to deep LSTMs or temporal convolutions, this Google team stepped up with something radical: discard both recurrence and convolutions entirely. They christened their architecture the Transformer, asserting that the attention mechanism is not merely an auxiliary feature to remember sequences, but the sole engine required to process human language.
The Bottleneck of Sequential Computation
The dominance of recurrent neural networks (RNN, GRU, and LSTM) rested on an intuitive assumption: language is inherently temporal. We read and listen word by word, from left to right. Consequently, models computed their hidden state $h_t$ by chaining each incoming token with the memory of the preceding step:
$$ h_t = f(h_{t-1}, x_t) $$
That temporal dependency creates two severe bottlenecks when confronting actual hardware:
- Incompatibility with GPU parallelism: To compute the fiftieth word in a sentence, the graphics card must sit idle waiting for the previous forty-nine steps to finish sequentially. Modern GPUs, built to crunch linear algebra and thousands of simultaneous matrix operations, spend substantial clock cycles stalled by serial execution.
- Signal decay across long distances: For information from the first token to reach the last, the signal must survive dozens of matrix multiplications and non-linear activation functions. Even with LSTM gating mechanisms, the hidden state vector struggles to retain distant syntactic dependencies.
The Transformer discards this sequential paradigm: it feeds the entire sentence into the network at once. Rather than iterating word by word through a forced temporal loop, it calculates cross-dependencies across all words simultaneously. The path length between any arbitrary pair of tokens within a layer drops from $O(n)$ to exactly $O(1)$.
The Mechanics of Self-Attention
At the mathematical core of the Transformer lies Scaled Dot-Product Attention.
The formulation draws on a classic information retrieval metaphor. Every word in the input sequence projects three distinct vectors via weight matrices learned during training: a Query ($Q$), a Key ($K$), and a Value ($V$).
Consider processing the sentence:
"The park bank is broken."
To resolve the contextual meaning of "bank", its query vector $Q$ performs a dot product against the key vectors $K$ of every word in the sentence. The higher the vector affinity between two concepts in that context, the larger the resulting score.
The paper's core formula is:
$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$
The scaling factor $\sqrt{d_k}$ addresses a purely numerical issue: when the vector dimension ($d_k$) is large, the dot products grow substantially in magnitude, pushing the softmax function into flat saturation zones where gradients become vanishingly small. Dividing by the square root keeps variance at unity and stabilizes training.
import numpy as np
def scaled_dot_product_attention(Q, K, V, mask=None):
"""Conceptual implementation of Scaled Dot-Product Attention (Vaswani et al., 2017)."""
d_k = Q.shape[-1]
# 1. Matrix multiplication between Queries (Q) and transposed Keys (K)
scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(d_k)
# 2. Optional masking for decoder (prevents looking ahead at future tokens)
if mask is not None:
scores = np.where(mask == 0, -1e9, scores)
# 3. Softmax over the last dimension to normalize affinities between 0 and 1
exp_scores = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
attention_weights = exp_scores / np.sum(exp_scores, axis=-1, keepdims=True)
# 4. Weighted matrix product against Values (V)
context_vectors = np.matmul(attention_weights, V)
return context_vectors, attention_weights
Multi-Head Attention: Distinct Perspectives in Parallel
Instead of computing a single attention pass across a 512-dimensional space, the authors project queries, keys, and values into $h = 8$ distinct representation subspaces of dimension $d_k = d_v = 64$.
This allows the network to jointly attend to different types of syntactic and semantic relationships simultaneously. While one attention head focuses on linking pronouns to distant subjects, another can capture grammatical agreement or prepositional phrases. Upon completion, the eight outputs are concatenated and projected back to the primary model dimension through an output matrix.
The Order Enigma: Positional Encodings
Discarding recurrence and convolutions introduces an immediate hurdle: matrix multiplications carry no intrinsic sense of temporal order. To a pure self-attention layer, "the dog bit the cat" and "the cat bit the dog" contain identical token sets and yield identical representations. The model is order-invariant.
To restore sequence awareness without resorting to recurrent loops, the authors inject Positional Encodings directly into the input embeddings. Rather than learning fixed vectors, they leverage sinusoidal functions across varying frequencies:
$$ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right) $$ $$ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right) $$
Each position dimension corresponds to a distinct wavelength, functioning like a continuous geometric clock. This mathematical formulation allows the model to extrapolate relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$.
Translation Benchmarks and Hardware Costs
The results on the WMT 2014 translation benchmark are unambiguous: - On English-to-German, the big Transformer reaches a BLEU score of 28.4, outperforming previous state-of-the-art results (including multi-model ensembles) by more than 2.0 BLEU. - On English-to-French, it sets a record score of 41.0 BLEU.
What truly stands out from an infrastructure perspective is the drastic reduction in training compute. While GNMT required weeks of compute across substantial hardware clusters, the base Transformer was trained in roughly 12 hours on eight NVIDIA Tesla P100 GPUs. The big model completed training in 3.5 days.
By removing the sequential barrier, GPUs can finally saturate their matrix computation pipelines with entire text blocks in parallel. Google has open-sourced the implementation under the tensor2tensor repository on GitHub, enabling researchers to inspect and train the architecture without proprietary frameworks.
What This Signals for the Field
This paper was engineered strictly for machine translation using a six-layer encoder and six-layer decoder stack. However, the architectural implications reach far beyond translation.
If deep linguistic representation can be solved via algebraic matrix products and normalization layers, the compute bottleneck that held back NLP has been breached. Teams with access to extensive document corpora and GPU clusters will soon train architectures with parameter counts unthinkable under recurrent designs.
Ditching the temporal axis in language computing sounded like heresy; after reading this paper, recurrence appears to be living on borrowed time.