GPT-2 Architecture

def. GPT-2 is OpenAI’s first public Large Language Model

Note

• GPT in 60 Lines of NumPy | Jay Mody
• But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - YouTube and Attention in transformers, visually explained | Chapter 6, Deep Learning - YouTube

Parameters §

Hyperparameters §

• Context Length: number of tokens allowed in input
  • Sequence length < Context Length. Sequence is the actual length of text input
• Vocabulary Size: the tokenizer’s size of vocabulary
• Embedding Dimension: the number of dimensions in the word embedding
• Head Count: number of attention heads in each attention layer

Learnable Parameters §

• wte (Vocab Size → Embedding Dimension): lookup table from token to embedding
• wpe (Context Length → Embedding Dimension): lookup table for positional embedding
• $\gamma ,\beta$ in the Normalization layer
• w,b (N → N), (N): Weights and biases in the linear (=linear) layers
• Attention
  • Query, Key, and Value matricies, each (Sequence Length → Embed. Dim.):

Word Embeddings §

1. Given a string of text, a tokenizer will split this text into chunks (tokens)
2. Learnable embedding matrix wte and wpe encodes this into an embedding These vectors $v_1,v_2$, embedded in the embedding space will:

• Be similar in meaning if cosine similarity $-1<\frac{v_1\cdot v_2}{|v_1||v_2|}<1$ is larger
• Difference in vectors $v_2-v_1$ i.e. the direction embed meaning that is the “difference in meaning” between $v_1$ and $v_2$

Attention §

1. Each token in string is multiplied with a Query matrix $Q_1=W_Q\cdot v_1$
   1. The resulting vector encodes the question, e.g. “are there adjectives in front of me?”
   2. The resulting vector
2. Each token in string is multiplied with a Key matrix $K_1=W_K\cdot v_1$
   1. The resulting vector encodes the answer, e.g. “I’m an adjective!”

$$Q=\left[\begin{array}{ccc}—& Q_1& —\\ —& Q_2& —\\ & ⋮& \\ —& Q_n& —\end{array}\right],K=\left[\begin{array}{ccc}—& K_1& —\\ —& K_2& —\\ & ⋮& \\ —& K_n& —\end{array}\right],V=\left[\begin{array}{ccc}—& V_1& —\\ —& V_2& —\\ & ⋮& \\ —& V_n& —\end{array}\right]$$

3. The cosine similarity of a query-key pair encodes “how well the question is answered”
   1. each column of the result passes thru a Softmax and Sigmoid function

$$\text{softmax}\left(\frac{\stackrel{\text{ dot similarity }}{\overbrace{Q{K}^T}}}{\underset{\text{ nice to compute }}{\underbrace{\sqrt{d_k}}}}\right)V$$