24. Distillation and Merging

Distillation

Learning objectives

• Finish module on fine tuning
• Learn to train a “student” with a “teacher”

• It would be nice to have a simple tutorial here

Sources

Following the Gen AI Handbook, we looked at

• blog post by Victor Sanh (Hugging Face)
• blog post by Matt Casey (Snorkel AI)
• blog post by Cheng-Yu Hsieh, et al. (Google Research)

Math Updates

Temperature

Another hyperparameter is the softmax temperature

\[p_{i} = \frac{exp(z_{i}/T)}{\sum_{j} exp(z_{j}/T)}\]

• \(T \rightarrow 0\): one-hot target vector
• \(T \rightarrow \infty\): uniform distribution (random guessing)

KL Loss

Kullback-Leibler loss is defined as

\[\begin{array}{rcl} KL(p||q) & = & \text{E}_{p}[\log \frac{p}{q}] \\ ~ & = & \displaystyle\sum_{i} p_{i} \cdot \log(p_{i}) - \sum_{i} p_{i} \cdot \log(q_{i}) \end{array}\]

BERT

Transformers

• Encoder-only: BERT

bidirectional encoder representations from transformers

• Decoder-only: GPT

generative pre-trained transformer

Encoders vs Decoders

BERT vs GPT

• image source: Ronak Verma

Applications

• BERT

  • text classification
  • data labeling
  • recommender
  • sentiment analysis

• GPT

  • content generation
  • conversational chatbots

Distillation

Fine Tuning

distillation motivation

• image source: Google Research

Teacher and Student

teacher and student

Researchers at the University of California at Berkeley found a way to change this, which they call “context distillation.” They built heavily-engineered prompts that ended with simple questions, such as “add these numbers.” Then, they stripped the prompt of its engineering and reduced the response to only its final answer to create a new data set that they used to fine-tune the model.

• image source: Snorkel AI

Chain of Thought

chain of thought

• image source: Snorkel AI

DistilBERT

DistilBERT Performance

DistilBERT performance