19. Instruct Fine-Tuning

Learning objectives

• Introduce Fine Tuning
• Motivate Instruct Fine Tuning

• The non-math side of training LLMs

Sources

Following the Gen AI Handbook, we looked at

• blog post by Sebastian Ruder (Meta)
• video by Shayne Longpre (Google)

Fine Tuning

Module

fine tuning

• “… to add a small set of parameters to a pretrained LLM. Only the newly added parameters are finetuned while all the parameters of the pretrained LLM remain frozen.” — Sebastian Raschka (author, article)

Where were we?

Wei, et al., 2022

Instruct Fine Tuning

Definition Seeking

getting definitions

Instruction Guidance

providing instructions

Alignment Tuning = Instruction Tuning

• open-ended tasks
• creative generation
• human feedback

Instruction Data

• Mixing few-shot settings

• Task diversity

• Data Augmentation

  • e.g. turning a question-answering task into a question-generation task

• Mixing weights

Recent Developments

Natural Instructions: Q

question generation, Mishra et al., 2022

193k instruction-output examples sourced from 61 existing English NLP tasks. Crowd-sourcing instructions from each dataset are aligned to a common schema. Instructions are thus more structured compared to other datasets.

Natural Instructions: A

answer generation, Mishra et al., 2022

Unnatural Instructions

Honovich, et al., 2023

Exemplars

Flan 2022

Input Inversion

improve task diversity with input inversion