the system prompt does: sets persistent behavior instructions for the conversation

Reasoning model

RLMs excel in logic, math, and programming tasks

Prompt engineering

The GenAI model learns tasks from examples in the prompt

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tunes on (instruction, response) pairs

the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles

[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction

ring attention does: distributes long sequences across multiple devices

Ring attention distributes long sequences across multiple devices

Masking (behavior)

Causal masking prevents attention to future tokens in the decoder