gradient checkpointing trades: recomputes activations to save memory

gradient accumulation simulates larger batch sizes without more memory

Gradient accumulation reduces memory usage by dividing a large batch into smaller mini-batches, accumulating gradients before updating model weights

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Overlapping subproblems

Dynamic programming solves overlapping subproblems by storing results of subproblems to avoid redundant calculations

weight initialization matters: Xavier/He init keeps activation variance ≈ 1 across layers

Weight initialization stabilizes learning by maintaining consistent activation variance

Masking (behavior)

Causal masking prevents attention to future tokens in the decoder

Flashbulb memory

Flashbulb memories are vivid but not always accurate