fused kernels do

kernel fusion reduces memory bandwidth bottleneck

Kernel fusion reduces memory bandwidth bottleneck by combining multiple operations into a single kernel, minimizing data transfers

operator fusion does at the compiler level: merges adjacent ops to reduce memory traffic

Operator fusion merges adjacent operations to optimize execution and reduce memory traffic

Overlapping subproblems

Dynamic programming solves overlapping subproblems by storing results of subproblems to avoid redundant calculations

Triton auto-tunes BLOCK_SIZE: different sizes optimize for different hardware

Triton auto-tunes BLOCK_SIZE for hardware efficiency, optimizing memory access patterns and computational throughput

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

Flashbulb memory

Flashbulb memories are vivid but not always accurate