```
Image: ChrisRuvolo, Public domain, via Wikimedia Commons
```
to write a fused softmax kernel in Triton: load row, compute max, subtract, exp, sum, divide
`output = exp(row - max_val) / sum(exp(row - max_val))`
tl.load and tl.store do in Triton: read/write tensors from/to GPU global memory
`tl.load` reads tensors from GPU memory; `tl.store` writes tensors to GPU memory
tl.dot does in Triton: block-level matrix multiply using tensor cores
tl.dot performs block-level matrix multiplication using tensor cores in Triton
a Triton kernel is
Triton kernel: Python-based GPU programming that compiles to PTX
a Triton @triton.jit decorator does: compiles a Python function into a GPU kernel
@triton.jit decorator compiles Python function into a GPU kernel
Triton differs from CUDA
Triton uses block-level programming, while CUDA uses thread-level programming
One email a day: 5 concepts + the 5 stories that matter →
Swipe through 100 ML concepts daily
Open TickerNews