a parity check bit does: detects single-bit errors by making total 1s even/odd

Rate-distortion theory: minimum bits to represent data within distortion D

Rate-distortion theory: minimum bits to represent data within distortion D = R(D)

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Bloom filter

Bloom filters check if an element is possibly in a set with high probability, avoiding false negatives

Randomized algorithm

Randomized algorithms use random bits for expected polynomial time

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

BPE tokenization merges frequent adjacent byte pairs iteratively

Encoder vs decoder: encoder sees all tokens bidirectionally, decoder sees only past tokens

Encoder: Sees all tokens bidirectionally; Decoder: Sees only past tokens