autoencoders learn the data manifold

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

the embedding layer does: maps discrete token IDs to dense learned vectors

Embeddings convert token IDs to dense vectors for neural network processing

weight tying does in language models: shares embedding and output projection matrices

Tying reduces the number of parameters by sharing embedding and output projection matrices

kernel fusion reduces memory bandwidth bottleneck

Kernel fusion reduces memory bandwidth bottleneck by combining multiple operations into a single kernel, minimizing data transfers

Vector quantization

Product quantization compresses vectors by splitting them into subvectors and quantizing each subvector independently

soft targets carry more information than hard labels: they encode class similarities

Soft targets carry more information than hard labels because they encode class similarities