What the compute-optimal training ratio is: roughly 20 tokens per parameter

What BPE tokenization does: iteratively merges the most frequent byte pairs

BPE tokenization merges the most frequent byte pairs iteratively to create subword units

What the context window limit means: maximum number of tokens the model can process at once

Context window limit restricts the model's input size to a fixed number of tokens for processing

What AdaGrad does: divides learning rate by sqrt of sum of squared gradients

AdaGrad adapts learning rates based on historical gradients, reducing for frequently updated features

Greedy vs beam search decoding: greedy picks best token, beam maintains k candidates

Greedy decoding selects one token, while beam search retains multiple candidates

Reed-Solomon error correction: What is the mathematical formula representing the minimum number of redundant symbols required to correct a given number of symbol errors in a Reed-Solomon code?

Minimum redundant symbols = (2t + 1) * k, where t = (number of symbol errors)/(2t + 1) and k = (codeword length - data length)

What is the primary objective of using the gradient descent optimization algorithm in training machine learning models?

Minimize the loss function to find optimal model parameters