Brier score

Brier score measures mean squared error of probability predictions

The Brier score quantifies the accuracy of probabilistic predictions by calculating the mean squared error between predicted probabilities and actual outcomes. It is a strictly proper scoring rule, ensuring that better predictions always receive a lower score. The score is applicable to tasks with mutually exclusive outcomes, where probabilities must sum to one.

Example

If a weather forecast predicts a 70% chance of rain and it rains, the Brier score for this prediction would be calculated based on the squared difference between the predicted probability (0.7) and the actual outcome (1 for rain, 0 for no rain).

Understanding the Brier score helps in evaluating and improving the accuracy of probabilistic predictions in various fields.

Related concepts

denoising score matching does: learns to denoise, which equals learning the score

Denoising score matching learns to denoise by estimating the score (gradient of log probability) of data distributions

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

word error rate (WER) measures: edit distance between predicted and reference transcriptions

Word Error Rate (WER) measures the edit distance between predicted and reference transcriptions

BLEU vs ROUGE: BLEU measures precision of n-grams, ROUGE measures recall

BLEU measures precision of n-grams, ROUGE measures recall

expected calibration error (ECE) measures: gap between confidence and accuracy

Expected Calibration Error (ECE) measures the gap between predicted confidence levels and actual accuracy

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Log-loss penalizes confident incorrect predictions more heavily

One email a day: 5 concepts + the 5 stories that matter →

Swipe through 100 ML concepts daily

Open TickerNews