Dictionary of Applied Machine Learning

probability distribution

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

Definition

P-methodA classifier that flags incoming emails as spam is fitted to one collection of emails, and the emails it will be shown next are a different collection. More generally, consider a machine learning (ML) method that reads in a dataset $\dataset = \big\{ \datapoint^{(1)}, \,\ldots, \,\datapoint^{(\samplesize)} \big\}$ and delivers an output, such as a hypothesis $\learnthypothesis$ or the prediction it produces for a given data point. Feeding a different dataset into the same method delivers a different output. One way to analyze the method is therefore to study how its output behaves when the dataset consists of realizations of independent and identically distributed (i.i.d.) random variables (RVs) $\datapoint^{(1)}, \,\ldots, \,\datapoint^{(\samplesize)} \sim \probdist$ with a common probability distribution $\probdist$ (see Fig. 1).

P-typicalThe probability distribution $\probdist$ is what such an analysis rests on. It decides which data points are typical, and thereby how the output of the method varies from one dataset to the next. Two numbers attached to the method are then RVs in their own right: the training error it achieves on the dataset it was given, and the risk it incurs on a fresh data point drawn from the same $\probdist$. Generalization is the statement that these two are close, and it is a statement about $\probdist$ before it is a statement about the method. Note that $\probdist$ is never observed; only the dataset is.

P-specifyHow a probability distribution is specified depends on the values the RV takes. The probability distribution of a binary RV $\truelabel \in \{0,1\}$ is fully specified by the single probability $\prob{\truelabel = 0}$, since $\prob{\truelabel=1} = 1-\prob{\truelabel=0}$; more generally, an RV with finitely many values is specified by a probability mass function (pmf). The probability distribution of a continuous real-valued RV $\feature \in \reals$ might be specified by a probability density function (pdf) $\pdf{\feature}{\cdot}$, for which $\prob{ \feature \in [a,b] } \approx \pdf{\feature}{a} \, |b-a|$ holds for a short interval $[a,b]$. In the most general case, the probability distribution of an RV $\datapoint$ defined on a probability space $(\samplespace, \sigmaalgebra, \prob{\cdot})$ is the measure that $\datapoint$ induces on the set of values it takes: it assigns to a measurable set $\genericevent$ of those values the probability $\prob{\datapoint \in \genericevent}$. A measure obtained this way assigns the value $1$ to the set of all values, and is therefore called a probability measure (Billingsley, 1986; Gray, 2009).

See also: independent and identically distributed, realization, random variable, probability, probability density function, probability mass function, probability measure, probability space, expectation, generalization.

References

1. Billingsley (1986). Probability and Measure. Wiley.
2. Gray (2009). Probability, Random Processes, and Ergodic Properties. Springer Science+Business Media. doi.org/10.1007/978-1-4419-1090-5

Cite this entry

@misc{dictml_probdist,
  author = {Jung, Alexander},
  title = {probability distribution},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-21},
  url = {https://dictionaryofml.org/terms/probdist.html}
}