Dictionary of Applied Machine Learning

interpretability

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

The interpretability of a machine learning (ML) method is the extent to which a human user can comprehend its behavior. Interpretability is closely related to predictability: the user should be able to anticipate the predictions of the method on a test set. Interpretability differs from explainability, which concerns understanding with the help of provided explanations.

Definition

Consider a training set of monthly sales and a straight line fitted through it. Reading next month's prediction off that line takes a slope, an intercept and one multiplication, and a user can carry the computation out on paper. The same prediction delivered by a deep net arrives with no account a user could follow. Interpretability is what separates the two.

Interpretability is often defined, informally, as the ability to explain, or to present in understandable terms, to a human (Doshi-Velez and Kim, 2017). The international terminology standard for artificial intelligence (AI) does not define interpretability (Standardization and Commission, 2022). Its closest notion is predictability, the property of an artificial intelligence system (AI system) that enables reliable assumptions by users about the predictions delivered by a machine learning (ML) method (Standardization and Commission, 2022, Sect. 3.5.8 and 5.15.7).

B-linAn ML method is interpretable for a human user if they can comprehend the computational process of the method. Comprehension of a computation cannot be observed directly, so interpretability is judged through predictability: the user anticipates the predictions the method delivers, and the evaluations proposed in the literature measure how often those anticipations are right (Chen et al., 2018; Colin et al., 2022; Doshi-Velez and Kim, 2017; Hase and Bansal, 2020). Predictability is the weaker of the two: the terminology standard notes that a user may rely on the predictions of an AI system without being able to explain the factors behind them (Standardization and Commission, 2022, Sect. 5.15.7). A method whose computation the user can follow is predictable to that user; a method whose predictions the user anticipates need not be one whose computation they could carry out.

B-kinkFig. 1 depicts a training set and a test set. The training set is used by two different methods that learn hypotheses $\learnthypothesis$ and $\learnthypothesis'$. The ML method producing the hypothesis $\learnthypothesis$ is interpretable to a human user familiar with the concept of a linear map. Since $\learnthypothesis$ is a linear map, the user can anticipate the predictions of $\learnthypothesis$ on the test set. In contrast, the ML method producing $\learnthypothesis'$ is less interpretable: the behavior of $\learnthypothesis'$ deviates from the user's expectations.

Figure 1 of the entry interpretability
Figure 1: Predictability, the weaker half: a human user provides pseudo-labels (squares) for the test set $\dataset'$ by extrapolating the linear trend of the training set $\dataset$ (circles). The predictions of the hypothesis $\learnthypothesis$ agree with these pseudo-labels, while the predictions of another hypothesis $\learnthypothesis'$ deviate from them. The user anticipates outputs here without carrying out the computation, which comprehending the method would additionally require
The AI Risk Management Framework of the US National Institute of Standards and Technology distinguishes interpretability from explainability (Standards and Technology, 2023), the property pursued by explainable artificial intelligence (XAI) (Gunning and Aha, 2019). In contrast to interpretability, explainability requires that an explanation is provided along with each prediction. These explanations may take the form of saliency maps or reference examples from the training set. A challenge for XAI methods is that a computed explanation can be unfaithful to the learned hypothesis and thereby mislead the user (Rudin, 2019).

Interpretability can also be obtained by decomposing a learned hypothesis into components that are themselves interpretable (Lipton, 2018). The hypothesis $\learnthypothesis$ in Fig. 1 decomposes into a slope and an intercept: the slope states how the prediction changes with the feature. A trained deep net lacks such a decomposition, since the activation of an individual neuron typically does not correspond to a human-interpretable concept. Mechanistic interpretability aims to recover interpretable components from a trained deep net (Olah et al., 2020; Elhage et al., 2021) by decomposition methods such as sparse coding of activations (Cunningham et al., 2024).

See also: explainability, explainable artificial intelligence, interpretable machine learning, trustworthy artificial intelligence, regularization, local interpretable model-agnostic explanations, mechanistic interpretability, transparency.

References

  1. Doshi-Velez and Kim (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. arxiv.org/abs/1702.08608
  2. Standardization and Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology. www.iso.org/standard/74296.html
  3. Chen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. Proc. 35th Int. Conf. Mach. Learn.. proceedings.mlr.press/v80/chen18j.html
  4. Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. Adv. Neural Inf. Process. Syst.. proceedings.neurips.cc/paper_files/paper/2022/hash/13113e938f2957891c0c5e8df811dd01-Abstract-Conference.html
  5. Hase and Bansal (2020). Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?. Proc. 58th Annu. Meeting Assoc. Comput. Linguistics. aclanthology.org/2020.acl-main.491
  6. Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). doi.org/10.6028/NIST.AI.100-1
  7. Gunning and Aha (2019). DARPA's Explainable Artificial Intelligence (XAI) Program. AI Magazine. doi.org/10.1145/3301275.3308446
  8. Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. Nature Mach. Intell.. doi.org/10.1038/s42256-019-0048-x
  9. Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue. doi.org/10.1145/3236386.3241340
  10. Olah et al. (2020). Zoom In: An Introduction to Circuits. Distill. doi.org/10.23915/distill.00024.001
  11. Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. transformer-circuits.pub/2021/framework/index.html
  12. Cunningham et al. (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models. 12th Int. Conf. Learn. Representations. openreview.net/forum?id=F76bwRSLeK

Cite this entry

@misc{dictml_interpretability,
  author = {Jung, Alexander},
  title = {interpretability},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-21},
  url = {https://dictionaryofml.org/terms/interpretability.html}
}