P-noise writes when the script runsDictionary of Applied Machine Learning · label
Numerical companion to the entry label: it recomputes what the entry states and prints one line per check
One block per paragraph of the entry (marked [P...]): each block verifies numerically what the corresponding statement asserts. Self-contained (numpy/matplotlib only), fixed seed.
Run it with python3 label.py, from any directory — it writes its output files into the current directory. Requires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Download label.py
One cell per block of the script: the code, and what that code printed when it last ran here
"""
label.py — numerical companion to the glossary entry 'label'.
One block per paragraph of the entry (marked [P...]): each block verifies
numerically what the corresponding statement asserts. Self-contained
(numpy/matplotlib only), fixed seed.
Blocks
------
[P-choice] Feature vs label is a design choice: with the "oncologist
available" design the prognosis enters the feature vector
and improves prediction of the target; without it the
prognosis is the label to be predicted from the remaining
features.
[P-labelfun] The labeling function h-bar acts on the data point itself:
a deterministic function assigns each (fully observed)
synthetic patient its label.
[P-partial] The feature vector captures only part of the data point:
two data points with identical features carry different
labels, so no hypothesis reading only the features predicts
perfectly — the empirical error of the best such hypothesis
stays bounded away from zero, while a hypothesis with
access to the full data point achieves zero error.
[P-noise] The observed label is a noisy proxy: training on labels
flipped with increasing probability degrades the test error
of the learned hypothesis (label noise degrades training
and generalization).
Outputs
-------
label.png : preview figure (checking only).
Data generated by pythondemos/label.py.
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
rng = np.random.default_rng(42)
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
# synthetic patients: (weight, age, hidden_marker) fully describe the
# data point; the label depends on all three.
m = 600
weight = 60 + 20 * rng.random(m)
age = 20 + 50 * rng.random(m)
hidden = rng.normal(size=m) # never recorded as feature
z = np.stack([weight, age, hidden], axis=1) # the data point itself
label_fun = lambda Z: (0.03 * Z[:, 0] + 0.04 * Z[:, 1]
+ 1.5 * Z[:, 2] > 4.0) # labeling function h-bar
y = label_fun(z)
Feature vs label is a design choice: with the "oncologist available" design the prognosis enters the feature vector and improves prediction of the target; without it the prognosis is the label to be predicted from the remaining features.
print("[P-choice] feature vs label as a design choice")
prognosis = (0.5 * hidden + 0.2 * rng.normal(size=m) > 0) # expert output
target = y
def acc(X, yy):
Xb = np.c_[X, np.ones(m)]
w = np.linalg.lstsq(Xb, yy.astype(float), rcond=None)[0]
return np.mean((Xb @ w > 0.5) == yy)
acc_without = acc(np.c_[weight, age], target)
acc_with = acc(np.c_[weight, age, prognosis.astype(float)], target)
print(f" accuracy without / with the prognosis feature: "
f"{acc_without:.2f} / {acc_with:.2f}")
check("prognosis used as a FEATURE improves the prediction",
acc_with > acc_without + 0.05)
check("prognosis used as the LABEL is predictable from features "
"(better than chance)",
acc(np.c_[weight, age], prognosis) >= 0.5)
[P-choice] feature vs label as a design choice
accuracy without / with the prognosis feature: 0.64 / 0.84
[ok] prognosis used as a FEATURE improves the prediction
[ok] prognosis used as the LABEL is predictable from features (better than chance)
The labeling function h-bar acts on the data point itself: a deterministic function assigns each (fully observed) synthetic patient its label.
print("[P-labelfun] the labeling function acts on the data point")
check("h-bar is deterministic on data points",
np.array_equal(label_fun(z), y))
check("h-bar reads the whole data point (hidden part matters)",
not np.array_equal(label_fun(z),
label_fun(np.c_[z[:, :2], np.zeros(m)])))
[P-labelfun] the labeling function acts on the data point [ok] h-bar is deterministic on data points [ok] h-bar reads the whole data point (hidden part matters)
The feature vector captures only part of the data point: two data points with identical features carry different labels, so no hypothesis reading only the features predicts perfectly — the empirical error of the best such hypothesis stays bounded away from zero, while a hypothesis with access to the full data point achieves zero error.
print("[P-partial] identical features, different labels")
x_feat = np.round(np.stack([weight, age], axis=1)) # recorded features
# find twin data points with equal features but different labels
_, inv, counts = np.unique(x_feat, axis=0, return_inverse=True,
return_counts=True)
twin_exists = any(len(set(y[inv == g])) > 1
for g in np.nonzero(counts > 1)[0])
check("two data points share features but differ in the label",
twin_exists)
err_feat = 1 - acc(x_feat, y)
Xfull = np.c_[z, np.ones(m)]
w_full = np.linalg.lstsq(Xfull, y.astype(float), rcond=None)[0]
err_full = np.mean((Xfull @ w_full > 0.5) != y)
print(f" error with features only: {err_feat:.2f}; "
f"with the full data point: {err_full:.2f}")
check("features-only hypothesis cannot be perfect", err_feat > 0.05)
check("full-data-point hypothesis is (nearly) perfect", err_full < 0.02)
[P-partial] identical features, different labels
[ok] two data points share features but differ in the label
error with features only: 0.36; with the full data point: 0.01
[ok] features-only hypothesis cannot be perfect
[ok] full-data-point hypothesis is (nearly) perfect
The observed label is a noisy proxy: training on labels flipped with increasing probability degrades the test error of the learned hypothesis (label noise degrades training and generalization).
print("[P-noise] label noise degrades training and generalization")
Xtr, Xte = x_feat[:400], x_feat[400:]
ytr0, yte = y[:400], y[400:]
errs = []
for flip in (0.0, 0.2, 0.4):
errs_r = []
for _ in range(25):
ytr = ytr0.copy()
idx = rng.choice(400, int(flip * 400), replace=False)
ytr[idx] = ~ytr[idx]
Xb = np.c_[Xtr, np.ones(400)]
w = np.linalg.lstsq(Xb, ytr.astype(float), rcond=None)[0]
errs_r.append(np.mean((np.c_[Xte, np.ones(200)] @ w > 0.5)
!= yte))
errs.append(float(np.mean(errs_r)))
print(f" test error at flip = 0, 0.2, 0.4: "
f"{errs[0]:.2f}, {errs[1]:.2f}, {errs[2]:.2f}")
check("test error grows with the label-noise level",
errs[0] < errs[1] < errs[2])
# ------------------------------------------------------------ preview
fig, ax = plt.subplots(figsize=(4.6, 3.2))
ax.plot([0, 0.2, 0.4], errs, "o-")
ax.set_xlabel("label-flip probability"); ax.set_ylabel("test error")
ax.set_title("[P-noise] label noise degrades generalization")
fig.tight_layout()
fig.savefig("label.png", dpi=110)
print(f"\n{sum(ok for _, ok in report)}/{len(report)} checks passed")
assert all(ok for _, ok in report)
[P-noise] label noise degrades training and generalization
test error at flip = 0, 0.2, 0.4: 0.34, 0.34, 0.40
[ok] test error grows with the label-noise level
8/8 checks passed
P-noise writes when the script runs