Start here
- Teaches
- How a classifier turns scores into probabilities, why cross-entropy is the loss that trains it, and how to measure the result honestly.
- Prerequisites
- Derivatives & gradients · basic probability (a number in [0,1] that sums to 1)
- Path
- Scores & boundary
- Sigmoid & softmax
- Cross-entropy
- Confusion metrics
- ROC & AUC
- Level
- Core — read top to bottom; open “Go deeper” only if you want the derivations.
Every classifier does the same three things: it produces a score, squashes that score into a probability, and is trained by a loss that punishes confident mistakes. Metrics come last — they judge the finished model. We follow that exact order so each idea has somewhere to attach.
01 What a classifier predicts
Definition. A classifier maps an input to a real-valued score (a “logit”), and a threshold on that score decides the class. The score is “how strongly,” not yet “how likely.”
Picture a single dial. Slide an email far right → “spam”; far left → “not spam.” The dial reading is the logit $z$; the mark where the label flips is the decision boundary. Everything later is about calibrating that dial and grading where you put the mark.
For a linear model the score is a weighted sum of features:
$$ z = \mathbf{w}^\top \mathbf{x} + b, \qquad \hat{y} = \begin{cases} 1 & z \ge 0 \\ 0 & z < 0 \end{cases} $$| symbol | meaning | shape |
|---|---|---|
| $\mathbf{x}$ | input features | vector, $d$ |
| $\mathbf{w}$ | learned weights | vector, $d$ |
| $b$ | bias / offset of the boundary | scalar |
| $z$ | score (logit) | scalar, $(-\infty,\infty)$ |
$\mathbf{w}=[1.5,\,-2],\ b=0.5,\ \mathbf{x}=[2,\,1]$
$z = 1.5(2) + (-2)(1) + 0.5 = 1.5$
$z = 1.5 \ge 0 \Rightarrow \hat{y}=1$ (positive class)
- The raw output of a classifier is a score, not a probability.
- A threshold on the score is the decision boundary; moving it trades one error type for another (§4).
- $z=0$ is the default boundary only because we squash with sigmoid next.
Self-check
1. If $z = -0.2$ with a threshold at 0, what class is predicted?
Show answer
Class 0 — the score is below the boundary.
2. You raise the threshold from $0$ to $1$. Do you predict positive more or less often?
Show answer
Less often — fewer scores clear the higher bar, so fewer positives.
Go deeper
Nonlinear models (trees, nets) replace $\mathbf{w}^\top\mathbf{x}+b$ with an arbitrary $f(\mathbf{x})$, but the pattern is identical: produce a score, squash, threshold. The boundary $\{ \mathbf{x} : z=0 \}$ is a hyperplane for the linear case and a curved surface otherwise. Distance from the boundary is proportional to $|z|$ — the geometric root of the “margin” in SVMs.
02 Scores → probabilities: sigmoid & softmax
Definition. The sigmoid squashes one score into a probability in $(0,1)$; softmax turns a vector of scores into a probability distribution that sums to 1.
A score of $+4$ and a score of $+40$ are both “very positive,” but a probability must live in $[0,1]$. Sigmoid bends the infinite number line into that interval — big positive → near 1, big negative → near 0, and $z=0 \to 0.5$ (hence the default boundary). Softmax is the multi-class version: exponentiate, then normalise so the parts sum to a whole.
| symbol | meaning | range |
|---|---|---|
| $z$ | a single logit | $(-\infty,\infty)$ |
| $\sigma(z)$ | probability of the positive class | $(0,1)$ |
| $\mathbf{z}$ | vector of $K$ class logits | $\mathbb{R}^K$ |
| $\text{softmax}(\mathbf{z})_i$ | probability of class $i$ | $(0,1)$, sums to 1 |
Sigmoid: $z=1.5 \Rightarrow \sigma(1.5)=\frac{1}{1+e^{-1.5}} = \frac{1}{1+0.223} \approx 0.82$
Softmax: $\mathbf{z}=[2,\,1,\,0]$, $e^{\mathbf{z}}=[7.39,\,2.72,\,1]$, sum $=11.11$
$\Rightarrow [0.665,\ 0.245,\ 0.090]$ — class 0 wins, and the three sum to 1.
- Sigmoid = softmax for two classes. Use sigmoid for binary / multi-label, softmax for one-of-$K$.
- Softmax is shift-invariant: adding a constant to every logit changes nothing — the basis of the numerically-stable form.
- The argmax class is the same before and after squashing; the probability only adds calibrated confidence.
Self-check
1. What is $\sigma(0)$, and why does it make $z=0$ the natural boundary?
Show answer
$\sigma(0)=0.5$ — equal odds, so the class flips exactly at $z=0$.
2. Softmax of $[5,5]$?
Show answer
$[0.5, 0.5]$ — equal logits give equal probability.
Go deeper — numerical stability
$e^{z}$ overflows for large $z$. Because softmax is shift-invariant, subtract the max logit first:
$$ \text{softmax}(\mathbf{z})_i = \frac{e^{z_i - m}}{\sum_j e^{z_j - m}}, \qquad m=\max_k z_k $$Frameworks fuse this with the log for the loss (“log-sum-exp”), which is why you pass raw logits — not probabilities — to CrossEntropyLoss.
03 The loss: cross-entropy
Definition. Cross-entropy (log-loss) measures the distance between the predicted probability and the true label — small when you are confidently right, large and fast-growing when you are confidently wrong.
The penalty is $-\log(\text{probability you gave the true class})$. Give the truth $p=1.0$ → penalty $0$. Give it $p=0.01$ → penalty $\approx 4.6$. The $-\log$ curve is gentle near 1 and explodes near 0, so the model is punished hardest exactly when it is sure and wrong. That asymmetry is what drives learning.
$-\log p$: near-zero cost for a confident correct call, unbounded cost as $p\to 0$.
Binary, then the general $K$-class form:
$$ \mathcal{L}_{\text{bin}} = -\big[\,y\log p + (1-y)\log(1-p)\,\big] \qquad \mathcal{L} = -\sum_{i=1}^{K} y_i \log p_i $$| symbol | meaning | example |
|---|---|---|
| $y_i$ | true label, one-hot | $[0,1,0]$ |
| $p_i$ | predicted probability (from §2) | $[.2,.7,.1]$ |
| $\mathcal{L}$ | loss for one example | scalar $\ge 0$ |
$\mathbf{y}=[0,1,0]$, $\mathbf{p}=[0.2,\,0.7,\,0.1]$
Only the true class survives the sum: $\mathcal{L} = -\log(0.7)$
$\mathcal{L} \approx 0.357$ — modest, because the model put most mass on the right class.
- Only the true class’s probability enters the sum — cross-entropy rewards putting mass on the truth.
- It pairs with softmax/sigmoid; together their gradient collapses to a famously clean form (below).
- Use MSE for regression, cross-entropy for classification — MSE on probabilities trains slowly and can stall.
Self-check
1. Two models predict the true class with $p=0.9$ and $p=0.6$. Which has lower loss?
Show answer
$p=0.9$: $-\log 0.9\approx0.105$ vs $-\log 0.6\approx0.511$. More confident-and-correct → lower loss.
2. Why not just train on accuracy?
Show answer
Accuracy is a step function — its gradient is zero almost everywhere, so it can’t guide gradient descent. Cross-entropy is smooth.
Go deeper — the gradient is just $p - y$
With softmax probabilities $p_i$ and cross-entropy $\mathcal{L}=-\sum_i y_i\log p_i$, the gradient with respect to the logit $z_k$ is
$$ \frac{\partial \mathcal{L}}{\partial z_k} = p_k - y_k. $$The softmax Jacobian and the $\tfrac{1}{p}$ from $\log$ cancel exactly. This is why the whole pipeline trains stably: the error signal into the logits is simply “predicted minus actual.” It is the classification analogue of the residual $\hat{y}-y$ you meet in gradient descent.
04 Grading it: the confusion matrix
Definition. The confusion matrix splits predictions into TP, FP, FN, TN; precision, recall, and F1 are ratios of those four counts that expose what a single accuracy number hides.
On 1000 emails where only 20 are spam, a model that says “never spam” scores 98% accuracy — and catches nothing. You need to ask two separate questions: of the ones I flagged, how many were right (precision)? Of the ones that were spam, how many did I catch (recall)?
| term | meaning |
|---|---|
| TP / TN | correctly flagged positive / negative |
| FP | false alarm — flagged, but actually negative |
| FN | miss — not flagged, but actually positive |
| $F_1$ | harmonic mean of P and R (punishes imbalance between them) |
Spam filter: TP = 15, FP = 5, FN = 5, TN = 975
Precision $= 15/(15+5) = 0.75$; Recall $= 15/(15+5) = 0.75$
$F_1 = \tfrac{2(0.75)(0.75)}{0.75+0.75}=0.75$, while accuracy $=990/1000=0.99$ — accuracy flatters, $F_1$ tells the truth.
- Accuracy lies under class imbalance; report precision + recall (or $F_1$) instead.
- Moving the §1 threshold trades precision against recall — you can’t maximise both freely.
- Choose by cost: high recall for cancer screening (misses are deadly), high precision for spam (false alarms annoy).
Self-check
1. A model flags everything as positive. What are its precision and recall?
Show answer
Recall = 1 (catches every positive); precision = base rate of positives (low under imbalance). $F_1$ stays low.
2. Why the harmonic mean for $F_1$ rather than the average?
Show answer
The harmonic mean is dragged down by the smaller value, so you can’t win $F_1$ by acing one metric and tanking the other.
Go deeper — multi-class averaging
For $K$ classes, compute P/R per class, then aggregate: macro (unweighted mean — treats rare classes equally), micro (pool all TP/FP/FN — dominated by frequent classes), or weighted (mean weighted by class support). Report macro when rare classes matter.
05 Threshold-free quality: ROC & AUC
Definition. The ROC curve plots true-positive rate against false-positive rate across every threshold; AUC is the area under it — one number for how well the scores rank positives above negatives.
Precision/recall judge one threshold. But a good model should rank every real positive above every real negative, whatever cut-off you later pick. AUC measures exactly that ranking: it is the probability that a random positive gets a higher score than a random negative. $0.5$ = coin flip, $1.0$ = perfect separation.
| symbol | meaning |
|---|---|
| TPR | recall — fraction of positives caught |
| FPR | fraction of negatives wrongly flagged |
| $s^{+},s^{-}$ | score of a random positive / negative |
Scores — positives: {0.9, 0.6}; negatives: {0.7, 0.2}. Compare all $2\times2$ pairs:
$0.9{>}0.7$✓ $0.9{>}0.2$✓ $0.6{<}0.7$✗ $0.6{>}0.2$✓ → 3 of 4 correctly ranked
AUC $= 3/4 = 0.75$.
- AUC is threshold-free — it grades the ranking, not one operating point.
- AUC $=0.5$ is random; below $0.5$ means the scores are inverted.
- Under heavy imbalance, prefer the precision–recall curve (and its area) — ROC can look deceptively good.
Self-check
1. Your AUC is 0.3. What cheap fix roughly doubles performance?
Show answer
Flip the sign of the score — an AUC below 0.5 means the ranking is backwards, so inverting gives ~0.7.
2. Does improving AUC guarantee better accuracy at your chosen threshold?
Show answer
No — AUC summarises all thresholds; you still pick an operating point from the precision/recall trade-off in §4.
Go deeper — PR vs ROC under imbalance
FPR has the large negative count $TN$ in its denominator, so on a 1:1000 problem a flood of false positives barely moves FPR — ROC/AUC stays optimistic. Precision uses $FP$ against $TP$ directly, so the PR curve (and average precision) surfaces the pain. Rule of thumb: balanced → ROC-AUC; rare-positive → PR-AUC.
06 One-screen reference
The whole path, side by side — squash, train, then judge.
| Piece | Formula | Use it when | Output |
|---|---|---|---|
| Sigmoid | $1/(1+e^{-z})$ | binary / multi-label | $(0,1)$ |
| Softmax | $e^{z_i}/\sum e^{z_j}$ | one-of-$K$ | distribution |
| Cross-entropy | $-\sum y_i\log p_i$ | classification loss | $\ge 0$ |
| Precision / Recall | $\tfrac{TP}{TP+FP}$ · $\tfrac{TP}{TP+FN}$ | one threshold, imbalance | $[0,1]$ |
| $F_1$ | $2PR/(P+R)$ | one number, balance P&R | $[0,1]$ |
| ROC-AUC | $\Pr(s^+ > s^-)$ | threshold-free ranking | $[0,1]$ |
- $z$
- logit / raw score
- $p_i$
- predicted probability of class $i$
- $y_i$
- true label (one-hot)
- $\mathcal{L}$
- loss for one example
- TP,FP,FN,TN
- confusion-matrix counts
- TPR,FPR
- true / false positive rate
End-of-note quiz — can you reproduce these from memory?
1. Order the pipeline from raw input to a graded prediction.
Show answer
score $z$ → squash (sigmoid/softmax) → probability → cross-entropy loss (train) → threshold → confusion-matrix metrics / AUC (evaluate).
2. Why does the softmax + cross-entropy gradient reduce to $p-y$, and why does that matter?
Show answer
The softmax Jacobian cancels the $1/p$ from the log; the clean “predicted minus actual” signal is what makes the classifier train stably.
3. When does 99% accuracy mean nothing, and what do you report instead?
Show answer
Under class imbalance — report precision, recall / $F_1$, and PR-AUC.