If I can only track one metric during model development, which should it be?

It depends on the task, but for imbalanced classification, avoid accuracy — precision, recall, F1, or PR-AUC are all better starting points depending on which error type costs more. For regression, MAE or RMSE depending on outlier sensitivity. There is no single universal answer; the right metric is the one that actually reflects what a mistake costs in your specific problem.

Do I always need to check calibration, even if I only care about ranking?

Not always — if a system only ever uses the ranking (show the top-k results) and never the raw probability value itself, calibration matters much less. It becomes important the moment a downstream system uses the probability directly, such as thresholding on a specific confidence level or feeding the score into an expected-value calculation.

How long should an A/B test run before I can trust the result?

Long enough to reach the sample size decided on before the test started, which typically depends on the expected effect size, your traffic volume, and how confident you need to be. There is no universal duration — the key principle is deciding it in advance and not adjusting the stopping point based on how the results look partway through.

If online evaluation is more trustworthy, why not just A/B test every model candidate directly?

Online evaluation is slower, riskier (a bad model affects real users while the test runs), and more expensive in terms of traffic than offline evaluation. Offline evaluation exists specifically to filter down to a small number of promising candidates cheaply first — testing every candidate online would waste real user traffic on options that offline evaluation could have ruled out for free.

beginnerintermediateadvanced

AI Full-Stack · Section 7

Model Evaluation

Level: intermediate
Read: 65 min
Concepts: 8

How to know whether a model is actually good, and by which definition of good: regression metrics and their sensitivity to large errors; classification metrics — precision, recall, F1, ROC-AUC, PR-AUC — and why accuracy and ROC-AUC both mislead under class imbalance; reading a confusion matrix and choosing a decision threshold deliberately; calibration as a quality separate from ranking ability; ranking-specific metrics for search and recommendation; the gap between a model metric and a business metric; offline evaluation versus the online evaluation that actually reflects user behavior; and the statistical rigor an A/B test needs to be trustworthy.

What is true here

MSE/RMSE punish large errors much more than MAE; R² compares against a mean-only baseline
Accuracy and ROC-AUC both mislead under class imbalance — precision, recall and PR-AUC do not
A decision threshold (usually defaulted to 0.5) should match the real cost of a false positive vs a false negative
Calibration (does 80% confidence mean 80% correct) is separate from ranking quality (AUC)
A model metric can improve while the business metric gets worse, because the training objective is an approximation

What you will be able to do

Choose between MAE and MSE/RMSE based on how costly large errors are
Pick the right classification metric for a problem's class balance and error costs
Read a confusion matrix to find WHERE a model fails, not just how often
Explain why a well-ranking model can still be badly calibrated
Explain why offline and online evaluation are sequential, not interchangeable, and what makes an A/B test trustworthy

Scoring a model honestly

Regression and classification metrics, the imbalance and calibration traps that make simple metrics lie, ranking-specific scoring, and the offline/online/business-metric gap between a good score and a good outcome.

Regression metrics: MAE, MSE, RMSE, R²

standardintermediate

MAE (mean absolute error) is the average size of a prediction's error, ignoring direction. MSE (mean squared error) squares each error before averaging, punishing large errors more. RMSE is the square root of MSE, back in the original units. R-squared says what fraction of the target's variation the model explains, from 0 to 1.

Think of it as

MAE and MSE answer the same question — how wrong are the predictions — with a different weighting of large versus small errors. MAE treats every unit of error equally: an error of 10 counts exactly ten times an error of 1. MSE squares the error first, so an error of 10 counts a hundred times an error of 1 — this makes MSE far more sensitive to a few large mistakes, which is exactly the point when large errors are disproportionately costly (predicting delivery time badly by 5 minutes is fine, badly by 5 hours is not), and exactly the problem when a few genuine outliers are dominating the metric and hiding how the model does on typical cases. RMSE takes MSE back into the original units (square-rooting undoes the squaring), which makes it more interpretable than MSE while keeping the same sensitivity to large errors. R-squared is different in kind — it compares the model to a naive baseline that always predicts the mean, so an R² of 0.8 means the model explains 80% of the variation the mean alone could not.

python

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = mean_squared_error(y_true, y_pred, squared=False)
r2 = r2_score(y_true, y_pred)

Four regression metrics, compared

Four regression metrics, compared
Metric	Formula	Sensitive to outliers?	Same units as target?
MAE	mean(\|actual - predicted\|)	No	Yes
MSE	mean((actual - predicted)²)	Yes — heavily	No (squared)
RMSE	sqrt(MSE)	Yes — heavily	Yes
R²	1 - (error variance / total variance)	Inherits from underlying error	No — a 0-1 ratio

Remember: MAE weighs every error unit equally; MSE/RMSE punish large errors much more. RMSE is MSE back in interpretable units. R² compares to a mean-only baseline. Pick MSE/RMSE when large errors cost more; pick MAE when they do not.

Classification metrics: precision, recall, F1, ROC-AUC, PR-AUC

coreintermediate

Accuracy is the fraction of predictions that were correct — simple, and misleading when classes are imbalanced. Precision is: of everything the model flagged as positive, how much actually was? Recall is: of everything that actually was positive, how much did the model catch? F1 is a single number balancing precision and recall.

Think of it as

Every one of these metrics answers a different question, and picking the wrong one to optimize for is one of the most common real mistakes in applied ML. Accuracy answers 'what fraction did I get right overall' — and on a 99%-negative dataset, a model that predicts 'negative' every single time scores 99% accuracy while catching zero positives, which is why accuracy alone is close to useless under class imbalance. Precision answers 'when I say positive, how often am I right' — high precision matters when a false positive is costly (flagging a legitimate transaction as fraud annoys a real customer). Recall answers 'of all the real positives, how many did I catch' — high recall matters when a false negative is costly (missing an actual fraud case, missing an actual tumor). Precision and recall trade off against each other as you move a decision threshold: catching more positives (higher recall) generally means accepting more false alarms too (lower precision). F1 is the harmonic mean of the two, useful as one number when you need to balance them, but it can hide which of the two is actually being sacrificed. ROC-AUC summarizes performance across every possible threshold, but is itself optimistic under heavy imbalance, because it includes true negatives, which are trivially easy on an imbalanced dataset — this is why PR-AUC (precision-recall AUC) is the better choice specifically for imbalanced problems, since it never gives credit for the easy true negatives.

python

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score,  # average_precision_score ~= PR-AUC
)

precision_score(y_true, y_pred)
recall_score(y_true, y_pred)
f1_score(y_true, y_pred)
roc_auc_score(y_true, y_proba)              # needs probabilities, not labels
average_precision_score(y_true, y_proba)    # PR-AUC — prefer this under imbalance

What we're doing: Evaluate a fraud classifier on a heavily imbalanced dataset and see why accuracy is the wrong headline metric.

evaluate_imbalanced.pypython

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, average_precision_score,
)

# 10,000 transactions, 100 are fraud (1%)
# model predicts "not fraud" for everything except 40 flagged cases,
# 30 of which are real fraud (30 true positives, 10 false positives)

y_true = [1] * 100 + [0] * 9900
y_pred = [1] * 30 + [0] * 70 + [1] * 10 + [0] * 9890

print("accuracy: ", accuracy_score(y_true, y_pred))
print("precision:", precision_score(y_true, y_pred))
print("recall:   ", recall_score(y_true, y_pred))

12: Accuracy comes out above 99% here even though the model missed 70% of actual fraud — because the 9,890 easy true negatives dominate the count and hide the real failure.
13: Precision (30/40 = 75%) says three in four flagged transactions really were fraud — a reasonable rate for an analyst reviewing flagged cases.
14: Recall (30/100 = 30%) reveals the real problem accuracy hid: the model is catching less than a third of actual fraud, which is the number that should have driven the headline, not accuracy.

Why this works: This is the canonical shape of the accuracy trap: a metric that looks excellent while the model fails at the one thing it exists to do. Reporting accuracy alone on an imbalanced problem — without also reporting precision, recall, or PR-AUC — routinely hides exactly this failure from anyone reading only the headline number.

Reporting accuracy as the headline metric on an imbalanced classification problem

Wrong

python

acc = accuracy_score(y_true, y_pred)
print(f"Model accuracy: {acc:.1%}")  # "99.4% accuracy!"
# Ships based on this one number.

Better

python

print(f"precision: {precision_score(y_true, y_pred):.1%}")
print(f"recall:    {recall_score(y_true, y_pred):.1%}")
print(f"PR-AUC:    {average_precision_score(y_true, y_proba):.3f}")
# Accuracy reported too, but never as the headline
# on an imbalanced problem.

What you see: A fraud, spam, or anomaly-detection model reports a very high accuracy number in a demo, gets approved for production, and then catches almost none of the real positive cases it was built for — discovered only after real fraud/spam/anomalies keep slipping through.

Why: On an imbalanced dataset, the majority class dominates the accuracy count, so a model can score extremely well by simply ignoring the minority class — which is usually the class the model was actually built to catch. Precision, recall and PR-AUC are the metrics that cannot be gamed this way.

Confusion matrix for the fraud-detection example above

Which metric to lead with, by what a mistake costs

Which metric to lead with, by what a mistake costs
If a false positive is costly...	If a false negative is costly...	If both matter equally...
Optimize for precision	Optimize for recall	Use F1, or a weighted Fβ
e.g. flagging a legit purchase as fraud	e.g. missing an actual fraud case	e.g. general-purpose spam filtering

Remember: Accuracy is misleading under imbalance. Precision = correctness of positive predictions; recall = coverage of real positives; they trade off. F1 balances both in one number but can hide which is sacrificed. ROC-AUC is optimistic under imbalance; PR-AUC is the better choice there.

Confusion matrices and threshold selection

standardintermediate

A confusion matrix is a table showing every combination of predicted vs actual class — true positives, false positives, true negatives, false negatives — in one view. A classifier that outputs a probability needs a threshold to turn that probability into a decision, and every metric in the previous concept changes depending on where that threshold is set.

Think of it as

A confusion matrix is the raw material every classification metric (§7.2) is computed from — precision, recall and F1 are all just different arithmetic on the same four counts. Reading it directly, rather than only the metrics derived from it, shows WHERE a model fails, not just how often: a model can have identical accuracy in two very different ways, one confusing two classes that are genuinely hard to tell apart, another making a scattered mix of unrelated errors. Threshold selection is the decision most classifiers hide by default — most libraries pick 0.5 as the cutoff between 'positive' and 'negative' with no consideration of whether that is actually the right point for the problem, when the true trade-off between false positives and false negatives is a business decision, not a modeling one. Lowering the threshold catches more true positives at the cost of more false positives (higher recall, lower precision); raising it does the reverse. The right threshold is the one that matches the real cost of each mistake, chosen deliberately by looking at the precision-recall curve across thresholds, not left at whatever a library defaults to.

python

from sklearn.metrics import confusion_matrix, precision_recall_curve

cm = confusion_matrix(y_true, y_pred)
# [[TN, FP],
#  [FN, TP]]

precisions, recalls, thresholds = precision_recall_curve(y_true, y_proba)
# pick the threshold whose precision/recall trade-off
# matches the real cost of each kind of mistake

60 real positives, 400 real negatives, 200 real thresholds swept

Remember: A confusion matrix is the raw counts every classification metric is built from — read it to see WHERE errors happen. The decision threshold (usually defaulted to 0.5) should be chosen deliberately to match the real cost of a false positive vs a false negative.

Calibration and probability quality

standardintermediate

A model is well-calibrated when its predicted probabilities mean what they say — among all predictions the model gave 80% confidence to, roughly 80% should actually be correct. A model can rank examples perfectly (high accuracy, high AUC) while still being badly calibrated, if its confidence numbers do not match reality.

Think of it as

Calibration and ranking ability are separate qualities, and a model can have one without the other. Ranking quality (what ROC-AUC and PR-AUC measure) only asks whether positives tend to score higher than negatives — it does not care whether a score of 0.9 means 90% likely or 55% likely, as long as the ordering is right. Calibration asks specifically whether the probability number is trustworthy as a probability. This matters whenever a downstream decision uses the probability itself, not just a threshold — a fraud system that only flags transactions above 0.9 confidence, or a model whose output feeds into an expected-value calculation, needs those numbers to be real probabilities, not just a well-ordered score. Many powerful classifiers, especially tree ensembles and neural networks, are naturally overconfident or underconfident by default — a reliability diagram (predicted probability on one axis, actual observed frequency on the other) reveals this visually, and techniques like Platt scaling or isotonic regression can recalibrate a model's raw outputs after training, without touching the model itself.

python

from sklearn.calibration import CalibratedClassifierCV, calibration_curve

calibrated = CalibratedClassifierCV(base_model, method="isotonic").fit(X_train, y_train)

prob_true, prob_pred = calibration_curve(y_true, y_proba, n_bins=10)
# plot prob_pred vs prob_true: a perfectly calibrated model traces y = x

3,000 real predictions, 10 real probability bins

Remember: Calibration asks whether a predicted probability matches real-world frequency, which is separate from ranking quality (AUC). It matters whenever a system uses the probability value itself. Overconfident/underconfident models can be recalibrated after training without retraining.

Ranking metrics: MRR, MAP, NDCG

standardintermediate

MRR (mean reciprocal rank) scores how early the first correct result appears in a ranked list. MAP (mean average precision) scores how well ALL the relevant results are ranked near the top, not just the first one. NDCG (normalized discounted cumulative gain) additionally accounts for results having different degrees of relevance, not just relevant-or-not.

Think of it as

These three metrics fit different notions of what a 'good' ranked list looks like. MRR only cares about the first correct answer's position — perfect for a task with exactly one right answer, like 'find this specific document' — a correct result at position 1 scores 1.0, at position 2 scores 0.5, at position 10 scores 0.1; everything after the first hit is ignored. MAP is for tasks with MULTIPLE relevant results, where you want all of them ranked near the top, not just the first — a search for 'python tutorials' has many relevant pages, and MAP rewards a ranking that surfaces most or all of them early. NDCG goes one step further: rather than treating results as simply relevant or not, it uses a graded relevance score (a search result can be 'perfect', 'good', or 'somewhat relevant') and discounts a result's contribution the further down the list it sits — a highly relevant result at position 10 contributes much less than the same result at position 1.

text

MRR  = mean(1 / rank_of_first_correct_result)
MAP  = mean(average precision across all relevant results, per query)
NDCG = DCG / ideal_DCG   (DCG discounts graded relevance by position)

Three ranking metrics, by what task fits each

Three ranking metrics, by what task fits each
Metric	Fits a task with...	Relevance is...
MRR	One correct answer	Binary (correct or not)
MAP	Multiple relevant results	Binary (relevant or not)
NDCG	Multiple, differently-relevant results	Graded (a relevance score, not just yes/no)

Remember: MRR: only the first correct result matters. MAP: all relevant results should rank near the top. NDCG: like MAP, but relevance is graded, not binary. All three penalize relevant results sitting too low.

Business metrics vs model metrics

standardintermediate

A model metric (accuracy, F1, RMSE) measures how well a model fits its training objective. A business metric (revenue, churn rate, support cost) measures whether the model's deployment actually helped. A model can improve on its own metric while making the business metric worse.

Think of it as

The gap between the two exists because a model's training objective is always an approximation of what the business actually wants, chosen because it is measurable and optimizable — and approximations can diverge from the real goal in ways that only show up once the model is deployed. A recommendation model optimized purely for click-through rate can learn to recommend clickbait that hurts long-term retention, a business metric it was never trained against. A churn model that improves its F1 score by becoming more aggressive about flagging at-risk users can increase the number of costly retention offers sent to users who would not have churned anyway, hurting margin even as the model metric improves. The fix is not to abandon model metrics — they are still essential for comparing models during development — but to also track the actual business metric post-deployment, and treat a model-metric improvement that does not show up in the business metric as a signal the proxy has drifted from the real goal, not as a result to ignore.

text

model metric improves   +   business metric unchanged/worse
  -> investigate: has the proxy objective drifted from the real goal?

model metric improves   +   business metric improves
  -> the proxy is still a reasonable stand-in — keep tracking both anyway

Remember: A model metric measures fit to a training objective; a business metric measures real-world impact. They can diverge because the training objective is always an approximation — track both, and treat a model-metric win with no business-metric movement as a warning, not a success.

Offline vs online evaluation

standardintermediate

Offline evaluation scores a model against a fixed, historical dataset before it is deployed — fast, cheap, repeatable. Online evaluation measures a model against real, live traffic after deployment — slower and riskier, but the only evaluation that reflects real user behavior.

Think of it as

Offline evaluation is where iteration actually happens — it is fast enough to run many times a day, and repeatable enough that two runs on the same model and data give the same score, which is what makes it useful for comparing candidate models before committing to anything. Its blind spot is exactly what makes it fast: a fixed historical dataset cannot capture how real users would actually respond to a NEW model's outputs, because user behavior is a live reaction to what they are shown, not a static label. A recommendation model can score well offline (predicting the historical choices users made under the OLD system) while performing worse online (once it starts showing users different content and their behavior changes in response). Online evaluation — typically an A/B test or a gradual rollout — is what actually measures the thing offline evaluation can only approximate, at the cost of being slower, riskier (a bad model affects real users while the test runs), and noisier (real traffic has more variance than a fixed dataset). The two are sequential, not alternatives: offline evaluation filters down to a small number of promising candidates cheaply, and online evaluation makes the final call on the ones that survive.

text

offline: fixed historical data -> fast iteration, many candidates compared
   |
   v  (a small number of promising candidates survive)
online: real traffic, A/B test / gradual rollout -> the real decision, slower, riskier

Remember: Offline evaluation is fast and repeatable but cannot see how users react to a genuinely new model. Online evaluation (A/B test, gradual rollout) is the real measure but slower and riskier. Use offline to filter candidates, online to decide.

Statistical significance and A/B testing

standardintermediate

An A/B test compares two versions (the current model vs a new one) by showing each to a separate group of real users and measuring the difference in outcome. Statistical significance asks whether an observed difference is likely real, or could plausibly be random noise from which users happened to land in which group.

Think of it as

A/B testing is online evaluation (§7.7) made rigorous: instead of just watching a metric after a full rollout, users are split randomly into a control group (the existing model) and a treatment group (the new one), so any difference in outcome can be attributed to the model change rather than to which users happened to see which version. The core danger this concept exists to prevent is declaring a winner too early or from too small a sample — with few users, random variation alone can produce what looks like a meaningful difference, purely by chance. Statistical significance formalizes this: a small p-value means the observed difference would be unlikely if there were truly no effect, which is evidence the difference is real, not proof of it. A confidence interval reports the plausible range for the true effect size, which is more informative than a single point estimate — a reported lift of '+2%, 95% CI [-1%, +5%]' is meaningfully less certain than '+2%, 95% CI [+1.5%, +2.5%]', even though both report the same +2%. Running a test long enough to reach a large enough sample, and deciding the test's duration and success criteria in advance rather than stopping as soon as a result looks favorable, are what keep an A/B test's conclusion trustworthy.

text

users -> randomly split -> control (current model) / treatment (new model)
measure the same metric in both groups
compute: difference, p-value, confidence interval
decide BEFORE the test: sample size, duration, success threshold
  (deciding these after seeing early results invalidates the test)

Remember: An A/B test randomly splits users to isolate the model as the cause of a metric difference. A small p-value is evidence a difference is real, not proof. A confidence interval reports a range, not just a point. Decide sample size and duration in advance — stopping early on a good-looking result inflates false positives.