If deep learning can do everything, why would anyone still use classical ML models?

On structured, tabular data — which is a huge share of real business problems — tree-based models like gradient boosting remain hard to beat, often matching or exceeding deep learning while training faster, needing less data, and staying easier to tune. Deep learning's real advantages show up on raw, unstructured data (images, audio, text), which is a different problem shape entirely.

How do I know if I picked the right number of clusters for k-means?

There is no single definitive answer, but common approaches include the "elbow method" (plotting within-cluster variance against different k values and looking for where the improvement levels off) and checking whether density-based clustering, which does not need k specified, arrives at a similar structure.

Given the interpretability cost, when is PCA actually worth using?

PCA earns its keep specifically when compactness matters more than individual feature meaning — visualizing very high-dimensional data in 2-3 components, speeding up training on data with hundreds of correlated features, or removing redundancy before a model that assumes independent features.

Does the interpretability/training-cost/prediction-cost/data-volume framework only apply to classical ML models?

No — it applies just as much when choosing between a classical model and a deep learning model, or between two different neural network architectures. The four factors are about the practical consequences of a model choice, not specific to any one family of models.

beginnerintermediateadvanced

AI Full-Stack · Section 6

Classical Machine Learning

Level: intermediate
Read: 55 min
Concepts: 6

The non-deep-learning models that still win on most structured, tabular problems: the core supervised families — linear/logistic regression, decision trees, random forests, gradient boosting, nearest neighbors — compared on nonlinearity, scaling and interpretability; SVMs and Naive Bayes as two more classification approaches with very different philosophies; k-means, hierarchical and density-based clustering for unsupervised grouping; PCA for compressing correlated features; the fundamentals ranking and recommendation systems build on; and a trade-off framework for choosing a model on interpretability, training cost, prediction cost and data volume, not accuracy alone.

What is true here

Linear models are interpretable but assume near-linearity; trees/forests/boosting capture nonlinearity automatically
Gradient boosting usually wins tabular accuracy; k-NN needs no training but is slow to predict at scale
SVMs maximize the margin between classes; Naive Bayes works well despite its false independence assumption
K-means needs k chosen in advance; density-based clustering needs no k and handles outliers natively
Model choice trades interpretability, training cost, prediction cost and data volume — not accuracy alone

What you will be able to do

Choose between linear, tree-based and instance-based models for a described tabular problem
Explain why tree-based models need no feature scaling and linear models do
Pick a clustering method based on whether k is known and whether outliers must be handled
Explain what PCA compresses and what it costs in interpretability
Weigh interpretability, training cost, prediction cost and data volume when choosing a model, not accuracy alone

Classical ML models and their trade-offs

The supervised and unsupervised models that predate deep learning and remain the strongest choice for most tabular data — plus the trade-off framework for picking between them.

The core supervised model families

coreintermediate

Linear regression predicts a number as a weighted sum of features. Logistic regression predicts a probability for classification. A decision tree splits data by asking a sequence of yes/no questions. A random forest averages many trees. Gradient boosting builds trees one at a time, each correcting the last one's errors. Nearest neighbors predicts using the most similar training examples.

Think of it as

These six models split into two families with very different behavior. Linear regression and logistic regression are linear models: fast, interpretable (each feature's weight says how much it matters and in which direction), and correct only if the true relationship really is close to linear in the features you gave it — they need you to do the nonlinear work yourself, often through feature engineering. Decision trees, random forests and gradient boosting are tree-based models: they split on thresholds, so they need no scaling, capture nonlinear relationships and feature interactions automatically, and are usually the strongest choice for structured tabular data. A single decision tree overfits easily; a random forest averages many trees trained on random subsets of data and features, trading a little bias for much lower variance; gradient boosting builds trees sequentially, each one targeting the previous ensemble's errors, which usually reaches higher accuracy than a random forest but is more sensitive to hyperparameters and easier to overfit if not tuned. Nearest neighbors is neither — it does no real 'training' at all, just stores the data and, at prediction time, looks up the closest stored examples; simple and interpretable, but slow to predict on large datasets and sensitive to feature scale.

python

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.neighbors import KNeighborsClassifier

# same interface across all of them:
model.fit(X_train, y_train)
predictions = model.predict(X_val)

What we're doing: Compare four model families on the same tabular classification problem to see the trade-offs in practice.

compare_models.pypython

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
import time

models = {
    "logistic_regression": LogisticRegression(max_iter=1000),
    "random_forest": RandomForestClassifier(n_estimators=200),
    "gradient_boosting": GradientBoostingClassifier(),
    "knn": KNeighborsClassifier(n_neighbors=5),
}

for name, model in models.items():
    start = time.perf_counter()
    model.fit(X_train_scaled, y_train)
    train_time = time.perf_counter() - start
    acc = accuracy_score(y_val, model.predict(X_val_scaled))
    print(f"{name:20s} acc={acc:.3f}  train={train_time:.2f}s")

8: Logistic regression usually trains fastest here, and its accuracy is a useful baseline — if a tree-based model barely beats it, the relationship may genuinely be close to linear.
10: Gradient boosting typically reaches the highest accuracy on structured tabular data like this, at the cost of a longer training time and more hyperparameters worth tuning.
15: k-NN's "training" time is nearly instant — it just stores the data — but its prediction time grows with dataset size, which this loop does not even measure separately.

Why this works: No single model wins universally — the honest way to choose is to actually compare a small, representative set on the real data, because the theoretical trade-offs (linear vs nonlinear, interpretable vs accurate, fast-train vs fast-predict) only tell you the shape of the decision, not which model wins on this specific dataset.

Choosing a model family from its reputation instead of comparing it on the actual data

Wrong

text

"Gradient boosting always wins Kaggle,
so we'll use it here too."
# No baseline comparison. No check that this
# dataset resembles the ones where that's true.

Better

text

Start with a fast, interpretable baseline
(logistic regression or a single tree).
Compare 2-3 model families on the real data
and a real validation split before committing.
Pick based on the measured trade-off, not reputation.

What you see: A team spends significant tuning effort on a gradient boosting model that ends up barely outperforming a five-line logistic regression baseline that was never tried — reputation, not measurement, drove the choice.

Why: A model family's general reputation comes from its behavior across many datasets; any single dataset can be the exception — small, close-to-linear, or too small for a complex model to have an advantage. A cheap baseline comparison catches this in minutes; skipping it can cost days of tuning a model that was never the right choice.

Six models, compared on four practical axes

Six models, compared on four practical axes
Model	Handles nonlinearity?	Needs scaling?	Interpretability
Linear regression	No (without manual feature engineering)	Yes	High — weights are direct
Logistic regression	No (without manual feature engineering)	Yes	High — weights are direct
Decision tree	Yes	No	High — a readable rule sequence
Random forest	Yes	No	Medium — feature importance, not individual rules
Gradient boosting	Yes	No	Medium — feature importance, not individual rules
k-nearest neighbors	Yes	Yes	Medium — reasoning is "similar to these examples"

Remember: Linear/logistic regression: fast, interpretable, assumes near-linearity. Trees/forests/boosting: capture nonlinearity automatically, no scaling needed, boosting usually wins accuracy but is most hyperparameter-sensitive. k-NN: no real training, slow at prediction time, scale-sensitive. Compare on the real data — reputation is not a substitute for a baseline.

Support vector machines and Naive Bayes

standardintermediate

A support vector machine (SVM) finds the boundary between classes that leaves the widest possible margin to the nearest points of each class. Naive Bayes predicts a class using Bayes' theorem, assuming every feature is independent of every other given the class — an assumption that is usually false but works surprisingly well in practice.

Think of it as

SVMs and Naive Bayes solve classification from opposite philosophies. An SVM looks for the single boundary that maximizes the margin — the distance to the nearest point of each class — which tends to generalize well because it is not just any boundary that separates the classes, but the one furthest from both. With a kernel trick, an SVM can find nonlinear boundaries by implicitly mapping data into a higher-dimensional space where a straight boundary does separate the classes. Naive Bayes instead computes, for each class, how likely the observed features would be if that class were true (using Bayes' theorem), and picks the most likely class — its 'naive' assumption is that features are conditionally independent given the class, which is rarely exactly true (word order in text, for instance, violates it) but the method still performs well because it only needs the RANKING of class probabilities to be right, not their exact values. Naive Bayes is a classic choice for text classification (spam filtering) because it is fast, needs little data, and the independence assumption hurts less than it sounds like it should.

python

from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB

svm = SVC(kernel="rbf").fit(X_train, y_train)       # nonlinear boundary via kernel
nb = MultinomialNB().fit(X_train_counts, y_train)     # fast text-classification baseline

Same real data a straight line genuinely cannot separate

Remember: SVM finds the widest-margin boundary between classes, optionally nonlinear via a kernel. Naive Bayes uses Bayes' theorem with a (usually false) independence assumption, and still works well because it only needs the ranking right, not the exact probabilities.

Clustering: k-means, hierarchical and density-based

standardintermediate

K-means groups data into a fixed number of clusters, k, by repeatedly assigning points to the nearest cluster center and recomputing centers. Hierarchical clustering builds a tree of nested clusters, from every point alone up to one giant cluster, and you pick where to cut it. Density-based clustering finds clusters as dense regions separated by sparse ones, without needing to choose k in advance.

Think of it as

The three methods differ mainly in what you have to specify up front and how they handle irregular shapes. K-means requires choosing k, the number of clusters, before running it — and it assumes clusters are roughly round and similarly sized, which fails visibly on elongated or unevenly-sized real clusters. Hierarchical clustering avoids choosing k up front — it builds the full tree of nested groupings (a dendrogram) and lets you cut it at whatever level produces the number of clusters you want, which also lets you inspect the whole hierarchy of groupings at once, not just one flat answer. Density-based clustering (like DBSCAN) does not require k either, and it naturally finds clusters of arbitrary shape by growing outward from dense regions — its real advantage is handling noise: points in sparse regions are labeled as outliers rather than forced into the nearest cluster, which k-means and hierarchical clustering cannot do.

python

from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN

kmeans = KMeans(n_clusters=5).fit(X)               # must choose k
hier = AgglomerativeClustering(n_clusters=5).fit(X) # or cut a dendrogram
dbscan = DBSCAN(eps=0.5, min_samples=5).fit(X)      # no k; labels noise as -1

k-means, actually run: 180 real points, k=3

Three clustering methods, by what they require and handle

Three clustering methods, by what they require and handle
Method	Requires k?	Cluster shape assumption	Handles outliers?
K-means	Yes	Roughly round, similar size	No — every point assigned to a cluster
Hierarchical	No (choose cut level)	Flexible, inspectable at every level	No — every point assigned
Density-based (DBSCAN)	No	Arbitrary shape	Yes — sparse points labeled as noise

Remember: K-means needs k chosen up front and assumes round, similar-sized clusters. Hierarchical clustering builds a full tree, letting you choose the cut level. Density-based clustering needs no k, finds arbitrary shapes, and is the only one that separates outliers from real clusters.

PCA and dimensionality reduction

standardintermediate

Principal component analysis (PCA) compresses many correlated features into fewer new features, called components, that capture as much of the original variation as possible. Each component is a weighted combination of the original features, ordered so the first component captures the most variation, the second the next most, and so on.

Think of it as

PCA works because real features are often correlated — square footage and number of bedrooms both roughly track 'how big is this house' — so some of that redundancy can be compressed away with little information lost. PCA finds new axes (components), each a linear combination of the original features, chosen so the data's variation is spread out as efficiently as possible along the first few axes. Keeping only the first k components discards the axes with the least variation, which is usually the least informative. The trade-off is interpretability: 'component 2' has no obvious real-world meaning the way 'bedrooms' does, since it is a mix of several original features. PCA is most useful for visualization (compressing to 2-3 dimensions to plot data), for speeding up training on very high-dimensional data, and for removing correlated redundancy before feeding data to a model that assumes independent features.

python

from sklearn.decomposition import PCA

pca = PCA(n_components=0.95)   # keep enough components for 95% of variance
pca.fit(X_train)                # fit on train only
X_train_reduced = pca.transform(X_train)
X_val_reduced = pca.transform(X_val)   # apply, never refit

print(pca.explained_variance_ratio_)   # how much each component captures

200 real correlated points, reduced by real PCA

Remember: PCA compresses correlated features into fewer, ordered components that capture the most variation — useful for visualization and speed, at the cost of interpretability. Fit on training data only, like any other preprocessing step.

Ranking and recommendation fundamentals

standardintermediate

A ranking model orders a list of items by relevance rather than predicting one label per item. A recommendation system suggests items a specific user is likely to want, usually by combining collaborative filtering (what similar users liked) with content-based filtering (what this item is similar to, based on its own features).

Think of it as

Ranking is a different problem shape than classification or regression, because what matters is the ORDER of items relative to each other, not each item's individual score being exactly correct — a ranking model that consistently puts the best results first is doing its job even if its raw scores are miscalibrated. This is why ranking metrics (§7) compare ordering, not absolute error. Recommendation systems are ranking applied to 'what should this user see next,' and they typically blend two signals: collaborative filtering infers preference from patterns across many users ('people who liked X also liked Y'), which needs no understanding of item content but suffers on new items with no interaction history yet (the cold-start problem); content-based filtering recommends items similar to ones a user already engaged with, based on the items' own features, which handles new items fine but tends to over-narrow recommendations to what a user has already shown interest in. Most production systems combine both, plus additional signals (recency, popularity, business rules), rather than relying on either alone.

text

collaborative filtering: "users like you also liked..."   -> cold-start problem for new items
content-based filtering: "similar to items you engaged with" -> over-narrowing problem
production systems: blend both + recency + popularity + business rules

Remember: Ranking optimizes relative order, not absolute score accuracy. Collaborative filtering uses cross-user patterns (weak on new items); content-based filtering uses item similarity (weak on narrowing too tightly). Real systems blend both.

Model trade-offs and assumptions, as one decision

standardintermediate

Choosing a model is not "which one is best" — it is a trade-off between accuracy, interpretability, training cost, prediction cost, and how much data you have. The right choice depends on which of those actually matters for the problem at hand.

Think of it as

Every model family from this section trades these five factors differently, and a real choice weighs the ones the project actually cares about. Interpretability matters when a decision needs to be explained — to a regulator, a doctor, or a customer disputing a denial — and linear models and single decision trees win there; ensembles (random forest, gradient boosting) and neural networks trade interpretability for accuracy. Training cost matters when a model needs to be retrained often; linear models and k-NN train almost instantly, gradient boosting and deep learning can take much longer. Prediction cost matters at serving scale; k-NN's prediction cost grows with dataset size, while a trained linear model or tree predicts in constant time regardless of how much data it was trained on. Data volume matters because complex models (deep learning, large ensembles) need more data to avoid overfitting than simple ones. The failure mode this concept exists to prevent is picking a model on accuracy alone, in a benchmark, and discovering only in production that its prediction latency or its inability to explain a decision is what actually mattered.

text

pick a model by asking, in this order for THIS project:
  1. does a decision need to be explained to a person? -> weigh interpretability
  2. how often does it retrain?                          -> weigh training cost
  3. what's the serving-time latency budget?             -> weigh prediction cost
  4. how much labeled data do we actually have?          -> weigh data requirements
  5. only then: which model is most accurate on our data

The same 22 noisy points, three real polynomial fits

The five factors, and which models are strong on each

The five factors, and which models are strong on each
Factor	Strong	Weak
Interpretability	Linear/logistic regression, single decision tree	Random forest, gradient boosting, deep learning
Training speed	Linear models, k-NN (no real training)	Gradient boosting, deep learning
Prediction speed at scale	Linear models, trees, forests, boosting	k-NN (grows with dataset size)
Works with little data	Linear models, Naive Bayes	Deep learning, large ensembles

Remember: Model choice trades interpretability, training cost, prediction cost, and data requirements against accuracy — the right model is whichever wins on the factors this specific project actually needs, not whichever tops a generic benchmark.