Is there a limit to how many interaction features I should add?

There is no fixed number, but adding many interaction features multiplies the feature space quickly and can reintroduce the same overfitting risk that comes from any high-dimensional feature set — each one should be added because there is a real reason to expect the interaction matters, not exhaustively for every possible pair.

Should I also scale the label (target), not just the features?

For regression, sometimes — scaling the target can help some optimizers converge more smoothly, but predictions then need to be converted back to the original scale before they are usable. For classification, the label is a category, not a continuous quantity, so scaling does not apply to it at all.

Should I always run PCA before training, just in case it helps?

No — PCA trades interpretability for compactness and is worth it specifically when that trade makes sense: very high-dimensional data, a genuine need for speed, or visualization. Running it by default on a reasonably-sized, already-interpretable feature set adds a step and a loss of interpretability for no clear benefit.

Can I combine manually engineered features with a learned embedding in the same model?

Yes, and this is common in practice — a pretrained embedding for a text or image field can be used as one input alongside manually engineered tabular features (price, category, account age) in the same model, getting the benefit of both approaches at once.

beginnerintermediateadvanced

AI Full-Stack · Section 5

Feature Engineering

Level: intermediate
Read: 55 min
Concepts: 6

Turning raw data into the numeric inputs a model can actually learn from: the five feature types and what each needs; scaling numeric features so no single one dominates by raw magnitude alone; the four categorical encodings — one-hot, ordinal, target, embedding — and the leakage risk that comes specifically with target encoding; shrinking a feature set through selection or dimensionality reduction; reading feature importance as a diagnostic; and the manual-features-versus-representation-learning trade-off that decides how much of this work a person does by hand versus a model learns on its own.

What is true here

Every feature type needs its own preparation step before a model can use it
Scale numeric features for distance-/weight-sensitive models; tree models are scale-invariant
Target encoding is powerful but must be computed out-of-fold, or it leaks the label into its own feature
Feature selection keeps interpretable features; dimensionality reduction creates compact, usually uninterpretable ones
Manual feature engineering suits structured tabular data; representation learning suits raw, unstructured data

What you will be able to do

Match a feature type to its required preparation step
Choose between normalization and standardization for a given model type
Pick the right categorical encoding for a column's cardinality, ordering, and leakage risk
Compute target encoding without leaking the label into its own feature
Decide whether a problem calls for manual feature engineering, representation learning, or both

From raw data to model-ready features

The preparation every feature type needs, the encoding trade-offs for categorical data, and when to hand-design features versus let a model learn them.

Feature types

standardbeginner

A numerical feature is a measurable quantity, like age or price. A categorical feature is one of a fixed set of values, like a country code. A text feature is free-form language. A time-series feature captures how a value changes over time. An interaction feature combines two or more existing features into a new one.

Think of it as

Each feature type needs a different preparation step before a model can use it — a model that expects numbers cannot directly consume the string 'France'. Numerical features usually need scaling. Categorical features need encoding into numbers (§5.3). Text features need extraction — a word count, TF-IDF, or an embedding. Time-series features need windowing — a rolling average, a lag, a day-of-week flag — to turn a sequence of values into a fixed set of per-row features. Interaction features exist because a model that only sees individual features can miss relationships between them: 'price per square foot' carries information neither 'price' nor 'square footage' carries alone, and creating it explicitly can help a model that cannot easily learn the division itself.

text

feature type   -> needs
numerical      -> scaling
categorical    -> encoding
text           -> extraction (TF-IDF, embedding)
time-series    -> windowing (lag, rolling average)
interaction    -> explicit combination of other features

Five feature types and their typical preparation step

Five feature types and their typical preparation step
Type	Example	Typical preparation
Numerical	Age, price, square footage	Scaling / normalization
Categorical	Country code, product category	Encoding (one-hot, ordinal, target, embedding)
Text	A product review	TF-IDF, word counts, or an embedding
Time-series	Daily transaction count	Lags, rolling averages, day-of-week/seasonality flags
Interaction	price ÷ square footage	Explicit combination of two or more existing features

Remember: Every feature type needs its own preparation step before a model sees it: numerical → scale, categorical → encode, text → extract, time-series → window, interaction → explicitly combine.

Scaling, normalization and standardization

standardbeginner

Scaling puts numeric features on comparable ranges so no single feature dominates a model just because its raw numbers happen to be larger. Normalization rescales values into a fixed range, usually 0 to 1. Standardization rescales values to have mean 0 and standard deviation 1.

Think of it as

Many models are sensitive to the raw scale of their inputs, not just the pattern within them — a feature measured in the thousands (income) can numerically overwhelm a feature measured in single digits (years of experience) even if the smaller feature matters just as much. Scaling fixes this by putting every feature on a comparable footing before training. Normalization (min-max scaling) maps a feature's range onto [0, 1] — it is simple and bounded, but sensitive to outliers, since one extreme value stretches the whole range. Standardization (z-score scaling) maps a feature to mean 0, standard deviation 1 — it is far less sensitive to outliers and is the more common default for models that care about scale, like linear/logistic regression, SVMs, and neural networks. Tree-based models (decision trees, random forests, gradient boosting) are a notable exception — they split on thresholds, not distances, so they are scale-invariant and usually need no scaling at all.

python

from sklearn.preprocessing import MinMaxScaler, StandardScaler

norm = MinMaxScaler().fit(X_train)        # -> range [0, 1]
X_train_norm = norm.transform(X_train)

std = StandardScaler().fit(X_train)       # -> mean 0, std 1
X_train_std = std.transform(X_train)
X_val_std = std.transform(X_val)          # apply, never refit

Why a model sees "income" and drowns out "years"

Remember: Scale numeric features for distance-/weight-sensitive models (linear, SVM, neural nets); tree models rarely need it. Normalization bounds to [0,1] and is outlier-sensitive; standardization centers to mean 0/std 1 and is more robust. Always fit on training data only.

One-hot, ordinal, target and embedding encodings

coreintermediate

A model needs numbers, not category names, so a categorical feature has to be encoded. One-hot encoding makes one binary column per category. Ordinal encoding assigns an integer to each category, in a meaningful order. Target encoding replaces a category with the average label value for that category. Embedding encoding learns a dense vector per category.

Think of it as

Each encoding trades off differently between dimensionality, ordering assumptions, and leakage risk. One-hot encoding is safe and simple — no ordering is implied — but it explodes in width with high-cardinality categories (a 'zip code' column with 40,000 unique values becomes 40,000 columns), and it treats every category as equally unrelated to every other, which throws away information for categories that really are ordered. Ordinal encoding fixes the width problem and fits naturally when categories genuinely have an order (small/medium/large), but assigning an arbitrary order to unordered categories (like city names) tricks distance-based models into believing some categories are 'closer' than others when they are not. Target encoding solves both the width and ordering problems by encoding each category as a single number — the average label value for that category — but it directly uses the label to build a feature, which makes it one of the easiest ways to accidentally create target leakage if the average is computed using the very rows it will be applied to. Embedding encoding learns a dense, lower-dimensional vector per category during training, the same idea LLMs use for tokens — it captures similarity between categories automatically, at the cost of needing enough data per category to learn something meaningful.

python

import pandas as pd
pd.get_dummies(df["category"])              # one-hot

pd.Categorical(df["size"], ["S", "M", "L"], ordered=True).codes  # ordinal

# target encoding — must be computed out-of-fold, never on the full set
df["category_te"] = df.groupby("category")["label"].transform("mean")  # WRONG if df includes validation rows

# embedding: a learned nn.Embedding(num_categories, dim) layer in a neural net

What we're doing: Compute target encoding correctly, out-of-fold, to avoid leaking the label into the encoded feature.

target_encode_safely.pypython

from sklearn.model_selection import KFold
import numpy as np

def target_encode_oof(df, cat_col, target_col, n_splits=5):
    encoded = np.zeros(len(df))
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
    for train_idx, val_idx in kf.split(df):
        means = df.iloc[train_idx].groupby(cat_col)[target_col].mean()
        encoded[val_idx] = df.iloc[val_idx][cat_col].map(means)
    return encoded

df["category_te"] = target_encode_oof(df, "category", "label")

6: Each fold computes category means using only the OTHER folds' rows — the rows being encoded never contribute to their own encoding.
7: This is out-of-fold encoding: a row's target-encoded value never depends on that row's own label, which is what makes it safe from target leakage.
10: A naive one-line `groupby(...).transform("mean")` on the whole dataset computes each category's mean including the row being encoded — that single line is the most common target-encoding leakage bug in practice.

Why this works: Target encoding is uniquely leakage-prone among encodings because it is built directly from the label — computing it out-of-fold, the same discipline used for stacked/ensembled models, is the only way to use it without quietly teaching the model to read the label back from its own encoded feature.

Target-encoding a category using the full dataset, including the rows being encoded

Wrong

python

means = df.groupby("category")["label"].mean()
df["category_te"] = df["category"].map(means)
# Every row's encoding includes its own label
# in the average used to encode it.

Better

python

# Use out-of-fold encoding (see example above),
# or at minimum compute means on TRAIN only and
# apply, unchanged, to validation/test:
means = df_train.groupby("category")["label"].mean()
df_val["category_te"] = df_val["category"].map(means)

What you see: A model using target-encoded features scores implausibly well in cross-validation, then performs much worse on genuinely new data — because each row's own label leaked into its own feature value through the naive mean.

Why: A category's naive average label value is built partly from the row being encoded itself, especially for rare categories with few rows — that row's own answer is baked into its own input feature, which is target leakage in its purest form.

Four encodings, by width, order and leakage risk

Four encodings, by width, order and leakage risk
Encoding	Output width	Implies order?	Leakage risk
One-hot	One column per category	No	None
Ordinal	One column	Yes — often wrongly, for unordered categories	None
Target	One column	No	High if not computed with cross-validation/out-of-fold
Embedding	k columns (a chosen dimension)	No	None, but needs enough data per category

Remember: One-hot: safe, explodes in width. Ordinal: compact, implies order — only use it when order is real. Target: powerful but must be computed out-of-fold or it leaks the label. Embedding: learns similarity, needs enough data per category.

Feature selection and dimensionality reduction

standardintermediate

Feature selection picks a subset of existing features to keep, discarding the rest. Dimensionality reduction creates a smaller number of new features that summarize the information in the original ones. Both fight the same problem: too many features slow training, increase overfitting risk, and make a model harder to interpret.

Think of it as

The two techniques solve the same problem differently. Feature selection keeps the original, interpretable features — it just drops some of them, using a filter method (rank features by a statistic like correlation with the label, keep the top ones), a wrapper method (try different subsets, keep whichever a model performs best with), or an embedded method (some models, like Lasso regression or tree-based feature importance, select features as a byproduct of training). Dimensionality reduction, by contrast, creates new features (like PCA components, covered in §6) that are combinations of the originals — more compact, but usually not individually interpretable anymore. The choice depends on whether interpretability matters: selection keeps 'square footage' as a recognizable feature; reduction might produce 'component 3', a mix of square footage, bedroom count and age with no simple name.

python

from sklearn.feature_selection import SelectKBest, f_classif
selector = SelectKBest(f_classif, k=20).fit(X_train, y_train)  # filter method
X_train_selected = selector.transform(X_train)

from sklearn.decomposition import PCA
pca = PCA(n_components=10).fit(X_train)                        # dimensionality reduction
X_train_reduced = pca.transform(X_train)

Remember: Feature selection drops features, keeping the rest interpretable. Dimensionality reduction creates new, combined features that are more compact but usually uninterpretable. Both should be fit on training data only.

Feature importance and leakage-aware pipelines

standardintermediate

Feature importance measures how much a trained model actually relies on each feature. A leakage-aware pipeline is a feature pipeline built so that every step — scaling, encoding, selection — is fit only on training data, so it structurally cannot leak validation or test information into training.

Think of it as

Feature importance is a diagnostic, not a design tool: it tells you what a model learned to lean on after training, which is useful for debugging (an unexpectedly dominant feature is often a leakage symptom, per §4) and for explaining predictions to stakeholders — but it does not tell you what SHOULD matter, only what did. An unusually dominant feature deserves the same suspicion an unusually good validation score does. A leakage-aware pipeline is the structural fix for the leakage problem that runs through every feature-engineering step in this section: rather than trusting yourself to remember 'fit on train only' every single time, a pipeline object (like scikit-learn's `Pipeline`) enforces that fitting only ever touches training data, because validation/test data is only ever passed through `.transform()`, never `.fit()`. This turns a discipline that is easy to violate by accident into one that is structurally difficult to violate at all.

python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", RandomForestClassifier()),
])
pipe.fit(X_train, y_train)          # every step fits on train only
pipe.score(X_val, y_val)            # every step transforms, never refits

importances = pipe.named_steps["model"].feature_importances_

Remember: Feature importance shows what a model relied on, and an unexpectedly dominant feature is worth investigating as possible leakage. A `Pipeline` object makes "fit on train only" structural instead of a habit that can be forgotten.

Manual feature engineering vs representation learning

standardintermediate

Manual feature engineering means a person designs features using domain knowledge — "price per square foot," "days since last purchase." Representation learning means a model (usually deep learning) learns useful features directly from raw data, with no person hand-designing them.

Think of it as

The choice usually tracks the data type and how much labeled data is available. Structured, tabular data with clear domain semantics (a real-estate listing, a transaction record) tends to favor manual feature engineering — a human genuinely knows that 'price per square foot' matters more than either input alone, and tree-based models trained on well-engineered tabular features are still hard to beat. Raw, high-dimensional, unstructured data (images, audio, raw text) tends to favor representation learning — nobody can hand-design a feature that captures 'is there a cat in this image' the way a convolutional network learns to. Representation learning also generally needs more data to learn good representations from scratch, which is why transfer learning (starting from a model pretrained on a huge dataset) is so common — it reuses representations someone else already learned. The two are not mutually exclusive: a pipeline can use a pretrained model's learned embedding as one 'feature' inside an otherwise manually engineered tabular pipeline.

text

tabular, domain semantics known   -> manual feature engineering (often wins)
raw image/audio/text, no hand-designable feature -> representation learning
small labeled dataset for either   -> transfer learning (reuse a pretrained model's representations)

Remember: Manual feature engineering suits structured data with known domain semantics. Representation learning suits raw, unstructured, high-dimensional data. Transfer learning bridges both when labeled data is scarce, and the two can combine in one pipeline.