How much labeled data is actually enough to train a model?

There is no fixed number — it depends on the task's complexity, the model's capacity, and how much natural variation exists in the real inputs. A more useful question than "is this enough" is often "is my train/validation gap small," since a persistently large gap despite adding data points at a more specific, model-complexity-related problem rather than a pure data-quantity one.

Is oversampling the minority class always the right fix for class imbalance?

Not always — it is one of several options (alongside undersampling and class weighting), and naive oversampling (simply duplicating minority examples) can make overfitting to those specific examples worse. Techniques like SMOTE, which synthesize new minority examples rather than duplicating existing ones, are often a stronger choice, but the right approach genuinely depends on the dataset and problem.

Is data leakage mostly a beginner mistake that experienced teams do not run into?

No — leakage is genuinely easy to introduce even on experienced teams, especially as a pipeline grows more complex (more feature-engineering steps, more upstream data sources). It is one of the most common causes of a real production incident, not just a beginner's error, which is exactly why auditing for it is a standing practice, not a one-time check.

Does a small project really need formal data lineage tooling?

Not necessarily formal tooling — a small project might get by with clear naming conventions and a written record of each pipeline step. What matters is being ABLE to answer 'where did this value come from, and what touched it since' when something breaks or a deletion request arrives, however that answer is tracked.

beginnerintermediateadvanced

AI Full-Stack · Section 4

Data Fundamentals

Level: intermediate
Read: 60 min
Concepts: 7

What happens to data before it becomes a feature: collection, labeling, cleaning and quality checks; the structured-vs-unstructured split and what bridges it; splitting a dataset properly with stratification; the real fixes for imbalance, missing data, outliers and noisy labels; the single most damaging and most common bug in applied ML — feature, target and temporal leakage; and the versioning and lineage discipline that makes a training run reproducible and traceable.

What is true here

Collection sets what a model can ever learn; label quality caps model quality
Unstructured data (text, image, audio, video) needs a model to extract meaning before it has features
Stratified, group- or time-aware splitting keeps each split representative and independent
Class imbalance makes accuracy lie — evaluate with precision/recall, not accuracy alone
Leakage — feature, target, or temporal — inflates validation metrics without inflating real performance

What you will be able to do

List the stages raw data passes through before it is feature-ready
Decide whether a dataset needs structured feature engineering or a representation-learning model first
Split a dataset with the right strategy — stratified, grouped, or time-based — for the data at hand
Diagnose whether an unusually good validation score is real or leaked
Explain why a dataset needs an immutable, versioned snapshot rather than a live query

Getting data ready for a model

The stages every dataset passes through — and the failure modes, especially leakage, that fake a good result without one.

Data collection, cleaning and quality checks

standardbeginner

Before a model can learn anything, data has to be collected, often labeled by hand or by another process, cleaned of errors, normalized into a consistent format, and checked for duplicates — a model trained on dirty data learns the dirt along with the pattern.

Think of it as

Every one of these steps exists to answer the same question: does this row actually represent what it claims to? Collection is where bias first enters a dataset — who or what generated the raw data determines what the model can ever learn. Labeling attaches the ground truth a supervised model needs, and label quality caps model quality: a model cannot be more accurate than its labels are correct. Cleaning fixes malformed values — a negative age, a price stored as text in one row and a number in the next. Normalization makes formats consistent (dates, units, casing) so the same real-world value is not treated as several different ones. Deduplication matters more than it looks: a duplicate row that ends up split across train and validation silently leaks train-set information into validation, inflating the reported score. Quality checks are the habit of verifying all of the above actually happened, rather than assuming it did.

text

raw data -> collection -> labeling -> cleaning -> normalization -> dedup -> quality checks -> ready for features

quality check failing at any stage should block the pipeline,
not just get logged and ignored

Data quality checks worth automating

Data quality checks worth automating
Check	Catches
Schema validation	Wrong type, missing required column, out-of-range value
Duplicate detection	The same row (or near-duplicate) appearing more than once
Label distribution check	A sudden shift in class balance between batches
Null/missing rate	A column that silently started arriving empty

Remember: Collection sets what the model can learn; label quality caps model quality. Cleaning and normalization make values consistent; deduplication prevents silent leakage across splits; quality checks verify all of it actually happened.

Structured vs unstructured data

standardbeginner

Structured data fits neatly into rows and columns with a fixed schema, like a database table. Unstructured data — text, images, audio, video — has no fixed schema; its meaning has to be extracted, usually by another model, before a downstream model can use it.

Think of it as

The distinction matters because it determines how much work happens before a model ever sees the data. Structured data (a relational table, a well-formed event log) already has named, typed fields — feature engineering can start immediately. Unstructured data has no such fields: a JPEG is a grid of pixel values with no column called 'contains a cat', and a sentence is a string with no column called 'sentiment'. Turning unstructured data into something a model can use requires representation — a vision model extracts pixel patterns, an embedding model extracts semantic meaning from text — and that representation step is itself often a whole model. Event streams sit in between: individually well-structured (a timestamp, a user id, an event type), but their value usually comes from patterns across many events, not from any single row.

text

structured:   rows + typed columns          -> features directly
unstructured: text/image/audio/video        -> needs a model to extract meaning first
event stream: structured per-event          -> value comes from patterns across many events

Structured vs unstructured, with a typical extraction step

Structured vs unstructured, with a typical extraction step
Kind	Example	What extracts meaning from it
Structured	A rows-and-columns sales table	Direct feature engineering — already typed and named
Text	A product review	An embedding model or an LLM
Image	A product photo	A vision model (CNN or vision transformer)
Audio	A support call recording	A speech-to-text model, then text processing
Event stream	Clickstream events	Aggregation/windowing across many events

Remember: Structured data has a schema already; unstructured data needs a model to extract meaning before it has one. Event streams are structured per-event but valuable in aggregate.

Splitting and stratified sampling

standardbeginner

Splitting divides a dataset into training, validation and test sets, usually 60-80% training and the rest split between the other two. A plain random split can accidentally put almost all of a rare class into one split; stratified sampling prevents that by keeping each split's class balance close to the full dataset's.

Think of it as

A random split works fine when classes are roughly balanced and examples are independent, but two situations break it. First, imbalance: if a rare class is 2% of the data, a plain random split can put a wildly different rare-class rate into each split — one that happens to over-represent it, one that under-represents it — making both the training signal and the reported metric unreliable. Stratified sampling fixes this by splitting within each class separately, so every split keeps close to the original class proportions. Second, dependency: examples that are not truly independent — multiple rows from the same user, or time-ordered data — need a split that respects that structure (group-based or time-based splitting) or information leaks across the split, the same failure mode as data leakage. Stratification and independent splitting are two separate concerns and a real project frequently needs both.

python

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.4, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42
)

Splitting data the real way it actually gets used

Remember: Split ~60-80% train, rest for validation/test. Stratify to keep class balance consistent across splits, especially under imbalance. Group- or time-based splitting is needed when rows are not truly independent.

Class imbalance, missing data, outliers and noisy labels

standardbeginner

Class imbalance is when one label is far rarer than others, which can make a model that always predicts the common class look accurate while being useless. Missing data is a value that was never recorded. Outliers are extreme values that may be real or may be errors. Noisy labels are labels that are simply wrong.

Think of it as

Each of these four problems fails a model differently, so each needs its own diagnosis. Imbalance is dangerous specifically because accuracy lies about it: a dataset that is 99% negative lets a model that always predicts 'negative' score 99% accuracy while catching zero positives — this is why imbalanced problems need precision/recall-style metrics, not accuracy, and often need resampling or class weighting during training. Missing data forces a choice with real consequences: drop the row (loses information, can bias the remaining data if missingness is not random), or impute a value (introduces an assumption about what the true value probably was). Outliers need a judgment call before any technique is applied — a $50,000 purchase in a dataset of $20-average orders might be a data-entry error, or might be a real, important large customer, and treating it as noise versus signal changes the model's behavior in opposite directions. Noisy labels are the hardest to detect because they look like normal data — the fix starts with sampling and manually auditing a subset of labels, not with a purely automated technique.

text

99% negative, 1% positive dataset:
  always-predict-negative model -> 99% accuracy, 0% recall on positives
  -> use precision/recall/F1, not accuracy, to evaluate this problem

Four problems, four different fixes

Four problems, four different fixes
Problem	Why it hurts	A fix
Class imbalance	Accuracy looks high while the model catches nothing rare	Resampling, class weights, precision/recall metrics
Missing data	Dropping loses data; imputing assumes a value	Impute deliberately, or use a model that handles missingness natively
Outliers	Can be real signal or a data error — opposite fixes	Investigate before deciding to clip, transform, or keep
Noisy labels	Looks like normal data, caps achievable accuracy	Audit a labeled sample manually; use label-quality tooling

Remember: Imbalance makes accuracy lie — use precision/recall instead and consider resampling or class weights. Missing data forces drop-vs-impute. Outliers can be signal or error. Noisy labels look normal; audit a sample to find them.

Feature, target and temporal leakage

coreintermediate

Leakage is when information that would not be available at real prediction time sneaks into training, making a model's validation score look far better than its production performance will actually be. It is the single most common reason a model that scored well in testing fails once it goes live.

Think of it as

All three kinds of leakage share one root cause: the model sees something during training that it will not see at the moment it actually needs to make a prediction. Feature leakage is a preprocessing step — scaling, encoding, imputation — fit on the full dataset (train and validation together) instead of on training data alone, so validation statistics quietly influence how training data gets transformed. Target leakage is a feature that is itself derived from, or a proxy for, the label — a 'was_refunded' column when predicting fraud is nearly the same information as the label. Temporal leakage is using information from after the prediction point in time — training a churn model on 'total lifetime purchases' when the number keeps growing after the point you would have needed to predict churn. All three inflate metrics identically: the model looks accurate on data that secretly contains the answer, and the deception only becomes visible once production data — which genuinely does not have that information — hits the model.

python

# WRONG: scaler sees validation data too
scaler.fit(X)                    # X is the full dataset
X_train_scaled = scaler.transform(X_train)

# RIGHT: scaler only ever sees training data
scaler.fit(X_train)              # fit on train only
X_train_scaled = scaler.transform(X_train)
X_val_scaled = scaler.transform(X_val)   # apply, never re-fit

What we're doing: Find the leaked feature in a churn-prediction dataset before it reaches training.

audit_features.pypython

features = [
    "account_age_days",
    "monthly_spend",
    "support_tickets_opened",
    "total_lifetime_purchases",   # <- keeps growing after "today"
    "cancelled_subscription",      # <- this IS the label, renamed
]
label = "churned"

for f in features:
    if f == label or "cancel" in f.lower():
        print(f"SUSPECT (target leakage): {f}")
    if "total_lifetime" in f or "as_of_today" not in f and "lifetime" in f:
        print(f"SUSPECT (temporal leakage): {f}")

5: "total_lifetime_purchases" is still accumulating after the moment a real prediction would need to be made — the model gets a preview of the future it will not have in production.
6: "cancelled_subscription" and "churned" describe nearly the same event — training on this feature teaches the model almost nothing except how to read the label back.
11: A name-pattern audit like this is a cheap first pass, not a complete one — the more reliable check is asking, feature by feature, "would this value exist at the moment I actually need to predict?"

Why this works: Leakage is invisible in a metrics dashboard — it only shows up as an unreasonably good validation score, which is exactly the signal that gets celebrated instead of investigated. The only reliable defense is asking, for every single feature, whether it would genuinely be available at the real moment of prediction — not whether it happens to be in the training table.

Fitting a preprocessing pipeline before splitting the data

Wrong

python

from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)   # sees everything
X_train, X_val = train_test_split(X_scaled)

Better

python

X_train, X_val = train_test_split(X)
scaler = StandardScaler().fit(X_train)   # train only
X_train_scaled = scaler.transform(X_train)
X_val_scaled = scaler.transform(X_val)   # apply, don't refit

What you see: A model reports a suspiciously strong validation score — often noticeably better than a similar published benchmark on comparable data — and then underperforms once deployed, with no code changes between the two.

Why: Splitting after fitting preprocessing means the scaler's mean and standard deviation were computed using validation rows too, so the "unseen" validation set was never really unseen by the pipeline — only by the model's own parameters.

Three leakage types, with a concrete example each

Three leakage types, with a concrete example each
Type	What leaks	Concrete example
Feature leakage	A preprocessing statistic computed across train + validation	A scaler's mean/std fit on the full dataset before splitting
Target leakage	A feature that is a proxy for the label	Predicting fraud using a "chargeback_filed" column
Temporal leakage	A feature that uses future information	Predicting churn using "total purchases," which keeps growing after prediction time

Remember: Leakage is when training sees information that would not exist at real prediction time. Feature leakage: preprocessing fit on validation data too. Target leakage: a feature that is really the label in disguise. Temporal leakage: a feature from after the prediction point. All three inflate metrics without inflating real performance.

Data versioning

standardintermediate

Data versioning means recording exactly which snapshot of a dataset a model was trained on — the same discipline as version-controlling code, applied to data. Without it, "retrain the model" silently means training on a dataset that has since changed, and nobody can explain why results shifted.

Think of it as

Code is normally version-controlled by default, but the dataset a model trains on usually is not, and that asymmetry is the problem. A dataset that grows daily, gets corrected, or has rows deleted for privacy reasons is a moving target — if a model's training run only records 'trained on the production database,' there is no way to reproduce the exact conditions later, compare two models fairly, or debug why a metric changed. Data versioning fixes this by treating a dataset snapshot as an immutable, named artifact — the same dataset id can always be fetched again in the exact state it was in in week the model was trained, separate from whatever the live dataset looks like today. This pairs directly with experiment tracking (§8): a real experiment record needs the model version, the code version, and the data version together, not just the first two.

text

bad:  train.py reads from "production_db.users" (changes daily)
good: train.py reads from "users_snapshot_v2026-09-10" (immutable, named)

experiment record = {code_version, data_version, model_version, metrics}

Remember: A live dataset changes over time; a versioned snapshot does not. Without a named, immutable data version, a retrain silently trains on different data and nobody can reproduce or compare results honestly.

Data lineage and provenance

standardintermediate

Provenance is where a piece of data originally came from. Lineage is the full chain of transformations it passed through to reach its current form — which tables it was joined from, which pipeline steps touched it, and in what order.

Think of it as

The two terms answer related but different questions. Provenance answers 'where did this originate' — which source system, which API, which point of collection produced the raw value. Lineage answers 'what happened to it since' — every join, filter, aggregation, and transformation a value passed through on its way to becoming a feature. Both matter for the same practical reason: when a model's output looks wrong, or a metric suddenly shifts, tracing lineage backward is how you find which upstream step actually changed, instead of guessing across the whole pipeline. It also matters for compliance — if a user asks for their data to be deleted, you need lineage to find every derived table and feature store entry that data ever touched, not just the original record.

text

provenance: raw_event (source: mobile_app_api)
lineage:    raw_event -> cleaned_events -> daily_aggregates
            -> user_features -> training_dataset_v7

a metric shift -> walk the lineage graph backward
                -> find which step actually changed

Remember: Provenance is where data came from; lineage is everything that happened to it afterward. Both are what let you trace a bad output back to its real cause, and find every derived copy when data must be deleted.