Machine Learning Intermediate: Tuning & Advanced

Parameters are learned by the model (weights); hyperparameters are set by you (max_depth, learning_rate, n_estimators). Tuning means systematically searching for the best hyperparameter combination. The danger is tuning too aggressively on a single validation set — you can overfit the validation data itself. Always use cross-validation inside your search, and hold out a final untouched test set.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20],
    'min_samples_split': [2, 5]
}
grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)
print('Best:', grid.best_params_)

Parameters are learned during training; hyperparameters are set before training.

GridSearchCV tries EVERY combination in your grid — thorough but explodes exponentially as you add parameters. RandomSearchCV samples random combinations — less exhaustive but far more efficient in high dimensions, and research shows it often finds equally good solutions in less time. Rule of thumb: small grids → GridSearch; large search spaces → RandomSearch.

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Exhaustive
param_grid = {'max_depth': [3, 5, 10], 'n_estimators': [50, 100]}
grid = GridSearchCV(model, param_grid, cv=5)

# Random
param_dist = {'max_depth': list(range(1, 20)), 'n_estimators': list(range(10, 500))}
random = RandomizedSearchCV(model, param_dist, n_iter=30, cv=5, random_state=42)

When the hyperparameter space is too large for exhaustive search.

Grid and random search waste time on bad regions. Bayesian optimization learns from previous trials — it builds a model of 'which hyperparameters give good scores' and focuses search there. Optuna is the modern library for this: define an objective function, let it sample intelligently, and it converges to optimal hyperparameters in far fewer trials. This is what serious teams use.

import optuna

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 10, 500)
    max_depth = trial.suggest_int('max_depth', 1, 30)
    model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)
print('Best:', study.best_params)

It learns from previous trials and focuses on promising regions.

Plain k-fold is wrong for many real datasets. StratifiedKFold preserves class proportions (critical for imbalanced data). GroupKFold keeps related samples together (patients, users — prevents leakage). TimeSeriesSplit respects chronological order (no future data in training). Using the wrong split leaks information and inflates your scores dishonestly.

from sklearn.model_selection import StratifiedKFold, TimeSeriesSplit, GroupKFold

# Imbalanced classes -> stratify
skf = StratifiedKFold(n_splits=5)
# Time-ordered data -> chronological
tss = TimeSeriesSplit(n_splits=5)
# Related samples -> group
gkf = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, cv=skf, groups=groups)

TimeSeriesSplit — never use future data to train.

Feature engineering is where domain knowledge meets ML — and it often matters more than model choice. Interaction features combine variables (price × quantity). Target encoding replaces categories with their average target value (powerful but risks leakage — must be done inside CV). Binning, ratios, and temporal aggregations all create signal. The best feature engineer beats the best algorithm.

# Interaction feature
import pandas as pd
df['price_per_unit'] = df['price'] / df['quantity']
df['income_age_ratio'] = df['income'] / (df['age'] + 1)

# Target encoding (inside CV only!)
import category_encoders as ce
encoder = ce.TargetEncoder()
X_encoded = encoder.fit_transform(X_train, y_train)

price × quantity, or age / income.

When one class dominates (fraud, rare disease), models learn to predict the majority and ignore the minority. Fixes: resample — SMOTE generates synthetic minority examples; undersample the majority; or use class_weight='balanced' so minority mistakes cost more. Critically, evaluate with precision/recall/F1, not accuracy. Imbalanced learning is a whole subfield because real-world data is rarely balanced.

from imblearn.over_sampling import SMOTE
from sklearn.ensemble import RandomForestClassifier

smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

model = RandomForestClassifier(class_weight='balanced')
model.fit(X_resampled, y_resampled)

It creates synthetic minority examples so the model sees more of the rare class.

Real datasets mix numeric and categorical columns needing different preprocessing. ColumnTransformer applies different transforms to different columns, and Pipeline chains preprocessing + model into one object. Together they make your workflow clean, reproducible, and leakage-proof — preprocessing is automatically applied correctly to train and test. This is the professional way to structure ML code.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier

preprocessor = ColumnTransformer([
    ('num', StandardScaler(), ['age', 'income']),
    ('cat', OneHotEncoder(), ['city', 'job'])
])
pipeline = Pipeline([
    ('prep', preprocessor),
    ('model', RandomForestClassifier())
])
pipeline.fit(X_train, y_train)

Preprocessing is fit only on training data and applied consistently to test.

Sometimes you need preprocessing scikit-learn doesn't have — a domain-specific cleaning, a custom scaling, a feature creation step. Custom transformers (classes with fit/transform) plug into pipelines seamlessly, keeping your logic encapsulated and reusable. This is how production teams encode their specific business logic into the ML workflow.

from sklearn.base import BaseEstimator, TransformerMixin

class LogTransformer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None): return self
    def transform(self, X):
        return X.apply(lambda col: np.log1p(col))

pipeline = Pipeline([
    ('log', LogTransformer()),
    ('model', model)
])

fit() and transform().

Stacking takes ensembling further: instead of voting, a meta-model learns how to best combine base models' predictions. You train diverse base models (RF, SVM, KNN), then train a meta-model on their out-of-fold predictions. Blending is a simpler version using a holdout set. Stacking frequently wins competitions because the meta-model learns which base model to trust when.

from sklearn.ensemble import StackingClassifier, RandomForestClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

base_models = [
    ('rf', RandomForestClassifier()),
    ('svm', SVC(probability=True)),
    ('knn', KNeighborsClassifier())
]
stack = StackingClassifier(base_models, final_estimator=model)
stack.fit(X_train, y_train)
print(stack.score(X_test, y_test))

Voting averages; stacking trains a meta-model on base predictions.

A model may predict 0.9 'probability' but be right only 60% of the time on such predictions — that's miscalibration. Calibration fixes probabilities so they reflect true likelihood, which matters for risk scores, fraud alerts, and any decision threshold. Random forests and SVMs are often miscalibrated; logistic regression is typically well-calibrated. CalibratedClassifierCV wraps a model to fix this.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import SVC

model = CalibratedClassifierCV(SVC(), method='isotonic', cv=5)
model.fit(X_train, y_train)
probs = model.predict_proba(X_test)

It's actually correct about 80% of the time.

Binary classifiers don't directly handle 3+ classes, so strategies adapt them. One-vs-Rest (OVR): train one binary classifier per class (class vs all others). One-vs-One (OVO): train a classifier for every pair of classes. OVR is efficient; OVO is accurate for small class counts but quadratic cost. scikit-learn handles this automatically for most models, but knowing the strategy helps tune and debug.

from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC

model = OneVsRestClassifier(SVC())
model.fit(X_train, y_train)
pred = model.predict(X_test)
print('Classes predicted:', pred[:5])

10 pairwise classifiers.

Applying ML to time series requires special care. Walk-forward validation trains on past data and tests on the next window — sliding forward, never peeking into the future. Lag features (yesterday's value, last week's average) turn time dependencies into features ML can learn. This hybrid 'time features + ML model' approach often beats classical forecasting on messy real-world data.

# Walk-forward validation
from sklearn.model_selection import TimeSeriesSplit

tss = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tss.split(X):
    model.fit(X[train_idx], y[train_idx])
    score = model.score(X[test_idx], y[test_idx])
    print(score)

A feature using past values (e.g., yesterday's sales) to predict today.

Anomaly detection finds data points that don't fit the pattern. Isolation Forest isolates anomalies quickly (fewer splits to separate them). One-Class SVM learns a boundary around normal data and flags anything outside. Both are unsupervised — train on normal data, detect deviations. Applications: fraud, equipment failure, network intrusion, manufacturing defects.

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.05, random_state=42)
model.fit(X)
anomalies = model.predict(X)  # -1 = anomaly, 1 = normal
print('Anomalies detected:', (anomalies == -1).sum())

Anomalies are rare/unknown; learning normal patterns lets you flag deviations.

PCA preserves global variance but often hides clusters. t-SNE and UMAP are non-linear techniques that preserve LOCAL structure — nearby points stay nearby in 2D, revealing clusters PCA misses. They're visualization gold for exploring high-dimensional data (images, embeddings), but they're NOT for feature extraction (don't train on their output — they're for seeing, not feeding models).

import umap

reducer = umap.UMAP(n_components=2, random_state=42)
embedding = reducer.fit_transform(X)
print('Embedding shape:', embedding.shape)

Local structure — neighboring points stay neighbors.

AutoML automates the tedious parts — trying dozens of models, tuning hyperparameters, building ensembles — and returns the best. Libraries like AutoGluon and auto-sklearn can beat hand-tuned models on many tabular tasks in minutes. AutoML is a baseline accelerator: use it to get a strong starting model quickly, then improve from there. It doesn't replace understanding, but it saves enormous time.

from autogluon.tabular import TabularPredictor

predictor = TabularPredictor(label='target').fit(train_df)
predictions = predictor.predict(test_df)
print(predictor.leaderboard())

No — it accelerates baseline building; understanding is still needed.

With dozens of algorithms, choosing is overwhelming. A systematic strategy: start simple (logistic regression, decision tree) as baselines, try ensembles (random forest) next, then gradient boosting (XGBoost/LightGBM) — the usual winner on tabular data. Compare with cross-validation, pick top few, tune them, and ensemble the best. This disciplined process beats random trial-and-error.

models = {
    'Logistic': LogisticRegression(),
    'Tree': DecisionTreeClassifier(),
    'RF': RandomForestClassifier(),
    'XGB': XGBClassifier(eval_metric='logloss')
}
for name, m in models.items():
    score = cross_val_score(m, X, y, cv=5).mean()
    print(f'{name}: {score:.3f}')

Gradient boosting (XGBoost/LightGBM).

Leakage is when training data accidentally contains information about the target that won't be available at prediction time — future data, target encoding done wrong, preprocessing fit on the full dataset. The result: great validation scores, terrible production performance. Leakage is the #1 silent killer of ML projects. Prevention: pipelines, proper CV, and never touching test data until the very end.

# LEAKAGE BUG (never do this)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # fit on ALL data incl test

# CORRECT (inside pipeline, fit on train only)
pipeline = Pipeline([('scaler', StandardScaler()), ('model', model)])
pipeline.fit(X_train, y_train)

Fitting preprocessing on the full dataset (including test).

Tuning an ensemble is different from tuning a single model — you're balancing individual model strength against diversity. More estimators reduces variance but has diminishing returns; deeper trees add strength but reduce diversity. The art is finding the combination that maximizes the ensemble's collective accuracy. Techniques: tune base model individually, then tune ensemble aggregation.

param_grid = {
    'n_estimators': [100, 200, 500],
    'max_features': ['sqrt', 'log2', None],
    'max_depth': [None, 10, 20],
    'min_samples_leaf': [1, 2, 4]
}
grid = RandomizedSearchCV(RandomForestClassifier(), param_grid, n_iter=50, cv=5)
grid.fit(X, y)

The number of trees — more reduces variance with diminishing returns.

Big models are expensive to deploy. Compression shrinks them: pruning removes low-importance tree branches, quantization reduces numerical precision (float32 → int8), and distillation trains a small 'student' model on a big 'teacher's' predictions. For tree ensembles, limiting depth and feature count already compresses. Smaller models = faster inference, lower memory, cheaper serving.

# Compress via depth limiting + feature subsampling
compact = RandomForestClassifier(
    n_estimators=50,
    max_depth=5,
    max_features='sqrt'
)
compact.fit(X_train, y_train)
print('Size vs accuracy trade-off balanced')

Faster inference, lower memory, cheaper cloud serving.

Your capstone: a Kaggle-style competition project. You'll build a full solution — load data, engineer features, build pipelines, tune with Optuna, stack models, validate with proper CV (avoiding leakage), and document your approach. This mirrors exactly how winning Kaggle solutions are built and is the strongest possible portfolio piece for ML roles.

pipeline = Pipeline([
    ('prep', preprocessor),
    ('model', StackingClassifier([
        ('rf', RandomForestClassifier()),
        ('xgb', XGBClassifier(eval_metric='logloss')),
        ('svm', SVC(probability=True))
    ]))
])
study = optuna.create_study(direction='maximize')
study.optimize(lambda t: objective(t, pipeline, X, y), n_trials=50)
print('Final best score:', study.best_value)

Verify no data leakage — your validation must reflect true generalization.