ML Chapter 5 | Train-Test Split, K-Fold Cross-Validation, Overfitting Prevention

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print("Train size:", len(X_train))
print("Test size:", len(X_test))

from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier()

scores = cross_val_score(model, X, y, cv=5)
print("Scores:", scores)
print("Average Score:", scores.mean())

# First split: train vs temp
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Second split: validation vs test
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42
)

print(len(X_train), len(X_val), len(X_test))

from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)

for train_index, test_index in split.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

Assignments

Assignment 1 – Simple Train/Test Split Practice

Take any supervised dataset. Split it into train and test sets (e.g. 80-20 or 70-30). Train a basic model and evaluate on the test set. Report train accuracy and test accuracy.

Hint: Use `train_test_split()` from scikit-learn with `random_state` to ensure reproducibility. :contentReference[oaicite:2]{index=2}

Assignment 2 – Impact of Split Ratio

Use the same dataset, but try different test-size splits (e.g. 10%, 20%, 30%, 40%). For each split, train and test the model. Compare how the performance metrics (accuracy, error) change with split ratio.

Hint: Smaller test set → more training data but less reliable test estimate; larger test set → vice versa. :contentReference[oaicite:3]{index=3}

Assignment 3 – K-Fold Cross-Validation vs Single Split

Take a dataset and apply both (a) single train/test split and (b) K-Fold Cross-Validation (e.g. k = 5). Compare the results (mean accuracy, variability across folds, train vs test performance).

Hint: Use `KFold` or `cross_val_score`. K-Fold usually gives more stable estimates and reduces bias from random split. :contentReference[oaicite:4]{index=4}

Assignment 4 – Cross-Validation on Small Dataset

Select or simulate a small dataset (few hundred records). Apply 5-fold cross-validation and observe how validation scores vary across folds. Explain why CV is especially helpful on small datasets.

Hint: Single split might give misleading high or low score due to unlucky partition; CV spreads risk across multiple folds. :contentReference[oaicite:5]{index=5}

Assignment 5 – Overfitting Detection via Train vs Validation Performance

Train a model on a dataset and record training accuracy and cross-validation (or test) accuracy. Try increasing model complexity (e.g. depth of decision tree, number of features) and observe when overfitting appears.

Hint: Overfitting occurs when train accuracy is high but test/validation accuracy drops. This shows the model memorizes training data instead of generalizing. :contentReference[oaicite:6]{index=6}

Assignment 6 – Use Stratified Sampling (for Classification)

Take an imbalanced classification dataset. Use stratified train/test split and stratified K-Fold cross-validation. Compare performance vs simple random split. Report class distribution and model metrics.

Hint: Stratified splitting ensures the class proportions are preserved in both splits, avoiding bias from class imbalance. :contentReference[oaicite:7]{index=7}

Assignment 7 – Hyperparameter Tuning with Cross-Validation

Pick an algorithm with hyperparameters (e.g. decision tree max_depth, number of neighbors in KNN). Use K-Fold CV to tune the hyperparameter. Show how CV helps pick a better hyperparameter than a single train/test split.

Hint: For each hyperparameter value, compute CV scores and choose the one with best average performance. :contentReference[oaicite:8]{index=8}

Assignment 8 – Compare Overfitting vs Underfitting with Split and CV

Design two models on the same dataset: one simple (underfitting), one very complex (overfitting). Use both single split and K-Fold CV to evaluate. Compare which evaluation method better reveals overfitting or underfitting.

Hint: CV tends to reveal instability or poor generalization for overfitted models more reliably than single split. :contentReference[oaicite:9]{index=9}

Assignment 9 – Nested Cross-Validation Conceptual Assignment

Read about nested cross-validation (train/validate/test inside CV). Describe when and why nested CV is useful (e.g. hyperparameter tuning + unbiased evaluation). Propose a scenario/dataset where nested CV would benefit.

Hint: Nested CV helps avoid selection bias — important when tuning many hyperparameters or comparing many models. :contentReference[oaicite:10]{index=10}

Assignment 10 – Report: Choosing Evaluation Strategy for a Given Problem

Take a hypothetical or real ML problem (with dataset description). Based on dataset size, class balance, and problem type, decide whether to use simple train/test split, K-Fold CV, or nested CV — and justify your choice in a short write-up.

Hint: Consider dataset size (small vs large), class balance, overfitting risk, and computational budget. :contentReference[oaicite:11]{index=11}