What Is Hyperparameter Tuning?
Hyperparameter tuning is the process of finding the best settings for a machine learning algorithm so that the trained model performs well on unseen data.
The important distinction is:
Model parameters are learned from data. Hyperparameters are settings you choose before or around training.
Simple example
Suppose you're training a neural network.
You might choose:
Learning rate = 0.001
Batch size = 32
Epochs = 20These are hyperparameters.
You can try different combinations:
Learning rate Batch size Epochs
-----------------------------------
0.1 32 20
0.01 32 20
0.001 32 20
0.0001 32 20You train and evaluate each configuration, then select a configuration based on performance on a validation set or another appropriate tuning procedure.
Parameters vs Hyperparameters
This is one of the most important concepts to understand.
Parameters
Parameters are learned by the model during training.
For a simple linear model:
y = wx + bw and b are parameters.
The training algorithm learns them from the data.
Hyperparameters
Hyperparameters control how the learning process or model structure works.
Examples:
Learning rate
Number of trees
Maximum tree depth
Batch size
Number of epochs
Number of hidden layers
Dropout rate
K in KNNYou generally specify these rather than having ordinary training directly learn them from the training examples.
Why Do We Tune Hyperparameters?
Imagine training the same model with three learning rates:
Learning Rate Validation Accuracy
------------------------------------
0.1 72%
0.01 89%
0.001 94%Here, 0.001 performed better on this validation setup.
But choosing a hyperparameter should not simply mean selecting whatever gives the highest score on one dataset split; repeated validation or cross-validation is often used, and the final test set should remain separate for unbiased evaluation.
Example: Random Forest
A Random Forest model has hyperparameters such as:
n_estimators = 100
max_depth = 10
min_samples_split = 2You could test:
n_estimators:
50
100
200
500and:
max_depth:
5
10
20
NoneThis creates different model configurations.
The tuning process evaluates them and identifies configurations that work well according to your chosen validation metric.
Example: KNN
K-Nearest Neighbors has an important hyperparameter called k.
k = 1
k = 3
k = 5
k = 7
k = 11Different values can produce very different behavior.
Conceptually:
Small k
↓
More sensitive to individual examples
Large k
↓
Smoother decision boundaryThe appropriate value depends on the dataset and objective.
Example: Neural Network
A neural network might have:
Learning rate = 0.001
Batch size = 32
Epochs = 50
Hidden layers = 3
Dropout = 0.2You might tune:
Learning rate:
0.1
0.01
0.001
0.0001
Batch size:
16
32
64
128Then compare the resulting validation performance.
Common Hyperparameter-Tuning Methods
1. Manual Search
You choose values yourself:
Try 0.1
Try 0.01
Try 0.001This is easy but can become inefficient.
2. Grid Search
Grid search systematically tests every combination you specify.
For example:
Learning rate = [0.01, 0.001]
Batch size = [32, 64]It evaluates:
0.01 + 32
0.01 + 64
0.001 + 32
0.001 + 64So there are 4 combinations.
For n hyperparameters, each with many candidate values, the number of combinations can grow rapidly.
3. Random Search
Instead of testing every possible combination, random search samples configurations from defined ranges or distributions.
For example:
learning_rate = random value between 0.0001 and 0.1Random search can be more efficient than grid search when only a few hyperparameters have a strong effect on performance.
4. Bayesian Optimization
Bayesian optimization uses results from previous experiments to decide which configuration to try next.
Conceptually:
Try configuration
↓
Measure performance
↓
Learn which regions look promising
↓
Choose next configuration
↓
RepeatThis can reduce the number of expensive training runs.
Hyperparameter Tuning Workflow
A typical process looks like:
Dataset
↓
Split data
↓
Choose model
↓
Define hyperparameter search space
↓
Try configurations
↓
Train models
↓
Evaluate on validation data
↓
Select configuration
↓
Final evaluation on test dataFor example:
100 configurations
↓
Validation
↓
Best configuration
↓
Final test evaluationWhy Not Tune Using the Test Set?
Suppose you repeatedly experiment using the test set:
Try model A → test
Try model B → test
Try model C → test
...Eventually, your choices can become indirectly tailored to that test set.
That makes the final test score less trustworthy as an unbiased estimate of performance on truly unseen data.
A better structure is:
Training set
↓
Learn model parameters
Validation set
↓
Tune hyperparameters
Test set
↓
Final evaluationCross-validation can also be used during the tuning stage, especially when the dataset is not large.
Hyperparameter Tuning vs Training
These are different processes.
Training
The model learns parameters:
Training Data
↓
Algorithm
↓
Learn Parameters
↓
ModelHyperparameter tuning
You search for good configuration settings:
Possible Hyperparameters
↓
Train/Evaluate Models
↓
Compare Validation Results
↓
Choose ConfigurationTogether:
Hyperparameters
↓
Training Process
↓
Learned Parameters
↓
Trained ModelA Real Python Example
Using scikit-learn, grid search can look like this:
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
model = RandomForestClassifier(random_state=42)
param_grid = {
"n_estimators": [100, 200, 500],
"max_depth": [5, 10, 20, None],
"min_samples_split": [2, 5, 10]
}
search = GridSearchCV(
estimator=model,
param_grid=param_grid,
cv=5,
scoring="accuracy",
n_jobs=-1
)
search.fit(X_train, y_train)
print(search.best_params_)
print(search.best_score_)The important part is:
search.best_params_It gives the hyperparameter configuration selected by the search procedure based on the cross-validation results.
An Easy Analogy
Imagine you're cooking rice.
The ingredients are your training data.
The recipe/process is your algorithm.
The amount of:
Water
Cooking temperature
Cooking timeacts like hyperparameters.
You try different settings:
Too much water → Poor result
Too little water → Poor result
Correct amount → Better resultHyperparameter tuning is essentially systematically searching for settings that produce better results.
The Core Idea
Remember this:
Parameters
→ learned from data
Hyperparameters
→ chosen/tuned by the practitioner or tuning procedureAnd:
Hyperparameter Tuning
↓
Try different configurations
↓
Train models
↓
Evaluate on validation data
↓
Select a suitable configurationSo, hyperparameter tuning is the optimization of the model's training/configuration settings, not the direct learning of the model's parameters themselves.