Hyperparameters

Hyperparameters are configuration settings defined before training a machine learning model. They shape how the model learns and how complex it can become. Unlike model parameters, they are not learned from the data; instead, they guide the training process and influence how efficiently and accurately the model discovers patterns.

Role

Determines the strategy of learning:

• Learning rate - How fast or slow the model learns.

• Trees, leaves - How big or small the model is.

• Overfitting or Underfitting - How well the model fits the data without overfitting or underfitting .

• Include or Not - How much time and resources the training will requires.

Examples

• Learning Rate:
  This setting controls the size of each learning step during optimization. If the value is too high, the model may diverge; if it’s too low, the learning process may be slow.

• Number of Trees:
  In a Random Forest or Gradient Boosting model, this parameter determines how many trees are constructed. Increasing the number of trees improves accuracy but slows down training.

• Regularization Strength:
  This applies a penalty to large weights to reduce overfitting. If the penalty is too strong, the model may underfit; if it’s too weak, overfitting may occur.

  Note:
  Hyperparameters must be defined before the training process begins.
  The model does not adjust these values automatically; you must choose or tune them yourself.
  It affects the model’s accuracy, training speed, overall training time, and ability to generalize to new data.
  Choose select manually through trial and error.

Different types of hyperparameters

Hyperparameters has different types of algorithm and multiple categories that makes the algorithm stable:

1. Structural Hyperparameters

   Definition: This defines the model’s architecture or scale.

   Role: Determines the model’s complexity and its ability to capture patterns.

   Example: NumberOfTrees, NumberOfLeaves

2. Learning Rate & Step Size

   Definition: Controls how fast the model updates during training.

   Role: Impacts how quickly the model converges and the stability of the training process.

   Example: LearningRate

3. Regularization Hyperparameters

   Definition: Applies penalties to restrict model complexity.

   Role: Reduces overfitting and improves generalization.

   Example: L1Regularisation, L2Regularisation

4. Optimization / Training Control

   Definition: Settings that guide the optimization process.

   Role: Affects the speed and efficiency of convergence.

   Example: HistorySize, OptimisationTolerance

5. Data Sampling / Subsampling

   Definition: Determines fraction of features or data used per iteration.

   Role: Adds randomness to prevent overfitting and speed up training.

   Example: FeatureFraction

6. Early Stopping / Convergence

   Definition: Defines the criteria used to halt training before completion.

   Role: Saves computation and avoids overfitting.

   Example: ConvergenceCheckFrequency

7. Algorithm-Specific Hyperparameters

   Definition: Applies unique settings specific to an algorithm.

   Role: Enables or restricts special behaviors.

   Example: EnforceNonNegativity

8. General (Hardware Utilization)

   Definition: Controls CPU/thread usage during training.

   Role: Improves performance with parallel processing.

   Example: NumberOfThreads