What is a Learning Rate?
The learning rate is a critical hyperparameter that controls how large of an adjustment a machine learning model makes to its internal parameters in response to the errors it calculates during training.
How does it work?
When a model makes a prediction and realizes it was wrong, it updates its internal numbers to do better next time. The learning rate dictates the size of that update step.
- Too high: The model takes massive steps, wildly overcorrecting its mistakes, making the training process unstable and unable to settle on the right answer.
- Too low: The model takes microscopic steps, making the training process unnecessarily slow, or causing the model to get stuck on a suboptimal solution.
What is a simple example?
Imagine trying to find the lowest point in a dark valley using a flashlight. The learning rate is the size of your steps. If your steps are huge (high learning rate), you might step completely over the lowest point and end up on the opposite hill. If your steps are tiny (low learning rate), it will take you a month to reach the bottom.
Why does it matter?
There is no single ideal learning rate that works for every model. Finding the "goldilocks" value—or using advanced optimizers that dynamically adjust the rate during training—is one of the most important steps in successfully training an AI.