Gradient descent is one of the most widely used optimization methods in machine learning. It works by repeatedly updating parameters in the direction that reduces a loss function. While the core idea is simple, real success depends on whether the algorithm converges reliably and efficiently. In convex optimisation, where the loss landscape has a single global minimum, convergence is theoretically well understood. Yet practical outcomes still vary because learning rates and their schedules can make the updates stable, unstable, slow, or oscillatory. Many learners encounter this gap between theory and practice early in an AI course in Delhi, especially when they move from equations to training models on real datasets.
Why Convergence Matters in Convex Optimisation
In a convex function, any local minimum is the global minimum. This is a strong guarantee. However, gradient descent does not “teleport” to that minimum. It takes steps, and the size of those steps determines behaviour.
If the learning rate is too large, updates can overshoot the minimum and bounce back and forth, sometimes even diverging. If it is too small, the algorithm becomes stable but painfully slow, taking many iterations to make meaningful progress. Convergence, therefore, is about balancing two priorities:
- Stability: Avoiding oscillations and divergence.
- Speed: Reaching a good solution in fewer iterations.
Convex theory tells us gradient descent converges under certain learning-rate conditions, but choosing a rate that is both stable and efficient is still a practical skill taught in an AI course in Delhi.
The Role of the Learning Rate in Stability
The learning rate (often written as η) scales the gradient update:
wt+1=wt−η∇f(wt)w_{t+1} = w_t – \eta \nabla f(w_t)wt+1=wt−η∇f(wt)In convex optimisation, a common assumption is that the gradient is L-Lipschitz smooth, meaning the function does not change too abruptly. Under this condition, stability improves when η is chosen below a bound related to L (often η ≤ 1/L, with variants depending on the analysis). Intuitively:
- With smooth convex functions, a moderate learning rate can guarantee steady decrease in the objective.
- With steeper curvature (large L), the same learning rate may become too aggressive, leading to oscillations.
This is why the same learning rate can work well for one problem and fail for another. Understanding how curvature affects safe step sizes is a key part of optimisation fundamentals in an AI course in Delhi.
Common Learning Rate Schedules and Their Convergence Behaviour
A learning rate schedule changes η over time. In convex settings, schedules are often used to combine fast early progress with stable late-stage convergence.
Constant learning rate
A fixed η is simple and can converge quickly when chosen well. In strongly convex, smooth problems, a constant learning rate can give fast progress and even linear convergence in some cases. The risk is sensitivity: if η is slightly too large, you may see oscillations near the optimum; if too small, training drags.
Best use: well-conditioned convex objectives where you can tune η reliably.
Step decay
This schedule reduces η by a factor at predefined milestones (for example, halve it every N iterations). Early iterations use a larger step for speed; later iterations use a smaller step for stability.
Impact: improves stability near the optimum and reduces “bouncing” behaviour.
Exponential decay
η decreases smoothly over time (ηₜ = η₀·γᵗ). This can be more gradual than step decay and may reduce abrupt changes in training behaviour.
Impact: tends to stabilise convergence, though it can shrink too quickly and stall progress if γ is too small.
Inverse time / polynomial decay
ηₜ = η₀ / (1 + kt) or ηₜ = η₀ / tᵅ. In convex optimization, decaying schedules like these are often associated with theoretical convergence guarantees, especially when noise is present (as in stochastic gradient descent).
Impact: strong stability guarantees over time, but can converge slowly late in training if decay is aggressive.
Warmup + decay
Warmup starts with a smaller η and increases to a peak before decaying. While warmup is discussed more in deep learning, it can still be useful when gradients are initially unstable or poorly scaled.
Impact: reduces early instability and then transitions to stable convergence.
How Schedules Affect Stability in Practice
Even in convex optimization, practical training often involves mini-batches, noisy gradients, and imperfect scaling. Learning rate schedules influence stability through three main mechanisms:
- Overshoot control near the minimum
- As parameters get closer to the optimum, gradients shrink, but curvature can still cause oscillations. Lowering η reduces overshoot and helps settle.
- Noise reduction in stochastic updates
- With mini-batches, gradients are noisy. A decaying η reduces variance in the updates, making the path toward the minimum smoother.
- Conditioning sensitivity
- If a convex problem is ill-conditioned (very different curvature in different directions), a learning rate that is safe in one direction may be too large in another. Schedules reduce the chance of persistent oscillations, but the underlying conditioning still matters.
These ideas are easier to grasp when you visualise loss curves and parameter trajectories—something many learners practise in an AI course in Delhi through hands-on optimisation labs.
Practical Guidelines for Choosing a Schedule
Here are simple, reliable rules for convex optimisation tasks:
- Start with a moderate constant η and check whether loss decreases smoothly.
- If you see oscillations or spikes, reduce η or add decay.
- For noisy SGD-style training, consider inverse-time decay to stabilise late training.
- Use step decay when you want simple, controlled transitions.
- Track not just loss, but also gradient norms and update magnitudes to diagnose instability.
A useful habit is to treat learning-rate tuning as a controlled experiment: change one variable at a time and compare curves across runs.
Conclusion
Gradient descent convergence in convex optimization depends heavily on learning rate choices. A learning rate that is too large can cause oscillations or divergence, while one that is too small leads to slow progress. Learning rate schedules help balance speed and stability by using larger steps early and safer steps later, especially when gradients are noisy or curvature varies. With a clear understanding of constant rates, decay schedules, and their stability effects, you can train convex models more reliably and diagnose optimisation issues faster—skills that are commonly strengthened through an AI course in Delhi.