Stochastic Gradient Descent: Foundations, Variants, and Applications

Overview of Stochastic Gradient Descent (SGD), a foundational optimization algorithm in machine learning. It traces SGD's historical roots back to the Robbins-Monro algorithm, explaining its evolution from a theoretical concept to the dominant method for training large-scale models like deep neural networks.

The text compares SGD to Batch and Mini-Batch Gradient Descent, highlighting their trade-offs in computational cost, memory, and convergence stability, and emphasizing Mini-Batch GD as the current standard due to its hardware compatibility.

Furthermore, it details the critical role of the learning rate and various decay schedules, along with the mathematical conditions for convergence.

Finally, the sources discuss advanced SGD variants like Momentum and Adam, their applications across diverse fields, and future research directions, including automated learning rate selection and distributed training.