Arcadia AI — Master's Program
EN
РусскийO'zbekchaEnglishDeutschSrpskiहिन्दी
Program  /  Semester 2 — Modern AI Stack
Course

Optimization for ML: SGD, Adam, Regularization, Schedules

Optimization as the engine of learning: from theory to modern adaptive methods

Instructor to be announced

About the course

This course covers the optimization methods underlying neural network training. It examines theoretical convergence guarantees, the geometry of the loss landscape, and practical techniques for stabilizing training. Understanding optimization is critical for the LLM and ML Engineering courses, and for writing original research papers.

What you'll learn

✓Derive convergence conditions for SGD and Adam on convex and non-convex problems
✓Choose and tune a learning-rate schedule based on the properties of a specific problem
✓Diagnose optimization issues (exploding/vanishing gradients, stagnation) and apply corrective techniques
✓Implement hyperparameter search using Bayesian optimization

Key topics

•Convex optimization: properties, convergence theorems
•SGD, Momentum, Nesterov; mini-batch theory
•Adaptive methods: AdaGrad, RMSProp, Adam, AdamW
•Learning-rate schedules: cosine annealing, warmup, cyclical LR
•Neural network landscapes: saddle points, plateaus, sharp/flat minima
•Gradient clipping and gradient normalization
•Regularization via the optimizer: weight decay, SWA, SAM
•Constrained optimization and proximal methods
•Hyperparameter search: Bayesian optimization, Population-Based Training
This description was generated automatically and has not yet been reviewed by an instructor — it's a draft for discussion.