Optimization in AI

Department of Industrial Engineering, Tsinghua University

How does the structure of a learning problem shape the optimizer—and how does that choice affect training? This course connects optimization theory, algorithm design, and the practical behavior of AI training. We will study gradient and stochastic methods, momentum, adaptive optimizers, and weight decay, with attention to convergence, conditioning, gradient noise, and the compute and memory constraints of modern models.

Course information · Schedule · References

Course information

  • Instructor: Jianhao Ma · jianhao@tsinghua.edu.cn
  • Teaching assistant: Yuzhen Chen (陈禹臻), Department of Industrial Engineering
  • Class time: Wednesdays, 9:50–11:25.
  • Location: 六教 6B113.
  • Office hours: Wednesdays, 1–2 pm.

Schedule

Topic Coverage Materials
Introduction Course overview; problem structure and optimizer design  
Mathematical foundations Matrix calculus, backpropagation, autodiff, convexity, optimality, and smoothness  
Gradient descent Convergence, stepsizes, conditioning, and smooth and nonsmooth problems  
Stochastic gradient descent Gradient noise, minibatches, learning-rate schedules, and convergence  
Momentum Heavy-ball dynamics, acceleration, and stochastic averaging  
Adaptive optimizers AdaGrad, RMSProp, Adam, and Muon  
Weight decay L2 regularization, decoupled weight decay, and AdamW  
Selected advanced topics Topics chosen based on course progress and recent developments, including second-order methods  

Guest lecture: Yushun Zhang (DeepSeek). Date and topic to be announced.

References