Séminaire MODAL'X : Rafal Kulil (Université d'Ottawa)

Résumé : Stochastic Gradient Descent (SGD) has become the most popular optimization algorithm used in Machine Learning. It was empirically observed that some characteristic of SGD exhibit heavy tails.
In this talk we will (partially) explain the mechanisms that lead to emergence of heavy tails in both online and offline SGDs.Furthermore, we will study SGD through the lenses of stochastic processes obtaining stable convergence. The main tools to prove relevant results are borrowed from the theory of heavy-tailed time series.