Yaroslav Bulatov·Oct 14, 2023Introduction to residual correctionBetter formated version (LaTex) of this post on Ghost
Yaroslav Bulatov·Feb 16, 2023Gradient descent under harmonic eigenvalue decayBetter formatted version on ghost (due to Latex support)
Yaroslav Bulatov·Jan 30, 2023Critical batch-size and effective dimension in Ordinary Least SquaresNote: a better formatted version (due to lack of LaTeX support on medium) is here
Yaroslav Bulatov·Dec 27, 2021optimal learning rate for Gradient Descent on a high-dimensional quadraticA better formatted version of this article is on Ghost, which has proper Latex support…
Yaroslav Bulatov·Dec 15, 2021How many matmuls are needed to compute Hessian-vector products?Suppose you have a simple composition of d dense functions. Computing Jacobian needs d matrix multiplications. What about computing Hessian…
Yaroslav Bulatov·Jul 9, 2021How to do matrix derivativesSuppose you have the following scalar function of matrix variable W.A response icon1A response icon1
Yaroslav Bulatov·Aug 30, 2019Using “Evolved Notation” to derive the Hessian of cross-entropy lossI was recently reminded of a lesson learned at Stephen Boyd’s Convex Optimization class at Stanford a few years ago, back when Google was…
Yaroslav Bulatov·Jul 21, 2019Large-scale AI and sharing of modelsBackgroundA response icon1A response icon1
Yaroslav Bulatov·Jun 25, 2019ICLR Optimization papers IIISelf-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response FunctionsA response icon1A response icon1