Abstract

The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objec tive function. All of this appears to call for a f batch approach, but since small batch sizes give rise to faster algorithms with better generalizati properties, L-BFGS is currently not considered an algorithm of choice for large-scale machine learning applications. One need not, however, choose between the two extremes represented by the full batch or highly stochastic regimes, and may instead follow a progressive batching approach in which the sample size increases during the course of the optimization. In this paper, we present a new version of the L-BFGS algorithm that combines three basic components — progressive batching, a stochastic line search, and stabl quasi-Newton updating — and that performs well on training logistic regression and deep neural networks. We provide supporting convergence theory for the method.