## Breaking the Span Assumption Yields Fast Finite-Sum MinimizationRobert Hannah, Yanli Liu, Daniel O'Connor, and Wotao Yin NeurIPS’18: ## OverviewIn this paper, we show that SVRG and SARAH (two stochastic algorithms for convex finite sum minimization) can be modified to be fundamentally faster than all of the other first-order stochastic algorithms that minimize the sum of smooth functions, such as SAGA, SAG, SDCA, and SDCA without duality. Most finite sum stochastic algorithms follow what we call the “span assumption”: Their updates are in the span of a sequence of component gradients chosen in a random IID fashion. In the big data regime, where the condition number , the span assumption prevents algorithms from converging to an approximate solution of accuracy in less than iterations. SVRG and SARAH do not follow the span assumption since they are updated with a hybrid of full-gradient and component-gradient information. We show that because of this, they can be up to times faster. In particular, to obtain an accuracy for and , modified SVRG requires iterations, whereas algorithms that follow the span assumption require iterations. Moreover, we present lower bound results that show this speedup is optimal, and provide analysis to help explain why this speedup exists. With the understanding that the span assumption is a point of weakness of finite sum algorithms, future work may purposefully exploit this to yield even faster algorithms in the big data regime. ## Citation
« Back |