Abstract

We study least squares linear regression over N uncorrelated Gaussian features that are selected in order of decreasing variance. When the number of selected features p is at most the sample size n, the estimator under consideration coincides with the principal component regression estimator; when p > n, the estimator is the least 2 norm solution over the selected features. We give an average-case analysis of the out-of-sample prediction error as p, n, N with p/N and n/N , for some constants and . In this average-case setting, the prediction error exhibits a “double descent” shape as a function of p. We also establish conditions under which the minimum risk is achieved in the interpolating (p > n) regime.