Classic bias-variance trade-off in modern statistical learning context: a position paper and theoretical review

Abstract

Classical statistical learning theory suggests that models learn data intricacies by fitting a parsimonious model, which can lead to irreducible error components. Overfitting occurs when a model fits the data so well that it becomes too good to be true, and when assessed for unseen data, it performs poorly. The bias-variance trade-off deals with balancing complexity and generalization error. Increasing the number of parameters in models increases the chances of poorly sampling in specific directions, leading to higher variance. This phenomenon is known as benign overfitting. This paper presents a position paper and brief theoretical review that synthesizes key analytical results from the growing literature on benign overfitting. It focuses on overparameterized linear regression analysis, which has two major benefits: decreasing the likelihood of overfitting and uncovering hidden trends in data. However, the generalization bounds for overparameterized models do not explain important empirical observations, and the case of dataset shift remains unexplored in this regime. Rather than proposing a new algorithm or empirical method, the paper aims to clarify conceptual mechanisms underlying benign overfitting and to highlight limitations of current theoretical explanations, particularly under dataset shift. The paper concludes by identifying open theoretical questions relevant to the foundational understanding of modern machine learning systems.