We consider a contextual stochastic optimization (CSO) problem, where one has observations of the uncertain parameters together with concurrent observations of covariates, and the goal is to choose decisions that minimize expected cost conditioned on new covariate observations. The empirical residuals-based sample average approximation (ER-SAA) of the CSO problem constructs scenarios of uncertainty by combining regression predictions with residuals. This approach implicitly assumes that the regression model captures the relationship among covariates, uncertain factors, and decisions with reasonable accuracy. However, since this relationship is fundamentally unknown, identifying a suitable regression model to approximate it remains a central challenge, commonly referred to as \textit{model uncertainty}. Overlooking such uncertainty can yield poor approximations and suboptimal decisions. To address this, we introduce a model-uncertainty–aware residuals-based SAA framework (MUR-SAA), formulated as a distributionally robust optimization problem on the regression model. By treating the regression function as a random element and optimizing over the worst-case expected cost across all plausible distributions of the unknown regression function, MUR-SAA produces solutions that are robust to model misspecification. We establish, for the first time, efficient mechanisms for quantifying the stability of general residuals-based stochastic programs and our MUR-SAA model. Moreover, we identify conditions under which our MUR-SAA model enjoys desirable asymptotic convergence and finite-sample properties. We derive equivalent tractable reformulations of MUR-SAA and develop a decomposition algorithm to solve them. We illustrate our theoretical results and demonstrate the advantages of our MUR-SAA model using a contextual portfolio optimization problem.