Rahul Mazumder

Sparse Regression at Scale: Branch-and-Bound rooted in First-Order Optimization

  • Hussein Hazimeh
  • Rahul Mazumder
  • Ali Saab
    • Categories (Mixed) Integer Nonlinear Programming, Branch and Cut Algorithms, Second-Order Cone Programming Tags , , , ,

    We consider the least squares regression problem, penalized with a combination of the L0 and L2 norms (a.k.a. L0 L2 regularization). Recent work presents strong evidence that the resulting L0-based estimators can outperform popular sparse learning methods, under many important high-dimensional settings. However, exact computation of L0-based estimators remains a major challenge. Indeed, state-of-the-art mixed … Read more

    Computing Estimators of Dantzig Selector type via Column and Constraint Generation

  • Rahul Mazumder
  • Stephen Wright
  • Andrew Zheng
    • Categories Linear Programming, Statistics Tags , ,

    We consider a class of linear-programming based estimators in reconstructing a sparse signal from linear measurements. Specific formulations of the reconstruction problem considered here include Dantzig selector, basis pursuit (for the case in which the measurements contain no errors), and the fused Dantzig selector (for the case in which the underlying signal is piecewise constant). … Read more

    A convex integer programming approach for optimal sparse PCA

  • Santanu S. Dey
  • Rahul Mazumder
  • Guanyi Wang
    • Categories (Mixed) Integer Nonlinear Programming, 0-1 Programming

    Principal component analysis (PCA) is one of the most widely used dimensionality reduction tools in scientific data analysis. The PCA direction, given by the leading eigenvector of a covariance matrix, is a linear combination of all features with nonzero loadings—this impedes interpretability. Sparse principal component analysis (SPCA) is a framework that enhances interpretability by incorporating … Read more

    Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods

  • Robert M. Freund
  • Rahul Mazumder
  • Paul Grigas
    • Categories Convex Optimization, Data-Mining, Statistics Tags , , ,

    Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists when the data is … Read more

    Learning a Mixture of Gaussians via Mixed Integer Optimization

  • Hari Bandi
  • Dimitris Bertsimas
  • Rahul Mazumder
    • Categories 0-1 Programming, Applications - OR and Management Sciences, Statistics Tags ,

    We consider the problem of estimating the parameters of a multivariate Gaussian mixture model (GMM) given access to $n$ samples $\x_1,\x_2,\ldots ,\x_n \in\mathbb{R}^d$ that are believed to have come from a mixture of multiple subpopulations. State-of-the-art algorithms used to recover these parameters use heuristics to either maximize the log-likelihood of the sample or try to … Read more

    Sparse principal component analysis and its l1-relaxation

  • Santanu S. Dey
  • Rahul Mazumder
  • Marco Molinaro
  • Guanyi Wang
    • Categories (Mixed) Integer Nonlinear Programming, Data-Mining

    Principal component analysis (PCA) is one of the most widely used dimensionality reduction methods in scientific data analysis. In many applications, for additional interpretability, it is desirable for the factor loadings to be sparse, that is, we solve PCA with an additional cardinality (l0) constraint. The resulting optimization problem is called the sparse principal component … Read more

    The Trimmed Lasso: Sparsity and Robustness

  • Dimitris Bertsimas
  • Rahul Mazumder
  • Martin Copenhaver
    • Categories Convex and Nonsmooth Optimization, Nonlinear Optimization, Statistics

    Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural properties and in doing … Read more

    An Extended Frank-Wolfe Method with “In-Face” Directions, and its Application to Low-Rank Matrix Completion

  • Robert M. Freund
  • Rahul Mazumder
  • Paul Grigas
    • Categories Convex Optimization, Data-Mining Tags , , , , ,

    We present an extension of the Frank-Wolfe method that is designed to induce near-optimal solutions on low-dimensional faces of the feasible region. We present computational guarantees for the method that trade off efficiency in computing near-optimal solutions with upper bounds on the dimension of minimal faces of iterates. We apply our method to the low-rank … Read more

    A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives

  • Robert M. Freund
  • Rahul Mazumder
  • Paul Grigas
    • Categories Data-Mining, Nonsmooth Optimization, Statistics Tags , , , ,

    In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FS-epsilon) and least squares boosting (LS-Boost-epsilon), can be viewed as subgradient descent to minimize the loss function defined … Read more