We consider the portfolio optimization problem with contextual information that is available to better quantify and predict the uncertain returns of assets. Motivated by the regime modeling techniques for the finance market, we consider the setting where both the uncertain returns and the contextual information follow a Gaussian Mixture (GM) distribution. This problem is shown to be equivalent to a nominal portfolio optimization problem where the means and the covariance matrix are adjusted by the contextual information. We then apply robust optimization and propose the robust contextual portfolio optimization problem, which reduces the sensitivity of model parameters used in the Gaussian Mixture Model (GMM). A tractable reformulation is derived to approximate the solution of the robust contextual portfolio optimization problem. We conduct a numerical experiment in the US equity markets, and the results demonstrate the advantage of our proposed model against other benchmark methods.