In several application domains, discrete choice models have become a popular tool to accurately predict complex choice behavior within the classical predict-then-optimize paradigm. Due to a variety of possible error sources, however, estimated choice models may be subject to ambiguity, which may induce different optimal decisions of highly varying quality. While previous studies focused on reducing the uncertainty within a nominal choice model, this study approaches the issue of ambiguity from a different angle by directly mitigating choice model ambiguity associated with a given set of predictive models in terms of their ability to yield optimal decisions. To this end, we propose a framework and a set of performance metrics that aim to assess the reliability of choice models and their induced decisions, therefore enabling the decision-maker to identify choice models that are likely to produce high quality decisions. While the framework can be applied to any context in which decisions are optimized based on choice models, we explicitly focus on the case of rank-based choice models for assortment optimization. In this context, we propose a stochastic and a robust optimization model, both of which can directly optimize on an ambiguity set, composed of different choice models. For the robust optimization model, we provide a decomposition method to solve large problem instances in a fraction of the original computing times by solving the model directly. Finally, we provide extensive sets of computational experiments, quantifying the ambiguity present in our application context, as well as the advantage of using our proposed approaches. Specifically, we show that (i) revenue estimates can vary by as much as 60% among assortments optimized on different choice models estimated under the same conditions, averaging 82% of the optimal revenue; (ii) choice models that score best according to any of the proposed metrics induce assortments with 4%-13% higher revenues than the average revenue estimates stemming from the individual choice models; (iii) both the stochastic and the robust models reliably produce assortments improving the revenue an additional 3-12% of the optimal revenue value, and (iv) selecting only a few choice models according to their scores from the performance metric can further improve the revenue by 13% to an average of 95% of the optimal revenue.