We consider a distributionally robust optimization problem where the ambiguity set of probability distributions is characterized by a tractable conic representable support set and expectation constraints. Specifically, we propose and motivate a new class of infinitely constrained ambiguity sets in which the number of expectation constraints could potentially be infinite. We show how the infinitely constrained ambiguity set can be used to incorporate covariance and entropic dominance in its description. In particular, we demonstrate that our proposed entropic dominance approach can improve the characterization of stochastic independence over existing approach based on covariance information. Although the corresponding distributionally robust optimization problem may not necessarily lead to tractable reformulations, we approach the problem by solving a sequence of tractable distributionally robust optimization problems, each over a relaxed and finitely constrained ambiguity set. When incorporating covariance information in the ambiguity set, we show that the subproblems are in the form of a second order conic program, which is a more computationally attractive format than a positive semidefinite program. We show favorable results in our computational study that this approach converges reasonably well.