In this paper, we propose submodular ambiguity sets for distributionally robust optimization and showcase its usefulness in modeling discrete and continuous uncertainty. From a modeling perspective, we propose two new tractable submodular ambiguity sets that combine marginal distributions and covariance information for discrete uncertainty and mean and covariance information for continuous uncertainty. From a computational perspective, for discrete uncertainty, we show that a class of distributionally robust optimization problems is solvable in polynomial time and in special cases, as a polynomial sized linear program. With continuous uncertainty, we show that it is solvable approximately up to an additive error in pseudo-polynomial time and in special cases, as a polynomial sized semidefinite program. We provide numerical evidence on the usefulness of incorporating covariance information through these ambiguity sets in resource allocation problems where fairness is of concern and limited data is available. In addition, we provide theoretical bounds in stylized settings to quantify the improvement provided by these ambiguity sets. The paper highlights that submodular ambiguity sets form the natural discrete counterpart of convex ambiguity sets and supplements it for continuous uncertainty, both in modeling and computation.