Quantifying the risk of unfortunate events occurring, despite limited distributional information, is a basic problem underlying many practical questions. Indeed, quantifying constraint violation probabilities in distributionally robust programming or judging the risk of financial positions can both be seen to involve risk quantification, notwithstanding distributional ambiguity. In this work we discuss worst-case probability and conditional value-at-risk (CVaR) problems, where the distributional information is limited to second-order moment information in conjunction with structural information such as unimodality and monotonicity of the distributions involved. We indicate how exact and tractable convex reformulations can be obtained using standard tools from Choquet and duality theory. Our reformulations can be embedded conveniently into higher-level problems such as distributionally robust programs. We make our theoretical results concrete with a stock portfolio pricing problem and an insurance risk aggregation example.