We introduce a new framework, Bayesian Distributionally Robust Optimization (Bayesian-DRO), for data-driven stochastic optimization where the underlying distribution is unknown. Bayesian-DRO contrasts with most of the existing DRO approaches in the use of Bayesian estimation of the unknown distribution. To make computation of Bayesian updating tractable, Bayesian-DRO first assumes the underlying distribution takes a parametric form with unknown parameter and then computes the posterior distribution of the parameter. To address the model uncertainty brought by the assumed parametric distribution, Bayesian-DRO constructs an ambiguity set of distributions with the assumed parametric distribution as the reference distribution and then optimizes with respect to the worst case in the ambiguity set. We show the consistency of the Bayesian posterior distribution and subsequently the convergence of objective functions and optimal solutions of Bayesian-DRO. Our consistency result of the Bayesian posterior requires simpler assumptions than the classical literature on Bayesian consistency. We also consider several approaches for selecting the ambiguity set size in Bayesian-DRO and compare them numerically. Our numerical experiments demonstrate the out-of-sample performance of Bayesian-DRO in comparison with Kullback-Leibler-based (KL-) and Wasserstein-based empirical DRO as well as risk-neutral Bayesian Risk Optimization. Our numerical results shed light on how to choose the modeling framework (Bayesian-DRO, KL-DRO, Wasserstein-DRO) for specific problems, but the choice for general problems still remains an important and open question.