Markov decision process (MDP) is a decision making framework where a decision maker is interested in maximizing the expected discounted value of a stream of rewards

received at future stages at various states which are visited according to a controlled Markov

chain. Many algorithms including linear programming methods are available in the literature

to compute an optimal policy when the rewards and transition probabilities are deterministic.

In this paper, we consider an MDP problem where the transition probabilities are known and

the reward vector is a random vector whose distribution is partially known. We formulate

the MDP problem using distributionally robust chance-constrained optimization framework

under various types of moments based uncertainty sets, and statistical-distance based uncertainty sets defined using phi-divergence and Wasserstein distance metric. For each type of

uncertainty set, we consider the case where a random reward vector has either a full support

or a nonnegative support. For the case of full support, we show that the distributionally robust

chance-constrained Markov decision process is equivalent to a second-order cone programming problem for the moments and phi-divergence distance based uncertainty sets, and it

is equivalent to a mixed-integer second-order cone programming problem for an Wasserstein distance based uncertainty set. For the case of nonnegative support, it is equivalent to

a copositive optimization problem and a biconvex optimization problem for the moments

based uncertainty sets and Wasserstein distance based uncertainty set, respectively. As an

application, we study a machine replacement problem and illustrate numerical experiments

on randomly generated instances.

## Article

View Distributionally robust chance-constrained Markov decision processes