Convergence of Controlled Models for Continuous-Time Markov Decision Processes with Constrained Average Criteria

Abstract

This paper attempts to study the convergence of optimal values and optimal policies of continuous-time Markov decision processes (CTMDP for short) under the constrained average criteria. For a given original model $\mathcal{M}$_$∞$ of CTMDP with denumerable states and a sequence {$\mathcal{M}$_$n$} of CTMDP with finite states, we give a new convergence condition to ensure that the optimal values and optimal policies of {$\mathcal{M}$_$n$} converge to the optimal value and optimal policy of $\mathcal{M}$_$∞$ as the state space $S$_$n$ of $\mathcal{M}$_$n$ converges to the state space $S$_$∞$ of $\mathcal{M}$_$∞$, respectively. The transition rates and cost/reward functions of $\mathcal{M}$_$∞$ are allowed to be unbounded. Our approach can be viewed as a combination method of linear program and Lagrange multipliers.