Infinite-horizon risk-sensitive Markov decision processes (MDPs) under the discounted cost criterion are challenging to solve because the optimal policy may be non- stationary. Existing solution methods reformulate the problem as a continuous-state (risk-neutral) MDP and solve it using state-discretization or value function approximation. Such approaches typically lack explicit stopping conditions or error bounds. In this paper, we present approximate variants of value iteration, linear programming (LP), and policy iteration approaches that can directly solve risk-sensitive MDPs without the need for reformulation. We show that the approximate value functions obtained by value iteration and LP converge to the optimal value function, and a near-optimal policy can be obtained. The policies generated by policy iteration also converge in value to the optimal policy. Importantly, in addition to proving asymptotic convergence, we provide explicit stopping conditions to achieve arbitrarily close approximations to the optimal value function.