We study the convergence rate of moment-sum-of-squares hierarchies of semidefinite programs for optimal control problems with polynomial data. It is known that these hierarchies generate polynomial under-approximations to the value function of the optimal control problem and that these under-approximations converge in the $L^1$ norm to the value function as their degree $d$ tends to infinity. We show that the rate of this convergence is $O(1/\log\log\,d)$. We treat in detail the continuous-time infinite-horizon discounted problem and describe in brief how the same rate can be obtained for the finite-horizon continuous-time problem and for the discrete-time counterparts of both problems.