Tailored mixed-integer optimal control policies for real-world applications usually have to avoid very short successive changes of the active integer control. Minimum dwell time constraints express this requirement and can be included into the combinatorial integral approximation decomposition, which solves mixed-integer optimal control problems by solving one continuous nonlinear program and one mixed-integer linear program. Within this work, we analyze the integrality gap of mixed-integer optimal control problems under minimum dwell time constraints by providing tight upper bounds on the mixed-integer linear program subproblem. We suggest different rounding schemes for constructing minimum dwell time feasible control solutions, e.g., we propose a modification of sum-up rounding. A numerical study supplements the theoretical results and compares objective values of integer solutions with a priori lower bounds.