A method is presented that automatically improves upon previous treatment plans by optimization under reference dose constraints. In such an optimization, a previous plan is taken as reference and a new optimization is performed towards some goal, such as minimization of the doses to healthy structures, under the constraint that no structure can become worse off than in the reference plan. Two types of constraints that enforce this are discussed: either each voxel or each dose-volume histogram of the improved plan must be at least as good as in the reference plan. These constraints ensure that the quality of the dose distribution cannot deteriorate, something that constraints on conventional physical penalty functions do not. To avoid discontinuous gradients, which may restrain gradient-based optimization algorithms, the positive part operators that constitute the optimization functions are regularized. The method was applied to a previously optimized plan for a C-shaped phantom and the effects of the choice of regularization parameter were studied. The method resulted in reduced integral dose and reduced doses to the organ at risk while maintaining target homogeneity. It could be used to improve upon treatment plans directly or as a means of quality control of plans.