Large language models (LLMs) trained with reinforcement learning from human feed-
back (RLHF) struggle with complex instructions that bundle multiple, potentially con-
icting requirements. We introduce constraint decomposition, a framework that separates
multi-objective instructions into orthogonal componentssemantic correctness, structural
organization, format specications, and meta-level requirementsand optimizes each in-
dependently before hierarchical combination. Our approach addresses the fundamental
limitation of monolithic reward models: their inability to distinguish which specic con-
straint failed when multiple requirements conict. We train decomposed reward models
with aspect-level human preferences and demonstrate that explicit constraint separation,
combined with conict-aware weight adaptation, enables more eective multi-objective op-
timization. On the IFEval benchmark, our method achieves 73.8% prompt-level accuracy
(±1.6%), a 32.6 percentage point improvement over standard RLHF (41.2%). Ablation
studies show that constraint decomposition contributes 54% of the total improvement, with
hierarchical combination adding 17%, weight prediction 15%, and conict detection 14%.
Our method generalizes to GSM8K (+15.6 points), HumanEval (+11.1 points), and MT-
Bench (+1.4 points). Code and data are available at https://github.com/epaunova/
constraint-decomposition-llm.