We develop a symmetry-compatible framework for first-order methods on matrix optimization problems. The central principle is that the update rule for a matrix variable should be equivariant with respect to the natural symmetry group acting on that variable. For matrix representations of linear operators, this leads to bi-orthogonal equivariance under left and right orthogonal changes of basis. We show that this coordinate-free consistency requirement characterizes spectral matrix-gradient methods, whose update maps preserve singular-vector structure and act only on singular values. This viewpoint gives a unified interpretation of stochastic spectral descent, Muon, Scion, and polar gradient methods. We then extend the same principle to matrix variables with one-sided, discrete, or affine symmetry structures, leading to right-spectral, row-normalized, centered, and hybrid row-norm/spectral updates. We establish descent and convergence guarantees under smoothness, alignment, and Polyak–Łojasiewicz conditions, including variants with momentum and inexact spectral or polar oracles. As a motivating application, we instantiate the framework for matrix-valued parameters in modern neural network architectures. The results suggest that equivariance provides a unifying principle for designing geometry-aware matrix-gradient methods.