We propose a novel approach using shape derivatives to solve inverse optimization problems governed by Maxwell’s equations, focusing on identifying hidden geometric objects in a predefined domain. The target functional is of tracking type and determines the distance between the solution of a 3D time-dependent Maxwell problem and given measured data in an $L_2$-norm. Minimization is conducted using gradient information based on shape derivatives. We describe the underlying formulas, the derivation of appropriate upwind fluxes and the derivation of shape gradients for general tracking type objectives and conservation laws. Subsequently, an explicit boundary gradient formulation is given for the problem at hand, leading to structure exploiting data efficient transient adjoints. Numerical results demonstrate the practicability of the proposed approach. Typical applications are non-invasive analysis of diverse materials and remote sensing.