Derivative-free algorithms seek the minimum value of a given objective function without using any derivative information. The performance of these methods often worsen as the dimension increases, a phenomenon predicted by their worst-case complexity guarantees. Nevertheless, recent algorithmic proposals have shown that incorporating randomization into otherwise deterministic frameworks could alleviate this effect for direct-search methods. In particular, the best guarantees and practical performance were obtained for direct-search schemes using a random vector uniformly distributed on the sphere and its negative at every iteration. This approach effectively draws directions from a random one-dimensional subspace, yet the properties of such subspaces have not been exploited in direct search, unlike for other derivative-free schemes. Moreover, existing theory is by design limited to bounded directions, and thus does not fully account for the numerous possibilities for generating random directions (such as drawing from a Gaussian distribution). In this paper, we study a generic direct-search algorithm in which the polling directions are defined using random subspaces. Complexity guarantees for such an approach are derived thanks to probabilistic properties related to both the subspaces and the directions used within these subspaces. Our analysis crucially extends previous deterministic and probabilistic arguments by relaxing the need for directions to be deterministically bounded in norm. As a result, our approach encompasses a wide range of new optimal polling strategies that can be characterized using our subspace and direction properties. By leveraging results on random subspace embeddings and sketching matrices, we show that better complexity bounds are obtained for randomized instances of our framework. A numerical investigation confirms the benefit of randomization, particularly when done in subspaces, when solving problems of moderately large dimension.
Citation
L. Roberts and C. W. Royer, Direct search based on probabilistic descent in reduced spaces. Technical report arXiv:2204.01275v2, January 2023.
Article
View Direct search based on probabilistic descent in reduced spaces