Not quite right. A more accurate statement would be "for everything else, there is gradient-free (zeroth-order) optimization." RL is when there is a sequential decision process and what you see depends on previous actions you took.

We use different definitions of RL. In mine, any problem can be phrased as a one-step MDP (such as in arxiv.org/abs/1611.01578), and zeroth-order optimization is a special case. Can debate definitions, but I use mine because algos like PPO are doing RL regardless of MDP used.