I see what you’re saying but I believe there’s an issue at the silicon level where the gates literally use more power to do the same work at higher frequencies.
It's a well-known feature of CMOS gates that they consume almost all of their power at the moments of switching. AFAIK only recently has static leakage become substantial, necessitating power gating.
