There's a big difference between not using JavaScript at all and doing it correctly with progressive enhancement. Any site which doesn't depend on a ton of JavaScript to render and uses lazy-loading where appropriate will see the same benefits and will be significantly more robust as well.