Try out rubberband. I find that with rubberband and mpv I can get close to 3x and still understand many (but not all) speakers. It depends on their particular accent/annunciation/cadence though. Some accents, like RP, are very clear to me at 1x but degrade more rapidly (for me) than others at high speeds.
Good question. By default mpv will automatically insert the 'scaletempo' filter to adjust pitch when speeding up or slowing down audio, and it does a reasonable job at modest speed changes. And it's fast; very little CPU load. However at more dramatic speed changes it starts to sound choppy with parts of words getting cut out which limits how fast it can go while maintaining comprehensibility. Scaletempo sounds choppy at high speeds because the way it works is by playing small snippets of the audio stream at 1x, skipping ahead past some audio to keep pace with the sped up video. There is also some blending of those snippets, but in principle it's a very simple approach that generally works well enough.
How precisely rubberband works is something I've not looked into deeply, but the end result is increased CPU load and less choppy audio at high speeds. If you throw together a little script to switch back and forth between rubberband and scaletempo in realtime, I think the difference is pretty dramatic.
https://breakfastquay.com/rubberband/