Yeah, that's a major issue with more loaded machines, especially with older/worse virtualization technologies. It seems tolerable on Twilio, and I've run asterisk and freeswitch on virtualized machines (with HVM) which work ok.

My reason for not wanting VoIP gateway on AWS is that I want to use it for AWS-hosted-app error reporting, so having dependencies on AWS is a problem.

Things word good when the throttle is not pushed on the VM. Also try our conference calls, you will see the difference :)