The main takeaway is that landline networks offer impressive reliability and quality of service. If you have people calling into conference calls from mobile phones you have similarly annoying problems.
However VoiP can work amazingly well, it just puts more responsibility in the callee's hands. But if you have reliable internet, with ethernet wiring to your machine, and a decent headset, VoiP can provide better-than-the-average-telephone quality.
All telephony advancements since the POTS line have been about increases in efficiency and additional features. In my opinion, the balance between quality and efficiency occurred at the point where most calls were routed over TDM based circuits. An ideal call would use something like PRI to the end user and SS7, over TDM circuits (T1, SONET) from end-to-end.
These days, an increasing portion of the PSTN is routed over IP networks. Even if you have a POTS line at your home/office, there's a good chance your call will be routed over an IP network if you're calling someone who is not on the same tandem. In many markets, IP reaches all the way to the individual COs that terminate copper to customers.
VoIP can work very well, but it requires a well engineered and controlled network with adequate bandwidth.
With TDM, you split your switching rate in to individual buckets. With T1, you get 24 buckets. End of story. You can't even try to push a 25th channel on T1. There's no such thing. Contrast this with VoIP.
With VoIP, the controls on the number of concurrent transmissions are all software. Channel control and routing are very well separated from the transport layer. You can attempt to stuff 100 G.711 calls on 1.54 mbps of bandwidth. There's nothing implicit about the voice stack that will stop you. The call quality will be non-existent, but you can try it.
The crux of the issue with VoIP boils down to one thing: humans are sensitive to conversational delays. We think that echo is unique to VoIP, but it's not. You actually want callers to hear themselves on the handset. If the listening portion of the handset falls completely silent, the caller will probably think that the line has dropped. On a circuit switched phone call, you hear yourself speak, but you hear yourself in almost real-time. Your brain does automatic echo cancellation.
It is extraordinarily rare for a circuit switched call to experience one-way latency higher than 100ms. That means 200ms maximum RTT. Your average individual starts to notice the delay at 300ms RTT. This means that your VoIP audio (called media) must transport over a link that has an RTT of less than 300ms, including any buffering. Nominally, you want no more than 100ms of consistent latency to support good quality voice.
So, here I go down the rabbit hole, and I know only a little bit about VoIP. I'm not an engineer, but I did stay at a Holiday Inn Express last night :)
I'll sum it up like this: you could take another step back and say that VoIP sucks because we all keep trying to do it over infrastructure that was never designed for it. VoIP at your home/office requires the cooperation of your carrier to get the IP transport for voice media to the carrier termination point in an expedient manner. It's not enough to put QoS on your LAN and expect voice to work well. The voice packets have to be expedited all the way to the other end, and that requires cooperation on the backhaul.
However VoiP can work amazingly well, it just puts more responsibility in the callee's hands. But if you have reliable internet, with ethernet wiring to your machine, and a decent headset, VoiP can provide better-than-the-average-telephone quality.