This was really interesting. I'm thinking about moving to Kubernetes and have wondered how to gracefully deal with websocket connections.
I'm curious though, if the rollout was over a couple of hours for example, why would reconnections be that big of a problem? We host about 10,000+ websocket on a $20 VPS, and the Go server hosting it crashes from time to time. A surge of 10,000 reconnections instantly afterwards has never lasted for more than a minute or so, so why is it so bad? Moments of peak load aren't that big of a deal, or?
In our case it's not the websockets that are the problem, it's the XMPP connection that each websocket connection creates. Logging in thousands of users takes several minutes. While a user reconnects, any conversations that the users are having with their website visitors are disrupted.
(work with OP on the same team) Basically there are a lot of other things that happen when a websocket connection is established and we don't necessarily have the capacity to handle that volume in a complete reconnect scenario, especially if the system is already near the daily load peak. We have hopes that autoscaling some things in the future will make it possible to handle peaks like this more gracefully.
I'm curious though, if the rollout was over a couple of hours for example, why would reconnections be that big of a problem? We host about 10,000+ websocket on a $20 VPS, and the Go server hosting it crashes from time to time. A surge of 10,000 reconnections instantly afterwards has never lasted for more than a minute or so, so why is it so bad? Moments of peak load aren't that big of a deal, or?