Having a central load balancer is going to turn into a nightmare once you start managing a reasonable number of servers. Hardware goes bad (especially in the cloud), and having a single point of failure leaves you at it's mercy.
A load balancer should never be a single point of failure. You should always have multiples.
Also, if the response to this is then 'but it's still a central point of failure', they haven't really removed that in this solution. If the zookeeper cluster dies you lose everything.
Generally if a clustered load-balancer dies and another takes over there's a half second or a couple seconds of transition, but you're back up and running with a very simple architecture. If all of your load-balancers die, something much bigger to worry about is going on.
Really all this does is place the failure mode into a higher-level service with a much greater potential for failure. Zookeeper even makes you specify the size of your cluster and in my experience it's difficult to live update this. I've read they're working on that, but still.
Clustered load-balancers (using Pacemaker/Corosync, keepalived or similar) are very well understood these days. Pacemaker/Corosync can even run within EC2 now, since a couple years ago they added unicast support thus obviating the multicast issues present within EC2.
Additionally, if we want to talk about load then a well configured haproxy/Nginx load-balancer can handle hundreds of thousands of connections a second. If your installation needs more than this then I'm certain you could get a layer to distribute the load-balancing between a set. Obviously another problem to introduce, but still not one you'll reach until you probably have even more traffic than airbnb gets.
For accuracy it's worth pointing out that if Zookeeper dies only registration/deregistration goes down. The local HAProxy processes will continue to run, and applications will continue to be able to communicate with services. Zookeeper isn't a single point of failure for communication; it's just a registration service.
Indeed it does! I use it for less complicated fail-over situations myself, but if you start to need more complicated topologies (or something which has good interaction with IPVS/LVS) then Corosync!
So... I might be a bit biased (co-founder at Fondu http://fondu.com), but this really is a major problem.
Most restaurant owners we've talked to are afraid of Yelp and the power that it wields. From an investor standpoint it does a great job monetizing it's community. The problem is that Yelp essentially weights the scales, depending upon who is paying them ad money.
Whereas Google did a great job by separating church and state (search and ads), Yelp happily blends the two together. The end result is a little bit of fact and a little bit of fiction.
YMMV, but we think discovering through trusted friends rather than group averages is how the future looks.
If you don't like the import *, don't use it. Python supports it, so why shouldn't this library? (not that it's a great idea)
If you'd like to handle missing system executables, catch the exception. You should be writing in that style anyways.
Honestly, this cleans up a ton of system scripting code, making it way more readable/maintainable.
Maybe the code could be cleaned up, but this is the direction that Python should be heading. Abstract away the complications when possible, keep low-level stuff around for when it's absolutely needed.
The search for a co-founder has been on as a serious milestone for the last month. Solo is something I chose when I started because I didn't have a co, and didn't want to let that be a blocking issue. I'm fully aware what a great co could do for my biz.
If anybody knows of a rockstar Python dev in NYC, let me know.
Thanks to everyone giving me the "Just 2 months? That's nothing" comments.
I don't mean that facetiously. It's actually pretty inspiring that you guys made it through that honeymoon phase and into something real. Hopefully this venture will be around a year from now, and I'll be able to dish out the same kind of inspiration.
I'd agree entirely on that sentiment. Just in the last month alone I can feel a lot changing in my mind regarding the whole thing.
I have a feeling that as it becomes more all-consuming that roller-coaster is going to come to bear in a stronger way. Once your ego becomes tied to the success of your venture, small fluctuations can have powerful mental impacts.
