Square is a great option for selling crafts at a market once a month. I tried to use it for a proper multi-channel retail store and it immediately fell apart:
- the e-commerce integration (Weebly?) is very limited and the resulting sites are dog slow
- the POS itself and the backend don't work when you have hundreds of SKUs and many variants
- there's very little customization or support
My business wasn't huge but we were doing ~300k revenue annually online and in-store. We started on Square, tried Lightspeed (also garbage) and finally ended up on Shopify (best of a bad lot).
Despite making noise about "supporting small business" Shopify makes most of their money from e-commerce for giant brands. They've tried to juice returns from small customers with merchant cash advances but my sense is they make more doing professional services for big e-commerce brands
It seems like this is an approach that trades off scale and performance for operational simplicity. They say they only have 1GB of records and they can use a single committer to handle all requests. Failover happens by missing a compare-and-set so there's probably a second of latency to become leader?
This is not to say it's a bad system, but it's very precisely tailored for their needs. If you look at the original Kafka implementation, for instance, it was also very simple and targeted. As you bolt on more use cases and features you lose the simplicity to try and become all things to all people.
> It seems like this is an approach that trades off scale and performance for operational simplicity.
Yes, this is exactly it. Given that turbopuffer itself is built on the idea of object storage + stateless cache, we're all very comfortable dealing with it operationally. This design is enough for our needs and is much easier to be oncall for than adding an entirely new dependency would have been.
> Failover happens by missing a compare-and-set so there's probably a second of latency to become leader?
Conceptually that makes sense. How complicated is it to implement this failover logic in a safe way? If there are two processes, competing for CAS wins, is there not a risk that both will think they're non-leaders and terminate themselves?
7. On failure due to CAS, fail active requests and terminate
The client should have a retry mechanism against the broker (which may include looking up the address again).
From the brokers PoV, it will never fail a CAS until another broker wins a CAS, at which point that other broker is the leader. If it does fail a CAS the client will retry with another broker, which will probably be the leader. The key insight is that the broker reads the file once, it doesn't compete to become leader by re-reading the data and this is OK because of the nature of the data. You could also say that brokers are set up to consider themselves "maybe the leader" until they find out they are not, and losing leadership doesn't lose data.
The mechanism to start brokers is only vaguely discussed, but if a host-unreachable also triggers a new broker there is a neat from-zero scaling property.
This is the hardest part because you can easily end up in a situation like you're describing, or having large portions of clients talking to a server just to have their writes rejected.
Further, this system (as described) scales best when writes are colocated (since it maximizes throughput via buffering). So even just by having a second writer you cut your throughput in ~half if one of them is basically dead.
If you split things up you can just do "merge manifests on conflict" since different writers would be writing to different files and the manifest is just an index, or you can do multiple manifests + compaction. DeltaLake does the latter, so you end up with a bunch of `0000.json`, `0001.json` and to reconstruct the full index you read all of them. You still have conflicts on allocating the json file but that's it, no wasted flushing. And then you can merge as you please. This all gets very complex at this stage I think, compaction becomes the "one writer only" bit, but you can serve reads and writes without compaction.
The average person does not know how to fly a plane or what a plane autopilot does. It's a ridiculous superficial comparison. Planes have professional pilots who understand the capabilities and limits of aviation autopilot technology.
Tesla has had it both ways for ages - their stock price was based on "self-driving cars" and their liability was based on "asterisk asterisk the car cannot drive itself".
According to your analogy. Certified pilot = Certified driving license holder. Its not like Tesla is advertising non driving license or in eligible person can drive using Autopilot. I wonder how can you even justify your statement
Autopilot is part of a private pilots license and systems are approved by the FAA. Tesla autopilot isn't part of a driving license, nor did it undergo review by the NHTSA prior to launch because Elon considered it "legal by default".
No. You don't need to know the autopilot to get your PPL. You do however need to know how to follow the POH (pilot operating handbook, which may include manufacturer guidelines for the autopilot) and perform basic instrument flying in an emergency. I don't recall any significant expectations of autopilot usage at the PPL level though.
> if you do it in an aircraft equipped with autopilot
There's also a (stupid, imo) tendency for APs to conveniently become inop right before a checkride. It's not accurate to say that all pilots, or even all pilots that have taken an IR ride, are "pilots who understand the capabilities and limits of aviation autopilot technology."
For the PPL specifically, the focus is on basic airmanship in VFR conditions, and that means eyeballing the six pack (or digital equivalent) and looking out of the window. The instrument flying expectations is primarily for emergencies and preparation for future instrument rating.
I'm not sure it's been codified, but I was told I would need to understand how to use the VOR and autopilot if the plane I was in had one.
In the fleet at the school I was learning in (Cessna 162) only one plane had an autopilot, which meant nobody practiced with it, so they never scheduled this plane for a check ride.
Autopilots in airplanes are kind of dumb (keep heading, speed, and altitude, they won’t do much anything else), which is why Tesla doesn’t use the name as branding for its full self driving software. People at least know that much.
But then again even on HN people like parent think that autopilot is the same as full self driving, when it is and always has been just smarter cruise control. The payout was for autopilot (a feature that most new cars have these days under various names), not full self driving.
> Autopilots in airplanes are kind of dumb (keep heading, speed, and altitude, they won’t do much anything else)
That is absolutely false.
A 20 year old avionics suite in General Aviation (GTN 450) does much more than maintain altitude, speed & heading - you input a flight plan including an approach, it will fly the flight plan, capture the approach signals (VOR/localiser/whatever - which is far more complex than “keeping course”) all the way down to approach minimums.
Autolanders are separate systems from autopilots, and there are definitely planes in production today with autopilots (pretty universal) and no autolanders (like almost all Cessnas, you need the garmin auto land system, and it’s only for emergencies). Autopilots have been a thing since the 1920s, when they were just a rope tied to a stick, they definitely didn’t do auto landing back then.
If you are trying to claim that all autopilots come with auto landers, thats absolutely not true. Even most, and again, they are always separate systems even if the auto lander can access the same servos used by the autopilot. Additionally, autolanders, unlike the autopilot, require the runways to support it (well, the ones used on commercial airplanes where it’s used sometimes in low visibility situations).
I really think only a few people on HN don’t get that autopilots are actually very much simple systems that have been around forever and do one thing well (keep the plane going in one direction at a specific altitude and speed).
If the average person does not know what an autopilot does, why would they expect Tesla's 'autopilot' to take such good care of them? I am reminded of a case many years ago when a man turned on the cruise control in his RV and went to the back to make himself lunch, after which the RV went off some sort of hill or cliff.
Rudimentary 'autopilots' on aircraft have existed for about a century now, and the earlier versions (before transistorization) only controlled heading and attitude (if conditions and other settings allowed it), with little indication of failure.
I see you regret Datadog but there's no alternative - did you end up homebrewing metrics, or are you just living with their insane pricing model? In my experience they suck but not enough to leave.
Currently going through leaving DD at work. Many potential options, many companies trying to break in. The one that calls to me spiritually is: throw it all in Clickhouse (hosted Clickhouse is shockingly cheap) with a hosted HyperDX (logs and metrics UI) instance in front of it. HyperDX has its issues, but it's shocking how cheap it is to toss a couple hundred TB of logs/metrics into Clickhouse per month (compared to the kings ransom DD charges). And you can just query the raw rows, which really comes in handy for understanding some in-the-weeds metrics questions.
VictoriaMetrics stack. Better, cheaper, faster queries, more k8s native, etc.
Easy to run with budget saved from not being on Datadog + attracts smart and observability minded engineers to your team.
"No alternative" isn't quite right anymore, though I understand the feeling. The real problem with Datadog isn't the pricing - it's that their per-host model incentivizes you to care about infrastructure topology rather than user-facing behavior. You end up with 10,000 dashboards and still can't answer "is checkout broken right now?"
The open source stack has gotten genuinely viable: Prometheus/VictoriaMetrics for metrics, Grafana for viz, and OpenTelemetry as the collection layer means you're not locked into anyone's agent. The gap used to be in correlation - connecting a metric spike to a trace to a log line - but that's narrowed significantly.
The actual hard part of leaving DD isn't technical, it's organizational. DD becomes load-bearing for on-call runbooks, alert routing, and team muscle memory. Migration is less "swap the backend" and more "retrain your incident response."
If you're evaluating: the question I'd ask isn't "which vendor has the best dashboards" but "can I get from alert to root cause in under 5 minutes with this tool?" That's the metric that actually correlates with MTTR, and it's where most monitoring setups (including expensive ones) fail.
I think the fundamental tension is that AI produces a high volume of low quality output, and the human in the loop hates reviewing all the slop. So people want to just let the AI interface directly, but when you let slop into the real world there are consequences.
Even if you don't believe in the capabilities of Israeli intelligence, it's well documented that Israel supported Hamas as a hard-line alternative to the PLO to avoid a two-state solution. The Israeli right has for decades intensified the conflict to justify total war against Palestinians. Allowing a domestic attack to gin up support for aggression is in line with their behaviour for the past 3 decades.
I'm old enough to remember when Arafat was well-respected in the west and a two-state solution was the mainstream view amongst Americans. Once Netanyahu came to power in 1996 (30 years ago!) he worked to delegitimize the PLO and pursue an aggressive genocide against the Palestinians.
For my entire career in tech (~20 years) I have been technically good but bad at identifying business trends. I left Shopify right before their stock 4xed during COVID because their technology was stagnating and the culture was toxic. The market didn't care about any of that, I could have hung around and been a millionaire. I've been at 3 early stage startups and the difference between winners and losers was nothing to do with quality or security.
The tech industry hasn't ever been about "building" in a pure sense, and I think we look back at previous generations with an excess of nostalgia. Many superior technologies have lost out because they were less profitable or marketed poorly.
I think you’re being unduly harsh on yourself. At least by the Shopify/COVID example. COVID was a black swan event, which may very well have completely changed the fortunes of companies like Shopify when online commerce surged and became vital to the economy. Shortcomings, mismanagement and bad culture can be completely papered over by growth and revenue.
Right place, right time. It’s too bad you missed out on some good fortune, but it’s a helpful reminder of how much of our paths are governed by luck. Thanks for sharing, and wishing you luck in the future.
I've been in this situation before. Anthropic has a stupid business model but the market can stay irrational longer than you can stay solvent. If you get in there you will be aligned with people who structurally do not lose.
Matt Levine has put this forward in his newsletter - if you're moderately influential you can go on TV and tell people that "X industry will be dead in 10 years" because of AI and then profit from the inevitable stock dip.
Because we live in the worst possible timeline the end result for AI companies does seem to be "too big to fail", where these massive investments will get foisted on working class people via a bailout or an IPO and index inclusion.
I don't see why any AI company would ever be "too big to fail". I can't see why any government would be motivated to take the political hit of bailing them out.
- the e-commerce integration (Weebly?) is very limited and the resulting sites are dog slow
- the POS itself and the backend don't work when you have hundreds of SKUs and many variants
- there's very little customization or support
My business wasn't huge but we were doing ~300k revenue annually online and in-store. We started on Square, tried Lightspeed (also garbage) and finally ended up on Shopify (best of a bad lot).
Despite making noise about "supporting small business" Shopify makes most of their money from e-commerce for giant brands. They've tried to juice returns from small customers with merchant cash advances but my sense is they make more doing professional services for big e-commerce brands
reply