Welp. Glad to see Li Shen's using the last fifteen years of my work to automate away my job. :-/
-- edit --
I've seen clients and some colleagues working on things like this, and I can't seem to put into words how disheartening it is. With the exception of some private analysis work, I've shared everything I've built, with everyone, for free. Papers like Elle took years to think through, implement, test, and write. That's free. High-quality checkers, Knossos, Jepsen itself, and the analyses I've put my life into: all public, all free. I put a lot of time into docs and support; essentially all unpaid. I teach classes and give conference talks to make these techniques broadly accessible because I want other engineers to be able to make high-quality systems.
At the same time, I've got a giant pile of debt from an old house that just won't quit throwing curveballs at me, and it's gonna be a few more decades before I can retire. The fact that my clients are willing to pay for this work is why I can invest so much time in R&D and give it all away. When I see someone roll in and just tell an LLM "Go use Jepsen and Elle and figure this out", it's like... well fuck. Is this even possible any more?
Thankfully, LLMs are still really bad at my job, but I don't know if, or how long, that will last. They also don't need to be good to be useful.
And if these LLM tools work, it's good, right? They find bugs, systems get safer. I want systems to be safer. On the other hand, I'm motivated to share what I do because I really want to help people. If it's just LLMs... it feels hollow. I think about this every time I've tried to work on open-source in the last few months. When I spend hours trying to figure out how to keep naming consistent, how to preserve compatibility over a decade, how to make complex code approachable through quality documentation... I have a person in mind. Someone I'll never meet, but they'll see that work, and their life will be a little easier, and maybe they'll smile. I've been talking with my therapist about it: how the work I used to do thinking about other human beings now feels purposeless. How the effort I put into making these tools and ideas accessible will inevitably cannibalize my own employment, because someone, somewhere, is going to tell an LLM "Hey, go do that", and I work in a very, very small niche. It feels like incipient depression.
Recently I've been thinking about taking Jepsen and its supporting libraries closed-source, and changing the way I write reports--instead of teaching people how to test and what to look for, just telling people the results. I don't want to do this. It's bad for everyone, but maybe it buys me a few years of runway. Enough to pay down some of the debt and figure out what I can do next with this body.
Yeah, I feel ya man. It’s a weird time to be in tech when it often times everything feels like the last years of our work are now instantly reproducible.
I’m not sure i wanna stay on the ride much longer, at least in a corp setting. I guess i don’t have much of a choice.
Thanks for Jepsen, though, it’s made a couple of my applications much better in ways I wouldn’t have managed without it; even if I have to relearn clojure every time I pick it up, and those applications resulted in real jobs and careers for a bunch of people. It’s not going to pay for your house, but it’s all I’ve got.
I've been specializing in distributed systems for nearly 35 years. I've read your work, and it's shaped my thinking. When you say you have a person in mind when you write, I am that person. Thank you for what you've done.
I don't think this replaces you. The hard part of reliability is understanding the failure modes in the context of the business. No one has unlimited time or money, we always have to make tradeoffs. Only experienced humans have both the ability to interrogate the stakeholders and a vision broad enough to understand what to pursue versus what to give up.
Tools like this make the grind part of the job easier. They do not replace the holistic view you need to be able to confidently tell someone "worry about X, do not worry about Y".
It’s shocking to me that you of all people should have financial issues. You are a legend in the community! By all means, take things closed source if it even just helps a little bit. As a profession we’ve all been hurt by over-sharing.
Why is it shocking? It is a well documented fact that people very rarely donate $ to open source project they use daily. And if they do it is typically a very low amount.
What is shocking to me is that people give away their work for free and then get upset when they have financial problems. It should be kinda obvious that one leads to the other. Except in unusual cases (Linus for example).
Yes I agree it would be nice if that was not the case. If people were always generous and always did the right thing. However that is not how people operate in general. Pretending otherwise is crazy.
There are a lot of create open source projects that have a paid infrastructure product that lets people pay to use the core tech. Perhaps you could productize your system. I would personally to pay to have something set up and useable so I don’t have to. I think the pattern you describe is clearly valuable
I honestly think the rise of LLMs will be the death of open source in the long run. Already, apparently, quality of OSS has dropped significantly since 2025 (so most models stop training on github after this).
I don't think a lot of OSS authors quite understand the extent to which models like claude/codex rely on their work. I'd bet money there are extensive curated tasks using your tooling for post-training. With 0 attribution or anything, these models are using your work wholesale to build sophisticated agents that can do your job.
Yeah it's depressing as hell. I guess it's the same thing for artists and musicians and writers.
P.S. I can symphathise with the old house issues! I bought a 1901 terraced property, it's an absolute money pit.
I discuss a lot of stuff, but that is because I am a nerd at heart, and rather play with technology, read papers, podcasts and stuff, than whatching depressing TV content.
However in the world of enterprise consulting a similar trend has been happening during the last 20 years.
First offshoring, then raise of cloud based infra, serverless, SaaS and iPaaS, and now AI based orchestrations on top of iPaaS and serverless.
Meaning for the same kind of requirements, a team playing puzzle with those kind of products can be reduced to one third of what it used to be required about a decade ago.
Then what happens to the other two thirds that now don't have anything to do, and whose salary is used instead on those licenses?
Thank you for all your work. I love your reports (even if they fly over my head when the timelines get squirrly), I love that you release them under an ethical reporting guideline, and I don't blame you if you take your test harness private. (sqlite does that and we still love them). (I hope it stays open but the world is the world and we gotta eat. If it's any consolation, I assume the therapist is also concerned about their job prospects...)
I would buy a book on distributed transactions (or databases in general) if you wrote one.
I have no idea if it would be fun or a good return on investment for you, but I would happily pay for a digital version of your distsys class aimed at practitioners. Some kind of ebook, perhaps with accompanying whiteboard/lectures.
> And if these LLM tools work, it's good, right? They find bugs, systems get safer. I want systems to be safer. On the other hand, I'm motivated to share what I do because I really want to help people. If it's just LLMs... it feels hollow.
I get that you have a financial issue, but perhaps you don't need to be conflicted about about open-sourcing your work as far as helping people goes? LLMs are tools for people. Code, research, standards, etc... are all means to an end. Maybe the agent operator doesn't read or understand your work, but the guy who built the agent skills likely did. Progress moves upward, while standing on the work of those who came before us.
LLMs have lowered the barrier to creating software and can hide a lot of source material, but your work is clearly having an impact here. If your goal is to help people make better software, that's still what's happening. The industry shift is happening regardless, so we might as well embrace the positives instead of focusing on the negatives IMHO.
Moving to a closed-source model for financial reasons is a totally separate issue IMO, and I wish you good luck and prosperity regardless of your decision.
You're kicking a man when he's feeling down. Have some sympathy for a man who loves to do things, teach people about the things he's found, and just hopes that good things will come back to him.
OK fair enough. That wasn't my intention at all but I can see how it might be interpreted that way.
My comment was more a meta comment on the trap a lot of open source developers fall into thinking that the world will fairly reward them for their work.
I've put considerable time into this, including speaking with Ofcom directly. The guidance Ofcom issued for small site operators last year was that they did intend to target "one-man bands", and that there would be no guidance on specific numbers that constituted the "significant number" of UK visitors which triggers Part 3 and 5 provider restrictions.
I would! I grew up in a low-density suburb near Portland, then lived in small-town Minnesota, Madison, SF, Chicago, and Cincinnati. I didn't own a car until my 30s, and I currently drive about once a month---camping, Costco, lumber, that sort of thing. Pretty much all my day-to-day travel is and has been by bicycle (now an e-bike), foot, train, or bus.
Situations vary, obviously! I'm no stranger to rural life, I wound up in a car-dependent suburb with terrible bus service for a bit, and my partner is in the trades. Private vehicles are sensible and essential answers to lots of problems.
But as the Netherlands illustrates, it's not all-or-nothing: reductions in car utilization and car infrastructure have real benefits. Broadly speaking I think we can and should disincentivize private car use, increase public transit frequency, and build networks of protected infrastructure for pedestrians, cyclists, and other non-car means of getting around.
I talked about this in the colophon of the piece, but perhaps you'd like an example? This is Pandoc Markdown--basically Markdown, a little bit of YAML front matter, a smattering of inline LaTeX, and some very small shell scripts. Here's the the build scripts and a little bit of fronttmatter if you'd like to do the same:
This series is seriously the best thing I have read about AI. Thank you thank you thank you for doing so much hard thinking and taking the time to write it all up. It's a monumental work and extremely valuable.
The next time someone asks me where I think AI is going, I'll just point them at this series.
I have read every post in the series and really appreciated it.
I've had a tremendous amount of respect for you since I first encountered the Jepsen analyses, but your breakdown of the likely impacts of LLMs and ML may impress me more.
You've articulated very well several concerns of mine that I haven't seen anyone else mention, and highlighted other issues I had not previously recognized.
Thank you for publishing this now, when it could still have some influence, rather than polishing and researching and refining until it was thoroughly rigorous and too late to be relevant.
> I think you can combine 'Incanters' and 'Process Engineers' into one - 'Users'
I wanted to talk about this more but couldn't quite figure out how to phrase it, so I cut a fair bit: with "incanters" I'm trying to point at a sort of ... intuitive, more informal practitioner knowledge / metis, and contrast it with a more statistically rigorous approach in "statistical/process engineers". I expect a lot of people will fuse the two, but I'm trying to stake out some tentpoles here. Users integrate a continuum of approaches, including individual intuition, folklore, formal and informal texts, scientific papers, and rigorously designed harnesses & in-house experiments. Like farming--there's deep, intuitive knowledge of local climate and landraces, but also big industrial practice, and also research plots, and those different approaches inform (and override) each other in complex ways.
I put... I'd guess around 60 hours into editing this piece, and had review from a dozen-odd friends, and I am still finding and fixing errors. I imagine that asking an LLM for a copyediting pass probably would have been helpful, but goshdarnit, I want to show that we can still write somewhat-passable prose by hand.
Thank you for this! I really wanted to go deeper on human factors, and I think there's a lot to be said about CRM and sociotechnical systems design, especially when ML gets used for decision support. Ultimately wound up truncating that section (along with more of the economic critique) because the piece was already far too long.
You're welcome! I imagine you already know this one as well but just in case.
Learning to Learn by the late Dr Richard Hamming. See especially Chapter 2.
A point Hamming makes is that when transitions from hand to machine production occurred, usually what is built ends up changing as the old techniques don't transfer 1:1 from the old world.
So for instance, we went from nuts and bolts to rivets and welding (Dr Hamming's literal example). This required builders to produce an equivalent product to the old, built with different techniques - and crucially! - under tighter control limits.
The reason things are going all over the place with AI at the moment is that it's speed, speed, speed. They had an all hands at my company recently where the top brass talked about AI. The only thing mentioned was speed - go faster, do more, etc. Not a single soul talked about quality.
But if you know your software engineering wisdom you know that you can only pick two when it comes to speed, scope, or quality. It's going to get real dumb for a while until people realize/remember quality is how you achieve speed.
There’s a paper out there, on designing IT systems from god knows when. It is incredibly dry, except for a line in it that stood out: All IT systems are political systems, because they decide how information and decisions flow.
I can only guess as to how much content you would have to explore on that axis.
-- edit --
I've seen clients and some colleagues working on things like this, and I can't seem to put into words how disheartening it is. With the exception of some private analysis work, I've shared everything I've built, with everyone, for free. Papers like Elle took years to think through, implement, test, and write. That's free. High-quality checkers, Knossos, Jepsen itself, and the analyses I've put my life into: all public, all free. I put a lot of time into docs and support; essentially all unpaid. I teach classes and give conference talks to make these techniques broadly accessible because I want other engineers to be able to make high-quality systems.
At the same time, I've got a giant pile of debt from an old house that just won't quit throwing curveballs at me, and it's gonna be a few more decades before I can retire. The fact that my clients are willing to pay for this work is why I can invest so much time in R&D and give it all away. When I see someone roll in and just tell an LLM "Go use Jepsen and Elle and figure this out", it's like... well fuck. Is this even possible any more?
Thankfully, LLMs are still really bad at my job, but I don't know if, or how long, that will last. They also don't need to be good to be useful.
And if these LLM tools work, it's good, right? They find bugs, systems get safer. I want systems to be safer. On the other hand, I'm motivated to share what I do because I really want to help people. If it's just LLMs... it feels hollow. I think about this every time I've tried to work on open-source in the last few months. When I spend hours trying to figure out how to keep naming consistent, how to preserve compatibility over a decade, how to make complex code approachable through quality documentation... I have a person in mind. Someone I'll never meet, but they'll see that work, and their life will be a little easier, and maybe they'll smile. I've been talking with my therapist about it: how the work I used to do thinking about other human beings now feels purposeless. How the effort I put into making these tools and ideas accessible will inevitably cannibalize my own employment, because someone, somewhere, is going to tell an LLM "Hey, go do that", and I work in a very, very small niche. It feels like incipient depression.
Recently I've been thinking about taking Jepsen and its supporting libraries closed-source, and changing the way I write reports--instead of teaching people how to test and what to look for, just telling people the results. I don't want to do this. It's bad for everyone, but maybe it buys me a few years of runway. Enough to pay down some of the debt and figure out what I can do next with this body.
Fuck.