That's interesting research, but I think a more important reason that you don't have access to them (not even via the bare Anthropic api) is to prevent distillation of the model by competitors (using the output of Anthropic's model to help train a new model).
If you can't trust a company, don't use their api or cloud services. No amount of external output will ever validate anything, ever. You never know what's really happening, just because you see some text they sent you.
Do you not see that the next (or previous) logical step would be a "commercial ban" of frontier models, all "distilled" from an enormous amount of copyrighted material?
Agree. This is such a good balanced article. The only things that still make the insights difficult to apply to professional software development are: this was greenfield work and it was a solo project. But that’s hardly the author’s fault. It would however be fantastic to see more articles like this about how to go all in on AI tools for brownfield projects involving more than one person.
One thing I will add: I actually don’t think it’s wrong to start out building a vibe coded spaghetti mess for a project like this… provided you see it as a prototype you’re going to learn from and then throw away. A throwaway prototype is immensely useful because it helps you figure out what you want to build in the first place, before you step down a level and focus on closely guiding the agent to actually build it.
The author’s mistake was that he thought the horrible prototype would evolve into the real thing. Of course it could not. But I suspect that the author’s final results when he did start afresh and build with closer attention to architecture were much better because he has learned more about the requirements for what he wanted to build from that first attempt.
This wasn't even just greenfield work, it included the exact type of work where AI arguably excels: extracting working code from an extant codebase (SQLite) as a reusable library. (It also included the type of work AI is really bad at: designing APIs sensibly.)
I agree with you take the there isn’t a lot of specialist work for data scientists to do with using off-the-shelf LLMs that can’t be done by an engineer. As an AI-aware software engineer myself… this stuff wasn’t that hard to pick up. Even a lot of the work on the Evals side (creating an LLM judge etc.) isn’t that hard and doesn’t require serious ML or stats.
But aren’t there still plenty of opportunities for building ML models beyond LLMs, albeit a bit less sexy now? It’s not like you can run a business process like (say) AirBnB’s search rankings or Uber’s driver marching algorithms on an LLM; you need to build a custom model for that. Or am I missing something here? Or is that point that those opportunities are still there, but the pond has shrunk because so much new work is now LLM-related? I buy that.
> I agree with you take the there isn’t a lot of specialist work for data scientists to do with using off-the-shelf LLMs that can’t be done by an engineer.
Conversely, data scientists are doing software engineering, including webdev. It’s an interesting time. I think it’s less about the job title demarcation now, and more about output.
As an ex historian I love how this famous 350yo work of political philosophy is just sitting at #7 on HN with absolutely no context on why it was submitted.
The great debate of political philosophy coming out of the 17th century was between Hobbes (anarchy is horrible, humans aren’t nice to each other, best to give up your freedoms to a strong sovereign/state for protection) and Locke (liberty is best, people are reasonable, limit government). I will say that like most of us I probably side more with Locke but as a pessimist about human nature I find Hobbes’s argument fascinating too.
While Hobbes is dark, he is giving an interesting explanation of how political power actually work, so that even when people are not nice, they can act in a civilized way.I only read a small parts of it and some summaries, from what I understand the crux of the argument doesn't necesserily force democracy or autocracy(although he seem skeptical of democracy) rather it explains the concept of sovereignity, even in a democracy.
I once quoted Leviathan in a course assignment to explain why Gandhi's method is effective :)
Niceness is the wrong lens to use for acting in a civilised way. Game theory generally recommends cooperation; in practical real-world situations most of the games we play are ones where the best situation comes from negotiation. The issue is more the truly enormous number of actors who either have remarkably short short time preferences, an unreasonable tolerance for risk or who are just unpredictable. That is one of the central themes of the whole liberal project, of course. How to minimise the amount of force required to contain irrational actors.
An easy example is that the scariest people to run in to in a dark ally are the drugged up types; because the problem is they don't have the ability to make decisions while considering the pros- and cons- over a couple of months and their normal behaviour isn't predictive of what they are about to do.
Someone who is truly horrible and comfortable with the idea of barbarism is actually pretty easy to get along with if they're happy to work with long term goals and are predictable in their deployment of violence. Their social place is probably in the military or police force. Or dentistry if they want more consensual torment.
> An easy example is that the scariest people to run in to in a dark ally are the drugged up types; because the problem is they don't have the ability to make decisions while considering the pros- and cons- over a couple of months and their normal behaviour isn't predictive of what they are about to do.
One can argue they can’t help it. But another strategy is to mimic that to gain an upper hand. Let’s imagine someone doesn’t want folks going down their street, they could pretend to act randomly and crazy. Even seasoned barbarians would stay away from that alley, not to even mention dentists ;-)
I'd argue this is too expansive of a view. It's a debate specifically coming out of the English Civil War, and specifically focusing on the tension between Parliament and the Monarchy. If you read Clarendon it becomes extremely obvious. Hobbes (like Clarendon) took the royalist view defending the king, and Locke set for an argument for parliament.
Some of it doesn't translate super well into modern times. For example, Locke barely touches upon judiciary. The modern notion of separation of powers came (I believe) from Montesquieu.
I will say that Hobbes gives a far more comprehensive argument than Locke does. And some of Locke's details, including his anthropology of the origin of commonwealths, is demonstrably false.
Oh for sure, both thinkers were products of the English Civil War and its aftermath (and see my comment below about reading Quentin Skinner for all the context on Hobbes). I’d add that Locke (who was writing later than Hobbes) was all wrapped up in the 1688 “Glorious Revolution” too.
But some works transcend the specific details of their historical origins and authorship and contain ideas that echo down the centuries. Locke’s ideas were instrumental in founding the United States and feed into much of modern liberalism. And I can read Hobbes here today in the 21st century and still find the pessimistic core of his book powerful and relevant, even while ignoring much of the book because it’s full of the parochial concerns of 17th century England. That was really what I was getting at: not “this is the exact meaning of these works in the 17th century”, but “here is the tension of ideas these books bequeathed to us.”
Beautiful comment. Thanks for sharing. "Homo homini lupus" comes to mind, used by Locke in De Cive ("on the citizen") [1]. Cive, root Civis, is where the word civilization comes from.
This is just a very casual, even pop cultural characterization of how these two thinkers are commonly seen. I would expect people to have come across more substantive characterizations during an early high school history class, but perhaps I'm overgeneralizing my experience.
In any case, if you're looking for an approachable yet good book, I recommend reading Edward Feser's "Locke"[0]. The focus is obviously on Locke, but you can't really appreciate Locke without also getting into some Hobbes, which the books does.
GP here, I agree with you, my characterizations were both pretty casual to the point of flippancy. I could write y’all a deeper essay on this stuff, but hey, I have LLMs to herd, the 17th century wasn’t my period anyway, and there is already a massive amount of insightful writing about these two thinkers to dive into.
I would say Hobbes in particular is a complex and difficult and frankly eccentric thinker; don’t make the mistake of believing you understand him; he is weird. If you really want to grok the guy in the context of his culture and historical moment, you should just read Quentin Skinner. That’s hardcore intellectual history though; for the basics I’d just go for the clear and brief and informative Oxford Hobbes: A Very Short Introduction.
To continue this discussion and to tie it into the original link, worth looking at this YouTube Video where "Jon Pike interviews Quentin Skinner about Thomas Hobbes' masterpiece Leviathan"...
Machiavelli. Not just the Prince but his other works. He reads remarkably modern. There are many "Machiavelli Readers" that will provide a curated selection.
Is there a middle ground argument? Something along the lines of humans are horrible to one another unless there is a social state that provides reasonable protection, at which point we can afford to be nice?
Economist magazine editor once said in an interview that Republican/conservative are open regulations for businesses and closed on people. Labour/democrats are tight on business and more welcoming to the people.
Economist editorial attempts to be open on both sides.
The question is not what state humans arein, but what state other humans would be when interacting with them. In other words, are other humans nice to me? I like it when they are nice to me. In return, I will also be nice to them.
Oh totally. I actually don’t like Locke’s position much either, he’s too libertarian for my taste (I would like the state to provide healthcare &c &c). But if I had to choose I’d choose Locke over Hobbes. Hobbes is… real dark.
&c as an abbreviation for etc was very common historically. For example, look at the OP. It would not normally be used for an et that is not the latin et (and), as in et cetera. Its use for an 'and' in latin carried over to english, for some reason, and that usage has stayed with us.
It is easier to see in other fonts, but yes, I am aware of that. However as far as I am aware, it was never used to join an e and t that were not the latin et.
I like it a lot, and it makes me happy to see someone using the ligature of "e t"[0] ("&") not only as "and" but also as it's original "et" in the abbreviation "etc".
To me it reads like someone playing with words in a fun way, which is not that common in my parts of the internet
Just because I prefer Locke to Hobbes if you forced me to choose doesn't mean I'm some sort of anti-regulation libertarian. Far from it. But if you actually read Hobbes you will see that:
* He thinks everyone should be compelled to worship in the state-sanctioned religion
* Censorship of publications, teaching, etc. is necessary because ideas can be dangerous.
* Separation of powers (e.g. between executive, legislature, judiciary) is bad; he wants a single unitary sovereign with unlimited power.
* The sovereign is above the law
* Resisting a tyrannical sovereign is bad
...and that's why I'd pick Locke over Hobbes. And I think most of us would too.
But those public institutions only give power to individuals. Bad actors are gonna orbit to those positions and abuse them as much as they can.
Private security is a different thing, as the power given is just wages in exchange for a good service, so individuals seeking power wouldn't want to "just have a job" at those.
Don't confuse anarchy with barbarism. Anarchy gives you the linux kernel while barbarism is being sacrificed to your neighbours gods when it doesn't rain in a long time.
The black vans with federal agents that are snatching people from the streets and killing those that are opposing them pacifically surely scream of government good.
This is great - always worth reading anything from Sebastian. I would also highly recommend his Build an LLM From Scratch book. I feel like I didn’t really understand the transformer mechanism until I worked through that book.
On the LLM Architecture Gallery, it’s interesting to see the variations between models, but I think the 30,000ft view of this is that in the last seven years since GPT-2 there have been a lot of improvements to LLM architecture but no fundamental innovations in that area. The best open weight models today still look a lot like GPT-2 if you zoom out: it’s a bunch of attention layers and feed forward layers stacked up.
Another way of putting this is that astonishing improvements in capabilities of LLMs that we’ve seen over the last 7 years have come mostly from scaling up and, critically, from new training methods like RLVR, which is responsible for coding agents going from barely working to amazing in the last year.
That’s not to say that architectures aren’t interesting or important or that the improvements aren’t useful, but it is a little bit of a surprise, even though it shouldn’t be at this point because it’s probably just a version of the Bitter Lesson.
> On the LLM Architecture Gallery, it’s interesting to see the variations between models, but I think the 30,000ft view of this is that in the last seven years since GPT-2 there have been a lot of improvements to LLM architecture but no fundamental innovations in that area.
After years of showing up in papers and toy models, hybrid architectures like Qwen3.5 contain one such fundamental innovation - linear attention variants which replace the core of transformer, the self-attention mechanism. In Qwen3.5 in particular only one of every four layers is a self-attention layer.
MoEs are another fundamental innovation - also from a Google paper.
Thanks for the note about Qwen3.5. I should keep up with this more. If only it were more relevant to my day to day work with LLMs!
I did consider MoEs but decided (pretty arbitrarily) that I wasn’t going to count them as a truly fundamental change. But I agree, they’re pretty important. There’s also RoPE too, perhaps slightly less of a big deal but still a big difference from the earlier models. And of course lots of brilliant inference tricks like speculative decoding that have helped make big models more usable.
I'd push back slightly on the "no fundamental innovations" read though — the innovations that stuck (MoE, GQA, RoPE) are almost entirely ones that improve GPU utilization: better KV-cache efficiency, more parallelism in attention, cheaper to serve per parameter. Mamba and SSM-based hybrids are interesting but kept running into hardwar friction.
RWKV is definitely worth including - they've done an excellent job with the accompanying research, provide plenty of reference diagrams and visualization.
Thanks for saying this. It never ceases to amaze me how many people still talk about LLMs like it’s 2023, completely ignoring the RLVR revolution that gave us models like Opus that can one-shot huge chunks of works-first-time code for novel use cases. Modern LLMs aren’t just trained to guess the next token, they are trained to solve tasks.
Forget 2023 - the advances in coding ability in just last 2-months are amazing. But, they are still not AGI, and it is almost certainly going to take more than just a new training regime such as RL to get there. Demis Hassabis estimates we need another 2-3 "transformer-level" discoveries to get there.
The article mentions Mach numbers, but it leaves out what is most interesting about Mach’s place in the history of science, which is as a bridge to Einstein and General Relativity. Essentially Einstein read Mach and took a bunch of mind-bendingly profound but vague philosophical ideas like Mach’s Principle[0] and put together General Relativity out of it. And this self portrait gives that side of Mach too - the philosopher obsessed with phenomenology and how local perception relates to the large scale universe out there.
That is not what your source says. It gives one quote by him that may be misinterpreted in this context, but later clears up that Einstein was not really opposed. He merely thought that pure math was a valid way to discover new scientific insight. But even that point of view, while radical at the time, is pretty much in line with logical positivism and has turned out to be true many times since then.
I think your comment really captures some of the reasons behind the differences between people’s reactions to Claude pretty well.
I will add though, on 2 and 3, during most of the coding I do in my day job as a staff engineer, it’s pretty rare for me to encounter deeply interesting puzzles and really interesting things to learn. It’s not like I’m writing a compiler or and OS kernel or something; this is web dev and infra at a mid sized company. For 95% of coding tasks I do I’ve seen some variation already before and they are boring. It’s nice to have Claude power through them.
On system design and architecture, the problems still tend to be a bit more novel. I still learn things there. Claude is helpful, but not as helpful as it is for the code.
I do get the sense that some folks enjoy solving variations of familiar programming puzzles over and over again, and Claude kills that for them. That’s not me at all. I like novelty and I hate solving the same thing twice. Different tastes, I guess.
> In my experience, tech employment is incredibly bimodal right now. Top candidates are commanding higher salaries than ever, but an "average" developer is going to have an extremely hard time finding a position.
That sounds good for many of us (and don’t we all like to think we’re top candidates here on HN…) but is there any data to back this up? Or it just anecdata (not to dismiss anecdata, still useful info).
reply