Human behaviour is goal-directed because humans have executive function. When you turn off executive function by going to sleep, your brain will spit out dreams. Dream logic is famous for being plausible but unhinged.
I have the feeling that LLMs are effectively running on dream logic, and everything we've done to make them reason properly is insufficient to bring them up to human level.
Isn’t a modern LLM with thinking tokens fairly goal directed? But yes, we hallucinate in our sleep while LLMs will hallucinate details if the prompt isn’t grounded enough.
The thing about dream logic is that it can be a completely rational series of steps, but there's usually a giant plot hole which you only realise the second you wake up.
This definitely matches my experience of talking to AI agents and chatbots. They can be extremely knowledgeable on arcane matters yet need to have obvious (to humans) assumptions pointed out to them, since they only have book smarts and not street smarts.
A prompt for an LLM is also a goal direction and it'll produce code towards that goal. In the end, it's the human directing it, and the AI is a tool whose code needs review, same as it always has been.
LLMs are literally goal machines. It’s all they do. So it’s important that you input specific goals for them to work towards. It’s also why logically you want to break the problem into many small problems with concrete goals.
What I'm surprises me about the current development environment is the acceleration of technical debt. When I was developing my skills the nagging feeling that I didn't quite understand the technology was a big dark cloud. I felt this clopud was technical debt. This was always what I was working against.
I see current expectations that technical debt doesn't matter. The current tools embrace superficial understand. These tools to paper over the debt. There is no need for deeper understanding of the problem or solution. The tools take care of it behind the scenes.
It’s not. LLMs are just averaging their internet snapshot, after all.
But people want an AI that is objective and right. HN is where people who know the distinction hang out, but it’s not what the layperson things they are getting when they use this miraculous super hyped tool that everybody is raving about?
The etiquette, even at the bigtech place I work, has changed so quickly. The idea that it would be _embarrassing_ to send a code review with obvious or even subtle errors is disappearing. More work is being put on the reviewer. Which might even be fine if we made the further change that _credit goes to the reviewer_. But if anything we're heading in the opposite direction, lines of code pumped out as the criterion of success. It's like a car company that touts how _much_ gas its cars use, not how little.
By now, a few years after ChatGPT released, I don't think anyone is thinking AI is objective and right, all users have seen at least one instance of hallucination and simply being wrong.
Sorry I can think of so many counter examples. I also detect a lot of “well it hallucinates about subject X (that the person knows well, so can spot the hallucination)” but continue to trust it on subjects Y and Z (which the person knows less well so can’t spot the hallucinations).
> Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know.
Sure, Gell-Mann amnesia exists, but remember that its origin is actually human, in the form of newspaper writers. So, how can we trust humans the same way? In just the same way, AI cannot also be fully trusted.
that doesn’t mean the future won’t herald a way of using what a transformer is good at - interfacing with humans - to translate to and interact with something that can be a lot more sound and objective.
You're falling into the extrapolation fallacy, there is no reason to think that the future won't have the same issues as today in terms of hallucinations.
And even if they were solved, how would that even work? The world is not sound and objective.
You can fire employees who don't review code generated though, because ultimately it's their responsibility to own their code, whether they hand wrote it or an LLM did.
It seems to me that it's all a matter of company culture, as it has always been, not AI. Those that tolerate bad code will continue to tolerate it, at their peril.
The volume is different. Someone submitted a PR this week that was 3800 lines of shell script. Most of it was crap and none of it should have been in shell script. He's submitting PRs with thousands of lines of code every day. He has no idea how any of it actually works, and it completely overwhelms my ability to review.
Sure, he could have submitted a ill-considered 3800 line PR five years ago, but it would have taken him at least a week and there probably would have been opportunities to submit smaller chunks along the way or discuss the approach.
It’s harder when the person doing what you describe has the ability to have you fired. Power asymmetry + irresponsible AI use + no accountability = a recipe for a code base going right to hell in a few months.
I think we’re going to see a lot of the systems we depend on fail a lot more often. You’d often see an ATM or flight staus screen have a BSOD - I think we’re going to see that kind of thing everywhere soon.
Humans have a 'world model' beyond the syntax - for code, an idea of what the code should do and how it does it. Of course, some humans are better than others at this, they are recognized as good programmers.
Could you please cite these papers. If by AI you mean LLMs, that is not supported by what I know. If you mean a theoretical world-model-based AI, that's just a tautological statement.
> My experience is that people who weren't very good at writing software are the ones now "most excited" to "create" with a LLM.
My experience is the opposite. Those with a passion for the field and the ability to dig deeply into systems are really excited right now (literally all that power just waiting to be guided to do good...and oh does it need guidance!). Those who were just going through the motions and punching a clock are pretty unmotivated and getting ready to exit.
Sometimes I dream about being laid off from my FAANG job so I have some time to use this power in more interesting than I'm doing at work (although I already get to use it in fairly interesting ways in my job).
I wouldn’t say the pessimists fall into that category.
In my experience they are mostly the subset of engineers who enjoyed coding in and of itself and ——in some cases—— without concern for the end product.
I'm using an LLM to write queries ATM. I have it write lots of tests, do some differential testing to get the code and the tests correct, and then have it optimize the query so that it can run on our backend (and optimization isn't really optional since we are processing a lot of rows in big tables). Without the tests this wouldn't work at all, and not just tests, we need pretty good coverage since if some edge case isn't covered, it likely will wash out during optimization (if the code is ever correct about it in the first place). I've had to add edge cases manually in the past, although my workflow has gotten better about this over time.
I don't use a planner though, I have my own workflow setup to do this (since it requires context isolated agents to fix tests and fix code during differential testing). If the planner somehow added broad test coverage and a performance feedback loop (or even just very aggressive well known optimizations), it might work.
They pay use and payroll taxes. Actually, just moving more of the tax burden over to graduated income and wealth taxes would go a long way to equality. Just taking health insurance into the public system would help. There are lots of easy things we can do, millionaire taxation is kind of just a distraction/strawman.
> Juniors are still getting hired because they're still way cheaper and they're just as capable as using AI as anyone.
Seniors have much more advantage right now in using AI than Juniors. Seniors get to lean in on their experience in checking AI results. Juniors rely on the AI's experience instead, which isn't as useful.
China looks like the good guy now, but if Xi decided to “reassert control” over Taiwan, it would quickly become an international pariah and everyone would forget about Trump immediately, the country would immediately be isolated from everyone other than their closest (geographically speaking) allies. Is China ready to do that? Not today, maybe in a decade or two (when they’ve replaced the USA as the top economic/military power, there won’t be severe consequences). Xi is smart enough to wait, taking Taiwan now wins them nothing and loses them everything.
> We'd just cut off all of our goods manufacturing and leave the shelves empty? I don't think it's likely.
All bets are off if China attacks Taiwan now, I think, it would be hard but there would be a response like that. In a decade or two, probably not, but more due to China's dominance in the world by that point rather than just their ability to make things clout.
Xi isn't dumb, he isn't going to stir the pot right now, he doesn't have to, China doesn't have much to gain from it. China has nothing but patience.
They could have used inside government legal analysis that other people didnt have. You could have predicted this with higher certainty if you knew the justices well enough.
They already kind of do, but I think anyone who was into US money has already left for it, and the money China is throwing at the problem is pretty good also. You can also have a lot more influence in a Chinese company without having to adopt a weird new American corporate culture.
Anyone reading this I am begging to please thoroughly test anything that comes out of ASUS before committing. Maybe only purchase with a generous return policy and possibly insurance. They are decent panels but everything around the panel is horrendous. Random connection errors with different machines, poor UX for switching inputs, takes a millenium to boot up and connect to the screen, forget about any support, if you have built in speakers you'd be better off with a tin can connected to your computer.
I’ve frankly have had worse experience with Samsung and better experiences with LG. The model I have is pretty bare bones, which is much better than the Samsung 27 inch 5k I had that just died on me after a couple of years. The LG 28 inch 4k is going on its 6th year. I think if I buy a 6K, I’ll wait for the LG to come down in price a bit ($2k for LG vs $1300 for Asus on Amazon).
They all suck in their own ways. In my experience LG, has random hardware failures (like one audio channel just dying how? I dont know), still kinda slow booting but this has gotten better, and their designs can be hit or miss(terrible stands, aesthetics are not ergonomic enough etc.). Samsung has been better for me but suffers from variations of the above.
These brands all have glowing fans online pushing their products(the flamewars about ASUS made me even hesitate to comment) but they burn their reputations customer by customer and I guess enough have been burned that Apple is able to maintain enough sales.
You’ll see a lot of MacBooks in Beijing’s zhongguangcun where all the tech companies are, but they also have a lot of students there as well, so who knows. You need to go out to the suburbs where Lenovo has offices to stop seeing them. I know Apple is common in Western Europe having lived there for two years (but that was 20 years ago, I lived in China for 9 years after that).
It wouldn’t surprise me if the deepseek people were primarily using Mac’s. Maybe Alibaba might be using PCs? I’m not sure.
I would also expect that the Deepseek devs are using MacBook. If not they may be using Linux - Windows is possible of course but not likely imho. I have no knowledge about that area though so would be interesting to here any primary sources or anecdotes.
Deepseek is in Hangzhou, so I guess they are. GDP/capita in Zhejiang is pretty high, even more so for HZ. If you ever visit, it feels like a pretty nice place (especially if you can get a villa around xihu). I also visited ZJU once, and it was pretty Macbooky, but I don't have as much experience there as Beijing's Zhongguancun.
reply