I’ve asked Astra to build me an app for a prototype I created quickly using Sonnet.
It’s been 2 days and it made no real progress on the actual app. It created docs, scripts, workflows, and it’s doing a bunch of reviewing on every PR.
I told it that I just need an MVP.
I’m pretty sure an average senior engineer would have finished that task much quicker, and guaranteed with more readable, higher-quality code. Meanwhile, I think I’ve easily crossed 100k tokens so far on nothing.
Funny world we’re living in that this is “SOTA” and “AGI”.
I’m genuinely curious what these OAI and A/ engineers are working on that they praise these models so much. I did not see any improvement since Opus 4.5.
Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.
I don't know how to make apps or evaluate code, but with astra I having been making an iOS app on my own for the first time and it's going great. my app is not terribly complex but requires using bluetooth and other intricacies which I thought would be tough. but it's going really well. I'm not asking it to one-shot it though, I'm going feature by feature, testing and building up.
yes, at first it would run simulator tests on all font sizes but it stopped after I asked it not to do that until UI review
maybe sol would have done the same thing, idk. but I find the whole process to be really nice with astra. I use it on high unless it says something is impossible then i go max and ask it to find alternatives (happened once)
I've been making a macos app with opus 4.8-5 and at first it was great, everything materialized in a week, but when I started tuning stuff and fixing performance problems I have spent a very frustrating month refactoring code where I had to constantly catch llm red-handed and explain and sometimes push obvious ways how to make things work properly (a general knowledge from a completely different stack). In the process CLAUDE.md and memory grew exponentially explaining what it should and what it should never do.
were you letting it run for hours like the OP, or doing short tasks and reviewing/testing each one? every once in a while I also ask it to consolidate/summarize docs and stuff like that. we'll see what happens in a week though
I have started with generating very detailed "feature" spec and going over it many times until it looked good to me. Then I made it write an "architecture doc" and plan how to implement all which was about 15 parts. Then I was making it implement one part and then tested it and made it fix 20 things and then again, and consolidate docs too, so after each part it looked and worked well enough.
>Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.
If a person, or team of people, can build a demo quickly then it's good odds that they can build the real version (though, famously, not a guarantee). However, it turns out that a machine that can spit out 100 demos of whatever can't actually build the real thing.
Similarly, a chess engine rated to 1000 Elo doesn't play like a 1000 rated human being. The mistakes that each make to reach the equivalent level are different in size, frequency and kind. The thing that makes a human reach a good demo is very close to the skillset to reach the finished article. This isn't so for LLMs but we have yet to update our priors.
This resonated with me. "Developing ideas and artifacts using AI breaks our normal intuitions along many meaningful axes and we've yet to update" is a really clean idea.
For me it also produces totally overengineered tests that are tightly coupled to the implementation. For example testing existence of css classes (in a template based go prooject ...) instead of behaviour.
Not GP, but IME it's not fixable by model selection, but being zealous about guiding output and vision, and pushing back on all the bad habits LLM in general has (eg verbose output as a band-aid for emergent intelligence). As soon as something is introduced into your codebase, it will continue being picked up into context until you remove it and any reference to it from any potential context entrypoint. If you don't any model will keep venturing down wrong/bad paths.
The CLI tools for automation are still a pain to use and missing functionality, which also means LLMs use it less; Claude gives up when it sees Chrome isn’t installed unless I specifically tell it to try and use Firefox.
I created an automation tool recently for taking screenshots (for SOC2 reporting, hopefully tptacek is proud), and getting it working with FF on the CLI was very painful (but necessary since I needed my cookies!)
I will never buy another laptop with less than a 500 nits display, no matter its specs/price.
I can’t believe virtually every other manufacturer is still making laptops with 250-300 nits.
Spending a full day looking at a dim, washed-out screen is just the worst experience ever. The fact that out of all manufacturers, Apple, which is considered the “expensive” and “premium” brand, made the Neo with 500 nits at that price point is just amazing to me. The rest have no excuse for putting these displays in laptops anymore.
Using a laptop outside is fantastic. I love to sit on my balcony and work there instead of inside. My work laptop is an HP with zero ability to do this.
I think the idea of leveraging AI to find useless PRs is right.
I didn’t like the second part though where it starts banning people for “bad behavior”.
I get that AI resistance will be a thing in the next few years, but realistically engineers who plan on being around for the next 20-30 years just need to embrace it instead of being sour about it.
Because of machines, things have arguably improved for humanity as a whole. People who seek more elevated/finer products can still turn to handcrafted products.
Mass software will be produced by AI, and it will scale just fine.
> Because of machines, things have arguably improved for humanity as a whole.
That's a very narrow view of the situation. I agree that this side of the coin exists, but there's also the other side where drinkable tap water is becoming a luxury in much of the global North, giant tornadoes and fires are sweeping the Earth, and temperatures are growing out of control.
So, to make lives more comfortable with machines for some millions of privileged people, they triggered the 6th mass extinction. That's the other side of the coin.
We're already at the end of the 6th mass extinction FWIW. Over the last 200 years most species have gone extinct. There isn't that much left to make extinct. We'll manage it though.
Can you say more about this? The first link below [1] says there are an estimated eight million species today, and the second link [2] says the number of known extinct species since 1500 is 905.
> I didn’t like the second part though where it starts banning people for “bad behavior”.
I think you failed to notice the ban would be a reaction to recurrent bad behavior, such as ignoring maintainer requests.
What's your solution to handle accounts which spam projects with nonsense PRs?
> get that AI resistance will be a thing in the next few years (...)
This is a very lazy opinion to express. If you pay attention to the post you're replying to, you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers.
You'd have the same problem if some guy decided to spam the project with a constant barrage of small low-effort nonfunctional PRs.
You should really read the comments you are replying to. The whole point of the proposal is to provide actionable feedback to the people operating these drive-by PR bots, and kick out repeat offenders to prevent them from generating noise.
> What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry.
I think you're trying too hard to come up with imaginary scenarios. Also, keep in mind that the objective is to steer contributions to be aligned with the project goals and help maintainers not waste time with low-quality noise. The goal is not to impose a fundamentalist militant stance against AI.
I did read it. It is not a plan to kick repeat offendrs. It is not a plan to give actionable feedback.
This plan will kick no bots, because bots dont mind discussion with another bot. However, falsely flagged people will realize they are talking with bot prompted to be obtuse, get annoyed anf then will be banned
> the objective is to steer contributions to be aligned with the project goals
I just think when you close PR's outright, you tend to get angry responses. And as a maintainer, you don't want to deal with angry responses. People who post bot PR's but can't accept having their PR's closed by a bot, shouldn't be visible to the maintainers.
Which part of the second part do you dislike more?
That AI will be given the power to ban people?
Or that AI will be the ones getting banned?
If it's the former, expect an invite from the AI resistance. :)
If it's the latter, well, that can be fixed by spending more tokens and writing better PRs so that the AI moderator doesn't think they're "low-effort, AI-like". I've seen my share of low-effort PRs from humans, too. Low-effort is low-effort, whether it's from humans or AI.
Agreed. People seem to hold views that getting visas to the US is a human right.
It’s no fun, but these are adults (and often very privileged adults) that decided to move to a new country for more upside at the cost of no residency.
Conflating this with “inhumane” is just a basic misunderstanding of immigration and the contract these people signed up for.
Very uncomfortable, yet a risk these people took, and hopefully a conscious one.
Democratic government must also allow the citizen body to voice their displeasure with the activity of the government, and to give their reasons for that displeasure
I dont think the moral/legal conflation is even the main tension here.
I think this distinction would help as well.
Inhumane != human rights violation.
Both of these things can be true at the same time:
1. It can be inhumane to halt someone's visa application
2. Visa applicants have no human right to have their application processed
One side hammers the first point because they disagree with the government. The other side hammers the second point because they agree with the government.
Everyone is right, except for the implication that their point disproves the other side's point.
Sure. Unless you prefer autocratic rule by a moral paragon, democracy involves doing what people want, not what is moral. Often what is nominally moral is not good for the country or its people.
> People seem to hold views that getting visas to the US is a human right.
Come on, there's nothing to be gained by misrepresenting people's views.
The argument is a simple one: if you are currently within the immigration process you should be able to move through that process in a timely manner, with clear communication of expectations. Freezing appointments is a political hack: they're not changing the law, they're not altering what anyone is entitled to. They're just leaving people in limbo.
You're saying "these people took a risk", and you're right. But that doesn't mean we can't criticize the people ensuring their bet is a losing one in the least moral way possible.
> “Just ‘putting a pause’ on people with existing appointments with no answer for when they might resume is.... Inhumane”
And now you're saying:
> “if you are currently within the immigration process you should be able to move through that process in a timely manner, with clear communication of expectations.”
Sure it sucks. But do you understand there is no right to have this visa application interview nor a yes/no decision in a timely manner? Applying for a visa is a privilege. You aren't being denied some human right if the process is interrupted.
Actually, there is. I am a bit tired of correcting these types of misunderstandings that people have. Unreasonable delay, unlawful witholding, arbitrary and capricious are all well defined terms in the law and courts have routinely held and even forced the government to take action (i.e., do their job) when they just pause or suspend things. For a few recent decisions read this:
We’re talking about two different things and I don’t understand why you’re conflating them.
“It isn’t a right” and “it’s immoral” are not mutually exclusive statements. No, there is no human right to a visa application response. That doesn’t mean that leaving people in indefinite limbo is morally right.
By owned by other people you meant the Ottomans, right?
Cause they seemed to be just fine selling that land.
Also “ignoring the partition”? Literally the day the partition was adopted, the Arab league attacked Israel. 6 nations to 1, yet Israelis outsmarted them all (which is why you keep hating us).
Afaik it was forbidden to sell lands in Palestine to Jewish people, though I don't think it was that effective in stopping lands being bought out by Jewish people due to weak central authority and worsening economy.
"Anyone who wants to thwart the establishment of a Palestinian state has to support bolstering Hamas and transferring money to Hamas" —Benjamin Netanyahu
Wasn't Hamas technically funded by Natanyahu for the explicit goal of dividing Palestinian leadership and to ensure that extremists had both money and power so that he always had justification to wipe out Palestine ?
Look at how evil this group is -> genocide of their entire state is good! (Don't look at the fact that I demanded that this group is explicitly funded - that is not important)
“Heh, we’re fucking awful, but check out what THOSE GUYS (who aren’t a first world country with international relationships and funding, i.e. where upholding a similar standard is laughable) are doing”
It’s been 2 days and it made no real progress on the actual app. It created docs, scripts, workflows, and it’s doing a bunch of reviewing on every PR.
I told it that I just need an MVP.
I’m pretty sure an average senior engineer would have finished that task much quicker, and guaranteed with more readable, higher-quality code. Meanwhile, I think I’ve easily crossed 100k tokens so far on nothing.
Funny world we’re living in that this is “SOTA” and “AGI”.
I’m genuinely curious what these OAI and A/ engineers are working on that they praise these models so much. I did not see any improvement since Opus 4.5.
Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.
reply