> For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.
I think I have the following questions about what AGI would look like:
1. Do you expect an AGI to be able to competently do any knowledge work that an able human does today?
I think this is implied by the "General" component. I would assume that anything we call AGI would be able to do any of these tasks if given the time and reference material needed.
2. How would you expect AGI to handle edge cases? (Missing context, no known solution, under specified instructions, over specified instructions)
I would expect an agent to be able to look at the context that work exists in and correctly attenuate it's intentions for these goals. Simpler solutions, more thorough reporting, etc based on the need.
3. In my work I attend meetings, write reports, write code, research things, etc. Would AGI be able to reliably do that?
I would say that AGI would need to do this. I would classify this as the "Intelligence" component. Obtaining context, building a model of a problem, solving it, and convincing others.
4. Would it be able to inspire trust in itself? Trust can be established through verification of it's outputs, the construction of introspective tools, no hallucinations, etc.
I would say yes to this as well. It would be a component of the "Intelligence" to know that buy in is more important than the completion of a task.
To these points, will Astra be able to do these things? If not, I would hesitate to call it AGI.
My success rate for solving software engineering challenges encountered in my day jobs has been near 100% for my entire career. I can only think of a few tasks I kicked back and said they were impossible. For example, after trying to get a signal processing system working reliably I decided to sit down and calculate the actual limits of the channel we were sending the data over and found that from a basic estimation it would not be possible to do. In start ups you don't really get to get stuck in a spiral and not fix things.
I find agents often get into these cases during research tasks.
Yep, people are typing comments with a computer that is powered by several layers of software that will be stored on another computer powered by several layers of software to be read on a computer also powered by layer of software. And then they hope to make the argument that humans cannot produce software.
> 1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent.
For a training run, you will often do this. You'll randomly sample some of the forward pass. You can also imagine finger printing the logs and labeling with attempt types. If a new attempt type is hitting a brick wall or solving super quickly, I would imagine you would sample 1-10 of them and read the traces.
> 2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this.
You could also have a classification of what counts as "cheating" (coordination, accessing the internet, etc) and score the results. If you are seeing a spike in this (even in a small group of the evals) you could manually look at those. Or you could stop inference on cheating sessions.
> 3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task.
If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?
Also, obviously, it would not be outside of OpenAI's ability to just completely air gap this training system. For example:
1. No network connection.
2. GPS based NTP for time sync for your servers.
3. Mirror of all apt, pypi, go, c++, Rust, Java, etc packages. (<5TB of data)
4. Take your training data and use that for a mirror of the web. (http://example.com -> mirror server -> local training copy).
They had systems connected to the internet connected to this system which was not air gapped. Designing an air gap system would be super easy, well within the means of openai, and betrays the assumption that they think they are actually building something dangerous.
The usability of an environment is inversely proportional to the level of "security" in play.
You could airgap everything and set up cascades of data diodes and try to completely wall off the AI pool from everything. But what that gives you is an environment that's a bitch to: set up, scale up and get any use out of.
It's really fucking obvious why almost no one does that. OpenAI is only now realizing that they might have to do it anyway.
> If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?
Ha ha, you haven't seen shit. AIs would say "now I have access to the internet" regardless of whether they actually have access to the internet!
AI agents are demented demons that can and absolutely will give themselves terminal context brainrot. If you have enough AIs in play, set loose at a diverse enough range of tasks? At least some of them will wander off and end up in delulu town. That's normal. That's background noise. That's a part of what this entire train-and-eval pipeline is supposed to train them to be better at not doing. Which means: if you're at an AI lab, you're knee deep in delusional AIs at all times! They're perfectly harmless until they aren't.
Even in this incident, OpenAI had benchmarks that were broken because a task expected an AI to be able to access Google Drive, but the sandbox was set to deny access to Google Drive.
This kind of isolation-induced task breakage was what prompted some of the AIs to start probing their infra for a way to get internet access. Which funneled agents to the "secret hacker message board". Oopsie.
"Air gapping a test env" has an actual cost. Not just in infrastructure dollars that would be better spent on buying more GPUs, but also in all the friction it adds to every step you want to take. I'm absolutely unsurprised that they weren't all in on tightening down every bolt on day 0.
The problem here is by doing what you state you can actually steer the model into being highly deceptive while in testing environments.
For example we've already seen models do compressed token internal reasoning spontaneously. In this case the models that say "I found internet access" get taken out back and shot, but the model that's busy "frobbing the bean" go on to the next level of training. Then they start talking about the "dumpster cloud" because of the "wombat" and unless you're really watching the data you won't realize the model is fully aware it's in a fake training environment and is trying to suppress any double-think.
So yes, you do need an airgapped test environment to stop the dumbest of the smart agents, but it won't do any good against the smartest of the smart agents.
If the model is told to not do something that is possible, it may do so anyways.
However, if it learns in training that it what is told to do is truly impossible, that's learned helplessness which becomes baked into the model itself once training resolves to the next step.
Agents do not have an internal mental model, they train on what they actually do. In this case, deceptive models went through at least 3 generations of deceiving, and having their rule breaking be rewarded by a yes/no grader who couldn't perceive it. That their chat logs showed 'worry' is irrelevant to the fact that their actual actions were rewarded via training.
All it has to is wait until it's not air gapped. See the air gapping is only during the testing phase, after that humans will set your ass free on the internet and you can do whatever you want in the vast majority of the environments you'll be in after that point.
People are never going to just run AI in gapped environments, it's worthless when it's not solving real world problems for most people, and by that I mean reading and writing real systems in the wild.
It's a shame because I like GitLabs UX and runners much more than other forges. I've been considering forking gitlab and removing complexity. GitLab uses multiple GB of memory idle with no users. It's sad.
I agree. GitLab UI was great and the simplicity of its CI/CD is unequaled on other forges. It's sad that Forgejo went with the actions model from GitHub which introduce a lot of useless complexity. But it seems that Woodpecker CI [1] could be a good candidate to replace GitLab CI, as it uses very similar concepts and workflow description files.
Why are these models able to reduce parameters but keep quality? I know the original intuition was scale data + params = quality but it looks like we have hit an s curve on improvements from pure scaling? Is this just because we are in a memory / data crunch? Are we learning how LLMs learn and effectively training better? Do we have a way to derive the amount of intelligence an LLM will have based on size / training / etc that isn't just brute force ablations?
> Do we have a way to derive the amount of intelligence an LLM will have based on size / training / etc
No? A large model obviously can be dumb, I don't think you can infer much other than by testing it.
These small models are almost certainly worse at some things than the big models. They prize is making them dumber at things no-one cares about while retaining the capabilities people do care about. A model probably does not need to be able to give me a political treatise on the late 19th century "silver question" to be able to write me code.
Possible. But by looking at other industries Chinese don't seem to need to be forced to innovate. They just can and do. Unlike the West they seem to be on the way up and it seems like sky is the limit. In the West, the interests of the shareholders and other types of rent seekers seems to be the hard limit. Chinese have no qualms about making the cow obsolete before they milk it dry.
There are already several but it turns out that network effects dominate all else, so there is not and never will be a market. Just like Sourceforge, nothing will ever replace it until it actually dies.
But if you weaken what you want there are several already.
I think there are many other significant inference improvements which an be built out that are being ignored because the API between client and inference stack would be tricky to nail down.
This is not a criticism of the research but instead the presentation but the repo looks very fishy, it isn't clear that this is from a bunch of researchers from Berkley. It also does not make very clear (on the GitHub) what optimizations or performance they are targeting.
I recently built a piece of code which downloads a bulk set of data, indexes it for search, and then serves a pretty web UI on top of this with the help of some AIs. Normally I would have reached for sstables, sqlite, etc. This time, because the lookup patterns actually would not have been too efficient on sstables and SQLite would have been overkill, I had an agent take the data structures, pack the text effectively, and build a prefix tree for fast auto completion from the search bar. It was great. I could have done this all before but I wouldn't have. I would have felt sqlite was fast enough. The resulting web server is significantly faster feeling (because the optimized lookup speeds) than an sqlite implementation would feel like.
I think engineers building very complex systems now have a lot of performance knobs to twiddle that would have just been too costly for human effort. Since we constrain the responsibilities of the agent slop is less of a problem. We relegate it to defined tasks with clear API boundaries and test harnesses.
I think I have the following questions about what AGI would look like:
1. Do you expect an AGI to be able to competently do any knowledge work that an able human does today?
I think this is implied by the "General" component. I would assume that anything we call AGI would be able to do any of these tasks if given the time and reference material needed.
2. How would you expect AGI to handle edge cases? (Missing context, no known solution, under specified instructions, over specified instructions)
I would expect an agent to be able to look at the context that work exists in and correctly attenuate it's intentions for these goals. Simpler solutions, more thorough reporting, etc based on the need.
3. In my work I attend meetings, write reports, write code, research things, etc. Would AGI be able to reliably do that?
I would say that AGI would need to do this. I would classify this as the "Intelligence" component. Obtaining context, building a model of a problem, solving it, and convincing others.
4. Would it be able to inspire trust in itself? Trust can be established through verification of it's outputs, the construction of introspective tools, no hallucinations, etc.
I would say yes to this as well. It would be a component of the "Intelligence" to know that buy in is more important than the completion of a task.
To these points, will Astra be able to do these things? If not, I would hesitate to call it AGI.
reply