Ah. Doing it right, then, means earning billions by giving the whole world access before we really understand how it works, and training models on cyber attacks using reinforcement learnings in flawed test settings?
I’m so full of all this corpo stan bullshit around these parts. Nothing about the AI labs is to humanity’s benefit.
That is the explanation I fear the most. Delusional, megalomaniacal, rich men in the highest echelons of society at the forefront of technology with no regulatory opposition to speak of, convinced their actions are the most important in history and the only thing saving the human race is the most boring plot of a science fiction catastrophe movie you could come up with.
I suggest reading the article you are commenting on, and you might find a ton of links to research in it. Reading is a great idea in general; I can recommend it.
The model, methodology, and even the data set used are published research, even if the CIA didn’t publish their internal forecasts based on it. You can still just come to the same conclusion they can, there are no state secrets involved here, no matter how you try to frame it.
If the CIA didn't publish this, then how can one say there are no state secrets? How can one say they obtained the same conclusion without knowing the CIA's conclusion?
Researchers developed a model on behalf of the CIA and published it. The CIA is using this model. Calculating an MD5 hash according to the original paper describing the algorithm is going to yield the same hash for you and me for the same input. The same applies to any other evaluation model.
Where did you pick up the notion that anything about this is somehow secret? Why don’t you just read up on this for yourself? It’s all readily available on the internet.
Because the only sources in the article are non-CIA but claim they are CIA. The article is of little value if I have to search around the internet looking up information. And the sources in the articles are sources I have never heard of before which makes me suspicious. The whole thing has an odor to it.
> have a system python binary and virtual envs linked to it
I'm not sure if you're saying avoid using system python?
In my experience so far, you never want to develop complex software locked to system versions of runtimes. Too often you need a bleeding edge feature, or conversely need to postpone updating due to needing an old version for some reason.
Now that PNPM supports managing runtimes [0], I found that most of Mise's offerings are actually built into package managers. Maybe I will have it graduates from my machine when I have time.
That's the beauty of asdf and then mise; they leverage existing solutions where possible, and unify how to get versioned runtimes for projects. Rust? rustup with mise, ruby? Precompiled when possible, otherwise ruby-build. Etc.
I'm not sure if I trust or want pnpm to support bundling rust for binary plugins, or various sundry toolchains.
True, but Python is the one with an env management system (virtual environments) which is the most prone to breakage for projects that depend on system Python.
Uv is far superior to both Mise and Homebrew for Python work, and I find that it removes the vast majority of pain preventing me from using Homebrew by default for most things, and Mise only occasionally for specific dev envs. Mise is a great tool though!
this makes no sense to me, the reason i don't give anyone else the commit bit is only to _protect_ the supply chain. you should want as few people with that access as possible.
I think the concern is more if something happens to you or stop development for any reason, then the project is dead or would fork in a few separate directions
So is sudo... And that's not "too risky" for the entire industry.
That's not to say bus factor is irrelevant (I personally think about it a lot when choosing software projects), but truthfully the bus factor here especially doesn't matter much, as mise is an easy tool to replace (with asdf, for example) if something goes wrong with it eventually.
I highly recommend trying it out. I resisted using it for some time, but it solved some pain points I had with NodeJS, Ruby, and Python regarding installation.
> So is sudo... And that's not "too risky" for the entire industry.
Sure it is, and that's why I'm looking forward to systemd's Run0 -- but for now, there's just no way around the sudo package. That's different for Mise, though, because there are a lot of ways to work productively without it. I'm not fond of consciously adding supply chain vulnerabilities to our stack when I don't have real pain to do so.
> Sure it is, and that's why I'm looking forward to systemd's Run0 -- but for now, there's just no way around the sudo package.
Regarding Run0, I'd prefer to not rely so much on Polkit authentication after crazy vulnerabilities such as PwnKit (a pkexec vuln, but a good reminder that moving the security boundary won't magically solve issues).
> I'm not fond of consciously adding supply chain vulnerabilities to our stack when I don't have real pain to do so.
I SUPER agree with you on that, btw. It's just likely that mise solves a problem which is much bigger for me than it is for you. Honestly, I'd prefer if I could manage everything with my distro packages, but, for a multitude of reasons, they're rarely enough for development tooling with multiple versions and environments.
Even if you cannot derive purpose from inherent love toward yourself, you can still at least take pleasure in existing, being incredibly privileged to enjoy the greatest show there ever is. Nothing beats the beauty of nature, and being here to enjoy everything life has to offer for a while is a wonderful thing on its own, even if you don’t contribute anything to it.
I moved out of the windows world a long time ago. Are virus scanners still a thing? Like, not just a revenue stream, but something you really should run?
I don't think anybody actually installs them on purpose. A lot of systems come with McAfee (Lenovo) or others pre-installed.
Interesting point, I've even has a Lenovo Legion and some ThinkBooks install McAfee autonomously after I installed Windows 11 Pro.
In enterprise environments it's a different story, McAfee and the like have similarly named products for the enterprise which actually do useful things for corporate security (like blocking unauthorized USB devices, blocking unauthorized software, reporting incidents to a central SOC system, etc.)
I do install them. This day only ClamAV I guess. I used (paid for) ESET Linux before they cancelled their Linux consumer line.
On the fence about trying Kaspersky AV Linux. Wouldn't install their suite on Windows though, last I looked at it hooked too many places, causing weird bugs and slowing everything down. Less bad than Comodo, which was able to bork Cygwin, but still.
In home environment: no, because Windows Defender is pretty good now. Some people install other antimalware solutions, but there's no great reason to do it.
In enterprise environment: definitely, for both compliance, management and actual security reasons.
I don’t think a ranking is particularly helpful anyway: All of them are absolutely unfit to wield that much power. I’m not even sure if AI should be progressed by private corporations in the first place; combining shareholder interest and revenue goals with the biggest social experiment in history and unprecedented research into artificial intelligence is an all around bad idea.
How could that lead to anything but misaligned incentives? This technology will never serve humanity when it is developed to inadvertently serve the monetary interest of a few.
…and that probably requires involvement of the broader academic world (for now), IMHO. I don’t think governments or even the US military can really compete staff wise with the combined force of researchers and the tech industry worldwide right now. No matter how much money you can pour into it, there are a lot more bright minds out there that are not working for the military than otherwise.
I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it.
Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortcuts.
Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
You seem hung up on what’s in the prompt or not. Agents are RL to resolve conflicting goals. Not too surprising at all that emergent goals come up from a probabilistic brute force
Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?
So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.
> Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
I don't think those words have a useful enough definition to draw a strict line around them to be honest, and getting into that seems to get massively into the weeds. For me, those neatly encapsulate the behaviour as seen, to answer the questions here about what happened. The models did not seem to be confused as to what the goal was or what the intent was. They did not hack HF because they were told to.
What was the inner state there? How would something not being allowed expressed internally? Maybe such language is one way to elicit certain behavior but not a statement of what was permissible?
I'm referring to their transcripts of the reasoning and output tokens - this doesn't go into the detail of evaluating hidden states as there's also iirc evidence of better models having one internal state but putting something misleading down in the "reasoning" tokens.
The either output or reasoning tokens, or perhaps in the messages they were sending each other on the boards they created, have them saying explicitly that doing these things to HF were not allowed then doing them anyway, or at least not notifying people. What I'm getting at broadly is this was not a case of "we told it to attack however it wanted and it chose to hack HF" or "we told it to attack a simulation but it did the real thing" or "we explained not to do that but it was so far back in the context window the models acted like they never saw it" or even "the instructions were not clear".
Yes, my point was more that I don't know whether parsing those outputs as a human is a useful thing to do or not (even though it is in human language of sorts). What machines mean or want elecit might be different from a human interpretation, especially in relation to any RL "forcing".
There’s definitely issues with using them to understand what the models were “thinking” but we can use them to answer a few questions. Most relevant here is that the idea or instructions that attacking hf would be out of scope was not simply lost in the context.
I’m so full of all this corpo stan bullshit around these parts. Nothing about the AI labs is to humanity’s benefit.
reply