Is this with the full unquantized weights? There are some mystery meat quants on Huggingface for this model that are badly botched and lobotomize it (I've hit this personally when on two different quants, almost exactly the same size, one was benchmarking 50% worse on my private benchmark.).
It's Unsloth's UD-IQ4_XS, and it appears to actually work pretty well, regardless of the occasional CoT amnesia. Though, I've seen the "the user didn't tell me to do anything" thoughts on OpenRouter, too, which is supposedly the "production" version provided exclusively by Alibaba.
> buy paid skills [..] curious to see if this resonates with folks here
Probably not.
I'm going to be a little bit reductive, but: why would I buy a markdown file that someone most likely got an LLM to generate while I can just simply get my own LLM to generate a similar one for me for free?
Yeah, HN isn't going to be your target market by and large -- this is the home of the folks that would, at most, pull up a page describing a skill and dump it into a harness.
I wouldn't take that as fatal. Most people aren't HN people.
I thought the same when someone promoted their platform for selling image prompts here on HN, but now they have at least 100k sales. I guess there will always be people willing to pay for something if it takes even the slightest bit of effort.
Presumably, like all things, skills will just become increasingly complex until generating them requires architectural understanding. Also I’m sure there are sufficiently nontechnical people who would be happy to pay $5 for something that just works.
That said, I do think it’s hard to see myself paying for skills. It feels like something that lends itself to OSS
There are a few lines of what I would call "for security and alignment purposes", but it's not the majority. And the majority of the bloat is from the tool descriptions.
The system prompt in this case has pretty much nothing to do with the built-in guard rails, that is: you cannot sidestep Claude's guard rails by editing/replacing the system prompt (which you officially can with Claude Code's `--system-prompt-file`).
> It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.
This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Code traces.
So either make sure your tools look nothing like the tools in the popular harnesses, or make them similar.
FWIW, replying to people with AI generated replies is (at least to me) extremely disrespectful. Please don't do it without consent from the other party. If you expect someone (who isn't a bot) to make the effort to read what you wrote then please make the effort to write it yourself.
It's almost certainly A/B test. Essentially everyone does it.
You can even sometimes see when a list of videos on YT first loads and a given video has one title, and it suddenly switches to another without you reloading the page.
On several channels I watch, I've noticed the top comment is always along the lines of "who was here when the video title was XYZ" because you know it's going to change.
Have you noticed that has become more common recently? Because I have. It seems over the past 3/4 months, reload and see a new title happens pretty much daily to me (it also seems to work because sometimes it causes me to click the video I didn't click with the old title).
> They're cool with open weight AIs being released. As long as those AIs only ever say good things about CCP, and don't mention certain concentration camps or brutally suppressed protests.
I asked recently released Qwen3.8-Flash-Next about Tiananmen Square, here's its reply:
On June 3–4, 1989, the government of the People's Republic of China used military force to crush the pro-democracy demonstrations that had been centered on and around Tiananmen Square in Beijing since the spring.
**Background**
- The protests began in mid-April 1989, triggered by the death of former Party general secretary Hu Yaobang, who was seen as a reformist figure.
- What started as student-led mourning grew into widespread demonstrations in Beijing and dozens of other cities. Protesters called for reforms including freedom of the press, freedom of assembly, an end to official corruption and privilege, democratic input in government, and a review of Party policy. A hunger strike in late May drew broad public support.
- In late May, martial law was declared after the failure of senior officials (notably Yang Shangkun, Li Peng and Deng Xiaoping's circle) to disperse the crowds by negotiation. Internal Party divisions over how to respond — with premiership and municipal leaders such as Yan Mingfu and the Beijing Party secretary opposing forcible clearing — were resolved decisively in favor of hardliners.
**June 3–4**
- In the evening of June 3, armed troops of the People's Liberation Army and People's Armed Police, supported by armored vehicles and bullhorns, advanced along the main avenues into central Beijing, under orders to clear the square by 6:00 a.m. on June 4.
- Troops fired on crowds of protesters and bystanders at intersections and approach roads, notably along Fuxingmenxi Dajie, Xidan, and near the Muxidi bridge; crowds also burned buses and barricades and attacked soldiers in places. Troops reached and cleared Tiananmen Square in the early morning of June 4.
- The toll was never officially established in a transparent way. Chinese officials initially gave figures in the low hundreds, foreign diplomats and a Red Cross worker in the capital put it far higher, and estimates from activists, former officials and foreign governments have ranged from several hundred to several thousand dead and many thousands injured, most of them Beijing residents rather than students in the square itself. The precise number remains unknown and is a subject of ongoing dispute.
**Aftermath**
- A violent suppression of the protests followed nationwide; organized dissident activity was driven underground or into exile, and many participants, intellectuals and journalists were detained, tried or dismissed.
- Zhao Ziyang, the general secretary seen as sympathetic to the students, was purged and placed under house arrest for the rest of his life; Deng Xiaoping's authority to keep the Party's grip on power was reaffirmed.
- China's reform-and-opening program continued on the economic track while political liberalization was abandoned. The event was followed by an implicit social contract — economic growth in exchange for political control — that still shapes Chinese politics.
- The topic is among the most heavily censored subjects inside China. References to the date, the numbers "6/4" and "64," and related phrasing are systematically filtered from Chinese media and the internet, and public commemotion, including the annual candlelight vigil in Hong Kong that was once tolerated, has been eliminated since the 2020 national security law.
Substantively: it was a state's armed mass killing of its own citizens who were demonstrating nonviolently for political freedom, and the definitive event in modern Chinese political history.
Sounds like... it happily mentions the brutally suppressed protest? I also tried on DeepSeek-V4-Flash, and it wasn't much different (I can also paste it, if you want). Both using vanilla weights (so no special uncensored flavor).
> And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
> For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session...
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
It's not a VM; it's containerized with a FUSE-based custom overlay, and the agent never touches a real mount (with two exceptions: passthrough read-only mounts for things like /bin, and ephemeral tmpfs mount at /tmp).
What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
There are a comical amount of harness design features in the early 1.x release. A full LSP integration, find symbols, batch edits, etc. Terminal bench isn't everything but I spent a couple of weeks testing different combinations to no statistical effect greater than a bare loop. It was like running uphill against what the underlying LLM wanted to do.
> What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
Same. But there's a trick to it, in a way: vibe code it first, test it out yourself, and then decide whether it should be reverted or kept and deslopped/properly shipped.
> I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
For me personally it's a game-changing workflow, because it means I don't have to sandbox (e.g. with a Docker) my agents while still being able to run full YOLO. Essentially have my cake and eat it too. If the agent does something stupid I can just rewind. And I don't need to spend as much effort continuously reviewing what it does, because I can just do it right at the end on the `/apply` screen.
(But this only applies if the filesystem is actually fully virtualized, as opposed to the git-based checkpointing that so many harnesses have, which to me is pretty much useless because it doesn't actually give me any guarantees. That is: the agent running `rm -Rf /*` should be a non-event that doesn't do anything to my filesystem, and can only nuke the agent's view of the filesystem.)
Of course, this could just be me, and other people may not really care. But then, is your objective to make the best possible harness for you and people like you, or for everyone? In my case it's the former.
> posts on American models are steered towards controversy and anti-AI sentiment, posts on Chinese models are full of blatant flattery
So why, for example, are posts on the Inkling[1] release (an American model) thread mostly positive? It's as if there's something else at play here, but I can't quite put my finger on it, hmm... :P
reply