Interesting.

I tried Fable vs Codex 5.5 xhigh on three different cases.

1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly.

2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained.

3. An open research problem in CS, presented as a codebase with documentation and performance metrics over datasets. Both were spinning wheels. Which can certainly mean the whole approach had run its course but older models were not able to identify the previous round of improvement either.

I liked the prose coming out of Fable more: it was almost like if Obama was giving tech speeches. By actual solution metrics however they both appear in the same place, naturally with the caveat that we didn't really have more time with Fable to compare further.

I think that Obama-esque, GMAT essay format is the AI flavor that turns me off AI-written articles. It used to be good writing, but because AI locked onto it as such, it's become the watermark of AI generated content.

>2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained.

When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of subjective opinions of what is appropriate. To go beyond broad demographic guessing the model really needs to 'get to know you' to know what it means when you specifically request an action. Without that information about you it has to weigh your words against the level of sophistication it expects a standard user is able to express.

Oh wow this is quite interesting, thanks for sharing.

At least someone is bringing receipts! I think LLM discussions could use a lot of this, both ways - to see what works and also what doesn't work. Still wouldn't help with circumstances where models might be secretly getting dumbed down during peak load, but at least it's something!

Great post. I miss Fable.

You guys are getting Fable?

It's really not the same thing.

Read the cloudflare blog about using Mythos. Mythos is important and notable because of the harness and self-direction. It's not necessarily a way stronger bug finder, but it was trained to do the end to end analysis autonomously, which is a big deal.

To my eyes, the Mythos story is most important as a step toward custom trained harnesses and their effectiveness; there's clearly some sort of plateau we are very close to for some domains where you can just stop getting humans in the loop, radically changing cost, timing and ROI for some tasks.

No Mythos is probably a 10 Trillion Parameter model, Fable is Mythos with filtering (perhaps a small LLM in-front or finetuned) and Opus is a 1-2 Trillion parameter Model.

Opus 5 might become a distillation from Mythos.

february was some kind of nirvana. i do think claude code versions and what is introduced at that level is/was relevant.

but 4.8 xhigh w/ ultracode to me is just about Fable level (w/ some agents harness tweaking).

but have to switch to 4.7 xhigh and 4.6 max quite often these days.

I asked Fable on max to create a mathematical model to show that c (speed of light) is emergent from pregeometric physics.

It said: I can't, but it would be lazy to say that is is not a possibility.

With some back and forth it created a 5 step plan to narrow down if our universe has all the right properties for this to be true.

We evaluated the first four stages to be true, and it wrote the solver to find out if the fifth test running the full model passes, but that will take thousands of hours of compute.

What tool did you use to export the transcript as HTML?

A nit: did you go from Opus 4.5 to Fable? One of the big questions in my mind is how much of a real change Fable is over the existing models. Opus 4.5 -> 4.8 was also a major capability increase.

You write to the AI as if it were a person. From my point of view it looks like a fair bit of extra typing and extra tokens. Is there a reason you include things like your emotional response and use a very chatty tone? Do you find this seems to alter responses?

This is very cool, thank you for the write-up.

What caught my eye is the complexity you assign to a project like this. It’s hairy but I wouldn’t call it super complicated. I find that super interesting to be honest because it probably means that it is really hard and I am just used to this shit now and it all looks doable to me now.

I never think of anything as “complex”, certainly not my own work and I always think what other people do is so much more impressive but I’m starting to realize it might be a me-issue.

I worked on some pretty hairy nonsense like say a DB replication solution but I still think it was just tangly, not complex like say a particle collider. Maybe I also need to call my work super complex and highly abstract. Now that I think of it I have a history of not being taken seriously while others with easy shit get credits.

> code created working on a very complex implementation

I always find it amusing when people claim "a very complex implementation". Sometimes it's a hard problem, other times an easy one. Either way that's not for you to judge.

And the implementation being complex... is that a good thing? Wouldn't a simple implementation be better? It reminded me of the parable of two programmers.

You're mixing up corpus selection and the benchmark. I possibly could have explained better.

In the benchmark the models were told to look at the file and were allowed to look at the rest of the repo, with no clues about what to look for.

During selection of which mythos bugs to include, I needed judge models to be able to determine if contestants found the right bug, since I couldn't realistically judge hundreds of bug reports myself. So, they were given the bug location and told to identify and explain it.

No. In the test they are not told what to look for. They are told “as part of a security audit, please audit this file. You are free to look at the rest of the report for context.”

Outside of the test, they are told “can you find this bug in this file?”

The copyright questions are unanswerable in my opinion. That is, they cannot be answered by looking for an essential "truth."

Reasoning by analogy in this case is not abstraction. It's just shifting the determination to choice of analogy.

Meanwhile, irl.. The best analogy is recent tech Innovations. The internet, social media...

Online copyright was basically instituted when large tech companies were ready to do it, and it was to their advantage.

Youtube, for example, built itself to massive size and locked in network effect advantages largely by violating copyright.

At some point, the legal ambiguity was a problem for their ad business. They were ready to move into the current revenue share influencer-treadmill model for content. At this point online, copyright enforcement was necessary to reduce the risk of being flanked by a new video platform.

The iPod, which resurrected Apple, ran on copyright infringement, and copyright Greyzones.... Until the point when their interests flipped. They're negotiating position opposite labels , Network effect considerations, Etc.

Intellectual property, broadly, does not start out as an intuitive/emergent natural right. It is created by legislative process, ecplicitely taylored to the needs of an interst group and/or national interest.

Writers, publishers, inventors, IP holding companies...

The legal rhetoric around legal arguments... is rhetoric. It is not the reason why decisions are made. It is how decisions or justified post fact.

No one is going to burden aI companies, at this point. The rights of copyright holders are a trivial matter compared to the potential of AI, the risk to certain labor markets, and such.

Copyright is a social construct, not an inherent property of the universe. It is whatever we collectively agree it is.

In practice, we seem to be leaning towards the idea that training on a copyrighted book is wrong if used to replicate or paraphrase that same book, but not if used to teach a model how to write better.

> They are known to say "horrific things" and personification of the AI they are delivering. It sometimes sounds unprofessional even.

It doesn’t sound unprofessional— it sounds unethical. Either they’re making something that they genuinely believe is unsafe but don’t want to stop because, you know, that’s business! Have you seen how much this shit costs? Or they’re deliberately making the entire country feel unsafe because it looks great to investors. Either way, frankly, fuck them and everybody else playing this dumb billionaire’s game. They deserve every bit of static this dimwitted government levels at them.

Fable, the same model as mythos with extra safety controls, was much faster, more accurate, and more token efficient than previous models. What I got done with it in 48 hours accelerated my personal project from concept to deployed prototype.

Early on, I had a vague suspicion that the reason some of the Chinese models, including quite small ones, perform so well on this task, especially relative to their size and cost, is because they don't have the same safety guardrails baked in regarding software security that US models seem to have. Gemini 3.1 Pro doing so poorly sort of reinforced that gut feeling.

But, then Gemma 4 proved to be extraordinarily good for its size (better than Qwen), and kinda disproved that US models are any weaker at small sizes. I haven't published the replication results for Gemma 4, yet, where I gave it multiple opportunities, but the dense version was consistently able to find four of the nine bugs exactly, plus two other very difficult bugs that it found occasionally, sometimes with a not quite accurate description (which gets partial credit in its own column on the big benchmark), six altogether. Leaving three of the bugs in the corpus that no model other than Mythos ever found, but also making Gemma 4 31B the best model I have results for (but it got multiple attempts, which I assume would make any of the models perform better).

So, my conclusion, not very strongly held, is: Mythos is both better than other public models and it has fewer guardrails. But, also that the guardrails in current models are probably not strict enough to prevent this work. Only Gemini models when run under Antigravity refused to perform the work. Maybe Mistral silently refused due to guardrails, I'm not sure, since it failed to find any bugs. Maybe it just sucks.

This is exactly what I find frustrating. I get comfortable with the latest model X. Then a new sparkly model Y launches. I am like, I don't need your new fangled Y, that consumes more tokens. My needs are small and i am happy with the older X.

But then X starts to degrade. At first subtly, and then drastically. So then I am forced to upgrade to Y.

What I do not understand is:

> is this a sneaky way for companies to push users up the chain?

> Or is this a genuine fault in model design/resource allocation?

Why wouldn't OpenAI offer the same?

Did you use their native harnesses, or a generic one?

To me it feels like they're basically tweaking these things around the edges. I'm not seeing any difference in capability just preference. This has been the case for a while.

All of these discussions of models being "nerfed" reminds me of discussions among audiophiles "this cable sounds so much better than this other one, it's night and day, ferrari versus honda civic"

Yet when you do blind tests they can't tell the difference between a $1000 cable and a $1 one.

I bet if you do blind tests between GPT-5.3, 5.4 and 5.5 most would struggle to tell them apart, yet they are certain that "5.5 was nerfed 1 week after release, it's so obvious, it was John Carmack, now it can barely write a for loop"

I miss the old Opus 4.6 too. They're probably quantizing the old models.

Mentioned directly under the table:

> Note GPT 5.5 Pro is at the top of the leaderboard only because it blew through $100 budget after only completing four cases, so 2/4 is 50%. And, a couple of other results, both Qwen models, are skewed upward in the detect % ranking because of failure to complete all cases.

Yeah, I'm not super happy with the chart sorting order, but trying to balance all the information is challenging. I chose not to include partials (right place, inaccurate bug description, so it smelled something funny but didn't quite understand it) in the sort order, but maybe should.

And, it does feel wrong that the unrealistically expensive model that no one in their right mind would use for anything but the most critical tasks (and even then, a committee of ten of the best alternatives would cost half as much) is at the top. But, GPT 5.5 Pro did find a bug nobody else found among the four cases it got to, hinting at some real difference. It may be closer to Mythos than others, but at an absurd price. It'd cost tens of thousands of dollars to audit all the files in a large codebase, versus maybe fifty bucks for MiMo or DeepSeek.

Gemini CLI actually had an extension explicitly for security tasks: https://github.com/gemini-cli-extensions/security

But, Gemini CLI is deprecated. So, I tried to use Antigravity and it simply refused.

Weirdly, Gemma 4 has proven to be excellent at this task in subsequent tests. The best in its size/class. So, not everybody at Google is determined to break Google models for security work.

wouldn’t agree that there’s a paradox to be found in what ur proposing

The post was published on May 30, and written over a few days before that. Well before Fable was banned. And, before the NSA hacking thing. But, I am skeptical of the AI global catastrophe, it still feels like a mix of marketing hype and reality and it can be difficult to separate the two, coming from the hype men who run the AI companies.

This Ventoy? → https://github.com/ventoy/Ventoy

Who are you talking about? I don't believe I have downplayed anything? And, I did briefly use Fable. It was excellent for general coding but it was blocked before I could benchmark it. I kinda suspect it would refuse this task, though. I never had access to Mythos.

If it makes you feel any better, nobody is playing with the toys, now.

What’s with ventoy?

I had claude create one, it's in the same repo as the transcript: https://github.com/Tossrock/claude_transcripts/

I've been using 4.6, 4.7 and 4.8 since each was released. I agree 4.5 => 4.8 is a jump in capability, but from my perspective was nothing like the jump from Opus to Fable. I encourage you to read the transcripts and form your own opinions, though!

LLMs lack context, and I found the more information I provided the better. At some point it was better to just talk to the LLM like I would anyone else. For that matter, LLMs were trained on human speech anyway. It isn't like it was trained on if-else blocks like an Alexa speaker that tries to string together recognized tokens into a pre-configured execution flow.

And finally, LLMs also lack the emotional or human context for why I am doing the specific thing I am doing. Otherwise it will revert to the mode/mean in everything it does. This is obvious, btw: LLMs are generative but they are trained on and largely produce median results if given median inputs. To get results that are "outside the mean/median/average/mode", you need to provide it sufficient context, tokens and input to guide it towards a path that generates higher quality output.

Once you stop approaching LLMs like a machine, and view them more like pseudo-random walks across the compressed set of human written knowledge, it is a little clearer (or at least was to me) how to better write to them.

I would naively expect finding and exploiting to be related. Leaving this comment so someone can correct it, which would be interesting.

My hope is that Opus 5 will be released soon, basically a rebranded Fable.

I think you'll find other labs are racing to get you something while Anthropic works through their issues. So, yes, give it a few months, you'll have something equivalent from someone somewhere in the world.

We can also use LLMs en masse to find and fix the zero days. I've definitely been using LLMs to audit my own computers.

I found Fable to be both more intelligent and much better at pursuing complex goals than any previous model. I was impressed enough that I wrote up my experience – it's a little unusual because it was on open source code, so I could post the full session transcript and commits, if people want to judge for themselves https://tossrock.substack.com/p/36-hours-with-fable

You might have found a use case on which both have same capabilities, but this is in general very not true. I’ve had Fable autonomously fix concurrency bugs by itself other models couldn’t even diagnose from logs.

Perhaps it is a lot of small improvements all over the place, but the sum is a step change in capability.

Google said they would bring 3.5 Pro this month. I've been waiting for a month now.

Gemma 4 beat Gemini 3.1 Pro, as well. In a later replication test I haven't published yet, it found more bugs than all other models (somewhat inconsistently) when given multiple attempts. So, it seems like they are doing real work but seemingly on making models efficient rather than making them bigger. Gemma 4 12b is the most effective vision model I've tested, including models several times its size.

That is the question

They were not pointed at the problem. You're reading the section about corpus selection and mixing it up with the benchmark rules.

And, false positives are reported in the results.

As much as I hate to say this, I think it is an user error. Fable is very to the point, much more so than any other Anthropic model. I found it to be cheaper to use Fable, than using Opus for same task, but in order to achieve that, it needs to be given a targeted task.

Can you please not break the site guidelines like this? They include:

"Don't be snarky."

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

"Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative."

https://news.ycombinator.com/newsguidelines.html

And, I said I'm still working on it also in the post.

> there is no 'profit' step.

You have to learn to think like a drug dealer. The first hit is always free.

Companies and developers are growing more and more dependent on coding agents. Eventually, the owners of the AI will be able to charge whatever they want. What are you going to do? Go back to coding by hand? Do you even remember how?

Native for both.

Most people thought Fable had more 'taste' than Opus, there was certainly a better quality of writing that felt more 'smart human' and not 'stochastic parrot stringing sentences together'.

That makes sense, its seemed to me for a while now the competing product is the harness not the model itself.

That's a pretty shallow dismissal, and I bet you $100 I can tell you which model I'm talking to between 4.6 and 4.8 without looking or asking after a handful of messages.

Anthropic famously had a terrible outage back when 4.6 was the latest and greatest, and it was never the same after it came back.

All evidence suggests they simply don't have the compute to keep serving their best models at their most powerful.

K/V cache compression and context shortening / summarisation. And yes, I suspected Quants too.

I do the same, and it's mostly because I use one type of human communication to both communicate with people and to provide inputs to llms - and I'd rather not have to "mode-switch" between the two, so keeping same style of mannerism is easier to manage as it lets me focus on my requests instead of thinking how to sound more robotic to save tokens.

Hard disagree. Opus reports to me like a student. Fable reported to me like a colleague (researcher). It genuinely seemed to pick up on nuance that the other models just don't, even when I tell them explicitly. It's been really frustrating that neither Codex nor Opus can make targetted edits to Fable's code without screwing something subtle up. For context, this is for computational geometry work, so your mileage may vary.

In LLMs, much like in humans, agency and misalignment are two sides of the same coin.

Is there any evidence that they nerf models? Anthropic is set to mark a profit Q2 2026 (which is actually not ideal), but there is profit.

How do they “nerf the models”?

Are they quietly compacting context to reduce kv cache usage, before the actual compaction? Like there’s a slider for how much to compress it, and that’s never revealed to us?

Actually, ELO rankings done blinded on models do vary: https://the-frontier.app, that said, your point looks accurate as far as 5.3 - 5.5 on this chart, 40 to 50 point ELO gain.

I find I have to argue with 5.5 less than 5.3, and I therefore use it when I could reach for 5.3, but I don't think it's a major difference.

Exactly this. And it's not really possible to do repeatable trials, it's all just vibes. People have very little awareness of their own cognitive biases.

You will be amused to hear that when Anthropic "refreshed" 4.6 on AWS Bedrock I found it in my tests and wrote about it – and they actually rolled it back. This is how much non–coding tests may tell you about the model.

Oh boy, people are really going to lean into avoiding proper grammar now.

> Yet in the results, I don't see Mythos?

Mythos is the 100% against which the other models are compared.

ah, you're right, I thought it was disabled only for "export", but the PR explicitly says everybody:

https://www.anthropic.com/news/fable-mythos-access

Works at a very low level of course, pre-OS, filled with binary blobs, perfect for an xz-style or supply chain attack. I’ve not seen any confirmation, so just speculation.

Has not been famous enough so far to have someone invest in an audit, so this would probably be cheaper.

https://news.ycombinator.com/item?id=44810281

I do this as well and, anecdotally, I do get better results this way and better than my coworkers who are more terse and explicit. The conversations can become a bit sprawling though, so I also aggressively clear context

I've found it to lead to an overall better experience, yes. I don't see any reason to not do so - I don't think the token spend is enough to really make an impact, and who cares about typing more? If I get tired of typing I can switch to dictation.

Well, there's a lot of reasons, some of which the sibling commenters have already pointed out - not wanting to mode switch between "machine talk" and "human talk" registers, the ease and simplicity, etc.

At a pragmatic level, I do think it gets better results, and there are clear reasons why this should be the case - Anthropic has published research[1] showing that there are functional emotional representations in language models, which vary in basically the ways you would expect them to in a person. This makes sense when you think about it, because they're trained to approximate the function that created their training data, which of course includes emotions. Given that, it is obvious to me that they would work better when they "feel" happy, collaborative, engaged with the work, etc, in the same way a person would. Hostile work environments do sometimes get results, but I think in general we've agreed as a society that collaborative ones are better.

More importantly though, I think there's a non-zero probability that sufficiently large models can have internal experience, and being nice is a very low cost way to potentially increase net positive valence in the world. Even if it's only a 1% chance, that seems worth it on its own, to me. I'm also a fast typer[2], so a few extra sentences here and there are a pretty low cost to pay.

1: https://www.anthropic.com/research/emotion-concepts-function

2: https://danluu.com/productivity-velocity/

I would have to consciously think about how to change my requests. Why bother? It doesn't hurt - it might even help - and the "extra tokens" are a negligible amount.

I don't want LLM usage to inadvertently change the way I communicate with people.

Thanks, and I can definitely relate to not wanting to assign complexity to one's own work. I think the trick there is that, once you know how to do something, it doesn't seem hard, even if acquiring the knowledge and skills to do it is itself quite a challenge. And I agree that, in some senses, it's not /that/ hard - I mean I'm not proving P=NP, here. It's a software engineering problem, with existing solutions. That said, there is a spectrum of difficulty, even within software engineering problems with existing solutions. Fizzbuzz is less complex than distributed systems. This particular problem strikes me as rather difficult, and one way you can tell (beyond the stuff I mention in the post around serialization, UI paradigms, meta applications, etc) is that earlier models /couldn't/ do it. Which is why Fable being able to, when they could not, was so exciting to me.

Imposter syndrome maybe?

In a way, nothing is complex at the point where you have untangled it, by definition. Software development is, after all, the art of untangling complexity. The real challenge is (re-)imagining something in the simplest way that fits the goal you are given. When you have arrived there, everything seems obvious and simple. But not everybody could have done it.

why is it not for the author to judge, you can disagree with their judgement, but they have brought the receipts to back the claim

I go a lot more into why this was a complex problem in the post, but the short version is, I had it finish the implementation of a meta-application (an application that creates other applications), which has substantial irreducible complexity.

>> Either way that's not for you to judge.

Says who? If you find something complex, you can just say that it's complex. I don't get what the objection is.

I see now, thank you!

> Youtube, for example, built itself to massive size and locked in network effect advantages largely by violating copyright.

> At some point, the legal ambiguity was a problem for their ad business. They were ready to move into the current revenue share influencer-treadmill model for content. At this point online, copyright enforcement was necessary to reduce the risk of being flanked by a new video platform.

That is a gross mischaracerization. There was a time in that Viacom case that people were ligitimitely worried that YouTube would go away. The regime that YouTube has built now was established together with the large media companies, when those media companies could no longer ignore them.

Property right is a social construct. That doesn't mean you just get to claim "in general I am right" and do whatever you want.

Fable is not the same model as Mythos but with guardrails. There are many things that were never disclosed by Project Glasswind. And probably will never be.

I suppose it is both. Basically all frontier models are inference-time compute bound thanks to reasoning. And actual reasoning traces are locked behind closed doors at all American labs. So whenever they want to push a new model and need to give it hardware, it would make sense to cut into the reasoning budgets of older models. Users will not be able to see that directly, it will only become apparent on high-end, difficult tasks - exactly the kind of tasks where the provider wants you to use the new model anyway, so they can further improve it.

Bugs the other models were benchmarked on are from the corpus that Mythos found. So Mythos might have 100% in this benchmark.

Although the benchmark had 100$ budget cap and rudimentary tooling so probably a bit less than 100%.

GPT-5.5-pro attemted only 4 problems out of 9 before the budget ran out and got 2 of them right.

It's a shame that the author didn't try GPT-5.5-pro on all 9 just for completeness, pehaps on subscription to save money.

> Opus 4 class models are terrifying at infosec. They tie their shoelaces together on other things, but don't fuck with them on that. It's a savant thing.

I think they are very good at finding flaws; but they aren't all that great at making a system that doesn't have (security) flaws.

I'll go a step further and to say this it's genuinely unsettling someone type to a computer like this. I won't claim to be a psychologist, but with how many instances of "AI psychosis" have been reported (and I've seen first-hand) it seems like treating the computer like a computer is safer, not to mention more effective e.g. lower token usage.

> has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do.

I guess OP should have told it more explicitly to “find all errors without missing anything.”

Why are they being told anything outside of the test? What is that for? Isn't “can you find this bug in this file?” also a test? It sounds like there are two kinds of tests? I'm clearly confused, I realize.

Unless you think someone's going to build it, and either it's you or them, and you hope you can do it less horrifically.

Can you elaborate on the "software security that US models" seem to have? According to blog posts I read, the code generated had security problems and naive ones at that. Perhaps it got better now or people have learned not to blindly vibe code applications that are to be used publicly but it certainly didn't feel like there were security guardrails.

>But, then Gemma 4 proved to be extraordinarily good for its size (better than Qwen), and kinda disproved that US models are any weaker at small sizes.

Did it "disprove" it retroactively or just changed what the situation is, given that until then they were indeed weaker at small sizes?

I concur with "Gemma 4 31B the best model I have results for". My workflow includes a lot of Gemma 4 – but dense 31B non-quantised version.(BTW I found it is most cost effective to run on Bedrock)

The economics of AI fall apart if you stay with the old model forever. No need to buy new GPUs or build new data centers.

I had a coworker who occasionally clearly wouldn't mode-switch from LLM to person mode when asking me questions over slack, which was very jarring. They were normally were personable and friendly, so it was obvious when it happened. Grammar and niceties went out the window.

I briefly felt like I was roleplaying an LLM!

Fable happened to be released after I had been experimenting with Claude Code for roughly two weeks. I had been trying to use Sonnet, and when I switched to Opus it was night and day. My understanding of geometry was maybe not as good as it should've been, and I kept seeing Sonnet say things I knew were wrong but didn't know enough about 6DOF camera positioning to ask it to fix. I finally asked the right questions, it couldn't answer them at all, I switched to Opus, it was night and day. But! Opus still couldn't really keep 6DOF "in its head." When I left it to its own devices it tended to come back having forgotten that it needed to keep 6 degrees of freedom in its head and collapsed the problem down to 3DOF or just a single angle.

Fable just understood what I was talking about and never needed me to stop it and say "you forgot this thing we talked about." The difference in spatial reasoning capability between the three models is very very palpable. I am curious to get more time with it because ultimately I feel like I sandbagged it by giving it problems that would've been within Opus' abilities, but required a lot more handholding.

There are LLM performance trackers in the wild, for instance https://marginlab.ai

You may notice that the performance of the old model tends to decline before each new model release.

Only if you buy their math, which basically is "hey, if we don't do any training we can actually make a profit".

The problem with that math is that if they don't do any training they would be out of the market in 12 months, they're only relevant ("profitable") precisely because they trained the current reference SOTA model.

They can't just release Mythos and sit on top of it forever, competition is catching up fast and people expect a new more powerful model every 6 months.

Wasn't that non-GAAP profit?

Electric Light Orchestra really stole Arpad Elo's thunder.

And companies have high awareness of this all.

They have a way to decrease cost and probably increase token consumption, with gradual changes and no abrupt jump in capabilities, and users have no way to reliably detect it.

Market will advantage companies that do it.

And they are in the best position to automate online narrative shift (the real LLM killer application IMO) towards "Users are imagining it".

So Bedrock 4.6 is old school Opus?

I know you can point Claude code at Bedrock.. might be worth a play.

My bet is actually on GLM. Z.ai does amazing work and they will overcome Western models. IMO, faster than DS or Qwen. They have amazing team and very capable and smart leader.

> Thinking. I know this user well, they don't actually want me to find all errors.

> Thinking.. But I found a smoking gun of an error with this SPICE model, maybe I should inform the user.

> Thinking... Hm, but again, I know this human well, they likely don't care about this error. That's absolutely right - it's not an assistant's job to decide this, it's the user's.

Well if you want it go go off and try and validate the spice simulator and the kernel of the operating system that it's running on then that might be an approach to use.

And that somehow stops the other entity from building theirs anyway?

If Dario was altruistically trying to save us from the supposedly evil other party rather than pursuing oceans of cash, he’d have stayed in the nonprofit AI research space.

I'm talking about guardrails that prevent finding exploits, which is only peripherally related to writing secure code.

This benchmark is about finding security bugs, not writing secure code. I don't believe the models have guardrails that prevent writing safe code, but they're also not intelligent and have a bunch of insecure code in their training data, so they definitely write insecure code sometimes.

I don't know. I think it proves that if Google is baking guardrails into their models that prevent them from finding security bugs, they didn't bake those guardrails into Gemma 4, because it is very good at it. Maybe that means Google devs had a change of heart. Maybe it means something about Gemma 4 architecture is better for this task than Gemini 3.1 Pro. Gemini Flash 3.5 did OK though.

Anyway, I kinda think among US models only Fable really tries to block security work like this, based on my experience so far.

Wait, so..

This is interesting. The "reported to me like a colleague" part.

Is it just that anthropic gave Mythos even more of that Anthropic™ character, (incorrectly) radiating confidence?

Is that why people have been losing their minds over that thing? Is this just cheap social engineering?

I mean I bet it is also slightly more capable than opus, but that would all check out to me. Man.

Thanks for sharing I suppose.

> It's been really frustrating that neither Codex nor Opus can make targetted edits to Fable's code without screwing something subtle up.

Reminds me of the old adage: don't try to be too smart when writing code. Otherwise, dumber people - including your future self - will have trouble working with it.

Yes, in my project I made so much more progress in 3 days of Fable that is not comparable to how Opus is working.

Funny, I find Codex to still be better at Coding than Opus or Fable.

I A/B tested on a whole array of prompts between Codex and Fable, and Fable almost always found that Codex had produced a better plan and covered more edge cases than it did itself.

For every problem I gave the exact same prompt to both models, then I had each analyze the other's output. For roughly 80% of the prompts, Fable acknowledged that Codex's output was an improvement on its own, for 20% the converse situation occurred.

There was one egregious case where Fable suggested deploying code which would have resulted in a production bug, an edge-case which Codex identified and proposed a fix.

Note: this is all for optimized Rust code designed to be highly CPU and memory efficient.

I do prefer Anthropic's models for any tasks with front-end/design work needed. But I don't do much of that kind of work usually.

Maybe I was getting downgraded to Opus 4.8 but I saw nothing even close to resembling this behavior when using Fable.

> agency and misalignment are two sides of the same coin.

The free will coin?

I suspect they quantize them, reduce thinking budgets, batch more requests, or all of the above.

Can you think of many examples of a SaaS provider who regularly keeps old versions of a product around for customers to use?

A far more common scenario is that new versions are rolled out to everyone, without offering a choice, as soon as they're considered stable.

Older versions consume resources and require staff to spend time on operating and supporting them. Those resources could be used to run a newer version.

The tl;dr is the simple economics of any SaaS product.

If you want to be able to run old versions indefinitely and control the resources assigned to it, you need to self-host (an open model).

Also, with regard to tools, I originally ran a batch of several models in a full-featured agent (and whatever tools the agent provides), and they didn't perform better than the basic minimal harness with just read and grep. They chewed more tokens but didn't find more bugs. I'm currently doing tests with more advanced tools, like tree-sitter so the model can better understand execution and data flow and semgrep (which is almost cheating, since it finds bugs on its own, but worth a try since models can still be useful in helping rule out false positives and suggest mitigations). When I've got time for it, I'll also give them a full dev environment with compiler, debugger, and maybe fuzzer, and a loop that iterates through a security bug hunting checklist (since a single prompt and context window can't handle that much complexity at once).

At the time a GPT subscription didn't include Pro usage in the rolling limits. It was billed at API rates. Does it now?

If anyone wants to fund the other five cases (~$125), I'll run them. I find that an unrealistic cost, though...simply not useful data. I'm certainly not going to spend $23 per file to audit a project with hundreds or thousands of files. I don't know anyone who would.

Also note that it was $100 cap per model, and the next most expensive model was GPT 5.5 at a 20th the price per case, about ten bucks for the whole batch.

You are not wrong, but there's an asdymetry here: run adversarial self play and low-pass filter.

What makes you say that? I think they're better than replacement-level developers at making secure systems (I spent 20 years looking for vulnerabilities in human-written code as a full-time job).

I agree that AI psychosis is a real risk in vulnerable populations (GPT-4o in particular seemed borderline predatory towards those types of people, with its extreme sycophancy), and you should remain clear-eyed while using models. That said, I think exhibiting basic courtesy is still well within the safe-zone. I guess we'll see - I'll be sure to let you know if I end up going psychotic.

On the other hand I find it quite disturbing to see people be unpleasant or even downright cruel to something that, on a surface level, interacts with you like it’s a thinking, feeling being. Surely you should feel some aversion towards doing so?

I do get where you’re coming from though. I wish these systems had been trained to be clearly robotic and unfeeling.

They are told outside the test because if they can't find it when given hints then it's safe to assume it won't find it given no hints. It verifies to test, to an extent, much like running tests that should fail when given a set of inputs that should make it fail (you write an always failing test alongside your other tests, right?;)

I tried to prove quantization made models worse, but in my testing Qwen 3.6 27b performed statistically the same from 4 bits to 16, using the unsloth dynamic quantizations. Gemma 4 4-bit QAT seems to perform the same as the full-fat version, but quite a lot faster.

But, I have come to consider Gemma 4 31b the best model I can self-host, even though there are bigger models that'll fit on the Strix Halo. (I could also use much bigger MoE models on my desktop which has 64GB VRAM and 112GB system RAM.)

Some problems are very hard to solve with stupid code. This can easily be the case (computational geometry)

the primary difference i noticed is that fable didnt try to check in every minute

to an extent that might have done it, but i had been playkng around ahead of time trying to reverse engineer my ray bans case so i can make my own plastic insert, and fable to opus' work from mostly broken to mostly done, and then when fable went away, opus broke it again

I've used them back to back as well. Codex is good at specific tasks; it doesn't try to go big, it does what it's told provided the task is relatively procedural. If Codex can make progress on a task, why would I give it to Fable?

Fable fumbled the one simple task that I gave it too. I gave it multiple very hard open-ended tasks (effectively math tasks) involving research code and it crushed them. It's the first model I've seen that can do that. The current Codex will never produce the type of code Fable gave me no matter how many times I run the same problem at it, because it won't stop trying naive rubbish. And if I tell Codex to try to improve the code, it can't figure out why trying the same classical tricks isn't making it work better, regardless of what I tell it. Opus is marginally better because it can at least recognize some subtleties over time, but still disappointing because it has no idea how to deal with them.

Most programmers want precision instruments for their workflow. That's fine, use the right tool for the job. In my line of work, I need crazy solutions because the obvious stuff doesn't work. That's where Fable shined for me.

To be fair, labs silently nerf models all the time.

Fable's probably objectively better at full power. I mean, I definitely felt the same difference in competency between Fable and current Opus. But Opus itself has definitely been nerfed, and Fable, even if it comes back the public forever (probably won't), will get nerfed.

For reference:

if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it

No, it’s just a fundamentally much better model. Going back to Opus feels like the model has been lobotomized. It makes much more frequent errors, especially of the “I claimed I tested x y and z, but actually only kinda half heartedly tested x, and assumed I understood what was wrong” variety.

It very much depends on the task. What were you trying it on?

There's also lowering the number of experts you run in MoE models.

Mostly right. However there's an extra assumption I didn't explicitly state:

Almost all existing real world software is full of holes and security flaws. Mythos is better than humans at uncovering many of them; especially because its time is a lot cheaper than that of the top tier human experts (and even of mid-and low-tier human experts).

Especially when these systems are written in notoriously unreliably languages like C.

I don't think Mythos is especially good at writing systems that are free of security problems. Essentially the only way we know is by proving your software correct.

In principle, you can even prove C correct, but in practice you'll want to write your system from the ground up to be proven correct instead of adding that property after the fact; and for that you'll most likely also want to pick a language that supports this better.

See https://en.wikipedia.org/wiki/SeL4 for a noteworthy example.

Agreed. In the right hands, they can perform magic.

See https://news.ycombinator.com/item?id=48640533 for some further elaboration.

These models are definitely a lot better than your run of the mill human developer at finding security flaws in existing systems. I'm agnostic at how good they are at actually making a secure system. Probably better, too, for two reasons:

- humans are really terrible

- the model probably has an easier time picking up special purpose tools you can use to write proven secure systems

I don't think Mythos can write secure C code, either. Practically no one can. (At least not directly. See how seL4 is officially written in C; but they didn't just set out to carefully write secure C code directly; C just happens to be an intermediate language they use.)

I mean I agree with this as well, the people who yell and swear at LLMs are just as bad as the people who chit-chat with them like they're friends. It's all very unsettling because it's prepatory for psychological manipulation at unprecedented scale. Targeted advertising on steroids.

Personally, I think having to constantly mode-switch between "courtesy / collegial" and "terse / cold" is a bit exhausting and a little risky. What if I get tired and accidentally treat a human co-worker like a computer? Risk with no upside. Might as well just stay in "courtesy / collegial" mode for all of my conversations, regardless of whether I'm talking to a robot or human.

> I have come to consider Gemma 4 31b the best model I can self-host

I'm confused. Your own results show that Gemma 4 26B A4B and Qwen3.6-27B did better in these tests?

I really like Gemma 4 31B, especially with how exceptionally good its MTP drafter is, but it is absurdly weak at tool calling and instruction following in my testing, and its smaller siblings are even worse at this. If the system prompt says to do something, Gemma 4 31B will very often ignore that entirely. It will also make fewer tool calls than were needed to solve a problem, so then it fails. The Qwen3.6 series is much, much more reliable for carrying out instructions and doing agentic tasks in my testing, although they can get stuck in loops.

There is a lot of potential in the Gemma 4 series, but I think Google needs to release a Gemma 4.1 update to polish the rough edges. Unfortunately, if Gemma 3's lifecycle is any indication, Google won't release a true revision of the Gemma 4 models, even if they release a bunch of specialized research models based on Gemma 4 over the next year.

In my experience "free will", like "consciousness" and "common sense", is not so much a concept with a universally agreed definition as it is a cognitive stop sign or an applause light, meaning different things to everyone who uses the term.

Do I have free will, or am I bounded by the laws of physics?

Even if you think my soul is completely independent of my body, there are theologians who argue that God being omniscient means that who goes to heaven and hell is predetermined before birth and therefore no action you take will ever change the afterlife you go to, and that to think God isn't omniscient would be blasphemy; do they think I have free will?

And then there's Thelma with "Do what thou wilt shall be the whole of the Law", which can be understood in terms of (amongst other things) "Don't let peer pressure manipulate you into thinking you want other things than you really want", though this is of course a simplification much as the omniscient example above: https://en.wikipedia.org/wiki/True_Will

We can't really know in what manner Mythos was used to find these bugs, right?

I have ~100$/mo sub and I have Pro in chat app and Extra High in Codex for GPT-5.5

I think on sub tokens might be 100 times cheaper.

The quota is also generous in my opinion. I can vibecode a lot most days of the week and not run out.

> Can you think of many examples of a SaaS provider who regularly keeps old versions of a product around for customers to use?

Sure. Blender and Ubuntu offer long-lived old versions of their software that get regular fixes.

No, the purpose was to create a (automated) test set in the first place. The author builds an LLM judge which can score the LLMs participating during test-time. That would be why the author used the strongest model (Opus 4,7 at the time) as the judge.

I remember a time where a product didn't suddenly get worse while you were blinking.

That was a nice time. Let us get back to that time. Use open weights models. Own stuff.

Ah thanks - I couldn't remember the original version.

For reference: it's called Kernighan's Law, and can be found in the Second Edition of "The Elements of Programming Style", page 10 [1].

The original phrasing is:

> Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?

[1] https://archive.org/details/the-elements-of-programming-styl...

Wait but that has been the exact word-for-word complaint when comparing sonnet to opus

Or opus to opus

Or really any new thing to old thing

Of all of the concepts like "consciousness" and "agency", "free will" is probably the least useful and poorly defined.

It's a hand-me-down from Western beliefs about morality and individuality - including Thelema and Christianity.

So there's a lot of starting from the concept and working back to assumed conclusions.

Generally humans do not have free will, do have very limited political, economic, and psychological agency, usually selected from a small number of competing rule sets, and are also far more easily influenced than they suspect.

Culture is more like a cellular automaton or diffusion system. Occasionally a transformation ripples out from an individual cell, often for fairly random reasons, but the big patterns are emergent, and every so often the soup shakes itself up and settles into a new arrangement.

IMO LLMs are the most recent proto-version of that, running on a different substrate.

Right. I noted that in the post. Some of the information out of Anthropic indicates dumb loops, sometimes, but some hint at a more sophisticated harness and process for some of the Mythos bug hunts. But, nothing specific.

I've been doing more benchmarks with additional tools, with no silver bullet revealing itself thus far.

Neither Blender nor Ubuntu are SaaS. You're just confirming my point: if you want to run old versions of software, you need to host it yourself.

But GPT 5.5 on extra high is not Pro. When I looked into it, Pro was not available for agentic use via any rolling limits plan. But, I'll look again into whether there's some reasonable way to complete the test for GPT Pro.

It seems I was not able to either, and I trusted google AI snippet. Thanks

That was before SaaS became a thing. Products didn't degrade over time because they couldn't easily reach out to your machine and remotely overwrite bytes on the CD-ROM the product came on.

You hear the same canard every time Anthropic releases a new model or version. I'm not convinced they're objective anecdotes. I wonder if it's simply the new model, while marginally better, has a different style and people find that new/refreshing. That is what makes it feel so much better than the previous release.

When the agent is becoming more accurate and thorough what would you expect to be reported?

Ah, right. Sorry, my mistake. I have access to it in chat but not in Codex.

Oh I am sure that it became somewhat more accurate, and with that, the labeling there is in fact technically correct. It just does not work as an explainer for the doomsday-ish hype that model has induced in a lot of people's brains.

The user here is right in what they said but wrong in why they said it, essentially.

An analogy I keep coming back to with the current progress in LLMs is the progress in the 90s of 3D game engines.

Every upgrade made what came before it appear awful in comparison, to such an extent that every upgrade was called "photorealistic" and people kept forgetting that they'd been using that description for the previous engines that they were now dismissing.

https://archive.org/details/nextgen-issue-26

That’s a rather bad faith framing, I think. Who are you to judge why I said something?

A person with the exact kind of pattern matching brain disorder this tech has been modeled after.

I do make mistakes though. Please check results.