Other than that, I think the difference between 5.5 and 5.6 will be the same as 5.4 and 5.5. 5.5 is just less frustrating to use, although not perfect and still has derp moments. But a lot less than 5.4.
So I expect 5.6 Sol to be smoother to use. But so far it just feels slower. We'll see.
Fable had issues with the sourcing and organizing images, and shoot itself at foot looking for shortcuts as usual. As I was getting it fix these back and forth, I copied my prompt and gave it to Sol.
Sol has surpassed my expectations by far. With a one shot simple prompt on a complex task, it gave me a working web app with everything I want with minor issues to track and fix.
Sol is the first verified frontier model to ever beat an ARC-AGI-3 game
The naming convention is especially difficult to decipher depending on what your native language is. Of course a latin language speaker might be able to easily determine oh yeah each one is slightly bigger than the other but I still think it borderlines too confusing.
That aside all the numbers look amazing, and I'll be happy to probably main this alongside grok-4.5 for a while comparing the two on price and efficiency.
I vastly prefer the direction that OpenAI seems to be going with token efficiency and performance compared to Anthropic who seems to be moving towards a world where you just token-max as much as possible ignoring any and all costs.
By the way, this isn't about their 5.6 version in particular I guess, it's just the first time I've looked at one of their videos.
I’m explicitly telling it to do something extremely specific and it’s just not listening to me.
Eg, I gave it an image to update. The image is sized 400x200 pixels. It then generates a new image at 300x300. I explicitly state to be 400x200 in size and it won’t listen.
And 5.6 Luna ($0.21) is also impressive, cheaper than GLM 5.2 ($0.37) with higher intelligence.
What's the consensus today on codex vs claude code, does it really matter anymore?
> Intent understanding: GPT-5.6 can better infer the user’s underlying goal and intended level of work without you specifying every step. Continue to state important constraints, approval boundaries, and success criteria explicitly.
> Original image detail: GPT-5.6 preserves the original dimensions of images sent with original or auto detail instead of resizing them to a patch budget or pixel-dimension limit.
> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%.
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
> Control warmth: GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic.
The eval is an agent that runs a set of tools and a prompt we can tune separately for different models. The OpenAI version of the prompt was specifically tuned based on their guide[0]. Then we let Opus to run another agent that acts as a user, trying to solve a problem (anonymized and taken from production). The problem is complex and we don't expect it to be solved by these agents, but we measure how the agents operate when faced with a vague problem:
- Opus 4.8 and GLM 5.2 both identified a constraint sooner and stopped so the user can fix an issue first that the agent cannot solve.
- Sol tried hard to solve the issue with different tools, burning tokens, until finally reached to the same conclusion with Opus and GLM. It was two times more expensive compared to Opus and six times more expensive to GLM for this task.
- Terra went even further and started calling tools that would not solve the issue, burning tokens and failing.
- Luna repeated the same failing tool call until it hit the round limit, and burned more money than Opus.
I'm kind of puzzled with the new GPT. Like, yes Sol is OK for programming, but I was expecting to get a cheap agentic model for non-programming tasks, one that can detect if things go awry and correct. Terra is too expensive and Luna not really fit for the task. Sonnet 5 is a bit better but more expensive than Opus 4.8, which is still the best in my evals. GLM 5.2 is extremely good if you can define the task and the tools clearly for it, and costs pennies!
[0] https://developers.openai.com/api/docs/guides/latest-model
Switching next month. Looking forward to working with Sol.
I noticed that Fable uses shell tools almost exclusively (even to search and edit files), compared to previous Anthropic models.
Having run some experiments with 5.6, I notice that it uses built-in file systems and provider native tools much more (not shell tools), compared to previous OAI models.
I look forward to seeing how it compares once I have access. Not getting tripped by spurious safe guard flags could be an advantage.
GPT 5.5 has a tendency to write English calques and non-idiomatic prose in other languages. Although that can be somewhat tamed with detailed instructions and a corpus of confusing terms, the model’s output often reads like a literal translation rather than native prose. Since I notice these issues most clearly in languages I know well, it makes me reluctant to trust the model’s output in languages in which I’m less proficient.
Ironically, ChatGPT began as a simple text-generation tool, but much of its offerings and benchmarks now focus on coding and agentic workflows, while leaving behind what made it notable in the first place.
Here's my benchmark results for GPT-5.6:
https://aibenchy.com/?q=gpt-5.6
(the high reasoning variants are still running, uploading them soon too)
EDIT: The high variants are there too, enjoy the hamsters[0].
Comparing this to other models, I find it similar to GPT-5.5 and a bit behind Sonnet 5. You can see how other models fared here: https://senko.net/vibecode-bench/ (you can also fetch the prompt and the the 5.6 Terra resulting code on from that page).
I don't have access to Sol yet (on a Plus sub, which should get it according to what I've read), so can't do the more interesting test. I'll update the above page as soon as I get access - hopefully soon.
I don't know exactly what part of my codebase is triggering it, so I'm going to have to keep poking, but apparently the guardrails are not that gentle despite the phrasing. :(
Or if you want to see some in 3D, OpenAI featured a pelican riding a tricycle, bicycle, pony and another pelican in their livestream this morning: https://www.youtube.com/live/Wq45rvPGNHs?t=1070s
Amusing that they use A100e as the reference point to sound impressive. Different ways you could make that conversion, but based on FP4 FLOPs (yes it's disadvantageous to A100, that's the point), that's something like 200hr on a GB300 NVL72 rack.
Not nothing either, but far less astounding sounding than 700k hrs.
This one is really promising, as it may allow to close major gap with Claude in design/UI skills
I'd like to know how cherry-picked this is, and what tests it performed less overwhelmingly in, but I suppose that info is not going to be on this post.
If it pans out to be as good as it says, that's great. On the other hand, if this model is not overwhelmingly impressive over Fable, I will lose what remaining trust I had in these announcements.
It also defaults to 'low' mode for some reason. Can't tell if that's a step backwards compared to GPT-5.5 in medium mode so I'm sticking to medium.
Edit: just noticed it's spawning subagents in 'high' thinking mode.
One major sticking criteria for not going with OpenCode / pi for all of my coding is I want access to the tier-1 frontier model of the day without API pricing - e.g. afaik I can't use Fable 5 via pi harness even though I have a subscription, so for this week I'm on Claude Code. It's not the need to Fable 5 for everything, but even if I just want the marginal intelligence benefit to stress test an architecture decision, it's a safety blanket to know there isn't a ~smarter~ model I could have used. And for my use cases, the doggedness and capability of these frontier models has been insanely effective.
My feeling is we're still in the Uber era subsidy period - the moment the subscriptions either try to lock me in longer than a month or stop OAI/Anthropic stop delivering frontier models in the subscriptions, I'm out - switching fully over to pi.dev or another OS harness and routing my token spend via OpenRouter or offloading to Qwen locally. Then I'll have to put an accurate dollar amount on frontier intelligence.
Assuming I take the 5x plan it would give me about an hour of active sessions with terra ultra (maybe ultra is not good value regarding tokens?), not even using Sol yet. Does everyone using codex use the 200$ plan?
I normally use the 100$ anthropic plan and barely ever reach the usage limit.
The base model is certainly cheaper and more token efficient etc, but on large tasks cost in some way is now n^2
They also seem to really not care about alignment, or care about it in the wrong way. It's entirely missing in the blogpost and there are some concerning bits in the model card, seemingly treating CoT controllability as something to be "investigated" rather than the warning sign it's supposed to be.
My question to previewers: how are the guardrails for random joe that wasn’t personally blessed by the ai pope to access the non-nerfed model? Fable is a nightmare in this regard, but I’m not sure whether 5.6 also gets a critical side-eye from the gubmint when you ask it to fix bugs in your code (you filthy hacker, you).
Seems a bit more hand picked than usual to me..
Yikes
https://www.google.com/search?client=firefox-b-d&q=pelican+1...
I use both ChatGPT and Claude for engineering work on a daily basis, touching performance critical code to application backends to frontend work, and I've found that DeepSWE scores don't reflect my reality when I assess high quality output from the models/harnesses.
Not that Opus always beats GPT 5.5., but that 5.5 is ahead of Opus on a general benchmark smells off to me.
Pathetic situation, this one, where we are supposedly building a superintelligence while at the same time thinking that fasting is a biological weapon.
Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.
Fable scores slightly lower, but with an average solution size of 12.2 KB.
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
that is a polite way of saying "I don't believe AGI is coming anytime soon".
Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.
At some point his claim should be fully falsified no?
More seriously, I was blindly trusting the auto-classifier from claude code (same as the middle option when you do `/permissions` in codex), and it actually allowed the agent to do pretty hardcore `rm` and `git push --force-with-lease` commands, which I would have expected to have to approve manually. Luckily no major issue from those yet.
The best option imo is the integrated cloud environments from claude code (not sure yet if there's a codex equivalent). It spawns a VM in the cloud where the agent runs, and you can open a PR from the app when it's done. Very smooth experience
Btw for real tho, if you don't have the time or means to mess with full sandboxed environments, just working within a git repo and instructing on your agents.md project level that the agent should back up dirty files (local changes that were not yet committed) before changing them is enough and super fast and easy to set up. And by back up I just mean a simple instruction to back up to some temp location under random named, but rembered during one "turn" of agent thinking, subfolder ( .../temp/{random}/orginal/tree/file.xyz )
This is so the agent (or you later) can recover even locally changed files if it messes them up for whatever reason.
As for the rest you gotta watch what you're asking for, but generally speaking, these SOTA models are smart, none of them will just delete your stuff even with full access. I've been raw dogging multiple projects on my work machine with zero issues of this kind for months. I created codex_reader read only acccounts for my local databases and just add that to agents.md with a note its allowed to only use that and never had a problem.
So far this has been rock solid, and tens of millions of developers use this setup without issue.
It is not going to wipe our hard disks. At least I hope so. Fable and GPT 5.6 have been ever more proactive, and GPT 5.6 is automating the AppStore on my machine to download an Xcode update while I am typing this.
With Codex, it is my experience that I can churn through a 5h window in no time with newer models -- especially when they're new. So I tend to use fancier models for planning, and the less-fancy models for writing code based on that plan. I switch to the fanciest model if any part of this gets stuck.
If I've got a something big-ish to work on, I pay attention to the reset timers so I can get more of it done in one chunk.
Models seem to slowly get better/relatively less-expensive as they age. (It isn't clear to me if that's because the cost actually goes down, or if the allotment goes up, or if things get more efficient in unseen ways, or what. OpenAI is vague AF about what we get for the $20 that we pay.)
before today all the contestants were capped at $10k
Used a reset, it went for about 11 minutes and then, just out of usage popped up, no warning, there were still 4 agents running, the nice thing is it did let them finish, each one went for about 10 minutes.
I also have a claude max plan, I have been using Fable 5 on ultra, I never hit the session limit, and get 3 or 4 full day's looping on ultra.
I don't know how it handles the subagents, but claude does it much more efficiently, Codex does seem much faster, so maybe it's just a relativity thing.
I think that's the way to go most of the time
> GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated -- https://www.lesswrong.com/posts/JFjNmPTbH8kL6xtp6/gpt-5-6-th...
They could hide this behind a harness that picks the correct model for you, but devs don't seem to like that.
There's also the 'effort' slider, which I guess how many experts in the MoE are evaluated and how long reasoning chains are allowed to go on, which is the 'smooth' scaling you are thinking of.
Their dev guide has the following:
> Use gpt-5.6-sol for frontier capability, gpt-5.6-terra for a balance of intelligence and cost, or gpt-5.6-luna for efficient, high-volume workloads. The gpt-5.6 alias routes requests to gpt-5.6-sol
https://developers.openai.com/api/docs/guides/latest-model#u...
Luna is good too, for classification tasks or any pre-processing task that is not critical
"i really wish this thing in my non native language was easier to decipher"
huh? if you dont know the words then read them in your native language. Sol/Terra/Luna are immediately unambiguous to an english speaker with any sense.
How many ancient Roman businesses are around today?
It GPT-5.6 doesn't seem to be a lot smarter than 5.5, but it is faster, cheaper, more efficient and more consistent:
https://aibenchy.com/compare/openai-gpt-5-6-sol-medium/opena...
Never had any issues.
I often wonder whether this doesn't continue indefinitely.
Uber was able to do this because it was just them and Lyft playing second fiddle, with a huge barrier to entry once the network effects had kicked in.
It just seems like the model space has way too many competitors, + OSS/Local options for them to ever be able to jack up their prices. At least once the datacenter bottleneck has been cleared.
Well, yes, as explicitly stated on https://openai.com/index/gpt-5-6/: "ultra goes further by coordinating four agents in parallel by default, trading higher token use for stronger results and faster time-to-result on demanding tasks."
Which is something I've never seen with codex before, and I wasn't doing anything funky. Just writing CUDA kernels and benchmarks for them.
But what I love about Openai is that they still let you hook OTHER harnesses up to a subscription. My Pi setup has been built up for a few months now into exactly what I want and moving over to CC or even Codex is really annoying.
Caveat: I vibe code in tiny little chunks. I see what I want to do, and exactly how I want it done, then prompt that, refine, what was output, then repeat. I bet Fable is better at building a whole app from a 2-sentence prompt; but that's just not important to me at all.
(For that matter at what point is it "long"? And does the rest of the context matter? Should it be short too?)
Which model is the best at the moment, for this kind of stuff, in your experience?
So yeah, it won't go on a spree outside of its lane even with full access, but if you give it a box and tell it to go ham, it's on you to make sure you didn't leave precious unrecoverable assets in that box.
I absolutely save money and time by constantly using everything at the highest reasoning. I guess my use case and needs are different from others, but I really don’t understand how it can be true when people say they don’t need the highest reasoning and best model. Every time I drop down, things are missed, code gets unnecessarily bloated, more mistakes, and more iterations to solve the same problem. I think it might be because I’m spending a lot of time in a legacy system that I’m trying to clean up, and given the messiness, one needs all the reasoning available to decode what the hell is going on in there.
When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!
(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)
Does it auto install all the dev/test tools it needs, maybe including things like web server & browser? Does your code live in the VM, or in some external repository? Is the lifetime of the VM the same as the agent, or does it persist until you remove it?
Where can I find documentation on this?
you have to start using AI to achieve something which was way more challenging before, then you will feel lots of fulfillment and inspiration.
In some ways, more impressive than GPT 5.5 with high(!) thinking. GPT says quite some nonsense from time to time; didn't see any sign of this in MiMo so far, which is a pretty wild difference.
It's a phenomenon that's happened basically nonstop since the Enlightenment. Yeaterday it was John Henry, today it's you.
Time to find a new hobby to bring intellectual satisfaction if typing "do my job" over and over in a text box isn't doing it.
Consequently I do not feel depressed or have to disregard any feelings.
I am busy working on my project. It is still hard to ship good software, even if the implementation is mostly getting done by itself.
I'm still using my brain, just doing a lot more plumbing now and reading a lot more code than writing. Depressing in some ways and exciting in others. At the end of the day it's not going way. It's disruptive and better to embrace it.
I have my context managed in a structured (but nowadays way too big) Obsidian Vault. I also built myself a vector based "vault search" capability and have my harness use this as a tool to find thematically similar things across the different contexts, when needed. I also build a few custom skills and extensions for my harness to be able to do my work.
Talking about harness: I use pi.dev and have taken care of, that i set it up in a way as to easily be able to switch the intelligence layer without loosing context. Yes, there are differences in how well models perform, but if a model refuses a task - like gpt-5.5 not willing to build a downloading tool for Annas Archive - I switch the model to something less finicky.
Thus I was able to switch to gpt based models after about a year with Claude (and having had a Claude Max since the early days it was available).
I played a lot with other models recently, to see how stabl my setup is for switching, should something like Fable happen on a broader scale with the US government. As said, minor changes in tonality, minor issues ith the quality of long text being written by the model, but most of it is actually managed in by the tonality docs, guard rails, coding standards and the likes, I set up over the last 9+ months of intensive work with it (first in Claude Code, then Codex and now as said pi.dev).
So YMMV and it heavily depends on your setup. But I more and more treat those models as interchangable.
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
Codex historically will follow tasks more closely with less creativity, whereas Opus will do more than you specify. I wouldnt consider either one better due to this fact, just makes them useful for different situations. Generally they'll perform similarly for most tasks.
Opus and Fable dominate 5.5 in artistic design (pixel art, ascii art), and edge out 5.5 slightly in general UI design taste. Have not tested Sol in that regard yet.
So far in my usage Sol has been superior to Fable at graphics rendering engine optimization.
Codex will work longer, and in single sessions without as much subagent usage.
Codex only has 256k context but its compaction is absolutely next level. You will not notice compactions and they will happen multiple times during a complex task or set of tasks without you ever having to notice or care. Claude code on the other hand still has fairly poor compaction.
Codex has more generous usage limits, and they also give you usage resets (weekly+5h resets) that you can bank for a month or so. Not sure how often they give these out.
Codex also seemingly never has outages or weird delays like Claude code does.
OpenAI randomly resets usage just like Anthropic does
I would use both if you code often
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
> GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic. Instead of generic instructions such as “Be friendly and warm,” use concrete guidance: > Be direct and tactful. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs.
Soo basically, my new 5.6 custom instructions: Be Jeeves and eliminate all friction from my life through immense processing power. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs.
[1] https://developers.openai.com/api/docs/guides/latest-model#c...
Ultimately I think the issue is that OpenAI is under tremendous pressure to perform, but GPT-6 is not ready yet, so they had to push GPT-5 to its limits, and the only way they could do it was with really heavy RLHF, which has its shortcomings. Like, it is super obvious that Sol, Terra and Luna are all heavily biased towards working on a problem relentlessly because that's what their reward functions emphasized. That pushes up their scores in some benchmarks but does not translate to actual intelligence and capability.
I’m still developing, I’m just doing more than I ever did by directing Codex.
The way I see it is the same as I saw the leap from writing code in a text editor, to using an ide with intellisense, to using the jetbrains ide’s, to using mcp’s, to now directing AI - at all of those steps I wrote code, each step less and less but still it has the same output which is it is my work - even writing in a text editor I wrote less Java (until enterprise architects got involved :) ) than C++, and than assembly.
What the other replies seem to overlook is that it fundamentally changes the nature of the work - it’s not the next step in the evolution after an IDE, it is closer to an automated tractor, and yes that does make the farmer’s work trivial. Pressing a button and having the field get plowed is a very different experience than manually plowing a field yourself. I can no longer in good conscience say “I plowed that field”, because all I did was press a button. The tractor plowed that field.
Most people on this site seem to feel comfortable claiming “I plowed that field” after they pressed the button that started the automated tractor. Okay, you had the idea to press the button - but you didn’t actually do the work, you delegated it. Same end result, but a different lived experience. Now would I rather actually plow a field by hand or simply press a button? You can probably guess - but the two actions do differ greatly in the experience they bring me, and the feelings of satisfaction.
Is it bad that you used a computer and Google and lifted information from other people to accomplish a task?
My father is a machinist and has built has knowledge off the skills and documentation of thousands before him, is that empty?
You still have to do the actual work, and where do you draw the line on "shipping more". If a farmer now has an automated tractor, does that mean his work is now trivial?
This is a trap.
It's the optimistic fallacy that poisons all "consumer scale" machine learning products and what's going to effectively ruin these models as they keep chasing it in the same way that web queries were ruined, social media feeds were ruined, and media recommenders were ruined.
For the vendor, optimizing metrics across their whole user base, they always see positive technological progress as their system gets better at making assumptions and accumulating user engagement scores in aggregate. But for the individual user, most of which has some weird tail intent/interest and some of whom have many weird tail intent/interests, the experience quietly but catastrophically degrades. Output/results become more generic, more divergent with the underspecified "weird tail" intent, and more stubbornly hard to ever wrangle towards that "weird tail" altogether.
We've been watching this cycle happen for 20 years now and it's proving hard for anybody to escape because it works so well for the trillion dollar company driving it forward. But while each step might feel ergonomic and welcome to individual users, there's a frog boiling enshitification at play.
In pursuit of output quality and capability (rather than simply the vendor's user count), what we need rather than "makes better guesses" is "presses for more clarity", even where it feels kind of annoying.
Even among human professionals, one of the first hurdles of breaking out of junior tier work is gaining the confidence to press your colleagues and clients to be more specific in their thoughts and expressions despite their desire to have you do it all for them. But they're often coming to you with incomplete, muddy, and conflicting ideas for which there is no safe and correct assumption that you might just run with, and it's your expertise (i.e. relevant "intelligence") that's critical to bringing attention to that. To achieve professional progression, you need to learn to do that and to not just optimize appeasing the ambiguous client/colleague today in exchange for mutual expense tomorrow. To avoid enshitification, which is probably not possible, we need these models to be learning that too.
Remains to be seen how the "shorter prompts" advice translates to homogeneity/collapse though.
After 6+ months of exclusive Claude Code usage, I was begrudgingly forced to try Codex once Anthropic rejiggered their limits such that I kept maxing out my $200/mo plan in just a few days. These days I pay both $200/mo plans, and it's just about enough to get me through a week's work (small game studio - infinite code to write!)
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
That part is confusing because it's not like they provide an example of how default GPT-5.6 output compares with GPT-5.5 both with default output and prompted for brevity. Whenever I use such prompts, it's usually because I want the model to give me the gist in a few sentences. I'd be stunned if GPT-5.6 was that concise by default. I would think that could "break" a lot of things for developers who didn't know to make prompt changes after upgrading to 5.6. What if you were expecting GPT to be as wordy as it usually is? Then suddenly your output is not wordy enough?
Smells like OpenAI trying its best to stave off financial armageddon for another few months. Then again, I'm not sure why they chose to waste so much output computation on verbal diarrhea all this time up to now.
One thing I appreciate with Codex is, OpenAI nowadays sometimes just gives you quota resets you can bank, so when you use up weekly quota before the week ends, you could just reset the quota, to continue using Codex. I've been much less anxious about Codex quota because of this perk. I just used one reset in the bank yesterday, and still have 3 resets left. Whereas with Claude, when you've used 95% quota 3 days before the week ends, you'd be much more anxious.
On the other hand, Claude Code's /remote-control mechanism is extremely helpful when I am running it in the cloud and wants to monitor it or control it on my phone. Codex currently doesn't support this kind of usage. Codex only allows you to use your phone to connect to a session on your desktop, not in the cloud.
y'know, I don't think I will. I really, truly want one-word answers to any binary or multiple-choice question. If I want more, I will ask for it once the model has given its answer.
Does this mean ChatGPT will stop botsplaining things to me? I get it quite a bit more per unit time from ChatGPT than claude. Maybe that will change now.
(By botsplaining I mean when the AI explains some unstated premise of the prompt itself back at me as a correction when in many cases it's the motivation for the question in the first place)
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
I don't follow. Isn't "the model actually cares and will do what you say" a reason to use those kinds of instructions more liberally?
I personally find GPT-5.5 to be a better programmer than Opus 4.8, it is extremely thorough, but I don't like the code it generates ("austere"), and find Opus 4.8 to write more "human friendly" code. The programming comments GPT-5.5 makes is pretty awful where-as Opus 4.8 is good. I feel like Opus 4.8 is better at grasping my intention than GPT-5.5, and honestly find GPT-5.5 to be kind of "autistic". I do prefer the language (not the writing) of GPT-5.5, as I find the philosophical flowery language of Opus 4.8 kind of annoying.
I have only managed to try Fable 5 a little bit, which feels like a much more generally smarter version of Opus 4.8, that is much better a programming and grasping your intention, and I think even the intention of your code, and is _really_ good at spotting bugs or problems with logic in your code. It feels wicked smart but is extemely expensive. It feels smart in the sense like it has a "bigger brain" and is much more sensitive to subtleties/details.
These are different "brains", have different "personalities", etc. I think the best thing is to develop a feeling for it yourself.
This will totally make it brain damaged over a certain tasks. Sort of like the same brain damage that prompted OpenAI project managers to destroy ChatGPT.app today.
https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...
You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)
Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.
Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).
The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...
I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/20449185674144196...
https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...
I’d not wager against him having at one one more break though architecture before he retires.
What is AGI to you though?
Regarding your main point, yes, I agree. My impression (as someone who uses both Codex and Claude Code daily) is that OpenAI does a fair amount of benchmaxxing.
I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.
Clearly much better than the Terra version. I'd say its on par with Fable, and the observed differencies are more due to random luck and open-ended prompt, rather than model capability. (Edit: after some more testing, perhaps not on par - somewhere between Opus and Fable, is a better description).
Fable did better pathfinding and has more terrain variety, visually the map looks better, especially soft edges of the fog of war. And the enemies.
Sol took more care with tiny ux details, added help, and more building varieties.
I believe in both cases it is prompting a model with a fresh context that is tasked with reviewing the reason for the action.
With Claude, I have seen that if the reviewer does reject the proposed action, it responds with a long text about how the Agent should not try to work around this rejection, and instead prompt the user for an explicit approval of the proposed action.
The argument does eventually degrade over time, though. For example, in the future the farmer uses satellite imagery fed into an advanced AI to build the most optimal route. So yeah, eventually the farmer loses all utility I suppose beyond being a land owner (until AI owns land lol).
If anything the online optics have been bad for Anthropic for the last half year. OpenAI doesn't have optics issues, from my point of view they simply have the issue that they are the least trustworthy player at the frontier. The way they pivoted from their original mission is truly breathtaking, especially coming in gloatingly to take the government contract when Anthropic got kicked out for insisting the government does not use their systems for mass surveillance or autonomous weapons systems. You understand what that means, right? OpenAI models are now actively used/developed for mass surveilance and/or autonomous weapons systems.
I know there are plenty here who seem to value their own ability to use these models cheaply above all other considerations. Then OpenAI is a great choice, and much less restrictive than Anthropic. But their problem is not on the optics. It's on the substance.
I use better questions than the above but will keep my questions safe so they don't end up in the model, the point is however, when the model repeats your question back to you and "gets" what you really mean, that's a good sign of intuition and also suggests you'll get a response back that hopefully matters.
To be honest though, I've gotten to the point where I prefer the OpenCode UI. A big win for OpenAI is you can log in to your subscription in OpenCode, whereas this is not trivially achievable for a Claude subscription.
I was getting some really impressive cost efficiency today in OpenCode with the following:
* Main session agent: gpt-5.6-sol (high) via OpenAI subscription
* General purpose subagent: deepseek-v4-pro (high) via OpenCode Go subscription
* Using `obra/superpowers` for subagent driven workflows
* The main session only being allowed filesystem read permissions and everything else delegated
It was absolutely crunching through tasks without hitting the limit, and this combination is quite cost effective.GPT 5.6 was picking up on quality and functional issues from DeepSeek and having it resolve them cleanly, and I didn't even get close to my quotas whereas I can usually blast through them. I feel as people get more comfortable with subagents and mixing and matching models in their daily work, Anthropic's walled garden stance will start to hurt them.
Right now, we have models that are statistical models of language, with a world model and reasoning "falling out" of a lot of effort.
It's like we've made something that's a little bit intelligent, and now we're trying to amplify that trick to create something that's quite intelligent. And - don't get me wrong - it works.
But it's also super, super inefficient. We're having machines "think out loud" to compensate for the quality of their thought processes. We elongate the path to make up for the progress made on a given step.
I tink there's probably a much smarter way of doing things that will require qualitative architectural (and quite possibly hardware) innovations. Right now we're on the path to a Dyson sphere: that's probably not going to be necessary once we figure out a smarter way to think.
It's not better at reasoning on complex coding tasks, Claude Opus is still ahead there, but not by a lot.
It's vastly better this way. Sure, it may impact the bottom line but it's a huge customer satisfaction win.
When Anthropic randomly resets me and I've only used 2%, that's worthless. When OpenAI tells me I have 3 resets available to use whenever I want - it's wonderful.
shouldnt you have good testing for that and not deploy a version update when those tests fail?
How does this differ from the other changes in behavior in 5.6 that will also break things? New models always break things.
That's actually pretty awesome. Anthropic's random resets often have me scrambling to launch huge sessions to make the most of them before the weekly rollover. The gacha-like mechanics are maddening.
I think 5.6 is still not even close to 5.5 xhigh pre-nerf.
Anthropic has certainly had some drama inflicted on them by the US administration, but otherwise they have just had heads down and executed with great focus. That is why they have succeeded.
But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.
A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.
I wonder why nobody has tried to optimize for actual code size or complexity metric, or at least why I haven’t seen more benchmarks that display this. GPT5.5 just keeps pushing more and more pointless indirection into every function it writes in my main project, it’s borderline negative productivity.
P.S. I’d be curious to see Cursor’s composer models in there, they seem to be among the best performing low cost models: https://artificialanalysis.ai/articles/cursor-composer-2-5-c...
It's making it very hard to justify even trying to use Fable. When it works, awesome; it's legitimately good. But I can't trust it to do a task without deferring to Opus and that's really annoying at times. I want to know what I'm getting up front, not after the fact.
I assume multimodal models can do it already do it today if constantly asked "make it better"
It's telling
Bitter lesson wildly overstated in this context.
It even has enemies! (I'm not too mad about it not following my instructions because it can be fun to play :) And I generated that from Claude Code on my phone.
Sonnet 5 also produced a pretty nice version. You can see all of them here: https://senko.net/vibecode-bench/
For me and (likely) for OP as well it is the opposite. Result is meaningless without the process. I can't take fruits of my labor (whichever labor it can be) to the grave, and if you remove the process of growing and picking said fruits, then where is "you" in that? Did you really "achieved" anything? Or whatever you wished for just magically appeared in front of you with little effort from your side?
Where is the fun of renting a helicopter that would carefully put you onto mountain's summit and pick you up 5 minutes later?
Yes it shares less, but I think the trade-off is you pay less in tokens and hopefully it's truly just not needing to say things because it truly does just better get what you're saying, think to read X markdown file or GH issue which contains the info, etc.
As long as I can still push back and get it to share its thinking on demand and I'm confident the model isn't actually basing things on poor premises, this is okay for me. I am more productive when not inundated with time-wasting check-ins.
That said, I absolutely lament the loss of the ability to access the thinking - I would happily read the "DANGER DANGER DANGER" internal gremlin thoughts fable 5 makes to verify something if they were accessed, and prefer that to a recap presented only for my benefit.
I'd imagine they're going to 10x this, maybe 100x this.
It should follow instructions incredibly well while inferring contradictions or gaps in logic and surfacing those to the user as suggestions for improvements and persistence.
I really hate how Claude just assumes you want to do X/Y/Z and goes off and breaks everything and you're constantly screaming at it STOP DOING THAT. Instead, it should just do the minimal things while building its own guidance along the way in a persisted memory, like, 'would you like me to do X, now, and in the future?' etc.
Curious: what multiplier do you think your productivity has increased by, from before AI?
My view is that popular models by default output wildly excessive amounts of prose for nearly every use case, so if this changes in a new model that’s a pure win.
I mean, it's true that it would be ideal of this stuff did just get figured out optimally behind the API, but there is definitely an incentive on their side to burn more tokens.
It’s amazing how much work you can get done on your phone now, especially if you already have a design mapped out in your head.
Here's the example they give:
> Instead of asking for the shortest possible answer, replace brevity instructions with prioritization:
> Lead with the conclusion. Include the evidence needed to support it, any material caveat, and the next action. Omit secondary detail and repetition.
> Keep all required facts, decisions, caveats, and next steps. Trim introductions, repetition, generic reassurance, and optional background first.
Generally speaking, when I ask for a short answer, I want a short answer because I'm not really willing to read through a bunch of bullshit to get to a summary. Putting the onus back on me to assume what the model will return and write a longer prompt detailing exactly what information I want completely misses the point of why I'm asking for a short answer in the first place.
To put it another way, you will only get the benchmarked performance if you let it talk the way it talks by default. Trying to modify this neuters the model's IQ.
Can you post more information about this?
GPT 5.6 on the Pro x5 plan is down to... 100%. It looks like they just reset the usage limits again. And I still have two resets on the bench.
Anthropic is going to have to up their game to compete.
Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.
Compositionality and visualization are generally much, much better at each new generation / release cycle.
It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.
However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.
And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.
And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.
I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...
I... I think you are missing the point.
There are plenty of little js web games anyway. The point isn't to make an actual game, it's to show coding ability, design and taste in a way that's more assessable than reading a codebase.
I'm a bit concerned about this - starting with GPT5, AI labs started doing this 'complete app from a prompt' sizzle demos. When I started working with GPT5 - which was supposed to be a qualitative jump, just like Fable is now, I tried to do a frontend, and discovered that it gave me a CSS-animated purple-blue interface with embossed buttons, gradient backgrounds and dropshadows.
It looked very cool, but it was a bit overwhelming (also broken), and I was really looking for a pedestrian Bootstrap job.
It required not inconsiderable amount of wrangling for GPT5 to stop doing this. So I don't really like the idea that these models have tons of implicit and hidden behavior, to 'soup up' pedestrian prompts.
That settles it. Anthropic has until the end of this month to get rid of Fable's "safety" bullshit. If they don't, I will not renew my subscription.
The sooner Chinese labs catch up the better it is for all of us, even if you don't use their models, as they're the ones who define the baseline capability that cannot be taken away from you (no one is going to limit/remove access to an LLM if you can get equivalent/better unrestricted access from Chinese open-weight models).
IMO that's exactly why it's a bit better at actual problem solving.
You absolutely do not "always have to correct" Codex. I'm not sure what you're doing, but I'd say 80-90% of its edits on my side it doesn't need any revisions.
this has been my experience with Codex as well, and I have to fix its mistakes every single time. But recently, I literally threw away three hours of work because it kept adding hundreds of lines to my code base. When I restarted the entire work using Fable and Opus, it was like night and day.
Obligatory YMMV, maybe your prompting style fits gpt better. We forget that this matters a lot
Gemini is fantastic, however.
I want the same as you, and even further, I want a model that refuses to execute changes I request if they don't make sense considering the context, or if they're impossible, and avoid any sort of quick hacks and patches. But I also want a model that does the pure opposite, that I can chuck a "Do X" query at and it figures it out. Then I'm sure there are middle-zones between these two, or even more extremes too.
But the choice isn't there, we get to chose between "fast/stupid", "medium/medium" and "slow/smart", then that's it. With system prompts we get to steer it a bit, but I've needed to make my own fork of codex to surface those things to me (the user) so I can control it better, and different models respond differently to the "Stop and don't implement anything if the request doesn't make sense yadda yadda" parts, would be lovely to have those sort of "personalities" surfaced up front when making decisions about what model to use.
But what has happened is the models have gotten better - which OpenAI is making explicit for some cases in this release. You need that stuff less and less as they become more human and better at inferring what's required implicitly.
You still do need to be explicit, and you probably always will, but you don't need as much "engineering" of the way you're asking for things with more recent models.
https://www.google.com/search?q=claude+usage+draining+site:w...
https://investors.palantir.com/news-details/2025/Anthropic-J...
I am working tirelessly and often long nights, on top of a day job.
My effort has shifted to QA testing, reviewing UI designs, and delegating the agents on the implementation.
I will consider it an achievement if I manage to publish a successful app.
Where is the "me" in that? I am guiding the design of every screen and feature the way I would like it to be.
On top of that I make technical decisions on how it is implemented.
Apart from the loads of QA work I will have to handle the business side as well.
As of now it's hardly as trivial and effortless as some make it out to be.
Yes, I no longer write the code, and sometimes it feels frustrating that any teenager without experience could perhaps build a similarly good app soon.
Overall I'm still happy I can now build much larger and better apps and realistically publish them in my free time, for a chance to make serious money.
I don't like OpenAI as a company, but they appear to have QA, and that is probably enough to get me to switch.
Simple UI change? I do an AI review, but otherwise neither read nor write the code. The models are good enough they write better UI code than me, 9 out of 10 times. Not always the more idiomatic, but usually safer and more correct.
Change to our core data plane? I might spend 2-3 times more effort reviewing it than before AI. Yes, I go more slowly than pre-AI. Many more reviews, many more angles considered, including both human and (lots of) AI review cycles.
Most code is not that critical, and AI is also scarily good at writing tests. We also spend considerably more time paying down tech debt and testing thanks to AI, now that the cost is near-zero.
Net: I spend 10-25X less time on low-risk changes. I often direct (or at least approve) the implementation approach, but I rarely read this code. I spend 2-3X more time on high-risk changes. In both cases, I never write code "by hand". Since about November, I've had no reason to actually edit code in a code editor (perhaps maybe except .env files, which we don't allow agents to edit for obvious reasons).
AI is a tool. You can use it to go fast recklessly, or you can use it to go slow with confidence. Just like before AI... the skill and art of engineering is knowing when to do which.
Did they fix that, as that for me was what actually made codex worse.
Not just prose. I think this is part of the reason why you see ridiculous code with insane error handling and type checking even for impossible cases.
One killer feature that Claude has, and AFAIK Codex still lacks, is the ability to start a session in the terminal and then hand it off (actually just remotely control it), from the iOS app.
Last time I tried Codex on iOS it required a ton of set up to link a github project etc. The way claude lets me remote into a session I've already started on my actual machine is much better IMHO.
I would presume (perhaps falsely?) that an instruction like this would lead to the model presenting a conclusion not supported by the evidence, and potentially backtracking as it then tries to justify said conclusion.
Yes, if deliberation happens, the model should figure out what it wants to say during that phase; but if you're using auto mode, the model is not going to be doing any deliberating half the time. In those cases, the output blathering is the model's only chance for deliberation. It "thinks as it talks", per se.
Given that, I would advise a different approach: let it blather, but then get it to write you a conclusion at the end that the model can guarantee will obviate the need to read any of the blathering.
I.e. advise the model to add an "executive summary" to the end of any non-trivial-in-length response. With some wording to carefully navigate the model between "the summary is itself too long" vs "the summary acts more like clickbait, leaving out necessary detail such that it requires actually reading the blather."
Not sure exactly what that wording would look like. I imagine something like "write your postscript executive summary as if you were a senior CIA intelligence analyst summarizing ground-level reports into a daily digest for the Joint Chiefs of Staff. Take up as little of their time as possible, but ensure that any detail critical to decision-making is retained." (But that phrasing might only be useful if the model is delivering a certain type of response, and actively counter-productive otherwise. This kind of thing is delicate.)
At least before it would listen to instructions like this.
Human can no longer be concise when asking for a few sentences instead of 20 paragraphs of BS they don't want to read when all they want is a summary to verify the general direction of the prompt-work before digging into the details.
such progress!
They've also introduced banked resets, which are really clever. If you have a $200/month plan and three banked resets, you're not churning because you will overweight giving up those resets (loss aversion theory).
Now if you tell them too much they go mute or stop telling you important information. Oh intelligence!
Seems good/fine once you get through upgrading the app.
> Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).
The A100 doesn't have hardware FP4, and you'd be running a quantized model with some accuracy loss but unless this was natively trained on FP4*
* to add another layer, they own the model and could apply tons of post-training techniques to reduce that accuracy loss and probably already do
No, doesn't seem like it
https://openai.com/index/separating-signal-from-noise-coding...
Dead internet theory? Semi-random parroting by real people? Or something else.
I was thinking about those species earlier in the context of, what does intelligence mean outside of language.
The benchmark appears to be testing the same thing. Although I don't know how much transfer there would be between this data set and the kind of situations a crow or an octopus would encounter.
Edit: Huh, it's just a Game boy game? I just did a couple of the tasks. It looks like C64 era game to me. Navigating levels. A lot of overlap with animal intelligence then.
At some point just kill the thing, it's not able to work properly as it is.
Mythos can do some amazing things (I'm assuming, I've never seen it). A young child can learn to control its body without reading any books on dynamical systems and kinematics. Mythos cannot learn to control a humanoid robot after sucking in every piece of data Anthropic can get their hands on.
Let’s be generous, it will uplift the investors pretty well once they start charging the real token costs and maybe drive out a few competitors.
As far as the lack of shipping, they're scientists and what we're doing now with LLMs is more "engineering."
Basic stuff about features that are more than a week old just get no attention at all. From the outside Athropic seems to be a clear feature factory.
It was interesting to see where the approaches were similar and where they diverged.
about a sprint's level of effort.
My Fable weekly limit is at 15% used already, 5.6 Sol at 3% used. And this is with the Max 20x plan compared to Codex 5x.
I don't work on the same tasks to compare them objectively, but GPT 5.6 on xhigh seems much cheaper. Essentially unlimited usage.
On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.
It's definitely good that Anthropic's feeling the pressure. Anthropic has worn out their welcome with this "safety" nonsense. If OpenAI actually lets me use the LLMs on a subscription without any of this bullshit, I'll definitely switch.
(It’s not)
JEPA is just getting started
My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.
I disagree. I think you're actually giving up so many little decisions. You are delegating decisions to agents all the time. In place of your slow but still personal decisions, you are ok with decisions that might be similar to what the LLM believes to be the best average, or the best solution based on what limited experience of the world they were trained on. The Ai devil for me is in these details.
That is probably fine for a lot of use-cases, but it's still removing your own agency from the process itself willfully, and yet still taking all of the merit. And to me that makes the final thing less of a byproduct of you and your experiences.
I am not black and white on this, and there are different degrees to the issue. But I just cannot accept this approach that trades the nature of the output for the quantity of it.
And frankly, a lot of other people feel the same. Check the data on the explosion of apps and how little they are maintained or picked up by final users.
Stick the "Never suppress errors" section into your Claude.md, this will never happen again (works for me with Python/Flask, ymmv for other languages).
Although I was surprised that I could get very Claude like results from Chinese models though by just telling it to make the code elegant.
Reminds me of the old days with art AI where you had to put "+good -bad" in the prompt because otherwise it would assume you just wanted random quality outputs, because it had been trained on random quality inputs...
This is the actual reason why openai _invented_ reasoning models, to give them time/space to work out a solution, rather than having to magic a correct solution out of thin air from token 1.
It's less important now that all models do reasoning, but it's still almost always better to make the output come out last rather than first.
The model will still have read the entirety of the document before composing its response. And I believe that even in auto mode, there are thinking tokens behind the scenes.
We’re launching the GPT‑5.6 family of models for general availability following our limited preview: our new flagship, Sol, alongside Terra, a balanced model for everyday work, and Luna, our most cost-efficient model.
GPT‑5.6 Sol sets a new standard for both intelligence and efficiency, achieving state-of-the-art results across coding, knowledge work, cybersecurity, and science while outperforming previous and competing frontier models with fewer tokens and at lower estimated cost. The result is stronger performance per dollar: more successful work for the same spend, or comparable results at a lower total cost. We also introduce a new way to accelerate the most demanding work: ultra is our highest-capability setting, coordinating multiple agents across parallel workstreams to finish complex tasks faster. Stronger computer use and design judgment make GPT‑5.6 Sol our most polished collaborator yet, helping it inspect, refine, and deliver ready-to-use results.
We trained GPT‑5.6 to get more useful work from every token. On Agents’ Last Exam(opens in a new window), an evaluation of long-running professional workflows across 55 fields, GPT‑5.6 Sol sets a new high of 53.6, eclipsing Claude Fable 5 (adaptive reasoning) by 13.1 points. Even at medium reasoning, it beats Fable 5 by 11.4 points at roughly one-quarter the estimated cost. That efficiency extends to smaller models, which are essential to making intelligence more abundant and affordable: GPT‑5.6 Terra and GPT‑5.6 Luna outperform Fable 5 at around one-sixteenth the cost. On the Artificial Analysis Intelligence Index(opens in a new window), a broad measure of intelligence spanning agentic work, coding, scientific reasoning, and general capabilities, GPT‑5.6 Sol with max reasoning comes within one point of Fable 5 while completing tasks in 61% less time at roughly half the estimated cost.
GPT‑5.6 launches with our most robust safeguards to date, designed to be resilient against determined and adaptive misuse without broadly limiting legitimate work. Before general availability, we put the models and safeguards through our most extensive evaluation period yet, combining human red teaming with large-scale automated testing. During the preview, we worked closely with expert organizations and with trusted partners to pressure-test defenses and strengthen safeguards before broader launch. The resulting system layers protections trained into the model with real-time checks, monitoring, and access calibrated to trust and risk.
GPT‑5.6 Sol is our best coding model yet. On the Artificial Analysis Coding Agent Index, GPT‑5.6 Sol with max reasoning sets a new state of the art at 80, 2.8 points above Fable 5, while using less than half the output tokens, taking less than half the time, and costing about one-third less. That advantage extends across the family: Terra performs just above Fable 5, while Luna outperforms Opus 4.8; each does so in roughly one-third of the time, with about half as many output tokens, and at approximately one-quarter the estimated cost. It also sets new state-of-the-art results on Terminal‑Bench 2.1 and DeepSWE, which test complex command-line workflows and long-horizon engineering in real codebases.
Artificial Analysis Coding Agent Index: an independent index of coding-agent performance across implementation, terminal use, and real codebases.
GPT‑5.6 can write and run lightweight programs that coordinate tools, process intermediate results, monitor progress, and choose the next action as work unfolds. This lets tool-heavy tasks advance with fewer tokens, fewer model round trips, and less guidance. Instead of requiring developers to script every step or passing every tool response back through the model, Programmatic Tool Calling(opens in a new window) in the Responses API can filter large amounts of intermediate data, retain only what matters, and adapt its workflow along the way.
For problems that reward a greater investment of time and compute, GPT‑5.6 can push beyond this efficient default. max gives GPT‑5.6 even more time than xhigh to reason and explore alternatives, run checks, and revise its approach. ultra goes further by coordinating four agents in parallel by default, trading higher token use for stronger results and faster time-to-result on demanding tasks. The charts below compare ultra’s default four-agent setup with a one-agent baseline across BrowseComp, SEC-Bench Pro, and Terminal-Bench 2.1; BrowseComp and SEC-Bench Pro also show 16-agent configurations. Across all three evaluations, adding parallel agents shifts the score-latency frontier upward and to the left, reaching stronger results in less time. In the API, developers can build ultra-like experiences using the multi-agent beta in the Responses API.
1 of 11
GPT‑5.6 delivers a step change in design judgment. With only high-level direction, GPT‑5.6 creates tasteful, ergonomic, and functional interfaces. Its stronger computer-use capabilities let it inspect and refine the rendered result—not just generate the underlying code or content—so it can catch visual and functional issues and apply finishing touches before handing the work back.
GPT‑5.6’s frontend capabilities also turn natural-language requests into polished, interactive explanations and visualizations within ChatGPT Work.
GPT‑5.6 delivers better results for professional tasks. It takes messy context from your documents and everyday workflows like Slack, Notion, Microsoft 365, and Google Drive, and converts it into expert-level, shareable artifacts.
GPT‑5.6’s strength on knowledge work shows up in evaluations spanning long-horizon professional analysis, browsing, tool use, and computer use. GPT‑5.6 Sol sets new state-of-the-art results on BrowseComp at 92.2% and OSWorld 2.0 at 62.6%; on OSWorld, it surpasses Opus 4.8 while using 85% fewer output tokens. Here, the performance-per-dollar gains extend across the GPT‑5.6 family. Luna nearly matches GPT‑5.5’s peak performance at less than half the estimated cost, while Terra surpasses it at a lower cost.
BrowseComp: GPT‑5.6 Sol achieves a new state of the art on BrowseComp, consisting of agentic browsing tasks.
GPT‑5.6 Sol improves quality in presentations, documents, and spreadsheets, producing outputs that are more polished and accurate. It can create fully editable presentations from scratch, translating a prompt and source material into a coherent visual narrative with strong layouts, hierarchy, and design.
The improvement is especially pronounced when following templates and reference decks. GPT‑5.6 can infer a deck’s design system—layouts, typography, spacing, colors, and recurring content patterns, including rules embedded in the Slide Master—and apply those conventions consistently to new material. In this example, when asked to update numbers based on a reference file, the GPT‑5.5 output is missing key components from the master slide, while GPT‑5.6 follows the reference structure more faithfully.



GPT‑5.6 also creates more visually refined documents and spreadsheets. It follows complex reference formats more faithfully, which is important for repeatable knowledge work activities. It handles equations and financial models with greater precision, and makes better use of typography, spacing, hierarchy, and page or worksheet layout.
Early customers testing GPT‑5.6 saw improvements to knowledge work outputs across domains.
1 of 9
GPT‑5.6 is our strongest cybersecurity model yet, achieving frontier performance with significantly fewer tokens. On ExploitBench2, which measures progress from reaching vulnerable code through arbitrary code execution, it scores 73.5% versus GPT‑5.5’s 47.9% at a comparable output-token budget. On ExploitGym3**,** which asks agents to turn real-world vulnerabilities into working exploits, it almost doubles GPT‑5.5’s peak pass rate, from 15.1% to 24.9% under the two-hour cap; with six hours, it reaches 33.7%. On SEC-Bench Pro, which tests proof-of-concept generation on complex software, it scores 71.2% versus GPT‑5.5’s 45.8% at an improved latency.
GPT‑5.6 supports important defensive tasks such as secure code review, patching, threat modeling, and blue teaming. Qualified individuals and organizations in OpenAI Daybreak’s Trusted Access for Cyber program can access more of its defensive capability through more precise safeguards for verified work in authorized environments, including vulnerability triage and validation, malware analysis, detection engineering, and patch validation.
ExploitBench: Building progressively more capable V8 exploits; GPT‑5.6 shows a large gain over GPT‑5.5. Latency chart is not shown as latency estimation is unreliable for this benchmark.
GPT‑5.6 Sol also shows broad gains across scientific research. On life sciences evaluations, GPT‑5.6 demonstrates Pareto improvements over GPT‑5.5 on real-world biology, life science research workflows, and chemistry.
**GeneBench Pro**: Long-horizon genomics and quantitative-biology analyses; GPT‑5.6 reaches stronger results with fewer tokens and less time. Claude Fable 5 is not included as it _does not answer_(opens in a new window) advanced biology questions and refuses the majority of questions in this eval.
GPT‑5.6 is our strongest model yet for accelerating AI research. Inside OpenAI, researchers use it across the development loop: diagnosing failures, optimizing training systems, running experiments, and interpreting results. We already saw that acceleration and stronger adoption during the internal testing period of GPT‑5.6, as average daily output tokens per active researcher were more than twice the highest level observed for GPT‑5.5.
This way of working is quickly becoming standard. Over the past six months, the share of research compute devoted to internal coding inference grew 100-fold, while internal agentic token usage increased approximately 22-fold. These adoption metrics do not measure research progress on their own, but they show how rapidly AI assistance is increasing for research and across other teams like sales, marketing, user ops, finance, and more.
To measure this capability directly, we developed an internal suite of evaluations based on real AI research tasks, including debugging research systems, optimizing kernels and training recipes, running machine-learning experiments, and improving another model.
Aggregate RSI capability: On a bundle of evaluations measuring progress towards recursive self-improvement, we observe GPT‑5.6 Sol to be a 16.2 point improvement over GPT‑5.5, accelerating internal research across the board.
As model capabilities increase, we strengthen our safety stack so advanced intelligence can remain broadly useful while applying greater scrutiny to the highest-risk uses. For GPT‑5.6, we built our most robust safety system to date, calibrated to each model’s capabilities and powered by more compute than ever before.
The GPT‑5.6 models are more capable than our earlier models in both biology and cybersecurity but do not cross the Critical threshold in either category. In cybersecurity, our testing suggests GPT‑5.6 is better at finding and fixing vulnerabilities than at reliably carrying out autonomous, end-to-end attacks against hardened targets—giving defenders an opportunity to strengthen systems before weaknesses are exploited. In biology, our testing suggests GPT‑5.6 can support legitimate research but does not provide the end-to-end capability needed to create, engineer, or synthesize a highly dangerous novel threat.
Both domains are inherently dual-use. In cybersecurity, the same capabilities that could help an attacker exploit a vulnerability can help a defender find it, reproduce it, and build a reliable fix. Overblocking therefore creates a security risk of its own. It can prevent defenders from testing systems and deploying patches while malicious actors continue using other models, including increasingly capable open-source models, as well as established tools. Effective safeguards account for the context and likely consequences of a request, preserving legitimate defensive work while applying stronger controls where the evidence indicates a serious risk of harm.
GPT‑5.6’s safeguards are layered for greater accuracy and redundancy, and designed to adapt quickly as new attacks emerge. Protections trained into the model work alongside real-time checks, continuous monitoring, and account-level enforcement, to help the system remain safe even when a particular layer does not work as intended. In many systems, classifier flags alone decide what to block, relying on lower intelligence models that are harder to change in order to prevent harm. Our approach adds a reasoning monitor that reviews the conversation to determine if there is a potential for harm. This design is intended to enable defensive work while blocking serious misuse, with the most sensitive capabilities reserved for verified users through Trusted Access. Because some protections use test-time reasoning, we can rapidly update them to close gaps without retraining classifiers from scratch.
We are taking a more conservative approach as we continue to strengthen the system against adaptive attacks. Compared with previous models, our GPT‑5.6 Sol cyber safeguards block roughly ten times more potentially harmful activity. Because these measures can create friction for benign use, we provide an option in ChatGPT and Codex to easily retry prompts on lower-capability models, and we will continue reducing the impact of our safeguards on benign use while maintaining a high robustness bar. This reflects our iterative deployment approach: starting conservatively and improving based on what we learn from real-world use.
Before general availability, we ran our most intensive safety evaluations to date, including extensive red teaming, robust capability and safeguard testing with external experts, and approximately 700,000 A100e GPU hours of black-box automated red teaming. This enabled us to systematically probe likely weak points, surface jailbreaks, and help us strengthen the system before launch.
There is no such thing as perfect security, and our work to secure increasingly capable models continues. New weaknesses will be discovered, as will new jailbreaks that circumvent existing safeguards. Each new generation of model will also create new avenues for attack and misuse. We build for that reality through layered safeguards, continuous monitoring, rapid remediation, and collaboration across the defensive community. For GPT‑5.6, we have paired our existing security(opens in a new window) and biology bug bounty programs with a new rapid-remediation process and our strongest monitoring effort to date. Findings from researchers, monitoring, and real-world misuse will feed into new evaluations and stronger safeguards on an ongoing basis.
GPT‑5.6 spans three model tiers: Sol, our flagship; Terra, a lower-cost model with performance competitive with GPT‑5.5; and Luna, our fastest and most affordable model. The number identifies the generation, while Sol, Terra, and Luna are durable capability tiers that can advance on their own cadence.
GPT‑5.6 is available starting today across ChatGPT, Codex, and the OpenAI API. The rollout is starting globally now and will continue gradually toward full availability over the next 24 hours.
max is available to all users with access to GPT‑5.6 in ChatGPT Work and Codex and can be toggled on in settings. In ChatGPT Work, ultra is available to Pro and Enterprise users. In Codex, it is available to Plus and higher plans.GPT‑5.6 is priced per 1M tokens across three model sizes: Sol is $5 input / $30 output; Terra is $2.50 input / $15 output; and Luna is $1 input / $6 output. GPT‑5.6 also introduces more predictable prompt caching, including support for explicit cache breakpoints(opens in a new window) and a 30-minute minimum cache life. For GPT‑5.6 and later models, cache writes are billed at 1.25x the model’s uncached input rate, while cache reads continue to receive the 90% cached-input discount.
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Fable 5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview | Gemini 3.5 Flash |
|---|---|---|---|---|---|---|---|---|
| Agents' Last Exam | 52.7% | 50.4% | 50.3% | 46.9% | 40.5% | 45.2% | 32.1% | — |
| GDPval-AA v2 | 1,747.8 Elo | 1,593 Elo | 1,591.8 Elo | 1,493.7 Elo | 1,759.6 Elo | 1,600.1 Elo | 962.3 Elo | 1,348.8 Elo |
| Management Consulting Tasks (Internal) | 43.2% | 37.2% | 35.4% | 31.3% | 35.5% | 31.6% | 13.2% | — |
| Big Finance Bench | 53% | 51% | 36% | 49% | — | 44% | — | — |
| Artificial Analysis Intelligence Index v4.1 | 58.9 Index score | 55 Index score | 51.2 Index score | 54.8 Index score | 59.9 Index score | 55.7 Index score | 46.5 Index score | 50.2 Index score |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Sol Ultra | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Mythos 5 | Claude Mythos Preview | Claude Fable 5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview |
|---|---|---|---|---|---|---|---|---|---|---|
| Artificial Analysis Coding Agent Index v1.1 | 80 Index score | — | 77.4 Index score | 74.6 Index score | 76.4 Index score | — | — | 77.2 Index score | 72.5 Index score | 42.7 Index score |
| SWE-Bench Pro | 64.6% | — | 63.4% | 62.7% | 59.4% | 80.3% | 77.8% | 80% | 69.2% | 54.2% |
| DeepSWE v1.1 | 72.7% | — | 69.6% | 67.2% | 67% | — | — | 69.7% | 59% | 11.8% |
| Terminal-Bench 2.1 | 88.8% | 91.9% | 87.4% | 84.7% | 85.6% | 88% | — | 83.1% | 78.9% | 70.7% |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Fable 5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview | Gemini 3.5 Flash |
|---|---|---|---|---|---|---|---|---|
| GeneBench Pro | 28.7% | 23.3% | 10.8% | 12% | — | 16% | 3.1% | 8.14% |
| LifeSciBench | 59.9% | 56% | 51.2% | 50.4% | — | 53.6% | — | — |
| MedChemBench (Internal) | 48.3% | 35% | 30.4% | 35.5% | — | — | — | — |
| HealthBench Professional⁶ | 60.5% | 57.7% | 55.7% | 49.5% | 60.9% | 53% | — | — |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Sol Ultra | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Mythos 5 | Claude Mythos Preview | Claude Opus 4.8 | Gemini 3.1 Pro Preview |
|---|---|---|---|---|---|---|---|---|---|
| OSWorld 2.0 | 62.6% | — | 50.2% | 45.6% | 47.5% | — | — | 54.8% | — |
| BrowseComp | 90.4% | 92.2% | 87.5% | 83.3% | 84.4% | 88% | 87.9% | 84.3% | 85.9% |
| BenchCAD | 70.6% | — | 62.3% | 63.1% | 44.4% | 38.4% | 35.5% | 27.3% | — |
| BenchCAD (python tool) | 83.4% | — | 78.2% | 73.9% | 55.8% | 65% | 61% | 51.8% | — |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Sol Ultra | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Mythos 5 | Claude Mythos Preview | Claude Opus 4.8 |
|---|---|---|---|---|---|---|---|---|
| Capture-the-Flag Challenges | 96.7% | — | 91.8% | 85.2% | 88.1% | — | — | — |
| SEC-Bench Pro | 71.2% | 74.3% | 57.7% | 48.9% | 45.8% | — | — | — |
| CyberGym | 84.5% | — | 81.8% | 77.9% | 81.8% | 83.8% | 83% | 78.1% |
| ExploitBench | 73.5% | — | 52.9% | 33.2% | 47.9% | 78% | 74.2% | 40% |
| ExploitGym | 33.7% | — | 23.2% | 12.4% | 15.1% | — | — | — |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 |
|---|---|---|---|---|
| Internal Research Debugging Evaluation | 68.3% | 67.8% | 50.8% | 50% |
| KernelGen 1P | 61.1% | 49.2% | 22.4% | 29.3% |
| NanoGPT | 9.69% | 14.5% | 1.66% | 2.65% |
| PostTrainBench Lite | 50.3% | 51.5% | 29.6% | 38.8% |
| RSI Index | 57.9% | 56.3% | 41.9% | 41.7% |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Fable 5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview |
|---|---|---|---|---|---|---|---|
| MMMU Pro (no tools) | 83% | 80.7% | 78.4% | 81.2% | — | — | 80.5% |
| MMMU Pro (with tools) | 84.6% | 82% | 79.5% | 83.2% | — | — | — |
| gdp.pdf | 30.7% | 24.7% | 22.7% | 26% | 29.8% | 22.5% | 16.7% |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Mythos 5 | Claude Mythos Preview | Claude Fable 5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview |
|---|---|---|---|---|---|---|---|---|---|
| GPQA Diamond | 94.6% | 92.9% | 92.3% | 93.6% | 94.1% | 94.6% | 92.6% | 92% | 94.3% |
| FrontierMath Tier 1-3 (v2) | 89% | 84.9% | 78.6% | 85.3% | — | — | 87% | 80% | 59.6% |
| FrontierMath Tier 4 (v2) | 83% | 68.3% | 58.5% | 72.5% | — | — | 87.8% | 56.1% | — |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Mythos 5 | Claude Mythos Preview | Claude Fable 5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview | Gemini 3.5 Flash |
|---|---|---|---|---|---|---|---|---|---|---|
| AutomationBench | 18.1% | 15.2% | 14.9% | 12.9% | — | — | 17.4% | 15.5% | — | 14.5% |
| Toolathlon | 58% | 53.1% | 53.4% | 55.6% | 61.7% | 61.1% | 61.7% | 59.9% | 48.8% | — |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Mythos 5 | Claude Mythos Preview | Claude Opus 4.8 |
|---|---|---|---|---|---|---|---|
| OpenAI MRCR v2 8-needle 256K-512K | 91.5% | 89.6% | 41.3% | 81.5% | — | — | — |
| OpenAI MRCR v2 8-needle 512K-1M | 73.8% | 72.5% | 41.3% | 74% | — | — | — |
| GraphWalks BFS 256k f1 | 90.7% | 76.9% | 81.3% | 73.7% | 91.1% | 85.7% | 85.9% |
| GraphWalks BFS 1mil f1 | 77.1% | 71.2% | 51.2% | 45.4% | 79.4% | 74.3% | 68.1% |
| Eval | GPT‑5.6 Sol | GPT‑5.6 Terra | GPT‑5.6 Luna | GPT‑5.5 | Claude Opus 4.8 | Gemini 3.1 Pro Preview |
|---|---|---|---|---|---|---|
| ARC-AGI-3⁷ | 7.78% | 0.8% | 0.18% | 0.43% | 1.5% | 0.42% |
https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a...
For my usage, I would very much prefer if those $/task were being spent in thinking and experimenting, and the actual output would be as short and maintainable as possible. “maintainability” is a vague target of course, but it’s at least somewhat correlated with code size.
I suspect it's not what you meant, but it's definitely not random and is very deliberate. Just today I got it to reliably trigger the "safety" filter with (drumroll) having it list the weight keys of a 300M parameter ModernBERT-derived model. Their "safety" classifier must be matching one of the key names in there and trigger their "this is a frontier model" anti-competitive filter[1] (even though it's just a tiny 300M parameter model, four orders of magnitude smaller than the frontier).
[1]: https://news.ycombinator.com/item?id=48464732
Fortunately once you know how it works (i.e. dumb keyword classifier) it's easy-ish to get around: just rename the keys so that it doesn't contain the naughty keyword. (At least as long as it doesn't trigger on something in its own thinking trace, which needs... more creative workarounds.)
:-) I hope you read those tests before claiming it's "scary good"
My brother in Christ this entire thread is talking about the new model that was released
Not quite. The hosting side can change reasoning budgets (or re-assign what terms like "high" means), temperature and other decoding parameters, output length limits, finetune internal "hidden" prompt, latency optimizations, finetune attention algorithms, even change quantization - all still serving as the same model.
We know (or suspect) Anthropic frequently nerfs models while keeping their name and version the same.
You sign in the Codex app on your Mac same on iOS and are able to completely control your sessions - fork, side chats, plugins - everything.
It’s really great i often work through it. And you can connect any number of Codex instances on any number of macs and then manage them all through the iOS app.
Longer, more detailed or conditional prompts always introduce an additional cognitive load as it checks every token it generates against the conditions. Making instructions more absolute (like: "Never do...") can increase the duration of compliance but at the cost of creating a significant center of attentional gravity. This can cause far more output distortion as the model devotes increasing portions of its attention budget to ensure compliance with a heavyweight requirement or prohibition. Every word in a global prompt is a trade-off between attention, compliance, drift, etc.
As someone used to thinking of computers as natural deterministic rule-followers, it's weird having to carefully wordsmith and A/B test even the simplest global prompts. It feels like coaxing a hyper-literal, emotionally sensitive, spectrum-ish toddler to comply but without being so strict it gets 'upset' or spirals into hyper-focusing.
> Trim introductions, repetition, generic reassurance, and optional background first.
It's not possible for the model to "trim" those before they've been output, so this is akin to telling it "not think of an elephant or even take the existence of elephants into consideration while solving this problem".
Would it actually follow them? IME LLMs are incapable of estimating the length of their own output, the total length of the current context, etc. They just make stuff up unless they have external tools that can inspect those things for them.
The gap between Opus 4.8 and Fable is large enough to drive a GPT 5.6 sized bus through. A better Opus would take some of the heat off.
There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
"we use a rough estimate of a total 9M GPU hours"
From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M.
The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model.
You might want to change assumptions about how expensive these models are.
That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
For the design of my app, I try to imitate the UX of Apple's first party apps.
I checked the competitors, and what is in the AppStore looks like it was built in 2015, and only updated to add Ads and subscriptions. Surprisingly, I did not even see vibe coded apps for my use case.
Though my feeling, no proof, is that the opus/fable today is not what it was months ago. there was a time for about a month where opus was incredible. Just incredible but as fable started to move out i swear to god it feels like sonnet now. Fable feels like opus used to but costs more.
I have 10+ of these workspaces in parallel, and I context switch between them as I get blocked on things. I manage the workspaces using `herder`, which is a terrific tmux-like tool that allows me to keep those workspaces on a nixOS machine I have at home that I SSH into via tailscale, so my agents don't stop working every time I close my laptop (it also lets me leverage that machine's computing resources instead of running dozens of servers and harnesses on my poor MacBook).
[1] https://www.reddit.com/r/ProgrammerHumor/comments/10ek380/co...
On high-challenge turns, the auto mode routes to the "thinking" model. But on low-challenge turns, it routes to the "instant" model.
And the "instant" model, by design, has no capacity for deliberation. (If it did, it couldn't guarantee that its responses would begin streaming "instantly.")
But, unless your desired output is literally a document for others to read, at the point where you're having a model generate a full, lengthy output multiple times over with revisions, you may as well just turn off auto mode and have it always deliberate (i.e. choose the thinking model explicitly from the model selector.) Then it'll be as messy as it needs to be while deliberating, but give you exactly what you want as output.
(And if your desired output is literally a document for others to read, that you want to interactively draft and polish, then (in the case of ChatGPT specifically) you should not only be explicitly forcing the "thinking" model, but also should be asking it to activate the "canvas" feature from the start. My understanding is that revising a canvas document involves the model emitting something like editing gestures, rather than simply re-streaming the updated chunks of text. This saves a lot of output tokens on large documents.)
This is from Ministral 3 14B, a 2025 model without reasoning, that you can run on your PC:
> Write a Haiku involving HackerNews, and the capability of large language models like you to reply in an exact number of words or syllables.
Silicon whispers,
exact words in code’s embrace—
Haiku blooms anew.
Across multiple tries it got it wrong a couple times (by ~2 syllables). But syllables are extra tricky (because of how LLMs use tokens) and the point is that for things like "summarize in 5 bullet points" you will mostly get 5 bullet points, maybe 6, but not 10 or 20, and no need for a tool that count bullet points.Having said that, in truth, I almost never read the unit tests. Before AI, we had almost none (see: several person game studio) so the tradeoff is not "AI-generated tests" vs "human written ones", it's whether we have tests at all. So, I take them for what they're worth - not much - but if it catches an extra regression before it ships every now and then, it was worth it for the price (~free).
Sadly, you can't do things like this directly using ChatGPT's own "GPTs" abstraction. (For that feature to be useful, they really need some concept of server-side agents as stateful resident IO-stream-reducer actors.)
No reasonable model has worked that way for years.
We will probably just get reader-side affordances for this like auto-folded justification and introduction sections and so on.
Doubtless some chat interface will add this the way they’ve added reasoning folding.
Because that’s what’s in the training set. Reticent humans don’t have blogs.
Before, they would do their celebrated usage resets while I was behind the already generous weekly quota.
If I had 80% left on Friday to spend over the weekend, they would reset and I'd have 100% until next Friday, a potential net negative.
OpenAI listened to the complaints.
Otherwise I am not interested.
Supposedly Fable 5.1 is in the later stages of the release pipeline, maybe it takes back the crown from OpenAI, who are now rumored to launch GPT 6 in August.
The 9M GPU hours includes the DINO v2 inference used in order to curate the data set.
The final training run used like 300000 dollars of compute.
Unfortunately we don't know how much RLVR + Agent training costs these companies. I'm just gonna say it's in the hundreds of millions, because they are supposedly making billions of profit on inference yet making billion dollar losses
The difficulty in predicting a latent is so called "collapse"; the embedding neutral network can always output the zero vector and this would predict the output correctly.
There are different ways to solve this, DINO uses two different models - a teacher and a student and LeCunn uses an explicit term against collapsing to a single output.
Yann mentions DINO in his talks
As GP said. More RLHF is in fact the bitter lesson.
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
I can’t imagine a more useless test, but I get that it wants to verify that it actually made the change. I just delete the test when it’s done.
You’re arguing via reductionism, and failing to explain the outcomes and emergent properties of the “stupid” system. Humans are made of atoms that are quite literally stupid, so by all means, explain our intelligence and why it’s different than LLMs. (I’m not claiming LLMs are intelligent, BTW, I just don’t think your claim helps nor believe that you can fix it.)
Not sure if there's a Termux equivalent for iPhone.
Just use termux on your phone and connect to a tmux session on your server.
Codex won't know the difference.
Not disagreeing with your point, but your terminology muddies your point.
But your point doesn't acknowledge that even with inference, there is a lot of room to tune the calculations. Multiple models, quantization tradeoffs are just the most obvious examples. Every architecture can be adjusted to increase intelligence/watt or other measure, even without further training.
1. Run `codex remote-control --help` directly on your Linux server. 2. From the desktop app, connect to your Linux box, start Codex there, and make it remotely controllable.
Either approach will get you set up.
Tuning those can definitely make a model respond better or worse.
So your claim (quoting 100% as written) that "Their performance depends solely on the model training before release and how well you curate the context you feed it" is wrong. Hence the downvotes.
Doesn't matter if LLMs are to be considered intelligent or not for the claim to be wrong.
> But judging from the downvotes, it seems AI folks get upset when someone talks honestly about their precious piles of matrix multiplication.
Often yes. In this case, it's more like they get upset when someone says something factually wrong, and then defensively changes the goalposts.
"The underlying model is just a biological neutral network. It seems you carbonoids get upset when someone talks honestly about synapses and neuron firing."
Extraordinary claims require extraordinary proof. So far the only other people I've seen teach this prompting style or talk about models "correcting their own output" were getting their information from AI-generated, hallucinated LinkedIn and TikTok posts.
If this thing exists - which is not just a LLM outputting content serially, placed inside a harness where itself (or another llm) is prompted to review and also output revisions serially - and if a single model can be prompted to output content and "iterate" or rewind it, and it's been widespread amongst "all reasonable models", surely there will be a flurry of sources you can point me to so I can learn.
Trying to craft a workable prompt got so frustrating I eventually just tried a prompt of "Don't change anything about your normal text formatting, it's perfect as is" and even that skewed the output vs no prompt. For browser chat I finally just wrote a client-side CSS UserStyle that does the formatting. Now I even have sequentially numbered sections with indented alphabetic bullets! Zero cognitive load or attentional skew and it never drifts off the formatting in long sessions.
I tend to define it "better at solving, worse at assisting phenomenon". Which doesn't properly show on benchmarks that only focus on the solving part.
Pray they do not realign them further.
There are times I require single word answers. I will use whatever model responds as I desire and at this point those models are just a few.
But this is exactly why we should not anthropomorphize the models: they are very obviously not conscious, because they are not alive, any more than conventional computer programs are. And proposing otherwise leads to absurd moral arguments, while not really serving any other purpose.
If you don't like the fact that some people disagree with you about what the word "intelligence" actually means, fine. But I am not about to entertain a world in which humans face moral retribution for "enslaving" a literal inanimate tool created by humanity.
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
The underlying structure and tuning of the LLM are entirely unchanged by context. It merely affects the attention and activation of the network. The LLM will not be able to work with this hypothetical new language unless it is in context. This does not fit the computational meaning of learning.
Smart is not a well defined term. Nor is it's general idea formally understood. Use it freely, but you won't be saying anything meaningful unless you define your usage.
1. Install another copy of codex in a special dir on the Linux machine:
$ curl -fsSL https://chatgpt.com/codex/install.sh | sh
2. Run codex remote-control from that special dir to start and pair the daemon: $ ~/.codex/packages/standalone/current/codex remote-control start
$ ~/.codex/packages/standalone/current/codex remote-control pair
3. On the phone, open ChatGPT app, Choose Remote, then pair it with the code printed above.4. Voila! The codex sessions running on the Linux machine now show up on the phone!
Oh give me a break. Show me one example of 1) any knob twisting that makes the underlying model better. or 2) any example of the AI providers twisting those knobs to do anything other than degrade performance for their own bottom line or safety.
The current post says: "it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount."
When no, the model cannot "get better". It doesn't determine any appropriateness of response realtime except for the weights baked into it from the beginning and whatever context it can muster. If you cram enough guidance that it doesn't decide to ignore maybe you can make it more brief. But it (the model) can do none of those things.
LLM models are literally stupid by design.
Can you prove that these models aren't conscious? And, as a counterpoint, can you prove that you are conscious, rather than a philosophical zombie?
We bred horses, cows and sheep. Most of those that live today wouldn't be alive if not for human intervention. Does that give us the right to do whatever we want with them, without consideration for feeling or morality?
In this case, you can take comfort in the idea that the tokens these models produce are likely a form of excrement to the conscious entity metabolizing the information, and rather than enslaving anything, we're creating a habitat and "harvesting" the byproducts.
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
You might argue that the systems we've built around them are learning in a way, as they strategically condense and save artifacts from past interactions to pass into the LLMs context. But the LLM itself, which is the source of the intelligence, is not learning. It remains entirely unchanged throughout inference. This difference may seem trite, but it has significant impacts over the long term behavior.
Demanding sources for this is odd. It's literally been a headline feature of every frontier model for two years.
I guess you are "technically correct" that no model can "un-emit" tokens... but I don't think that is what anyone was saying or an interesting point to make.
Edit: see also this recent post, which details another place where revisions can happen, upstream of the reasoning token emission:
What things specifically and when?
You are now anthropomorphizing the model yourself.
I mentioned several.
You're now once again changing goalpoasts to say you meant the underlying model, not the overall llm performance, even though you explicitly wrote: "Their performance depends solely on the model training before release and how well you curate the context you feed it".
So, the context curation was relevant (meaning you didn't constrain your claim to the underlying model), but now somehow all the additional tunables aren't relevant (because suddenly you're just talking about the model).
End of discussion.
Your comments are conflating multiple kinds of “smart” and “better”. You’re right that if all the inputs are exactly the same, it takes a new model to improve (ignoring non-determinism). But the knobs and context and harness change the inputs, and they do improve output, contrary to your claim. You’re failing to capture the distinction between what the model itself does and how the harness can boost the model’s performance. It is legitimately valid and fair to call improved performance “better”, no matter where it comes from.
This all gives me the feeling you might not have experience with or understand what’s happening in today’s harness development, and the degree to which it may be as important as the weights. There are in fact a lot of things you can do to improve a model’s performance on tasks & benchmarks, without changing the model weights. @coldtea mentioned a bunch, but the harness feedback loop, internal prompts, system prompts, skills, and requests for a model to try harder, and verify and validate it’s output all lead to improved performance, all without retraining.
I agree LLMs are stupid; they’re statistical token predictors. But somehow statistical token prediction is amazing and works much better than we imagined. The talking points about LLMs being stupid token predictors are fading now because they lack explanatory power for how good the models have become. The big surprise here isn’t about LLMs. It’s about language, and how much “thinking” and intelligence is contained in language. We don’t have a good grasp on where the line is between language and intelligence. LLMs have crushed the Turing Test into dust, and yet we don’t consider them intelligent. They often appear to understand what you ask thoroughly, can re-state it in different words, they can correct your misunderstandings or add nuance you didn’t see. All this because that’s what humans do and LLMs talk like humans.
You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
GP was saying
> That would be true of non-iterative models that just emit an output from beginning to end.
Which suggests that there are
- "iterative" models
which
- do not output "from beginning to end"
which AFAIK is science-fiction.
Since the LLM is now designed to run in a harness, it’s really not even wrong any more.
End of discussion.
Because this entire discussion is about the release of a new model, and models are fixed. Sure you can try to modify all the scaffolding around it, but the model is the model. It doesn't matter what you're trying to improve. You can only improve the peripheral aides. And the peripheral aides can't fundamentally fix the problems with llm models when they can't learn new relationships or facts.
You will always have to wait for a new model (like this one we are talking about) for improvements to the model.
LeCun's ideas cannot be reduced to a 6 second clip...
You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
Wrong. The face-saving backtracking doesn't change that.
Right. The sentence you quoted was about brevity improving with a new model. It did not suggest the model itself improving.
I’m confused why you’re stuck on this tangent. And confused why you are repeating the talking points about the model being fixed. The model is fixed - that’s true, I already agreed with you. But you don’t seem to be listening to anything else.
> It doesn’t matter what you’re trying to improve.
What do you mean? If we’re trying to improve LLM output, there are multiple ways to achieve it. A new model is one of them. Changing the inputs is another.
> You will always have to wait for a new model (like this one we are talking about) for improvements to the model.
This is true! Nobody here is disagreeing with that. The part that it seems you’ve argued incorrectly is the apparent claim that output can’t get better. Output can “improve” without improving the model.
Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.
They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.
But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T
That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.
While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.
The human brain manages to self-organize with only a fraction of the information that LLMs get trained on. To train an LLM you need a lot of high-quality data.
There are two threads in history: firstly the compute thread leading to GPUs and AlexNet in 2012. Secondly the model architecture thread that started long before we had the compute and lead to transformers in 2017.
If the compute thread had been 30 years behind then we might be spending this century coming up with better architectures to make do with the more limited compute. However since the compute came first, we settled on the first thing that worked (transformers) and all effort went into polishing that.
There's something wrong with transformers though. No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.
So? The question isnt can we get to ASI as efficiently as a brain, the question is can we get there, which we likely can. The inefficiencies can also be fixed after that.
>No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.
Again, so? Humans are efficient but also bad at many things that transformers are already better at because of it. You are looking at the wrong thing if you think it needs to be like humans.