Bravo, and I hope it has the impact on the AI safety field it deserves to have.
They being the US and China and by agreement.
It would be ideal, but there’s far too much money on the table to overcome human nature.
So my hope is we hit some kind of limits naturally.. Wishful thinking?
My early analysis of the analysis:
https://lifearchitect.substack.com/p/the-memo-special-editio...
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
If you won't even so much as acknowledge the possibility of error, your argument is hollow and empty. All the "choices" presume these labs are being completely honest and acting in some degree of good faith (relative to the systemic incentives of society in its present form), while in reality we're still just building and refining probability models with increasing accuracy of output and flexibility of processing (namely agents) but still lack actual "intelligence" of any real sort.
Show me a paper that doesn't merely presume inevitability of LLM-based AGI/ASI, and instead actually lays out the core paths that history suggests we're likely to encounter with any "world changing technology":
* In the best case, that the technology really will revolutionize the world and do everything promised by its biggest boosters (papers like this one)
* In the middle case, that it becomes just another tool in our collective toolkit, and the consequences of a revolution built on external investment fizzling out
* In the worst case, that the tool itself is so niche in its utility that investment collapses rather than fizzles out; what do we do with all this compute, now? Who owns the debt? Who foots the bill? How can we mitigate those existential risks?
I'm just rather nauseated by the continued trot of inevitablism masquerading as academia rather than an actual, neutral, bias-controlled-and-disclosed study that paints potentialities instead.
---
Having finished skimming through it, another comment springs to mind: Jesus Christ these things continue to be jingoist as absolute fuck. It's a fancier set of makeup for the same shitty western chauvinism worldview of American excellence and Manifest Supremacy.
Nah, I'm done with this trite garbage. Go proselytize to idiots, I'm not one of them.
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
But unlike some of the others, I’m hearing anti-AI sentiment from a wide range of people who don’t even use social media.
> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.
> ...
> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.
> ...
> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.
I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".
The jingoism is off putting. I think Daniel says it's a political necessity: https://x.com/DKokotajlo/status/2075261194978640096
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
Edit: Also, definitely not a Chinese op. The authors are prominent Americans, and are the folks responsible for the AI 2027 forecast that has pretty accurately predicted the current state of affairs today: https://ai-2027.com/
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
If you don't care about getting the drone back, it does simplify the problem somewhat.
Meanwhile, golf courses are a traditional green space where people in a community gather for both work and leisure. They're not ideal themselves, but they at least provide some benefit against which their negatives can be weighed.
If all you hear from critics of data center building is water use complaints, that's strictly because you've chosen not to listen to people.
If the equivalent numbers for electricity and water usage were being being used for streaming video, I seriously doubt people would be demanding no more Netflix data centers. The news story would immediately die.
Personally I would happily close down all golf courses and put them to better use as literally anything else.
Even just making them public parks would be great
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
We already have at least 5 companies only in the US. Your whole premise is false.
I am not sure if alternative reality fiction is the best way to approach real and serious AI risks.
I am also not sure, with the amount of emdashes and the style of prose, that the entire article was not AI generated.
AI is going to be a mature scientific field. There are going to be efficiency improvements in training and inference. New paradigms are going to emerge with better multimodality, real time streaming and real time interfaces. Models are going to converge on the limits of our data available for pre and post training, improvements will be incremental and spiky in domains.
I am not sure who the AI 2040 article is for. I suspect it is intended to be a digestible piece of media for the financial class.
AI is going to be a useful technology and its impacts across the economy and global will be broadly distributed. Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones. Maybe the argument is that in verifiable domains, such as model training, AI models can supercede humans. I don't think so. A human's high level thinking, our incredibly more efficient semantic/neural compression, our ability to switch tasks and achieve the creative insight is not replicated through the current paradigm.
For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)
Without massive amounts of investment, AI development stops dead.
In this post, they hand wave about the USA being able to acutally 1) build concensus locally for regulation and 2) the rest of the world actually follows suit.
It fails to understand that actually the progress of AI is not actually the gift of the USA. It requires a constant supply of things from china.
Also its assuming that having 74 billion agents doesn't cause economic distortion. Like what value are these agents generating that justifies them being run?
I really wish people would just ignore this for what it is: bad sci-fi with an incomplete world.
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
An easy choice to make if the alternative is everyone dying instead.
Neither of these come to pass. The first because we don't live in a science fiction universe. The second because AI is a completely open source technology. You can literally make your own models and train them yourselves using state of the art techniques published in free papers. All you need is GPUs and smart people who can read (and now that AI can code, you don't need the second bit). These people are doomer quacks too caught up in fear and excitement to think rationally.
Their expectations and assumptions are all wrong, from the basic understanding of AI in general (like how you don't need a billion dollars or "American Brains" to make a half decent model), to the misunderstanding of market realities and competition in China and Europe. The US doesn't have a monopoly on chips, or on smart people; Huawei chips work fine for AI training and probably a fifth of the US tech sector's best workers aren't American to begin with. There's many forms of AI in many places; munition, economic driver, laborer, generic tool. It's now a part of the world, the way the Internet is. There's no keeping the genie in the bottle. AI doesn't take over the world; it's integrated with it. It's boring.
In a few years people will forget the doomer predictions, the way every doomer prediction that never came to pass is forgotten. We didn't nuke ourselves, the internet and television didn't rot our brains, the radio and telephone didn't corrupt the souls of Americans, the newspaper didn't incite anarchist riots. Every new technology freaks people out, and we adapt to every new technology and make it boring and normal. That's humanity.
Plan A seems like a good start and am glad an effort is actually being made to address any potential dangers. The only weak link I see is that there is no way for inaccessible, third-world countries, non-aligned states, and malicious wealthy rogue agents to be regulated. All I hear is a way for regulating companies that, themselves, legally have to answer or are bound to their host nations. Basically, I don't see a way to hold non-aligned states accountable.
I see a lot of focus on the well-being and protection of AI, which is important to country economies but, the folks that have been affected by layoffs, not necessarily due to AI, plus the workforce that are now feeling the negative consequences of the AI burn are justified in being worried. Anyone that feels the need to criticize them, clearly is not being affected by AI in the same way. Job loss will lead to economic and population destabilization, far worse than anything that has transpired in modern times. Hopefully, those being squeezed now won't be ignored.
They are also claiming that China may go to war with the US if our AI is better than theirs. They are coming up with scary scenarios which realistically won't happen.
>The problem with an intelligence explosion is the "explosion" part.
It's not a literal explosion. If we explosively ended world hunger that would be a good thing. Similar to having an abundance of food for everyone, having an abundance of intelligence is wildly beneficial to society. The article doesn't mention it but an explosion isn't guaranteed we could just see a plateau of capabilities due to bottlenecks of resources needed to power AI, time needed to run AI, and limited interaction with the real world. AI can't run science experiments on its own by the 2030 doomsday timeline
I imagine any populism movement will require rampant fearmongering to get a result. Considering the rough present alignments, presumably blue tribe focused propaganda will involve climate and inequality focused fear and red tribe focused propaganda will involve job loss. Grey tribe positioning is the P(doom) meme where everyone is rewarded for a high-P(doom) estimate.
Whether they actually actively oppose those things to the point of impacting building permits, that's a completely different matter. It really doesn't take much legislation to make golf courses economically unviable and force them to close, especially if you've got enough population within 30 minutes to support 22 of them (I speak from experience, I helped write a water reclamation ordinance that shut down at least one in my SoCal city)
"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”
Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.
Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote
The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.
Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?
A solution is strict legal liability. Corporations must be strictly liable for harms. That liability should be higher when AI is involved. Such liability may not be waived by contract, forced into arbitration, or devolved upon a third party service.
Then we let the plaintiff's bar and the insurance adjusters price that risk.
Something like this turns on in the EU in October 2026.[1]
[1] https://cybernews.com/security/eu-will-hold-tech-companies-l...
One where humans are increasingly pushed to the periphery to make room for data centers and the human population is subsumed by robots.
Who is this future serving? Not me. Fuck this.
The only choice here is to go to sea and get away from the crowds and the bots. Bots don't like salt water much, so I'll see you out there.
- have you seen the actual problems in other countries outside america?
- have you been to any country outside USA and the countries in europe?
- have you taken a trip to any developing country and stayed there for a month?
- have you seen what sort of daily struggles, political systems, bureaucracy and work exists in developing countries?
Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities. Furthermore, AI agents lack any notion of an identity, so certainly are not capable of attaining legal personhood or being sued or fired, or owning property. I think slop burnout, cybersecurity, loss of privacy, even environment issues are far more concerning and real issues arising from AI than alignment or the prospect of mass labor displacement due to AI.
The Plan A proposal estimates that the ownership of ~96% of AI relevant compute hardware can have its ownership traced, since the companies selling are very few.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
Why would it be inflationary?
https://www.tobyord.com/writing/inefficiency-of-reinforcemen...
This is similar to that other exponential, which happened with CPUs - we ran out of true geometric scaling in the mid 2000s, and everything else supporting Moore's Law has been cleverness that arrived in the nick of time, supported by a bit of marketing, and very optimizable benchmarks, far from guaranteed gains coming from making a single physical metric better.
You say it like it's a fact, but in reality everyone sees the phenomenon of AI slop.
P.S. Information search and retrieval if the best and most direct way to use LLMs.
because it is. Previously: https://news.ycombinator.com/item?id=43571851 / https://ai-2027.com/
Which predicts that explosive growth of robot production will lead to problems such as
> a deflationary debt spiral, where the AI and robot companies can’t pay back loans in dollars because the robots and AIs are worth nominally less than the loans written the year before.
In other words, the companies go bankrupt because they produced an oversupply of cheap goods, the bubble pops, and there's less new investment for a while. Plenty of precedent for such a development.
But instead of adjusting their predicted output growth downwards accordingly, they instead propose that
> One way to solve this could be for the loans to be denominated in AI and robots, so the companies pay back the loans with some percentage of the AI and robots instead of dollars.
Try doing this today with a battery factory for example. You expect that battery prices will fall to the point where the revenue from selling batteries won't ever cover the cost of building the factory. So you propose to a bank that they'll be the ones to build the factory, and you'll borrow it from them (not paying rent?), make your batteries, then give back the factory when you're done. All the profit is yours, all the risk is theirs! Which is of course why a real bank won't agree to this, all you're going to get is a dollar loan with the factory as collateral.
I think this is the reason why you have the tendency to propose some freeze-all policies, full control or similar. If you want to find the equilibrium, you need to accept that it will be a controlled equilibrium, most likely on a saddle point, with underlying process changing all the time, requiring fast changes in regulations. Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
The question (7:35) is "Where would you like to see people investing more time?" And Sam seems to be saying AI safety, I guess? This is 2015, and he refers to founding OpenAI. Based on his actions since then, yeah, seems like it's not a joke to him. This is Altman we're talking about.
I am extremely skeptical about whether this is even possible and even less convinced it’s a likely scenario but it seems like a good path.
I think it should be obvious that he understands that LLMs are trained via next-token prediction.
[0]: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...
There are multiple teams working on adding long-term memory to AI. This is not a fundamental problem.
Fine-tuning to store memory in the weights is something that almost works right now, btw.
I'm confused if this is satire, sarcasm, or genuine belief. If this was the case, then AI companies should absolutely remove the "it may make mistakes", because doing mistakes would imply that "the very best human knowledge and expertise" is what actually fails, and not the AI.
With that being said, I'll still urge people to visit a professional therapist for health problems and I generally still trust human knowledge workers for critical scenarios. I will reconsider your claim when chatGPT can effectively play Yu-Gi-Oh! (or at the very least respond with the correct rules appropriately), which is a significantly lower stakes scenario than betting your entire company on its aptitude.
Would the US government not pour enormous resources in AI labs if needed, knowing that China might be doing the same? What happens if an adversary develops an AI capable of finding and implementing exploits in every software run by your country's strategic infrastructure?
Your comment also implies that those who haven’t participated in colonization might not fear being colonized. Seems… off.
Even then, you can just compare the progress in open models. Leaps and bounds from where they were 6 months ago.
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
Not only this, but in the GPT 5.0 era, agents had 0 taste. Nothing looked good. It was the agentic version of the twitter bootstrap era, but worse somehow. Now, I would argue the average agent frontend beats the average human frontend. This isn't even getting into 3D applications in the GPT 5 era
Anyway, the models now reliably execute more than a human can fit into their own context. It's magic
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
Just purely organic YouTube Comments circa early '20s alone surely outslop any "AI" by a giant margin.
Everyone sees the markers, and it's a hot topic. There are maybe a thousand from-scratch trained models, and just few mainstream ones produce most of human-targeted content. In today's world, no surprise everyone knows the common patterns of those. That sloppy landscape is not just load-bearing em-dashes — it's a humble testament to their reinforcement learning.
Humans produce tons of texts, with all sorts of nonsense in it, without thinking it through. Our slop is just a lot more diverse. And mostly just spoken out loud.
> P.S. Information search and retrieval if the best and most direct way to use LLMs.
Yes, but not directly, if they don't know something they tend to hallucinate like mad, even today. YMMV, but in my experience they work best as actual "cheap" reasoning for building queries and checking out search engine results. Even if they misinterpret some result, more and more results will still steer it towards correct conclusions and it can point at some results that relate well enough to be useful.
> “Politics is the art of the possible”
I agree with your last statement.
> Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.
Without totally derailing the thread, this is also obviously why climate and biosphere collapse is not (and likely will continue not) to be addressed, e.g. Timothy Morton's Hyperobjects
https://jodavaho.io/posts/ai-jobpocolypse.html
The difference in the unemployment vs efficient employment model is mostly user driven adoption vs company mandated adoption, or centaurs vs reverse centaurs.
https://pluralistic.net/2026/07/02/canonization/#operate-ite...
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
I mean they might, but its not clear how they would do it, especially as they are reaching the point where its going to be expensive to borrow.
For anything health related all AI models show high levels of anchoring bias. I would not use it as a confidant, and be skeptical of claims. Even so, human doctors are also fallible and prone to cognitive bias.
I think the obfuscation is because human intelligence has been projected onto AI model capability. AI models only have a limited dimension of human intelligence, and in some axes orthogonal, and when I say distillation I refer to this.
Its really not controlled by central banks. Its influenced, but not controlled.
When central banks "print" money, they effectively just add money to the accounts of investment banks
But investment banks are also "printing" money. Double accounting effectively uses assets to double the available pool of money. If you then sell off those loans based on those assets, then you crystallise that new money. Investment banks are inflationary.
AI companies are racing to build AIs that are smarter than humans in every way. In AI 2027, we predicted that this would result in either extinction or irreversible concentration of power.1
Plan A is our positive vision for what should happen instead.
In this scenario, humanity delays the development of superintelligence until 2040, makes all AI research public, allows dozens of companies globally to catch up to the frontier, and intentionally enters a regime of mutually assured compute destruction.
America has two workforces now. The first is people, 165 million of them. The second is AI agents: millions of copies spun up and shut down every hour, working around the clock at superhuman speeds.
Most of their work is slop. But enough of it is good that people are paying ten billion dollars a month for AIs that can, in theory at least, do anything on a computer that an employee can.
There is one job the AI companies want to automate more than any other—their own. They haven’t succeeded yet; no recursive self-improvement so far.14 But they seem to be getting closer, and they’re pulling up the ladder behind them: the strongest coding AIs refuse to help competitors with AI R&D.15 Even as the most bullish employees admit that things are taking a bit longer than planned, the skeptics notice that their usual dismissals are starting to ring hollow. Why exactly will AI never be able to do my job? What’s the barrier again?
Congress is starting to pay more attention. They’ve long been hearing about AI: datacenters using too much water,16 chatbots encouraging suicide, Mythos hacking NSA systems—and of course, tech industry lobbyists warning that any whiff of regulation will make America immediately lose the race with China and spend the rest of history as a CCP tributary state.17
Now they step back and ask: Where are we going with this? What does the world look like five, ten, or fifteen years from now? Will there still be jobs? What if there aren’t?
One question weighs especially heavily on their minds: Who will control all these AIs?
Congress settles on an important part of the answer: Probably not us.18
They hold a series of tense hearings on AI. They read the 2016 OpenAI emails discussing how OpenAI was founded in order to prevent Demis Hassabis from becoming dictator.19 But who is preventing Sam or Elon from becoming dictator? Congress is unsatisfied with existing responses.
The result of this wakeup is the AI Transparency Act of 2027, an omnibus bill that does many things, some good and some bad, but doesn’t fundamentally change the situation.20
Incremental AI Policy Wishlist
Our main recommendation is to begin negotiating something like Plan A as soon as possible. But in this scenario, we depict Plan A happening imperfectly and only in the nick of time. So here is a list of less ambitious ideas that still help.
The 2028 election cycle is heated, as usual. AI is the biggest topic. The datacenters now under construction cost twice as much as the entire US military budget.23
Most white-collar professions are seeing disruption like software engineering saw in 2026; such jobs now heavily involve managing AI agents. AI companies have industrialized the training process: Executives say “let’s move into [profession] this year” and then the company interviews professionals, buys data, creates training environments, etc. until their AIs get traction. Then the AIs rapidly improve as they are used more widely in the field and accumulate more real-world data.
Other countries are starting to get scared and angry. It seems like a handful of US and Chinese companies are on track to automate all the white-collar jobs. Power is concentrating in the US, and in particular in the President plus a handful of tech CEOs.
AI experts warn that the intelligence explosion is near. By speeding up AI research, the AIs will become even more competent, speeding up research even faster, making them even more competent, and so on. There are complicated dynamics about bottlenecks and hardware limits governing how fast this process goes and where it ends, but it seems like it might go very fast and end somewhere very far away.
On the default path, the next presidential term will see AIs that are far beyond human level, created entirely by AIs, themselves created entirely by other AIs, without any human in the loop since several generations back. Will those AIs be obedient, aligned, etc.? Why? Who will control them if so? How exactly is all of this supposed to end well?
Having put humanity on this path, the AI companies find it acceptable. But most people don’t. Forget thinking about his legacy—the President is starting to think about what’ll happen to him after he leaves office and the world gets transformed.24 Both presidential candidates keep getting asked what they’ll do about AI, and try out increasingly dramatic ideas on the campaign trail. The discourse bounces back and forth across all of the options displayed below, and more.
Eventually the President and his protégé converge on one plan; the opposition candidate converges on another. Then it’s Election Day.
Most of the discussion around AGI is highly speculative. I am not saying AGI could not exist, and it is a term that has historically been loosely defined. Decades of coming science and research will tell.
A recursively self-improving AI has strong first-mover effects. That isn’t fundamentally incompatible with commoditisation if there is literally only one path to super-intelligence and you can have AIs at different rings on that ladder co-existing. (Not technically commoditised at that point. There are still different rings. But close enough.)
But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.
This is only true at a given AI capability level, no? e.g., if AI at the GLM-5.2 level is commoditized, all that suggests is that there's no recursive self-improvement easily possible at the capability level of GLM-5.2. (And with the harnesses for it that exist so far, etc etc.)
If I observe commoditization of a given tier of model capabilities at a given point in time, this seems to say little about what's possible with models six months later, or models that are undergoing proprietary deployments at that very moment inside the major labs, or even models that are notionally available for public use but have had recursive self-improvement adjacent capabilities intentionally nerfed (e.g., Fable).
(I might be misinterpreting your comment tbc - if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree.)