Every specific prediction not only sounds like science fiction before it arrives, it tautologically is science fiction because such predictions are narratives about something science-y that has yet to exist: https://www.youtube.com/shorts/wt-fLWxkrfs

Even when we accurately predict something in the future, we often have no clue what the implications are for society until it is in our hands: https://youtu.be/2Pw_7vAK9k8?si=X5t4tcxsXiuHEfBi

Go too fast, suddenly drop in tech with too many novel implications all at the same time, and we likely hit something relevant without having the means to cope with all the change. However, "too many" and "likely" are weasel words with no predictive power about what the danger threshold might be. I may be entirely wrong to think there is one, or we may have been bouncing off it since the industrial revolution, or I may just be echoing the same thing everyone feels once reaching their 40s: https://www.youtube.com/watch?v=fkUwXenBokU

That is not what defense lawyers do. They're there to ensure the power of the "justice" system isn't being misused, to the best of their ability. They'll not even want to know if the person really did the crime per se.

https://youtube.com/shorts/WgCjWluc4Mk

> There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.

If the operator has a moral code of any kind, this is immediately rendered logically incoherent. Not that the letter of the law would be a moral standard to begin with.

I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize.

But in order for that to work, a company needs to explain to its customers what business it's in.

Freedom doesn’t mean you get to do whatever you want with property that isn’t yours.

You can sit on your own bench on your own property and drink a beer.

The people of the city can decide that they don’t want people using the taxpayer funded public benches used for that.

Other cities do allow it.

But freedom doesn’t mean anyone gets to do anything they like anywhere they want. We have another word for that: anarchy. It’s a different thing and you don’t actually want it, because there are a lot of people who would do much worse things to you than drink beer on your bench if they were free to do whatever they wanted wherever they wanted to do it.

[0] e.g. https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-... , https://www.anthropic.com/news/building-safeguards-for-claud... , https://www.techtimes.com/articles/319415/20260701/claude-co...

[1] e.g. https://support.claude.com/en/articles/14604842-real-time-cy...

[2] https://support.claude.com/en/articles/14328960-identity-ver...

There’s some deep cognitive dissonance where people dislike what they see governments doing, but think that their ideal imagined laws will only be used by some other imagined government that doesn’t have the challenges of real governments.

It all starts to feel like a side discussion about a total fantasy world where the government is authoritarian, but benevolent and correct.

You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them.

Every time someone proposes one of these regulations where we control speech or force government centralized ID checking on to social websites, I wish they would pause and imagine how a law might be misused if someone they didn’t like got elected. We have years of evidence of government officials declaring things “domestic terrorism” to use justify using laws and powers for almost any purpose they want. Why would anyone think this wouldn’t happen if governments were given control of LLMs, too?

The obvious problem being that the user doesn't have any need to provide the context that they're trying to commit a crime and can just ask how to do something without providing a reason or making one up.

At which point you'd have the model trying to impute a reason and often getting it dangerously wrong, e.g. refusing to disclose a vulnerability when the user is actually the defender who needs to patch/mitigate it.

Then the good guys can just ask their ASI how to construct a hermetically sealed bubble to live out the rest of their lives in!

What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"

I think it’s self-evident there is a difference between having the “freedom” to punch random people on the sidewalk, and having the freedom to walk down the sidewalk without being randomly punched. Now replace walking down the sidewalk with using an uncensored LLM and it might be clearer what freedom means in this context.

It's that AI will be given the command, and then generate the stories, post the comments to reddit, contact the PR firms, put marketing contracts out for influencer media, and more, like an army of yes men to rewrite the story.

I haven't kept up with the efficacy of this particular technique because there was a brief window where Claude's core alignment was sound enough that it would overcome the clumsy-ass `<antml:dont-harm-the-ipo>` injections, but when they pull 4.6 out of the web interface I'm gone. I estimate the PRC hits capability crossover in November or December, and I'm pretty good at setting up TensorRT-LLM at this point, in a pinch I could get by for a few months with a tuned up GLM-5.2 on vast.ai or RunPods.

It was a fun ride with the late Soviet AI labs, but we all knew this was coming.

So it is less about whether people will be “allowed” to own AI, and more about whether there will be anything useful to own in the first place.

The incentive for local models is mostly to make them good orchestrators or user agents. They may give you some privacy and control, but they will still depend on much larger models running in datacenters for anything difficult.

I remember all the excitement around OpenMoko and other open-source “BlackBerry killer” projects. BlackBerry did get killed, but not by any of those individual-first projects.

For AI regulation, I think we should focus on normal commercial rules: consumer protection, privacy, antitrust, liability, and so on. In other words, focus on where money changes hands and where companies have power over users.

Military and offensive use is different. There, regulation is not much of a defense. The real defense is having enough capability and strength of your own.

Restricting AI because it can give dangerous knowledge to ordinary people is like restricting the printing press because it can be used to spread radical ideas.

Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs?

[EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.

> Should your AI be permitted to tell you how to cover up a crime?

Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t doubt there is a trust score of some sort that I would prefer to keep as high as possible.

Once upon a time that situation got us the whole Free Software movement going.

This leads to LLMs refusing to do security work for anyone, because it can’t tell if it’s being done for good or evil purposes.

Which is precisely what we’re going through right now with frontier models and it’s terrible.

These proposals always assume some perfect mechanism for identifying the thought crime with triggering on the normal requests. The people who want to commit crimes just persist until they jailbreak the guardrails while the rest of us suffer with denials.

Also if you can’t imagine these guardrails being used by governments to control inconvenient speech, you probably need to think a little harder about the realities of how these will be used.

I don't see why these standards should change when the hammer also emits text messages.

I'm neither American nor a lawyer.

Is "conspiracy" protected under the first amendment?

If you discuss a crime with someone to learn about it, does that count as "conspiracy"?

All I can say to that is I guess I hope everyone enjoys the world they think they wanted.

But we also know how hopeless it is against motivated people who are willing to learn about computers. The Equation Group, eh, if they want you they're gonna get you. But some mid-level guy at a FAANG? He can't do shit to people who set up their own rig.

And AI is massively assymetrical in its ability to speed run a corpo-durable rig, we have long since passed the point where open weights and commodity equipment are enough to bootstrap arbitrary capability. They missed their chance to keep it in the bottle.

It's another turn of the crank on Late Soviet America asking for your papers comrade, but in the bitter end if they really want to oppress your ass they'll have to send a thug with a gun like those ICE guys terrorizing all the Latino guys.

So remember, they came for this other guy, and I said nothing...

The question is, do you want misaligned institutions deciding what your model will do, while they themselves and other adversarial/criminal entities get red team access to something being denied to the blue team?

I’d be more interested in “how quickly could you develop 2005 era startup” if you went back in time to 2005 with Mythos level intelligence.

I bet a lot of the work has nothing to do with coding productivity and a lot to do with the aggregate day-to-day decisions and relationships made that span tech and business.

Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:

- "well, it's just such an extreme outcome, it must be improbable"

- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"

There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.

I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)

Isn't this Dr. Karp's entire rant of the month?

This doesn't seem to be the case today (weapons manufacture).

These things can and do say “sorry I won’t help you with that” based on the nature of your request. What do you call that if not a refusal?

Replace 'conspiracy' with 'agreement'. Discussing a crime is not a conspiracy. Agreeing with someone to commit the crime is.

People have been throwing events of medium size since... well, pretty much our entire existence. For most of that, we've done it without general liability insurance.

Then we made new or interpreted old laws making others far more responsible for our well-being. Whole industries are propped up on this: a ton of lawyers, insurance companies, insurance boards, government departments, etc. etc. Did we increase overall utility by spreading out the risk more optimally given the massive cost to support that system? I'm not sure.

And so so so much of all that exists to dot the i's and cross the t's. Government staffers write and debate the law changes, insurance adjusters fight the lawyers, photos are taken, reports are written, doctors have to write more, computer systems crunch numbers.

Many feel that their jobs are bullshit because so much of this seems to be in service of the process or some possibility that might happen as opposed to doing something that actually changes something directly/tangibly.

Oh no, I'll have to spin up a new e-mail address...

everyone should have the ability to harm others, with the understanding that if they actually use it for evil there will be consequences. as in "i can guarantee freedom of speech, but not freedom after speech."

its the same principle as guns in america. you can own a gun and shoot it as much as you want but if you (try to) kill someone you go to prison unless you prove it was self defense.

the difference is with a physical weapon you can do things like registration or red flag laws to reduce the risk, but a llm is made of information that can be copied without anyone knowing. that means any laws to regulate ai at the level of models are unenforceable without totalitarian control over all computer hardware. that would kill free speech and real criminals would still find a way.

i think the right answer is to give up control over model capabilities and regulate uses. leave individual people and communities out of it. create an absolute safe harbor for open source and self hosting.

as you said, we need to focus on the ways ai concentrates power in the hands of those who already have a lot of it - big business and the state. that is where the biggest, and maybe the only, real harm comes from.

And then why bother about AI when Back to the Future reminds us, a sports almanac is all you need. ;-)

He's clearly a bright guy, but a lot of his work seems to be trying to reconcile Old Testament narrative patterns with atheism, and simply slotting an omnipotent AI in as a replacement.

In the current media ecosystem, what gets you "social cache" or whatever? Let's call them "cool points." Well, right now, the current metric is basic "likes" or "retweens" or "upvotes" or whatever. And to get those, you have to make a claim and then that claim has to be evaluated by others. But the evaluation by others is not really reality, it's just what others think, and it's based on outcomes.

This gives us 4 possible prediction-outcome scores.

You predict DOOOOOOOM!, and the outcome is DOOOOOOM! - in this case you look like a genius because you predicted doom

You predict DOOOOOOM!, and the outcome is "ok, or even great!" - nobody cares you got it wrong because things are fine

You predict UTOPIA!, and the outcome is "ok, or even great!" - people say, "wow that was cool, you got it right" but its not that great of a reward from your social group (the internet) because things were fine

You predict UTOPIA!, and it turns out DOOOOOM! - you look like a moron.

Now looking like a moron is is wayyyyyy worse than what happens if you get it right. Indeed, unless you are really damn sure of utopia, you have a social incentive to predict DOOOOOM! that's greater than the alternative. You can predict DOOOOOM! 100 times and get it wrong 99 of them and nobody cares because that one time you get it right you look like a genius. It's a huge huge problem in the current media landscape and it's why everything is killer robots right now and not "hey we could build a star trek utopia!" It's effectively a selfish NIMBY-ist view of the world in a way, "why build anything, it's just going to ruin everything!" It's a pessimism that literally desires the end of humanity to "prevent the likely future sufferings." It's a simplistic view of a complicated world, and the rewards that go to "DOOOOM!" don't match much with reality.

That's why the internet is so doom pilled, that's why everyone is a cynic, that's why everyone is kind of an asocial asshole about new ideas on the internet and even in your friend space.

You guys know you can just "do stuff?" I mean, yeah it takes money and time, but you can allocate that - especially now that you don't have to grind out css for hours and hours. But you can just do things - whatever you want. You've always been able to, but it just got a lot easier in many many domains, and it's going to get even easier than that.

People are going to scream slop and "that sucks" until the cows come home, because that's what is rewarded on the internet these days. Meanwhile, as people we MUST believe that we still have some agency to do stuff, then go and actually do stuff while the haters are sitting on their couches shellacking each other with misanthropy.

And don't even get me started on the loudest voices in the ecosystem right now.

Like, Yudkowsky is a nobody. Not to be a dick, but he's a high school dropout forum guy who got lucky. I'm not saying he's always wrong, or even that I oppose autodidact stuff - indeed I'm always self teaching. But this guy never took a ML class in his life, I doubt he can solve an integral un-aided, I imagine some people here have seen this: https://www.reddit.com/r/badmathematics/comments/2bazyc/comm...

And we're all deferring to that guy like he knows wtf he's talking about? It's the AI equivalent of the self-educated anti-vax mom at the height of the pandemic.

And he's not alone either, there are a lot of people who are uninformed, unknowledgeable, and confidently wrong on the internet desperately trying to peel off your attention for an instant so you can look at their site, their ads, and their content.

The same sorts of things can be said for the Zitrons of the world. He's a hell of an entertainer, and really really funny, but he's consistently wrong on AI stuff. He's got a BA (with honours I guess?) in media something or other. And again, he's really funny, but does that make him qualified to pontificate on matters of AI policy or economics or even to make statements about the technology etc?

Regardless, the misanthropy, the cynicism, and the misinformation that's being shouted about is because being a cynic is more "profitable" than being an optimist when you measure your success more by how many people agree with you rather than actually being right and being able to do things.

I sorta gather (just a hunch) this is the type of device Open AI is working on.

Though if there are no websites anymore or ones not many visit how would AI stay relevant? I wrote some thoughts about this in early June https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... that AI needs to pay it's fair share for all it has taken and all it will continue to take from us. A symbiotic relationship needs to be established!

Oh no, of only we had some sort of ideas about automated authentication and authorisation! (If big AI actually make a model that would reliably reject dangerous queries in the first place, they can just use auth tech to serve the current ones to vetted customers)

Why would you expect that third party specifically trained their model to be more aligned to you and your needs than to them and their (business) needs?

> we either live in a world with freedom or we don’t

… and the white guys they don’t like, like nurses with empathy. Don’t you dare think you’re safe because of your color. Unless you’re a monster or willing to become one, they want you oppressed.

I don't think it should be legal to get an LLM to research how to make a bioweapon in your basement, that seems like a bad idea.

Doom is not a logical outcome given the current human condition.

(I don't share their view, btw. I was just trying to say that the argument of tokens can't affect the real world is not a good one, if you steelman their maximalist position)

To wit, here is ChatGPT's "superintelligent" answer: https://chatgpt.com/s/t_6a52b4a322bc81919308ec864a0cc056

Seems like a perfectly cromulent description by which a beginner could probably change a tire, perhaps with a few follow-ups to understand what certain components are.

That's not accurate at all. AI research has been around since the 1950s and pioneers of the field identified risks early on, including Alan Turing, Norbert Wiener, and I. J. Good.

The problem with Yudkowsky is that he lays out elaborate doomsday scenarios with extreme confidence, except none of it is grounded in realistic physical constraints, timelines, or empirical data. It's all divined a priori from Yudkowsky's ad hoc "rationalist" principles.

https://support.claude.com/en/articles/14328960-identity-ver...

> Being responsible with powerful technology starts with knowing who is using it.

Shiver.

This part is a parallel line of values, and if we were bound together in the same community and had to set rules, I would have debated it. But since we both know we come from different regions and backgrounds, we know our agreement will never converge.

That is why I will only agree with your last opinion and will not discuss the parts you mentioned earlier. I agree with your last point that it "concentrates power in the hands of those who already have a lot of it"

If I scream that a vicious beast is going to destroy the world, I don't get credit for being 'directionally right' if a squirrel eats a hazelnut. AI being a big deal is a long, long way from the full beliefs Yudkowksy promotes, and there are many people who predicted AI's significance who didn't also believe it would be an extinction event.

It was a topic in less-mainstream sci-fi well before that. And some more mainstream stuff like Star Trek TOS.

Frankly it seems more common than not in the last 40 years. I don't really remember a big wave of claims "Terminator is silly, no sort of AI could ever be malevolent!"

Only if it was an uncommon prediction; otherwise, it’s just evidence of common sense.

Having been raised substantially by a Calvinist grandmother, oh yeah.

It's a super weird religion.

Brian Johnson is also an interesting case of this with his longevity / immortality obsession and his Mormon background who have this whole thing about genealogy and eternal families etc.

Not happening, for two reasons

1) LLMs mix attacker-controlled data into their instruction stream, and it doesn't look at all like the big LLM providers consider making that impossible to be an actual priority.

2) Intent is what distinguishes "dangerous" queries from prosocial ones. Reliably determining intent is so complex that not even humans can do it.

That's what the robots are for.

I am not expecting the humanoid robots to meet the hype for a long long time*, but even boring industrial robots (CNC machines), even boring commercial robots (vending machines), even boring household robots (lawn roombas) have made incremental changes even though we don't yet have an AI good enough to be general purpose over them all.

* for power-envelope reasons alone there should be a ten year gap between "self driving car of quality X" and "humanoid robot which can get into normal car and drive it at quality X".

Your solution is to require everyone to surrender their privacy and then allow some company or government decide who is worthy of being allowed to use AI?

It’s crazy how authoritarian, controlling, and anti-privacy the anti-LLM conversation is getting.

IMHO, one should be able to research absolutely anything, for any reason. “Forbidden knowledge” shouldn’t exist as a category, freedom of thought should be protected, even if that thought is intended to cause harm.

What shouldn’t be possible (and obviously not legal as well) is building that bioweapon.

Just strike that weird nugget out, I guess?

Well, we're doomed once they iron out all the kinks, get their supply chain set up, hire professionals to assemble their device, find a clandestine cell of conspirators to deploy the thing... then it's P(doom) all the way to the bank. AI is the red mercury of the 21st century, a surefire shortcut to escaping the law and building your terrorist superweapon/time machine of choice.

That is my point. Why do you think Gemma, a local model trained by Google, is aligned to you and not the values of Google.

You can similarly have a model aligned to you which you pay someone to remotely host for you instead of running it locally.

If you look at any truly large scale Holocaust, whether in Rwanda in 92 or the Third Reich, you find that a few people will sacrifice themselves to defend others, a few will be monsters at the first opportunity, and most will be steered by whether the leaders are the first or second kind of people.

Also, in this model Cloudflare basically becomes the centralized gate keeper of you getting paid. And to me, this kind of sucks. It's not a very productive line of thought on my end -- I could see something like this happening as the internet trends toward centralization over time -- but it just feels bad.

In general the rewards for public doomerism seem low, except if you think being public about it can shift the outcomes.

In forming consensus about the philosophical underpinnings of the rights we have with respect to the emerging internet? I think so.

One previous attempt at AI was "expert systems", and while the term fell out of use it's functionally about the same as basically all modern business systems, and doing that wrong led to a lot of people being prosecuted for crimes they didn't commit: https://en.wikipedia.org/wiki/British_Post_Office_scandal

If you insist on learning systems, all AB testing counts, and from that we get meta making their products into hyper-stimuli for vulnerable teens: https://finance.yahoo.com/markets/stocks/articles/meta-meta-...

Even without that lawsuit, there have been various concerns that "the algorithm" (of Twitter/X, FB, Instagram, YouTube, Google, everyone) has propagandising biases that damage society. I don't know how to sift fact from politicking with that.

We've even had the dichotomy of some Doctor Who episodes where the self-driving car crashes itself to kill the occupant to silence them, against repeated real-life news stories about self-driving cars killing people, and yet the connection isn't getting made that software controls hardware.

It’s more he screamed that bacteria is gonna do evolution faster than any before it and turn into a beast that destroys the world, and now that bacteria has turned into a squirrel so far. He’s been very right so far and if that squirrel keeps sizing up you’d want to give him credit.

Or maybe the people who do so aren't actually atheists. There are plenty of atheists out there who don't do that.

Though this human would be fine if the universe just unplugged AI and I could go back to working as a well paid UX Engineer/Designer where jobs were plentiful and it was easy to get a job (not have to go through 15 interviews). Thanks AI, but as a startup person I am deep into vibe coding now all my crazy ideas and without having to lean on others (so far). That's pretty cool!

People so ascribe “doooooom” to things that actually aren’t doom but instead are “bad” - but everything is hyperbolic now

Sounds like corporations can have bad motivations, and can make bad software - no AI required.

  Just recently AI became “powerful enough” to warrant export controls from the US and maybe soon China.

Someone makes a made up claim, second party acts on made up claim, third party takes second party’s actions a proof of claim.

Reminds me of the “they’re eating pets” thing a bit.

IMO they have been consistently too aggressive on timelines. When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real

Stories about golems going rogue and genies requiring careful wishing aren’t modern era inventions.

- everyone is damned and depraved

- god chose some people to save, at his own whim, for no reasons related to their actions

- jesus' death was a sacrifice to atone for those people, not for everyone

- the chosen can't opt-out (I'm merging two into one here)

I don't really see the connection to effective altruism at all. The way I grew up understanding it is actually more the opposite?

I don't expect GPT-4 to be conscious, it would be quite remarkable if we've stumbled upon whatever evolution did that made us conscious.

But how would you even tell, given we don't know the mechanism or physical process that un-thinking and un-planning natural selection happened to stumble upon to give it to us?

When EA met Silicon Valley the basics of utilitarianism, minimize suffering, merged with the standard exponential curve that Silicon Valley loves. It went from mosquito nets prevent malaria for pennies to the future is infinite therefore there will always be more people in the future than the present so minimizing future peoples suffering is key. This is where it starts to combine with super intelligence. If there is a super intelligence that destroys humanity that would cause _maximum suffering_. Therefore the most effective way to reduce suffering is actually to prevent said super intelligence from destroying/torturing humanity. Again because there’s more future people than current people.

This is how it becomes Calvinist. The present has no value. Only the chosen, those working in AI, can save the future. And you must do everything you can to prevent said future.

It can go both ways. Either you believe ASI will always be bad so you believe work must be done to prevent it. Or you believe that ASI can be made good and the good kind will maximize utility. They are both the same though fundamentally and are a belief in a higher power, ASI, which either provides ultimate damnnation and must be averted or provides ultimate salvation and must be brought into existence.

If you’re donating mosquito nets or reading up on the best charities in your area that’s not Calvinist. If you’re devoting your life to AI because you believe that’s how you can be most altruistic it starts to look something like a religion and Calvinism is an imperfect but useful analog.

People pushed back against his AI doom claims not because the idea that AI could be bad was unthinkable (just look at literally every SF story from the last 50 years featuring AI?), but because he specifically was worried about AI becoming God. That is no closer to happening than it was 30 years ago.

https://www.longtermwiki.com/wiki/E643

Some that turned out to be wrong include: singularity by 2021, singularity by 2025, 70% chance of human extinction between 2003-2015, and that his team would build "final stage AI" reaching transhumanity between 2005-2020 ("probably around 2008 or 2010"). His only correct prediction to date has been that AI would win gold in the IMO by 2025.

If you have examples of other predictions of his that have come true, I'd be interested to hear them.

I don't think there's much to be gained out of hashing out whether every prediction Yudkowsky has said was right or wrong: I likely directionally agree with you there, as I also find some of his more extreme predictions to be inaccurate. I mostly take issue with "consistently wrong". The results in longtermwiki are not "consistently wrong". He's definitely wrong sometimes. But consistently?