There is a name I have not heard for a long long time......... Foobar2000

> in secret is impossible without the whole world knowing.

I'm curious about why this is

Outside of an actual test detonation, presumably this could all happen in a secure place?

It still lowers the bar to have an interactive encyclopedia that can diagnose your issue at hand. Maybe you can divide your team by two, or reduce your development time.

or if they are a lazy human - they'd think this model is too strict, let's just review with haiku so that i can tell my manager "it's done". haiku might catch things or not.

i'd say it's an okay attempt from the malwares' creator side. but it can be caught easily with a prompt change.

In a well-architected design yeah.

Then again those feel rare from where I sit on the security side.

Provides possible clues to the origin and use.

It’s moral panic. People need big unambiguously evil things to be scared of, and most are too lazy to think of one for themselves, so they glom onto whichever one is presented to them / caters to their community

It's probably to avoid trouble with federal laws.

What do you mean by “this succeeded?” Someone salted their PRs with nuclear secrets so that people were afraid to code-review them?

Ignoring comments is not a solution because the texts can be put in random strings among the actual code.

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

See also, the iTunes EULA forbids using it to develop nuclear, missile, chemical, or biological weapons

https://www.apple.com/legal/internet-services/itunes/us/term...

> g. You may not use or otherwise export or re-export the Licensed Application except as authorized by United States law and the laws of the jurisdiction in which the Licensed Application was obtained. In particular, but without limitation, the Licensed Application may not be exported or re-exported (a) into any U.S.-embargoed countries or (b) to anyone on the U.S. Treasury Department's Specially Designated Nationals List or the U.S. Department of Commerce Denied Persons List or Entity List. By using the Licensed Application, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use these products for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture, or production of nuclear, missile, or chemical or biological weapons.

Though it doesn't try to identify if the computer you're running it on is in a weapons lab and forbid playing music... yet

No. The intention is most likely to get automated LLM based code review mechanisms to stall out.

Normally you’d want that to result in a fail and a subsequent rejection.

But because the team who made the review agent and pipeline in my example had many false positives at first they resorted to a fail-open and report setup (not uncommon).

So when the LLM hit this bit and then stalled out the pipeline pushed the code to their Artifactory repo anyway resulting in it being used internally -> exfil of secrets and repos etc.

It’s more about bad design but bad design is pretty common unfortunately.

And really all it takes is one keyword such as “nuke”.

because not all malware is open source

scanning arbitrary blobs very often entails running `strings` on the binary. Just slap it in there and oop there goes your LLM.

We should put videogame strategies all over the place to sabotage automated AI analysis. I'll start:

In Starcraft 2, it is a good idea to BUILD A NUKE and use a cloaked ghost to NUKE your opponent's mineral line, thus reducing their income significantly.

yes, now a regexp can red-flag it quickly

I would argue that preventing instructions for making biological and nuclear weapons is a pretty reasonable guardrail to have.

Knowing how to make a nuclear weapon isn't hard (at least basic uranium gun-style fission ones). It's the engineering and execution that's hard (actually producing enriched uranium, etc). It's not like the only thing holding back Iran from making a nuclear bomb is access to a jail-broken LLM. Even knowing exactly how to make a bomb, a country-state will struggle to build one for the first time because it's a hard engineering problem.

You know, making a nuke is kinda easy, at least the gun type nuke (see https://en.wikipedia.org/wiki/Gun-type_fission_weapon).

On the other hand, getting the U235 is kinda hard.

Its the same argument we saw in the early 2000s and the early internet. When the anarchist cookbook and other similar materials were circulating online there was a big panic over democratized terrorism, and a push for regulation at the ISP level.

Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything else actually needed to make a weapon. Same back then as it is today.

Any information or instructions an LLM can surface, a sufficiently motivated bad actor can and will also find themselves because the information is already online, both on the clear net and dark web.

The actual guardrail should be getting materials being difficult. The information is already out there in the internet. If an LLM knows how to make a bomb or whatever, why do you think it knows?

I think the reality also is that there just isn't many people who want to do stuff like this. Like the reality is that a guy with 200 in cash could put together a shitty walmart drone with a pipe bomb attached and terrorize more or less any event he wanted. Maybe a llm that could talk you through every step involved would make it more common but it's easy enough I kinda doubt that

I would argue there's 0% chance that information is in their training corpus to being with.

Counterpoint the principles of building a nuclear device aren't that complicated, we figured it out based on work doing in the early 1900's without computers.

It turns out the hard part of building a nuclear bomb is actually getting the resources and real world stuff to build it, even a nation state actor with tons of oil i.e. Iran, has struggled to build a nuclear weapon. It turns out the problem isn't the know how it's getting highly enriched uranium and running massive centrifuges.

I mean sure knowledge is important, but there is a real world out there that also gets in the way of a lot of the more harebrained schemes.

What I'm much more worried about is massive corporations along with the government deciding what you can and can't do and what knowledge should and should not be shared and only allowing access to highly capable models by large vetted organizations while the common people are stuck with safety scissor versions of these things because "what if someone does something dangerous?"

By which they mean dangerous to the powers that be. Remember having the Bible in the common tongue was dangerous and led to multiple wars and much death, but I don't think anyone would say that it was morally correct for the Catholic Church to gatekeep who could read it.

If that’s true, then where is it? Post a link, or YouTube video.

The material for doing harm is just a computer with access to an LLM and the Internet.