E.g. research paper, law makers, lawyers, state policies, notaries,...
These are much longer content and thus statistically they will disclose a better guess at AI generated content.
Asking another AI to paraphrase will not erase the mark (which they are unaware about) but rather cumulatively add their own mark and make it easier to detect.
The problem is not to use AI, but to endorse the responsibility of the content you (as a human) deliver and somehow make sure that fake-news, biased content or unverified output is detected as early as possible.
Maybe some part of it is that the deluge of slop is uncovering how poorly/sloppily these social institutions were working in the first place.
Whatever happened to just delivering the best product or service? Why must tech be full of ninnying nannies that act against their users, "for their 'safety'‽"
It would look like a lot of little signatures on little bits of text, and then larger signatures on a collection of those chunks once the larger chunk exists. It's not that hard. It's just a lot of signatures.
But this scheme could only ever prove that this bit of text was made by a given AI, and validate anything else ever included in the signature hasn't been tampered with. It's not hard to work up a scheme that proves (within reason) a text was generated no earlier than some date by incorporating some sort of information that could only have been known at that date so that could be validated. But this isn't even a step in the direction of proving that something was made by a human. And that's assuming the private keys stay private, which is its own tricky problem. If a private key ever leaks anything signed with it becomes invalidated.
I was assuming it was something like SynthID rather than just sneaky invisible unicode but it's hard to tell from the description.
But if we're talking about deterministically taking some watermarked LLM output and having a function removeWatermark(text), it won't necessarily be "trivial" to remove, because the watermark function itself need not be public. Only the API that tests for the watermark need be public, right?
Anthropic's magic watermark could be, like the article mentions, something like "every 7th semicolon has a N% chance to be a comma where N is the sum of the last X characters mod Y, and every character in the bit range q1...q2 has a Z% chance to..." etc etc etc. And if Anthropic controls those variables, it would be very difficult to determine the rule, even with some pretty advanced analysis (I would assume). And keep in mind, that example rule I mentioned is pretty naive, too. I expect the actual rule would be way more advanced and not so straightforward as "swap every <charX> for a <charY>"
People underestimate the value of rules that only take malice and a little knowledge to break.
And they tend to exaggerate that underestimation if they... don't like the rule.
It's not clear to me that it's impossible, or even especially difficult, to make something that survives a casual LLM paraphrase. Remember, all you need to encode is a single bit of info. There's a lot of space to redundantly encode that signal.
just make an API that returns the string distance between a previously generated paragraph and the query?
that would sidestep this whole problem class.
regulators could even specify how that has to work.
what am i missing?
Currently I can recognize AI text because I read thousands of ai generated text. I know that 110% of yahoo finance news is generated. I don't want to read an AI generated personal blog, but if I do what's the problem really? Other than the companies distinguishing AI text for getting better training data, how do people benefit from watermarked text exactly?
Here's what I grasp: The AI system scores each token and then selects tokens based on those scores. If we encode something in the token selection routine ('in order choose the 1st, 3rd, 1st, 5th, 2nd, then 1st highest scored tokens'), we can identify AI-generated text by comparing sample text (ST) to the expected text (ET) for that prompt.
1) How do we score the tokens for the ET without the original prompt? Even a Markov-like process needs to start somewhere.
2) To recreate ET don't we need to maintain, until the end of time, the AI state - entire model and code - at the time of ST output?
3) Doesn't #2 require maintaining all states for all AIs? Often you won't know when and from which AI system the ST might have been generated. What happens when an AI vendor goes out of business?
4) To recreate ET, don't we effectively have to rerun the prompt? Won't rerunning it for every verification increase most costs of AI output by an order of magnitude? Most of what AI vendors do would be ST validation.
Isn’t the field notably corrupt. I’ve seen some pretty batshit stuff make it at least briefly into reputable journal. Anyone remember the “5G Covid” and “The world is a living ball of flesh” schizo dermatologist? That guy at least briefly got his stuff published by the NIH.
We don't appear to be talking about cryptographic signing here. (That would never work for the problem because everyone expects unsigned text anyway.) We're talking about:
> It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated.
As in, trying to force ChatGPT output to contain intentionally crafted ChatGPT-specific LLMisms that a human is unlikely to imitate, even one who reads a lot of ChatGPT output.
But how is this implemented? It's a few lines of code to implement a basic "identify and strip/replace any non printable ascii, unicode ..." or whatever. A screenshot/OCR will also do this.
SO at the end of the day you're left with some dumb rules like "you used `load-bearing` more than once per 500 words, that's AI!"
For Anthropic, it’s highly likely that the watermark is a SynthID type mark similar to the one that Sean is talking about (I actually ran the analysis here https://johnjwang.com/post/2026/08/12/how-claude-watermarkin...). When we get confirmation of whether all models actually are watermarked, I think we’ll be even more confident.
Of course it’s always possible that Anthropic has come up with a proprietary scheme, but I think it’s definitely harder to implement.
I think the game will be a cat and mouse game similar to LinkedIn and other websites trying to block scrapers: each iteration makes it harder for someone to figure out the watermarking scheme, but likely not impossible
They store some prompts and responses, not all, that's what you're missing.
Yes, this watermark will be easy to strip. It is still valuable for the vast majority of times where people just don't.
Most people are generally lazy. They upload text to LinkedIn full of "genuine", "honest" and "load-bearing".
They claim to only do this when you agree to this in your personal settings. Though Google does say they will train on it, unless you disable history and only use ephemeral chats. Anthropic has a setting for it and claims not to train by default.
Also it would be vulnerable to attacks and privacy problems. You could search for substrings about some suspected information, like "John Smith's medical records show advanced cancer" etc. Of course you'd have to guess the phrasing but still.
2. Attacker asks the LLM for the opening sentences of the book, it goes into the generated responses database.
3. Later, a malicious user shows that the first few sentences of the authors book are identical to a previously generated response.
Really, Anthropic (and the other labs that follow) are just trying to satisfy the requirements of the law so they can continue to serve the EU. Wether it's actually effective is something else entirely.
"The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around."
I'm reacting to the idea that "plain old regular text" can't be "signed" because they aren't "files". I'm observing that you can sign a stream of text, and probably other metadata, no problem. To my reading this really is about signing and not stegonographic watermarking, so we're in a context where for some reason the users in question want to carry the certificate of generation by AI and so the fact that this is trivially strippable isn't the issue at hand.
I read it this way because it seems to me clear that it isn't any particularly harder to do the stenographic stuff on a stream than a file (per zahlman's comment), so it only makes sense to be talking about this if we are actually talking about signing.
2. same answer
3. no, it just needs the previous text, private key, and the matrix math (CPU is fine)
4. no, see above.
The people that make the laws and the people complaining about AI don't know or care about the reality of the situation nearly as much as they care about being re-elected and feeling good about their social posture.
This wouldn't be helped by an AI watermark. What would help is if the reviewer used AI to look up the authors and see the authorship claims are dubious. The papers are still up.
I do think the academic publishing field is corrupt, which is why I'm not convinced it should be on AI providers to help bail it out of doing its one job (verification and trust).
- watermark-free generation
- the stripping of watermarking from the output of SAAS models
Any discussion of watermarking is dead in the water in a world where we are permitted to have these things. I fear for the future.
Additionally, think of cases like paying a lawyer or an expert for an extensive report or opinion on something. Wouldn’t you want to know if that is actually their carefully assembled professional assessment rather than the output of an LLM prompt?
1. So we can't score the first paragraph (or similar-sized block), and not short texts? Not deal-breaker, but a limitation.
2. Doesn't the score vary by each AI system state - its model, programming, harness, etc.? Claude's output today doesn't match Gemini's, nor Claude from 2 years ago.
Of course I would want my code formatting tool to normalize that all to plain 0x20 spaces. But it would still be a helpful "brown M&M" test of did you even read CONTRIBUTING and run the code formatter before submitting this PR?
Key phrase. And I'm not saying fraud in the legal liability sense. If you're not trying to hide the fact that something was LLM generated, then you have no reason to remove it. If you are trying to hide it, then there's probably a reason, i.e. you would face consequences for doing so, therefore it is fraud.
- not being locked into a provider
- not being forced to have your prompts saved by a possible competitor
- an alternative to the duolopy we quickly see forming
- offline access
I fear for the future without local models, much more than the future with them, and would rather everyone had access to a local model than be certain we catch everyone copy+pasting LLM responses. Watermarking would be cool, but it's not worth losing local for.
No, I will not chill. The war on general purpose computing is gonna get real hot real soon.
The European Union AI Act will begin to be enforceable in August 2026, one month from now1. One of the biggest new requirements is Article 50, which requires all AI outputs to be “detectable as artificially generated”. In other words, if LLM providers want to do business in the EU, they will have to apply a watermark to their outputs2: some hidden signature that can be used to identify AI content.
LLM text watermarking is a fascinating problem. Like the best engineering problems, it is theoretically hard to solve perfectly, but has multiple partial solutions: for instance, Google’s SynthID, and (as I’ll argue) some quiet Unicode trickery from OpenAI and Anthropic. It will be interesting to see how the AI labs navigate these tradeoffs before the end of the year.
I wrote about AI watermarking at the end of last year in AI detection tools cannot prove that text is AI-generated. It’s easy to watermark an image, because digital images contain lots of noise that the human eye can’t really see. For instance, you could apply a watermark like “these twenty pixels in these exact spots will always share a color”. Text is much, much harder. Unlike images, text is a very compressed medium: you cannot make any change to a sentence that a human wouldn’t notice (with one exception, which we’ll get to later). So how are you supposed to watermark it?
It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated. Any changes you make to apply the watermark will compromise the quality of the output. For instance, “every fifth letter is an ‘e’” would be a good watermark, but applied naively would make the AI output full of typos. Could you just let the model figure out how to fit the watermark? Strong AI models are smart enough to juggle this kind of constraint3, but it’d still consume reasoning time that would be better spent on the user’s problem, and make the model sound much less capable than it is4.
Do you really need a watermark? If you’re Anthropic, and you’re required to be able to verify whether your models produced a particular block of text, can’t you simply run the text through each model, measuring as you go how closely the model’s predicted tokens match each token from the text?
Not really. The space of “all possible Claude Sonnet answers to a question” is way larger than the space of “all possible watermarked answers to a question”. In other words, you’d get too many false positives for human text that reads like it was AI-written. It’s way more likely for a human to accidentally write like Claude than it is for a human to accidentally reproduce a watermark.
It would also be prohibitively expensive to run every Anthropic model against a piece of text in order to watermark it. The EU AI Act will eventually require labs like Anthropic to offer free watermarking services to every EU citizen (see Commitment 2). You couldn’t do that with the “run the model” approach.
As far as I know, the only AI provider to say they watermark text output is Google, who use a tool called SynthID. Here’s how it works.
When an LLM generates text, it’s generating a series of tokens (words or chunks of words). At each step, the model itself doesn’t output a single token, but instead outputs a full list of all (say) 100,000 tokens in its vocabulary, each annotated with the probability that that token will be the next one. Tools like ChatGPT or Claude Code will pick semi-randomly from the most likely options in order to get their outputs. This semi-random sampling process can be influenced in a detectable way.
For instance, we could choose a sampling strategy like “we pick the second most likely token, then the first, then the second, then the first, and so on”. That would still produce high-quality output, but you’d be able to re-run the model against the generated text to verify that the pattern holds. However, that’d make verification really expensive, and any slight tweaks to the output would break the pattern and thus break the fingerprint. Is there a better way?
Yes. SynthID is a process for assigning each token a “score” based on its previous tokens (for instance, sum the token’s ID with the IDs of its previous three tokens then take mod 5)5. To apply the watermark, the model adopts a sampling strategy like “out of the top five most likely tokens, pick the one with the top SynthID score”6. The watermark can then be detected by calculating the aggregate SynthID score of a block of text. If it’s suspiciously high, it’s very likely to have been AI-generated.
This is basically a version of the common advice that you can identify LLMs by use of the em-dash, except that instead of a list of keywords, it relies on subtle mathematical relationships between words that humans can’t identify. Because the process for assigning the score is trivial, it’s very cheap to run watermark detection.
Google have a complicated mathematical rationale for why SynthID doesn’t make the model dumber: supposedly the SynthID scoring is random enough to act like a normal pseudo-random token sampler, just one that leaves a detectable fingerprint on the outputs. But of course this is suspicious. For instance, it’s common to do inference setting temperature to zero, which always picks the model’s most likely next token. In that case, you can’t leave a fingerprint at all (or you have to ignore the user’s preference and pick the second or third choice anyway).
If you can’t alter the model outputs, can you still fingerprint the content? Well, kind of. I’m pretty sure OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you might go through and replace your normal ” ” spaces (unicode U+0020) with a three-per-em ” ” space (unicode U+2004), or a CJK ideographic ” ” space (unicode U+3000). These are called “homoglyphs”, and you can find more of them here.
Of course, lots of human-generated text uses homoglyphs. But it’s trivial to encode a pattern of homoglyphs (say, “every third space becomes a three-per-em”) that is much less likely to occur in the wild. Like the SynthID watermark, a homoglyph-based watermark can be detected very cheaply. A homoglyph-based watermark is cheaper to apply than SynthID: you could even do it entirely on the client.
I don’t think this is a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the ’ character in “Today’s date”, though they’ve since walked that back). In the last few years, I’ve noticed that when I copy blocks of text from ChatGPT and paste them into VSCode, sometimes VSCode marks some or all of the spaces as unusual Unicode characters7. Are OpenAI and Anthropic using homoglyphs as an AI-generated watermark? I’m not sure. But they’re definitely using homoglyphs.
The AI Act (specifically, its associated Code of Practice) requires watermarking to be “embedded within the content in a manner that is difficult for it to be separated from the content”. However, text watermarks can be trivially removed.
To remove unicode homoglyph watermarking, you simply have to replace all the homoglyphs with their “real” character equivalents. If you have access to even a relatively weak un-watermarked LLM8, you can strip out SynthID watermarking by asking that LLM to paraphrase the text content. Because the watermark is inherent to subtle vocabulary choices, re-wording the content will remove the watermark. You could even do it by hand, although at that point it’s not really AI-generated content anymore. Since there will be some kind of free public watermark testing tool, you can just keep tweaking until it comes back negative.
Moreover, the AI Act requires watermarking techniques to be “interoperable… as far as this is technically feasible”. That means AI providers would have to publish their watermarking process, and potentially even attempt to standardize on applying the same kind of watermarks. I just don’t see how this is compatible with the kind of security-by-obscurity that LLM text watermarking depends on. Unlike image and video watermarks, text watermarks will always be trivial to remove.
The AI Act and Code of Practice talk a lot about “digitally signed metadata”. The idea here is that you can include an AI disclosure in the file’s metadata itself, ideally in a way that cannot be tampered with (for instance, by signing a hash of the file’s contents). This signed-metadata process is basically C2PA Content Credentials. While you can remove C2PA metadata, you (theoretically) can’t fake it, so a file with “created by a human” metadata can be trusted, and files with no metadata at all can be held in suspicion.
This post is already too long to get into what I think about C2PA, but I do want to say that C2PA is not a substitute for text watermarking. It only really applies to files. In the words of the Code of Practice, that’s “a data format that supports attaching metadata (e.g., an audio, image, video, or containerised text)“. The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around.
I think it’s a fascinating question whether Claude Code has to C2PA-sign any HTML files or PDFs it generates for you. That seems kind of tricky to get right. But in any case, the AI Act also mandates some kind of actual watermarking as well.
So what’s going to happen this year? If I had to guess, I’d say that each AI provider (not just labs like OpenAI or Anthropic, but third-party providers like Fireworks or Groq) will stick a SynthID token sampler in front of their inference stacks. This might be limited to users in the EU, but it might not be, since SynthID is at least as good as a normal top-k token sampling approach.
AI providers will then offer a “check for watermark” page that re-tokenizes user-provided text, runs the scoring, and checks whether it’s above a certain threshold. Depending on how seriously the interoperability clause is taken, providers might even standardize on the same SynthID setup, in which case there could be a single EU-hosted “watermark this text” page.
I don’t think unicode-based watermarking is going to be considered compliant with the AI Act, but some providers which don’t want to set up SynthID might try it. Either way, technical users will be able to strip out the watermark at will, and there will be a plethora of tools that non-technical users will use for this purpose.
Well, for new systems; existing ones get until December.
I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks.
Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.”
I leave the analogy with AI safety guardrails as an exercise for the reader.
That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens.
The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works.
When this became public knowledge, OpenAI claimed it was just a model quirk, which is certainly possible.
All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text.
If you liked this post, consider subscribing to email updates about my new posts, or sharing it on Hacker News.
Here's a preview of a related post that shares tags with this one.
AI inference is obviously profitable
Many people claim that AI inference is unprofitable to serve, and thus must be subsidized by an ocean of dumb money from investors who believe that some future AI model will come to dominate the world economy. When that dumb money goes away, so will AI products. According to this view, LLMs are just inherently too expensive (in terms of money, power, and water) to be used in consumer products. In fact, they can only be used today by externalizing the costs: money onto VC funds and now retail ETF investors, power onto electric utility consumers, and water onto the communities where datacenters are built.
Continue reading...