The European Union AI Act will begin to be enforceable in August 2026, one month from now1. One of the biggest new requirements is Article 50, which requires all AI outputs to be âdetectable as artificially generatedâ. In other words, if LLM providers want to do business in the EU, they will have to apply a watermark to their outputs2: some hidden signature that can be used to identify AI content.
LLM text watermarking is a fascinating problem. Like the best engineering problems, it is theoretically hard to solve perfectly, but has multiple partial solutions: for instance, Googleâs SynthID, and (as Iâll argue) some quiet Unicode trickery from OpenAI and Anthropic. It will be interesting to see how the AI labs navigate these tradeoffs before the end of the year.
I wrote about AI watermarking at the end of last year in AI detection tools cannot prove that text is AI-generated. Itâs easy to watermark an image, because digital images contain lots of noise that the human eye canât really see. For instance, you could apply a watermark like âthese twenty pixels in these exact spots will always share a colorâ. Text is much, much harder. Unlike images, text is a very compressed medium: you cannot make any change to a sentence that a human wouldnât notice (with one exception, which weâll get to later). So how are you supposed to watermark it?
Itâs basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated. Any changes you make to apply the watermark will compromise the quality of the output. For instance, âevery fifth letter is an âeââ would be a good watermark, but applied naively would make the AI output full of typos. Could you just let the model figure out how to fit the watermark? Strong AI models are smart enough to juggle this kind of constraint3, but itâd still consume reasoning time that would be better spent on the userâs problem, and make the model sound much less capable than it is4.
Do you really need a watermark? If youâre Anthropic, and youâre required to be able to verify whether your models produced a particular block of text, canât you simply run the text through each model, measuring as you go how closely the modelâs predicted tokens match each token from the text?
Not really. The space of âall possible Claude Sonnet answers to a questionâ is way larger than the space of âall possible watermarked answers to a questionâ. In other words, youâd get too many false positives for human text that reads like it was AI-written. Itâs way more likely for a human to accidentally write like Claude than it is for a human to accidentally reproduce a watermark.
It would also be prohibitively expensive to run every Anthropic model against a piece of text in order to watermark it. The EU AI Act will eventually require labs like Anthropic to offer free watermarking services to every EU citizen (see Commitment 2). You couldnât do that with the ârun the modelâ approach.
As far as I know, the only AI provider to say they watermark text output is Google, who use a tool called SynthID. Hereâs how it works.
When an LLM generates text, itâs generating a series of tokens (words or chunks of words). At each step, the model itself doesnât output a single token, but instead outputs a full list of all (say) 100,000 tokens in its vocabulary, each annotated with the probability that that token will be the next one. Tools like ChatGPT or Claude Code will pick semi-randomly from the most likely options in order to get their outputs. This semi-random sampling process can be influenced in a detectable way.
For instance, we could choose a sampling strategy like âwe pick the second most likely token, then the first, then the second, then the first, and so onâ. That would still produce high-quality output, but youâd be able to re-run the model against the generated text to verify that the pattern holds. However, thatâd make verification really expensive, and any slight tweaks to the output would break the pattern and thus break the fingerprint. Is there a better way?
Yes. SynthID is a process for assigning each token a âscoreâ based on its previous tokens (for instance, sum the tokenâs ID with the IDs of its previous three tokens then take mod 5)5. To apply the watermark, the model adopts a sampling strategy like âout of the top five most likely tokens, pick the one with the top SynthID scoreâ6. The watermark can then be detected by calculating the aggregate SynthID score of a block of text. If itâs suspiciously high, itâs very likely to have been AI-generated.
This is basically a version of the common advice that you can identify LLMs by use of the em-dash, except that instead of a list of keywords, it relies on subtle mathematical relationships between words that humans canât identify. Because the process for assigning the score is trivial, itâs very cheap to run watermark detection.
Google have a complicated mathematical rationale for why SynthID doesnât make the model dumber: supposedly the SynthID scoring is random enough to act like a normal pseudo-random token sampler, just one that leaves a detectable fingerprint on the outputs. But of course this is suspicious. For instance, itâs common to do inference setting temperature to zero, which always picks the modelâs most likely next token. In that case, you canât leave a fingerprint at all (or you have to ignore the userâs preference and pick the second or third choice anyway).
If you canât alter the model outputs, can you still fingerprint the content? Well, kind of. Iâm pretty sure OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you might go through and replace your normal â â spaces (unicode U+0020) with a three-per-em âââ space (unicode U+2004), or a CJK ideographic âăâ space (unicode U+3000). These are called âhomoglyphsâ, and you can find more of them here.
Of course, lots of human-generated text uses homoglyphs. But itâs trivial to encode a pattern of homoglyphs (say, âevery third space becomes a three-per-emâ) that is much less likely to occur in the wild. Like the SynthID watermark, a homoglyph-based watermark can be detected very cheaply. A homoglyph-based watermark is cheaper to apply than SynthID: you could even do it entirely on the client.
I donât think this is a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the â character in âTodayâs dateâ, though theyâve since walked that back). In the last few years, Iâve noticed that when I copy blocks of text from ChatGPT and paste them into VSCode, sometimes VSCode marks some or all of the spaces as unusual Unicode characters7. Are OpenAI and Anthropic using homoglyphs as an AI-generated watermark? Iâm not sure. But theyâre definitely using homoglyphs.
The AI Act (specifically, its associated Code of Practice) requires watermarking to be âembedded within the content in a manner that is difficult for it to be separated from the contentâ. However, text watermarks can be trivially removed.
To remove unicode homoglyph watermarking, you simply have to replace all the homoglyphs with their ârealâ character equivalents. If you have access to even a relatively weak un-watermarked LLM8, you can strip out SynthID watermarking by asking that LLM to paraphrase the text content. Because the watermark is inherent to subtle vocabulary choices, re-wording the content will remove the watermark. You could even do it by hand, although at that point itâs not really AI-generated content anymore. Since there will be some kind of free public watermark testing tool, you can just keep tweaking until it comes back negative.
Moreover, the AI Act requires watermarking techniques to be âinteroperable⊠as far as this is technically feasibleâ. That means AI providers would have to publish their watermarking process, and potentially even attempt to standardize on applying the same kind of watermarks. I just donât see how this is compatible with the kind of security-by-obscurity that LLM text watermarking depends on. Unlike image and video watermarks, text watermarks will always be trivial to remove.
The AI Act and Code of Practice talk a lot about âdigitally signed metadataâ. The idea here is that you can include an AI disclosure in the fileâs metadata itself, ideally in a way that cannot be tampered with (for instance, by signing a hash of the fileâs contents). This signed-metadata process is basically C2PA Content Credentials. While you can remove C2PA metadata, you (theoretically) canât fake it, so a file with âcreated by a humanâ metadata can be trusted, and files with no metadata at all can be held in suspicion.
This post is already too long to get into what I think about C2PA, but I do want to say that C2PA is not a substitute for text watermarking. It only really applies to files. In the words of the Code of Practice, thatâs âa data format that supports attaching metadata (e.g., an audio, image, video, or containerised text)â. The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so canât be signed. What would it even look like to sign ChatGPT outputs? Thereâs no artifact to pass around.
I think itâs a fascinating question whether Claude Code has to C2PA-sign any HTML files or PDFs it generates for you. That seems kind of tricky to get right. But in any case, the AI Act also mandates some kind of actual watermarking as well.
So whatâs going to happen this year? If I had to guess, Iâd say that each AI provider (not just labs like OpenAI or Anthropic, but third-party providers like Fireworks or Groq) will stick a SynthID token sampler in front of their inference stacks. This might be limited to users in the EU, but it might not be, since SynthID is at least as good as a normal top-k token sampling approach.
AI providers will then offer a âcheck for watermarkâ page that re-tokenizes user-provided text, runs the scoring, and checks whether itâs above a certain threshold. Depending on how seriously the interoperability clause is taken, providers might even standardize on the same SynthID setup, in which case there could be a single EU-hosted âwatermark this textâ page.
I donât think unicode-based watermarking is going to be considered compliant with the AI Act, but some providers which donât want to set up SynthID might try it. Either way, technical users will be able to strip out the watermark at will, and there will be a plethora of tools that non-technical users will use for this purpose.
Well, for new systems; existing ones get until December.
I donât think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that theyâre looking for watermarks.
Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an âeâ, but GPT-5.5-Pro produced this puzzling koan: âThese hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.â
I leave the analogy with AI safety guardrails as an exercise for the reader.
Thatâs a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens.
The tokens are scored in a multi-round knockout against each other, but I think thatâs more of an implementation detail and not required to get the core intuition behind why SynthID works.
When this became public knowledge, OpenAI claimed it was just a model quirk, which is certainly possible.
All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text.
If you liked this post, consider subscribing to email updates about my new posts, or sharing it on Hacker News.
Here's a preview of a related post that shares tags with this one.
AI inference is obviously profitable
Many people claim that AI inference is unprofitable to serve, and thus must be subsidized by an ocean of dumb money from investors who believe that some future AI model will come to dominate the world economy. When that dumb money goes away, so will AI products. According to this view, LLMs are just inherently too expensive (in terms of money, power, and water) to be used in consumer products. In fact, they can only be used today by externalizing the costs: money onto VC funds and now retail ETF investors, power onto electric utility consumers, and water onto the communities where datacenters are built.
Continue reading...