Let’s assume that a hundred companies offer inference for the same open model, such as DeepSeek. They all individually decide to implement watermarking on their own in order to comply with Europe’s AI Act. Each provider marks the output it generates with its own key, which a detector uses to check whether content contains that provider’s watermark. This means that there are 100 watermarking keys. So if someone encounters a piece of content on social media, they will have to go to 100 different systems, each run by the inference provider, to check if it contains the watermark of that specific provider.
Having every provider embed a watermark does not, by itself, give people a usable way to recognise AI-generated content. They also need to know which provider to actually ask if the content contains a watermark. So the individual systems of all the providers need some way of working together.
I write about the dynamics of governance of decentralised social systems, and it turns out that AI is kind of just that.
What a working system needs
Let’s start with the obvious, of what problem this is trying to solve. The goal is that people can know that output they see online (whether that’s text, images or audio) is created by an AI system. This means that a system of watermarking needs the following components:
- a way to apply a watermark to AI-generated output
- a way for people to see if arbitrary content contains an AI watermark
- a way for people to have a reasonable sense that the large majority of AI-generated content they encounter contains a watermark
- a system of compatible watermarks: the goal is to give an indication if content is created by any AI system, not just whether a piece of content is generated by one system
It is in these final two requirements that my interests are, because those turn out to contain the hard governance challenge. Furthermore, these hard governance challenges display dynamics that are fairly similar to the challenges of governance of decentralised social networks.
Recognising a watermark and drawing conclusions from its absence are two different things. The second depends on how much of the ecosystem participates. Identifying marked output remains useful even when some providers do not participate, but it does not give people the same ability to draw conclusions about arbitrary content they encounter.
The AI Act does account for interoperability in principle. Article 50 requires marking solutions to be “effective, interoperable, robust and reliable”, within the limits of technical feasibility. The Code of Practice sets out several ways for these systems to work together: routing detection queries to the relevant providers, adding a readable signpost to content that identifies the right detector, or creating a shared detection service. So the problem is not that European policymakers forgot to ask for compatible systems.
How watermarking works
But to get to those points, first some basics on watermarking. I’m not going into all the details here or offering you a full explanation of how this works.
The main system used is SynthID, which is used (and created by) Google, and used by companies like Anthropic.
When an LLM generates text, it assigns probabilities to the possible next tokens, which are words or parts of words. The system then selects a token, usually using some randomness in that selection. SynthID changes how those selections are made. It uses a secret key together with the preceding text to introduce a statistical pattern into the choices.
To someone reading the text, those choices look ordinary. But a detector configured with the same key can check whether the choices follow the expected pattern. There are no hidden characters attached to the text, as the watermark is in the selection of the words themselves.
There are a few aspects to know about SynthID:
- SynthID is not part of a model; it is applied during inference. So a model is not watermarked, the specific output of a model is watermarked, provided it is turned on during inference. For all intents and purposes you can imagine that any inference provider has a setting that they can tick for each prompt individually whether watermarking is included, yes or no. This means that regulating the company that releases a model does not automatically determine what happens when another company runs the model.
- SynthID needs a key, and you need to know the key in order to determine if the output contains a watermark.
Both aspects turn out to be hugely impactful on how AI watermarking actually happens in practice in society.
When the same company controls both the model and its inference, this distinction is easy to miss. But this is a huge problem for open models, where it is really hard to get a sense how many inference providers even exist. (The lists of Artificial Analysis and OpenRouter only overlap like half, and they are both missing a few smaller providers I know about. I could not get a clear number, but once you account for the smaller ones hundreds seem likely.) These inference providers would need to implement watermarking like SynthID.
The key is the key here (side note, what I love about being a writer in the age of AI is that you use language badly and people like it because it means it’s not AI). It has a few specific properties that make implementation messy:
- You need to know the key to determine if the output contains a watermark. But once you know what the key is, you can also use it to manipulate the output. Once you have the key, you can easily figure out what the minimum amount of changes are to remove the watermark from AI-generated output. You can also use it to make human-created content have the watermark so it seems AI-generated.
- If each provider uses a different key, detecting its watermark requires a detector configured with that key.
These two aspects together make the requirement that people can easily figure out if any piece of content they encounter online contains an AI watermark surprisingly difficult, even if the content has been properly watermarked.
Providers can offer a checking service while keeping the key private. Once a key of a popular watermarking system leaks, the signal value of that watermark drops precipitously. Because from now on, two new uncertainties are introduced: a piece of content without that specific watermark might have been generated by the AI system whose key was leaked, with the watermark then removed because the key was available. And if a piece of content does contain the watermark of an AI system that has a leaked key, the content might be human-created but maliciously manipulated so it seems like it has been AI-generated.
Even willing participants need institutions
The question is what makes that compatibility work in practice. A legal obligation can require providers to cooperate, but the actual cooperation still needs to be organised and sustained. Providers need arrangements for discovering each other’s detectors, interpreting their responses and dealing with systems that change or disappear. It is here that watermarking becomes a problem of governing a decentralised ecosystem.
Keeping all the keys and detectors separate takes us back to the hundred-provider problem: someone encountering a piece of content does not know which provider to ask.
But these inference providers can of course collaborate together, and decide to share a single watermarking key. This makes lookup and verification much easier. But this in turn creates a new problem: the blast radius when the key leaks becomes much larger. Big Tech companies like Anthropic and OpenAI have comparatively good security practices (for wildly optimistic definitions of ‘good’). A meaningful number of inference providers on OpenRouter are nothing more than a hustlebro who rents a few GPUs and resells access to them for a markup, whose understanding of cybersecurity practices makes that of OpenAI actually seem good.
That puts the collaboration between inference providers in a bind: collaborating with a shared key for watermarking makes the system actually usable for people, but means collaborating with actors that are shady at best, and if they accidentally screw up it’s your problem too.
The third option is to create a shared institution that coordinates verification across providers. Providers could keep their keys separate while making their detectors accessible to a single shared interface. This allows people to go to a single place to check if a piece of content contains an AI watermark. Such a system is possible, but needs collaborative governance, a way to manage participation, maintain the directory of providers, operate the infrastructure, etcetera. This is somewhat similar to shared moderation services on open social networks: easy to imagine in the abstract, probably a net benefit for society if it exists, but in practice extremely hard to set up. That a collaborative system in a trustless decentralised environment would be beneficial is easy to say, but the history of the open social web is a long collection of events demonstrating how incredibly hard this is to set up.
And if the market actors can end up in a collaborative system for watermarking verification, a decentralised system ends up producing a centralised actor again. Whoever governs that institution helps determine which providers are included and whose detectors people can reach.
What if providers do not want to cooperate?
All of this assumes that the inference providers want to cooperate in the first place.
The problem is that watermarking incentivises defection. Providing AI inference is a competitive cutthroat business for a commodity, where there are very few ways to differentiate besides via price. Some customers want AI-generated output that cannot readily be recognised as AI-generated. For those customers, the absence of a watermark is a feature worth choosing a provider for. So if 99 DeepSeek inference providers are all complying with the AI Act and apply a watermark, there is a massive incentive for inference provider number 100 to offer unwatermarked output. In a system that ruthlessly competes on price, having any differentiation is valuable.
What matters here is not simply whether a provider is based outside Europe, but whether it can afford to forgo European customers. For providers that can, offering inference outside the EU without watermarking can be a way to compete with providers that apply watermarks globally. Giving up the European market does not necessarily mean giving up a viable business.
The real problem with defection is the impact it has on the entire ecosystem. Watermarking can still identify marked output from participating providers. But that is different from creating an ecosystem where a negative result provides reasonable assurance that the output is most likely not AI-generated. With every inference provider that defects, the grounds for drawing that conclusion become weaker. Even universal participation would not, by itself, resolve detection failures or the removal of watermarks.
And because the ecosystem is global, a European ecosystem with proper watermarking only has limited value for that broader goal. Text, images and audio generated outside the EU can circulate into the same feeds, websites and conversations as content generated inside it. A European reader does not encounter a neatly bounded European content ecosystem. A provider can forgo European customers without keeping its customers’ output out of European feeds. This is how non-participation elsewhere undermines what European readers can infer from the absence of a watermark.
EU regulation assumes that there are Big Tech companies for whom losing access to the European market is a larger cost than the cost of compliance with EU regulation. That works when there are a few global giants. It works when Anthropic decides that it is easier to just apply their watermarking system globally, and not just purely for their European customers. EU regulation works a whole lot less well when there is a massive decentralised global market, with a meaningful number of players for whom losing access to the European market is not a big loss at all. Even worse, for some of them not complying with European regulation might prove to be a valuable business opportunity.
If you read my article from a few months ago on how European platform regulation relates to decentralised social networks, you see the exact same pattern. The implied assumption of European regulation, whether that is the Digital Services Act for social media platforms or the AI Act for AI labs, is that there are a few global giant companies, and the Acts are written with them in mind. The system works reasonably well when the ecosystem indeed contains only a few giants. But whether it is social media or AI inference, the underlying assumptions of how European regulation actually works start to break down once the tech starts to decentralise, and instead of a few global giants we get hundreds of globally distributed companies.