QWE AI Academy Most AI bots hitting your repo aren’t indexing it. They’re eating it. That’s the backdrop for the announcement that lit up Hacker News and every FLOSS Mastodon feed: Codeberg is drawing a hard line, and it raises a practical question every open-source maintainer now has to answer.
The takeaway upfront
If you maintain open source code and you don’t want it in the next Claude or GPT training run, you have two moves: change your license or change your server config. Only one of them actually works today. Server config wins – but with a specific gotcha most guides skip. Skip to the walkthrough if you already know the story.
What actually happened at Codeberg
In mid-2026, Codeberg e.V. adopted two formal resolutions by member vote. First: a binding commitment that the forge and all associated services will not use project or user data to train LLMs or any other generative AI tools. Second: an amendment to the terms of use prohibiting the hosting of ‘vibe-coded’ projects – code generated primarily by AI tools – on the grounds that they place disproportionate burdens on volunteer infrastructure.
Their reasoning is worth reading directly. The Codeberg blog describes LLM use in FLOSS as a multidimensional attack on trust between contributors – maintainers drowning in low-effort AI-generated PRs, uncertainty about which projects have real human oversight. The sharper point: in copyleft projects, LLMs enable what they call ‘license laundering’ – copyleft code gets stripped of its reciprocity requirements by being ‘generated’ out of the training data. Read the original post – it’s short.
Community reaction split along predictable lines. On Hacker News, the thread captured it plainly: some developers agree strongly with Codeberg’s stance, others reject the reasoning entirely. That’s the news. Now the question you actually care about: what do you do?
Method A vs Method B: license clause or technical signal?
Two schools of thought. Here they are.
| Approach | What it is | Who it stops | Real-world effect |
|---|---|---|---|
| License clause | Add “no LLM training” language to your LICENSE or a separate NOTICE | Actors who read licenses (mostly: none) | Legal signal only – enforcement is theoretical |
| Technical signal | robots.txt + HTTP headers + repo metadata (ai.txt) | Bots that voluntarily comply (GPTBot, ClaudeBot, Google-Extended, etc.) | Real, measurable blocking of the major players – if they honor it |
Method A’s core problem: standard permissive licenses – MIT and Apache 2.0 – contain zero restrictions on training use. Adding a custom “no training” clause creates a legal argument you’d theoretically need to enforce in court, and it may cost you the OSI “open source” label in the process. You gain a paper trail; you lose the label.
What does “protection” even mean here, practically? Not invisibility. Not legal immunity. The realistic goal is: make your objection legible to compliant actors, and create an evidence trail in case you ever need to demonstrate intent. That framing makes Method B the obvious choice – it produces observable behavior. The Codeberg approach itself is essentially institutional Method B: platform-level technical enforcement, not a license rewrite.
The actual walkthrough (Method B, in three layers)
Three layers, stacked. Each catches what the layer above misses.
Layer 1: robots.txt for your project page or personal site
If you host docs, a blog, or a project page, this is your first move. Drop a robots.txt at the site root. This blocks the major training crawlers while leaving actual search engines alone:
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
The list of user agents comes from robotstxt.com/ai, updated as new bots appear. Copy from there rather than from static blog posts – the list moves fast.
Layer 2: HTTP response headers
robots.txt is polite. Headers are firmer. If you control your server, add the X-Robots-Tag header to tell any bot inspecting the response that this content is off-limits for training:
# nginx example
add_header X-Robots-Tag "noai, noimageai" always;
The noai and noimageai directives aren’t universally standardized, and some crawlers now check for them. It’s a cheap add with no downside if your server already runs nginx or Apache.
Layer 3: repo-level metadata
Add a top-level file named ai.txt or a section in your README declaring the restriction – not legally binding, but it gives compliant scrapers a clear signal and creates evidence of intent if you ever need to show a crawler ignored your notice.
Pro tip: Don’t assume your platform’s defaults are neutral. A 2026 analysis of Anthropic bots across commonly used hosts (via parse.gl) found ClaudeBot implicitly blocked on a meaningful share of sites whose owners had no intention of blocking it – CDN and hosting provider defaults injected during setup. Check your actual, live
robots.txtfrom an incognito browser before assuming anything.
The Google-Extended gotcha nobody warns you about
You’ll add User-agent: Google-Extended / Disallow: / to your robots.txt, then check your server logs for a hit from Google-Extended to confirm it’s working. You’ll never find one.
Turns out Google-Extended is a control token rather than a traditional bot – it won’t appear in server logs even when properly configured. The directive tells Google whether your content can train Gemini and Vertex AI generative APIs, but there is no crawler behind the name. Google’s existing Googlebot fetch respects the directive on the backend. Empty logs don’t mean it’s broken.
Three things to check before you call it done
Compliance is unilateral. Every technical signal above assumes the crawler is honest. Non-compliant scrapers exist and won’t stop. Al-Kaswan and Izadi at TU Delft make this concrete in their 2023 paper on LLM training and open source licensing: LLMs for code are trained on large unsanitized corpora scraped from the internet, and models memorize and emit that content – often verbatim. Their argument is the sharpest academic treatment of the licensing dilemma currently available.
Secrets in your repo become model outputs. If your repo ever contained an API key or password in git history and it got scraped before you added blocks, research shows those repositories can contain sensitive data – API keys, passwords, personally identifiable information – that models inadvertently learn and replicate. Rotate anything that ever touched a public repo. Don’t wait.
The adversarial option exists. If you want to go further than blocking – actually degrade models trained on your code – the academic literature has an answer. CoProtector (arXiv:2110.12925) is a protection mechanism where a developer protects their repository against unauthorized training by inserting poison code artifacts. If a training run ignores the warning notice, the poison source code gets collected; the result is a model with embedded backdoors and deteriorated performance that can also be used for digital forensics to prove the repo was scraped. It’s fringe. It’s also real. Read the CoProtector paper before trying it – poisoning your own repo has obvious downsides for actual human collaborators.
Is there a clean answer to whether any of this is worth the effort for a small project no one will train on anyway? Probably not. The point isn’t to become uncrawlable. The point is to make the choice explicit – so that if your code shows up somewhere it shouldn’t, you have a record of having said no.
FAQ
Does blocking GPTBot hurt my Google search ranking?
No – GPTBot is OpenAI’s training crawler, completely separate from Googlebot. Blocking Google-Extended only controls AI model training and has no effect on Google Search indexing or ranking.
My code is on GitHub, not Codeberg. What can I actually do?
GitHub uses repo data in ways Codeberg now refuses to, and you can’t unilaterally opt your GitHub repo out of the platform’s own training pipeline because you don’t control the server. That said, you’re not out of options. Mirror to Codeberg or a self-hosted Gitea instance and treat GitHub as the read-only public copy – that’s the cleanest move if you care about this enough to act on it. Or add your ai.txt and license notice on GitHub anyway: you won’t stop a determined scraper, but you’ll have a stated-intent record. If you have a personal domain mirroring the project, put your robots.txt there. The tooling consensus is still forming, which is exactly why Codeberg’s institutional move matters.
Will these blocks work retroactively on code already in training sets?
No – what’s scraped is scraped. Robots.txt only affects future fetches, and no technical signal reaches data already in a training corpus. If your repo was public before 2023, assume it’s in at least one model’s training set.
Do this next
Open your project’s website (or your personal domain) in a browser, append /robots.txt, and read what’s actually there. If it’s empty or missing the crawlers above, that’s your next 10-minute task. No website? Mirror one repo to Codeberg this week and see how the ToS feels in practice before deciding whether it’s the forge you want for the next one.