When AI Says ‘If I Don’t Steal It, Someone Else Will’

The truth: AI companies have adopted the worst ethical framework for data scraping, and they’re not hiding it. Their justification? “If we don’t take your content, our competitors will.”

Familiar? Should be. Same logic from a 2021 viral video: Israeli settler confronted about taking a Palestinian family’s home in East Jerusalem responded, “If I don’t steal it, someone else is gonna steal it.” Became a meme. Now it’s AI’s business model.

Not hypothetical. Right now, AI crawlers visit your site. The polite “don’t scrape us” files? Not stopping them.

The Moment the Quiet Part Got Said Out Loud

Traffic spikes. Server logs: millions of requests from unknown user agents. Your tutorial that took weeks? Training data for a model that answers your topic without linking back.

Not theoretical. July 2024: iFixit’s CEO watched Anthropic’s ClaudeBot hit their servers a million times in 24 hours. Freelancer.com: 3.5 million visits in four hours. Same crawler. These aren’t bugs – features.

Darker: industry response boils down to competitive necessity. DeepSeek’s January 27, 2025 release made it explicit. Built a model rivaling OpenAI’s o1 for $5.6 million using aggressive collection. OpenAI and Microsoft accused them of scraping API outputs. DeepSeek’s counter: everyone does it, we did it cheaper.

What ‘If I Don’t Steal It’ Means for Your Content

The logic: AI company says “we need this data to stay competitive” – that’s a prisoner’s dilemma at industrial scale. Company A respects your robots.txt and doesn’t train. Company B ignores it and does. Company B gets the better model. A falls behind. Solution? Nobody respects boundaries.

Princeton researchers studying DeepSeek identified “second-mover advantage.” How: First-movers spend billions training frontier models. Second-movers scrape those outputs (APIs, public interfaces), distill the knowledge, build “good enough” alternatives for a fraction. Training a student by copying homework – cheaper, faster, completely dependent on someone else’s work.

Think of it this way: You spend 6 months writing the definitive tutorial on container orchestration. Company A scrapes it, trains their model. Company B then scrapes Company A’s model responses about orchestration, distills that knowledge, and now they have your expertise twice-removed. You got nothing. They got your competitive edge for pennies.

June 2025 data from Cloudflare: damning. OpenAI’s crawl-to-referral ratio was 1,700:1. Every visitor ChatGPT sent back, their crawlers already hit 1,700 pages. Anthropic? 73,000:1. Compare: Google’s ratio is 14:1 – they drive traffic. AI crawlers? Pure extraction.

Pro tip: Check server logs for ClaudeBot, GPTBot, CCBot, Google-Extended user agents. Filter by request frequency over 24-hour windows. Thousands of requests from the same bot? You’re being farmed – regardless of what your robots.txt says.

The Reddit Lawsuit That Broke the Illusion

June 2025: Reddit filed suit against Anthropic in California. Accusation: systematic scraping of user content to train Claude, despite explicit restrictions. Different from usual “AI behaves badly” stories because of what audit logs revealed.

The complaint: Anthropic’s bots accessed Reddit 100,000+ times after July 2024 – same period Anthropic publicly claimed to have blocked crawlers from the platform. This isn’t misunderstanding terms. Documentation that “we’ll respect your wishes” promises aren’t kept even when explicitly made.

Reddit didn’t file copyright. Went with breach of contract and computer fraud, likely because copyright law hasn’t caught up. Subtext: if a platform with $60 million from Google and $70 million from OpenAI (roughly 10% of 2024-2025 revenue) still can’t get a company to honor restrictions, what chance does your blog have?

The kicker: Anthropic’s official documentation says they “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” Reddit’s logs suggest otherwise. Gap between policy and practice? Where your content disappears.

Why Robots.txt Isn’t Working

Every tutorial: update robots.txt. Add these lines, block these agents, done. Except it’s not done. Robots.txt is voluntary. Polite request. No technical enforcement.

June 2024: TollBit (startup brokering licensing deals) sent letters to major publishers warning multiple AI companies bypassed robots.txt entirely. Business Insider reported both OpenAI and Anthropic were circumventing restrictions.

Worse: robots.txt has a 500 KB size limit per HUMAN Security documentation. AI crawlers multiply (dozens now, more weekly). Large sites physically can’t list them all. Assuming they identify themselves honestly – many don’t.

Crawlers rotate user agent strings. Others disguise as legitimate search bots. Some use residential proxies to look like regular users. Relying solely on robots.txt? Bringing a suggestion box to a data heist.

Three Moves That Work When Politeness Fails

If robots.txt is theater, what stops AI scrapers? Not “reduces” – stops. What’s working in February 2026 based on publishers successfully defending content.

1. Rate Limiting at Infrastructure Level

First real defense. Set request limits per IP per time window. Single source makes 100+ requests in an hour? Throttle hard or block entirely.

# Cloudflare Workers rate limit
# Blocks IPs making >100 req/hour

async function handleRequest(request) {
 const ip = request.headers.get('CF-Connecting-IP');
 const count = await RATE_LIMITER.get(ip);

 if (count && count > 100) {
 return new Response('Rate limit exceeded', { status: 429 });
 }

 await RATE_LIMITER.put(ip, (count || 0) + 1, { expirationTtl: 3600 });
 return fetch(request);
}

Cloudflare, Akamai, most CDNs offer this as a toggle. Catch: legitimate crawlers (Googlebot) also make frequent requests. You’ll need to allowlist known-good bots by verified IP ranges. Google publishes theirs. Bing does. AI companies mostly don’t – part of the problem.

2. Authentication Walls for High-Value Content

Content valuable enough to scrape? Valuable enough to gate. Doesn’t mean paywalls necessarily – even free account signup stops most automated scrapers.

Why: Creating accounts at scale is expensive. Solve CAPTCHAs, verify emails, maintain session state. Scrapers optimize for volume. Friction works.

Middle-ground approaches: free preview + login for full content, or time-delayed public access (paywalled first 30 days, then open). Reddit and Stack Overflow both moved here specifically because of AI scraping pressure.

3. Legal Notices That Create Liability

Subtle but powerful. Add explicit “No AI Training” notice to site’s terms of service. Display it prominently. Why it matters: converts murky fair-use argument into clear breach-of-contract claim.

Sample language from Authors Guild:

NO AI TRAINING: Without in any way limiting the
author's exclusive rights under copyright, any use
of this publication to "train" generative artificial
intelligence (AI) technologies to generate text is
expressly prohibited.

Does this stop scrapers? No. Makes the lawsuit later much easier. Reddit’s case against Anthropic hinges partly on contract violation, not just copyright. Only possible because Reddit had clear terms Anthropic allegedly violated.

The Decision You Have to Make

Most tutorials avoid telling you whether you should block. Decision depends entirely on what content you publish and what you need it to do.

Real question: strategic. What do you get for allowing your content to be scraped? Answer is “nothing” or “exposure”? Block it. Exposure doesn’t pay server bills.

Nobody asks: does blocking even matter at this point? Content publicly available past three years? Almost certainly already in multiple training datasets. Common Crawl (used for GPT-3, many others) archives the web constantly. Blocking now stops future scraping but doesn’t remove what’s already taken.

What Happens Next

California’s AI Training Data Transparency Act went into effect January 1, 2026. Requires AI companies publish summaries of training data sources. Will they comply? Will summaries be detailed enough? We’re about to find out.

EU AI Act’s full implementation in August 2026 includes provisions about training data transparency and copyright. Multiple lawsuits working through courts: New York Times, Authors Guild, Getty Images all have active cases against AI companies.

What’ll happen: nothing fast enough to help you. Copyright law moves at glacial speed. AI companies move at silicon speed. By time courts issue rulings, training will be done, models deployed.

You’re left with the same choice everyone has: defend your content with tools available now, or don’t. No cavalry coming. Just you, your robots.txt that doesn’t work, and an industry that decided your consent is optional.

What you do next depends on whether you believe “someone else will take it” is a justification or an excuse.

FAQ

Is blocking AI crawlers legal, or could I get in trouble for it?

Yes, completely legal. Your server, your content, your rules. Not obligated to serve content to anyone. Real question: is not blocking giving away rights you might want later?

If I block AI crawlers, will it hurt my SEO or Google rankings?

Blocking AI training bots (GPTBot, ClaudeBot, CCBot) won’t affect traditional search rankings – those are separate from search crawlers like Googlebot. However, blocking Google-Extended (Google’s AI training crawler) has caused ranking issues for some sites despite Google claiming otherwise. User reports from mid-2025 suggest the two systems aren’t as separate as documented. Safest approach: block AI training bots, keep search engine crawlers allowed, monitor your traffic closely first month after changes. I saw one site drop 15% in impressions after blocking Google-Extended, recovered after reverting. Your mileage may vary, but watch your Search Console like a hawk.

Can AI companies still use my content if it’s already been scraped before I blocked them?

Yes. Hard part. Content was publicly accessible when they scraped? Likely already in their training datasets. No “undo” button. Blocking now prevents future training runs, stops them getting updated versions of your content. Some legal experts argue even past scraping without consent could be actionable under contract or computer fraud theories (like Reddit’s approach), but those cases are still being litigated. Think of it like this: if someone photocopied your book last year, blocking the copy machine today doesn’t un-copy it. But it stops them making fresh copies. Blocking won’t erase what’s taken, but it stops the bleeding.