QWE AI Academy I needed a clean text corpus for a small classifier and almost copied the wrong playbook. Not Meta’s shadow-library pipeline – the louder internet script that treats every public URL as free training fuel. The Aaron Swartz vs Meta double standard is useful here for one reason: it shows how the same rough activity class ends differently when you have no legal department.
I used to treat the Swartz story as pure tragedy porn. Then the corpus deadline got real, and the only question that mattered was narrower: how do you gather AI data when prosecutors, ToS, and rate limits hit individuals harder than platform labs?
The double standard, once, as a risk map
Per the United States v. Swartz record and MIT/DOJ accounts, Swartz pulled roughly 4.8 million JSTOR articles (about 70GB in contemporary reports) in late 2010-early 2011 over MIT guest access. Turns out it wasn’t only “open pages in a browser” – a laptop in an unlocked network closet, a high-rate script, MAC rotation after blocks, and a multi-day JSTOR outage for MIT. Prosecutors stacked wire-fraud and CFAA counts (13 felonies after the superseding indictment) with a headline max of 35 years and $1 million; he died by suicide on January 11, 2013; charges were dropped. JSTOR did not drive a civil pile-on. His 2008 Guerilla Open Access Manifesto had already framed paywalled science as the thing worth resisting.
Court filings later showed Meta staff torrented at least 81.7TB from shadow libraries (LibGen, Anna’s Archive, Z-Library, and similar) for Llama training – Ars Technica’s February 2025 reporting on the unsealed material is the clean public summary, including internal unease about torrenting from corporate machines. In Kadrey v. Meta (N.D. Cal.), Judge Vince Chhabria’s June 25, 2025 order granted partial summary judgment that training on the books was fair use on that record, largely because plaintiffs failed to show concrete market harm. Civil noise continues. Criminal CFAA thunder like Swartz’s did not show up. A August 2026 comparison essay (curiousquail) later spiked on Hacker News; the legal facts above are what the argument sits on.
Pro tip: If your project cannot survive a prosecutor’s press release, design the data path so you never need a defense team to explain it.
Step-by-step: scrape like consequences exist
Personal or small-team data for fine-tuning or RAG. Public pages only. No shadow libraries.
- Confirm the data is public – not behind a login or paywall you do not own.
- Fetch and parse robots.txt before the first real request.
- Use a clear User-Agent with a contact URL or email.
- Rate-limit hard: start around 1 request every 1-3 seconds plus jitter (as of 2025-26 common guidance from practitioner and firm write-ups).
- Cache aggressively; stop cold on 429/403.
- Log fetches; be ready to delete on request.
Minimal Python pattern:
import time
import random
import urllib.robotparser
from urllib.parse import urljoin
import requests
BASE = "https://example.com"
rp = urllib.robotparser.RobotFileParser()
rp.set_url(urljoin(BASE, "/robots.txt"))
rp.read()
UA = "MyResearchBot/1.0 (+https://yoursite.com/bot; [email protected])"
headers = {"User-Agent": UA}
def allowed(path):
return rp.can_fetch(UA, urljoin(BASE, path))
def polite_get(path):
if not allowed(path):
print("robots.txt disallows", path)
return None
time.sleep(1.5 + random.random()) # jitter
r = requests.get(urljoin(BASE, path), headers=headers, timeout=20)
r.raise_for_status()
return r.text
# Only call polite_get on paths you truly need
Test one low-traffic public path first. API or bulk export exists? Use it. I’ve killed more side projects by skipping a boring dump button than by any crawl-delay.
Pitfalls that hit solo scrapers harder
The catch is authorization after a soft ban. Ignore crawl-delay, keep hammering, and a ToS fight starts sounding like CFAA “without authorization” once access was revoked – the same statute family that crushed Swartz. Closet hardware and stretched guest credentials raise temperature faster than polite GETs.
- One IP, no identity, high rate – looks like resource exhaustion. Corporate AI pipelines with compliance theater rarely get the same instant black-hole treatment.
- Republishing full copyrighted text is a different animal from training embeddings you never ship as the book.
- “Public” is not a blanket commercial-AI license. Market-harm evidence failed plaintiffs in Kadrey (June 2025); another record could land differently.
- Empty User-Agent → Cloudflare or counsel, not a polite “please slow down” mail.
robots.txt is still widely treated as non-binding courtesy. Treating it as optional when you are a named individual is not the same risk profile as a company that can staff the aftermath.
Safer alternatives to DIY mass scraping
| Approach | Effort | Risk | Best for |
|---|---|---|---|
| Official APIs / bulk downloads | Low | Lowest | Structured public data |
| Common Crawl WARC archives | Medium | Low (already crawled) | Broad web text for research |
| Hugging Face / academic dumps | Low | Low-medium (check licenses) | Ready LLM-style corpora |
| Polite targeted scrape (code above) | Medium | Medium | Niche sites with no dump |
| Shadow libraries / book torrents | Low setup | High for individuals | Avoid |
Common Crawl’s monthly archives exist so every lab does not need its own spider army – hundreds of billions of pages, filter by language or domain, move on. Pair with license-aware sets and you stay nearer the research-shaped fair-use arguments labs actually brief in court. Remember the 81.7TB headline? You do not need a personal remix of that path for a classifier.
Is the system fair? No. Does unfairness grant a solo builder a free pass to act like Meta until the subpoena arrives? Also no – the office that charged Swartz still knows how to write a press release.
FAQ
Was Aaron Swartz really just “scraping”?
No. Closet access, service disruption, MAC evasion. That is why “it was only downloading” fails as a full defense narrative.
Did Meta face zero consequences?
Authors still sued. Picture a status conference where fair use for training sticks because market harm was not proven on that record (Kadrey, June 25, 2025) while separate publisher pressure and reputation costs keep humming. What never matched Swartz was criminal CFAA exposure.
Can I legally scrape for my own small AI project?
People often hear “public page = free training set.” Wrong frame. Lowest-risk lane most practitioner guides describe (as of 2025-26): publicly available pages, robots respected, hard rate limits, no technical circumvention, no full-text dump of other people’s books, eyes on ToS and copyright. US CFAA and contract claims are real; other countries differ. Prefer Common Crawl, licensed corpora, or written permission. Not legal advice – if the corpus is core to a paid product, hire counsel who has read the recent training opinions, not a Twitter thread.
Open a notebook. Hit robots.txt on a site you actually need. Write polite_get before the first bulk loop. That habit is how the double standard stays a cautionary tale instead of your case caption.