AI Slop Wins $25K DeepMind Kaggle Prize: Build Your Own

A $25,000 DeepMind Kaggle Grand Prize just went to a benchmark that half the internet is calling AI slop, and the other half is calling a legitimate contribution to AGI evaluation. Both sides have a point. And if you spent the weekend rage-scrolling instead of submitting, here’s the useful part: the tool the winners used is free, the SDK is trivial, and the bar for a decent submission was lower than the shouting suggests.

This piece isn’t a news recap. It’s a hands-on walkthrough of Kaggle’s Community Benchmarks – the platform the whole hackathon ran on – plus an honest read on when “slop” is a fair critique and when it’s just a reflex insult.

The takeaway in one paragraph

Kaggle announced the winners of the Measuring Progress Toward AGI Hackathon after 1,000+ teams submitted benchmarks across five cognitive tracks. Winners included GAUGE, which tests whether models know when to abstain versus answer under uncertainty, and Metaproteus, which tests whether models can accurately predict their own likely responses. The critics’ complaint: some of these read like a clever prompt wrapped in 30 lines of Python. The counter: that’s kind of the point – good evals are supposed to be simple and reproducible. Either way, you can build one this afternoon.

Background: why this hackathon existed at all

DeepMind published a paper called Measuring Progress Toward AGI: A Cognitive Taxonomy and simultaneously ran a Kaggle hackathon to actually build evals against it. The prize pool was $200,000: $10,000 awards for the top two submissions in each of the five tracks, and $25,000 grand prizes for the four absolute best overall submissions. Submissions were open March 17 through April 16, 2026.

The five tracks weren’t random. DeepMind picked learning, metacognition, attention, executive functions, and social cognition because that’s where the evaluation gap is largest – meaning current benchmarks are getting saturated and nobody quite knows how to test these capabilities rigorously.

Pro tip: If you missed this hackathon, read the winning benchmarks first before designing your own. Kaggle publishes the code. Reading two or three winners takes 20 minutes and will show you exactly what “acceptable rigor” looks like on this platform.

Method A vs Method B: two ways to build a benchmark

Here’s the real fork in the road – and the source of the slop debate. You can go two directions.

Method A: The gimmick benchmark. Pick a cute idea (“does the model know when it’s guessing?”), write one task, hard-code five to ten examples, ship it. Total time: maybe two hours. This is what the “AI slop” crowd is mad about. It can win if the framing is novel enough.

Method B: The rigorous benchmark. Pick a capability nobody has measured well, design 100+ tasks with proper distractors, collect human baselines, document your methodology. Total time: a week or two. Much less likely to be called slop. Also much more likely to actually influence how models are evaluated.

My honest read: Method A wins prizes more often than defenders of “rigor” want to admit, because judges are humans who reward novelty. Method B builds a portfolio piece you can actually cite in a paper. Which one you pick depends on whether you want the check or the credibility.

Walkthrough: build a benchmark with the kaggle-benchmarks SDK

The SDK is embarrassingly small. Here’s the pattern – Kaggle’s Community Benchmarks announcement shows it in full, but this captures the mental model:

import kaggle_benchmarks as kbench

@kbench.task(name="simple_riddle")
def solve_riddle(llm, riddle: str, answer: str):
 """Asks a riddle and checks for a keyword in the answer."""
 response = llm.prompt(riddle)
 kbench.assertions.assert_contains_regex(
 f"(?i){answer}",
 response,
 expectation="LLM should give the right answer."
 )

solve_riddle.run(
 llm=kbench.llm,
 riddle="What gets wetter as it dries?",
 answer="Towel",
)

A @kbench.task decorator wraps a function. Inside, you prompt the LLM and assert something about the response. Chain a bunch of these together, register them as a benchmark, run them across models. Done.

Step-by-step

1. Install the SDK inside a Kaggle notebook (it’s preinstalled in the Community Benchmarks environment).
2. Write one task first. Use assert_contains_regex or a custom assertion – whatever catches the failure mode you care about.
3. Run it locally against kbench.llm (the default model) to sanity-check.
4. Add 20-50 more task instances. Vary difficulty. Include adversarial cases.
5. Group tasks into a benchmark and submit. Kaggle then provides a leaderboard that tracks and compares results across models.

What model access you actually get

You can run your benchmarks against models from Google, Anthropic, and DeepSeek for free (within quota). Notice who’s missing: OpenAI. This isn’t an oversight – Kaggle’s roadmap explicitly says “we don’t currently support OpenAI” (as of the January 2026 launch). If your benchmark idea depends on comparing GPT-4/5 against Gemini, you’re stuck exporting to your own use.

Edge cases nobody in the news coverage mentioned

Three things that will bite you the moment you start building:

1. The quota is undocumented. “Within quota” is the official phrasing across every announcement, but no number is published. A benchmark with 100 tasks × 3 models × 5 runs = 1,500 API calls, and you’ll find out mid-execution if that’s too many. Start with 10 tasks, expand once you’ve confirmed nothing throttles.

2. Judging is subjective in a way leaderboards aren’t. This hackathon didn’t rank by score – it ranked by benchmark quality, judged by humans. That’s why the winners list includes ideas that read like tweets (Metaproteus: does the model predict its own answers?). If you optimize for a numeric leaderboard, you’ll build the wrong thing.

3. “AI slop” is now a reflex insult, not a diagnosis. A joint University of Oslo and American University of Sharjah study analyzed 25 million Reddit and Hacker News comments and found something worth internalizing before you post your work anywhere: by 2026, “AI slop” accounted for 94% of pejorative AI accusations, and the volume of such accusations increased more than tenfold – often with no evidence the content was AI-generated. The researchers concluded these accusations increasingly function as an emergent form of social gatekeeping, not as a reliable way to identify AI. Translation: if you ship anything, someone will call it slop. Ship anyway.

So – was the winning benchmark actually slop?

“Slop” meaning mass-produced content designed to game engagement? No – these are small, hand-designed evaluations with real methodology behind them. Could some have been written in an afternoon? Yes. But Andrej Karpathy, as he noted publicly and as Kaggle cited in their 2026 Community Benchmarks announcement, spent roughly a third of his time at Tesla on evals – good evals are hard because they’re small and surgical, not despite it. A two-hour benchmark that asks a question nobody operationalized before is more valuable than a 200-task suite testing something already saturated.

The winners weren’t crowned for token count. They were crowned for asking a question nobody had operationalized yet. That’s a low bar and a high bar at the same time.

FAQ

Can I still submit if the hackathon is over?

The hackathon window closed April 16, 2026. But Community Benchmarks itself is a permanent Kaggle product – you can build and publish benchmarks anytime, and they’ll appear on public leaderboards. Follow Kaggle’s official channels for announcements about future competitions in this space.

Do I need a machine learning background to build a decent benchmark?

Not really. The bottleneck isn’t ML skill – it’s domain skill. A doctor who can design 50 medical reasoning tasks that trip up models will build a better benchmark than an ML PhD guessing at what makes medical questions hard. Pick the field you know weirdly well, then wrap it in the ten-line SDK pattern above. Weird domains win.

What’s the difference between a task and a benchmark?

A task is one test – one function that prompts the model and asserts something. A benchmark is a bundle of tasks scored together to produce a leaderboard.

Next step: Open Kaggle’s Community Benchmarks announcement, copy the riddle task above into a new Kaggle notebook, and run it. If it executes, you’re already 60% of the way to a submission-ready benchmark. Pick a domain you know weirdly well and write ten more tasks by tomorrow.