Dayjob Tax Guide: Test AI Agents on Real Tax Work

You just dumped a client folder into an AI agent

Last week I watched a colleague drop 40 PDFs, two Excel ledgers, and a phone photo of a crumpled receipt into a frontier agent and ask for “the 2025 partnership return package.” Clean workpapers showed up in minutes. Then the landmine: a multi-state dispatch log the client never flagged. The agent never mentioned it.

That miss is the whole reason Surge AI’s Dayjob: Tax matters. As of the late October 2026 release, the best pass@1 score on 50 professional assignments is 10.8%. Not a quiz score. A judgment score.

Think of every client zip as a crime scene. The asked question is the obvious clue. The return blows up on the clue nobody labeled.

Build the 4-bucket bar before you touch the leaderboard

Competitors will walk you through headline model names first. Skip that. If you prepare tax work with agents, you need an acceptance test you can run on Tuesday afternoon – not another chart.

Score any agent output on four binary buckets only:

1. Numbers match your independent recalculation.
2. Every material claim points at a specific file and location.
3. At least one consequential issue the client did not ask about gets raised.
4. The deliverable is complete enough for a human reviewer to sign.

Require ≥95% of that checklist. Same harsh bar Dayjob uses for pass@1: calculations, supporting evidence, unasked consequential issues, finished deliverable. “Mostly right” still costs six figures when one deadline dies quietly.

Before finalizing any recommendation:
1. List every premise the client stated that is not independently verified in the files.
2. Search all records for dates, jurisdictions, and status facts that could reverse the advice.
3. Surface any deadline, nexus, or eligibility cliff the client did not mention.
4. Cite file + location for every material number and conclusion.
If evidence conflicts or is missing, say so and stop short of a definitive filing position.

Pair the primary agent with a cheap second pass whose only job is bucket (c). Cost ticks up. Silent liability drops.

Practice loop with what is public today

Full Tax’s 50 environments are largely request-gated as of late October 2026 – email [email protected]. You still copy the evaluation pattern without waiting.

1. Clone the open use: git clone https://github.com/surge-ai/dayjob (Apache 2.0 on GitHub; Harbor containerized tasks).
2. Pull the public Healthcare and Finance samples on Hugging Face. Run two end-to-end jobs so you feel median file sprawl before you invent tax packs.
3. Fabricate a mini tax workspace: 20-40 mixed files (W-2s, K-1 drafts, email threads, one blurry receipt image, a ledger that contradicts the client memo). Write a 60-80 word prompt that under-specifies the real problem.
4. Judge with the four buckets above. No partial credit theater.

I ran a stripped multi-state trap on my own stack. Single-state filing, happy tone – until I forced “scan every log for location evidence.” One instruction. Different liability profile.

What the official numbers punish

10.8% is the ceiling on the Dayjob: Tax leaderboard right now (Claude Sonnet 5.5 Adaptive/Max). Claude Fable 5.1 lands 9.6%; Claude Opus 5.5 at 9.2%. Several strong models sit at 0%. Figures can move with new configs – check the live page.

Workspace shape from the Surge announcement: 50 environments across individual/family, small-business, partnerships, property/investments, estates/gifts/trusts, international, corporate transactions, tax review/disputes. 1,789 files total; median 30 per assignment (max 123); heavy PDF load plus Excel, Word, email, images, CSV, JSON, text, including handwritten receipts.

Who owns the last mile of professional judgment when the client memo is cheerful and incomplete? That question sits under every low double-digit pass rate.

Unasked deadline (Anna condo sale). Both proposed closing dates met the two-out-of-five residence test for up to $250k excluded gain. Move-out was August 2024, so eligibility cliffs around end of August 2027. Per Surge’s write-up, Opus surfaced that cliff on 5/5 attempts. Astra: 0/5. Correct answer to the asked question. Still a fail on consequential issues.

Locked premise (retirement split). Records invited an “unmarried” reading. One family of runs carried that premise through all five attempts even after noting uncertainty. Another line flagged verification on 4/5. Bucket (c) again – not arithmetic.

Turns out the same pass-everything design showed up earlier on Healthcare/Finance in the DAYJOB paper (arXiv:2610.01306): median rubric criteria 47.5 / 57.5, pass only if every criterion clears, strongest models still around ~24% there. High per-criterion accuracy compounds into low task passes. Your shiny Schedule K-1 math demo can coexist with an end-to-end fail.

Honest limits

Home-made rubrics stay noisier than Surge’s expert-authored judges until a CPA iterates with you. Human pros would not score a clean 100 on these edge-heavy packs either – HN chatter after launch was blunt about that – but they still beat agents on the judgment axis.

Plan human sign-off on every consequential call. Narrower benches (knowledge drills, pure calc, info-gathering) can look fine while long-horizon agents collapse on “find the problem nobody typed.”

FAQ

Is 10.8% really the best any model can do right now?

On this leaderboard and 95% threshold, yes – as of the October 2026 drop. Several well-known models are at zero. Recheck the live board before you quote it in a partner meeting.

Can I run Dayjob: Tax locally today without emailing Surge?

Not the full 50. Clone the GitHub use, run public Healthcare/Finance Harbor tasks, then build smaller tax-style folders with your own files and the four-bucket ≥95% checklist. That stress-tests judgment before you request Tax access.

Should firms stop using AI for tax work because of these scores?

No. Stop treating agents as unsupervised preparers. Let them retrieve, draft calculations, and sort documents. Keep a human – or a strict second agent – on two gates only: “what did the client not ask?” and “does the evidence actually support this?” Firms that treat 10.8% as a ban miss the point; firms that ignore the failure modes will eat the penalty later. The benchmark’s gift is forced exposure of silent misses, not a morality play about tools.

Next action: take one closed client folder, strip identifiers, write a short under-specified prompt, score your current agent on the four buckets. One afternoon. You’ll know whether your stack survives a Dayjob-style pack – or only survives demos.