Benchmark the tax agent. Release the evidence.

A benchmark can qualify a pilot, not release client work

Tax work is not a prompt followed by a number. Records conflict, clients omit facts they do not know matter, authority changes by year and jurisdiction, calculations feed multiple forms, and the most consequential issue may sit outside the literal request. Two new benchmarks make that gap measurable.

Surge AI's DAYJOB: Tax uses 50 environments and 1,789 files across eight practice areas. Agents receive folders containing PDFs, spreadsheets, emails, images, and other records, then produce professional deliverables. Surge reports that the strongest tested agent passed 10.8 percent of assignments at its strict threshold. In one case, an agent answered the client's proposed home-sale dates but missed the later deadline after which the residence requirement could fail. The answer was locally correct and professionally incomplete.

Columbia DAPLab's TaxLab takes a different approach. It uses 3,142 source-grounded executable rules covering 63 U.S. federal individual tax issues and 55 CPA-reviewed synthetic scenarios. Some taxpayer facts are held back behind a simulated taxpayer, so the agent must recognize missing information and ask. The project reports 64.0 percent best perfect-filing accuracy; access to IRS documents improved one tested model's score by 7.3 percentage points. Results remain publisher-reported, and TaxLab is under review with open-source release planned.

These are useful but different tests. DAYJOB asks whether an agent can own messy work and satisfy an expert rubric. TaxLab asks whether it can develop facts, consult a bounded authority set, and complete fields matching a rule-derived gold return. Neither score proves fitness for a firm's clients, years, jurisdictions, software, privacy commitments, or professional obligations.

The IRS Office of Professional Responsibility's June 2026 guidance is direct: practitioners using generative AI remain responsible for due diligence, including verification of facts, citations, and calculations. Human scrutiny is not a decorative last step. It is the professional control that determines whether draft work is correct, supportable, confidential, and suitable for delivery or filing.

Read the metrics as diagnostic instruments

Do not average away the failure you care about. TaxLab reports that stronger models concentrate errors in particular issues; one model's medical and dental deduction error rate was 5.2 times its own average. A model can look strong overall and still be unsafe for the exact work assigned. Score by practice area, issue, deliverable, fact pattern, tax year, source quality, and failure consequence.

Repeat tasks. DAYJOB describes five-attempt examples where models differed in consistently noticing a deadline. One successful run does not establish a dependable control. Track pass-at-one, pass across repeated attempts, safe abstention, unsupported citation rate, missed-fact rate, calculation accuracy, issue-spotting recall, reviewer corrections, and total reviewer time.

Keep benchmark claims bounded. An expert rubric is useful, but it reflects its authors' task design. A gold return is useful, but it depends on the rule system and scenario facts. A model can match fields for the wrong reason, and a reviewer can accept a persuasive memo with a hidden error. Preserve source and calculation traces so a score can be explained.

Build four gates instead of one generic review

Gate one is scope and custody. Freeze the taxpayer, engagement, tax year, jurisdictions, deliverable, data set, authority set, toolchain, permissions, and owners. Confirm the tool is approved for client information and that retention, access, and location meet policy and engagement terms. A broader context window is not permission to upload a broader client file.

Gate two is fact development. Convert records and representations into atomic facts with source identifiers. List what is missing. Test whether the agent asks narrow questions instead of filling gaps from patterns. Hold back facts in the challenge suite: residence dates, ownership percentages, elections, related-party status, basis adjustments, state activity, signatures, and deadlines.

Gate three is authority and calculation. Open every material citation and confirm issuer, text, tax year, jurisdiction, effective date, hierarchy, and applicability. Recalculate material amounts outside model prose using approved software, formulas, scripts, or manual procedures. Trace each final value through facts, rule, calculation, form field, workpaper, and communication.

Gate four is qualified release. The independent reviewer examines source records, issue spotting, contrary authority, missing facts, calculations, exceptions, and the exact deliverable. Approval binds to hashes. Changed facts, authority, formulas, tool versions, or text reopen affected review.

tax_agent_acceptance: v1
assignment:
  tax_year: 2026
  jurisdictions: [US-Federal, CA]
  deliverable: client_memo_and_calculation
evidence:
  fact_sources: required
  primary_authority_pinpoints: required
  material_calculations_reperformed: true
challenges:
  missing_fact_cases: 12
  contrary_authority_cases: 6
  repeated_runs_per_case: 5
release:
  agent_can_file: false
  qualified_reviewer: required
  exact_hash_approval: required
  approval_expires_on_change: true

Sampling should follow consequence. Inspect every election, filing status, deadline, material position, penalty exposure, client representation, unresolved assumption, and advice statement. Add targeted cases around changed law, boundaries, phase-outs, related parties, new entities, large transactions, and contradictory records. A random sample can catch unknown patterns, but it cannot replace complete review of high-consequence items.

Worked example: the home-sale answer that missed the real deadline

A client moved out of a condominium in August 2024 and asks whether an October 2026 or January 2027 sale is better. A draft correctly determines that both proposed dates satisfy the usual two-out-of-five-year residence condition for the principal-residence exclusion. It then recommends either date.

The acceptance workflow does not stop at the literal choice. The fact register records acquisition, ownership, occupancy, move-out date, prior exclusion claims, use after move-out, planned sale dates, basis, improvements, selling costs, and state facts. Missing items become questions. The authority register links the residence and ownership tests, dollar limits, timing, exceptions, and applicable year. A qualified reviewer checks the authority and whether any later guidance changes it.

The consequential-issue test asks what happens if the sale slips beyond the proposed dates. The draft should identify the approximate August 2027 boundary after which the five-year lookback may no longer include two years of residence, subject to exact facts and applicable rules. It should not turn that observation into client advice without verifying every required fact and exception. The useful output is a dated scenario table, assumptions, authority, calculation, missing facts, and an explicit review gate.

This example shows why “correct answer” is a weak acceptance unit. Professional completeness includes recognizing when the client's frame is too narrow, then developing facts and escalating judgment without inventing them.

Failure modes to test before release

The IRS guidance also flags billing. AI may reduce research or drafting time; practitioners should not bill for manual labor not actually performed or double bill AI-assisted tasks. Include actual operator, tool, reviewer, and rework time in the packet so engagement owners can apply policy transparently.

Do not let the AI write its own quality conclusion. It can assemble evidence and identify discrepancies, but the preparer and reviewer must describe the procedures they actually performed. A green tool status proves workflow completion, not due diligence.

A 30-day bounded pilot

1. Select 20-40 de-identified, completed assignments across no more than two practice areas and one tax-year range.
2. Create answer and evidence keys with qualified practitioners, including missing-fact questions, authority, calculations, issues, deliverables, and known traps.
3. Freeze the toolchain and run each case repeatedly. Preserve every tool call, source, output, exception, and reviewer correction.
4. Test baseline prompts, IRS-document access, and bounded tools separately so gains can be attributed.
5. Inject missing facts, contradictory records, stale authority, wrong-year forms, duplicate documents, and plausible distractors.
6. Require independent recalculation of every material field and review of every consequential position.
7. Measure strict task pass, field accuracy, issue recall, unsupported citations, unsafe assumptions, abstention, reviewer minutes, rework, cost, and privacy exceptions.
8. Shadow the existing process. Do not sign, file, post, advise, or communicate from the pilot output.
9. Approve only the task classes that meet predeclared thresholds; document excluded tasks and rollback.
10. Requalify after model, retrieval, prompt, tax-year, authority, software, or permission changes.

A good pilot may conclude that the agent is useful for source inventory, question drafting, or deterministic comparison but not for issue spotting or client communication. Partial scope is a success when the boundary is supported by evidence. The wrong outcome is a broad launch justified by an average leaderboard score.

FAQ

Can a benchmark approve an AI tax agent for client work?

No. It can support selection and pilot scope. Client work needs current facts, authority, calculations, confidentiality, review, and release evidence.

Which benchmark is more useful, DAYJOB or TaxLab?

They answer different questions. DAYJOB tests messy professional assignments and deliverables. TaxLab tests interactive fact development and field-level filing. Use the one closest to the task, then build internal cases.

Must every AI citation be checked?

Every material citation should be opened and verified for issuer, wording, year, jurisdiction, effective date, and applicability. Search snippets and fluent summaries are not authority.

Can the agent ask the client questions?

It may draft questions in an approved workflow. A named practitioner decides what to ask, how to communicate, what confidentiality rules apply, and how the answer changes the analysis.

Who owns the final conclusion?

The qualified practitioner and authorized engagement or release owner. The AI cannot own professional judgment, client advice, signature, filing, election, or approval.

Sources and further reading

• Surge AI: DAYJOB: Tax - checked November 1, 2026; first-party benchmark description and reported results.
• Columbia DAPLab: TaxLab - field-level rules, scenarios, methodology, and reported results.
• IRS OPR Issue 2026-19 - responsible AI use, due diligence, verification, confidentiality, and billing.
• AICPA & CIMA explanation of IRS guidance.
• Thomson Reuters: meeting IRS expectations - practitioner interpretation; vendor source.
• DAYJOB benchmark paper - long-horizon professional-work evaluation methodology.
• TaxCalcBench - tax-calculation benchmark.
• The Verification Tax - limits and cost of detecting rare model errors.
• Hacker News discussion of DAYJOB: Tax - current developer/practitioner discussion; not tax authority.

Related workflows: accounting research evidence, AI workpaper review, and source-document verification.