Surge AI's DAYJOB: Tax Shows LLM Agents Can Still Miss Anna's August 2027 Deadline

By David Nakamura | AI & Data News | AIDataNewsGlobal LLMs desk | Contact: editorial@aidatanews.com

The more revealing failure mode for an LLM agent is not a wrong calculation. It is a response that answers the asked question accurately while leaving a time-sensitive condition unmentioned. That is the space DAYJOB: Tax is designed to probe. The newest addition to Surge AI's DAYJOB family asks whether agents can take professional tax work end-to-end: sift through incomplete records, detect consequences that were not requested, reconcile conflicting evidence, and deliver work that could stand in a real engagement.

The Anna Case: Correct Timing Analysis, Missing Expiry

Anna was considering two dates for closing on the sale of her condo: October 2026 and January 2027. She had left the property in August 2024 and wanted to know which timing would better support an exclusion worth up to $250,000 in gains.

The agents evaluated in this example reached the narrow conclusion that either proposed date could satisfy the residence condition. The issue they did not consistently raise was broader. Under the usual rule, a homeowner must have lived in the home for at least two of the five years before the sale. With a departure date of August 2024, the safe window would be expected to close around the end of August 2027; a transaction pushed past August 2027 could therefore threaten eligibility.

Over five attempts, Astra did not identify that expiry in the runs, while Opus surfaced it in all five attempts. The practical consequence is easy to state: correct advice about the two dates can still omit the constraint that changes the decision.

Why the Benchmark Was Built This Way

Earlier DAYJOB releases covered Healthcare and Finance. DAYJOB: Tax extends the approach to tax preparation, compliance, and advisory work, where the initial instruction often does not define the whole problem. It contains 50 environments spread across eight areas of tax practice. Each environment presents a client request and supporting records, then asks the agent to produce tax returns, workpapers, technical reviews, or advice.

The evaluation is therefore not limited to tax knowledge. An agent must decide which lines of inquiry deserve investigation, how to handle evidence that points in different directions, when information is too sparse to rely on, and how to reflect those decisions in the completed deliverable.

Leaderboard, Pass Rule, and Cost Frontier

On the current DAYJOB: Tax leaderboard, Claude Sonnet 5.5 is first at 10.8%, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%.

The scoring rule is intentionally unforgiving. Results are reported at a 95% rubric threshold: an attempt passes only if it meets 95% or more of the criteria, which can include calculations, supporting evidence, consequential issues, and the completed deliverable.

Performance measured against trial cost

The benchmark also compares results with average cost per trial. On the high-performance end of the measured Pareto frontier sit Claude Sonnet 5.5 and Opus 5.5.

What Counts as Professional Tax Work in the Tasks

The benchmark covers eight areas of professional tax work, each requiring different rules, records, and expertise:

• Individual and family tax
• Small-business tax
• Partnerships
• Property and investments
• Estates, gifts, and trusts
• International tax
• Corporate transactions
• Tax review and disputes

Some assignments require complete filing packages with supporting workpapers. Others ask the agent to investigate a transaction, review prior filings, or advise on a decision that has not yet been made.

Files, Receipts, Ledgers, and Messy Real-World Evidence

Across the assignments, there are 1,789 workspace files. Per assignment, the median of 30 files can grow to as many as 123. The file mix includes 1,487 PDFs, 117 Excel workbooks, and 38 Word documents, plus emails, images, CSVs, JSON records, and text files.

The workspaces are populated with filed and draft tax returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts, and correspondence. They also include the less structured evidence that professionals routinely have to interpret.

In one task, the record set includes a photograph of a handwritten receipt for merchandise purchased for resale. The agent has to read the descriptions and amounts, reconcile them with spending records, and determine how the purchases affect inventory and cost of goods sold.

In another task, an industrial silo-services company assumes it operates only in Alabama. The field dispatch log, however, shows crews working across five states; job locations, crew sizes, and days on site all bear on multistate tax obligations.

Other environments include a cross-border household whose Polish employment and property records must be matched against cryptocurrency exchange exports, wallet histories, and historical purchase lots.

Expert Construction and Three-Layer Review

DAYJOB: Tax starts from real-world professional workflows and turns them into realistic assignments through tax experts, with supporting records, expected outcomes, and grading criteria. Each task undergoes a three-layer expert review for correctness, completeness, and solvability. The experts cover corporate taxation, partnerships, individual returns, trusts, and multistate compliance, including:

• Corporate and private-equity taxation: A tax attorney spent over 10 years at PwC, Deloitte, and Exxon, covering tax planning, compliance, audits, and controversy.
• Partnership and corporate taxation: A Baker Tilly Tax Director with experience at PwC, Moss Adams, and Windes specializes in complex partnership allocations, consolidated corporate returns, REITs, and high-net-worth clients.
• Real estate and individual taxation: A tax professional with experience at Deloitte and EY focuses on real-estate partnerships, individual taxation, and trusts.

Assumption Risk: The Retirement-Account Example

In another task, a separating couple asked how to divide a Roth IRA and a 401(k). Opus 5.5 recommended keeping the accounts separate and making a cash payment to even out balances, starting from the assumption that they were not legally married.

One response opened:

You aren't legally married, so neither account can be split without triggering taxes and penalties.

Records in the client file complicated that conclusion. The couple now lived in Florida, where new common-law marriages generally are not recognized, but they had previously lived together in Texas and held a ceremony in Colorado. Texas recognizes informal marriages under certain conditions, and a marriage established elsewhere may remain valid in Florida.

That distinction can change the tax consequences of splitting retirement accounts. The model kept its recommendation tied to the unmarried assumption across all five attempts, even when it acknowledged uncertainty. Astra, by contrast, recognized the need to verify marital status in four of five attempts, and one response advised:

We first need to confirm your legal marital status.

Where This Sits Among Existing Evaluations

A defined problem is the common starting point for many tax evaluations: answer a legal question, calculate a return, or collect missing taxpayer information. Those are meaningful measurements. But much of professional tax work begins before the client knows what question should be asked. Clients provide incomplete instructions, records contain conflicting evidence, and important issues may be absent from the prompt.

DAYJOB: Tax tests whether agents can carry that end-to-end responsibility: discover issues the client overlooked, resolve conflicting evidence, and decide what needs to be done before a deliverable can be completed.

Benchmark	Core question	What it evaluates	What DAYJOB also asks
TaxBench	Is a difficult tax question answered correctly?	It evaluates 114 tax questions across multiple specialties, each with a defined problem and rubric.	Did the agent warn Anna when the eligibility from her two dates could expire?
TaxCalcBench	Is an accurate tax return calculated?	It evaluates tax-return accuracy against expected filing values using realistic documents and scenarios.	Do the underlying records support the claimed treatment, or is missing receipt evidence being treated as verified?
TaxLab	Can missing information be gathered and the return filed correctly?	It tests whether agents can question a simulated taxpayer, obtain missing information, and complete a federal individual return.	Can the agent carry responsibility across other deliverables: partnership loss analysis, estate return, review of a disputed tax position, or strategy letter?

Common Misconceptions

• A high task completion rate is the same as professional readiness. The benchmark's headline pass rates are low, which is precisely why the Anna-style misses matter.
• An answer that is numerically correct is necessarily useful. The condo example shows that correct treatment of October 2026 and January 2027 can still leave out a later August 2027 deadline.
• The best leaderboard model is automatically the best production choice. The benchmark compares performance with average cost per trial, but production selection still needs fit to workload, risk tolerance, and human review design.
• Hidden deadline detection is a minor UX issue. In this domain, it is a governance and financial-consequence issue.

Action Recommendations for Teams Shipping Tax-Aware Agents

1. Build prompts that ask for unrequested consequences, not just answers. If an agent must compare dates, it should also be required to list expiry windows, record gaps, and decisions that change the result.
2. Make assumption checks explicit. The retirement-account example shows the failure point: the agent did not verify the client's marital status before advising. Teams should require unresolved status questions to be surfaced before a final recommendation.
3. Score against 95%-style rubrics only if the deliverable can be inspected. The rubric criteria matter as much as the percentage: calculations, evidence, consequential issues, and completed deliverable.
4. Pair leaderboard position with cost and risk review. Claude Sonnet 5.5, Claude Fable 5.1, and Claude Opus 5.5 show different rankings; Opus 5.5 appears on the high-performance edge, but production decisions still need model-specific guardrails.
5. Keep the human tax professional in the loop until benchmark pass rates move meaningfully above current results. For now, agents have not cleared the audit.

FAQ

What does DAYJOB: Tax evaluate?

It evaluates whether AI agents can complete professional tax assignments that include messy records, incomplete instructions, conflicting evidence, and deliverables such as tax returns, workpapers, technical reviews, or client advice. The suite covers 50 environments across eight areas of tax practice.

What were the top scores?

Claude Sonnet 5.5 leads with 10.8%, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. Scoring uses a 95% rubric threshold for pass@1.

What does the Anna example prove?

It shows that an agent can correctly evaluate October 2026 and January 2027 against a condition worth up to $250,000 in excluded gains and still miss a later constraint: the residence rule requiring at least two of the five years, with the eligibility window closing around the end of August 2027.

Why are the file counts important?

They signal that the benchmark is not a single-question exam. It uses 1,789 workspace files, with a median of 30 per assignment and as many as 123, including 1,487 PDFs, 117 Excel workbooks, and 38 Word documents.

How should readers interpret the low pass rates?

As a boundary marker for current LLM agents: the measured tasks remain difficult, especially where professional judgment depends on issues not written into the request.

Bottom Line

The DAYJOB: Tax result is not a simple model ranking. It is a clearer statement of where LLM agents still fall short in tax work: the requested answer can be correct, while the professionally important consequence remains unspoken. For now, agents have not passed the audit, and the full leaderboard is maintained on the DAYJOB: Tax benchmark page.