Catalogue

Tool-using agents serve a simulated user in retail and airline domains under written policy; graded on the final database state.

Dual-control follow-up to τ-bench: the user can act on tools too, with a new telecom troubleshooting domain.

Browser agents complete knowledge-work tasks on a live ServiceNow instance: forms, lists, catalogs and dashboards.

Customer-service tasks over a realistic Salesforce org with interconnected objects, from case routing to policy questions.

175 tasks inside a simulated software company with GitLab, chat, file storage and project tools; checkpoint-based partial credit.

Office automation across documents, spreadsheets, email and calendar apps, chaining several applications per task.

Everyday tasks across nine simulated apps exposing 457 APIs, graded by state-based unit tests that also catch collateral damage.

Berkeley Function Calling Leaderboard: single, parallel and multi-turn function calls checked by AST match and execution.

912 real spreadsheet manipulation questions from user forums, each checked against several test spreadsheets.

Enterprise text-to-SQL workflows over large BigQuery, Snowflake and local databases, often needing multi-step queries.

Text-to-SQL over 95 real databases with dirty values and external knowledge, scored by execution accuracy.

Numerical reasoning over earnings reports that mix tables and text, with gold calculation programs.

Conversational follow-ups on FinQA reports, where later questions depend on earlier answers.

Hybrid table-and-text questions from financial reports, including arithmetic, counting and sorting.

Open-book questions about public company filings, with evidence pages; an open sample of 150 is public.

162 legal reasoning tasks written with legal professionals, from issue spotting to rule application.

510 commercial contracts with expert labels for 41 clause types, such as change of control and non-compete.

Document-level inference over NDAs: is each of 17 hypotheses entailed, contradicted or not mentioned, with evidence spans.

Real work products from 44 occupations graded by industry experts; a gold subset is public.

Real GitHub issues from Python repositories, resolved when the hidden tests pass in a per-instance container.

Long-horizon web tasks on self-hosted shopping, forum, GitLab, CMS and map sites with functional checks.

Computer-use tasks across real desktop apps in full virtual machines, scored by execution-based scripts.

Drawing navigation and cross-document project coordination. Curated answers and structured checks; some defects are injected into public project records.

Drawing perception, OCR, counting and spatial questions. A drawing-reading benchmark rather than a CAD editing environment.

Civil-engineering drawing revision tasks. Grades recorded tool arguments; final CAD geometry is not independently checked by that scoring method.

Industrial CadQuery generation, reasoning and targeted editing. 17,900 execution-verified programs across 106 families; unseen-family transfer remains difficult.

18,000 evaluation samples across five input modalities. Measures reconstruction, executable programs and compactness. Separate from BenchCAD.

Parametric parts, controlled edits, assemblies and declared physics checks. 600 part tasks; only 164 parts support full L3 FEA. Solver assumptions are benchmark-assigned.

Simulation-executable building models in OpenSeesPy. Grades execution and period checks against empirical references; passing is not engineering certification.

CAD generation and editing with public inputs and private leaderboard reference artifacts. Official grading requires its hosted evaluation setup.