/
TauBenchBadge
Tags
Information
Organization
BenchFlow
Release Date
April 18, 2025
Paper: https://arxiv.org/abs/2406.12045
| Strategy | Pass^1 | Pass^2 | Pass^3 | Pass^4 |
|---|---|---|---|---|
| TC (claude-3-5-sonnet-20241022) | 0.460 | 0.326 | 0.263 | 0.225 |
| TC (gpt-4o) | 0.420 | 0.273 | 0.220 | 0.200 |
| TC (claude-3-5-sonnet-20240620) | 0.360 | 0.224 | 0.169 | 0.139 |
| TC (mistral-large-2407) | ?? | ?? | ?? | ?? |
| TC (gpt-4o-mini) | 0.225 | 0.140 | 0.110 | 0.100 |
| Act (gpt-4o) | 0.365 | 0.217 | 0.160 | 0.140 |
| ReAct (gpt-4o) | 0.325 | 0.233 | 0.185 | 0.160 |
| Strategy | Pass^1 | Pass^2 | Pass^3 | Pass^4 |
|---|---|---|---|---|
| TC (claude-3-5-sonnet-20241022) | 0.692 | 0.576 | 0.509 | 0.462 |
| TC (gpt-4o) | 0.604 | 0.491 | 0.430 | 0.383 |
| TC (claude-3-5-sonnet-20240620) | 0.626 | 0.506 | 0.435 | 0.387 |
| TC (mistral-large-2407) | ?? | ?? | ?? | ?? |
| TC (gpt-4o-mini) | ?? | ?? | ?? | ?? |
| Act (gpt-4o) | ?? | ?? | ?? | ?? |
| ReAct (gpt-4o) | ?? | ?? | ?? | ?? |
*TC = tool-calling strategy (the function-calling strategy reported in the paper)
git clone https://github.com/sierra-research/tau-bench && cd ./tau-bench
pip install -e .
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...
MISTRAL_API_KEY=...
Run a tool-calling agent on the τ-retail environment:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --user-model gpt-4o --user-model-provider openai --user-strategy llm --max-concurrency 10
Set max concurrency according to your API limit(s).
To run specific tasks, use the --task-ids flag. For example:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --user-model gpt-4o --user-model-provider openai --user-strategy llm --max-concurrency 10 --task-ids 2 4 6
This command will run only the tasks with IDs 2, 4, and 6.
By default, we use gpt-4o as the user simulator with strategy llm. You can use other models by setting the --user-model flag, or other strategies by setting the --user-strategy flag. For example, run a tool-calling agent with a claude user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model claude-3-5-sonnet-20240620 --user-model-provider anthropic --user-strategy llm
Other strategies:
To run react user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model gpt-4o --user-model-provider openai --user-strategy react
Example of a react user response:
Thought:
I should provide my name and zip code as I wasn't given an email address to use.
User Response:
Sure, my name is Yusuf Rossi, and my zip code is 19122.
To run verify user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model gpt-4o --user-model-provider openai --user-strategy verify
This strategy uses a subsequent LLM verification step to check if the user simulator's response is satisfactory. If not, the user simulator will be prompted to generate a new response.
To run reflection user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model gpt-4o --user-model-provider openai --user-strategy reflection
This strategy uses a subsequent LLM verification step to check if the user simulator's response is satisfactory. If not, the user simulator will be prompted to reflect on its response and generate a new response.
Often times, it is difficult and time consuming to manually identify specific error locations in trajectories as they can be long and the constraints can be complex. We have provided an auto error identification tool that can do the following:
Both of the labels are accompanied with a description.
To run the auto error identification, run:
python auto_error_identification.py --env <airline/retail> --platform openai --results-path <the path to your results file here> --max-concurrency 16 --output-path test-auto-error-identification --max-num-failed-results 10
Please note that this feature utilizes an LLM, which may lead to inaccurate error identifications.
*Notice: If an error is raised due to the structure of your results file, you may have to rerun the benchmark to produce a new results file. We have recently rewritten the benchmark to be more type-safe and extensible.
τ-bench might be expensive to run. We have provided a set of historical trajectories for the airline and retail environments in ./historical_trajectories.
If you would like to contribute your historical trajectories to this benchmark, please submit a PR!
See ./LICENSE.
Please submit issues or pull requests if you find problems with the benchmark.
@misc{yao2024tau,
title={$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains},
author={Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},
year={2024},
eprint={2406.12045},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2406.12045},
}