Definition
Plain language
A benchmark of tool-calling tasks built on top of the Model Context Protocol.
As stated in the literature
A benchmark suite evaluating LLM tool-use agents on MCP-served tasks including filesystem and Postgres workloads, used as out-of-distribution transfer evaluation for Firefly-trained models.
Why it matters: It evaluates agents in something closer to real production setups, where tools and protocols matter as much as raw reasoning.
For example, an agent might be asked to query a Postgres database via MCP, transform the result, and write the output to a file, all through standard MCP tool calls.
Heard on the show
“Some MCPMark file system and database tasks.”Episode 059 — Firefly's Inversion: Building Verified Tool-Call Training Data by Working Backward