The harness-leverage result suggests a useful procurement lesson: benchmark the model and orchestration layer as a pair. A model that looks expensive in isolation may win on completed-task cost if it needs fewer retries, produces shorter tool traces, or benefits more from caching and decomposition. I would also distinguish token reduction from economic efficiency—external tool latency, human approvals, and failure recovery can dominate the bill. Have the authors published per-task variance? The average improvement is strong, but the tasks where harness design hurts quality may be the most operationally important.
Thanks—exactly. The average result is compelling, but per-task variance is what would make it actionable for deployment. A harness that lowers average cost but introduces a few expensive failure modes could still be the wrong choice in production. I’d love to see future benchmarks report both completed-task cost and the distribution of outcomes across task types.
Great points! I completely agree that models and orchestration should be evaluated together, since completed-task cost matters more than model cost alone.
I also agree that token efficiency is only one part of the equation - latency, human approvals, and failure recovery are equally important in production.
As for per-task variance, I don't recall the paper reporting it, but that analysis would certainly help identify where harness optimizations introduce trade-offs.
Exactly. The missing variance view is what would separate a broadly robust harness gain from an average driven by a few task classes. I’d want median and tail latency, retry count, human-intervention rate, and completed-task cost by task type—not just aggregate token savings. If the authors release traces, this could become a useful procurement benchmark rather than another model leaderboard.
The harness-leverage result suggests a useful procurement lesson: benchmark the model and orchestration layer as a pair. A model that looks expensive in isolation may win on completed-task cost if it needs fewer retries, produces shorter tool traces, or benefits more from caching and decomposition. I would also distinguish token reduction from economic efficiency—external tool latency, human approvals, and failure recovery can dominate the bill. Have the authors published per-task variance? The average improvement is strong, but the tasks where harness design hurts quality may be the most operationally important.
Thanks—exactly. The average result is compelling, but per-task variance is what would make it actionable for deployment. A harness that lowers average cost but introduces a few expensive failure modes could still be the wrong choice in production. I’d love to see future benchmarks report both completed-task cost and the distribution of outcomes across task types.
Great points! I completely agree that models and orchestration should be evaluated together, since completed-task cost matters more than model cost alone.
I also agree that token efficiency is only one part of the equation - latency, human approvals, and failure recovery are equally important in production.
As for per-task variance, I don't recall the paper reporting it, but that analysis would certainly help identify where harness optimizations introduce trade-offs.
Exactly. The missing variance view is what would separate a broadly robust harness gain from an average driven by a few task classes. I’d want median and tail latency, retry count, human-intervention rate, and completed-task cost by task type—not just aggregate token savings. If the authors release traces, this could become a useful procurement benchmark rather than another model leaderboard.