What does the Cost Benchmark skill do?

The Cost Benchmark skill runs performance tests on AI models against a corpus, providing empirical results that validate claimed efficiencies.

How can I customize the benchmarking process?

You can customize the benchmarking by setting various environment variables to choose models and adjust output paths.

Where are the results of the benchmarks stored?

Results are stored in `docs/benchmarks/runs/`, with both the latest results and historical records available.

What should I do if the win rate is below the threshold?

If the win rate falls below the set threshold of 0.80, you may need to investigate the model's performance or adjust the benchmarking parameters.

Can I compare different AI models using this tool?

Yes, you can compare different AI models by running benchmarks with various baseline configurations, including models like Gemini and Anthropic.

Cost Benchmark

Name: Cost Benchmark
Availability: InStock
Author: ruvnet

Free

Verify and measure AI cost performance effectively.

by ruvnet67.6k stars on ruvnet/ruflo

1 views

Updated Aug 10, 2026

Get this skill

Free · Opens the source repo

What Cost Benchmark does

Cost Benchmark is a tool designed for developers and data scientists who need to validate the performance of AI models against specific benchmarks. It runs a series of tests using scripts/bench.mjs on a structural and adversarial corpus, producing detailed results that are stored in a structured format. This allows users to compare the measured performance of different models against their claimed capabilities, ensuring that any assertions made about model efficiency are backed by empirical data.

The skill is particularly useful in scenarios where AI models are being evaluated for release or when modifications are made to the benchmarking corpus. By running the benchmark tests, users can confirm that the performance of AI models has not regressed and that any new cases are handled correctly. This is crucial for maintaining the integrity of model performance claims, especially in competitive environments where cost and efficiency are key factors.

Additionally, the tool supports various configurations through environment variables, allowing users to customize their benchmarking process. Users can choose to run baseline comparisons against models like Gemini or Anthropic's Sonnet and Opus, providing flexibility in evaluating cost-effectiveness across different AI solutions. The results are outputted in a markdown format, making it easy to review and share findings with stakeholders.

Cost Benchmark is ideal for teams involved in AI development who need a reliable way to validate performance metrics before making decisions based on model capabilities. It serves as a verification gate that enhances trust in the performance claims made by AI models, which is essential for both internal assessments and external communications.

When to use it

Use this tool before releasing a new version of an AI model or after modifying the benchmarking corpus to ensure performance claims are accurate.

When not to use it

This skill is not suitable for general-purpose benchmarking outside of the specified AI models or if empirical validation is not required.

What you can build with it

Pre-release Validation

Run benchmarks to ensure that the latest model version maintains its performance claims before public release.

Corpus Expansion Testing

After updating the benchmarking corpus, use this skill to verify that new cases are processed correctly without regressions.

Performance Auditing

When auditing performance claims, run the benchmarks to confirm that the model meets the required standards and can be marked as verified.

How to install Cost Benchmark

View source

1. Install with the skills CLI

npx skills add ruvnet/ruflo/cost-benchmark --agent claude-code

2. Or install it manually

Download the skill folder and drop it into ~/.claude/skills/ for all projects, or .claude/skills/ to scope it to one repo. Restart Claude Code so it picks up the new skill.

Anthropic's agentic coding CLI, and the reference implementation of Agent Skills. Drop a skill folder into ~/.claude/skills and Claude Code loads it automatically whenever a task matches the skill's description. Claude Code docs

Inside SKILL.md

Written by ruvnet

Cost Benchmark

Runs scripts/bench.mjs against the structural+adversarial corpus and writes per-case + summary results to docs/benchmarks/runs/. This is the verification gate that backs every measurable claim in cost-booster-edit / cost-booster-route.

When to use

Before publishing a release — verify booster win rate didn't regress.
After expanding bench/booster-corpus.json — confirm new cases route correctly.
When auditing a "claimed upstream" tag — flip it to "verified" once the bench supports it.
On a cost question ("is Sonnet 4.6 cheaper than Opus 4.7 for these tasks?") — re-run with BENCH_ANTHROPIC=1.

Steps

Run the bench from v3/ (where agent-booster resolves):

( cd v3 && node ../plugins/ruflo-cost-tracker/scripts/bench.mjs )                  # booster only — free, ~85 ms
( cd v3 && BENCH_LLM_BASELINE=1 node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Gemini 2.0 Flash (cheap)
( cd v3 && BENCH_LLM_BASELINE=1 BENCH_ANTHROPIC=1 \
     node ../plugins/ruflo-cost-tracker/scripts/bench.mjs )                          # + Sonnet 4.6 + Opus 4.7

Inspect the markdown summary printed to stdout. The gate metric is winRate (Tier 1 cases). Adversarial cases are tracked separately as escalationRate.
Persisted output lands at:
- docs/benchmarks/runs/latest.json — pointer to the most recent run
- docs/benchmarks/runs/<ISO-timestamp>.json — historical record
Read it back in subsequent skills (e.g. cost-report step 2 reads latest.json for live tier-spend numbers).

Smoke gates

winRate ≥ 0.80 on Tier 1 cases (smoke step 23). Lower the threshold by editing scripts/smoke.sh.
escalationRate is reported but ungated — adversarial cases are diagnostic.

Env overrides

Env var	Default	Purpose
`BENCH_LLM_BASELINE`	unset	`=1` runs the OpenAI-compat baseline
`BENCH_LLM_MODEL`	`models/gemini-2.0-flash`	Override the OpenAI-compat model
`BENCH_LLM_BASE_URL`	Gemini OpenAI shim	Override endpoint
`BENCH_ANTHROPIC`	unset	`=1` runs Anthropic baseline (Sonnet 4.6 + Opus 4.7)
`BENCH_ANTHROPIC_MODELS`	`claude-sonnet-4-6,claude-opus-4-7`	Comma-separated Claude IDs
`BENCH_OUT`	timestamped file	Override output path
`BENCH_QUIET=1`	unset	Suppress markdown summary

API keys auto-pulled from gcloud secrets (GOOGLE_AI_API_KEY, ANTHROPIC_API_KEY); override with BENCH_LLM_API_KEY / BENCH_ANTHROPIC_API_KEY.

Cross-references

ADR-0002 §"Decision 1" / §"Riskiest assumption" · cost-booster-edit/SKILL.md (verification table consumes this skill's output) · cost-report/SKILL.md step 2 (reads runs/latest.json).

Frequently asked questions about Cost Benchmark

Similar skills

Heap Snapshot Analysis

Investigate V8 heap snapshots for memory issues.

VS Code Performance Workflow

Automate performance investigations in VS Code.

Memory Leak Audit

Prevent memory leaks with effective coding patterns.

CPU Profile Analysis

Analyze V8 and Chrome performance profiles for optimization.

Chat Performance Testing

Benchmark and validate chat UI performance in VS Code.

Vercel React Best Practices

Optimize your React and Next.js applications for performance.