NeMo Evaluator Plugin
OfficialFreeEvaluate models and datasets efficiently with NeMo.
Free · Opens the source repo
What NeMo Evaluator Plugin does
The NeMo Evaluator Plugin is designed to facilitate the evaluation of models, datasets, and agents using NVIDIA's NeMo framework. It provides a structured approach to selecting evaluation interfaces and metrics, validating examples, and submitting evaluation jobs to the NeMo Platform. By using this plugin, developers can streamline their evaluation processes, ensuring that they choose the right metrics and execution interfaces for their specific needs.
To get started, users must clarify the type of evaluation they require—whether it is dataset-driven or task-driven. The plugin supports both types, allowing for flexibility in how evaluations are structured. After selecting the appropriate evaluation type, users can define the metrics that will be used to score the evaluation, ensuring that the behaviors of interest are accurately measured. The plugin also provides guidelines for creating minimal examples to validate the metrics before scaling up to larger evaluations.
The NeMo Evaluator Plugin is particularly useful for developers and data scientists working with machine learning models who need to assess performance in a systematic way. It enables quick iterations on metric selection and allows for integration with the NeMo Platform, which can handle more complex evaluation tasks. The CLI commands provided make it easy to access various functionalities, such as listing available metrics, validating inputs, and submitting jobs for evaluation.
While the plugin offers robust capabilities for model evaluation, it is essential to be aware of its limitations. For instance, certain environment variables must be configured correctly, and users should be cautious of HTTP errors during job submissions. Overall, the NeMo Evaluator Plugin is a valuable tool for those looking to enhance their evaluation workflows in machine learning projects.
When to use it
Use this plugin when you need to evaluate machine learning models or datasets systematically, especially in a NeMo environment.
When not to use it
Avoid this tool if you're not working within the NeMo framework or if your evaluation needs are very simple and do not require structured metrics.
What you can build with it
Validating Metrics
Use the NeMo Evaluator Plugin to validate your metrics with minimal examples before scaling your evaluations.
Submitting Evaluation Jobs
Submit evaluation jobs to the NeMo Platform for both dataset-driven and task-driven evaluations.
Iterating on Metrics
Quickly iterate on metric selection and validation to refine your evaluation processes.
How to install NeMo Evaluator Plugin
View source1. Install with the skills CLI
npx skills add nvidia/skills/nemo-evaluator-plugin --agent claude-code2. Or install it manually
Download the skill folder and drop it into ~/.claude/skills/ for all projects, or .claude/skills/ to scope it to one repo. Restart Claude Code so it picks up the new skill.
Anthropic's agentic coding CLI, and the reference implementation of Agent Skills. Drop a skill folder into ~/.claude/skills and Claude Code loads it automatically whenever a task matches the skill's description. Claude Code docs
Inside SKILL.md
Written by nvidiaEvaluator Plugin
The Plugin CLI entrypoint is uv run nemo evaluator.
Purpose
Use this skill to choose an evaluation interface and metric, validate a minimal example, submit a NeMo Platform evaluation job, and retrieve its results.
Inputs
Establish these inputs before building an evaluation:
- Evaluation interface: dataset-driven vs. task-driven agentic evaluation
- Execution interface: standalone SDK evaluation or a durable NeMo Platform job.
- Pass/fail dataset examples: the smallest representative pass and failure cases.
- Metrics: the behaviors to score and the template fields they consume.
- Target: no target for offline scoring, or the model, agent, runner, or precomputed trials that produce outputs.
Instructions
- Clarify whether the input is dataset-driven rows or task-driven agent work.
- Choose the simplest metric that measures the requested behavior. Prefer deterministic metrics when possible.
- Build a tiny smoke case with one expected pass and one expected failure.
- Validate metric behavior with the standalone SDK and inspect row-level output plus aggregates.
- Fix field mappings, prompts, parsers, or task definitions before scaling.
- Submit the platform job only after the input and scoring shape works.
Read Metric Selection before choosing a metric for a rubric, RAG workflow, or tool-calling evaluation.
Choose the execution interface
| Need | Interface |
|---|---|
| Fast metric iteration without NeMo Platform | nemo_evaluator_sdk.Evaluator |
| Dataset-driven platform job | client.evaluator.submit(...) or nemo evaluator evaluate submit |
| Multiple inline/stored metric refs in one job | nemo evaluator evaluate submit with an EvaluateInputSpec |
| Task-driven platform job | nemo evaluator agent-evaluate submit |
| Reusable platform definitions and result indexes | client.evaluator.metrics, .tasks, .tasksets, .eval_results, .agent_eval_results |
Default to submit for every plugin evaluation. The plugin's local execution
path — client.evaluator.run() and the nemo evaluator ... run CLI verb — is
being retired, so do not build on it even though --help still lists it. For
fast metric iteration without the platform, use the standalone
nemo_evaluator_sdk.Evaluator instead.
- Read SDK Execution for datasets, targets, configuration, field mapping, job lifecycle, and custom metric packaging.
- Read Stored Resources for persisted definitions and result queries.
Limitations
api_key_secretis an environment-variable name standalone but a NeMo Platform secret name onsubmit. See API Auth.- HTTP 409 from a submission often means a referenced platform secret is missing, not a duplicate job. Read the response body.
intentis grader metadata and is never shown to the agent; onlyinputsreaches it.- Metric templates use
item.*for dataset rows butreference.*,sample.*, andinputs.*in agent evaluation. - Metric progress can reach 100 percent before the platform job is terminal.
Always call
job.wait_until_done()before retrieving results or downloading artifacts.
CLI Interface
Prerequisites
All commands in this file assume that the shell's working directory is the root of the NVIDIA-NeMo/nemo-platform repository.
In a NeMo Platform repository checkout, run commands through the workspace:
# confirms plugin readiness and lists the registered evaluator jobs.
uv run nemo evaluator info
# lists available metric names; add a metric name to print its schema.
uv run nemo evaluator metric-types
# next two commands print the dataset-driven and task-driven job input and
# output schemas - can be very large, use with caution to avoid filling up the context window.
uv run nemo evaluator evaluate explain
uv run nemo evaluator agent-evaluate explain
When the skill and plugin are installed, use the installed nemo command
without assuming a repository root or manually activating .venv.
Resolve bundled assets relative to this skill directory. In this repository the
canonical path is skills/nemo-evaluator-plugin; an installed skill may live
under a different skills root.
Bundled assets
| Path | Use |
|---|---|
assets/specs/exact_match_metric.json | Two-row offline smoke spec; submit as-is |
assets/specs/llm_as_judge.json | Online generation + judge; local-first (NVIDIA_API_KEY) |
assets/specs/fabric_agent_eval.json | Task-driven Fabric runner spec |
assets/examples/plugin_sdk_examples.py | Copyable SDK snippets for each plugin surface |
Available Scripts
| Script | Purpose | Arguments |
|---|---|---|
scripts/generate_example_specs.py | Generate or drift-check bundled specs | --check, --write |
In this repository, NeMo uses the displayed workspace command:
uv run --frozen python skills/nemo-evaluator-plugin/scripts/generate_example_specs.py --check
Do not assume a client-specific run_script() helper; use the displayed
uv run command.
Examples
Dataset-driven evaluation examples
- Follow Validate standalone, then submit to the platform. for the two-row pass/fail smoke test and its CLI submission.
- Follow Map noncanonical fields
when dataset columns need
field_mapping. - Follow Getting job results for submission, terminal waiting, result retrieval, and artifact download.
- Follow Store a metric, task, and taskset for reusable definitions, and Query persisted results for result lookup.
Task-driven agent evaluation examples
Standalone SDK evaluation
Use AgentEvaluator().run(...) for standalone task-driven SDK evaluation. Its
target can be a Model, a GenericAgent, or a direct AgentTaskRunner.
Platform job evaluation
Use the plugin agent-evaluate submit job for platform task evaluation. Its
target is a ModelTarget, AgentTarget, CodexRunnerTarget,
FabricRunnerTarget, or HarborRunnerTarget; alternatively provide
precomputed trials. Provide exactly one of target or trials.
Submission accepts inline tasks or a stored TasksetRef. Stored tasksets are
resolved in the target workspace.
Read Agent Evaluation for inline tasks,
TasksetRef, concurrency, fail-fast behavior, result artifacts, and runner
configuration.
Prepare Fabric in a repository checkout
Fabric runner examples and tests need the optional harness adapters and the matching Relay gateway:
uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact
script/dev-install-fabric.sh
The install script downloads the checksum-verified nemo-relay binary that
matches the locked Python bindings. Add its reported directory to PATH, then
use uv run --frozen --no-sync ... for Fabric checks so uv does not remove the
optional adapters.
Output Format
Report a completed platform evaluation in this form:
Job: <job-name>
Status: <terminal-status>
Metrics: <metric-names>
Mean: <aggregate-mean>
Artifacts: <downloaded result or artifact location>
Errors: <error messages>
Read specialized references
- Read Evaluator API Auth before using a model, agent, remote metric, or durable submission.
- Read LLM Judge before writing judge scores, prompts, or parsers.
Troubleshooting
Read Evaluator troubleshooting when schema, authentication, job, result, or runner behavior fails.
Follow security best practices
Never print, serialize, or commit secret values. Store only environment-variable names or platform secret references in specs and examples.
Frequently asked questions about NeMo Evaluator Plugin
Similar skills
Heap Snapshot Analysis
Investigate V8 heap snapshots for memory issues.
VS Code Performance Workflow
Automate performance investigations in VS Code.
Memory Leak Audit
Prevent memory leaks with effective coding patterns.
CPU Profile Analysis
Analyze V8 and Chrome performance profiles for optimization.
Chat Performance Testing
Benchmark and validate chat UI performance in VS Code.
Vercel React Best Practices
Optimize your React and Next.js applications for performance.