i4h Workflow — Annotate Dataset

Purpose

Use a VLM to verify whether each episode satisfies the env's task description. Use when the user asks to annotate, label episodes, filter demos, or gate finetuning on a success classifier.

Base Code

These steps drive the i4h-workflows base code (the workflows/agentic/ tree). To reuse an existing checkout, set I4H_WORKFLOWS to its path (no clone happens). Otherwise this resolves the current repo, or clones to ~/i4h-workflows — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• Annotation is optional. Do not run it during validation unless the user requests labels.
• For natural-language prompts such as "Run Annotation on all recorded episodes", annotate all episodes in one selected HDF5 recording, not every historical HDF5 under workflows/agentic/runs/. If the user does not name an HDF5, choose the latest annotatable recording: first look inside runs/.latest when it contains an HDF5, otherwise pick the newest non-annotation .hdf5 under workflows/agentic/runs/. Only batch across multiple HDF5 files when the user explicitly asks for all historical recordings, every HDF5 file, or a batch annotation run.
• Env config (source of truth): the annotator reads the success criterion (policy.task_description) from workflows/agentic/config/environments/<env>.yaml. Pass --task-description to override.
• Talks to an OpenAI-compatible endpoint via --base-url (default http://localhost:8000/v1) and --model (default Qwen/Qwen3-VL-8B-Instruct). Point both at a running vision-model server. Do not use text-only/code models such as qwen3-coder-next; offline annotation sends image inputs and requires a VLM.
• Keep every annotation artifact inside workflows/agentic/runs/<run>/. Do not create or access /tmp/annotate_* or other external temp directories.

Start VLM

Skip this section if an OpenAI-compatible endpoint serving a vision model is already running — just set VLM_BASE_URL/VLM_MODEL in Run to point at it. A local-agent server running qwen3-coder-next does not qualify because it is text-only. annotator/vllm.sh defaults to port 8000, so starting it on top of an existing server collides; don't.

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

For readiness, use workflows/agentic/annotator/vllm.sh ensure. Do not replace it with raw docker ps, fixed sleeps, ad hoc model-listing HTTP probes, or separate manual wait steps; the helper owns the start-and-wait policy.

Step 1 — start VLM (if needed)

Run this exact command. Do not add sleep, status, curl, or shell control operators around it:

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
"${REPO_ROOT}/workflows/agentic/annotator/vllm.sh" ensure

Run (Offline HDF5)

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

Step 1 — setup and resolve HDF5

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
ENV_ID=scissor_pick_and_place
RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"

# LLM endpoint + model (OpenAI-compatible vLLM). Defaults match annotator/vllm.sh; override to use an
# external vision server — e.g. VLM_BASE_URL=http://localhost:8000/v1 VLM_MODEL=qwen3-vl-32b
VLM_BASE_URL="${VLM_BASE_URL:-http://localhost:8000/v1}"
VLM_MODEL="${VLM_MODEL:-Qwen/Qwen3-VL-8B-Instruct}"

# Point HDF5_PATH at a real recording (absolute path). Recordings come from teleop, mimic, or
# validate (which writes data/verify.hdf5 under each runs/eval_* dir). If HDF5_PATH is not set,
# choose one recording: an HDF5 inside runs/.latest if present, otherwise the newest non-annotation
# HDF5 under runs/. "All recorded episodes" means all episodes inside this one HDF5.
HDF5_PATH="${HDF5_PATH:-}"
if [ ! -f "${HDF5_PATH}" ]; then
  LATEST_RUN="$(readlink -f "${RUNS_ROOT}/.latest" 2>/dev/null || true)"
  if [ -n "${LATEST_RUN}" ] && [ -d "${LATEST_RUN}" ]; then
    HDF5_PATH="$(
      find "${LATEST_RUN}" -name '*.hdf5' -type f -printf '%T@ %p\n' 2>/dev/null \
        | sort -nr | awk 'NR==1 { $1=""; sub(/^ /, ""); print; exit }'
    )"
  fi
fi
if [ ! -f "${HDF5_PATH}" ]; then
  HDF5_PATH="$(
    find "${RUNS_ROOT}" \( -path '*/annotate_*' -o -path '*/.latest' \) -prune -o \
      -name '*.hdf5' -type f -printf '%T@ %p\n' 2>/dev/null \
      | sort -nr | awk 'NR==1 { $1=""; sub(/^ /, ""); print; exit }'
  )"
fi
if [ ! -f "${HDF5_PATH}" ]; then
  echo "annotate: set HDF5_PATH to an existing .hdf5 (got '${HDF5_PATH:-<unset>}'). Candidates:" >&2
  find "${RUNS_ROOT}" \( -path '*/annotate_*' -o -path '*/.latest' \) -prune -o \
    -name '*.hdf5' -type f -printf '%TY-%Tm-%Td %TH:%TM  %p\n' 2>/dev/null | sort -r | head
  exit 1
fi

RUN_DIR="${RUNS_ROOT}/annotate_${ENV_ID}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${RUN_DIR}/data" "${RUN_DIR}/logs" "${RUN_DIR}/tmp"
ln -sfn "${RUN_DIR}" "${RUNS_ROOT}/.latest"

Step 2 — annotate offline

TMPDIR="${RUN_DIR}/tmp" "${REPO_ROOT}/workflows/agentic/annotator/run.sh" \
  --env "${ENV_ID}" \
  --base-url "${VLM_BASE_URL}" \
  --model "${VLM_MODEL}" \
  --output "${RUN_DIR}/annotations.jsonl" \
  offline \
  --hdf5-path "${HDF5_PATH}" \
  --filter "${RUN_DIR}/data/filtered.hdf5" \
  > "${RUN_DIR}/logs/annotator.log" 2>&1

Step 3 — summarize annotations

SUCCESS_COUNT=$(grep -c '"success": true' "${RUN_DIR}/annotations.jsonl" 2>/dev/null || true)
FAILURE_COUNT=$(grep -c '"success": false' "${RUN_DIR}/annotations.jsonl" 2>/dev/null || true)
printf 'annotations: success=%s failure=%s\n' "${SUCCESS_COUNT}" "${FAILURE_COUNT}"
grep -E "Traceback|Error|FAILED" "${RUN_DIR}/logs/annotator.log" || true

Step 4 — stop VLM (only if you started it in Start VLM)

Skip when using an external server (e.g. the local-agent one) — it would kill that server.

"${REPO_ROOT}/workflows/agentic/annotator/vllm.sh" stop

Live Mode

Annotate the latest camera frames from a running policy/Arena session over Zenoh (cameras default to the env config). Use only when such a session is already up and the user asks for live judging.

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

Step 1 — setup

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
ENV_ID=scissor_pick_and_place
RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"
VLM_BASE_URL="${VLM_BASE_URL:-http://localhost:8000/v1}"
VLM_MODEL="${VLM_MODEL:-Qwen/Qwen3-VL-8B-Instruct}"
RUN_DIR="${RUNS_ROOT}/annotate_live_${ENV_ID}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${RUN_DIR}/tmp"

Step 2 — annotate live

TMPDIR="${RUN_DIR}/tmp" "${REPO_ROOT}/workflows/agentic/annotator/run.sh" \
  --env "${ENV_ID}" \
  --base-url "${VLM_BASE_URL}" \
  --model "${VLM_MODEL}" \
  --output "${RUN_DIR}/live.jsonl" \
  live \
  --count 5 \
  --interval 2.0 \
  --timeout 30.0

• --count 0 runs forever; --interval is seconds between snapshots; --timeout is how long to wait for first frames from every camera.
• --min-success-frames N (needs a finite --count) exits non-zero unless at least N sampled snapshots pass — use it as a gate.
• --dump-frames-dir DIR saves sampled frames; add --dump-frames-only to dump without calling the VLM.
• --cameras a,b overrides the env's Zenoh camera names.

Verify

• annotations.jsonl exists.
• Filtered HDF5 exists when --filter was passed.
• Tally success/failure counts from the JSONL before reporting.

Prerequisites

• Workflow set up via [[i4h-workflow-setup]] (the .venv must exist).
• An existing HDF5 recording to annotate (set HDF5_PATH to an absolute path; the Run block lists candidates if it's unset or wrong).
• A reachable OpenAI-compatible endpoint serving a vision model — either start the annotator's own (annotator/vllm.sh start) or point VLM_BASE_URL/VLM_MODEL at an existing vision server. The current qwen3-coder-next local-agent endpoint is not sufficient because it is text-only.
• Annotation is optional — only run it when the user requests labels.

Limitations

• Annotation is optional and is not run during validation unless requested.
• Requires a reachable OpenAI-compatible vLLM server; defaults to localhost:8000/v1.
• Live mode applies only when a policy/Arena session is already running and the user requests live judging.
• The annotator reads task text from the env YAML; override per-run with --task-description.

Troubleshooting

• Error: .venv not found / module import fails - Cause: workflow not set up. Fix: run [[i4h-workflow-setup]] first.
• Error: connection refused at localhost:8000/v1 - Cause: no vLLM at VLM_BASE_URL. Fix: start one (annotator/vllm.sh start) or set VLM_BASE_URL/VLM_MODEL to a running server.
• Error: model not found / 404 from the endpoint - Cause: VLM_MODEL is not the id the server actually serves. Fix: set VLM_MODEL to the served name (e.g. qwen3-vl-32b; check curl ${VLM_BASE_URL}/models).
• Error: image input unsupported / bad request from a text model - Cause: the endpoint is serving a text-only/code model such as qwen3-coder-next. Fix: use a vision model endpoint such as Qwen3-VL for annotation.
• Error: input HDF5 not found - Cause: HDF5_PATH unset or not a real file. Fix: pick an absolute path from the candidates the Run block prints.
• Error: filtered HDF5 missing - Cause: --filter was not passed. Fix: add --filter <path> to write the filtered dataset.

Final Response

Report env, input HDF5, annotations path, filtered HDF5 (if any), success/failure counts, VLM blockers.