Evaluator Hand-off

Each verified publication includes artifacts/tasks/registry.json, a native Harbor registry for terminal-curator@1.0. Evaluator can consume it through its existing harbor_registry provider. The checked-in SWE benchmark remains the default until you choose this registry in an evaluation run profile.

Configure one terminal evaluation

In a full Evaluator config or private EVAL_CONFIG profile, set these fields under runtime_info.input:

task_source:
  provider: harbor_registry
  dataset_name: terminal-curator
  version: "1.0"
  registry_path: ../terminal-curator/artifacts/tasks/registry.json
  no_hack: false

harbor_job:
  n_tasks: 2
  n_concurrent: 2
  max_retries: 0

job_analysis:
  enabled: false

This snippet updates the relevant fields; retain the other required settings from Evaluator's full config. Set its endpoint, model, and supported agent runtime for the model being evaluated. Supply secrets through environment variables. n_tasks: 2 is a small initial run; use null to evaluate the whole published batch.

Keep no_hack: false: that setting is a SWE-bench Verified dataset remap, and Evaluator rejects it for terminal-curator. SWE gold-dataset analysis expects metadata that these terminal verifiers do not provide, so leave job_analysis.enabled: false. Harbor still records per-task rewards, exceptions, and agent trajectories for the Evaluator dashboard.

Run /evaluator:check or $evaluator-check, then the authorized /evaluator:run or $evaluator-run. The registry and its referenced task paths must be readable on the configured Docker host. This workflow uses the existing Evaluator agent and LiteLLM pipeline; it does not launch another curation batch.

What the registry contains

The block writes the registry inside the new immutable publication before switching the artifacts/tasks symlink. Its shape is:

[
  {
    "name": "terminal-curator",
    "version": "1.0",
    "description": "Terminal tasks verified with Harbor Oracle=1 and NOP=0.",
    "tasks": [
      {
        "name": "terminal-example",
        "path": "/absolute/block/path/artifacts/tasks-versions/batch-id/terminal-example"
      }
    ]
  }
]

The example path illustrates the format. Actual paths are generated for the execution host. Each task entry names a local task, with no Git URL, and only IDs in the verified manifest are included. Later publications preserve the task paths already read by an evaluation job. Retain old published versions while those jobs or their reproduction records refer to them.

The conditional block edge is terminal-curator.output.eval_registry → evaluator.input.task_source.registry_path, active when Evaluator selects dataset_name: terminal-curator. Tracer's producer selection is independent. Inspect the actual evaluation result.json and per-trial results before reporting a scored or solved evaluation; Oracle/NOP curation evidence is recorded separately.