Proofs are the unit of account

The seventeen proof kinds

Proof is a closed union at types.ts:12–29. It is not extensible at runtime; adding a proof means editing the type, which forces every switch and manifest to be re-checked by the compiler. The names fall into four rough groups, though the source itself imposes no grouping.

Four of the gameplay proofs exist because of that sentence. The proof vocabulary is deliberately weighted toward things a program can observe from outside the artifact — pixels, exit codes, HTTP responses, frame timings — rather than toward things an agent can assert about its own work.

The registry in full

Each manifest is a plain object literal. Six fields are required by EvaluatorManifest (types.ts:55–70); the rest — publicTests, hiddenTests, mutationTests, artifacts, antiCheatChecks, scoreWeight, command — are optional annotations.

What the manifests do carry, and carry seriously, is anti-cheat intent. Every non-trivial evaluator declares mutation tests that must fail on a deliberately broken artifact. The browser-game manifest is the most explicit:

41    mutationTests: [
42      "disabled movement must fail",
43      "disabled collision must fail",
44      "fake telemetry without pixel evidence must fail"
45    ],
46    artifacts: ["screenshots", "video", "frame stats", "gameplay transcript"],
47    antiCheatChecks: [
48      "cross-check telemetry against canvas pixels",
49      "store random seed",
50      "reject animated noise as gameplay"
51    ],
52    scoreWeight: 0.35

“Reject animated noise as gameplay” is the kind of rule that only gets written by someone who has watched an agent produce a canvas full of moving pixels and call it Doom. These strings are instructions to a human or agent writing the evaluator, not assertions the registry can enforce.

Selection is two array filters

112export function selectEvaluators(domain: Domain, requiredProofs: Proof[]): EvalSelection {
113  const candidates = evaluatorRegistry.filter((e) =>
113      e.domains.includes(domain) || e.domains.includes("unknown"));
114  const selected = candidates.filter((e) =>
114      e.proves.some((proof) => requiredProofs.includes(proof)));
115  const covered = new Set(selected.flatMap((e) => e.proves));
116  const uncoveredProofs = requiredProofs.filter((proof) => !covered.has(proof));
117
118  return { domain, requiredProofs, evaluators: selected, uncoveredProofs };
124}

Line 115 also has a quiet consequence. covered is built from the full proves array of every selected evaluator, not from the intersection with requiredProofs. Selecting api-contract for its api_contract proof therefore also marks security_baseline covered, whether or not the goal asked for it. Coverage is transitive through selection.

Selection outcome for every domain’s default proof set

Eight of nine domains are fully covered by construction, which is unsurprising: the default proof lists and the evaluator manifests were written together. infra is the exception and the only place the missing-evaluator path fires without a hand-edited contract.

Asking for the gap directly

The missing-evals command compiles a goal and returns a proposal for each proof no selected evaluator claims. Because infra is the one domain with a real gap, it is the natural demonstration.

$ daedalus missing-evals "deploy a kubernetes cluster with terraform"
# classifyDomain -> "infra"   (keywords: kubernetes, terraform)
# defaultProofsForDomain("infra") -> builds, tests_pass, security_baseline
# selectEvaluators -> [shell-core]   uncoveredProofs -> [security_baseline]

[
  {
    "id": "eval_proposal_IkuDsIk1",
    "proof": "security_baseline",
    "domain": "infra",
    "status": "proposal",
    "purpose": "Create a reproducible evaluator for security_baseline.",
    "suggestedFiles": [
      "evals/custom/evaluator.test.ts",
      "evals/custom/fixtures/"
    ],
    "validationPlan": [
      "define observable evidence for the proof",
      "write a passing check for a correct artifact",
      "mutate or mock a broken artifact",
      "confirm the evaluator fails on the broken artifact"
    ],
    "antiCheatChecks": [
      "avoid checking only static text",
      "prefer artifact behavior over agent trace claims",
      "add hidden randomized cases where possible"
    ]
  }
]

Field values transcribed from the fallback branch at evaluatorRegistry.ts:176–191; the identifier suffix is nanoid(8) and varies per invocation.

Note what the fallback branch does not know. It has no idea that a security baseline for Terraform means Checkov or tfsec, or that RESEARCH.md:241 already lists Semgrep, gitleaks, OSV, Trivy, npm audit and pip-audit as the intended security tooling. The proposal is domain-blind boilerplate outside the two hand-written branches.

Evaluator packs

Six JSON documents sit under evaluator-packs/, one per pack. They restate the domain-to-evaluator mapping and add a list of recommended file paths. No TypeScript source imports them, and package.json has no YAML or glob dependency, so they function as documentation for whoever writes the evaluators by hand.

The game pack carries one field the others do not: a test_api_guidance object naming window.__DAEDALUS_GAME_TEST__ as the sanctioned telemetry hook, with the rule “Test telemetry is allowed, but it must be cross-checked against screenshots/canvas/video evidence.” That is the same distrust of self-reported state that runs through the whole design — a game can be instrumented to say it is working, so the instrument is only admissible alongside pixels.

There is no infra pack and no security pack, which matches the one red cell in Fig. 2.2 and the outstanding item at RESEARCH.md:319.