Validation Lab — Dottie harness & training progress
Read-only scorecard for the Dottie orchestration stack: the live harness API and the distilled router model trained from run traces. Every number on this page is recomputed from committed event logs and reports — history, not live telemetry — and unmeasured quantities render as unmeasured.
Champion router model — orch-mlp-v1-v4
Promotion is eval-gated and fail-closed: a champion ships to routing only after the gate passes on measured data. Until then it serves in an advisory role; the badge carries the gate's exact reason.
Hill-climb ladder — 8 variants, real CPU training runs
Advantage-weighted training on the orchestration corpus (seed-pinned, ≤60s budget per variant; actual 1.5–7.6s). Bars share one hue — one measure across variants; hover any bar for its config.
Promotion gate — fail-closed
| Measured held-out coverage (n ≥ 10) | ✓ PASS — 61 of 10 required measured held-out records |
| Beats frequency-prior baseline on measured hold-out | ✓ PASS — champion 82.0% vs baseline 24.6% |
| Beats heuristic router on measured hold-out | ✕ FAIL — champion 82.0% vs baseline 83.6% |
Unmeasured renders as unmeasured, never as pass. The gate unlocks as real harness runs accumulate measured hold-out records.
Validation queue
| Method | Site | Status |
|---|---|---|
| Barlow Twins | research | awaiting pilot |
| InfoNCE domain tune | research | in execution |
Candidates advance pilot → charter → deploy; verdicts publish here with their evidence references.
Training corpus — 1613 records by provenance
Provenance is labeled per record. The measured slice grows with every harness run; simulated records come from a seeded, deterministic battery (seed 20260809).
Corpus by routing tier
| Tier | Records |
|---|---|
| agentic_epic | 490 |
| deep_research | 451 |
| action_operator | 309 |
| deterministic | 253 |
| llm | 110 |
Label sources
| Label source | Records |
|---|---|
| simulated | 820 |
| measured-behavior | 752 |
| measured-outcome | 41 |
Each record's label provenance is tracked separately from its feature provenance. The promotion gate's champion-vs-heuristic comparison only becomes meaningful when the measured hold-out contains labels the heuristic did not make (measured-outcome or operator-corrected).
Correction queue — operator review
| Run | Label | Source | Status | Correction command |
|---|---|---|---|---|
20260813T173943611486Z | action_operator | measured-behavior | ok | scout harness correct harness-run-20260813T173943611486Z <tier> --reason "…" |
20260813T173940773648Z | action_operator | measured-behavior | ok | scout harness correct harness-run-20260813T173940773648Z <tier> --reason "…" |
20260813T173936967593Z | agentic_epic | measured-outcome | fail | scout harness correct harness-run-20260813T173936967593Z <tier> --reason "…" |
20260813T173935751878Z | action_operator | measured-behavior | ok | scout harness correct harness-run-20260813T173935751878Z <tier> --reason "…" |
20260813T173934555198Z | agentic_epic | measured-outcome | fail | scout harness correct harness-run-20260813T173934555198Z <tier> --reason "…" |
20260813T173932399458Z | action_operator | measured-behavior | ok | scout harness correct harness-run-20260813T173932399458Z <tier> --reason "…" |
20260810T121725288196Z | agentic_epic | measured-outcome | fail | scout harness correct harness-run-20260810T121725288196Z <tier> --reason "…" |
20260810T121724195455Z | agentic_epic | measured-outcome | fail | scout harness correct harness-run-20260810T121724195455Z <tier> --reason "…" |
20260810T121723037562Z | agentic_epic | measured-outcome | fail | scout harness correct harness-run-20260810T121723037562Z <tier> --reason "…" |
20260810T121721866345Z | agentic_epic | measured-outcome | fail | scout harness correct harness-run-20260810T121721866345Z <tier> --reason "…" |
The ten most recent measured harness runs from the committed corpus.
A correction is ground truth: run the command with the tier that should have been
routed (goal text via scout harness checkpoint show --run-id …; the
corpus itself carries metadata only). Corrections land in
label_corrections.jsonl and the nightly retrain consumes them as
operator-corrected labels — the strongest signal against the gate's label ceiling.
Run scoreboard — mined from committed timelines
| Agent | Runs | Events | OK rate | p50 latency |
|---|---|---|---|---|
| builder | 122 | 122 | 100% | 0.02096000002893561 ms |
| critic | 122 | 122 | 100% | 0.10784499994542784 ms |
| executor | 124 | 124 | 100% | 0.04063200009341017 ms |
| mcp-operator | 25 | 25 | 68% | 120.91145499994127 ms |
| operator | 31 | 62 | 100% | 0.013480000006893533 ms |
| planner | 124 | 124 | 100% | 0.024464000034640776 ms |
| researcher | 3 | 3 | 100% | 30 ms |
| scout-lc-deep | 3 | 3 | 100% | 55 ms |
| scout-lc-langchain | 3 | 3 | 100% | 45 ms |
| scout-prime-coordinator | 3 | 3 | 100% | 0 ms |
| strategist | 158 | 158 | 100% | 0.02600900000970796 ms |
All runs to date succeeded — an OK rate of 100% is this period's data, not a claim; failure classes will appear here as they occur.
Venture artifacts — playbook-generated, provenance-checked
| Artifact | Path | Classification | Generated | Checksum |
|---|---|---|---|---|
| validation/dataset-card | workspace/artifacts/validation/dataset_card.md | REAL | 2026-08-10T02:28:03+00:00 | sha256:c45b2d9449f40044… |
| research/research-brief | workspace/artifacts/research/research_brief.md | REAL | 2026-08-10T02:28:02+00:00 | sha256:c0b675c7137d170e… |
| ops/changelog | workspace/artifacts/ops/changelog_entry.md | REAL | 2026-08-10T02:28:02+00:00 | sha256:b00597a9b069e96b… |
| ops/ops-digest | workspace/artifacts/ops/ops_digest.md | REAL | 2026-08-10T02:28:02+00:00 | sha256:2fefaf440f4f1f9f… |
| monitor/run-scoreboard | workspace/artifacts/monitor/scoreboard.json | REAL | 2026-08-10T02:28:01+00:00 | sha256:9c7c2e015dcdaa1f… |
| monitor/run-scoreboard | workspace/artifacts/monitor/scoreboard.md | REAL | 2026-08-10T02:28:01+00:00 | sha256:44fbaaf7bb6fc015… |
Generated by scripts/business/playbook.py run <venture> from
committed sources only; each row's checksum is copied verbatim from the venture's own
manifest, never recomputed here. Documents live under workspace/artifacts/
in the dottie repository.