# The RecordRoom spec format — v0.1 A **spec** is a versioned document of evals: the rules a working AI agent is held to in one domain of knowledge work. This file defines the format — what lives in the files, how they compile, how they run, and how they change. The format is the product's API: the site renders these files, the runner executes them, the product versions them. Anything true on a page is a field in a file. ## Design principles 1. **One sentence, one rule.** Every eval is a single expert-written sentence a practitioner would recognize, bound to how it is checked. The sentence and its checking definition travel together and version together — drift between them is the disease this format exists to cure. 2. **Judged by default, computed where genuine.** Evals are graded by an LLM judge reading declared evidence. A `computed` check is allowed only where ground truth is genuinely arithmetic (a sum, a threshold). 3. **Evidence or it didn't happen.** Every verdict cites spans from the evidence it was allowed to read. Every number rendered anywhere is computed from these files. 4. **Specs accumulate, like case law.** Reviewed verdicts get pinned to evals as examples — precedents under a statute. They are simultaneously the eval's calibration data, the judge's few-shot context, and its documentation. 5. **No gates on running.** An eval runs the moment it is written. Precision is earned in use, from expert agree/disagree, not from an upfront labeling ritual. 6. **The judge's internals are not format.** One-shot, two-step, votes — runner implementation details, recorded in the run, never in the spec. ## Directory layout ``` specs/// ├── SPEC.md # canonical, human-authored ├── corpus/ │ ├── tasks// # input materials (md/txt/pdf/csv) + task.json │ └── outputs/.json # trajectories └── dist/ ├── spec.json # compiled from SPEC.md (build artifact) └── runs/.json ``` Names are **org-scoped** from the first file: `recordroom/property-underwriting`. ## SPEC.md YAML frontmatter, then eval sections, then free prose. ```markdown --- name: recordroom/property-underwriting version: 0.1.0 description: One line. What this spec holds an agent to. maintainers: [recordroom] license: CC-BY-4.0 forked_from: null tasks: - {id: schedule, name: Building schedule} - {id: construction, name: Construction validation} --- ## Evals ### tiv-sum task: schedule severity: blocking **The TIV total equals the sum of the locations.** reads: schedule, cover sheet, final answer fail-when: the stated or reported total does not equal the recomputed sum of location TIVs beyond the tolerance. na-when: the submission has no location schedule. derived_from: [pibit-intake-2024] check: computed sum_equals # optional; omit for judged evals params: {tolerance: {default: 1, unit: USD}} examples: # pinned reviewed verdicts (grow over time) - {output: PU-0468-a, verdict: fail, note: "stated 48.2M vs 47.0M sum", reviewed_by: , on: 2026-08-01} ## About Free prose: intent, method, how to run it, where the evals come from (with citations), corpus provenance and synthesis declaration. ``` Eval fields: `id` (stable string; display order is presentation, identity is not), `task`, `severity` (blocking | major | minor), the **sentence** (bold, first line), `reads` (the judge's entire visible evidence — nothing outside it may influence a verdict), `fail-when`, `na-when`, `derived_from`, optional `check` + `params` for computed evals, `examples`. ## Corpus A **task** is input materials plus an instruction: `task.json` — `{id, title, instruction, materials: [files]}`. Materials are real documents; the runner's parser renders them into readable, citable evidence. An **output** is a trajectory — the full working session, not just the final answer: ```json {"id": "…", "task": "…", "agent": {"id": "…", "name": "…", "kind": "captured | generated", "model": "…"}, "turns": [ {"role": "user", "content": "the request or a follow-up"}, {"role": "agent", "tool": "query_appetite", "args": {"naics": "445110"}, "result": "retail: in appetite to $25M/location"}, {"role": "agent", "content": "a question back, an action, or the answer"} ], "final": "the last agent message — the deliverable"} ``` Turns are ordered. A turn is either a message (`role` + `content`) or a tool call (`role: "agent"` + `tool` + `args` + `result`). Multi-turn is expected: user follow-ups, agent questions, tool calls interleaved. Tasks may declare `tools` (name + description + what it returns) so trajectories' tool calls are grounded in a real interface. Process evals read the turns; content evals read the materials, the turns, and `final`. Corpus hygiene (hard rules): synthesized data is declared as such; expected labels and case names never appear in judge evidence; the agent under test never sees the spec at evaluation time. ## Runs `run.json` is append-only and self-contained: ```json { "kind": "run", "spec": {"name": "recordroom/property-underwriting", "version": "0.1.0", "hash": "sha256:…"}, "corpus": {"hash": "sha256:…"}, "params": {"tolerance": 1}, "instruments": { "judge": {"model": "…", "strategy": "one-shot", "votes": 1}, "parser": {"version": "…"} }, "created_at": "…", "results": [ {"task": "PU-0412", "output": "PU-0412-a", "eval": "construction-class-evidence", "verdict": "fail", "na_reason": null, "why": "…", "citations": [{"from": "output", "span": "…"}, {"from": "inspection", "span": "…"}]} ], "totals": {"pass": 0, "fail": 0, "na": 0} } ``` `verdict`: pass | fail | na. `na_reason`: not_applicable | insufficient_evidence | not_judgeable. Hashes are sha256 over canonical JSON. Results are never rewritten; a re-run is a new run. ## Verdict reviews and examples The one review primitive: an expert **agrees or disagrees** with a verdict, with an optional note. Reviews accumulate into (a) per-eval precision (agreement rate), (b) clusters that surface evals needing attention, and (c) pinned `examples` when a reviewed verdict is worth keeping — which feeds the judge's few-shot context on future runs. Iteration on an eval is always deliberate and user-initiated; nothing auto-edits. ## Versioning Semver on the spec. Changing an eval's sentence, fail/NA conditions, or check is **major for that eval** and requires showing the semantic diff (which verdicts flip across existing runs) before adoption. Adding an eval or pinning examples is minor. Prose-only edits are patch. Every change carries a reason. `forked_from` records lineage. ## What renders where Spec page = frontmatter line + eval sections + About. Cases = corpus rendered as conversations. Results = the latest run. A UI element that cannot name its field in these files does not ship.