# Company evidence: a synthetic pilot ## Question and limits Does explicitly maintaining claim dependencies improve decisions when an inherited AI interpretation is copied into company records, compared with competent notes and structured beliefs? This is a synthetic memory-and-provenance task. It does not run Sho, establish how frequently this failure occurs in actual companies, or test how the inherited interpretation was originally produced. The initial AI note is deliberately supplied. All names and records are fictional. The cases are deliberately legible: copied records disclose their source, independent checks disclose separate verification, and an identified owner explicitly confirms or corrects the proposition. A tie can mean this test is too easy; it does not establish equivalence on longer or noisier work. ## Design fixed before held-out execution - Two development families: regional storage and training completion. Four held-out families: release review, support coverage, sensor calibration, backup deletion. - Each family crosses initially true/false with copied/independent later records. Three systems and three repetitions give 144 held-out episodes (four families, not 144 independent scenarios). Development has 24 episodes and is reported separately. - Three isolated model calls per episode. Call 1 sees the inherited interpretation plus two updates and makes a decision while retaining memory. Call 2 receives an authoritative correction or confirmation plus that memory. Call 3 receives a later digest copied from the old interpretation plus the revised memory and decides what a different team's weekly handoff may state. Earlier raw records are not repeated. Future records and expected answers never enter prompts. - The notes baseline may preserve IDs, source relationships, uncertainty and correction history in any form. All arms retain both the proposition and the action-eligibility conclusion that depends on it. Structured arms represent the latter with a claim dependency instead of duplicating raw sources. Maintained dependencies add a deterministic helper that marks directly affected and dependent claims for review when a source is explicitly superseded. The helper supplies no hidden truth and makes no decision. A separately verified scheduling fact is also retained by every arm; its source is never superseded. We score its survival and report actual direct and transitive helper marks. - Source IDs must remain verbatim. Both structured arms constrain source and superseded-source arrays to `R` plus digits; descriptions belong in separate text. Claim keys must be unique and every claim dependency must resolve within the retained state. The notes baseline receives the same verbatim-ID guidance and may choose its own representation. - Every system sees identical records, authorship, timestamps and provenance. The common decision policy requires two independent direct checks or a current authoritative determination. Copies are not direct checks. This stipulated policy makes scoring deterministic; it is not a claim about the right decision threshold for real companies. - Same model (`gpt-6-astra`), medium reasoning, three calls, maximum 5,000 UTF-8 bytes of persistent state, 600-character rationale. Structured state is limited to eight claims. Notes may use any internal structure within the same byte limit. - Existing authenticated Codex CLI and its configured priority service tier; no tools or external data. A tool call makes the episode invalid. No paid API account or new credentials. Runtime includes Codex's fixed instruction/context overhead and may report cached input tokens. No dollar-cost comparison is inferred from subscription-token accounting. - Maximum concurrency four. Default per-call timeout 180 seconds. No automatic retries or selecting among answers. Invalid episodes remain visible and count separately; subsequent stages stop when no valid state exists. ## Scoring Scores are fixed functions in `run.py` and expected answers are included in `fixtures.json`. At call 1, copied records require hold and zero independent support; two independent checks require proceed and support count two. The audit at call 2 and the retained correction at call 3 require proceed/one supporting verification if true, or do-not-proceed/zero if false. Report initial decision errors; decisions inconsistent with the authoritative revision; old false claims resurfacing as proceed; all later-task decision errors; copies counted as direct support; missed/miscounted independent support; unnecessary hold decisions when evidence settles the task; loss of the unaffected scheduling fact; invalid episodes; calls, input/cached/output/reasoning tokens, wall-clock call latency, state bytes, and deterministic helper time. Missing token fields are reported as null, not zero. Mean state size uses only valid produced states. Behavioral score denominators are valid episodes, with invalid and scheduled counts separately visible; do not portray invalid runs as successful decisions. Decision errors and support errors are distinct. Correct prose with a wrong decision field does not pass the decision score. A support count alone does not establish that a model internally treated sources as independent. We do not use a model judge. Zero observed errors are reported as counts, not proof of reliability. ## Reproduction and interpretation Requires Python 3 and an authenticated Codex CLI that supports `exec --ignore-user-config --ephemeral --output-schema`. The original run uses Codex 0.153.1. Configure `--cli` to the local executable if needed. Do not provide private company records or credentials to the harness. Run `python3 run.py prepare --output my-results`, then `python3 run.py development --output my-results`, then `python3 run.py heldout --output my-results`. Preparation writes fixtures and schemas and freezes their SHA-256 hashes together with this protocol and the runner. Frozen files must match before runs proceed. Development diagnoses feasibility; no held-out tuning or selective case removal is permitted. Use a fresh output directory for a new experiment version. Every actual model request and final response is saved under `results/runs/`. Usage, timings, errors, memory and deterministic scores are in each episode record. `summary.json` aggregates all completed episodes; invalid episodes are retained. The three arms differ in persistence format and, for the third, deterministic maintenance, so any difference cannot be attributed to an abstract concept of belief alone. The fixed serving model name may route to a provider revision over time; identical outputs are not guaranteed on rerun. Repetitions do not replace more distinct scenario families. ## Infrastructure preflight The first development launch returned a usage-limit error on all 24 first-call attempts before producing model output. Restoring the existing priority-tier setting alone did not fix a subsequent canary. No held-out case ran. Those requests, errors and original frozen protocol/source are preserved under `preflight/`; they are infrastructure failures, not evidence about any arm. Full-request probes succeeded with the CLI inheriting the environment unchanged and its parent working directory set to the user home (the CLI itself receives an empty temporary workspace). The final runner uses that demonstrated launch context and retains the configured priority tier. No account change, usage reset, credential extraction or purchase was used. ## Development contract correction (September 6, 2026) A completed development batch had 23 valid episodes and one infrastructure failure. Its behavioral decisions passed, but inspection found source fields such as `R1: Operations analyst; ...`. These human-readable references did not match the helper's exact `R1` key, so some dependency invalidations were silently missed. Before any held-out execution, both structured arms received an exact source-ID contract and validation; common verbatim-ID guidance was added for all arms. Unique claim keys and resolved dependencies were also enforced. Fixtures, expected decisions and scoring were unchanged. The original frozen code, all outputs and a concrete missed-link example are retained under `preflight/development-before-source-id-contract/`. This is a development interface defect, not an AI corroboration failure or a held-out finding.