Phase 0: Establish canonical paper-faithful baselines and lock reference metrics

[georgepullen]

Purpose

Create a reproducible paper-faithful baseline package that all downstream tickets must compare against.

Mandatory Reading (blocking)

Before writing code, the agent must read and summarize in a first issue comment:

• reports/NL_IMPLEMENTATION_ORACLE.md (sections: 3, 4.1-4.7, 5, 6.1-6.4)
• reports/paper/NL-print.extracted.clean.txt (Eq. 21-24, 28-31 and Table 1 text)
• docs/PAPER_COMPLIANCE.md

The first comment must include:

1. 5-10 bullet summary of the current implementation state.
2. Exact list of code paths used for baseline runs.
3. Confirmed gaps that this ticket does not change.

Required Code Anchors

• src/nested_learning/training.py
• src/nested_learning/model.py
• configs/pilot_paper_faithful.yaml
• configs/pilot_selfmod_paper_faithful.yaml
• scripts/eval/zeroshot.py
• scripts/eval/niah.py
• scripts/eval/continual.py
• scripts/eval/passkey.py
• scripts/eval/pg19_perplexity.py

Scope

• Define canonical baseline run commands for:
  • hope_selfmod paper-faithful
  • hope_attention comparison
• Produce frozen baseline artifact bundle with checksums + evals.
• Add reports/ baseline table (short + explicit command provenance).

Runbook

• uv run python train.py --config-name pilot_selfmod_paper_faithful train.steps=2000
• uv run python train.py --config-name pilot_paper_faithful model.block_variant=hope_attention train.steps=2000
• Run full eval suite for both checkpoints.

Deliverables

• Baseline artifact bundle under artifacts/.
• Eval JSON set under eval/ for both baselines.
• Report file documenting exact commands + hashes + metrics.

Acceptance Criteria

• No NaN/Inf in training logs.
• All eval outputs exist for both baselines.
• scripts/checkpoint/verify.py passes on all published checkpoints.
• First issue comment includes mandatory reading summary.

[georgepullen]

Read summary (oracle + paper + compliance) before implementation:

• reports/NL_IMPLEMENTATION_ORACLE.md confirms strongest fidelity exists around Eq. 30/31 scheduling + CMS chaining, while largest optimizer divergence is Eq. 17-24 in src/nested_learning/optim/deep.py.
• Current train loop already emits loss, ppl, teach_signal_norm and per-layer update telemetry via HOPEModel.pop_update_metrics().
• Paper-faithful configs (pilot_paper_faithful, pilot_selfmod_paper_faithful) enforce key mechanism constraints: batch_size=1, use_fast_state=true, fail_if_paper_faithful_disabled=true, and cms_flush_partial_at_end=true.
• Eq. 31 frequency semantics are implemented with LevelSpec.update_period + chunk buffering and are already test-covered.
• Eq. 28-29 projection term is still a known gap; this ticket will not change optimizer mathematics.
• DDP/FSDP caveats in compliance docs confirm strict paper-faithful online+per-layer path is currently single-process (train.py) oriented.
• Evaluation surface required for baselines is present: zeroshot, NIAH, continual, passkey, PG-19 scripts.
• Checkpoint integrity sidecars/hashes are already implemented (write_checkpoint_metadata, scripts/checkpoint/verify.py) and should be reused in baseline packaging.

Code paths for Phase-0 baseline runs (canonical):

• train.py
• src/nested_learning/training.py
• src/nested_learning/model.py
• configs/pilot_paper_faithful.yaml
• configs/pilot_selfmod_paper_faithful.yaml
• scripts/eval/zeroshot.py
• scripts/eval/niah.py
• scripts/eval/continual.py
• scripts/eval/passkey.py
• scripts/eval/pg19_perplexity.py
• scripts/checkpoint/verify.py

Confirmed gaps NOT changed in this ticket:

• No implementation of Eq. 21-24/28-29 optimizer math changes.
• No backprop-through-online-writes boundary-state training.
• No DDP/FSDP parity rework for paper-faithful online/per-layer semantics.

[georgepullen]

Progress update:

Implemented canonical Phase-0 baseline execution + packaging scaffolding:

• Added canonical attention paper-faithful config: configs/pilot_attention_paper_faithful.yaml
• Added baseline orchestration script: scripts/compute/run_phase0_baselines.sh
• Added frozen bundle packager + provenance report generator: scripts/checkpoint/package_phase0_baselines.py
• Added baseline runbook doc: reports/phase0_baseline_plan.md
• README updated with canonical baseline entrypoint.
• Added config composition coverage for new attention config: tests/test_paper_faithful_configs.py

This now gives explicit command provenance, expected eval outputs, checkpoint verification, telemetry validation, and artifact checksums/manifest packaging.

Remaining to fully close this ticket:

• Execute the full 2k-step dual baseline runs on target hardware/data.
• Produce concrete artifacts/phase0_baselines/phase0_*/manifest.json and reports/phase0_baseline_table.md from real run outputs.