RunPod: Add reproducible pod setup, storage, and checkpoint-resume playbook

[georgepullen]

Purpose

Make RunPod execution reproducible, interruption-safe, and auditable for this repo.

Mandatory Reading (blocking)

First comment must summarize:

• reports/NL_IMPLEMENTATION_ORACLE.md section 6.3.3 and 6.3.4
• docs/FSDP_SCALING_GUIDE.md
• docs/release_checklist.md
• docs/env_matrix.md

Also include links reviewed from RunPod docs in the first comment.

Required Code Anchors

• scripts/compute/
• training entrypoints (train.py, train_fsdp.py, train_deepspeed.py)
• docs under docs/

Scope

• Add concrete RunPod playbook:
  • pod create settings
  • persistent storage conventions
  • SSH + transfer commands
  • checkpoint frequency guidance for spot/on-demand
  • forced stop/resume drill

Deliverables

• docs/runpod_execution.md
• helper scripts for setup/sync/checkpoint validation
• runbook used by Phase 5: Run RunPod pilot ablation matrix (5k/25k) for paper-fidelity paths #11 and Phase 5: Execute mid/target faithful runs and publish final alignment report #12

Acceptance Criteria

• Fresh pod can run smoke end-to-end using docs only.
• Resume drill validated with evidence.
• First issue comment contains mandatory reading summary.

[georgepullen]

Read summary before implementation:

• Oracle §6.3.3 confirms repo has scale configs and FSDP guidance but no fully reproducible large-scale runbook for interruption-safe execution.
• Oracle §6.3.4 flags hyperparameter uncertainty and therefore requires strict provenance capture (exact command, config, checkpoint hash) for every run.
• Existing checkpoint sidecar machinery (.meta.json, .sha256, .yaml) is sufficient to anchor resume audits if integrated into the runpod workflow.
• Current train.py path is single-process oriented; robust resume today is strongest in train_fsdp.py (checkpoint.resume_path) and DeepSpeed tag resume.

RunPod docs reviewed (Context7):

• Pod creation API and pod settings: https://github.com/runpod/docs/blob/main/pods/manage-pods.mdx
• Network volumes (persistent across pod lifecycle): https://github.com/runpod/docs/blob/main/pods/storage/create-network-volumes.mdx
• Storage type behavior: https://github.com/runpod/docs/blob/main/pods/storage/types.mdx
• SSH/SCP usage: https://github.com/runpod/docs/blob/main/pods/configuration/use-ssh.mdx
• Pod lifecycle (stop/start/restart/reset): https://context7.com/runpod/docs/llms.txt

PyTorch docs reviewed (Context7):

• AMP/autocast guidance: https://github.com/pytorch/pytorch/blob/main/docs/source/notes/amp_examples.rst
• Checkpoint save/load pattern: https://context7.com/pytorch/pytorch/llms.txt
• FSDP state-dict notes: https://github.com/pytorch/pytorch/blob/main/docs/source/distributed.fsdp.fully_shard.md

Implementation scope for this ticket:

• Add docs/runpod_execution.md with reproducible pod/storage/ssh/resume drill.
• Add helper scripts for sync/setup/checkpoint+telemetry validation.
• Add explicit quality gates used by tickets Phase 5: Run RunPod pilot ablation matrix (5k/25k) for paper-fidelity paths #11 and Phase 5: Execute mid/target faithful runs and publish final alignment report #12.

[georgepullen]

Progress update:

Implemented RunPod reproducibility playbook + helper scripts, with Context7 doc references embedded in the runbook:

• New runbook: docs/runpod_execution.md
  • pod/storage/ssh contract
  • sync workflow
  • checkpoint cadence recommendations
  • forced stop/resume drill and quality gates
  • RunPod + PyTorch source links
• Helper scripts:
  • scripts/compute/runpod_bootstrap.sh
  • scripts/compute/runpod_sync.sh
  • scripts/compute/runpod_validate.sh
  • scripts/compute/runpod_resume_drill.sh
• README and release checklist updated to reference RunPod execution and resume drill.

Blocking technical gap addressed while implementing this:

• train.py path now supports checkpoint resume via train.checkpoint.resume_path in shared loop (src/nested_learning/training.py), enabling practical interruption-safe resume drills.

Remaining for full closure:

• Run an end-to-end fresh pod execution and attach evidence artifacts (log excerpts + generated checkpoints/manifests) from this exact playbook.

[georgepullen]

RunPod playbook regression update:

• Re-ran forced stop/resume drill after telemetry + optimizer-path changes:
  • DEVICE=cpu bash scripts/compute/runpod_resume_drill.sh (pass)
• Drill validations passed end-to-end:
  • checkpoint integrity check pass on resume + final checkpoints
  • fidelity telemetry validator pass on both phase logs

This confirms the runbook scripts remain valid after the recent model/telemetry changes.