Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/1026.added
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added typed Stage 5 promotion result models around the existing release transaction engine.
16 changes: 16 additions & 0 deletions docs/engineering/pipeline-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -882,6 +882,14 @@ def fitted_weights_spec_for_scope(scope: FitScope | str) -> FittedWeightsSpec

Return the current fitted-weight spec for a regional or national scope.

### `policyengine_us_data.release_promotion.results.full.FullPromotionResult`

```python
class FullPromotionResult
```

Typed result for a full Stage 5 release promotion transaction.

### `policyengine_us_data.datasets.cps.extended_cps.ExtendedCPS._validate_housing_assistance_microsimulation`

```python
Expand Down Expand Up @@ -1450,6 +1458,14 @@ def stage_1_step_specs() -> tuple[DatasetBuildStepSpec, ...]

Return the canonical Stage 1 dataset-build substage specs.

### `policyengine_us_data.utils.release_promotion.promote_full_release_with_result`

```python
def promote_full_release_with_result(config: FullReleasePromotionConfig, deps: FullReleasePromotionDependencies) -> 'FullPromotionResult'
```

Run the existing transaction engine and wrap its output in a typed result.

### `policyengine_us_data.calibration.unified_matrix_builder.UnifiedMatrixBuilder`

```python
Expand Down
4 changes: 4 additions & 0 deletions docs/engineering/skills/documentation_review.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ Check that changed pipeline behavior has a durable documentation surface:
- Generated docs build when decorator, Pydoc, or map source changes. PRs that
change decorator metadata, Pydoc-facing source, or `docs/pipeline_map.yaml`
should refresh the checked-in generated artifacts in the same change.
- Pipeline documentation segment edits require the same treatment: if a PR
changes source text that feeds generated pipeline docs or AI-facing pipeline
guidance, verify whether `scripts/extract_pipeline_docs.py` output changes and
commit the refreshed generated artifacts when it does.
- Stale architecture names, folder names, and artifact names are not preserved in
durable documentation sources or generated output.

Expand Down
9 changes: 8 additions & 1 deletion docs/engineering/skills/pipeline_docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,14 @@ The generated JSON and Markdown files are published artifacts, not hand-authored
source. PRs should update decorators, docstrings, and `docs/pipeline_map.yaml`,
then regenerate the checked-in artifacts in the same change so reviewers see the
pipeline docs that will ship. On pushes to `main`, automation may refresh those
artifacts again with the version/changelog commit.
artifacts again with the version/changelog commit, but PR authors and AI agents
must not rely on that later automation for review correctness.

Any time a PR touches a pipeline documentation segment, a `@pipeline_node`
decorator, Pydoc-facing text that feeds the extractor, or
`docs/pipeline_map.yaml`, regenerate and commit the checked-in docs produced by
`scripts/extract_pipeline_docs.py`. Treat the generated docs as part of the same
change, even if the source edit is small.

## Annotation Rules

Expand Down
18 changes: 18 additions & 0 deletions docs/engineering/stages/release_promotion.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,3 +103,21 @@ perform Hugging Face writes, GCS uploads, Modal calls, staging cleanup, or
release-manifest publication. Keep those operations behind explicit adapters or
services so tests can exercise candidate shape and validation logic without
credentials or network access.

Use `FullPromotionResult` and its substep result objects when exposing Stage 5
promotion outcomes to contracts, status APIs, or orchestration summaries. The
current compatibility wrapper, `promote_full_release_with_result()`, must keep
calling the existing transaction engine first and only wrap its dictionary
output afterward so the promotion order remains unchanged.

Result objects should carry semantic public-output identity, not only counts.
Keep Hugging Face repo/type, staging prefix, promoted/no-op paths, and commit ID
when available on `HuggingFacePromotionResult`; GCS bucket, release version,
object paths, skipped paths, and failures on `GcsPromotionResult`; release
manifest and TRACE TRO paths on `ReleaseManifestPromotionResult`; version
manifest path/version/current version on `VersionManifestPromotionResult`;
completion marker path/tag/validity on `CompletionMarkerPromotionResult`; and
cleanup `status` as `skipped`, `completed`, or `failed` on
`CleanupPromotionResult`. Later contract, index, diagnostics, and status
writers should read this typed material instead of scraping logs or
reconstructing public paths independently.
68 changes: 68 additions & 0 deletions docs/generated/pipeline_api.json
Original file line number Diff line number Diff line change
Expand Up @@ -1228,6 +1228,40 @@
"signature": "def fitted_weights_spec_for_scope(scope: FitScope | str) -> FittedWeightsSpec",
"source_file": "policyengine_us_data/fit_weights/specs.py"
},
"full_promotion_result": {
"docstring": "Typed result for a full Stage 5 release promotion transaction.",
"id": "full_promotion_result",
"kind": "class",
"line": 62,
"metadata": {
"api_refs": [
"policyengine_us_data.release_promotion.results.FullPromotionResult",
"policyengine_us_data.release_promotion.results.full.FullPromotionResult"
],
"artifacts_in": [
"release promotion transaction output"
],
"artifacts_out": [
"typed promotion result"
],
"description": "Typed Stage 5 result model for full release promotion outcomes.",
"id": "full_promotion_result",
"label": "FullPromotionResult",
"node_type": "library",
"pathways": [
"5_validate_and_promote_release"
],
"source_file": "policyengine_us_data/release_promotion/results/full.py",
"stability": "moving",
"status": "transitional",
"validation_commands": [
"uv run pytest tests/unit/release_promotion/test_results.py"
]
},
"object_path": "policyengine_us_data.release_promotion.results.full.FullPromotionResult",
"signature": "class FullPromotionResult",
"source_file": "policyengine_us_data/release_promotion/results/full.py"
},
"geo_assign": {
"docstring": "Assign random census block geography to cloned\nCPS records.\n\nEach of n_records * n_clones total records gets a\nrandom census block sampled from the global\npopulation-weighted distribution. State and CD are\nderived from the block GEOID.\n\nArgs:\n n_records: Number of households in the base CPS\n dataset.\n n_clones: Number of clones (default 10).\n seed: Random seed for reproducibility.\n fixed_state_fips: Optional state FIPS per base record. Positive\n values constrain every clone of that record to blocks in the\n requested state; zero or missing values remain unrestricted.\n\nReturns:\n GeographyAssignment with arrays of length\n n_records * n_clones.",
"id": "geo_assign",
Expand Down Expand Up @@ -3889,6 +3923,40 @@
"signature": "def validate_area(sim, targets_df: pd.DataFrame, engine, area_type: str, area_id: str, display_id: str, dataset_path: str, period: int, training_mask: np.ndarray, variable_entity_map: dict, constraints_map: Optional[dict] = None) -> list",
"source_file": "policyengine_us_data/calibration/validate_staging.py"
},
"typed_full_release_promotion": {
"docstring": "Run the existing transaction engine and wrap its output in a typed result.",
"id": "typed_full_release_promotion",
"kind": "function",
"line": 171,
"metadata": {
"api_refs": [
"policyengine_us_data.utils.release_promotion.promote_full_release_with_result"
],
"artifacts_in": [
"staged release artifacts",
"release manifest inputs"
],
"artifacts_out": [
"FullPromotionResult"
],
"description": "Compatibility wrapper that returns typed Stage 5 promotion results from the existing transaction engine.",
"id": "typed_full_release_promotion",
"label": "Typed Full Release Promotion",
"node_type": "library",
"pathways": [
"5_validate_and_promote_release"
],
"source_file": "policyengine_us_data/utils/release_promotion.py",
"stability": "moving",
"status": "transitional",
"validation_commands": [
"uv run pytest tests/unit/release_promotion/test_results.py"
]
},
"object_path": "policyengine_us_data.utils.release_promotion.promote_full_release_with_result",
"signature": "def promote_full_release_with_result(config: FullReleasePromotionConfig, deps: FullReleasePromotionDependencies) -> 'FullPromotionResult'",
"source_file": "policyengine_us_data/utils/release_promotion.py"
},
"unified_matrix_builder": {
"docstring": "Build sparse calibration matrix for cloned CPS records.\n\nProcesses clone-by-clone: each clone's records get their\nassigned geography, are simulated, and the results fill\nthe corresponding columns.\n\nArgs:\n db_uri: SQLAlchemy database URI.\n time_period: Tax year for calibration (e.g. 2024).\n dataset_path: Path to the base extended CPS h5 file.",
"id": "unified_matrix_builder",
Expand Down
54 changes: 52 additions & 2 deletions docs/generated/pipeline_map.json
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,31 @@
"uv run pytest tests/unit/fit_weights/test_specs.py"
]
},
{
"api_refs": [
"policyengine_us_data.release_promotion.results.FullPromotionResult",
"policyengine_us_data.release_promotion.results.full.FullPromotionResult"
],
"artifacts_in": [
"release promotion transaction output"
],
"artifacts_out": [
"typed promotion result"
],
"description": "Typed Stage 5 result model for full release promotion outcomes.",
"id": "full_promotion_result",
"label": "FullPromotionResult",
"node_type": "library",
"pathways": [
"5_validate_and_promote_release"
],
"source_file": "policyengine_us_data/release_promotion/results/full.py",
"stability": "moving",
"status": "transitional",
"validation_commands": [
"uv run pytest tests/unit/release_promotion/test_results.py"
]
},
{
"api_refs": [
"policyengine_us_data.datasets.cps.extended_cps.ExtendedCPS._validate_housing_assistance_microsimulation"
Expand Down Expand Up @@ -1741,6 +1766,31 @@
"uv run pytest tests/unit/test_build_dataset_specs.py"
]
},
{
"api_refs": [
"policyengine_us_data.utils.release_promotion.promote_full_release_with_result"
],
"artifacts_in": [
"staged release artifacts",
"release manifest inputs"
],
"artifacts_out": [
"FullPromotionResult"
],
"description": "Compatibility wrapper that returns typed Stage 5 promotion results from the existing transaction engine.",
"id": "typed_full_release_promotion",
"label": "Typed Full Release Promotion",
"node_type": "library",
"pathways": [
"5_validate_and_promote_release"
],
"source_file": "policyengine_us_data/utils/release_promotion.py",
"stability": "moving",
"status": "transitional",
"validation_commands": [
"uv run pytest tests/unit/release_promotion/test_results.py"
]
},
{
"api_refs": [
"policyengine_us_data.calibration.unified_matrix_builder.UnifiedMatrixBuilder"
Expand Down Expand Up @@ -1966,9 +2016,9 @@
}
],
"metadata": {
"api_node_count": 95,
"api_node_count": 97,
"canonical_stage_count": 5,
"decorated_object_count": 144,
"decorated_object_count": 146,
"mapped_decorated_node_count": 49,
"stage_count": 17,
"substage_count": 17
Expand Down
16 changes: 16 additions & 0 deletions policyengine_us_data/release_promotion/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,15 @@
build_release_candidate_bundle_from_stage4_contract,
read_stage4_release_candidate_bundle,
)
from .results import (
CleanupPromotionResult,
CompletionMarkerPromotionResult,
FullPromotionResult,
GcsPromotionResult,
HuggingFacePromotionResult,
ReleaseManifestPromotionResult,
VersionManifestPromotionResult,
)
from .validation import build_release_candidate_shape_report
from .validation import (
DEFAULT_REQUIRED_RELEASE_ARTIFACT_FAMILIES,
Expand All @@ -39,11 +48,18 @@
"BASE_RELEASE_ARTIFACT_PATHS",
"DEFAULT_REQUIRED_RELEASE_ARTIFACT_FAMILIES",
"RELEASE_VALIDATION_SUBSTAGE_ID",
"CleanupPromotionResult",
"CompletionMarkerPromotionResult",
"FullPromotionResult",
"GcsPromotionResult",
"HuggingFacePromotionResult",
"ReleaseArtifactSpec",
"ReleaseCandidateInputBundle",
"ReleasePromotionContext",
"ReleaseCandidateValidationDependencies",
"ReleaseCandidateValidator",
"ReleaseManifestPromotionResult",
"VersionManifestPromotionResult",
"VALIDATION_REPORT_POLICY_PRESENCE_ONLY",
"VALIDATION_REPORT_POLICY_REQUIRE_PASSING",
"build_legacy_release_candidate_bundle",
Expand Down
33 changes: 33 additions & 0 deletions policyengine_us_data/release_promotion/results/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
"""Typed Stage 5 promotion result models."""

from .cleanup import (
CLEANUP_STATUS_COMPLETED,
CLEANUP_STATUS_FAILED,
CLEANUP_STATUS_SKIPPED,
CLEANUP_STATUSES,
CleanupPromotionResult,
)
from .destinations import (
GcsPromotionResult,
HuggingFacePromotionResult,
)
from .full import FullPromotionResult
from .manifests import (
CompletionMarkerPromotionResult,
ReleaseManifestPromotionResult,
VersionManifestPromotionResult,
)

__all__ = [
"CLEANUP_STATUS_COMPLETED",
"CLEANUP_STATUS_FAILED",
"CLEANUP_STATUS_SKIPPED",
"CLEANUP_STATUSES",
"CleanupPromotionResult",
"CompletionMarkerPromotionResult",
"FullPromotionResult",
"GcsPromotionResult",
"HuggingFacePromotionResult",
"ReleaseManifestPromotionResult",
"VersionManifestPromotionResult",
]
46 changes: 46 additions & 0 deletions policyengine_us_data/release_promotion/results/_coercion.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
"""Shared coercion helpers for typed release-promotion results."""

from __future__ import annotations

from collections.abc import Sequence
from typing import Any

from policyengine_us_data.stage_contracts._coercion import require_non_empty


def nonnegative_int(value: Any, field_name: str) -> int:
"""Return a non-negative integer or raise a contract validation error."""

if isinstance(value, bool) or not isinstance(value, int):
raise ValueError(f"{field_name} must be an integer")
if value < 0:
raise ValueError(f"{field_name} must be non-negative")
return value


def bool_value(value: Any, field_name: str) -> bool:
"""Return a boolean or raise a contract validation error."""

if not isinstance(value, bool):
raise ValueError(f"{field_name} must be a boolean")
return value


def string_tuple(value: Any, field_name: str) -> tuple[str, ...]:
"""Return a tuple of non-empty strings from a sequence value."""

if value is None:
return ()
if isinstance(value, str) or not isinstance(value, Sequence):
raise ValueError(f"{field_name} must be a sequence of strings")
items = tuple(value)
for item in items:
require_non_empty(item, f"{field_name} item")
return items


def require_type(value: Any, field_name: str, expected_type: type) -> None:
"""Validate a nested typed result object."""

if not isinstance(value, expected_type):
raise ValueError(f"{field_name} must be {expected_type.__name__}")
Loading