feat(extensions): canonical src/ bundle layout with libraries, init hooks, python modules and extensions deprecation#39
Merged
Conversation
Introduce pre_init and post_init hook folders executed via runpy around initialize_pipeline(), with extensions/libraries on sys.path and legacy flat extensions/ deprecated. - Add extension_loader (add_extensions_libraries_to_sys_path, run_init_hooks) - Extend constants for libraries/pre_init/post_init paths; resolve python function files under extensions/libraries before legacy extensions/ - Wire hooks in DLTPipelineBuilder; bump version; add unit tests - Update docs, scaffold, samples (bronze), Cursor assets/SKILL, VERSION
rederik76
changed the title
1 task
added 4 commits
May 16, 2026 13:22
…tensions/ Replaces the single-directory extensions/ convention with a structured src/ layout that separates cluster libraries, spec-referenced Python, and lifecycle init scripts. Renames extension_loader.py to bundle_loader.py to reflect its broader role. Layout changes - Pipeline bundles: src/libraries/ (wheels + sys.path), src/python/ (spec Python), src/init/pre/ and src/init/post/ (lifecycle scripts) - Framework bundle: custom code lives exclusively under src/local/ (libraries/, python/, init/pre/, init/post/) — no top-level src/ equivalents - FrameworkPaths constants renamed to LOCAL_* to reflect local/ prefix; all *sourcePath confs resolve to src/ so constants omit the src/ segment Deprecation - extensions/ on sys.path deprecated (v0.13.0), removed in v1.0.0 - extensions/ is registered unconditionally alongside src/python/ when both exist - extensions/libraries/ (never shipped) dropped with no deprecation window Code - extension_loader.py → bundle_loader.py; uses FrameworkPaths.LOCAL_* for framework paths and PipelineBundlePaths.* for bundle paths - dataflow_spec_builder: search paths updated to LOCAL_* framework constants - constants.py: FrameworkPaths gains LOCAL_* paths; PipelineBundlePaths paths corrected to omit redundant src/ prefix Samples & template - bronze_sample migrated: extensions/libraries/ → src/python/, extensions/pre_init|post_init/ → src/init/pre|post/ - New libraries demo: phonenumbers wheel in src/libraries/, installed via environment.dependencies; demonstrates full install→import chain - pipeline_bundle_template scaffolded with README.md for each new src/ directory - src/local/ scaffolded in framework src/ with README.md files Docs & ADRs - feature_python_extensions.rst rewritten: new title, bundle context table, src/python/ replaces src/extensions/ throughout, environment.dependencies replaces libraries: whl/pypi syntax - feature_python_functions.rst, feature_python_source.rst: updated to src/python/ with deprecation notices for src/extensions/ - build_pipeline_bundle_steps.rst, build_pipeline_bundle_structure.rst: updated directory diagrams and added steps for libraries/python/init - docs/decisions/0001-bundle-src-layout.md, 0002-extensions-deprecation.md: new ADRs
rederik76
changed the title
haillew
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the single-directory extensions/ convention with a structured src/ layout
that separates cluster libraries, spec-referenced Python, and lifecycle init scripts.
Renames extension_loader.py to bundle_loader.py to reflect its broader role.
Layout changes
src/init/pre/ and src/init/post/ (lifecycle scripts)
python/, init/pre/, init/post/) — no top-level src/ equivalents
*sourcePath confs resolve to src/ so constants omit the src/ segment
Deprecation
Code
framework paths and PipelineBundlePaths.* for bundle paths
corrected to omit redundant src/ prefix
Samples & template
extensions/pre_init|post_init/ → src/init/pre|post/
environment.dependencies; demonstrates full install→import chain
Docs & ADRs
src/python/ replaces src/extensions/ throughout, environment.dependencies
replaces libraries: whl/pypi syntax
with deprecation notices for src/extensions/
directory diagrams and added steps for libraries/python/init