Open
Conversation
- Add query_pandas_from_snowflake_via_s3_stage() for efficient large query results (>10M rows) - Add publish_pandas_via_s3_stage() for efficient large DataFrame writes (>10M rows) - Add make_batch_predictions_from_snowflake_via_s3_stage() for batch ML predictions - Support dev/prod environment switching via current.is_production - Add helper functions for S3 operations and SQL generation - Add metaflow_s3/utils.py with S3 utility functions - Add comprehensive functional tests - Integrate with existing Metaflow card system and cost tracking
… functional tests
…r schema from DataFrame
…te table creation logic
…function for improved readability and maintainability
…for improved clarity and performance
…in batch inference
…using S3 file retrieval function
…ient assignment for improved clarity
…for improved performance
…tegrate with multiprocessing
…roved parallel processing
…r for improved parallel processing
…ved concurrency in batch inference
…e and temporary file handling
…remove unused functions
- Introduced new documentation for `make_pydantic_parser_fn`, `publish`, `publish_pandas`, `query_pandas_from_snowflake`, and `restore_step_state` functions. - Removed the outdated `pandas.md` and `validate_config.md` documentation files. - Updated the Snowflake utilities README to reflect the integration with Metaflow and emphasize the use of high-level APIs.
…ount in query_and_batch step
…chInferencePipeline
…ation for clarity and consistency
…te_snowflake_to_s3_copy_query
…e formatting in publish_pandas documentation
…n for clarity and completeness
… of publish to avoid affecting other tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates:
use_s3_stageparameter inpublish_pandasandquery_pandas_from_snowflakeBatchInferencePipelinefor inference using s3 stage