Cache Improvements #29
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…zation demo script
… cache - Added comprehensive caching system inspired by VFB_connect to VFBquery. - Implemented memory and disk caching with configurable TTL and size limits. - Enhanced core functions with caching capabilities, including term info and instance retrieval. - Created a demo script to showcase caching functionality and performance improvements. - Developed a test suite to validate caching behavior and performance metrics. - Updated documentation to reflect new caching features and usage examples.
- Added solr_cache_demo.py to demonstrate caching benefits and cold start problem. - Created solr_cache_integration.py to integrate SOLR caching into existing VFBquery functions. - Developed solr_result_cache.py for server-side caching using SOLR, including metadata management and expiration handling. - Introduced test_solr_cache_enhanced.py to validate caching lifecycle, expiration, cleanup, and performance metrics.
… improving data integrity and retrieval efficiency. Add production tests for cache functionality and field preservation.
- Updated README.md to clarify caching features and examples, emphasizing server-side caching. - Removed SOLR_CACHING.md as the content is now integrated into README.md. - Deleted cache optimization demo scripts (cache_optimization_demo.py, native_caching_demo.py, production_cache_test.py, solr_cache_demo.py) as they are no longer needed. - Improved caching examples in README.md to reflect current functionality and usage.
…nd improve clarity
…in get_instances function
…n formatting for thumbnails
…atch expected Neo4j format
…nulls function to improve list filtering; adjust cache_data serialization in SolrResultCache
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements comprehensive caching improvements for VFBquery to address cold start performance issues and reduce response times from minutes to seconds. The changes introduce multiple layers of caching including in-memory, disk-based, and SOLR-based caching systems, all enabled by default with production-ready configurations.
Key changes:
- Added SOLR-based result caching with 3-month TTL for server-side optimization
- Implemented multi-layer memory and disk caching with intelligent LRU eviction
- Enhanced
get_instancesfunction with fallback SOLR implementation when Neo4j is unavailable - Added comprehensive performance testing infrastructure and monitoring
Reviewed Changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/vfbquery/solr_result_cache.py | New SOLR-based caching system for server-side result storage |
| src/vfbquery/cache_enhancements.py | Multi-layer caching with memory management and disk persistence |
| src/vfbquery/cached_functions.py | Enhanced VFBquery functions with integrated caching support |
| src/vfbquery/vfb_queries.py | Updated get_term_info and get_instances with caching and fallback mechanisms |
| src/vfbquery/init.py | Auto-enable caching on import with configurable environment controls |
| test_parsing.sh | Test script for GitHub Actions workflow parsing validation |
| .github/workflows/performance-test.yml | Automated performance testing and reporting workflow |
| src/test/term_info_queries_test.py | Added performance benchmarking test with timing thresholds |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.