fix: enhanced document upload, page limit, upload limit tests#839
fix: enhanced document upload, page limit, upload limit tests#839MODSetter merged 12 commits intoMODSetter:devfrom
Conversation
- Introduced a TaskDispatcher abstraction to decouple the upload endpoint from Celery, allowing for easier testing with synchronous implementations. - Updated the create_documents_file_upload function to utilize the new dispatcher for task management. - Removed direct Celery task imports from the upload function, enhancing modularity. - Added integration tests for document upload, including page limit enforcement and file size restrictions.
- Removed commented-out testing configuration from .env.example to streamline the file. - Updated markers in pyproject.toml to remove the e2e test marker, clarifying the purpose of the remaining markers.
…ation - Updated the embedding dimension in test configurations to use the value from the application config, enhancing maintainability and consistency across tests.
…integration tests
|
@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 30617c6..836d529
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (13)
• surfsense_backend/.env.example
• surfsense_backend/app/routes/documents_routes.py
• surfsense_backend/app/services/task_dispatcher.py
• surfsense_backend/pyproject.toml
• surfsense_backend/tests/conftest.py
• surfsense_backend/tests/e2e/conftest.py
• surfsense_backend/tests/integration/conftest.py
• surfsense_backend/tests/integration/document_upload/conftest.py
• surfsense_backend/tests/integration/document_upload/test_document_upload.py
• surfsense_backend/tests/integration/document_upload/test_page_limits.py
• surfsense_backend/tests/integration/document_upload/test_upload_limits.py
• surfsense_backend/tests/integration/indexing_pipeline/test_index_document.py
• surfsense_backend/tests/utils/helpers.py
⏭️ Files skipped (1)
| Locations |
|---|
surfsense_backend/tests/integration/document_upload/__init__.py |
… layout - Introduced SidebarSeparator component for improved sidebar organization. - Updated layout configuration to utilize the new SidebarSeparator in the documentation layout.
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 836d529..2468cc2
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (12)
• surfsense_web/app/docs/layout.tsx
• surfsense_web/app/docs/sidebar-separator.tsx
• surfsense_web/app/globals.css
• surfsense_web/app/layout.config.tsx
• surfsense_web/content/docs/connectors/meta.json
• surfsense_web/content/docs/docker-installation.mdx
• surfsense_web/content/docs/how-to/meta.json
• surfsense_web/content/docs/index.mdx
• surfsense_web/content/docs/installation.mdx
• surfsense_web/content/docs/manual-installation.mdx
• surfsense_web/content/docs/testing.mdx
• surfsense_web/lib/source.ts
2 participants
Description
document_upload/tests API behavior through public endpoints,indexing_pipeline/tests pipeline internals in isolationconftest.pywith automaticsurfsense_testdatabase overrideapp_config.embedding_model_instance.dimensionacross all test files.env.exampleandpyproject.tomlby removing unused testing config and thee2emarkerMotivation and Context
FIX #
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This PR refactors the document upload testing architecture by replacing end-to-end tests with integration tests that run the FastAPI app in-process via
ASGITransport. It introduces a task dispatcher abstraction (TaskDispatcherprotocol) that allows tests to process documents synchronously without requiring Celery/Redis, centralizes test database configuration in the rootconftest.pywith automatic test database override, and replaces hardcoded embedding dimensions with dynamic values fromapp_config. The testing configuration is cleaned up by removing thee2emarker and unused environment variables, while the test suite now covers document upload API behavior including multi-file uploads, duplicate detection, authentication, error handling, page limits, upload limits, and searchability.⏱️ Estimated Review Time: 30-90 minutes
💡 Review Order Suggestion
surfsense_backend/tests/conftest.pysurfsense_backend/pyproject.tomlsurfsense_backend/.env.examplesurfsense_backend/app/services/task_dispatcher.pysurfsense_backend/app/routes/documents_routes.pysurfsense_backend/tests/integration/conftest.pysurfsense_backend/tests/integration/document_upload/conftest.pysurfsense_backend/tests/utils/helpers.pysurfsense_backend/tests/integration/document_upload/test_document_upload.pysurfsense_backend/tests/integration/document_upload/test_page_limits.pysurfsense_backend/tests/integration/document_upload/test_upload_limits.pysurfsense_backend/tests/integration/indexing_pipeline/test_index_document.py