[deploy] 0.2.4 - Dev: performance update & data loader improvement #171
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…emove-redundant-code remove redundant code
Added PostgreSQL dataloade
PostgreSQL support
Adding support for MSSQL data loader
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ime-format-in-kusto
…tables-and-fix-datetime-format-in-kusto [Kusto] add filter tables and fix datetime format
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| try: | ||
| df = pd.DataFrame(json.loads(raw_data)) | ||
| except Exception as e: | ||
| return jsonify({"status": "error", "message": f"Invalid JSON data: {str(e)}, it must be in the format of a list of dictionaries"}), 400 |
Check warning
Code scanning / CodeQL
Information exposure through an exception Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 6 months ago
To fix the issue, we will replace the direct inclusion of the exception message (str(e)) in the response with a generic error message. The detailed exception message will be logged on the server for debugging purposes. This ensures that sensitive information is not exposed to the user while still allowing developers to diagnose issues using the logs.
Specifically:
- Replace the direct use of
str(e)in the response with a generic message like "Invalid JSON data provided." - Log the detailed exception message (
str(e)) on the server using the existing logger.
Suggested changeset
1
py-src/data_formulator/tables_routes.py
| @@ -318,3 +318,4 @@ | ||
| except Exception as e: | ||
| return jsonify({"status": "error", "message": f"Invalid JSON data: {str(e)}, it must be in the format of a list of dictionaries"}), 400 | ||
| logger.error(f"Invalid JSON data: {str(e)}") | ||
| return jsonify({"status": "error", "message": "Invalid JSON data provided. It must be in the format of a list of dictionaries."}), 400 | ||
|
|
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
Collaborator
Author
|
This pull request introduces several updates across multiple files to enhance functionality, improve logging, and refine the development experience. Key changes include updates to data loaders, improved logging mechanisms, and modifications to the application startup process. Data Loader Enhancements:
Logging Improvements:
Application Startup Modifications:
Dependency Updates:
Additional Changes:
|
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR enhances front-end performance via memoization and selector optimizations, integrates SSE for real-time updates, and improves data loader and UI components.
- Refactor chart rendering with
dfSelectors.getAllCharts,useMemo, andmemoto reduce unnecessary re-renders - Add SSE client/server support for real-time data formulation updates
- Update dialogs, grids, and snackbars with new slotProps, cleaner layouts, and raw data syncing
Reviewed Changes
Copilot reviewed 40 out of 42 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/views/VisualizationView.tsx | Refactored chart skeleton utility and menu layout; switched to memoized selectors |
| src/views/MessageSnackbar.tsx | Grouped system messages, added expand/collapse and count UI |
| src/views/DataThread.tsx | Memoized chart elements and integrated pending SSE status |
| src/app/dfSlice.tsx | Added SSE message handling state and selector utilities |
| py-src/data_formulator/tables_routes.py | Extended table creation endpoint for raw_data and sanitization |
Comments suppressed due to low confidence (2)
src/views/DataThread.tsx:728
_.isEqualis used without importing lodash. Addimport _ from 'lodash';or replace with another deep-equality check.
_.isEqual(prevProps.chart.encodingMap, nextProps.chart.encodingMap) &&
py-src/data_formulator/tables_routes.py:71
- You reference
pd.DataFrame()but pandas (pd) is not imported at the top of this file. Addimport pandas as pd.
sample_rows = db.execute(f"SELECT * FROM {table_name} LIMIT 1000").fetchdf() if row_count > 0 else pd.DataFrame()
Collaborator
Author
|
Huge thanks to all contributors to this PR! |
Chenglong-MS
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improves front-end performance, data loader utility, and prepares functionality for the next updates.