Add IQR-based environment fluctuation detection to benchmark timing stats#691
Merged
JewelRoam merged 1 commit intoPaddlePaddle:developfrom Apr 15, 2026
Merged
Add IQR-based environment fluctuation detection to benchmark timing stats#691JewelRoam merged 1 commit intoPaddlePaddle:developfrom
JewelRoam merged 1 commit intoPaddlePaddle:developfrom
Conversation
…tats This commit introduces IQR (Interquartile Range) based environment fluctuation detection to the timing statistics calculation in test_compiler_util.py. The feature helps detect unstable benchmarking environments by measuring the relative variation in timing results. Key Changes: - Enhanced get_timing_stats() to compute median, Q1, Q3, and IQR - Added environment variable GRAPH_NET_FLUCTUATION_DETECT_THRESHOLD for configurable fluctuation detection sensitivity - RuntimeError is raised when IQR/median exceeds the threshold - Extended return stats dictionary with new fields: median, iqr - Updated print_times_and_speedup() to use median for speedup calculation IQR/median Ratio: - Measures relative variability of timing measurements - Lower values indicate more consistent timing - Higher values indicate environment instability or interference Environment Variable Configuration: - GRAPH_NET_FLUCTUATION_DETECT_THRESHOLD (default: 0.2) - Controls the sensitivity of fluctuation detection Detection Algorithm: 1. Calculate median, Q1 (25th percentile), Q3 (75th percentile) 2. Compute IQR = Q3 - Q1 3. Calculate relative IQR = IQR / median 4. Compare against threshold 5. Raise RuntimeError with detailed diagnostics if exceeded Error Message Format: When fluctuation is detected, the error message includes: - IQR/median ratio and threshold - Raw timing values for manual inspection Use Cases: - Multi-user GPU environments where timing variance is common - CI/CD pipeline monitoring for performance regression detection - Manual benchmark verification in shared resources - Identifying external workload interference Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thanks for your contribution! |
lixinqi
approved these changes
Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
File change the same as https://github.com/PaddlePaddle/ai4c/pull/72/changes https://github.com/PaddlePaddle/ai4c/pull/75/changes