2742 hi goodtimes statistical filter 2

[subagonsouth]

Change Summary

Summary

Implements Statistical Filter 2 from the IMAP-Hi Algorithm Document Section 2.3.2.3, which detects and removes short-lived "pulses" of qualified counts that may be temporally correlated between sensors.

Changes

hi_goodtimes.py (+250 lines)

mark_statistical_filter_2: Main filter function that:

• Groups events by 8-spin set (esa_sweep, esa_step)
• Filters to qualified coincidence types (calibration products 1 or 2)
• Sorts events by event_met
• Finds clusters where ≥min_events occur within max_time_delta seconds
• Marks affected spin bins (with padding and wraparound) for all METs in that 8-spin set

_find_event_clusters: Vectorized helper that finds time clusters using:

• Window span computation: event_times[n-1:] - event_times[:-n+1]
• Transition detection via padded np.diff to identify contiguous regions

_compute_bins_for_cluster: Computes spin bins to cull with:

• np.unwrap to handle clusters spanning the 0/89 boundary
• Configurable bin padding with modular wraparound

utils.py (+29 lines)

Added to HiConstants:

• STAT_FILTER_2_MIN_EVENTS = 6
• STAT_FILTER_2_MAX_TIME_DELTA = 10.0 (seconds)
• STAT_FILTER_2_BIN_PADDING = 1
• DE_CLOCK_TICK_DURATION_SECONDS and DE_CLOCK_TICK_DURATION_MICROSECONDS (moved from hi_l1a.py)

hi_l1a.py, hi_l1b.py, hi_l1c.py, test_hi_l1c.py

• Refactored to use HiConstants.DE_CLOCK_TICK_* instead of local constants

test_hi_goodtimes.py (+465 lines)

• TestFindEventClusters: 7 tests for cluster detection (empty, too few, spread out, single/multiple clusters, merging, exact threshold)
• TestComputeBinsForCluster: 6 tests for bin computation (basic range, wrapping at 0/max, partial cluster, boundary spanning)
• TestStatisticalFilter2: 6 integration tests (no qualified events, no clusters, cluster detected, multiple clusters, bin padding with wrapping, custom parameters)

Closes: #2742

[Copilot]

Pull request overview

This pull request implements Statistical Filter 2 from the IMAP-Hi Algorithm Document Section 2.3.2.3, which detects and removes short-lived "pulses" of qualified counts that may be temporally correlated between sensors. These pulses are typically visible only at the highest few energy steps and are not caught by Statistical Filter 1.

Changes:

• Added mark_statistical_filter_2 function and helper functions (_find_event_clusters, _compute_bins_for_cluster) to detect event clusters and mark affected spin bins
• Added Statistical Filter 2 tuning parameters (min_events, max_time_delta, bin_padding) to HiConstants in utils.py
• Refactored DE_CLOCK_TICK constants from hi_l1a.py to HiConstants class for centralized access
• Added comprehensive test coverage for cluster detection, bin computation with wrapping, and filter integration

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
imap_processing/hi/hi_goodtimes.py	Implements Statistical Filter 2 with vectorized cluster detection, bin computation with wraparound, and marking of affected spin bins across 8-spin sets
imap_processing/hi/utils.py	Adds Statistical Filter 2 constants and consolidates DE_CLOCK_TICK constants from hi_l1a.py
imap_processing/hi/hi_l1a.py	Refactors to use DE_CLOCK_TICK constants from HiConstants
imap_processing/hi/hi_l1b.py	Updates to use HiConstants.HALF_CLOCK_TICK_S instead of local constant
imap_processing/hi/hi_l1c.py	Updates to use HiConstants clock tick constants instead of importing from hi_l1a
imap_processing/tests/hi/test_hi_l1c.py	Updates tests to use HiConstants.DE_CLOCK_TICK_S
imap_processing/tests/hi/test_hi_goodtimes.py	Adds comprehensive tests for cluster detection, bin computation, and filter integration including edge cases like boundary wrapping

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lacoak21]

LGTM! 💯

[lacoak21]

Isnt this a duplicate of lines 1697 and 1698?

[lacoak21]

why do you need flatnonzero? Isnt diff 1d?

[subagonsouth]

flatnonzero returns the array of indices directly instead of a tuple of array of indices. So, it avoids having to add the [0] on the end to index the first returned element.

[lacoak21]

Ohh - I see.

[lacoak21]

is it worth checking that the last bins arent flagged as well?