Skip to content

netann: make ChanStatusManager tests deterministic#10623

Open
erickcestari wants to merge 2 commits intolightningnetwork:masterfrom
erickcestari:make-chanstatusmanager-tests-deterministic
Open

netann: make ChanStatusManager tests deterministic#10623
erickcestari wants to merge 2 commits intolightningnetwork:masterfrom
erickcestari:make-chanstatusmanager-tests-deterministic

Conversation

@erickcestari
Copy link
Copy Markdown
Collaborator

@erickcestari erickcestari commented Mar 3, 2026

Replace wall-clock timing with injectable clock and ticker interfaces to eliminate flaky failures under CPU contention.

  • Add Clock and StatusSampleTicker fields to ChanStatusConfig
  • Replace time.NewTicker with ticker.Ticker interface
  • Replace time.Now() calls with injected clock
  • Replace time.Sleep and safeDisableTimeout with explicit tick/advanceTime/triggerDisable helpers
  • assertNoUpdates/assertUpdates no longer accept duration parameters

Before:

go test -run TestChanStatusManagerStateMachine ./netann/... -count=40
--- FAIL: TestChanStatusManagerStateMachine (0.00s)
    --- FAIL: TestChanStatusManagerStateMachine/add_new_channels (4.05s)
        chan_status_manager_test.go:781: received update for unexpected short chan id: 0:0:5479
FAIL
FAIL    github.com/lightningnetwork/lnd/netann  185.921s
FAIL

after:

go test -run TestChanStatusManagerStateMachine ./netann/... -count=1000 -race
ok      github.com/lightningnetwork/lnd/netann  47.401s

@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the reliability and determinism of the ChanStatusManager by decoupling its time-sensitive operations from the system's wall-clock. By introducing injectable clock and ticker interfaces, the manager's behavior can now be precisely controlled and simulated within tests, effectively resolving previously observed flaky test failures and ensuring consistent results.

Highlights

  • Deterministic Timing: Introduced clock.Clock and ticker.Ticker interfaces to replace direct usage of time.Now() and time.NewTicker, making time-related operations in ChanStatusManager deterministic.
  • Dependency Injection: The new clock and ticker interfaces are now injected into ChanStatusConfig and ChanStatusManager, allowing for precise control over time in testing environments.
  • Test Refactoring: Refactored chan_status_manager_test.go to utilize the injected clock and ticker, removing reliance on wall-clock time and introducing helper functions like tick(), advanceTime(), and triggerDisable() for controlled test execution.
  • Improved Test Reliability: Eliminated flaky test failures caused by CPU contention and non-deterministic timing, as evidenced by the before/after test results provided in the PR description.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • netann/chan_status_manager.go
    • Imported github.com/lightningnetwork/lnd/clock and github.com/lightningnetwork/lnd/ticker packages.
    • Added Clock and StatusSampleTicker fields to ChanStatusConfig for dependency injection.
    • Included a clock.Clock field in the ChanStatusManager struct.
    • Updated the statusSampleTicker field in ChanStatusManager to use the ticker.Ticker interface.
    • Modified NewChanStatusManager to initialize the clock and statusSampleTicker from the configuration, falling back to default implementations if not provided.
    • Added m.statusSampleTicker.Resume() in the start() method and m.statusSampleTicker.Stop() in the Stop() method.
    • Changed the statusManager goroutine to listen on m.statusSampleTicker.Ticks() instead of m.statusSampleTicker.C.
    • Replaced direct calls to time.Now() with m.clock.Now() in markPendingInactiveChannels and disableInactiveChannels.
  • netann/chan_status_manager_test.go
    • Imported github.com/lightningnetwork/lnd/clock and github.com/lightningnetwork/lnd/ticker packages for testing utilities.
    • Defined testStartTime for consistent test clock initialization.
    • Updated newManagerCfg to return and inject *clock.TestClock and *ticker.Force into the ChanStatusConfig.
    • Added clock, ticker, disableTimeout, and sampleInterval fields to the testHarness struct.
    • Modified newHarness to store the injected clock.TestClock and ticker.Force.
    • Introduced tick(), advanceTime(), and triggerDisable() helper methods to testHarness for programmatic control of test time.
    • Updated assertNoUpdates and assertUpdates functions to remove duration parameters and integrate with the new time control helpers.
    • Adjusted all calls to assertNoUpdates and assertUpdates within stateMachineTests to match their new signatures and leverage deterministic timing.
    • Replaced safeDisableTimeout usage with disableTimeout in testHarness and test cases.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the ChanStatusManager to use injectable clock and ticker interfaces, which is a great step towards making the tests fully deterministic and eliminating flakiness. The changes in chan_status_manager.go are well-implemented for dependency injection. The test file chan_status_manager_test.go is updated to use a test clock and ticker, with new helper functions to control time.

However, I've found that the new test implementation in chan_status_manager_test.go still contains sources of non-determinism by using time.Sleep and time.After. This goes against the primary goal of the PR. My review comments focus on these areas with suggestions for improvement to achieve fully deterministic tests.

Comment thread netann/chan_status_manager_test.go Outdated
Comment thread netann/chan_status_manager_test.go Outdated
@lightninglabs-deploy lightninglabs-deploy added the severity-medium Focused review required label Mar 3, 2026
@erickcestari erickcestari force-pushed the make-chanstatusmanager-tests-deterministic branch 2 times, most recently from e3e2b5e to 02beffb Compare March 3, 2026 16:40
@lightninglabs-deploy lightninglabs-deploy added severity-medium Focused review required and removed severity-medium Focused review required labels Mar 3, 2026
@erickcestari erickcestari force-pushed the make-chanstatusmanager-tests-deterministic branch from 02beffb to 4b371b3 Compare March 3, 2026 16:48
@lightninglabs-deploy lightninglabs-deploy added severity-medium Focused review required and removed severity-medium Focused review required labels Mar 3, 2026
@ziggie1984
Copy link
Copy Markdown
Collaborator

@erickcestari I think you introduced a race look at the CI test for more infos.

Replace wall-clock timing with injectable clock and ticker interfaces
to eliminate flaky failures under CPU contention.
- Add Clock and StatusSampleTicker fields to ChanStatusConfig
- Replace time.NewTicker with ticker.Ticker interface
- Replace time.Now() calls with injected clock
- Replace time.Sleep and safeDisableTimeout with explicit
  tick/advanceTime/triggerDisable helpers
- assertNoUpdates/assertUpdates no longer accept duration parameters
- Add dedicated no-op Sync method to the manager's event loop for
  side-effect-free test synchronization, replacing the prior approach
  of using RequestAuto which could silently mutate ManuallyDisabled
  channel state
- Add timeout guards on Force channel send and update channel receive
  to prevent indefinite hangs
- Reorder Stop() to call ticker.Stop() after the goroutine exits,
  eliminating a theoretical data race and fixing a pre-existing
  ticker resource leak
- Use time.Hour for Force ticker interval to prevent phantom real-time
  ticks during tests
@erickcestari erickcestari force-pushed the make-chanstatusmanager-tests-deterministic branch from dd11c6a to 0d9ffe6 Compare March 5, 2026 20:40
@erickcestari
Copy link
Copy Markdown
Collaborator Author

@erickcestari I think you introduced a race look at the CI test for more infos.

I believe those were other non-deterministic tests.

@saubyk saubyk requested a review from ziggie1984 March 24, 2026 17:00
@lightninglabs-deploy
Copy link
Copy Markdown
Collaborator

@ziggie1984: review reminder
@erickcestari, remember to re-request review from reviewers when ready

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

severity-medium Focused review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants