config agent: reduce full config check frequency from 5s to 1m and compare hashes instead#3028
Draft
config agent: reduce full config check frequency from 5s to 1m and compare hashes instead#3028
Conversation
nikw9944
changed the title
nikw9944
force-pushed
the
nikw/config-agent-cpu
branch
from
7260b33 to
e81a21e
Compare
Contributor
Author
BEFORE: Simple polling every 5 secondsAFTER: Hash-based polling (5s hash check, 5m full config fetch) |
nikw9944
force-pushed
the
nikw/config-agent-cpu
branch
5 times, most recently
from
6fa1ee4 to
2ab7f72
Compare
Contributor
Author
|
Here's a before and after comparison of total CPU usage by Arista EOS ConfigAgent when running all tests in parallel. Before change: After change: Note that the peak usage is about the same, but the steady state usage (after all tests have run and the config agents are just polling) is reduced from about 400% (consuming 4 cores) to about 200% (consuming 2 cores) across 29 EOS containers. |
nikw9944
force-pushed
the
nikw/config-agent-cpu
branch
2 times, most recently
from
b80e2f9 to
c8454d9
Compare
nikw9944
changed the title
nikw9944
changed the title
packethog
requested changes
Feb 27, 2026
|
|
||
| // Find unknown BGP peers that need to be removed | ||
| peerFound := func(peer net.IP) bool { | ||
| for _, tun := range device.Tunnels { |
Contributor
There was a problem hiding this comment.
The old code looped over deviceForRender.Tunnels which looks like it was a post-dedupe slice of tunnels. Should we still be doing that here?
| } | ||
|
|
||
| // Response containing only the config hash | ||
| message ConfigHashResponse { |
Contributor
There was a problem hiding this comment.
Why make a separate RPC for this as opposed to just checking the hash in ConfigResponse and throwing away the config payload if the last hash is the same? Other than bytes on the wire, doesn't it simplify things?
Contributor
Author
There was a problem hiding this comment.
Nope, just bytes on the wire, but when we had packet loss from Singapore to the controller it may have helped.
CHANGELOG.md
Outdated
| - CLI | ||
| - Remove restriction for a single tunnel per user; now a user can have a unicast and multicast tunnel concurrently (but can only be a publisher _or_ a subscriber) ([2728](https://github.com/malbeclabs/doublezero/pull/2728)) | ||
| - Device agents | ||
| - Reduce config agent network and CPU usage by checking config checksums every 5 seconds, and reducing full config check frquency to 1m |
| } | ||
|
|
||
| // GetConfigHash returns only the hash of the configuration for change detection | ||
| func (c *Controller) GetConfigHash(ctx context.Context, req *pb.ConfigRequest) (*pb.ConfigHashResponse, error) { |
Contributor
There was a problem hiding this comment.
Can you add tests for this?
| ).Inc() | ||
|
|
||
| hash := sha256.Sum256([]byte(configStr)) | ||
| getConfigDuration.Observe(float64(time.Since(reqStart).Seconds())) |
Contributor
There was a problem hiding this comment.
This is re-using the same histogram as the GetConfig RPC. We should create a new histogram for this RPC endpoint because otherwise it will skew the latency profile downward.
…ture docs Extract config generation logic into reusable functions: - generateConfig() - renders device config with deduplication - processConfigRequest() - validates request and finds unknown BGP peers This refactoring prepares for adding GetConfigHash endpoint that will share the same config generation logic. Also add architecture documentation with sequence diagram showing agent-controller communication flow.
…ange detection Add new GetConfigHash RPC that returns only the SHA256 hash of the device configuration (64 bytes) instead of the full config (~50KB). This enables agents to efficiently check for config changes without transferring the full configuration on every poll. Changes: - Add GetConfigHash RPC to controller.proto - Implement GetConfigHash() handler that reuses processConfigRequest() - Add controller_grpc_getconfighash_requests_total metric - Regenerate protobuf code
…meout Replace aggressive 5-second full config polling with hash-based change detection. The agent now: - Checks config hash every 5 seconds (64 bytes) - Only fetches and applies full config when hash changes - Forces full config check after timeout (default 60s) as safety net This dramatically reduces: - Network bandwidth (99%+ when config unchanged) - EOS device load (no config application when unchanged) - Agent CPU (hash computed only when fetching new config) Add --config-cache-timeout-in-seconds flag to control the forced full config check interval. Refactor main loop: - Split pollControllerAndConfigureDevice into focused functions - Add computeChecksum() helper for SHA256 hashing - Add fetchConfigFromController() to get config and compute hash - Add applyConfig() to apply config to EOS device - Rename variables: cachedConfigHash, configCacheTime, configCacheTimeout Add GetConfigHashFromServer() client function to call new gRPC endpoint.
nikw9944
force-pushed
the
nikw/config-agent-cpu
branch
from
c8454d9 to
ab051f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Controller
Agent
Testing Verification