log-gen with patterns in worktree #1582
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces an accumulator mechanism into the capture manager, allowing for a rolling-window delay of metrics. The goal here is to support historic metric uses cases, such as that present in a theoretical Datadog intake blackhole, necessary to support #1546 and #1530. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
This commit removes from discovered or user-provided labels any reserved fields. We do this to avoid a serialization bug where a label duplicates one of lading's always-present fields. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
This commit adds an additional fuzz harness to assert that capture files produced by lading_capture are valid. This runs the CaptureManager in afl++ in forking mode, meaning we exercise the global registry, historical metrics et al without needing to elide any part of the novel machinery here. This fuzz test is however very slow. We rely on tokio's pause/advance to manage time but it's still not quick to run. I have not found any validation issues in produced capture files. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
This PR introduces support for Datadog intake v2 in a new blackhole, the goal being to accumulate trace-agent metrics that are only transmitted to dogstatsd and then out to the intake. The construction method here is intended to make it easy to add support for new intake versions. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
TheSafo
force-pushed
the
jsaf/patterned-log-gen
branch
from
02fc1cb to
c8c96ef
Compare
Fix clippy lints for patterned log generator - Add arbitrary::Arbitrary trait for Config and LogFormat when feature is enabled - Fix clone_on_copy by dereferencing config - Fix trivially_copy_pass_by_ref for LogLevel::as_str - Fix uninlined_format_args by using inline format strings - Fix unused_self by making helper functions static - Fix unwrap_used by using expect or checking Option - Format code with cargo fmt Fix timestamp ordering issue by dropping historical metrics Previously, historical metrics (with timestamps before capture start) were clamped to tick 0, which caused multiple metrics to collapse to the same timestamp. This violated the ascending timestamp order requirement in captures.zstd files. Now, metrics with timestamps before capture start are dropped with a warning instead of being clamped. This prevents timestamp ordering issues while still handling historical metrics gracefully without panicking. Changes: - instant_to_tick() -> instant_to_tick_checked() returning Option<u64> - handle_metric_received() now checks for None and logs a warning - Updated test to verify metrics are dropped (not panicked on) - Added proptest as dev-dependency for state_machine tests
Dropping historical metrics was causing negative baseline errors in analysis. Now historical metrics are clamped to tick 0 (accepted) instead of dropped. The line_count increment ensures no duplicate timestamps even when multiple metrics are at tick 0.
Clamping all historical metrics to tick 0 caused them to pile up and flush together, making counters appear to reset (causing negative baselines). Now historical metrics are clamped to accumulator.current_tick, spreading them across the appropriate time window and preventing counter resets.
TheSafo
force-pushed
the
jsaf/patterned-log-gen
branch
from
c8c96ef to
7f1555e
Compare
TheSafo
force-pushed
the
stateful-logs-blackhole
branch
from
b61a8ce to
7542753
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Related issues
A list of issues either fixed, containing architectural discussions, otherwise relevant
for this Pull Request.
Additional Notes
Anything else we should know when reviewing?