Skip to content

[🐛 Bug]: Correlation Incident prefix intermittently ignored #5450

@hrabalvojta

Description

@hrabalvojta

Describe the bug
Correlation Incident prefix is not included in all Incident names.

So I ended up with this:

Status Incident Alerts Sources
Firing INC-9 - KubeDeploymentRolloutStuck 1 prometheus
Firing KubePodNotReady 2 prometheus
Firing INC-7 - KubeDeploymentReplicasMismatch 1 prometheus

To Reproduce
Steps to reproduce the behavior:

  1. Have Kubernetes cluster with kube-prometheus-stack deployed
  2. Configure provider prometheus
  3. Configure alertmanager to send alerts to keep
  4. Create simple manual correlation
Correlation name: default
Append to the same Incident if delay between alerts is below: 72 hours
Incident name template: {{ alert.alertname }}
Incident prefix: INC
Select attribute to group by: alert.alertname
Resolve on: all alerts resolved
Start incident on: Any condition met
Alerts threshold: 1
Filter:
  Source: prometheus
  1. Create fail deployment to create alerts
    I created deployment with non-existenting image spec.template.spec.containers[0].image: afdahf and spec.replicas: 2
  2. When alerts are triggered you should see incidents above.

Expected behavior
It is expected that correlation - Incident prefix will be present in all incidents.

Status Incident Alerts Sources
Firing INC-9 - KubeDeploymentRolloutStuck 1 prometheus
Firing INC-8 - KubePodNotReady 2 prometheus
Firing INC-7 - KubeDeploymentReplicasMismatch 1 prometheus

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Versions:

  • kubernetes: v1.34.1+k3s1
  • keephq helm chart: keep-0.1.93
  • kube-prometheus-stack helm chart: kube-prometheus-stack-78.5.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions