From 6fd7b862231db183ca7b2b53ccd8a301dd5f73ca Mon Sep 17 00:00:00 2001 From: xeniape Date: Wed, 30 Jul 2025 13:31:50 +0200 Subject: [PATCH 01/15] docs: document mTLS configuration for NiFi 2.x monitoring --- .../nifi/pages/usage_guide/monitoring.adoc | 121 +++++++++--------- 1 file changed, 58 insertions(+), 63 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 585df15a..ebbe8acc 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -2,8 +2,9 @@ :description: The Stackable Operator for Apache NiFi automatically configures NiFi to export Prometheus metrics. :k8s-job: https://kubernetes.io/docs/concepts/workloads/controllers/job/ :k8s-network-policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/ +:prometheus-operator: https://prometheus-operator.dev/ -In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`]. +In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`,window=_blank]. The NiFi `2.0.0` release changed the way of exposing Prometheus metrics significantly. The following steps explain on how to expose Metrics in NiFi versions `1.x.x` and `2.x.x`. @@ -11,10 +12,10 @@ The following steps explain on how to expose Metrics in NiFi versions `1.x.x` an == Configure metrics in NiFi `1.x.x` For NiFi versions `1.x.x`, the operator automatically configures NiFi to export Prometheus metrics. -This is done by creating a {k8s-job}[Job] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task]. +This is done by creating a {k8s-job}[Job,window=_blank] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task,window=_blank]. IMPORTANT: Network access from the Job to NiFi is required. -If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies], make sure to allow access from the Job to NiFi. +If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies,window=_blank], make sure to allow access from the Job to NiFi. See xref:operators:monitoring.adoc[] for more details. @@ -34,94 +35,83 @@ spec: == Configure metrics in NiFi `2.x.x` -The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507]. -Metrics are now always exposed and can be scraped using the NiFi Pod FQDN and the HTTP path `/nifi-api/flow/metrics/prometheus`. +The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507,window=_blank]. +Metrics are now always exposed and can be scraped using the NiFi `-metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`. -For a deployed single node NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under: +For a deployed NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under: ``` -https://simple-nifi-node-default-0.simple-nifi-node-default..svc.cluster.local:8443/nifi-api/flow/metrics/prometheus +https://simple-nifi-node-default-metrics..svc.cluster.local:8443/nifi-api/flow/metrics/prometheus ``` -IMPORTANT: If NiFi is configured to do any user authentication, requests to the metric endpoint must be authenticated and authorized. +NOTE: The above URL connects to one of the Pods, reachable through the specified Service, therefore scraping metrics produced by that Pod only. +To scrape metrics from a particular Pod the FQDN of the Pod and the `-headless` Service need to be used. For example: `\https://simple-nifi-node-default-0.simple-nifi-node-default-headless..svc.cluster.local:8443/nifi-api/flow/metrics/prometheus` -=== Authentication with NiFi `2.x.x` - -[IMPORTANT] -=== -The NiFi metrics endpoints are behind a strong authentication mechanism which require credentials for each individual pod. -=== +IMPORTANT: If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized. -To authenticate, you can use a bearer token created by your NiFi instance e.g. +=== Authentication with NiFi `2.x.x` -[source,bash] ----- -curl -X POST https://simple-nifi-node-default-0.simple-nifi-node-default..svc.cluster.local:8443/nifi-api/access/token -d 'username=&password=' -k ----- +To authenticate against the NiFi `2.x.x` API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between +Kubernetes Pods, check out the xref:home:secret-operator:index.adoc[Secret Operator documentation]. -where `-k` equals `verify=false` to allow self-signed certificates. The reply is your bearer token. +For the following example, showing how to configure a Prometheus scraper for NiFi, we use the afore-mentioned method of configuring mTLS +and utilizing the internally available `tls` xref:home:secret-operator:secretclass.adoc[SecretClass]. -The following example shows how to configure the Prometheus scraper to use the bearer token to authenticate against a NiFi pod. +To generate a client certificate signed by the `tls` SecretClass CA trusted in NiFi, add the following `volume` and `volumeMount` +to the Prometheus Pod. -[source,yaml] ----- ---- -authorization: <1> - type: Bearer - credentials: "" <2> -tls_config: - insecure_skip_verify: true -static_configs: - - targets: - - '..svc.cluster.local:8443' <3> -metrics_path: '/nifi-api/flow/metrics/prometheus' -scheme: https ----- -<1> Use the `authorization` property instead if the `basic_auth`. -<2> Add the previously obtained token here. -<3> Static targets only scrapes one pod. +IMPORTANT: If the {prometheus-operator}[Prometheus Operator,window=_blank] is being used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the https://github.com/prometheus-operator/prometheus-operator/issues/7709[Prometheus Operator Issues,window=_blank]. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank]. -or use it in a NiFi secret which should look like [source,yaml] ---- --- -apiVersion: v1 -kind: Secret -metadata: - name: nifi-authorization-secret -type: Opaque -stringData: - nifi_token: "" +prometheus: <1> + prometheusSpec: + volumes: + - name: tls + ephemeral: + volumeClaimTemplate: + metadata: + annotations: + secrets.stackable.tech/class: tls # <2> + secrets.stackable.tech/scope: pod,service=prometheus-kube-prometheus-prometheus # <3> + spec: + storageClassName: secrets.stackable.tech + accessModes: + - ReadWriteOnce + resources: + requests: + storage: "1" + volumeMounts: + - name: tls + mountPath: /stackable/tls # <4> ---- +<1> This given configuration is set in the {prometheus-operator}docs/api-reference/api/#monitoring.coreos.com/v1.Prometheus[Prometheus resource,window=_blank] for the {prometheus-operator}[Prometheus Operator,window=_blank] +<2> We use the `tls` SecretClass here, shipped with the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well +<3> We add the `service=prometheus-kube-prometheus-prometheus` scope for the Prometheus Service to be included in the `subjectAlternateName` of the TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank] +<4> The path where the mTLS certificates are mounted to inside the Prometheus Pod If you want to use a `ServiceMonitor` you'd need to configure it as follows: -// TODO: The ServiceMonitor should be switched to the -metrics service - [source,yaml] ---- --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: - name: scrape-nifi2 + name: scrape-nifi labels: stackable.tech/vendor: Stackable release: prometheus spec: endpoints: - - port: https - path: 'nifi-api/flow/metrics/prometheus' + - path: /nifi-api/flow/metrics/prometheus + port: https scheme: https - interval: 5s - tlsConfig: - insecureSkipVerify: true - authorization: - credentials: <1> - key: "nifi_token" - name: "nifi-authorization-secret" - optional: false - type: "Bearer" + tlsConfig: <1> + caFile: /stackable/tls/ca.crt + certFile: /stackable/tls/tls.crt + keyFile: /stackable/tls/tls.key relabelings: <2> - sourceLabels: - __meta_kubernetes_pod_name @@ -130,7 +120,7 @@ spec: - __meta_kubernetes_pod_container_port_number targetLabel: __address__ replacement: ${1}.${2}.${3}.svc.cluster.local:${4} - regex: (.+);(.+?)(?:-headless)?;(.+);(.+) + regex: (.+);(.+?)(?:-headless)?;(.+);(.+) # <3> selector: matchLabels: prometheus.io/scrape: "true" @@ -138,7 +128,12 @@ spec: any: true jobLabel: app.kubernetes.io/instance ---- -<1> Authorization via Bearer Token stored in a secret -<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of target pod +<1> In the TLS configuration of the ServiceMonitor, specify the paths to the mTLS files mounted to the Prometheus Pod +<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary for scraping NiFi since it requires a DNS name to address the NiFi REST API +<3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `-headless` Service, which is why we use the `-headless` Service instead of the `-metrics` Service for scraping NiFi metrics NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-headless` thus we need to regex this suffix. + +The described example is part of the https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus.yaml[Prometheus,window=_blank] +and https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[ServiceMonitor,window=_blank] manifests +being used in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank] of the https://github.com/stackabletech/demos[`demos` repository,window=_blank]. From 319b9d89120ab6409bb1566385853cbe08776483 Mon Sep 17 00:00:00 2001 From: xeniape Date: Thu, 31 Jul 2025 14:33:02 +0200 Subject: [PATCH 02/15] update ServiceMonitor manifest --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index ebbe8acc..f1530025 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -99,7 +99,7 @@ If you want to use a `ServiceMonitor` you'd need to configure it as follows: apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: - name: scrape-nifi + name: scrape-nifi-2 labels: stackable.tech/vendor: Stackable release: prometheus @@ -119,10 +119,11 @@ spec: - __meta_kubernetes_namespace - __meta_kubernetes_pod_container_port_number targetLabel: __address__ - replacement: ${1}.${2}.${3}.svc.cluster.local:${4} - regex: (.+);(.+?)(?:-headless)?;(.+);(.+) # <3> + replacement: ${1}.${2}-headless.${3}.svc.cluster.local:${4} # <3> + regex: (.+);(.+?)(?:-metrics)?;(.+);(.+) selector: matchLabels: + stackable.tech/vendor: Stackable prometheus.io/scrape: "true" namespaceSelector: any: true @@ -132,7 +133,7 @@ spec: <2> Relabel \\__address__ to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary for scraping NiFi since it requires a DNS name to address the NiFi REST API <3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `-headless` Service, which is why we use the `-headless` Service instead of the `-metrics` Service for scraping NiFi metrics -NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-headless` thus we need to regex this suffix. +NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-metrics` thus we need to regex this suffix. The described example is part of the https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus.yaml[Prometheus,window=_blank] and https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[ServiceMonitor,window=_blank] manifests From a153bf3d156ebba1aa0be4fa82e0991f99f05e33 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:45:44 +0200 Subject: [PATCH 03/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index f1530025..513136df 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -36,7 +36,7 @@ spec: == Configure metrics in NiFi `2.x.x` The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507,window=_blank]. -Metrics are now always exposed and can be scraped using the NiFi `-metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`. +Metrics are now always exposed and can be scraped using the NiFi `metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`. For a deployed NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under: From 789be8339f5a3b75dd0a2d1d8f2c6ec96aaa10df Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:46:15 +0200 Subject: [PATCH 04/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 513136df..2e83a3a7 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -45,7 +45,7 @@ https://simple-nifi-node-default-metrics..svc.cluster.local:8443/nifi ``` NOTE: The above URL connects to one of the Pods, reachable through the specified Service, therefore scraping metrics produced by that Pod only. -To scrape metrics from a particular Pod the FQDN of the Pod and the `-headless` Service need to be used. For example: `\https://simple-nifi-node-default-0.simple-nifi-node-default-headless..svc.cluster.local:8443/nifi-api/flow/metrics/prometheus` +To scrape metrics from a particular Pod, the FQDN of the Pod and the `headless` Service need to be used. For example: `\https://simple-nifi-node-default-0.simple-nifi-node-default-headless..svc.cluster.local:8443/nifi-api/flow/metrics/prometheus` IMPORTANT: If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized. From 2884dc7ad8a64904ce4ded49ec4ed4d7c6aa2ff0 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:47:02 +0200 Subject: [PATCH 05/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 2e83a3a7..e0fc9ebe 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -54,7 +54,7 @@ IMPORTANT: If NiFi is configured to do any user authentication, requests to the To authenticate against the NiFi `2.x.x` API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between Kubernetes Pods, check out the xref:home:secret-operator:index.adoc[Secret Operator documentation]. -For the following example, showing how to configure a Prometheus scraper for NiFi, we use the afore-mentioned method of configuring mTLS +The following example illustrates the configuration of a Prometheus scraper for NiFi, using the aforementioned method of configuring mTLS and utilizing the internally available `tls` xref:home:secret-operator:secretclass.adoc[SecretClass]. To generate a client certificate signed by the `tls` SecretClass CA trusted in NiFi, add the following `volume` and `volumeMount` From 146dd3879c492ad620e26eed73531d06073c0a15 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:47:28 +0200 Subject: [PATCH 06/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index e0fc9ebe..9bed59fd 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -60,7 +60,7 @@ and utilizing the internally available `tls` xref:home:secret-operator:secretcla To generate a client certificate signed by the `tls` SecretClass CA trusted in NiFi, add the following `volume` and `volumeMount` to the Prometheus Pod. -IMPORTANT: If the {prometheus-operator}[Prometheus Operator,window=_blank] is being used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the https://github.com/prometheus-operator/prometheus-operator/issues/7709[Prometheus Operator Issues,window=_blank]. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank]. +IMPORTANT: If the {prometheus-operator}[Prometheus Operator,window=_blank] is used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the https://github.com/prometheus-operator/prometheus-operator/issues/7709[Prometheus Operator Issues,window=_blank]. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank]. [source,yaml] ---- From a9574b8c7c4099d6874d13fc95e88e8a2bde8d06 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:48:04 +0200 Subject: [PATCH 07/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 9bed59fd..c4503411 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -87,7 +87,7 @@ prometheus: <1> mountPath: /stackable/tls # <4> ---- <1> This given configuration is set in the {prometheus-operator}docs/api-reference/api/#monitoring.coreos.com/v1.Prometheus[Prometheus resource,window=_blank] for the {prometheus-operator}[Prometheus Operator,window=_blank] -<2> We use the `tls` SecretClass here, shipped with the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well +<2> The `tls` SecretClass created by the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well <3> We add the `service=prometheus-kube-prometheus-prometheus` scope for the Prometheus Service to be included in the `subjectAlternateName` of the TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank] <4> The path where the mTLS certificates are mounted to inside the Prometheus Pod From 684375aa70994559242fedef82ec7cd9e88e65e7 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:48:37 +0200 Subject: [PATCH 08/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index c4503411..41bbe65b 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -88,7 +88,7 @@ prometheus: <1> ---- <1> This given configuration is set in the {prometheus-operator}docs/api-reference/api/#monitoring.coreos.com/v1.Prometheus[Prometheus resource,window=_blank] for the {prometheus-operator}[Prometheus Operator,window=_blank] <2> The `tls` SecretClass created by the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well -<3> We add the `service=prometheus-kube-prometheus-prometheus` scope for the Prometheus Service to be included in the `subjectAlternateName` of the TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank] +<3> The `service=prometheus-kube-prometheus-prometheus` scope is added to include the `subjectAlternateName` of the Prometheus Service in the generated TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank] <4> The path where the mTLS certificates are mounted to inside the Prometheus Pod If you want to use a `ServiceMonitor` you'd need to configure it as follows: From fc613475267c6c4240316cf76d9da117814b2762 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:48:55 +0200 Subject: [PATCH 09/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 41bbe65b..36236d06 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -89,7 +89,7 @@ prometheus: <1> <1> This given configuration is set in the {prometheus-operator}docs/api-reference/api/#monitoring.coreos.com/v1.Prometheus[Prometheus resource,window=_blank] for the {prometheus-operator}[Prometheus Operator,window=_blank] <2> The `tls` SecretClass created by the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well <3> The `service=prometheus-kube-prometheus-prometheus` scope is added to include the `subjectAlternateName` of the Prometheus Service in the generated TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank] -<4> The path where the mTLS certificates are mounted to inside the Prometheus Pod +<4> The path where the mTLS certificates are mounted inside the Prometheus Pod If you want to use a `ServiceMonitor` you'd need to configure it as follows: From 0e1db273d340c7f5ef1ab4179eaf67efc0629727 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:50:32 +0200 Subject: [PATCH 10/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 36236d06..7a60c1e7 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -108,11 +108,11 @@ spec: - path: /nifi-api/flow/metrics/prometheus port: https scheme: https - tlsConfig: <1> + tlsConfig: # <1> caFile: /stackable/tls/ca.crt certFile: /stackable/tls/tls.crt keyFile: /stackable/tls/tls.key - relabelings: <2> + relabelings: # <2> - sourceLabels: - __meta_kubernetes_pod_name - __meta_kubernetes_service_name From 547c0eb972bfdec66b0fd3c35fe08bc556365ca9 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:51:21 +0200 Subject: [PATCH 11/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 7a60c1e7..6873b384 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -129,7 +129,7 @@ spec: any: true jobLabel: app.kubernetes.io/instance ---- -<1> In the TLS configuration of the ServiceMonitor, specify the paths to the mTLS files mounted to the Prometheus Pod +<1> In the TLS configuration of the ServiceMonitor, specify the location of the cert and key files mounted into the Prometheus Pod <2> Relabel \\__address__ to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary for scraping NiFi since it requires a DNS name to address the NiFi REST API <3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `-headless` Service, which is why we use the `-headless` Service instead of the `-metrics` Service for scraping NiFi metrics From 493f14b85a229f2659a10ed98e77d8bb60f96705 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:52:05 +0200 Subject: [PATCH 12/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 6873b384..f554cfec 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -131,7 +131,7 @@ spec: ---- <1> In the TLS configuration of the ServiceMonitor, specify the location of the cert and key files mounted into the Prometheus Pod <2> Relabel \\__address__ to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary for scraping NiFi since it requires a DNS name to address the NiFi REST API -<3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `-headless` Service, which is why we use the `-headless` Service instead of the `-metrics` Service for scraping NiFi metrics +<3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `headless` Service, which is why we use the `headless` Service instead of the `metrics` Service to scrape NiFi metrics NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-metrics` thus we need to regex this suffix. From e2850324bb1f60389b71fc33aa2ed6176837773f Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 15:54:25 +0200 Subject: [PATCH 13/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index f554cfec..deb36264 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -137,4 +137,4 @@ NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP expo The described example is part of the https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus.yaml[Prometheus,window=_blank] and https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[ServiceMonitor,window=_blank] manifests -being used in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank] of the https://github.com/stackabletech/demos[`demos` repository,window=_blank]. +used in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[monitoring stack,window=_blank] of the https://github.com/stackabletech/demos[demos repository,window=_blank]. From 55dab63e5d50a650e4f28058f341a34459575ae9 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 16:49:45 +0200 Subject: [PATCH 14/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index deb36264..7d131a72 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -130,7 +130,7 @@ spec: jobLabel: app.kubernetes.io/instance ---- <1> In the TLS configuration of the ServiceMonitor, specify the location of the cert and key files mounted into the Prometheus Pod -<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary for scraping NiFi since it requires a DNS name to address the NiFi REST API +<2> Relabel `__address__` to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary to scrape NiFi, since it requires a DNS name to address the NiFi REST API <3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `headless` Service, which is why we use the `headless` Service instead of the `metrics` Service to scrape NiFi metrics NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-metrics` thus we need to regex this suffix. From 1ed137a12e69dfd5c8fd0f011eb657b2236d1169 Mon Sep 17 00:00:00 2001 From: Xenia Date: Wed, 6 Aug 2025 16:51:01 +0200 Subject: [PATCH 15/15] Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander --- docs/modules/nifi/pages/usage_guide/monitoring.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/monitoring.adoc b/docs/modules/nifi/pages/usage_guide/monitoring.adoc index 7d131a72..3a093f87 100644 --- a/docs/modules/nifi/pages/usage_guide/monitoring.adoc +++ b/docs/modules/nifi/pages/usage_guide/monitoring.adoc @@ -133,7 +133,7 @@ spec: <2> Relabel `__address__` to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary to scrape NiFi, since it requires a DNS name to address the NiFi REST API <3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `headless` Service, which is why we use the `headless` Service instead of the `metrics` Service to scrape NiFi metrics -NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-metrics` thus we need to regex this suffix. +NOTE: The SDP exposes a dedicated `metrics` Service since the xref:listener-operator:listener.adoc[Listener integration]. The described example is part of the https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus.yaml[Prometheus,window=_blank] and https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[ServiceMonitor,window=_blank] manifests