Skip to content

fead: add OpenTelemetry observability support for modular inputs#450

Open
wtobis-splunk wants to merge 10 commits intodevelopfrom
feat/add-observability
Open

fead: add OpenTelemetry observability support for modular inputs#450
wtobis-splunk wants to merge 10 commits intodevelopfrom
feat/add-observability

Conversation

@wtobis-splunk
Copy link
Contributor

@wtobis-splunk wtobis-splunk commented Mar 11, 2026

Issue number
ADDON-85636

Summary
This PR introduces a new solnlib.observability module that adds an ObservabilityService for Splunk modular inputs and a LoggerMetricExporter for local/debug metric export. The service initializes standard event count and event byte counters, reads TA metadata from app.conf, attempts secure OTLP export to the local Spotlight collector with env/IPC-based port discovery, and degrades gracefully when observability is unavailable.

It also adds unit tests covering exporter behavior and OTLP setup/fallback paths, exports the new module through solnlib.__init__, and updates project dependencies and MkDocs references so the new observability API is documented and packaged correctly, and formally drops Python 3.7 support from the package configuration.

Testing

  • Manual testing on MSCS
  • Unit tests added

@@ -0,0 +1,534 @@
#
# Copyright 2025 Splunk Inc.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Copyright 2025 Splunk Inc.
# Copyright 2026 Splunk Inc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, updated to 2026.

hec_config,
log,
net_utils,
observability,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking about this import. It means that observability will be imported even if it is not used.

I checked the otel modules and importing them takes around 0,5-0,6 sec.

It's just a note. If that's ok, then we can leave it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — removed observability from __init__.py to avoid the ~0.5s import overhead for users who don't need it. Users should import directly via from solnlib.observability import ....

from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

_Logger: TypeAlias = Union[logging.Logger, logging.LoggerAdapter]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_Logger: TypeAlias = Union[logging.Logger, logging.LoggerAdapter]
_Logger = Union[logging.Logger, logging.LoggerAdapter]

What about not adding a type to _Logger, in order to avoid typing_extensions? If we want to leave it, maybe we should add it as a dependency in pyproject.toml?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed typing_extensions dependency and TypeAlias annotation — using plain _Logger = Union[...] instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants