Add SQL Server OTel input package#17429
Conversation
Add SQL Server OpenTelemetry input package (sql_server_input_otel). WHAT: New input package that enables collection of SQL Server metrics using the OpenTelemetry Collector's sqlserverreceiver. The package provides configuration support for the receiver: connection settings (server, port, username, password), instance settings (instance_name, computer_name), collection settings (interval, initial delay), query sample collection, and top query collection. Includes Docker test environment (MSSQL 2019), policy tests, system test (skipped for CI), input template (input.yml.hbs), manifest, docs, and changelog. WHY: To allow users to collect SQL Server metrics via the Elastic Agent with the EDOT Collector using the standard OpenTelemetry SQL Server receiver, with a supported input package. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ Vale Linting ResultsNo issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
The package looks well-structured and follows the OTel input package conventions. The linter passes and policy test configurations appear correct.
Two minor items flagged via inline comments:
- Changelog link should reference this PR (#17429), not the related issue
- System test config may need the
service:field for proper docker-compose integration
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
|
@claude address pr review comments and vale linting suggestions. Include the package and codeowner into the .CODEOWNERS file |
|
Claude finished @teresaromero's task in 1m 21s —— View job Addressing PR Review Comments and Vale LintingAll requested changes have been addressed and pushed:
Changes committed in Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
- Fix Vale linting: replace "in order to" with "to" in README docs - Update changelog link to reference PR #17429 instead of issue #17152 - Add service field to system test config for docker-compose integration - Add CODEOWNERS entry for sql_server_input_otel package (@elastic/ecosystem) Co-authored-by: Tere <teresaromero@users.noreply.github.com>
- Fix Vale linting: replace "in order to" with "to" in README docs - Update changelog link to reference PR elastic#17429 instead of issue elastic#17152 - Add service field to system test config for docker-compose integration - Add CODEOWNERS entry for sql_server_input_otel package (@elastic/ecosystem) Co-authored-by: Tere <teresaromero@users.noreply.github.com>
- Updated the description to include log collection capabilities. - Bumped Kibana version requirement from 9.2.0 to 9.4.0. - Added configuration options for enabling query sample and top query events. - Updated policy templates and test configurations to support new log event features. - Enhanced documentation to reflect the new log collection capabilities. Co-authored-by: Tere <teresaromero@users.noreply.github.com>
There was a problem hiding this comment.
The elastic-package lint command fails due to a mismatch between docs/README.md and its source file _dev/build/docs/README.md. The Logs reference section was added to docs/README.md directly, but it should be in _dev/build/docs/README.md (the source), then regenerated via elastic-package build.
After applying the suggested fix, run:
elastic-package buildto regenerate docs/README.md from the source template.
Note: The unresolved thread on input.yml.hbs regarding logs pipeline appears to be addressed—the logs pipeline is now included in both the template and expected outputs.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
- Added dynamic signal types to the manifest for improved configuration. - Updated policy tests to include new log statement configurations for SQL Server metrics. - Enhanced expected output permissions to accommodate log collection capabilities. Co-authored-by: Tere <teresaromero@users.noreply.github.com>
There was a problem hiding this comment.
In addition to the existing unresolved thread about the missing Logs reference section in _dev/build/docs/README.md, I found a build-blocking issue:
The dynamic_signal_types: true field in the manifest is invalid. Running elastic-package build fails with:
Error: building package failed: invalid content found in built zip package: found 1 validation error:
1. file "manifest.yml" is invalid: field policy_templates.0: Additional property dynamic_signal_types is not allowed
This field does not exist in the package specification and is not used by any other package in the repository. It should be removed or replaced with valid fields.
Note: The @mrodm comment about logs support in input.yml.hbs appears to have been addressed — the logs pipeline is now included in lines 39-41.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
andrewkroh
added
New Integration
There was a problem hiding this comment.
The two critical blocking issues from prior reviews remain unresolved:
-
manifest.yml:30 -
dynamic_signal_types: trueis not a valid field in the package specification.elastic-package buildfails with:Additional property dynamic_signal_types is not allowed. Remove this field. -
_dev/build/docs/README.md - The Logs reference section exists in
docs/README.mdbut is missing from the source file_dev/build/docs/README.md. Copy the Logs reference section to the source file, then runelastic-package buildto regenerate.
Note: The review thread on input.yml.hbs regarding logs support appears to have been addressed—the logs pipeline is now included (lines 39-41). Consider replying to that thread to mark it resolved.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
…-server-input-otel
… options and documentation updates - Updated manifest to include additional settings for query sample and top query collection, specifying their applicability. - Added metrics configuration options to the input YAML template. - Expanded documentation to detail the new log event types and their configurations. - Introduced expected test files for scenarios with metrics enabled.
…-server-input-otel
…tation. Updated policy templates to include new 'datasource' field for connection strings, improved descriptions for connection settings, and added resource attributes configuration. Updated README to reflect new features and known limitations regarding resource attributes. Adjusted Dockerfile healthcheck command for better error handling. Added new test cases for datasource configuration and default variables.
- Introduced `init.sh` for database setup and initialization. - Added `workload.sh` for generating database workload. - Updated `Dockerfile` to include new scripts and modified healthcheck logic. - Enhanced `docker-compose.yml` to define a new service for workload generation with health checks. - Created `test-logs-config.yml` for system testing with relevant configurations.
There was a problem hiding this comment.
Found one issue that will cause incorrect runtime behavior:
top_query_lookback_time type mismatch: The manifest defines this as type: integer with default 60, but the upstream SQL Server receiver's lookback_time field is time.Duration. When parsed, integer 60 will be interpreted as 60 nanoseconds, not 60 seconds. Change to type: duration with default 60s.
Note: The expected test outputs will also need updating after this fix.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
…Changed wording from "disabled by default" to "turned off by default" for better clarity.
…configurations. Adjusted component IDs for logs and metrics to ensure correct mapping in policy templates.
|
ping to @elastic/obs-infraobs-integrations so they can take a look |
| @@ -0,0 +1,13 @@ | |||
| service: sql_server_input_otel | |||
| vars: | |||
| datasource: "sqlserver://SA:1234_asdf@{{Hostname}}:{{Port}}" | |||
There was a problem hiding this comment.
As this field could contain potentially the password in the string, should it be considered as a possible field to be secret too?
| @@ -0,0 +1,173 @@ | |||
| format_version: 3.6.0 | |||
There was a problem hiding this comment.
As a note, no stable versions (1.x.y) of this package could be released until this spec version as it is released. If it is tried to do so, this error will be raised when building the package:
package with GA version (1.0.0) is using an unreleased version of the spec (3.6.0-next)
…field in the manifest with a note on future changes regarding secret handling. Clarified connection requirements in the README to specify the use of either individual connection settings or a datasource connection string.
…from 1 to 10, enhancing test coverage for SQL Server Input.
💚 Build Succeeded
History
|
andrewkroh
added
the
Integration:sql_server_input_otel
|
Package sql_server_input_otel - 0.1.0 containing this change is available at https://epr.elastic.co/package/sql_server_input_otel/0.1.0/ |
* Add SQL Server OTel input package Add SQL Server OpenTelemetry input package (sql_server_input_otel). WHAT: New input package that enables collection of SQL Server metrics using the OpenTelemetry Collector's sqlserverreceiver. The package provides configuration support for the receiver: connection settings (server, port, username, password), instance settings (instance_name, computer_name), collection settings (interval, initial delay), query sample collection, and top query collection. Includes Docker test environment (MSSQL 2019), policy tests, system test (skipped for CI), input template (input.yml.hbs), manifest, docs, and changelog. WHY: To allow users to collect SQL Server metrics via the Elastic Agent with the EDOT Collector using the standard OpenTelemetry SQL Server receiver, with a supported input package. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review comments and Vale linting suggestions - Fix Vale linting: replace "in order to" with "to" in README docs - Update changelog link to reference PR elastic#17429 instead of issue elastic#17152 - Add service field to system test config for docker-compose integration - Add CODEOWNERS entry for sql_server_input_otel package (@elastic/ecosystem) Co-authored-by: Tere <teresaromero@users.noreply.github.com> * Enhance SQL Server OpenTelemetry input package - Updated the description to include log collection capabilities. - Bumped Kibana version requirement from 9.2.0 to 9.4.0. - Added configuration options for enabling query sample and top query events. - Updated policy templates and test configurations to support new log event features. - Enhanced documentation to reflect the new log collection capabilities. Co-authored-by: Tere <teresaromero@users.noreply.github.com> * Enhance SQL Server OpenTelemetry input package with log support - Added dynamic signal types to the manifest for improved configuration. - Updated policy tests to include new log statement configurations for SQL Server metrics. - Enhanced expected output permissions to accommodate log collection capabilities. Co-authored-by: Tere <teresaromero@users.noreply.github.com> * Update SQL Server OpenTelemetry input package manifest to version 3.6.0 * Enhance SQL Server OpenTelemetry input package with new configuration options and documentation updates - Updated manifest to include additional settings for query sample and top query collection, specifying their applicability. - Added metrics configuration options to the input YAML template. - Expanded documentation to detail the new log event types and their configurations. - Introduced expected test files for scenarios with metrics enabled. * Enhance SQL Server Input integration by updating manifest and documentation. Updated policy templates to include new 'datasource' field for connection strings, improved descriptions for connection settings, and added resource attributes configuration. Updated README to reflect new features and known limitations regarding resource attributes. Adjusted Dockerfile healthcheck command for better error handling. Added new test cases for datasource configuration and default variables. * Add SQL Server Input workload and initialization scripts - Introduced `init.sh` for database setup and initialization. - Added `workload.sh` for generating database workload. - Updated `Dockerfile` to include new scripts and modified healthcheck logic. - Enhanced `docker-compose.yml` to define a new service for workload generation with health checks. - Created `test-logs-config.yml` for system testing with relevant configurations. * Update SQL Server Input documentation to clarify log event settings. Changed wording from "disabled by default" to "turned off by default" for better clarity. * Update SQL Server Input expected test files to swap metrics and logs configurations. Adjusted component IDs for logs and metrics to ensure correct mapping in policy templates. * Update sample_event.json for SQL Server Input * Update SQL Server Input manifest and documentation. Added a 'secret' field in the manifest with a note on future changes regarding secret handling. Clarified connection requirements in the README to specify the use of either individual connection settings or a datasource connection string. * Update test-logs-config.yml to increase minimum count for assertions from 1 to 10, enhancing test coverage for SQL Server Input. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Tere <teresaromero@users.noreply.github.com>
4 participants
Proposed commit message
Add SQL Server OpenTelemetry input package (
sql_server_input_otel).WHAT: New input package that enables collection of SQL Server metrics and logs using the OpenTelemetry Collector's sqlserverreceiver. The package provides configuration for:
datasource) as an alternativeinstance_name,computer_namedb.server.query_sample) with configurablemax_rows_per_querydb.server.top_query) with configurable lookback time, max sample count, query count, and collection intervalIncludes Docker test environment (MSSQL 2019 with workload generator), 5 policy test scenarios (default, default-vars, datasource, logs-enabled, metrics-enabled), 3 system test configs (default, datasource, logs), input template (
input.yml.hbs) with both metrics and logs pipelines, manifest (format_version: 3.6.0, Kibana^9.4.0), docs, changelog, and CODEOWNERS entry.WHY: To allow users to collect SQL Server metrics and logs via the Elastic Agent with the EDOT Collector using the standard OpenTelemetry SQL Server receiver, with a supported input package.
Checklist
changelog.ymlfile.Author's Checklist
elastic-package lintpasses locally@elastic/ecosystemelastic-packageHow to test this PR locally
elastic-package lintfrom the package directory:Related issues
Screenshots