Skip to content

#585 Add Storm V2 metrics support with backward-compatible bridge#1846

Open
rzo1 wants to merge 1 commit intomainfrom
585
Open

#585 Add Storm V2 metrics support with backward-compatible bridge#1846
rzo1 wants to merge 1 commit intomainfrom
585

Conversation

@rzo1
Copy link
Copy Markdown
Contributor

@rzo1 rzo1 commented Mar 27, 2026

Thank you for contributing to Apache StormCrawler.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes

  • Is there a issue associated with this PR? Is it referenced in the commit message?

  • Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve?

  • Has your PR been rebased against the latest commit within the target branch (typically main)?

  • Is your initial contribution a single, squashed commit?

  • Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

  • Have you ensured that the full suite of tests is executed via mvn clean verify?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file?

Note

Introduce a CrawlerMetrics factory that routes metric registration to Storm V1, V2 (Codahale/Dropwizard), or both APIs based on the config property stormcrawler.metrics.version ("v1" default, "v2", "both"). This enables gradual migration from deprecated V1 metrics without breaking existing deployments or dashboards.

  • New metrics bridge infrastructure in core (ScopedCounter, ScopedReducedMetric interfaces with V1/V2/Dual implementations)
  • Migrated all bolt/spout metric registration across core and all external modules (opensearch, sql, solr, aws, tika, warc, urlfrontier)
  • Added V2 ScheduledStormReporter implementations for OpenSearch, SQL, and Solr that write the same document schema as V1 MetricsConsumer

@rzo1 rzo1 requested a review from jnioche March 27, 2026 18:09
@rzo1 rzo1 added this to the 3.5.2 milestone Mar 27, 2026
@rzo1 rzo1 force-pushed the 585 branch 3 times, most recently from b257bbc to 74598bd Compare March 27, 2026 19:01
Introduce a CrawlerMetrics factory that routes metric registration to
Storm V1, V2 (Codahale/Dropwizard), or both APIs based on the config
property `stormcrawler.metrics.version` ("v1" default, "v2", "both").
This enables gradual migration from deprecated V1 metrics without
breaking existing deployments or dashboards.

- New metrics bridge infrastructure in core (ScopedCounter,
  ScopedReducedMetric interfaces with V1/V2/Dual implementations)
- Migrated all bolt/spout metric registration across core and all
  external modules (opensearch, sql, solr, aws, tika, warc, urlfrontier)
- Added V2 ScheduledStormReporter implementations for OpenSearch, SQL,
  and Solr that write the same document schema as V1 MetricsConsumer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant