Improve documentation (missing configs, etc.)#1843
Merged
Conversation
…c files Fix inaccurate default values, wrong config key names, and a nonexistent class reference in the documentation: - protocols: add missing "file" to default - redirections.allowed → http.allow.redirects (default false, not true) - fetchInterval.error: -1, not 44640 - cacheConfigParamName → robots.cache.spec - errorcacheConfigParamName → robots.error.cache.spec - http.accept and http.accept.language: add actual defaults - protocol.md.prefix: default is "protocol.", not empty - FrontierSpout → Spout (class doesn't exist)
…ss documentation
- Fix wrong package names: org.apache.storm.crawler → org.apache.stormcrawler
- Fix outdated import: backtype.storm.Config → org.apache.storm.Config
- Fix non-existent class reference: StatusStreamBolt → DummyIndexer
- Fix method signature to match actual code (MetadataTransfer.getMetaForOutlink)
- Fix broken internal links (StatusStream, Configuration, HTTPProtocol anchors)
- Fix Selenium protocol links pointing to core/ instead of external/selenium/
- Fix incorrect GitHub line number references for crawler-default.yaml
- Fix incomplete references ("See blog post", "See example", "See default")
- Fix Storm UI URL protocol: https → http
- Fix TikaParser → ParserBolt (actual class name)
- Remove undocumented/unimplemented http.store.responsetime config entry
- Replace unexplained %HEAP-MEM% placeholder with concrete 2g value
- Fix typos: anonynmous, asomething, extra commas, grammar
- Fix AsciiDoc link syntax in powered-by.adoc
Configuration tables in configuration.adoc now cover all keys from crawler-default.yaml, including robots, protocol, OkHttp, parsing, sitemap, scheduling, and indexing options. Internals documentation now covers: - FeedParserBolt, URLPartitionerBolt, StatusEmitterBolt - FileSpout, MemorySpout, AbstractQueryingSpout - AdaptiveScheduler (with config example) - DelegatorProtocol (with config example) - SelfURLFilter Also fixes missing default values for indexer.text.fieldname and indexer.url.fieldname.
Add configuration tables for OpenSearch, Solr, SQL, URLFrontier, Tika, AWS, AI/LLM, Playwright, Language ID, WARC, and common spout options. Each module section includes all key config options with defaults and descriptions, plus links to the module source for full setup details. The Language ID module (which had no README) now has its parse filter configuration documented for the first time.
New extending.adoc covers: - Writing custom URL filters, parse filters, protocols, bolts/spouts with complete code examples and registration instructions - Politeness and rate limiting: queue modes, crawl-delay handling, robots.txt compliance - Error handling and retry logic: status lifecycle, retry mechanism, custom fetch intervals - Monitoring and metrics: fetcher metrics, MetricsConsumer setup - Scaling and tuning: thread config, queue sizing, connection pooling - Security: SSL/TLS, basic auth, proxy auth
rzo1
changed the title
jnioche
approved these changes
Mar 27, 2026
sebastian-nagel
approved these changes
Mar 27, 2026
Contributor
There was a problem hiding this comment.
Thanks, @rzo1! A great improvements.
See a few minor inline comments, mostly around protocol configurations.
But we can move these also to another issue / PR. I'll be also happy to change the docs directly.
Contributor
Author
|
thanks for reviewing. I have addressed the comments and added the suggest redirection section :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Comprehensive documentation audit and improvement across all StormCrawler docs.