feat: multi-backend fulltext search (Elastic, OpenSearch, Typesense)#10705
Open
dnplkndll wants to merge 5 commits intohcengineering:developfrom
Open
feat: multi-backend fulltext search (Elastic, OpenSearch, Typesense)#10705dnplkndll wants to merge 5 commits intohcengineering:developfrom
dnplkndll wants to merge 5 commits intohcengineering:developfrom
Conversation
|
Connected to Huly®: UBERF-16138 |
5 tasks
Add two new search backend adapters implementing FullTextAdapter: - @hcengineering/opensearch: Drop-in replacement using OpenSearch 2.x client. Removes deprecated type: _doc params, uses osErr error types. Fixes inherited bugs: updateMany error ID matching, remove() array mutation. - @hcengineering/typesense: Native Typesense adapter with proper query translation. Uses filter_by for workspace/class scoping, optional filter clauses for soft boosts (matching ES should semantics), read-modify-write for updateByQuery, and batch import for updateMany. Binary doc.data is skipped (relies on Rekoni extraction). Both adapters follow the same factory pattern as the existing elastic adapter and can be selected at runtime. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Don Kendall <kendall@donkendall.com>
Wire up the new OpenSearch and Typesense adapters in the fulltext pod. The FULLTEXT_BACKEND env var selects the adapter at startup: - "elastic" (default): existing Elasticsearch 7.x - "opensearch": OpenSearch 2.x (API-compatible drop-in) - "typesense": Typesense (lightweight C++ engine) Register both new packages in rush.json and add them as fulltext pod dependencies. ELASTIC_INDEX_NAME now defaults to 'huly_storage_index' instead of requiring it for all backends. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Don Kendall <kendall@donkendall.com>
The script only read version.txt when git describe succeeded, falling back to hardcoded "0.6.0" when no tags exist (e.g. in forks). This caused the model version to be 0.6.0 instead of the value in version.txt, making the fulltext indexer skip all workspaces. Now reads version.txt first, falls back to git tags, then "0.6.0". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Don Kendall <kendall@donkendall.com>
- Increase query_by_weights to 100,100,1 (matching ES boost ratios) - Enable num_typos: 2,2,1 for broader fuzzy matching - Add typo_tokens_threshold: 1 and drop_tokens_threshold: 1 - Use text_match_info.best_field_score for more granular scoring - Add sanitizeDoc() to coerce dynamic *_fields to string arrays (fixes 5 import errors from searchIcon_fields type mismatch) Search comparison results after tuning: - ES↔OpenSearch: 100% identical on all queries - ES↔Typesense: same top results, 5/5 overlap on class filter, 3/5 on prefix/fuzzy (different ranking, same docs found) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Don Kendall <kendall@donkendall.com>
When the fulltext pod starts and the Typesense collection has 0 documents (e.g. after pod restart with emptyDir), automatically enumerate all active workspaces and queue a fullReindex event for each. This makes Typesense self-healing without requiring a PVC. - Add optional getDocCount() to FullTextAdapter interface - Implement getDocCount() in TypesenseAdapter via collections().retrieve() - Add checkAndTriggerReindex() to WorkspaceManager.startIndexer() - Elastic/OpenSearch backends skip the check (no getDocCount method) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Don Kendall <kendall@donkendall.com>
dnplkndll
force-pushed
the
feat/multi-search-backend
branch
from
5b6f894 to
2cf6709
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add configurable fulltext search backend with three adapters and self-healing for non-persistent backends.
New packages
@hcengineering/opensearch— Drop-in OpenSearch 2.x replacement for ES 7.x (Apache 2.0 licensed, same wire protocol)@hcengineering/typesense— Lightweight C++ search engine (~3-5x less RAM, no JVM)Fulltext pod changes
FULLTEXT_BACKENDenv var selects adapter at runtime:elastic(default) |opensearch|typesensefullReindexfor each. Elastic/OpenSearch skip this check (persistent storage).getDocCount()method onFullTextAdapterinterface enables the detection.Build fix
show_version.js— readsversion.txtbefore git tags. Previously fell back to hardcoded0.6.0when no git tags exist (forks), causing model version mismatch that blocked the indexer entirely.Search Comparison Results
Tested with 565 docs, 7 query types across all 3 backends:
OpenSearch is a perfect drop-in (identical results). Typesense finds the same relevant documents with minor ranking differences (different scoring model).
Resource Comparison (production verified)
Commits
@hcengineering/opensearchand@hcengineering/typesensepackages implementingFullTextAdaptersanitizeDoc()for dynamic fieldsgetDocCount()on adapter +checkAndTriggerReindex()in managerBackward Compatibility
elastic— zero changes for existing deploymentsFullTextAdapterinterface change is additive only (getDocCountis optional)Related
Test plan
rush buildpassesFULLTEXT_BACKEND=invalidexits with clear error message🤖 Generated with Claude Code