Preprocessing enhancements: Safety assessment, Language detection #101

GFJHogue · 2026-01-20T21:22:03Z

This PR replaces #99 & #100 and partially replaces #97

The BaseGraphBuilder.preprocess() step (available for all chat profiles) now includes safety check and language detection tasks.

The safety check adds a safety string field to BaseState, which is expected to be equal to "true" when the user's input is considered to be appropriate and within Reactome scope, as well as an additional reason_unsafe string field.
- The React-to-Me profile now implements conditional routing; only researching and answering "safe" questions, while "unsafe" questions bypass the usual workflow to generate a contextual refusal response.
The language detection adds a self-explanatory detected_language string field to BaseState.

Breaking change:

since the detected language isn't passed to the RAG answerer and the rephrased question (for the React-to-Me profile, at least) is supposed to always be in English (for retrieval), then informed responses will always be in English.
- if detected_language isn't English, we need to translate the English RAG output and stream it instead

- Implement parallel execution of safety and scope check, query expansion, and language detection

… expansion and conversation history management

- Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance

… expansion and conversation history management

- Add type annotation for rrf_scores in retrieval_utils.py - Fix metadata dictionary comprehension in csv_chroma.py - Update retriever type annotations to use Any - Add isinstance check for BM25Retriever - Remove default values from TypedDict in base.py - Fix TypedDict expansion in postprocess method

- Implement parallel execution of safety and scope check, query expansion, and language detection

- Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance

…ination mitigation

* rewrite HybridRetriever class * fix types * fix HybridRetriever class inheritance issues * fix lint * multithread, as in Helia's code

…ss (#102) * feat: expand preprocessing to a multi-step workflow. - Implement parallel execution of safety and scope check, query expansion, and language detection * feat: Add new runnables for checking question safety and scope, query expansion and conversation history management * feat:improved hybrid retrieval - Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance * feat: Add new runnables for checking question safety and scope, query expansion and conversation history management * code quality check fixes * fix: Resolve mypy linter errors - Add type annotation for rrf_scores in retrieval_utils.py - Fix metadata dictionary comprehension in csv_chroma.py - Update retriever type annotations to use Any - Add isinstance check for BM25Retriever - Remove default values from TypedDict in base.py - Fix TypedDict expansion in postprocess method * remove: Remove reactome_kg directory from repository * feat: expand preprocessing to a multi-step workflow. - Implement parallel execution of safety and scope check, query expansion, and language detection * feat: expand preprocessing to a multi-step workflow. - Implement parallel execution of safety and scope check, query expansion, and language detection * feat:improved hybrid retrieval - Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance * feat:improved answer generation, in-line citation handling and hallucination mitigation * remove irrelevant docs * [WIP] clean up changes * [WIP] clean up changes (2) * revert retrieval changes * macos-intel actions runner * fix SafetyCheck type usage * stream unsafe response to user * black spacing * cross-db use detected_language from base state * pre-release docker push * new HybridRetriever class (#102) * rewrite HybridRetriever class * fix types * fix HybridRetriever class inheritance issues * fix lint * multithread, as in Helia's code * fix typing for #102 --------- Co-authored-by: Helia Mohammadi <helia.mohammadi01@gmail.com>

…ss (#102) * feat: expand preprocessing to a multi-step workflow. - Implement parallel execution of safety and scope check, query expansion, and language detection * feat: Add new runnables for checking question safety and scope, query expansion and conversation history management * feat:improved hybrid retrieval - Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance * feat: Add new runnables for checking question safety and scope, query expansion and conversation history management * code quality check fixes * fix: Resolve mypy linter errors - Add type annotation for rrf_scores in retrieval_utils.py - Fix metadata dictionary comprehension in csv_chroma.py - Update retriever type annotations to use Any - Add isinstance check for BM25Retriever - Remove default values from TypedDict in base.py - Fix TypedDict expansion in postprocess method * remove: Remove reactome_kg directory from repository * feat: expand preprocessing to a multi-step workflow. - Implement parallel execution of safety and scope check, query expansion, and language detection * feat: expand preprocessing to a multi-step workflow. - Implement parallel execution of safety and scope check, query expansion, and language detection * feat:improved hybrid retrieval - Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance * feat:improved answer generation, in-line citation handling and hallucination mitigation * remove irrelevant docs * [WIP] clean up changes * [WIP] clean up changes (2) * revert retrieval changes * macos-intel actions runner * fix SafetyCheck type usage * stream unsafe response to user * black spacing * cross-db use detected_language from base state * pre-release docker push * new HybridRetriever class (#102) * rewrite HybridRetriever class * fix types * fix HybridRetriever class inheritance issues * fix lint * multithread, as in Helia's code * fix typing for #102 --------- Co-authored-by: Helia Mohammadi <helia.mohammadi01@gmail.com> * revert pre-release branch stuff --------- Co-authored-by: Helia Mohammadi <helia.mohammadi01@gmail.com>

heliamoh and others added 17 commits September 27, 2025 17:51

feat: expnad preprocessing to a multi-step workflow.

2607236

- Implement parallel execution of safety and scope check, query expansion, and language detection

feat: Add new runnables for checking question safety and scope, query…

8b35029

… expansion and conversation history management

feat:improved hybrid retrieval

8b2578f

- Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance

feat: Add new runnables for checking question safety and scope, query…

b2cc4bb

… expansion and conversation history management

code quality check fixes

3b9e95d

remove: Remove reactome_kg directory from repository

f35f3e0

code quality fixes

ba01931

feat: expnad preprocessing to a multi-step workflow.

7f8d4c5

- Implement parallel execution of safety and scope check, query expansion, and language detection

feat: expnad preprocessing to a multi-step workflow.

67fcd60

- Implement parallel execution of safety and scope check, query expansion, and language detection

feat:improved hybrid retrieval

3ea2ba8

- Replace SelfQueryRetriever with efficient hybrid search (BM25 + vector) - Add RRF (Reciprocal Rank Fusion) support for query expansion - Implement parallel processing for improved performance

feat:improved answer generation, in-line citation handling and halluc…

5b82199

…ination mitigation

remove irrelevant docs

27e761c

[WIP] clean up changes

0f67db3

[WIP] clean up changes (2)

0f24963

revert retrieval changes

5ca38cf

spacing, macos-intel actions runner

02e7301

GFJHogue mentioned this pull request Jan 20, 2026

Safety preprocessing #100

Closed

GFJHogue self-assigned this Jan 20, 2026

GFJHogue mentioned this pull request Jan 22, 2026

React to me architectural upgrade - Advanced hybrid retrieval, preprocessing pipeline, and safety system #97

Closed

GFJHogue added 4 commits February 6, 2026 11:10

fix SafetyCheck type usage

337be49

stream unsafe response to user

4980410

black spacing

b202c3c

cross-db use detected_language from base state

9b01a48

GFJHogue changed the base branch from main to pre-release February 6, 2026 16:51

GFJHogue marked this pull request as ready for review February 6, 2026 16:52

GFJHogue requested review from adamjohnwright and heliamoh February 6, 2026 16:52

pre-release docker push

ebe6180

adamjohnwright approved these changes Feb 6, 2026

View reviewed changes

GFJHogue added 2 commits February 8, 2026 18:50

new HybridRetriever class (#102)

9232748

* rewrite HybridRetriever class * fix types * fix HybridRetriever class inheritance issues * fix lint * multithread, as in Helia's code

fix typing for #102

93644d5

GFJHogue merged commit 5dc7795 into pre-release Feb 9, 2026
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preprocessing enhancements: Safety assessment, Language detection #101

Preprocessing enhancements: Safety assessment, Language detection #101

Uh oh!

GFJHogue commented Jan 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Preprocessing enhancements: Safety assessment, Language detection #101

Preprocessing enhancements: Safety assessment, Language detection #101

Uh oh!

Conversation

GFJHogue commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking change:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GFJHogue commented Jan 20, 2026 •

edited

Loading