Add incremental embeddings mode to reduce costs

[mishig25]

Summary

• Adds incremental mode to populate-search-engine and add-gradio-docs commands
• Only processes new/changed documents by tracking document IDs in a HuggingFace dataset
• Reduces embedding inference costs by avoiding reprocessing unchanged content
• Workflow now defaults to incremental mode, with option for full rebuild when needed

How it works

Document IDs are generated deterministically: {library}-{page}-{sha256_hash_of_text[:8]}

In incremental mode:

1. Fetches existing document IDs from hf-doc-build/doc-builder-embeddings-tracker dataset
2. Reconstructs IDs for all current chunks from source docs (fast - no embedding needed)
3. Compares to find new/changed chunks (different hash = different content)
4. Generates embeddings only for new chunks
5. Uploads directly to main Meilisearch index (upsert behavior)
6. Updates tracker dataset with new IDs

Files

New files:

• migrations/init_embeddings_tracker.py - Initializes tracker dataset by reconstructing IDs from hf-doc-build/doc-build (no Meilisearch needed)
• src/doc_builder/embeddings_tracker.py - Helper functions for tracking document IDs

Modified files:

• src/doc_builder/commands/embeddings.py - Added --incremental and --hf_token flags to both commands
• src/doc_builder/build_embeddings.py - Updated add_gradio_docs to support incremental mode
• .github/workflows/populate_search_engine.yml - Default to incremental mode, with full_rebuild input option

Usage

Initialize tracker (one-time setup - already done):

uv run python migrations/init_embeddings_tracker.py --hf_token <token>

Incremental update (default in workflow):

uv run doc-builder populate-search-engine --incremental
uv run doc-builder add-gradio-docs --incremental

Full rebuild (when needed):

uv run doc-builder populate-search-engine
uv run doc-builder add-gradio-docs

Or trigger workflow with full_rebuild: true.

Setup required

• HF_TOKEN secret needs to be added to the repo for updating the tracker dataset

Test plan

• Run migration script to create initial tracker dataset (48,401 document IDs)
• Run workflow in incremental mode - verify it skips existing docs
• Modify a doc and run again - verify only the changed doc is processed
• Run with full_rebuild=true - verify full rebuild still works