Implement Semantic Joins

[dimitri-yatsenko]

Summary

This PR implements semantic matching for joins in DataJoint 2.0. Instead of matching attributes purely by name (natural join), DataJoint now tracks attribute lineage (origin) and only allows joins on attributes that share both the same name AND the same lineage.

📄 Full Specification - API reference, user guide, and implementation details

Key Changes

• Lineage tracking: Each attribute now has a lineage property indicating its origin (schema.table.attribute)
• ~lineage table: Hidden per-schema table storing lineage information, populated at table declaration time
• Semantic checks: Joins and restrictions error if namesake attributes have different lineages
• Schema methods: schema.rebuild_lineage() to restore lineage for legacy schemas, schema.lineage_table_exists property
• Removed operators: @ (permissive join) and ^ (permissive restriction) replaced by .join(semantic_check=False) and .restrict(semantic_check=False)
• dj.U * table removed: Join with universal set is no longer supported

New Files

• src/datajoint/lineage.py - Lineage management module
• tests/integration/test_semantic_matching.py - 21 comprehensive tests
• docs/src/design/semantic-matching-spec.md - Full specification with API reference and user guide

Modified Files

• condition.py - assert_join_compatibility() with semantic checking
• expression.py - Updated join/restrict methods, removed @/^ operators
• heading.py - lineage_available property, lineage loading from ~lineage table
• table.py - _populate_lineage() at declaration, cleanup at drop
• declare.py - FK attribute mapping for lineage tracking
• schemas.py - rebuild_lineage() method, lineage_table_exists property

Behavior Summary

Scenario	Action
Same name, same lineage	Match (join proceeds)
Same name, different lineage	Error
Same name, either lineage null	Error
~lineage table missing	Warning + skip semantic check

Migration for Users

# Removed operators
A @ B                    # Use: A.join(B, semantic_check=False)
A ^ B                    # Use: A.restrict(B, semantic_check=False)
dj.U('a') * B            # Removed (no longer needed with semantic matching)

# Rebuild lineage for legacy schemas
schema.rebuild_lineage()  # Then restart kernel

Test Plan

• All 21 new semantic matching tests pass
• Existing relational operand tests pass (63 tests)
• Pre-commit hooks pass (ruff, codespell, formatting)

🤖 Generated with Claude Code