Implement Codec Infrastructure

[dimitri-yatsenko]

Summary

This PR implements a comprehensive redesign of the custom type system, renaming "AttributeType/adapter" terminology to "Codec" and providing a cleaner, more intuitive API. It also simplifies the testing infrastructure with testcontainers.

Codec API Redesign

Key Changes

• Renamed AttributeType base class to Codec - The new name better reflects the purpose: encoding Python objects for database storage and decoding them on retrieval
• Auto-registration via __init_subclass__ - Codecs automatically register when their class is defined; no decorator needed
• New get_dtype(is_external) method - Codecs dynamically return their underlying storage type based on whether external storage is used
• Unified naming convention - All built-in codecs use <name> or <name@store> syntax consistently
• Renamed type category ADAPTED to CODEC in internal code

Built-in Codecs

Codec	Internal	External	Description
<blob>	bytes	<hash@>	DataJoint serialization for Python objects
<hash@>	N/A	json	Content-addressed storage with MD5 dedup
<object@>	N/A	json	Path-addressed storage for files/folders
<attach>	bytes	<hash@>	File attachments
<filepath@store>	N/A	json	Reference to existing external files

New Codec API

import datajoint as dj

class GraphCodec(dj.Codec):
    name = "graph"  # Auto-registers on class definition
    
    def get_dtype(self, is_external: bool) -> str:
        return "<blob>"  # Delegates serialization to blob codec
    
    def encode(self, graph, *, key=None, store_name=None):
        return {'nodes': list(graph.nodes()), 'edges': list(graph.edges())}
    
    def decode(self, stored, *, key=None):
        G = nx.Graph()
        G.add_nodes_from(stored['nodes'])
        G.add_edges_from(stored['edges'])
        return G

Testing Infrastructure: Testcontainers

Tests now use testcontainers to automatically manage MySQL and MinIO containers. No manual docker-compose up required.

New Developer Workflow

# Clone and install
git clone https://github.com/datajoint/datajoint-python.git
cd datajoint-python
pip install -e ".[test]"

# Run all tests - containers start automatically
pytest tests/

Benefits

• Zero setup - Just pip install and pytest
• Dynamic ports - No conflicts with other services
• Automatic cleanup - Containers stop when tests finish
• Simpler CI - No docker-compose orchestration needed

Fallback: External Containers

For development/debugging with persistent containers:

docker compose up -d db minio
DJ_USE_EXTERNAL_CONTAINERS=1 pytest tests/

Dead Code & Terminology Cleanup

Removed

• Backward compatibility aliases (AttributeType, register_type, list_types, get_type, etc.)
• Backward compatibility codec aliases (ObjectType, AttachType, XAttachType, FilepathType)
• Unused build_foreign_key_parser_old() function
• Legacy feature switches (ADAPTED_TYPE_SWITCH, FILEPATH_FEATURE_SWITCH)
• Unused enable_filepath_feature test fixture
• Misleading comments about non-existent DJBlobType/ContentType
• object-type-spec.md (implementation complete, info now in object.md)
• pytest-env dependency (testcontainers handles configuration)

Renamed

• Type category ADAPTED → CODEC in declare.py and heading.py
• Test files:
  • schema_adapted.py → schema_codecs.py
  • test_adapted_attributes.py → test_codecs.py
  • test_type_composition.py → test_codec_chaining.py
• Test classes and functions to use Codec terminology

Updated Terminology

• All comments/docstrings: "AttributeType" → "Codec", "Adapter" → "Codec"
• Fixed content_registry.py docstring: SHA256 → MD5

Documentation Updates

New Documentation

• codec-spec.md - Detailed API specification for creating custom codecs
• codecs.md - User guide with examples (replaces customtype.md)
• README.md - Comprehensive developer guide with test/pre-commit instructions

Updated Documentation

• mkdocs.yaml - Navigation updated: customtype.md → codecs.md
• attributes.md - Fixed dead links, updated terminology
• docker-compose.yaml - Clarified it's optional for tests

Removed Documentation

• object-type-spec.md (redundant with object.md)
• customtype.md (replaced by codecs.md)

Other Changes in This Branch

Settings System

• Simplified settings to pure Pydantic without backward compatibility shims
• Added recursive config file search and secrets separation
• Removed deprecated save_* methods and set_password function

Type System

• Added type aliases (int8, int16, int32, int64, uint8, etc.)
• Added decimal(n,f) to core types
• Renamed core type 'blob' to 'bytes' for cross-database portability
• Added text type and documented type modifier policy

External Storage

• Refactored external storage to use fsspec for unified backend support
• Implemented content registry with MD5 hashing (removed SHA256)
• Added garbage collection module for external storage cleanup
• Implemented <object@> type for managed file/folder storage

Infrastructure

• Dropped support for Python < 3.10 and MySQL < 8.0
• Reorganized tests into unit/ and integration/ directories
• Version bump to 2.0.0a7

Test Plan

• All 471 tests pass
• Verified codec auto-registration works correctly
• Verified external storage chain (<blob@> → <hash@> → storage) works
• Verified testcontainers starts/stops containers automatically
• Verified external container mode works (DJ_USE_EXTERNAL_CONTAINERS=1)
• Verified renamed test files execute correctly

🤖 Generated with Claude Code

[mweitzel]

Does it make sense to have any of the core datatypes support auto_increment? I would expect that, or at least to not generate a warning if used.

[dimitri-yatsenko]

auto_increment is not part of DataJoint's model for entity integrity. There are several reasons that are captured in the new docs but here is the main one (number 9).

9. Avoiding Identification System Design

Novice users reach for auto_increment as a shortcut to avoid the harder work of designing a proper identification system. They treat it as "row number" rather than "entity identifier"—conflating storage order with identity.

This reveals a conceptual gap: they're not thinking about what makes this entity unique in the real world, just how to get a number in the table. The result:

• Subjects identified by insertion order rather than lab naming conventions
• Sessions numbered globally rather than per-subject
• No ability to re-insert corrected data (new row = new ID)
• Foreign keys become meaningless integers rather than traceable lineage

DataJoint forces users to confront the question: "What uniquely identifies this entity in your domain?" This upfront design effort pays dividends in schema clarity, data integrity, and long-term maintainability.

---

The discomfort users feel when denied auto_increment is actually the discomfort of having to think carefully about their data model—which is precisely what they should be doing.

[dimitri-yatsenko]

This is very well described in the emerging docs in https://github.com/datajoint/datajoint-docs/

[mweitzel]

This document doesn't contain the words Object Augmented Schema, or the acronym OAS. Is that deliberate? I'd expect that to be grep-able.=

[dimitri-yatsenko]

yep, this is moved into datajoint-docs in #1305. datajoint-python will no longer contain the specs.

[mweitzel]

should this raise on successive calls if it fails on the initial call?

[dimitri-yatsenko]

It will raise an error even on the first time in the calling context if the plugin they are looking for is not registered.

[dimitri-yatsenko]

fixing..

[dimitri-yatsenko]

fixed 736860e

[mweitzel]

What is the default store? That is not really described anywhere.

[dimitri-yatsenko]

The settings structure provides a separate section for the default store and named stores. The default store has no name. I will be sure that the how-tos cover it well in the docs.

[mweitzel]

Why do some raise NotImplementedError where others are marked as @abstractmethod