Skip to content

Commit d36739d

Browse files
Use angle brackets for all AttributeTypes in definitions
All AttributeTypes (Layer 3) now use angle bracket syntax in table definitions: - Core types (Layer 2): int32, float64, varchar(255) - no brackets - AttributeTypes (Layer 3): <object>, <djblob>, <filepath@main> - angle brackets This clear visual distinction helps users immediately identify: - Core types: direct database mapping - AttributeTypes: encode/decode transformation Co-authored-by: dimitri-yatsenko <dimitri@datajoint.com>
1 parent 2a5d161 commit d36739d

File tree

1 file changed

+52
-47
lines changed

1 file changed

+52
-47
lines changed

docs/src/design/tables/storage-types-spec.md

Lines changed: 52 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ This document defines a three-layer type architecture:
1212
┌───────────────────────────────────────────────────────────────────┐
1313
│ AttributeTypes (Layer 3) │
1414
│ │
15-
│ Built-in: object content filepath@s <djblob> <xblob> │
16-
│ User: <custom> <mytype> ... │
15+
│ Built-in: <object> <content> <filepath@s> <djblob> <xblob>
16+
│ User: <custom> <mytype> ...
1717
├───────────────────────────────────────────────────────────────────┤
1818
│ Core DataJoint Types (Layer 2) │
1919
│ │
@@ -28,6 +28,10 @@ This document defines a three-layer type architecture:
2828
└───────────────────────────────────────────────────────────────────┘
2929
```
3030

31+
**Syntax distinction:**
32+
- Core types: `int32`, `float64`, `varchar(255)` - no brackets
33+
- AttributeTypes: `<object>`, `<djblob>`, `<filepath@main>` - angle brackets
34+
3135
### OAS Storage Regions
3236

3337
| Region | Path Pattern | Addressing | Use Case |
@@ -37,7 +41,7 @@ This document defines a three-layer type architecture:
3741

3842
### External References
3943

40-
`filepath@store` provides portable relative paths within configured stores with lazy ObjectRef access.
44+
`<filepath@store>` provides portable relative paths within configured stores with lazy ObjectRef access.
4145
For arbitrary URLs that don't need ObjectRef semantics, use `varchar` instead.
4246

4347
## Core DataJoint Types (Layer 2)
@@ -103,7 +107,7 @@ MySQL and PostgreSQL backends. Users should prefer these over native database ty
103107
AttributeTypes provide `encode()`/`decode()` semantics on top of core types. They are
104108
composable and can be built-in or user-defined.
105109

106-
### `object` / `object@store` - Path-Addressed Storage
110+
### `<object>` / `<object@store>` - Path-Addressed Storage
107111

108112
**Built-in AttributeType.** OAS (Object-Augmented Schema) storage:
109113

@@ -119,8 +123,8 @@ class Analysis(dj.Computed):
119123
definition = """
120124
-> Recording
121125
---
122-
results : object # default store
123-
archive : object@cold # specific store
126+
results : <object> # default store
127+
archive : <object@cold> # specific store
124128
"""
125129
```
126130

@@ -149,7 +153,7 @@ class ObjectType(AttributeType):
149153
)
150154
```
151155

152-
### `content` / `content@store` - Content-Addressed Storage
156+
### `<content>` / `<content@store>` - Content-Addressed Storage
153157

154158
**Built-in AttributeType.** Content-addressed storage with deduplication:
155159

@@ -208,7 +212,7 @@ class ContentType(AttributeType):
208212

209213
#### Database Column
210214

211-
The `content` type stores JSON metadata:
215+
The `<content>` type stores JSON metadata:
212216

213217
```sql
214218
-- content column (MySQL)
@@ -219,7 +223,7 @@ features JSON NOT NULL
219223
features JSONB NOT NULL
220224
```
221225

222-
### `filepath@store` - Portable External Reference
226+
### `<filepath@store>` - Portable External Reference
223227

224228
**Built-in AttributeType.** Relative path references within configured stores:
225229

@@ -236,9 +240,9 @@ updating data.
236240
```python
237241
class RawData(dj.Manual):
238242
definition = """
239-
session_id : int
243+
session_id : int32
240244
---
241-
recording : filepath@main # relative path within 'main' store
245+
recording : <filepath@main> # relative path within 'main' store
242246
"""
243247

244248
# Insert - user provides relative path within the store
@@ -254,13 +258,13 @@ ref.download('/local/path') # explicit download
254258
ref.open() # fsspec streaming access
255259
```
256260

257-
#### When to Use `filepath@store` vs `varchar`
261+
#### When to Use `<filepath@store>` vs `varchar`
258262

259263
| Use Case | Recommended Type |
260264
|----------|------------------|
261-
| Need ObjectRef/lazy access | `filepath@store` |
262-
| Need portability (relative paths) | `filepath@store` |
263-
| Want checksum verification | `filepath@store` |
265+
| Need ObjectRef/lazy access | `<filepath@store>` |
266+
| Need portability (relative paths) | `<filepath@store>` |
267+
| Want checksum verification | `<filepath@store>` |
264268
| Just storing a URL string | `varchar` |
265269
| External URLs you don't control | `varchar` |
266270

@@ -309,7 +313,7 @@ recording JSON NOT NULL
309313
recording JSONB NOT NULL
310314
```
311315

312-
#### Key Differences from Legacy `filepath@store`
316+
#### Key Differences from Legacy `filepath@store` (now `<filepath@store>`)
313317

314318
| Feature | Legacy | New |
315319
|---------|--------|-----|
@@ -334,7 +338,7 @@ column_name JSONB NOT NULL
334338
```
335339

336340
The `json` database type:
337-
- Used as dtype by built-in AttributeTypes (`object`, `content`, `filepath@store`)
341+
- Used as dtype by built-in AttributeTypes (`<object>`, `<content>`, `<filepath@store>`)
338342
- Stores arbitrary JSON-serializable data
339343
- Automatically uses appropriate type for database backend
340344
- Supports JSON path queries where available
@@ -354,15 +358,15 @@ class AttributeType:
354358

355359
**Resolution examples:**
356360
```
357-
<xblob> → uses content type → default store
358-
<xblob@cold> → uses content type → cold store
359-
<djblob> → dtype = "longblob" → database (no store)
360-
object@cold → uses object type → cold store
361+
<xblob> → uses <content> type → default store
362+
<xblob@cold> → uses <content> type → cold store
363+
<djblob> → dtype = "longblob" → database (no store)
364+
<object@cold> → uses <object> type → cold store
361365
```
362366

363367
AttributeTypes can use other AttributeTypes as their dtype (composition):
364-
- `<xblob>` uses `content` - adds djblob serialization on top of content-addressed storage
365-
- `<xattach>` uses `content` - adds filename preservation on top of content-addressed storage
368+
- `<xblob>` uses `<content>` - adds djblob serialization on top of content-addressed storage
369+
- `<xattach>` uses `<content>` - adds filename preservation on top of content-addressed storage
366370

367371
## User-Defined AttributeTypes
368372

@@ -480,17 +484,17 @@ class Attachments(dj.Manual):
480484

481485
| Type | dtype | Storage Location | Dedup | Returns |
482486
|------|-------|------------------|-------|---------|
483-
| `object` | `json` | `{schema}/{table}/{pk}/` | No | ObjectRef |
484-
| `object@s` | `json` | `{schema}/{table}/{pk}/` | No | ObjectRef |
485-
| `content` | `json` | `_content/{hash}` | Yes | bytes |
486-
| `content@s` | `json` | `_content/{hash}` | Yes | bytes |
487-
| `filepath@s` | `json` | Configured store (relative path) | No | ObjectRef |
487+
| `<object>` | `json` | `{schema}/{table}/{pk}/` | No | ObjectRef |
488+
| `<object@s>` | `json` | `{schema}/{table}/{pk}/` | No | ObjectRef |
489+
| `<content>` | `json` | `_content/{hash}` | Yes | bytes |
490+
| `<content@s>` | `json` | `_content/{hash}` | Yes | bytes |
491+
| `<filepath@s>` | `json` | Configured store (relative path) | No | ObjectRef |
488492
| `<djblob>` | `longblob` | Database | No | Python object |
489-
| `<xblob>` | `content` | `_content/{hash}` | Yes | Python object |
490-
| `<xblob@s>` | `content@s` | `_content/{hash}` | Yes | Python object |
493+
| `<xblob>` | `<content>` | `_content/{hash}` | Yes | Python object |
494+
| `<xblob@s>` | `<content@s>` | `_content/{hash}` | Yes | Python object |
491495
| `<attach>` | `longblob` | Database | No | Local file path |
492-
| `<xattach>` | `content` | `_content/{hash}` | Yes | Local file path |
493-
| `<xattach@s>` | `content@s` | `_content/{hash}` | Yes | Local file path |
496+
| `<xattach>` | `<content>` | `_content/{hash}` | Yes | Local file path |
497+
| `<xattach@s>` | `<content@s>` | `_content/{hash}` | Yes | Local file path |
494498

495499
## Reference Counting for Content Type
496500

@@ -539,8 +543,8 @@ def garbage_collect(project):
539543

540544
## Built-in AttributeType Comparison
541545

542-
| Feature | `object` | `content` | `filepath@store` |
543-
|---------|----------|-----------|------------------|
546+
| Feature | `<object>` | `<content>` | `<filepath@store>` |
547+
|---------|------------|-------------|---------------------|
544548
| dtype | `json` | `json` | `json` |
545549
| Location | OAS store | OAS store | Configured store |
546550
| Addressing | Primary key | Content hash | Relative path |
@@ -552,9 +556,9 @@ def garbage_collect(project):
552556
| Integrity | DataJoint managed | DataJoint managed | User managed |
553557

554558
**When to use each:**
555-
- **`object`**: Large/complex objects where DataJoint controls organization (Zarr, HDF5)
556-
- **`content`**: Deduplicated serialized data or file attachments via `<xblob>`, `<xattach>`
557-
- **`filepath@store`**: Portable references to files in configured stores
559+
- **`<object>`**: Large/complex objects where DataJoint controls organization (Zarr, HDF5)
560+
- **`<content>`**: Deduplicated serialized data or file attachments via `<xblob>`, `<xattach>`
561+
- **`<filepath@store>`**: Portable references to files in configured stores
558562
- **`varchar`**: Arbitrary URLs/paths where ObjectRef semantics aren't needed
559563

560564
## Key Design Decisions
@@ -564,20 +568,21 @@ def garbage_collect(project):
564568
- Layer 2: Core DataJoint types (standardized, scientist-friendly)
565569
- Layer 3: AttributeTypes (encode/decode, composable)
566570
2. **Core types are scientist-friendly**: `float32`, `uint8`, `bool` instead of `FLOAT`, `TINYINT UNSIGNED`, `TINYINT(1)`
567-
3. **AttributeTypes are composable**: `<xblob>` uses `content`, which uses `json`
568-
4. **Built-in AttributeTypes use JSON dtype**: Stores metadata (path, hash, store name, etc.)
569-
5. **Two OAS regions**: object (PK-addressed) and content (hash-addressed) within managed stores
570-
6. **Filepath for portability**: `filepath@store` uses relative paths within stores for environment portability
571-
7. **No `uri` type**: For arbitrary URLs, use `varchar`—simpler and more transparent
572-
8. **Content type**: Single-blob, content-addressed, deduplicated storage
573-
9. **Parameterized types**: `type@param` passes store parameter
574-
10. **Naming convention**:
571+
3. **AttributeTypes use angle brackets**: `<object>`, `<djblob>`, `<filepath@store>` - distinguishes from core types
572+
4. **AttributeTypes are composable**: `<xblob>` uses `<content>`, which uses `json`
573+
5. **Built-in AttributeTypes use JSON dtype**: Stores metadata (path, hash, store name, etc.)
574+
6. **Two OAS regions**: object (PK-addressed) and content (hash-addressed) within managed stores
575+
7. **Filepath for portability**: `<filepath@store>` uses relative paths within stores for environment portability
576+
8. **No `uri` type**: For arbitrary URLs, use `varchar`—simpler and more transparent
577+
9. **Content type**: Single-blob, content-addressed, deduplicated storage
578+
10. **Parameterized types**: `<type@param>` passes store parameter
579+
11. **Naming convention**:
575580
- `<djblob>` = internal serialized (database)
576581
- `<xblob>` = external serialized (content-addressed)
577582
- `<attach>` = internal file (single file)
578583
- `<xattach>` = external file (single file)
579-
11. **Transparent access**: AttributeTypes return Python objects or file paths
580-
12. **Lazy access**: `object`, `object@store`, and `filepath@store` return ObjectRef
584+
12. **Transparent access**: AttributeTypes return Python objects or file paths
585+
13. **Lazy access**: `<object>`, `<object@store>`, and `<filepath@store>` return ObjectRef
581586

582587
## Migration from Legacy Types
583588

0 commit comments

Comments
 (0)