Skip to content

Commit d66f76e

Browse files
committed
Merge claude/add-file-column-type-LtXQt into upgrade-adapted-type
2 parents 106f859 + b2bc219 commit d66f76e

23 files changed

+3869
-170
lines changed

docs/src/client/settings.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,3 +164,57 @@ Configure external stores in the `stores` section. See [External Storage](../sys
164164
}
165165
}
166166
```
167+
168+
## Object Storage
169+
170+
Configure object storage for the [`object` type](../design/tables/object.md) in the `object_storage` section. This provides managed file and folder storage with fsspec backend support.
171+
172+
### Local Filesystem
173+
174+
```json
175+
{
176+
"object_storage": {
177+
"project_name": "my_project",
178+
"protocol": "file",
179+
"location": "/data/my_project"
180+
}
181+
}
182+
```
183+
184+
### Amazon S3
185+
186+
```json
187+
{
188+
"object_storage": {
189+
"project_name": "my_project",
190+
"protocol": "s3",
191+
"bucket": "my-bucket",
192+
"location": "my_project",
193+
"endpoint": "s3.amazonaws.com"
194+
}
195+
}
196+
```
197+
198+
### Object Storage Settings
199+
200+
| Setting | Environment Variable | Required | Description |
201+
|---------|---------------------|----------|-------------|
202+
| `object_storage.project_name` | `DJ_OBJECT_STORAGE_PROJECT_NAME` | Yes | Unique project identifier |
203+
| `object_storage.protocol` | `DJ_OBJECT_STORAGE_PROTOCOL` | Yes | Backend: `file`, `s3`, `gcs`, `azure` |
204+
| `object_storage.location` | `DJ_OBJECT_STORAGE_LOCATION` | Yes | Base path or bucket prefix |
205+
| `object_storage.bucket` | `DJ_OBJECT_STORAGE_BUCKET` | For cloud | Bucket name |
206+
| `object_storage.endpoint` | `DJ_OBJECT_STORAGE_ENDPOINT` | For S3 | S3 endpoint URL |
207+
| `object_storage.partition_pattern` | `DJ_OBJECT_STORAGE_PARTITION_PATTERN` | No | Path pattern with `{attr}` placeholders |
208+
| `object_storage.token_length` | `DJ_OBJECT_STORAGE_TOKEN_LENGTH` | No | Random suffix length (default: 8) |
209+
| `object_storage.access_key` || For cloud | Access key (use secrets) |
210+
| `object_storage.secret_key` || For cloud | Secret key (use secrets) |
211+
212+
### Object Storage Secrets
213+
214+
Store cloud credentials in the secrets directory:
215+
216+
```
217+
.secrets/
218+
├── object_storage.access_key
219+
└── object_storage.secret_key
220+
```

docs/src/design/tables/attributes.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ together with DataJoint.
7676
NumPy arrays, dicts, lists, datetime objects, and nested structures. Compatible with
7777
MATLAB. See [custom types](customtype.md) for details.
7878

79+
- `object`: managed [file and folder storage](object.md) with support for direct writes
80+
(Zarr, HDF5) and fsspec integration. Recommended for new pipelines.
81+
7982
- `attach`: a [file attachment](attach.md) similar to email attachments facillitating
8083
sending/receiving an opaque data file to/from a DataJoint pipeline.
8184

@@ -94,6 +97,7 @@ libraries, making table definitions more readable and explicit about data precis
9497

9598
| Alias | MySQL Type | Description |
9699
|-------|------------|-------------|
100+
| `bool` | `tinyint` | Boolean value (0 or 1) |
97101
| `int8` | `tinyint` | 8-bit signed integer (-128 to 127) |
98102
| `uint8` | `tinyint unsigned` | 8-bit unsigned integer (0 to 255) |
99103
| `int16` | `smallint` | 16-bit signed integer (-32,768 to 32,767) |
@@ -117,6 +121,7 @@ class Measurement(dj.Manual):
117121
precise_value : float64 # double-precision measurement
118122
sample_count : uint32 # unsigned 32-bit counter
119123
sensor_flags : uint8 # 8-bit status flags
124+
is_valid : bool # boolean flag
120125
"""
121126
```
122127

0 commit comments

Comments
 (0)