Merge pull request #665 from objectstack-ai/copilot/update-metadata-documentation

Author: hotlong

packages/metadata/README.md

@@ -1,13 +1,245 @@
 # @objectstack/metadata
 
-The **Model Definition** layer. It provides classes and utilities for parsing and validating ObjectStack metadata schemas.
+> **Metadata Loading, Persistence & Customization Layer for ObjectStack.**
 
-## Features
+`@objectstack/metadata` is the central service responsible for loading, validating, persisting and watching all metadata definitions (Objects, Views, Flows, Apps, Agents, etc.) in the ObjectStack platform.
 
-- **Schema Loading**: Load metadata from JSON/YAML or TypeScript files.
-- **Reference Resolution**: Resolves cross-object references and inheritance.
-- **Validation**: Strict validation against the Zod schemas in `@objectstack/spec`.
+It implements the **`IMetadataService`** contract from `@objectstack/spec` and acts as the single source of truth that all other packages depend on.
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     IMetadataService                        │
+│              (Contract: @objectstack/spec)                   │
+├─────────────────────────────────────────────────────────────┤
+│                     MetadataManager                         │
+│            (Orchestrator: this package)                      │
+│                                                             │
+│  ┌─────────────┐  ┌──────────────┐  ┌───────────────────┐  │
+│  │  In-Memory   │  │   Overlay    │  │  Type Registry    │  │
+│  │  Registry    │  │   System     │  │  & Dependencies   │  │
+│  └─────────────┘  └──────────────┘  └───────────────────┘  │
+├─────────────────────────────────────────────────────────────┤
+│                      Loader Layer                           │
+│  ┌─────────────┐  ┌──────────────┐  ┌───────────────────┐  │
+│  │ Filesystem   │  │   Remote     │  │   Memory          │  │
+│  │ Loader       │  │   Loader     │  │   Loader          │  │
+│  │ (files)      │  │   (HTTP)     │  │   (test)          │  │
+│  └─────────────┘  └──────────────┘  └───────────────────┘  │
+│  ┌──────────────────────────────────────────────────────┐   │
+│  │ DatabaseLoader (planned — datasource-backed storage) │   │
+│  └──────────────────────────────────────────────────────┘   │
+├─────────────────────────────────────────────────────────────┤
+│                    Serializer Layer                          │
+│  ┌──────────┐  ┌──────────┐  ┌──────────────────────────┐  │
+│  │   JSON   │  │   YAML   │  │   TypeScript/JavaScript  │  │
+│  └──────────┘  └──────────┘  └──────────────────────────┘  │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Core Concepts
+
+### 1. Metadata Sources (Three-Scope Model)
+
+ObjectStack adopts a three-scope layered model for metadata:
+
+| Scope      | Storage      | Mutability   | Description                                |
+|:-----------|:-------------|:-------------|:-------------------------------------------|
+| `system`   | Filesystem   | Read-only    | Defined in code, shipped with packages      |
+| `platform` | Database     | Admin-editable | Created/modified by admins via UI          |
+| `user`     | Database     | User-editable  | Personal customizations per user           |
+
+Resolution order: **system** ← merge(**platform**) ← merge(**user**).
+
+### 2. Loaders
+
+Loaders are pluggable data sources that know how to read/write metadata from different backends. Each loader declares a `MetadataLoaderContract` with name, protocol, and capabilities:
+
+| Loader              | Protocol       | Read | Write | Watch | Status       |
+|:--------------------|:---------------|:-----|:------|:------|:-------------|
+| `FilesystemLoader`  | `file:`        | ✅   | ✅    | ✅    | Implemented  |
+| `MemoryLoader`      | `memory:`      | ✅   | ✅    | ❌    | Implemented  |
+| `RemoteLoader`      | `http:`        | ✅   | ✅    | ❌    | Implemented  |
+| `DatabaseLoader`    | `datasource:`  | ✅   | ✅    | ✅    | Planned      |
+
+### 3. Serializers
+
+Serializers convert metadata objects to/from different file formats:
+
+- **JSONSerializer** — `.json` files with optional key sorting
+- **YAMLSerializer** — `.yaml`/`.yml` files (JSON_SCHEMA for security)
+- **TypeScriptSerializer** — `.ts`/`.js` module exports (for `defineObject()`, `defineView()`, etc.)
+
+### 4. Overlay / Customization System
+
+The overlay system enables non-destructive customizations on top of package-delivered (system) metadata, following a delta-based approach (JSON Merge Patch):
+
+- **getOverlay** / **saveOverlay** / **removeOverlay** — manage customization deltas
+- **getEffective** — returns the merged result of base + platform overlay + user overlay
+- Overlays never modify the base definition — they are additive patches
+
+### 5. MetadataManager (IMetadataService Implementation)
+
+The `MetadataManager` is the main orchestrator. It provides:
+
+- **Core CRUD**: `register`, `get`, `list`, `unregister`, `exists`, `listNames`
+- **Convenience**: `getObject`, `listObjects`
+- **Package Management**: `unregisterPackage` — unload all metadata from a package
+- **Query / Search**: `query` with filtering, pagination, sorting by type/scope/state/tags
+- **Bulk Operations**: `bulkRegister`, `bulkUnregister` with error handling
+- **Import / Export**: `exportMetadata`, `importMetadata` with conflict resolution (skip/overwrite/merge)
+- **Validation**: `validate` — structural validation of metadata items
+- **Type Registry**: `getRegisteredTypes`, `getTypeInfo` — discover available metadata types
+- **Dependency Tracking**: `getDependencies`, `getDependents` — cross-reference analysis
+- **Watch / Subscribe**: `watchService` — observe metadata changes in real-time
+- **Loader Delegation**: `load`, `loadMany`, `save` — delegate I/O to registered loaders
+
+### 6. NodeMetadataManager
+
+Extends `MetadataManager` with Node.js-specific capabilities:
+
+- Auto-configures `FilesystemLoader` for local development
+- File watching via **chokidar** for hot-reload during development
+- Detects file add/change/delete events and notifies subscribers
+
+### 7. MetadataPlugin
+
+Integrates with the ObjectStack kernel plugin system:
+
+- Registers as the primary `IMetadataService` provider
+- Auto-loads all metadata types from the filesystem on startup (sorted by `loadOrder`)
+- Supports YAML, JSON, TypeScript, and JavaScript metadata formats
+
+## Metadata Types
+
+The platform supports **26 built-in metadata types** across 6 protocol domains:
+
+| Domain       | Types                                                                       |
+|:-------------|:----------------------------------------------------------------------------|
+| **Data**     | `object`, `field`, `datasource`, `validation`                               |
+| **UI**       | `view`, `app`, `dashboard`, `report`, `action`, `theme`                     |
+| **Automation** | `flow`, `workflow`, `trigger`, `schedule`                                 |
+| **System**   | `manifest`, `translation`, `api`, `permission_set`, `role`, `profile`       |
+| **Security** | `permission_set`, `role`                                                    |
+| **AI**       | `agent`, `rag_pipeline`, `model`, `prompt`, `tool`                          |
+
+Each type has a defined `loadOrder` (dependencies load before dependents), file patterns (e.g. `**/*.object.{ts,json,yaml}`), and overlay support flag.
+
+## Spec Protocol References
+
+This package depends on schemas and contracts defined in `@objectstack/spec`:
+
+| Spec Module                      | What It Defines                                     |
+|:---------------------------------|:----------------------------------------------------|
+| `spec/contracts/metadata-service` | `IMetadataService` — the async service interface   |
+| `spec/kernel/metadata-loader`    | Loader contract, load/save/watch schemas, `MetadataManagerConfig` |
+| `spec/kernel/metadata-plugin`    | Type registry, plugin manifest, capabilities        |
+| `spec/kernel/metadata-customization` | Overlay, merge strategy, customization policy   |
+| `spec/system/metadata-persistence` | `MetadataRecord` — DB persistence envelope        |
+| `spec/data/datasource`           | `DatasourceSchema`, `DriverDefinition`, capabilities |
+| `spec/contracts/data-driver`     | `IDataDriver` — database driver interface           |
+
+## Installation
+
+```bash
+pnpm add @objectstack/metadata
+```
 
 ## Usage
 
-Internal package used by `@objectstack/runtime` to process configuration.
+### Basic (Browser-Compatible)
+
+```typescript
+import { MetadataManager, MemoryLoader } from '@objectstack/metadata';
+
+const manager = new MetadataManager({
+  formats: ['json'],
+  loaders: [new MemoryLoader()],
+});
+
+// Register metadata
+await manager.register('object', 'account', { name: 'account', label: 'Account', fields: {} });
+
+// Retrieve
+const obj = await manager.get('object', 'account');
+
+// Query
+const result = await manager.query({ types: ['object'], search: 'account' });
+```
+
+### Node.js (with Filesystem)
+
+```typescript
+import { NodeMetadataManager, MetadataPlugin } from '@objectstack/metadata/node';
+
+const manager = new NodeMetadataManager({
+  rootDir: './src',
+  formats: ['typescript', 'json', 'yaml'],
+  watch: true,
+});
+
+// Load all objects from filesystem
+const objects = await manager.loadMany('object');
+
+// Watch for changes
+manager.watchService('object', (event) => {
+  console.log(`Object ${event.name} was ${event.type}`);
+});
+```
+
+### With Kernel Plugin
+
+```typescript
+import { MetadataPlugin } from '@objectstack/metadata/node';
+
+const plugin = MetadataPlugin({
+  rootDir: './src',
+  watch: process.env.NODE_ENV === 'development',
+});
+// Register with ObjectStack kernel
+kernel.use(plugin);
+```
+
+## Package Structure
+
+```
+packages/metadata/
+├── src/
+│   ├── index.ts                     # Main exports (browser-compatible)
+│   ├── node.ts                      # Node.js exports (filesystem, watching)
+│   ├── metadata-manager.ts          # MetadataManager (IMetadataService impl)
+│   ├── node-metadata-manager.ts     # NodeMetadataManager (+ file watching)
+│   ├── plugin.ts                    # MetadataPlugin (kernel integration)
+│   ├── loaders/
+│   │   ├── loader-interface.ts      # MetadataLoader contract
+│   │   ├── filesystem-loader.ts     # File I/O with glob, cache, ETag
+│   │   ├── memory-loader.ts         # In-memory store (tests/overrides)
+│   │   └── remote-loader.ts         # HTTP API loader with auth
+│   ├── serializers/
+│   │   ├── serializer-interface.ts  # MetadataSerializer contract
+│   │   ├── json-serializer.ts       # JSON format
+│   │   ├── yaml-serializer.ts       # YAML format
+│   │   └── typescript-serializer.ts # TS/JS module format
+│   └── migration/
+│       ├── index.ts                 # Barrel export
+│       └── executor.ts              # ChangeSet executor (DDL operations)
+├── package.json
+├── tsconfig.json
+├── vitest.config.ts
+├── README.md                        # This file
+└── ROADMAP.md                       # Development roadmap
+```
+
+## Related Packages
+
+| Package                 | Relationship                                     |
+|:------------------------|:-------------------------------------------------|
+| `@objectstack/spec`     | Protocol definitions (schemas, contracts, types)  |
+| `@objectstack/core`     | Logger, service registry, kernel utilities        |
+| `@objectstack/runtime`  | Uses this package to bootstrap metadata           |
+| `apps/studio`           | Visual metadata editor (consumes IMetadataService)|
+
+## License
+
+Apache-2.0 — see [LICENSE](../../LICENSE) for details.