Add connector protocols with authentication, sync, webhooks, and templates by Copilot · Pull Request #377 · objectstack-ai/framework

Copilot · 2026-01-30T08:01:20Z

Implements external system integration protocols with comprehensive authentication, data synchronization, and connector templates for SaaS, databases, file storage, and message queues.

Core Protocol (`integration/connector.zod.ts`)

Authentication - Discriminated union supporting:

OAuth2 (all grant types), JWT (HS/RS/ES algorithms), SAML (SSO)
API Key (header/query param), Basic, Bearer, None

Data Sync - Bidirectional with conflict resolution:

Strategies: full, incremental, upsert, append_only
Conflict resolution: source_wins, target_wins, latest_wins, manual
Real-time via webhooks, scheduled via cron, batch processing

Field Mapping - Source → target with transformations:

{
  sourceField: 'FirstName',
  targetField: 'first_name',
  dataType: 'string',
  transform: { type: 'uppercase' },
  syncMode: 'bidirectional'
}

Webhooks - Event-driven with HMAC verification:

Events: record.created, record.updated, sync.completed, etc.
Retry with exponential backoff, custom headers, signature algorithms

Rate Limiting - Multiple strategies:

token_bucket, sliding_window, fixed_window, leaky_bucket
Upstream rate limit header detection, burst capacity

Connector Templates

SaaS (integration/connector/saas.zod.ts):

Providers: Salesforce, HubSpot, Stripe, Zendesk, ServiceNow, etc.
OAuth token refresh, API versioning, object discovery, pagination
Examples: Salesforce (OAuth2 + incremental sync), HubSpot (API key)

Database (integration/connector/database.zod.ts):

Providers: PostgreSQL, MySQL, MongoDB, Snowflake, BigQuery, Redshift
CDC (log-based, trigger-based), connection pooling, SSL/TLS, read replicas
Examples: PostgreSQL (CDC + replication slot), MongoDB (change streams)

File Storage (integration/connector/file-storage.zod.ts):

Providers: S3, Azure Blob, GCS, Dropbox, Box, OneDrive, Google Drive
Multipart uploads, versioning, lifecycle policies, encryption
Content processing: text extraction, thumbnails, virus scanning
Examples: S3 (KMS encryption + thumbnails), Google Drive (OAuth2)

Message Queue (integration/connector/message-queue.zod.ts):

Providers: Kafka, RabbitMQ, SQS, Pub/Sub, Azure Service Bus, NATS
Consumer/producer configs, acknowledgment modes, delivery guarantees
DLQ, partitioning, schema registry (Avro), SASL auth
Examples: Kafka (consumer groups + exactly-once), RabbitMQ (routing keys)

Schema Generation

60+ JSON schemas auto-generated for IDE autocomplete and validation. All schemas follow camelCase for config keys, snake_case for machine names per ObjectStack conventions.

Directory Structure

All connector protocols are organized in the dedicated integration namespace:

Location: packages/spec/src/integration/
Exports: Available via @objectstack/spec/integration or Integration namespace
Organization: Keeps system directory focused on core system-level configurations

The integration namespace is established as the dedicated location for all external system connection protocols, separate from automation workflows and system configuration.

Original prompt

2.3 Connector Protocols / 连接器协议

In THIS Repo (Spec Definitions) / 在本仓库（规范定义）

External System Connector Protocol

Enhance packages/spec/src/system/connector.zod.ts
Authentication method schema (OAuth2, API Key, JWT, SAML)
Data synchronization configuration
Field mapping schema
Webhook configuration schema
Rate limiting and retry configuration
Specific Connector Templates

Create connector protocol templates
SaaS connector protocol template
Database connector protocol template
File storage connector protocol template
Message queue connector protocol template

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

vercel · 2026-01-30T08:01:25Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
spec	Ready	Preview, Comment	Jan 30, 2026 8:51am

…lates Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

hotlong · 2026-01-30T08:46:49Z

@copilot AI Prompt

Title: Refactor Connector Protocols to new integration directory

Body:Refactor the changes in this PR to move all Connector-related definitions out of the system directory and into a new top-level directory named integration. The current placement in system is overcrowding that namespace.

Requirements:

Create New Directory:

• Create a new directory: packages/spec/src/integration.

Move Core Definitions:

• Move packages/spec/src/system/connector.zod.ts (and any related files like authentication, sync configuration, rate limiting schemas) to packages/spec/src/integration/.

• Rename/Structure them appropriately (e.g., packages/spec/src/integration/connector.zod.ts).

Move Templates:

• Move all specific connector templates (SaaS, Database, File Storage, Message Queue) into the new packages/spec/src/integration directory.

• You may organize them into a subdirectory if appropriate, e.g., packages/spec/src/integration/templates/.

Update Exports:

• Update packages/spec/src/index.ts to export the definitions from the new integration directory instead of system.

• Ensure there are no lingering imports referencing the old system paths for these files.

Clean Up:

• Remove the connector-related files from packages/spec/src/system to ensure the directory remains focused on system-level configurations only.

Goal: Establish integration as the dedicated namespace for all external system connection protocols.

…ectory Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot · 2026-01-30T08:50:11Z

@copilot AI Prompt

Title: Refactor Connector Protocols to new integration directory

Body:Refactor the changes in this PR to move all Connector-related definitions out of the system directory and int...

Completed the refactoring. All connector protocols have been moved to the new packages/spec/src/integration directory:

Changes made:

Created new integration directory as a dedicated namespace for external system connection protocols
Moved connector.zod.ts, connector.test.ts, and all connector templates (saas, database, file-storage, message-queue) from system to integration
Created integration/index.ts with proper exports
Updated system/index.ts to remove connector exports
Added Integration namespace export to main index.ts

Verification:

✅ All 36 tests passing
✅ Build successful with documentation auto-generated for new integration category
✅ JSON schemas moved to packages/spec/json-schema/integration/

The system directory is now focused on system-level configurations only, and integration is the dedicated namespace for all external system connector protocols.

Commit: 95a23eb

github-actions · 2026-01-30T08:55:17Z

This PR is very large. Consider breaking it into smaller PRs for easier review.

packages/spec/src/integration/connector.test.ts

+import {
+  // Authentication Schemas
+  ApiKeyAuthSchema,
+  OAuth2AuthSchema,
+  JwtAuthSchema,
+  SamlAuthSchema,
+  BasicAuthSchema,
+  BearerTokenAuthSchema,
+  NoAuthSchema,
+  AuthenticationSchema,
+
+  // Field Mapping
+  FieldMappingSchema,
+  FieldTransformSchema,
+
+  // Data Sync
+  DataSyncConfigSchema,
+  SyncStrategySchema,
+  ConflictResolutionSchema,
+
+  // Webhook
+  WebhookConfigSchema,
+  WebhookEventSchema,
+
+  // Rate Limiting & Retry
+  RateLimitConfigSchema,
+  RetryConfigSchema,
+
+  // Base Connector
+  ConnectorSchema,
+  ConnectorTypeSchema,
+  ConnectorStatusSchema,
+
+  // Types
+  type Connector,
+  type ApiKeyAuth,
+  type OAuth2Auth,
+  type FieldMapping,
+  type DataSyncConfig,
+  type WebhookConfig,
+} from './connector.zod';


In general, to fix "unused import" issues, you remove the specific imported symbols that are never referenced in the file, leaving the rest of the import statement intact. This improves readability and avoids misleading future maintainers into thinking those items are tested or needed here.

For this file, the best fix is to edit the import block on lines 2–42 of packages/spec/src/integration/connector.test.ts and delete only the unused names: NoAuthSchema, FieldTransformSchema, SyncStrategySchema, ConflictResolutionSchema, WebhookEventSchema, BasicAuthSchema, and BearerTokenAuthSchema. We will keep all imports that are evidently used or could reasonably be used in the shown tests: ApiKeyAuthSchema, OAuth2AuthSchema, JwtAuthSchema, SamlAuthSchema, FieldMappingSchema, DataSyncConfigSchema, RateLimitConfigSchema, RetryConfigSchema, ConnectorSchema, ConnectorTypeSchema, ConnectorStatusSchema, and the imported types Connector, ApiKeyAuth, OAuth2Auth, FieldMapping, DataSyncConfig, WebhookConfig.

No new methods, definitions, or imports are required; we’re only trimming the existing import list.

Copilot

Pull request overview

This pull request adds comprehensive connector protocols for external system integration to ObjectStack. It introduces a new integration namespace containing authentication methods, data synchronization, field mapping, webhooks, and specialized connector templates for SaaS, databases, file storage, and message queues.

Changes:

Creates new integration namespace with core connector protocol (connector.zod.ts)
Adds 4 specialized connector templates (SaaS, Database, File Storage, Message Queue)
Implements comprehensive authentication schemas (OAuth2, JWT, SAML, API Key, Basic, Bearer, None)
Provides data sync configuration with conflict resolution and field mapping
Includes webhook configuration with HMAC verification and retry logic
Generates 60+ JSON schemas for IDE autocomplete
Adds comprehensive test coverage (connector.test.ts with 624 lines)
Updates documentation and exports

Reviewed changes

Copilot reviewed 68 out of 68 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`packages/spec/src/integration/connector.zod.ts`	Core connector protocol with authentication, sync, webhooks, rate limiting
`packages/spec/src/integration/connector/saas.zod.ts`	SaaS connector template (Salesforce, HubSpot, etc.)
`packages/spec/src/integration/connector/database.zod.ts`	Database connector template with CDC support
`packages/spec/src/integration/connector/file-storage.zod.ts`	File storage connector (S3, Azure Blob, Google Drive)
`packages/spec/src/integration/connector/message-queue.zod.ts`	Message queue connector (Kafka, RabbitMQ, SQS)
`packages/spec/src/integration/connector.test.ts`	Comprehensive test suite covering all schemas
`packages/spec/src/integration/index.ts`	Integration namespace exports
`packages/spec/src/index.ts`	Adds Integration namespace to main exports
`packages/spec/src/system/index.ts`	Documents connector protocol moved to integration
`content/docs/references/integration/*`	Documentation for connector schemas
`packages/spec/json-schema/integration/*`	Auto-generated JSON schemas (60+ files)

Copilot · 2026-01-30T09:15:47Z

packages/spec/src/integration/connector.zod.ts

+export const ConflictResolutionSchema = z.enum([
+  'source_wins',    // External system data takes precedence
+  'target_wins',    // ObjectStack data takes precedence
+  'latest_wins',    // Most recently modified wins
+  'manual',         // Flag for manual resolution
+]).describe('Conflict resolution strategy');


Duplicate schema definition: ConflictResolutionSchema is already defined in packages/spec/src/automation/sync.zod.ts (lines 82-88). The automation version includes a 'merge' option that this integration version lacks. This duplication will lead to maintainability issues and inconsistencies. Consider consolidating these into a shared schema or importing from automation if they serve the same purpose.

Copilot · 2026-01-30T09:15:47Z

content/docs/references/integration/connector.mdx

+| **certificate** | `string` | ✅ | SAML IdP certificate (X.509) |
+| **privateKey** | `string` | optional | SAML service provider private key |
+| **callbackUrl** | `string` | optional | SAML assertion consumer service URL |
+| **signatureAlgorithm** | `Enum<'sha1' \| 'sha256' \| 'sha512'>` | optional | SAML signature algorithm |


The SAML signatureAlgorithm property permits the weak sha1 option, which is considered cryptographically broken for signatures and can enable forging SAML assertions in realistic attack scenarios. An attacker who can influence or exploit SHA‑1 collisions could bypass SSO/authentication guarantees if SAML messages are signed with SHA‑1. Restrict this field to modern algorithms like sha256/sha512 (with a secure default) and deprecate or remove sha1 support to avoid relying on a broken primitive.

Suggested change

| **signatureAlgorithm** | `Enum<'sha1' \| 'sha256' \| 'sha512'>` | optional | SAML signature algorithm |

| **signatureAlgorithm** | `Enum<'sha256' \| 'sha512'>` | optional | SAML signature algorithm |

Initial plan

e2f4fff

Copilot AI assigned Copilot and hotlong Jan 30, 2026

Copilot started work on behalf of hotlong January 30, 2026 08:01 View session

vercel bot deployed to Preview January 30, 2026 08:02 View deployment

Add connector protocols with authentication, sync, webhooks, and temp…

2404f7b

…lates Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

vercel bot deployed to Preview January 30, 2026 08:09 View deployment

Fix Zod schema defaults and TypeScript compilation issues

82522e9

Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

vercel bot deployed to Preview January 30, 2026 08:14 View deployment

Copilot AI changed the title ~~[WIP] Enhance connector protocols with schemas and templates~~ Add connector protocols with authentication, sync, webhooks, and templates Jan 30, 2026

Copilot AI requested a review from hotlong January 30, 2026 08:18

Copilot finished work on behalf of hotlong January 30, 2026 08:18

Copilot started work on behalf of hotlong January 30, 2026 08:47 View session

Refactor: Move connector protocols from system to new integration dir…

95a23eb

…ectory Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

vercel bot deployed to Preview January 30, 2026 08:51 View deployment

Copilot finished work on behalf of hotlong January 30, 2026 08:53

github-actions bot added documentation Improvements or additions to documentation protocol:system tests size/xl labels Jan 30, 2026

github-advanced-security bot found potential problems Jan 30, 2026

View reviewed changes

hotlong marked this pull request as ready for review January 30, 2026 09:04

Copilot AI review requested due to automatic review settings January 30, 2026 09:04

hotlong merged commit edfb083 into main Jan 30, 2026
10 checks passed

Copilot started reviewing on behalf of hotlong January 30, 2026 09:04 View session

Copilot AI reviewed Jan 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add connector protocols with authentication, sync, webhooks, and templates#377

Add connector protocols with authentication, sync, webhooks, and templates#377
hotlong merged 4 commits intomainfrom
copilot/enhance-connector-protocols

Copilot AI commented Jan 30, 2026 •

edited

Loading

Uh oh!

vercel bot commented Jan 30, 2026 •

edited

Loading

Uh oh!

hotlong commented Jan 30, 2026

Uh oh!

Copilot AI commented Jan 30, 2026

Uh oh!

github-actions bot commented Jan 30, 2026

Uh oh!

Check notice

Copilot Autofix

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 30, 2026

Uh oh!

Copilot AI Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	\| signatureAlgorithm \| `Enum<'sha1' \\| 'sha256' \\| 'sha512'>` \| optional \| SAML signature algorithm \|
	\| signatureAlgorithm \| `Enum<'sha256' \\| 'sha512'>` \| optional \| SAML signature algorithm \|

Conversation

Copilot AI commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Core Protocol (integration/connector.zod.ts)

Connector Templates

Schema Generation

Directory Structure

Uh oh!

vercel bot commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hotlong commented Jan 30, 2026

Uh oh!

Copilot AI commented Jan 30, 2026

Uh oh!

github-actions bot commented Jan 30, 2026

Uh oh!

Check notice

Copilot Autofix

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jan 30, 2026 •

edited

Loading

Core Protocol (`integration/connector.zod.ts`)

vercel bot commented Jan 30, 2026 •

edited

Loading