Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/features/cloud-sync-s3/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Plan: 云同步(S3 兼容对象存储)

## 架构总览
云能力作为现有备份链路的**叠加层**,不重写本地备份/导入:

```text
DataSettings.vue ─► sync store ─► SyncClient ─► [route] ─► SyncPresenter ─► CloudStorageService
保存/测试/上传/拉取 │ │
ConfigPresenter R2 / S3 桶
(safeStorage 加密凭证)
```

- 上传 = 取本地最新 zip(`SyncPresenter.listBackups()`)→ `CloudStorageService.uploadBackup()`。
- 拉取 = `CloudStorageService.downloadLatest()` 落地同步文件夹 → 复用 `SyncPresenter.importFromSync()`。

## 关键文件
- `src/main/presenter/syncPresenter/cloudStorageService.ts`(新增):S3 客户端封装
(`forcePathStyle: true`、`region` 默认 `auto`),方法 `testConnection / uploadBackup /
listRemoteBackups / downloadLatest`,沿用 `backup-\d+\.zip` 文件名约定。
- `src/main/presenter/syncPresenter/index.ts`:新增 `testCloudConnection / uploadLatestBackupToCloud /
pullLatestBackupFromCloud`,从 ConfigPresenter 取解密后的 `ResolvedCloudSyncConfig` 构造服务。
- `src/main/presenter/configPresenter/index.ts`:`getCloudSyncConfig / setCloudSyncConfig /
getResolvedCloudSyncConfig / isCloudSafeStorageAvailable`;secret 经 `safeStorage` 加密,
视图脱敏(仅 `hasSecret`)。
- `src/shared/contracts/routes/sync.routes.ts` + `routes.ts`:新增 5 路由并登记。
- `src/main/routes/index.ts`:sync 段加 5 个 case 分发(上传/拉取复用 `recordSettingsActivity`)。
- `src/shared/types/presenters/legacy.presenters.d.ts`:`ISyncPresenter` / `IConfigPresenter` 新方法 +
`CloudSyncConfigView / CloudSyncConfigInput / ResolvedCloudSyncConfig / CloudSyncResult` 类型。
- `src/renderer/api/SyncClient.ts` + `src/renderer/src/stores/sync.ts`:5 个客户端方法 + store action。
- `src/renderer/settings/components/DataSettings.vue`:同步卡片内新增云同步区块。
- i18n:`sync.json`(success/error 云键)、`settings.json`(`data.cloudSync.*`),全语言补齐。

## 复用
- 备份/导入:`performBackup / importFromSync / listBackups / getBackupsDirectory`。
- 加密:`safeStorage`(参照 `databaseSecurityPresenter`)。
- IPC / 渲染数据流:`defineRouteContract`、`useIpcQuery/useIpcMutation`、pinia store。

## 依赖
- 新增 `@aws-sdk/client-s3`(与现有 `@aws-sdk/client-bedrock` 同源)。

## 边界与决策
- R2 必须 path-style + `region: 'auto'`。
- secret 留空 = 不修改既有值;非空才重新加密写入。
- 配置保存先完成 secret 加密,再写入本地设置;如果第二步失败,回滚已写入的 secret,避免
access key 与旧 secret 错配。
- 云上传只接受结构可导入的 `backup-\d+\.zip`,避免用户放入任意 zip 后被同步到云端。
- 活动记录复用既有 `backup_created` / `imported` action,不扩 schema(最小改动)。
36 changes: 36 additions & 0 deletions docs/features/cloud-sync-s3/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Spec: 云同步(S3 兼容对象存储)

## 背景与问题
DeepChat 现有的「同步」仅把数据库 + 配置打包为 `backup-<时间戳>.zip` 写入**本地**同步文件夹
(默认 `~/DeepchatSync`),导入也只从该本地文件夹读取。多设备(家 / 公司)之间数据不会自动流转,
必须手动搬运 zip。

## 目标
在**不改动**现有本地备份/导入逻辑的前提下,叠加一个最小的云能力:
1. **上传到云**:把本地最新备份 zip 推送到 S3 兼容对象存储。
2. **从云拉取最新**:下载云端最新备份 zip 到本地同步文件夹,复用现有导入流程还原。

主用例为 Cloudflare R2,通过 **S3 兼容协议** 实现,使同一套配置也能连 MinIO / AWS S3 / B2。

## 非目标(明确不做,避免过度设计)
- 不做定时/自动上传,纯手动按钮触发。
- 不做云端多版本管理、保留策略、冲突合并。
- 不做 WebDAV / R2 专有 Token API。
- 不引入新的导入合并语义,沿用现有 increment / overwrite。

## 用户故事
- 作为多设备用户,我在 A 机点「上传到云」,在 B 机点「从云拉取最新」,即可把聊天记录与配置带过去。

## 验收标准
- 设置 → 数据出现「云同步 (S3 兼容)」区块:endpoint / bucket / region / prefix / AK / SK + 保存 / 测试连接 / 上传 / 拉取。
- 填入有效 R2 凭证后「测试连接」成功;「上传到云」后桶内出现 `deepchat-backups/backup-*.zip`。
- 另一设备「从云拉取最新」后数据恢复。
- `secretAccessKey` 在 `app-settings` 中以 safeStorage 密文存储,渲染层永不收到明文。
- 同一套 UI 切换 endpoint/bucket 即可对接 MinIO(验证 S3 兼容)。

## 安全
- 凭证 secret 用 Electron `safeStorage` 加密后落盘(与 `databaseSecurityPresenter` 一致)。
- safeStorage 不可用时拒绝保存 secret 并提示(`sync.error.safeStorageUnavailable`)。

## 待澄清
- 无(方案、凭证存储、触发方式均已与用户确认)。
26 changes: 26 additions & 0 deletions docs/features/cloud-sync-s3/tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Tasks: 云同步(S3 兼容对象存储)

- [x] 安装 `@aws-sdk/client-s3` 依赖
- [x] 新增 `CloudStorageService`(testConnection / uploadBackup / listRemoteBackups / downloadLatest)
- [x] `ConfigPresenter` 加云凭证读写,secret 用 safeStorage 加密、视图脱敏
- [x] `SyncPresenter` 加 testCloudConnection / uploadLatestBackupToCloud / pullLatestBackupFromCloud
- [x] 新增 5 个 IPC 路由契约并在 `routes.ts` 登记
- [x] `legacy.presenters.d.ts` 加接口方法与 CloudSync* 类型
- [x] `main/routes/index.ts` 注册 5 个 case
- [x] `SyncClient.ts` + `stores/sync.ts` 加云方法/状态
- [x] `DataSettings.vue` 云同步 UI(表单 + 保存/测试/上传/拉取)
- [x] i18n:zh-CN / en-US 增云键,其余语言英文兜底,`pnpm run i18n` 校验通过
- [x] PR review:云配置写入失败可感知并回滚 secret
- [x] PR review:云上传/下载改为流式 IO
- [x] PR review:上传前校验备份包结构,跳过伪造 zip
- [x] PR review:导入时本地 settings 读取失败则回滚
- [x] PR review:云操作 busy guard 补齐
- [x] PR review:he-IL / id-ID 云同步文案本地化
- [x] 收尾:`pnpm run typecheck` / `format` / `lint` 全绿

## 待人工验证(需真实 R2 凭证)
- [ ] 填 R2 凭证 → 测试连接成功
- [ ] 上传到云 → 桶内出现 `deepchat-backups/backup-*.zip`
- [ ] 另一设备拉取最新 → 数据恢复
- [ ] 切换 MinIO endpoint 验证 S3 兼容
- [ ] 确认 `app-settings` 中 secret 为密文
33 changes: 33 additions & 0 deletions docs/issues/import-fts-shadow-table/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Issue: 增量导入时 FTS5 影子表报错

## 现象
增量(increment)导入备份时抛错并回滚:

```text
Failed to import database: Failed to import table deepchat_search_documents_fts_config:
table deepchat_search_documents_fts_config may not be modified
```

经云同步「从云拉取最新」复用 `importFromSync` 时触发,本地「导入数据」走相同路径也会复现。

## 根因
`DataImporter.getTablesInOrder()`(`src/main/presenter/sqlitePresenter/importData.ts`)用
`SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'` 取表,
未排除 FTS5 的虚拟表 `deepchat_search_documents_fts` 及其影子表
`_data/_idx/_docsize/_config`。对影子表执行普通 `INSERT` 会被 SQLite 拒绝("may not be modified")。

关键坑:FTS5 影子表在 `sqlite_master` 中**带有真实的 `CREATE TABLE` sql(非 NULL)**,因此
不能靠 `sql IS NULL` 识别,必须按「虚拟表名前缀」排除。

## 修复
先取出所有虚拟表(`sql LIKE 'CREATE VIRTUAL TABLE%'`),再排除名字等于虚拟表名、或以
`<虚拟表名>_` 开头的影子表。FTS5 采用外部内容表模式(`content='deepchat_search_documents'`)
+ 触发器,导入内容表 `deepchat_search_documents` 行时触发器会自动维护 FTS 索引,无需直接写 FTS 表。

## 影响范围
- 仅增量导入路径(`DataImporter`);覆盖导入走整库文件拷贝,不受影响。
- 修复后导入不再触碰 FTS 影子表,搜索索引由触发器重建。

## 验证
- 含 `deepchat_search_documents_fts*` 表的备份增量导入成功,无报错。
- 导入后对已导入文档执行搜索可命中(FTS 索引已由触发器填充)。
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
"@ai-sdk/openai-compatible": "^2.0.48",
"@ai-sdk/provider": "^3.0.10",
"@aws-sdk/client-bedrock": "^3.1057.0",
"@aws-sdk/client-s3": "^3.1062.0",
"@duckdb/node-api": "1.5.3-r.1",
"@e2b/code-interpreter": "^1.5.1",
"@electron-toolkit/preload": "^3.0.2",
Expand Down
142 changes: 141 additions & 1 deletion src/main/presenter/configPresenter/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ import {
AcpResolvedLaunchSpec,
ProviderDbRefreshResult
} from '@shared/presenter'
import type {
CloudSyncConfigView,
CloudSyncConfigInput,
ResolvedCloudSyncConfig
} from '@shared/presenter'
import { ProviderBatchUpdate } from '@shared/provider-operations'
import { SearchEngineTemplate } from '@shared/chat'
import {
Expand All @@ -38,7 +43,7 @@ import {
import ElectronStore from 'electron-store'
import { DEFAULT_PROVIDERS } from './providers'
import path from 'path'
import { app, nativeTheme, shell } from 'electron'
import { app, nativeTheme, shell, safeStorage } from 'electron'
import fs from 'fs'
import {
CONFIG_EVENTS,
Expand Down Expand Up @@ -1875,6 +1880,141 @@ export class ConfigPresenter implements IConfigPresenter {
this.setSetting('lastSyncTime', time)
}

// === Cloud sync (S3-compatible) settings ===
// Non-sensitive fields live in app-settings; the secret is encrypted via safeStorage.
private readonly CLOUD_SYNC_BASE_KEY = 'cloudSyncConfig'
private readonly CLOUD_SYNC_SECRET_KEY = 'cloudSyncSecret'

isCloudSafeStorageAvailable(): boolean {
try {
return safeStorage.isEncryptionAvailable()
} catch {
return false
}
}

private getCloudSyncBase(): {
enabled: boolean
endpoint: string
bucket: string
region: string
prefix: string
accessKeyId: string
} {
const stored = this.getSetting<{
enabled?: boolean
endpoint?: string
bucket?: string
region?: string
prefix?: string
accessKeyId?: string
}>(this.CLOUD_SYNC_BASE_KEY)
return {
enabled: stored?.enabled ?? false,
endpoint: stored?.endpoint ?? '',
bucket: stored?.bucket ?? '',
region: stored?.region ?? 'auto',
prefix: stored?.prefix ?? 'deepchat-backups',
accessKeyId: stored?.accessKeyId ?? ''
}
}

private getCloudSyncSecret(): string {
const wrapped = this.getSetting<string>(this.CLOUD_SYNC_SECRET_KEY)
if (!wrapped) {
return ''
}
try {
return safeStorage.decryptString(Buffer.from(wrapped, 'base64'))
} catch (error) {
console.error('[Config] Failed to decrypt cloud sync secret:', error)
return ''
}
}

getCloudSyncConfig(): CloudSyncConfigView {
const base = this.getCloudSyncBase()
return {
...base,
hasSecret: Boolean(this.getCloudSyncSecret()),
safeStorageAvailable: this.isCloudSafeStorageAvailable()
}
}

private setCloudSyncSetting<T>(key: string, value: T): void {
this.getSettingsStoreForKey(key).set(key, value)
eventBus.sendToMain(CONFIG_EVENTS.SETTING_CHANGED, key, value)
}

private deleteCloudSyncSetting(key: string): void {
this.getSettingsStoreForKey(key).delete(key)
eventBus.sendToMain(CONFIG_EVENTS.SETTING_CHANGED, key, undefined)
}

setCloudSyncConfig(config: CloudSyncConfigInput): CloudSyncConfigView {
const current = this.getCloudSyncBase()
const next = {
enabled: config.enabled ?? current.enabled,
endpoint: config.endpoint ?? current.endpoint,
bucket: config.bucket ?? current.bucket,
region: config.region ?? current.region,
prefix: config.prefix ?? current.prefix,
accessKeyId: config.accessKeyId ?? current.accessKeyId
}

// Only update the secret when a non-empty value is provided; empty/undefined keeps the existing one.
const currentWrappedSecret = this.getSetting<string>(this.CLOUD_SYNC_SECRET_KEY)
let nextWrappedSecret: string | undefined
if (typeof config.secretAccessKey === 'string' && config.secretAccessKey.length > 0) {
if (!this.isCloudSafeStorageAvailable()) {
throw new Error('sync.error.safeStorageUnavailable')
}
nextWrappedSecret = Buffer.from(safeStorage.encryptString(config.secretAccessKey)).toString(
'base64'
)
}

let secretWritten = false
try {
if (nextWrappedSecret !== undefined) {
this.setCloudSyncSetting(this.CLOUD_SYNC_SECRET_KEY, nextWrappedSecret)
secretWritten = true
}
this.setCloudSyncSetting(this.CLOUD_SYNC_BASE_KEY, next)
} catch (error) {
if (secretWritten) {
try {
if (currentWrappedSecret) {
this.setCloudSyncSetting(this.CLOUD_SYNC_SECRET_KEY, currentWrappedSecret)
} else {
this.deleteCloudSyncSetting(this.CLOUD_SYNC_SECRET_KEY)
}
} catch (rollbackError) {
console.error('[Config] Failed to rollback cloud sync secret:', rollbackError)
}
}
throw error
}
Comment thread
zerob13 marked this conversation as resolved.

return this.getCloudSyncConfig()
}

getResolvedCloudSyncConfig(): ResolvedCloudSyncConfig | null {
const base = this.getCloudSyncBase()
const secretAccessKey = this.getCloudSyncSecret()
if (!base.endpoint || !base.bucket || !base.accessKeyId || !secretAccessKey) {
return null
}
return {
endpoint: base.endpoint,
bucket: base.bucket,
region: base.region,
prefix: base.prefix,
accessKeyId: base.accessKeyId,
secretAccessKey
}
}

// Skills settings
getSkillsEnabled(): boolean {
return this.getSetting<boolean>('enableSkills') ?? true
Expand Down
22 changes: 19 additions & 3 deletions src/main/presenter/sqlitePresenter/importData.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,25 @@ export class DataImporter {
}

private getTablesInOrder(): string[] {
const tables = this.sourceDb
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'")
.all() as { name: string }[]
const allTables = this.sourceDb
.prepare(
"SELECT name, sql FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'"
)
.all() as { name: string; sql: string | null }[]

// Virtual tables (e.g. FTS5) and their shadow tables cannot be written by a plain column-copy
// INSERT — SQLite raises "table X may not be modified". Note FTS5 shadow tables
// (<vtab>_data/_idx/_docsize/_config/_content) DO carry a real CREATE TABLE sql in
// sqlite_master, so they must be excluded by name prefix, not by inspecting their sql.
// For external-content FTS the index is rebuilt by triggers when the content table is imported.
const virtualTableNames = allTables
.filter((table) => typeof table.sql === 'string' && /^CREATE VIRTUAL TABLE/i.test(table.sql))
.map((table) => table.name)

const isVirtualOrShadow = (name: string): boolean =>
virtualTableNames.some((vtab) => name === vtab || name.startsWith(`${vtab}_`))

const tables = allTables.filter((table) => !isVirtualOrShadow(table.name))

const preferredOrder = ['conversations', 'messages', 'attachments', 'message_attachments']
const preferredSet = new Set(preferredOrder)
Expand Down
Loading