-
Notifications
You must be signed in to change notification settings - Fork 674
feat(sync): S3-compatible cloud backup sync #1741
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Plan: 云同步(S3 兼容对象存储) | ||
|
|
||
| ## 架构总览 | ||
| 云能力作为现有备份链路的**叠加层**,不重写本地备份/导入: | ||
|
|
||
| ```text | ||
| DataSettings.vue ─► sync store ─► SyncClient ─► [route] ─► SyncPresenter ─► CloudStorageService | ||
| 保存/测试/上传/拉取 │ │ | ||
| ConfigPresenter R2 / S3 桶 | ||
| (safeStorage 加密凭证) | ||
| ``` | ||
|
|
||
| - 上传 = 取本地最新 zip(`SyncPresenter.listBackups()`)→ `CloudStorageService.uploadBackup()`。 | ||
| - 拉取 = `CloudStorageService.downloadLatest()` 落地同步文件夹 → 复用 `SyncPresenter.importFromSync()`。 | ||
|
|
||
| ## 关键文件 | ||
| - `src/main/presenter/syncPresenter/cloudStorageService.ts`(新增):S3 客户端封装 | ||
| (`forcePathStyle: true`、`region` 默认 `auto`),方法 `testConnection / uploadBackup / | ||
| listRemoteBackups / downloadLatest`,沿用 `backup-\d+\.zip` 文件名约定。 | ||
| - `src/main/presenter/syncPresenter/index.ts`:新增 `testCloudConnection / uploadLatestBackupToCloud / | ||
| pullLatestBackupFromCloud`,从 ConfigPresenter 取解密后的 `ResolvedCloudSyncConfig` 构造服务。 | ||
| - `src/main/presenter/configPresenter/index.ts`:`getCloudSyncConfig / setCloudSyncConfig / | ||
| getResolvedCloudSyncConfig / isCloudSafeStorageAvailable`;secret 经 `safeStorage` 加密, | ||
| 视图脱敏(仅 `hasSecret`)。 | ||
| - `src/shared/contracts/routes/sync.routes.ts` + `routes.ts`:新增 5 路由并登记。 | ||
| - `src/main/routes/index.ts`:sync 段加 5 个 case 分发(上传/拉取复用 `recordSettingsActivity`)。 | ||
| - `src/shared/types/presenters/legacy.presenters.d.ts`:`ISyncPresenter` / `IConfigPresenter` 新方法 + | ||
| `CloudSyncConfigView / CloudSyncConfigInput / ResolvedCloudSyncConfig / CloudSyncResult` 类型。 | ||
| - `src/renderer/api/SyncClient.ts` + `src/renderer/src/stores/sync.ts`:5 个客户端方法 + store action。 | ||
| - `src/renderer/settings/components/DataSettings.vue`:同步卡片内新增云同步区块。 | ||
| - i18n:`sync.json`(success/error 云键)、`settings.json`(`data.cloudSync.*`),全语言补齐。 | ||
|
|
||
| ## 复用 | ||
| - 备份/导入:`performBackup / importFromSync / listBackups / getBackupsDirectory`。 | ||
| - 加密:`safeStorage`(参照 `databaseSecurityPresenter`)。 | ||
| - IPC / 渲染数据流:`defineRouteContract`、`useIpcQuery/useIpcMutation`、pinia store。 | ||
|
|
||
| ## 依赖 | ||
| - 新增 `@aws-sdk/client-s3`(与现有 `@aws-sdk/client-bedrock` 同源)。 | ||
|
|
||
| ## 边界与决策 | ||
| - R2 必须 path-style + `region: 'auto'`。 | ||
| - secret 留空 = 不修改既有值;非空才重新加密写入。 | ||
| - 配置保存先完成 secret 加密,再写入本地设置;如果第二步失败,回滚已写入的 secret,避免 | ||
| access key 与旧 secret 错配。 | ||
| - 云上传只接受结构可导入的 `backup-\d+\.zip`,避免用户放入任意 zip 后被同步到云端。 | ||
| - 活动记录复用既有 `backup_created` / `imported` action,不扩 schema(最小改动)。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # Spec: 云同步(S3 兼容对象存储) | ||
|
|
||
| ## 背景与问题 | ||
| DeepChat 现有的「同步」仅把数据库 + 配置打包为 `backup-<时间戳>.zip` 写入**本地**同步文件夹 | ||
| (默认 `~/DeepchatSync`),导入也只从该本地文件夹读取。多设备(家 / 公司)之间数据不会自动流转, | ||
| 必须手动搬运 zip。 | ||
|
|
||
| ## 目标 | ||
| 在**不改动**现有本地备份/导入逻辑的前提下,叠加一个最小的云能力: | ||
| 1. **上传到云**:把本地最新备份 zip 推送到 S3 兼容对象存储。 | ||
| 2. **从云拉取最新**:下载云端最新备份 zip 到本地同步文件夹,复用现有导入流程还原。 | ||
|
|
||
| 主用例为 Cloudflare R2,通过 **S3 兼容协议** 实现,使同一套配置也能连 MinIO / AWS S3 / B2。 | ||
|
|
||
| ## 非目标(明确不做,避免过度设计) | ||
| - 不做定时/自动上传,纯手动按钮触发。 | ||
| - 不做云端多版本管理、保留策略、冲突合并。 | ||
| - 不做 WebDAV / R2 专有 Token API。 | ||
| - 不引入新的导入合并语义,沿用现有 increment / overwrite。 | ||
|
|
||
| ## 用户故事 | ||
| - 作为多设备用户,我在 A 机点「上传到云」,在 B 机点「从云拉取最新」,即可把聊天记录与配置带过去。 | ||
|
|
||
| ## 验收标准 | ||
| - 设置 → 数据出现「云同步 (S3 兼容)」区块:endpoint / bucket / region / prefix / AK / SK + 保存 / 测试连接 / 上传 / 拉取。 | ||
| - 填入有效 R2 凭证后「测试连接」成功;「上传到云」后桶内出现 `deepchat-backups/backup-*.zip`。 | ||
| - 另一设备「从云拉取最新」后数据恢复。 | ||
| - `secretAccessKey` 在 `app-settings` 中以 safeStorage 密文存储,渲染层永不收到明文。 | ||
| - 同一套 UI 切换 endpoint/bucket 即可对接 MinIO(验证 S3 兼容)。 | ||
|
|
||
| ## 安全 | ||
| - 凭证 secret 用 Electron `safeStorage` 加密后落盘(与 `databaseSecurityPresenter` 一致)。 | ||
| - safeStorage 不可用时拒绝保存 secret 并提示(`sync.error.safeStorageUnavailable`)。 | ||
|
|
||
| ## 待澄清 | ||
| - 无(方案、凭证存储、触发方式均已与用户确认)。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Tasks: 云同步(S3 兼容对象存储) | ||
|
|
||
| - [x] 安装 `@aws-sdk/client-s3` 依赖 | ||
| - [x] 新增 `CloudStorageService`(testConnection / uploadBackup / listRemoteBackups / downloadLatest) | ||
| - [x] `ConfigPresenter` 加云凭证读写,secret 用 safeStorage 加密、视图脱敏 | ||
| - [x] `SyncPresenter` 加 testCloudConnection / uploadLatestBackupToCloud / pullLatestBackupFromCloud | ||
| - [x] 新增 5 个 IPC 路由契约并在 `routes.ts` 登记 | ||
| - [x] `legacy.presenters.d.ts` 加接口方法与 CloudSync* 类型 | ||
| - [x] `main/routes/index.ts` 注册 5 个 case | ||
| - [x] `SyncClient.ts` + `stores/sync.ts` 加云方法/状态 | ||
| - [x] `DataSettings.vue` 云同步 UI(表单 + 保存/测试/上传/拉取) | ||
| - [x] i18n:zh-CN / en-US 增云键,其余语言英文兜底,`pnpm run i18n` 校验通过 | ||
| - [x] PR review:云配置写入失败可感知并回滚 secret | ||
| - [x] PR review:云上传/下载改为流式 IO | ||
| - [x] PR review:上传前校验备份包结构,跳过伪造 zip | ||
| - [x] PR review:导入时本地 settings 读取失败则回滚 | ||
| - [x] PR review:云操作 busy guard 补齐 | ||
| - [x] PR review:he-IL / id-ID 云同步文案本地化 | ||
| - [x] 收尾:`pnpm run typecheck` / `format` / `lint` 全绿 | ||
|
|
||
| ## 待人工验证(需真实 R2 凭证) | ||
| - [ ] 填 R2 凭证 → 测试连接成功 | ||
| - [ ] 上传到云 → 桶内出现 `deepchat-backups/backup-*.zip` | ||
| - [ ] 另一设备拉取最新 → 数据恢复 | ||
| - [ ] 切换 MinIO endpoint 验证 S3 兼容 | ||
| - [ ] 确认 `app-settings` 中 secret 为密文 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # Issue: 增量导入时 FTS5 影子表报错 | ||
|
|
||
| ## 现象 | ||
| 增量(increment)导入备份时抛错并回滚: | ||
|
|
||
| ```text | ||
| Failed to import database: Failed to import table deepchat_search_documents_fts_config: | ||
| table deepchat_search_documents_fts_config may not be modified | ||
| ``` | ||
|
|
||
| 经云同步「从云拉取最新」复用 `importFromSync` 时触发,本地「导入数据」走相同路径也会复现。 | ||
|
|
||
| ## 根因 | ||
| `DataImporter.getTablesInOrder()`(`src/main/presenter/sqlitePresenter/importData.ts`)用 | ||
| `SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'` 取表, | ||
| 未排除 FTS5 的虚拟表 `deepchat_search_documents_fts` 及其影子表 | ||
| `_data/_idx/_docsize/_config`。对影子表执行普通 `INSERT` 会被 SQLite 拒绝("may not be modified")。 | ||
|
|
||
| 关键坑:FTS5 影子表在 `sqlite_master` 中**带有真实的 `CREATE TABLE` sql(非 NULL)**,因此 | ||
| 不能靠 `sql IS NULL` 识别,必须按「虚拟表名前缀」排除。 | ||
|
|
||
| ## 修复 | ||
| 先取出所有虚拟表(`sql LIKE 'CREATE VIRTUAL TABLE%'`),再排除名字等于虚拟表名、或以 | ||
| `<虚拟表名>_` 开头的影子表。FTS5 采用外部内容表模式(`content='deepchat_search_documents'`) | ||
| + 触发器,导入内容表 `deepchat_search_documents` 行时触发器会自动维护 FTS 索引,无需直接写 FTS 表。 | ||
|
|
||
| ## 影响范围 | ||
| - 仅增量导入路径(`DataImporter`);覆盖导入走整库文件拷贝,不受影响。 | ||
| - 修复后导入不再触碰 FTS 影子表,搜索索引由触发器重建。 | ||
|
|
||
| ## 验证 | ||
| - 含 `deepchat_search_documents_fts*` 表的备份增量导入成功,无报错。 | ||
| - 导入后对已导入文档执行搜索可命中(FTS 索引已由触发器填充)。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.