ENG-1475 Export relations to imported nodes#817
ENG-1475 Export relations to imported nodes#817maparent wants to merge 1 commit intoeng-1476-repair-lint-ci-task-for-databasefrom
Conversation
|
Updates to Preview Branch (eng-1475-export-relations-to-imported-nodes) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
ad7af74 to
ce78692
Compare
30e40b2 to
bb38514
Compare
ce78692 to
b87afad
Compare
6784bf7 to
91b4992
Compare
9d6b14b to
1b31245
Compare
cc52d2f to
a6f2d9e
Compare
ce150f3 to
3169b1e
Compare
7489026 to
2df080d
Compare
There was a problem hiding this comment.
Devin Review found 2 new potential issues.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 12 additional findings in Devin Review.
There was a problem hiding this comment.
🔴 Missing legacy importedFromSpaceUri check in collectDiscourseNodesFromPaths allows imported files to be synced
Same root cause as in shouldSyncFile: collectDiscourseNodesFromPaths only checks frontmatter.importedFromRid but not the legacy frontmatter.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault at apps/obsidian/src/utils/getDiscourseNodes.ts:32-38 correctly checks both fields, but this function was not updated consistently.
Root Cause and Impact
This function is used by syncDiscourseNodeChanges (called from FileChangeListener). During the migration window or if migration fails, any file change event on a file that still has only the legacy importedFromSpaceUri will pass through both shouldSyncFile and collectDiscourseNodesFromPaths guards, causing the imported node to be synced to Supabase as a locally-created node. This is a second guard that should also check the legacy field for defense-in-depth.
(Refers to lines 717-720)
Was this helpful? React with 👍 or 👎 to provide feedback.
9121ca2 to
6aa3c39
Compare
2df080d to
21b378d
Compare
There was a problem hiding this comment.
Devin Review found 2 new potential issues.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 15 additional findings in Devin Review.
| const allNodes = await collectDiscourseNodesFromVault(plugin, true); | ||
|
|
||
| const changedNodeInstances = relationsOnly | ||
| ? [] |
There was a problem hiding this comment.
🔴 Imported nodes are unintentionally synced as Content and Concepts to the user's own space
Changing collectDiscourseNodesFromVault(plugin) to collectDiscourseNodesFromVault(plugin, true) causes allNodes to include imported nodes. When relationsOnly is false, these imported nodes are passed directly to buildChangedNodesFromNodes at line 430. Since imported nodes were never synced as Content to this space, detectNodeChanges (syncDgNodesToSupabase.ts:298-301) treats them as brand-new files (returning ["title", "content"]), so they end up in changedNodeInstances. They then flow into:
upsertNodesToSupabaseAsContentWithEmbeddings(line 444) — creating Content rows and fetching embeddings for nodes that belong to another spacediscourseNodeInstanceToLocalConceptviaconvertDgToSupabaseConcepts(line 452) — creating duplicate Concept rows in the user's space
Root Cause and Impact
The intent of passing includeImported: true was to populate allNodesById in convertDgToSupabaseConcepts so that relationInstanceToLocalConcept can look up imported nodes and use their importedFromRid for local_reference_content. However, the same allNodes array is also fed into buildChangedNodesFromNodes, which has no filter for imported nodes.
Impact: Every sync (when relationsOnly is false) will re-upload all imported nodes as Content with embeddings (wasting API calls and storage), and create duplicate Concept entries in the user's own space for nodes that are supposed to be cross-space references. This could also cause unique constraint violations on (space_id, name) if the imported node's file path collides with an existing concept name.
The fix should separate the two uses: pass only non-imported nodes to buildChangedNodesFromNodes, while passing all nodes (including imported) to convertDgToSupabaseConcepts for the allNodesById lookup.
(Refers to lines 426-434)
Prompt for agents
In apps/obsidian/src/utils/syncDgNodesToSupabase.ts, in the syncAllNodesAndRelations function (lines 406-469), the allNodes variable (line 426) now includes imported nodes via collectDiscourseNodesFromVault(plugin, true). This is needed for convertDgToSupabaseConcepts (line 452-459) but NOT for buildChangedNodesFromNodes (line 430-434). Fix by collecting two sets of nodes:
1. allNodes (with imported) for passing to convertDgToSupabaseConcepts
2. localNodes (without imported) for passing to buildChangedNodesFromNodes
For example, replace lines 426-434 with something like:
const allNodes = await collectDiscourseNodesFromVault(plugin, true);
const localNodes = allNodes.filter(n => !n.frontmatter.importedFromRid);
const changedNodeInstances = relationsOnly
? []
: await buildChangedNodesFromNodes({
nodes: localNodes,
supabaseClient,
context,
});
This ensures imported nodes are available in allNodesById for relation reference resolution but are not synced as Content/Concepts to the user's own space.
Was this helpful? React with 👍 or 👎 to provide feedback.
21b378d to
2df8ee0
Compare
There was a problem hiding this comment.
Devin Review found 2 new potential issues.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 18 additional findings in Devin Review.
| resourceIds.add(triple.id); | ||
| } | ||
| }); | ||
| if (!resourceIds) return; |
There was a problem hiding this comment.
🟡 !resourceIds is always false because a Set object is always truthy
The guard if (!resourceIds) return; at line 204 is intended to skip the Supabase upsert when there are no resource IDs to publish. However, resourceIds is a Set<string> (initialized on line 165), and a Set object is always truthy in JavaScript regardless of its size. This means the check never triggers an early return.
Root Cause and Impact
The condition !resourceIds evaluates the truthiness of the Set object itself, not its size. Since any object (including an empty Set) is truthy in JavaScript, !resourceIds is always false.
The correct check should be if (resourceIds.size === 0) return;.
Impact: When no matching relation triples are found, the function proceeds to call client.from("ResourceAccess").upsert(...) with an empty array ([...resourceIds.values()] yields []). Depending on how Supabase handles an empty upsert, this could be a no-op or could cause an unnecessary network request. More importantly, the function then proceeds to loadRelations, iterate over relations, and potentially call saveRelations — all unnecessary work.
| if (!resourceIds) return; | |
| if (resourceIds.size === 0) return; |
Was this helpful? React with 👍 or 👎 to provide feedback.
2df8ee0 to
de9dff5
Compare
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 18 additional findings in Devin Review.
https://linear.app/discourse-graphs/issue/ENG-1475/export-relations-to-imported-nodes
https://www.loom.com/share/527b312dca4f441083f9ae25987e5f57
Code note: the main work was to send the original node's koi identifier in upsertConcept, and have that function resolve the identifier to the original node.