ENG-1475 Export relations to imported nodes#817
Conversation
|
Updates to Preview Branch (eng-1475-export-relations-to-imported-nodes) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
ad7af74 to
ce78692
Compare
30e40b2 to
bb38514
Compare
ce78692 to
b87afad
Compare
6784bf7 to
91b4992
Compare
9d6b14b to
1b31245
Compare
cc52d2f to
a6f2d9e
Compare
ce150f3 to
3169b1e
Compare
7489026 to
2df080d
Compare
There was a problem hiding this comment.
Devin Review found 2 new potential issues.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 12 additional findings in Devin Review.
There was a problem hiding this comment.
🔴 Missing legacy importedFromSpaceUri check in collectDiscourseNodesFromPaths allows imported files to be synced
Same root cause as in shouldSyncFile: collectDiscourseNodesFromPaths only checks frontmatter.importedFromRid but not the legacy frontmatter.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault at apps/obsidian/src/utils/getDiscourseNodes.ts:32-38 correctly checks both fields, but this function was not updated consistently.
Root Cause and Impact
This function is used by syncDiscourseNodeChanges (called from FileChangeListener). During the migration window or if migration fails, any file change event on a file that still has only the legacy importedFromSpaceUri will pass through both shouldSyncFile and collectDiscourseNodesFromPaths guards, causing the imported node to be synced to Supabase as a locally-created node. This is a second guard that should also check the legacy field for defense-in-depth.
(Refers to lines 717-720)
Was this helpful? React with 👍 or 👎 to provide feedback.
9121ca2 to
6aa3c39
Compare
2df080d to
21b378d
Compare
21b378d to
2df8ee0
Compare
2df8ee0 to
de9dff5
Compare
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 18 additional findings in Devin Review.
6aa3c39 to
44ac414
Compare
de9dff5 to
0299cbf
Compare
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 18 additional findings in Devin Review.
| END; | ||
| $$; | ||
|
|
||
| CREATE OR REPLACE FUNCTION public.rid_or_local_id_to_concept_db_id(rid VARCHAR, default_space_id BIGINT) |
There was a problem hiding this comment.
i accidentally found a case where this might fail:
- if we clean up the orphaned node, then the stale relation instance won't be able to resolve to any db id -> cause this function to return null -> cause
concept.reference_content-> violate non-null constraints -> unable to upsert ANY concepts.
i discovered this bug after having ran supbase db reset, keeping the vault the same. this means that locally there will be old imported files with frontmatter field importFromRid, that we can't resolve in db. Hence, i received this error:
VM209 plugin:@discou…raph/obsidian:55074 Initial sync failed: Error: upsert_concepts failed: {
"code": "23502",
"details": null,
"hint": null,
"message": "null value in column "reference_content" of relation "Concept" violates not-null constraint"
}
this shouldn't be blocking any syncing operation
There was a problem hiding this comment.
Ok, this is a bit of a headache. I corrected the function to allow partial failures; but it means we cannot use the lastSync time anymore!
I think... we need to look at error values, i.e. negative Ids returned by upsertConcept, and possibly remove the offending node/relations. Let's discuss 1-1.
There was a problem hiding this comment.
Leaving a trail: we will think about error handling in another PR (ENG-1510.)
The issue of lastSync time may not be as pressing, since the nodes with an error probably cannot be inserted anyway.
0299cbf to
b0251a3
Compare
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 18 additional findings in Devin Review.
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 19 additional findings in Devin Review.
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing legacy importedFromSpaceUri check in shouldSyncFile allows imported files to be synced (apps/obsidian/src/utils/fileChangeListener.ts:101-103)
After the migration from importedFromSpaceUri to importedFromRid, shouldSyncFile only checks frontmatter?.importedFromRid but does NOT check the legacy frontmatter?.importedFromSpaceUri. The refactored collectDiscourseNodesFromVault in getDiscourseNodes.ts:32-38 correctly checks both fields:
if (
(frontmatter.importedFromRid || frontmatter.importedFromSpaceUri) &&
includeImported !== true
) { continue; }Root Cause and Impact
The FileChangeListener is initialized immediately (apps/obsidian/src/index.ts:56-57) via this.fileChangeListener.initialize(), while migrateImportedFromFrontMatter runs asynchronously inside the fire-and-forget void initializeSupabaseSync(this) call at apps/obsidian/src/index.ts:47. During the migration window (or if migration fails), files that still have only importedFromSpaceUri in their frontmatter will pass shouldSyncFile, be queued, and eventually be synced to Supabase as locally-created nodes. This could create duplicate concepts or corrupt data in the remote database.
View 18 additional findings in Devin Review.
https://linear.app/discourse-graphs/issue/ENG-1475/export-relations-to-imported-nodes
https://www.loom.com/share/527b312dca4f441083f9ae25987e5f57
Code note: the main work was to send the original node's koi identifier in upsertConcept, and have that function resolve the identifier to the original node.