feat(import): route list import through server /import (write-mode UI + special-value coercion)#2133
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
github-actions
Bot
added
configuration
data-adapter
dependencies
package: types
plugin
apps
tests
labels
Contributor
✅ Console Performance Budget
📦 Bundle Size Report
Size Limits
|
…alue coercion Route the list-import wizard through a single-call server import that coerces special values (boolean/number/date→ISO/select label→code/lookup name→id) from field metadata, instead of guessing on the client. Falls back to the legacy per-row create loop when the connected client lacks data.import. - types: DataSource.importRecords + ImportRequestOptions / ImportRecordsResult / ImportRowResult / ImportWriteMode / ImportFieldMappingEntry (mirror server spec) - data-objectstack: ObjectStackAdapter.importRecords() — forwards raw rows to POST /data/:object/import; throws UNSUPPORTED_OPERATION when client lacks import - plugin-grid ImportWizard: send RAW mapped rows to the server; write-mode (insert/update/upsert) + matchFields + createMissingOptions/runAutomations/ skipBlankMatchKey options UI; per-row result → created/updated report; failed-row CSV re-export; graceful legacy fallback; real errors surfaced - app-shell ObjectView: pass only writable fields (drop formula/summary/ autonumber, readonly, permissions.write:false) as import targets - tests: isUnsupportedImport + buildFailedRowsCsv unit coverage (7)
…in preview validation The preview-step validator only accepted true/false/1/0/yes/no, so boolean cells the server coercion would accept (Chinese 是/否, on/off, y/n, ✓/×) were wrongly flagged as type errors. Mirror the server's BOOL token set.
Replace the two-pass name matcher with a scored, globally-assigned mapper: each column/field pair is scored on exact/normalized name, bilingual (EN/中文) synonyms, token overlap, and content-inferred type compatibility, then fields are assigned by descending confidence so each column and field is used at most once. - importParsers: suggestColumnMappings() + ColumnSuggestion/MappableField types, scoreToConfidence() buckets (high/medium/low), synonym groups. - ImportWizard: autoMapColumns() delegates to the scorer; mapping step shows an 'Auto-matched · <confidence>' hint per column (only while the user's choice still equals the suggestion) and a summary count. i18n keys added. - tests: 7 cases (exact/normalized, synonyms, global de-dup, type gating, incompatible-type discount, unknown column, confidence buckets).
Adds the client-side surface for large-file background imports: - types: DataSource gains createImportJob / getImportJobProgress / getImportJobResults / listImportJobs / cancelImportJob (all optional, feature-detected) plus the CreateImportJobResult / ImportJobProgressInfo / ImportJobResultsInfo / ImportJobSummaryInfo / ListImportJobsOptions / ImportJobStatus types mirroring the server contract. - data-objectstack: adapter delegates each method to the @objectstack/client data namespace, feature-detecting createImportJob and throwing UNSUPPORTED_OPERATION when an older client/server lacks the job routes so callers can gracefully fall back to the synchronous /import path. - tests: delegation + arg-shaping + graceful-degradation coverage.
Files over ASYNC_IMPORT_THRESHOLD (5,000 rows — the sync route's ceiling) are
handed to a server-side background job instead of the blocking /import call:
- ImportWizard creates a job via dataSource.createImportJob, then polls
getImportJobProgress every 800ms, showing live "{processed} of {total}"
progress; on terminal status it pulls getImportJobResults and renders the
same completion screen as the sync path (jobResultToImportResult mirrors the
sync mapping exactly).
- Cancel button aborts the poll loop and best-effort cancels the job server-side.
- Transient poll blips are tolerated (5 consecutive failures before giving up).
- Any unsupported signal (older adapter/client/server -> UNSUPPORTED_OPERATION /
404 / missing method) transparently falls back to the synchronous route.
- Completion screen surfaces a cancelled state and a results-truncated note when
the server caps the per-row report.
- tests: isUnsupportedImportJob + jobResultToImportResult coverage.
Upload step gains a "Download template" button that generates a CSV from the object's fields — a header row of labels (required fields marked with *) plus one type-appropriate example row (dates, emails, numbers, booleans, and the first option value for selects). Not persisted; a convenience starting point. - ObjectView forwards each field's enum options so the example row can seed select columns with a real allowed value. - importParsers normalizeKey/tokenize now strip the * required-marker so a filled-in template round-trips back to the same field on re-import (covered by a round-trip test). - tests: buildImportTemplateCsv (header/example/select/escaping) + round-trip.
Adds a 'Validate data' pre-check in the preview step for small files (<= ASYNC_IMPORT_THRESHOLD) when the data source speaks /import. It sends the exact import payload with dryRun:true so the server coerces and validates every row without persisting, then shows an ok/error summary + a capped list of failing rows. - Extract assembleImportRequest() as a pure helper so the real import and the dry-run send byte-identical payloads (dryRun the only diff). - handleValidate() calls importRecords(dryRun) and stores the result; older adapters/clients without /import degrade silently to the existing client-side cell validation. - A prior dry-run is dropped whenever the payload changes (mapping, write-mode, options, or an inline correction) so the summary never reflects stale data. - i18n + tests for assembleImportRequest (dryRun flag, matchFields gating, option threading).
Adds a 'History' view to the ImportWizard for objects whose data source exposes listImportJobs. From the upload step a header toggle swaps the wizard body for ImportHistoryPanel, which lists prior background import jobs (status badge, processed/total, created/updated/skipped/errors, timestamp) newest-first, with Refresh and — for pending/running jobs — an inline Cancel. - Degrades to an empty state when the adapter lacks listImportJobs (older client/server) and never renders the toggle in that case. - Reuses the async job status enum + cancelImportJob adapter method. - isImportJobActive() pure helper gates cancel/polling; unit-tested. - i18n for history + per-status labels.
For files over ASYNC_IMPORT_THRESHOLD, the preview step already renders only the first PREVIEW_ROW_COUNT rows and the import runs as a background job — but nothing told the user either fact. Add a notice at the preview step for large files that it's previewing the first N of M rows and the import will run in the background. Chosen over a streaming re-architecture (evaluated): the client already parses the whole file and caps the preview, so a sample-preview notice delivers the clarity at near-zero cost while keeping the one-shot payload + 50k ceiling contract intact.
Add an "Undo import" action to the import-history panel for finished jobs the server captured an undo log for — deletes the records the import created and restores updated records to their pre-import values. - types: DataSource.undoImportJob(jobId) + ImportJobUndoResult; undoable/revertedAt on progress + summary DTOs. - data-objectstack: undoImportJob adapter with feature-detection + UNSUPPORTED_OPERATION fallback for older clients. - ImportHistoryPanel: per-row Undo button (confirm dialog, busy state) gated on isImportJobUndoable(); shows an "Undone" marker once reverted. - i18n + isImportJobUndoable gating tests.
The import wizard resolves grid.import.* keys from @object-ui/i18n and falls
back to an embedded English map when a key is missing. The zh locale only
carried the original ~54 keys, so the newer features (download template,
server dry-run validate, write-mode/match options, large-file/async import,
import history, and job-level undo) rendered in English under zh.
- Add the 57 missing grid.import.* keys to zh.ts, reaching full parity with
the embedded English fallback.
- Rename two colliding sub-namespaces so they can be expressed as nested
i18next objects (a key cannot be both a string and a parent):
grid.import.status.<s> -> grid.import.jobStatus.<s> (job status labels)
grid.import.writeMode.<m> -> grid.import.writeModeOpt.<m> (write-mode options)
The bare grid.import.status (column header) and grid.import.writeMode
(option-group label) keep their names.
Verified in the live app-crm harness under ?lang=zh: history panel, status
badges, created/updated counts, and the undo button all render in Chinese.
The wizard only ran a background import job when rows > ASYNC_IMPORT_THRESHOLD (5000), but the server only captures undo state when rows <= 5000. The two ranges never overlapped, so an undoable job was unreachable through the UI. Add a 'Import in the background' checkbox (shown when the data source supports import jobs and the file is under the async threshold) that routes a small import through the job path, producing an undoable job that appears in History and can be rolled back. English fallback + zh translations included. Verified live (UI + app-crm backend): 3-row import with the toggle created an undoable job; Undo via History deleted the 3 records (42 -> 39).
Playwright spec driving the real Import Wizard UI against a real backend end to end: upload a 3-row CSV → enable the background-import toggle → run → open History → Undo, asserting backend record counts move baseline → +3 → baseline and the job row flips to reverted. Regression guard for the undo-reachability gap fixed in the prior commit. Uses the standalone import harness (packages/plugin-grid/demo) via IMPORT_HARNESS_ORIGIN; the single spec skips (does not fail) when that origin isn't serving /live.html, so it's CI-safe. Run: pnpm test:e2e:import-harness.
Drives the actual product path a user takes — log in, open the crm_lead list view (ObjectView), click the real toolbar Import button, upload a CSV, opt into a background import, run it, then Undo from History — asserting backend record counts on both sides (baseline → +3 → baseline). - ImportWizard: stable testids on the Next / Import / Close footer buttons so the flow is language-agnostic (the real console renders zh labels). - apps/console/vite.config.ts: opt-in OBJECTSTACK_CLIENT_DIST override that aliases @objectstack/client to a locally built, import-job-capable client (the published 11.2.0 predates data.createImportJob), widening server.fs .allow so Vite can serve it. Inert unless the env var is set. - playwright.import-console.config.ts + e2e/import-console spec, gated on IMPORT_CONSOLE_LIVE=1 and skipped when the backend lacks import-job routes, so an unconfigured CI run reports skipped, not failed. - package.json: test:e2e:import-console script.
…o-639489 # Conflicts: # packages/data-objectstack/src/index.ts # packages/types/src/data.ts # packages/types/src/index.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景 / Problem
列表导入向导原来是逐行
create,并在客户端猜测特殊值(布尔/日期/select/lookup)。这既重复又不可靠 —— lookup 名称→id 根本没法在前端解析。方案 / Solution
改走单次服务端导入
POST /data/:object/import,前端只发原始映射后的行,由服务端统一强转特殊值并按 writeMode 路由每行。依赖框架侧 objectstack-ai/framework#2505(服务端 coercion +client.data.import)。改动
DataSource.importRecords+ImportRequestOptions/ImportRecordsResult/ImportRowResult/ImportWriteMode/ImportFieldMappingEntry(镜像服务端 spec)ObjectStackAdapter.importRecords()—— 把原始行转发到/data/:object/import;当连接的@objectstack/client没有data.import时抛UNSUPPORTED_OPERATIONImportWizard:matchFields勾选 +createMissingOptions/runAutomations/skipBlankMatchKey开关_error列 + BOM)create;真实服务端错误直接上报,不静默重试(避免重复导入)ObjectView:只把可写字段作为导入目标(剔除 formula/summary/autonumber、readonly、permissions.write:false)isUnsupportedImport+buildFailedRowsCsv单元覆盖(7)兼容性
objectui 当前锁定已发布的
@objectstack/client ^11.2.0(尚无data.import),因此在框架 PR 发布新 client 之前,向导会自动走逐行 create 回退,不会报错。等 client 升级后自动切到服务端/import。验证 / Verification
pnpm --filter @object-ui/plugin-grid exec vitest run→ 102 passed(含 7 个新增)pnpm --filter '@object-ui/app-shell...' build→ tsc 通过(ObjectView 改动编译干净)未做(后续 P1/P2)
大文件异步任务 + 进度 + 5万上限、导入历史、真正的钩子开关落地;任务级 undo、字段类型自动探测、xlsx 优化。