feat: add jsonb[] type support in export-meta.ts and csv-to-pg parser#853
feat: add jsonb[] type support in export-meta.ts and csv-to-pg parser#853pyramation merged 2 commits intomainfrom
Conversation
- Add 'jsonb[]' to FieldType union - Map '_jsonb' PostgreSQL type to 'jsonb[]' in mapPgTypeToFieldType - Add missing 'fields' column (jsonb[]) to secure_table_provision config - Update grant_privileges from 'jsonb' to 'jsonb[]' in secure_table_provision config Companion to constructive-db Phase 0 provision layer cleanup.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
| fields: 'jsonb[]', | ||
| grant_privileges: 'jsonb[]', |
There was a problem hiding this comment.
🔴 jsonb[] field type has no coercion handler in csv-to-pg parser, producing corrupted SQL output
The PR adds jsonb[] as a valid FieldType and uses it for secure_table_provision.fields and secure_table_provision.grant_privileges, but the downstream csv-to-pg parser (packages/csv-to-pg/src/parse.ts:461-472) has no 'jsonb[]' case in its getCoercionFunc switch statement. This causes jsonb[] values to fall through to the default handler, which calls String(value) on the raw value. Since node-postgres returns jsonb[] columns as JavaScript arrays of parsed objects, String([{"a":1},{"b":2}]) produces "[object Object],[object Object]" — corrupted, unrecoverable data in the generated SQL INSERT statements.
Prompt for agents
The csv-to-pg parser at packages/csv-to-pg/src/parse.ts needs a new case for 'jsonb[]' in the getCoercionFunc function (around line 460, before the default case). The handler should:
1. Get the rawValue from the record
2. Check for null tokens
3. Verify the value is an array
4. JSON.stringify each element of the array
5. Format as a PostgreSQL array literal string, e.g. using the psqlArray helper but with JSON-stringified elements, or a custom approach like: '{' + array.map(el => '"' + JSON.stringify(el).replace(/\\/g, '\\\\').replace(/"/g, '\\"') + '"').join(',') + '}'
6. Return it as a string constant node
Example case to add in packages/csv-to-pg/src/parse.ts after the 'jsonb' case (around line 460):
case 'jsonb[]':
return (record) => {
const rawValue = record[from[0]];
if (isNullToken(rawValue)) {
return makeNullOrThrow(fieldName, rawValue, type, required, 'value is empty or null');
}
if (Array.isArray(rawValue)) {
if (rawValue.length === 0) {
return makeNullOrThrow(fieldName, rawValue, type, required, 'array is empty');
}
const elements = rawValue.map(el => JSON.stringify(el));
const arrayLiteral = psqlArray(elements);
if (isEmpty(arrayLiteral)) {
return makeNullOrThrow(fieldName, rawValue, type, required, 'failed to format array');
}
const val = nodes.aConst({ sval: ast.string({ sval: String(arrayLiteral) }) });
return wrapValue(val, opts);
}
return makeNullOrThrow(fieldName, rawValue, type, required, 'value is not an array');
};
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Companion PR to constructive-db#629, which migrates
secure_table_provision.fieldsandsecure_table_provision.grant_privilegesfromjsonbtojsonb[].This PR updates the export-meta config to match the new column types:
'jsonb[]'to theFieldTypeunion and maps PostgreSQL's internal_jsonbtype name to itgrant_privilegesfrom'jsonb'to'jsonb[]'in thesecure_table_provisionconfigfields: 'jsonb[]'to thesecure_table_provisionconfig (this column existed in the DB but was previously missing from the export config)Updates since last revision
Added a
jsonb[]coercion handler inpackages/csv-to-pg/src/parse.ts(getCoercionFunc) to prevent corrupted SQL output. Without this,jsonb[]values fell through to thedefaulthandler, which calledString(value)on parsed JavaScript arrays — producing[object Object],[object Object]instead of valid PostgreSQL array literals.The new handler:
JSON.stringifys each element, then formats viapsqlArray()into a PG array literal like{"{\"name\":\"col\"}","{\"type\":\"text\"}"}parseJsonas-ismakeNullOrThrowReview & Testing Checklist for Human
jsonb[]coercion produces valid SQL: The newjsonb[]case incsv-to-pg/src/parse.tshas no test coverage. Manually verify thatpsqlArray+escapeArrayElementcorrectly handles JSON strings containing quotes, backslashes, and nested objects — the double-escaping of"inside PG array literals is error-prone. Consider adding a unit test inpackages/csv-to-pg.rawValueis a string (not an array), the handler passes it throughparseJsonwhich returns it as-is. For ajsonb[]column, this would produce a SQL string literal like'[{"a":1}]'— a JSON array, not a PostgreSQL array literal (ARRAY[...]or{...}syntax). Confirm whether PostgreSQL accepts this implicit cast, or if this path needs to produce a proper PG array literal instead.fieldscolumn is safe: Thefieldscolumn was missing from the export config before this PR. Verify this was an oversight (not intentional) and that adding it won't cause issues with existing export data or the Parser.jsonb[]before the export config expects them to be.Notes
buildDynamicFieldsfunction intersects the hardcoded config with actual DB columns, so both the static and dynamic paths now resolve_jsonb→'jsonb[]'rather than the old default of'text'.Link to Devin session: https://app.devin.ai/sessions/5d550def7a314c97854ec12fa09dd4ca
Requested by: @pyramation