Skip to content

Commit 9114935

Browse files
committed
Remove source-specific references from sink and SDK docs
1 parent eaf9dc5 commit 9114935

3 files changed

Lines changed: 13 additions & 19 deletions

File tree

core/connectors/sdk/README.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,9 +106,8 @@ preserve_structure = false
106106
- **`preserve_structure`**: Whether to preserve the original message structure during conversion
107107
- **`field_mappings`**: Mapping of field names for transformation (e.g., `"old_field" = "new_field"`)
108108
- **`unwrap_envelope`**: Extracts a nested JSON field and promotes it as the top-level payload.
109-
Required when a source emits envelope-wrapped records (e.g., Postgres `DatabaseRecord` with
110-
`table_name`, `operation_type`, `data`, etc.) and the downstream sink expects flat JSON
111-
matching the target table schema (e.g., Iceberg, Delta).
109+
Required when a source emits envelope-wrapped records (with metadata fields alongside a nested
110+
data object) and the downstream sink expects flat JSON matching the target table schema.
112111
- **`field`**: The envelope key whose value becomes the new payload (e.g., `"data"`). Must not be empty.
113112

114113
```toml

core/connectors/sdk/src/transforms/unwrap_envelope.rs

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,9 @@ use serde::{Deserialize, Serialize};
2323
/// Configuration for the UnwrapEnvelope transform.
2424
///
2525
/// Extracts a nested JSON field from an envelope object and promotes it
26-
/// to the top-level payload. For example, given a source envelope with
27-
/// shape `{ table_name, operation_type, timestamp, data: { ... },
28-
/// old_data }`, setting `field = "data"` replaces the entire payload
29-
/// with the contents of `data`.
26+
/// to the top-level payload. For example, given an envelope with shape
27+
/// `{ metadata_a, metadata_b, data: { ... } }`, setting `field = "data"`
28+
/// replaces the entire payload with the contents of `data`.
3029
#[derive(Debug, Serialize, Deserialize)]
3130
pub struct UnwrapEnvelopeConfig {
3231
pub field: String,

core/connectors/sinks/iceberg_sink/README.md

Lines changed: 8 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -83,23 +83,19 @@ Example:
8383

8484
## Source Compatibility
8585

86-
The Iceberg sink expects flat JSON where each top-level key maps directly to a column in the
87-
target Iceberg table schema. Sources that wrap row data in an envelope (e.g., the Postgres
88-
source emits `DatabaseRecord` objects with `table_name`, `operation_type`, `timestamp`, `data`,
89-
and `old_data` fields) are **not** directly compatible — the Arrow JSON reader will map envelope
90-
keys to table columns, producing nulls or schema errors.
86+
The Iceberg sink expects **flat JSON** where each top-level key maps directly to a column in the
87+
target Iceberg table schema. Sources that wrap row data in an envelope (with metadata fields
88+
alongside a nested data object) are not directly compatible — the Arrow JSON reader will map
89+
envelope keys to table columns, producing nulls or schema errors.
9190

92-
Use the `unwrap_envelope` transform to extract the inner data before it reaches the sink:
91+
If your source emits envelope-wrapped JSON, use the `unwrap_envelope` transform to extract the
92+
inner data field before it reaches the sink:
9393

9494
```toml
95-
[[streams]]
96-
stream = "postgres_stream"
97-
topic = "postgres_topic"
98-
schema = "json"
99-
10095
[transforms.unwrap_envelope]
10196
enabled = true
10297
field = "data"
10398
```
10499

105-
Alternatively, configure the Postgres source with `payload_column` and `payload_format = "json_direct"` to bypass the envelope entirely — see the Postgres source README for details.
100+
Set `field` to the envelope key that contains the actual row data. See the SDK README and your
101+
source connector's documentation for details on the envelope shape.

0 commit comments

Comments
 (0)