Skip to content

Commit 251b492

Browse files
authored
refine kafka schema registery subject name (#593)
1 parent 5180b4e commit 251b492

1 file changed

Lines changed: 35 additions & 20 deletions

File tree

docs/kafka-schema-registry.md

Lines changed: 35 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2,43 +2,58 @@
22

33
## Read Messages in Protobuf or Avro Schema {#read}
44

5-
To read Kafka data in Protobuf or Avro schema with a schema registry, you can create an external stream with `kafka_schema_registry_url` settings, e.g.
5+
To consume Kafka data using **Avro** or **Protobuf** via a Schema Registry, create an external stream using the `kafka_schema_registry_url` and associated settings.
66

77
```sql
88
CREATE EXTERNAL STREAM my_stream (
9-
-- columns goes here ...
9+
-- Define columns matching your schema (optional)
10+
column_name data_type,
11+
...
1012
) SETTINGS
1113
type = 'kafka',
12-
brokers = '...',
13-
topic = '...',
14-
data_format = '..',
14+
brokers = 'localhost:9092',
15+
topic = 'my_topic',
16+
data_format = 'Avro', -- or 'ProtobufSingle'
17+
subject_name_strategy = '..',
18+
schema_subject_name = '..',
1519
kafka_schema_registry_url = 'http://url.to/my/schema/registry',
1620
kafka_schema_registry_credentials = 'API_KEY:API_SECRET',
17-
kafka_schema_registry_skip_cert_check = true|false,
18-
kafka_schema_registry_private_key_file = '..',
19-
kafka_schema_registry_cert_file = '..',
21+
kafka_schema_registry_skip_cert_check = [true|false],
2022
kafka_schema_registry_ca_location = '..';
2123
```
2224

23-
Please note:
25+
### Key Configuration Details
26+
27+
**1. Authentication & URL**
28+
`kafka_schema_registry_url`: Must include the protocol (`http://` or `https://`).
29+
`kafka_schema_registry_credentials`: Optional. Omit if your registry does not require an API key or Basic Auth.
2430

25-
1. `kafka_schema_registry_credentials` is optional. Skip this if the schema registry server doesn't require authentication.
26-
2. Make sure to add `http://` or `https://` in the `kafka_schema_registry_url`. In Proton 1.5.3 or above, self-signed HTTPS certification is supported.
27-
1. One solution is to set `kafka_schema_registry_skip_cert_check` to `true`. This will completely skip the TLS certification verification. In this case, you don't need to specify the certification files.
28-
2. A more secure solution is to keep the default value of `kafka_schema_registry_skip_cert_check`, which is false. Omit this setting and specify the following 3 settings:
29-
1. `kafka_schema_registry_private_key_file`: the file path to the private key file used for encryption. Please use absolute file path and make sure Proton can access this file. If you are using Kubernetes or Docker, please mount the file systems properly.
30-
2. `kafka_schema_registry_cert_file`: the file path to the certificate file (in PEM format). If the private key and the certificate are stored in the same file, this can be empty if `kakfa_schema_registry_private_key_file` is specified.
31-
3. `kafka_schema_registry_ca_location`: the path to the file or directory containing the CA/root certificates.
31+
**2. TLS/SSL Security Options**
32+
You can configure secure connections in two ways:
33+
- **Basic (Insecure)**: Set `kafka_schema_registry_skip_cert_check = true`. This bypasses TLS verification and is recommended for local testing only.
34+
- **Advanced (Secure)**: Set `kafka_schema_registry_skip_cert_check = false` (default) and provide:
35+
- `kafka_schema_registry_ca_location`: Path to the CA/root certificates.
36+
- `kafka_schema_registry_private_key_file`: Absolute path to your private key. Ensure Proton has read permissions (mount correctly if using Docker/K8s).
37+
- `kafka_schema_registry_cert_file`: Path to the PEM certificate. Can be empty if the certificate is bundled with the private key.
3238

33-
3. Make sure you define the columns matching the fields in the Avro schema. You don't have to define all top level fields in Avro schema as columns in the stream. For example, if there are 4 fields in Avro schema, you can choose only 2 of them as columns in the external stream. But make sure the data types match.
34-
4. `data_format` can be `Avro`, or `ProtobufSingle`.
35-
5. Schema reference is not supported yet.
39+
**Subject Name Strategies**
40+
41+
The `subject_name_strategy` determines how the stream looks up schemas in the registry.
42+
43+
| Strategy | Behavior | Derived Subject Name | Typical Use Case |
44+
| :--- | :--- | :--- | :--- |
45+
| **TopicNameStrategy** | **Default.** Assumes one schema per topic. | `<topic>-value` | Standard topics where every message follows the same structure. |
46+
| **RecordNameStrategy** | Supports **mixed schemas** in one topic. | `schema_subject_name` | Consuming a specific record type from a stream containing multiple Avro types. `schema_subject_name` is usually full qualified record name like `com.x.y.z.RecordA` |
47+
| **TopicRecordNameStrategy** | Scopes record names to a specific topic. | `<topic>-<schema_subject_name>` | Mixed topics where you need to distinguish between same-named records in different environments. Consuming a specific record type from a stream containing multiple Avro types. `schema_subject_name` is usually full qualified record name like `com.x.y.z.RecordA`|
3648

3749
:::info
50+
Note on Selective Consumption: When using `RecordNameStrategy` or `TopicRecordNameStrategy`, the external stream specifically targets the Schema ID associated with your provided `schema_subject_name`.
3851

39-
For examples to read Avro message in various Kafka API compatitable message platforms, please check [this doc](/tutorial-sql-read-avro).
52+
1. Automatic Filtering: Any records in the topic that use a different Schema ID (i.e., different record types) are automatically discarded during decoding.
53+
2. Multi-Type Processing: To consume multiple record types from the same topic, you must create a separate external stream for each unique `schema_subject_name`.
4054

4155
:::
56+
4257
## Write Messages in Avro Schema{#write}
4358

4459
Writing Avro/Protobuf data with schema registry is not supported in Timeplus Proton.

0 commit comments

Comments
 (0)