You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/pulsar_namespace.md
+89-1Lines changed: 89 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,6 +29,8 @@ The `PulsarNamespace` resource defines a namespace in a Pulsar cluster. It allow
29
29
|`deduplication`| Whether to enable message deduplication for the namespace. | No |
30
30
|`bookieAffinityGroup`| Set the bookie-affinity group for the namespace, which has two sub fields: `bookkeeperAffinityGroupPrimary(String)` is required, and `bookkeeperAffinityGroupSecondary(String)` is optional. | No |
31
31
|`topicAutoCreationConfig`| Configures automatic topic creation behavior within this namespace. Contains settings for whether auto-creation is allowed, the type of topics created, and default number of partitions. | No |
32
+
|`schemaCompatibilityStrategy`| Schema compatibility strategy for this namespace. Controls how schema evolution is handled for topics within this namespace. Options: `AutoUpdateDisabled`, `Backward`, `Forward`, `Full`, `AlwaysCompatible`, `BackwardTransitive`, `ForwardTransitive`, `FullTransitive`. | No |
33
+
|`schemaValidationEnforced`| Controls whether schema validation is enforced for this namespace. When enabled, producers must provide a schema when publishing messages. If not specified, the cluster's default schema validation enforcement setting will be used. | No |
32
34
33
35
Note: Valid time units are "s" (seconds), "m" (minutes), "h" (hours), "d" (days), "w" (weeks).
34
36
@@ -70,6 +72,89 @@ This configuration overrides the broker's default topic auto-creation settings f
70
72
```
71
73
This explicitly disables topic auto-creation for the namespace, overriding any broker-level settings that might enable it.
72
74
75
+
## Schema Management
76
+
77
+
Pulsar provides powerful schema management capabilities at the namespace level, allowing you to control how schema evolution is handled and whether schema validation is enforced. This feature consists of two complementary settings: schema compatibility strategy and schema validation enforcement.
78
+
79
+
### Schema Compatibility Strategy
80
+
81
+
The `schemaCompatibilityStrategy` field controls how Pulsar handles schema evolution for topics within the namespace. This allows you to configure different compatibility requirements for different namespaces based on your use case.
82
+
83
+
#### Available Strategies
84
+
85
+
1. **AutoUpdateDisabled**: Disables automatic schema updates and requires manual schema management. This is the most restrictive strategy, suitable for ultra-stable environments where strict schema control is required and no automatic schema evolution is desired.
86
+
87
+
2. **AlwaysCompatible**: Allows any schema changes without validation. This is the most permissive strategy but may lead to compatibility issues. Suitable for development/testing environments.
88
+
89
+
3. **Backward**: New schema can read data written with the previous schema. This strategy supports consumer-driven schema evolution, such as adding optional fields or removing fields.
90
+
91
+
4. **BackwardTransitive**: New schema can read data written with any previous schema in the chain. This provides long-term backward compatibility across multiple schema versions.
92
+
93
+
5. **Forward**: Previous schema can read data written with the new schema. This strategy supports producer-driven schema evolution, such as adding fields that older consumers can ignore.
94
+
95
+
6. **ForwardTransitive**: Any previous schema can read data written with the new schema. This ensures new data is readable by any older schema version.
96
+
97
+
7. **Full**: Schema changes are both forward and backward compatible. Both new and previous schemas can read data written by either schema. This provides strict compatibility requirements in both directions.
98
+
99
+
8. **FullTransitive**: Schema changes are forward and backward compatible with all schemas. Any schema in the chain can read data written by any other schema in the chain. This provides maximum compatibility guarantees.
100
+
101
+
#### Usage Examples
102
+
103
+
**Development Environment**:
104
+
```yaml
105
+
schemaCompatibilityStrategy: AlwaysCompatible
106
+
```
107
+
108
+
**Production Environment**:
109
+
```yaml
110
+
schemaCompatibilityStrategy: Backward
111
+
```
112
+
113
+
**Critical Systems**:
114
+
```yaml
115
+
schemaCompatibilityStrategy: FullTransitive
116
+
```
117
+
118
+
### Schema Validation Enforcement
119
+
120
+
The `schemaValidationEnforced` field controls whether producers must provide a schema when publishing messages to topics within the namespace.
121
+
122
+
- **When enabled (`true`)**: Producers must provide a schema when publishing messages. Messages without schemas will be rejected. This ensures all data in the namespace has a defined structure and is recommended for production environments where data consistency is critical.
123
+
124
+
- **When disabled (`false`)**: Producers can publish messages with or without schemas. This allows for more flexibility in message publishing and is useful for development/testing environments or legacy integrations.
125
+
126
+
- **Default behavior**: If `schemaValidationEnforced` is not specified, the cluster's default schema validation enforcement setting will be used.
127
+
128
+
### Configuration Examples by Use Case
129
+
130
+
#### Development/Testing Environment
131
+
```yaml
132
+
schemaCompatibilityStrategy: AlwaysCompatible
133
+
schemaValidationEnforced: false
134
+
```
135
+
This configuration allows rapid schema iteration and flexible schema validation for experimentation.
136
+
137
+
#### Standard Production Environment
138
+
```yaml
139
+
schemaCompatibilityStrategy: Backward
140
+
schemaValidationEnforced: true
141
+
```
142
+
This provides a good balance between flexibility and safety, ensuring consumers can handle schema changes while enforcing schema validation for data consistency.
143
+
144
+
#### Mission-Critical Systems
145
+
```yaml
146
+
schemaCompatibilityStrategy: FullTransitive
147
+
schemaValidationEnforced: true
148
+
```
149
+
This configuration provides maximum compatibility guarantees with strict schema validation enforcement.
150
+
151
+
#### Legacy System Integration
152
+
```yaml
153
+
schemaCompatibilityStrategy: ForwardTransitive
154
+
schemaValidationEnforced: false
155
+
```
156
+
This ensures older systems can consume new data while allowing gradual migration without strict schema requirements.
157
+
73
158
## replicationClusters vs geoReplicationRefs
74
159
75
160
The `replicationClusters` and `geoReplicationRefs` fields serve different purposes in configuring replication for a Pulsar namespace:
@@ -144,7 +232,7 @@ Please note the following important points:
144
232
145
233
1. The fields `name` and `bundles` cannot be updated after the namespace is created. These are immutable properties of the namespace.
146
234
147
-
2. Other fields such as `backlogQuotaLimitSize`, `backlogQuotaLimitTime`, `messageTTL`, `maxProducersPerTopic`, `maxConsumersPerTopic`, `maxConsumersPerSubscription`, `retentionTime`, `retentionSize`, and `topicAutoCreationConfig` can be modified.
235
+
2. Other fields such as `backlogQuotaLimitSize`, `backlogQuotaLimitTime`, `messageTTL`, `maxProducersPerTopic`, `maxConsumersPerTopic`, `maxConsumersPerSubscription`, `retentionTime`, `retentionSize`, `topicAutoCreationConfig`, `schemaCompatibilityStrategy`, and `schemaValidationEnforced` can be modified.
148
236
149
237
3. If you want to change the `connectionRef`, ensure that the new PulsarConnection resource exists and is properly configured. Changing the `connectionRef` can have significant implications:
0 commit comments