Skip to content

Commit f62c87b

Browse files
committed
further edits
1 parent 189cb22 commit f62c87b

2 files changed

Lines changed: 44 additions & 121 deletions

File tree

CHANGES.md

Lines changed: 17 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -44,21 +44,18 @@ Complete refactor of the networking layer using a bespoke event-loop supporting
4444
* connection: short-circuit send/recv when closed (#2967)
4545

4646
### Transports and Proxies
47-
Built-in support for SSL, SOCKS5, and HTTP CONNECT proxy connections, plus sensible defaults for modern TLS.
4847
* Default SSLContext -> PROTOCOL_TLS_CLIENT; minimum version TLS 1.2 (#2807)
4948
* Enable TCP KeepAlive as default socket option (#2904)
5049
* kafka.net: HTTP CONNECT proxy support (RFC 7231 s4.3.6) (#2990)
5150

5251
### SASL Authentication
53-
* kafka.net: SASL authentication support (#2814, #2886, #2994, #3026)
5452
* kafka.net: Validate SASL/SCRAM iterations (#3026)
5553
* SASL: Prefer node hostname to IP address when building mechanisms (#3003)
5654

5755
### Timeouts and Connection Management
5856
Per-request and per-stage timeouts replace the old single client-wide timeout.
5957
* KIP-601: support socket_connection_setup_timeout_ms w/ exponential backoff (#3027)
6058
* kafka.net: Bootstrap if needed for metadata refresh (#2825)
61-
* kafka.net.manager: Close before stopping IO thread; fail pending bootstrap (#2934)
6259
* kafka.net: Raise UnknownBrokerIdError when connection fails because node_id is not in metadata (#2876)
6360

6461
### Futures and Wakeups
@@ -69,7 +66,7 @@ The `Future` primitive gains `__await__` and a faster slotted implementation; cr
6966
* WakeupNotifier primitive for cross-thread/task wakeups (#2925, #2933)
7067

7168
### Concurrency and Error Handling
72-
Defensive checks throughout the event loop and transport stack: improved socket I/O error handling, RuntimeErrors on misuse of the IO thread, and lock-based detection of concurrent access.
69+
Defensive checks throughout the kafka.net event loop and transport stack: improved socket I/O error handling, RuntimeErrors on misuse of the IO thread, and lock-based detection of concurrent access.
7370
* kafka.net.selector: Break scheduled heapq ties when tasks share scheduled_at (#3007)
7471
* kafka.net.selector: Monitor slow tasks with slow_task_threshold_secs (#2946)
7572
* kafka.net.selector: Use threading.Lock() to detect concurrent access to poll() (#2945)
@@ -80,31 +77,25 @@ Defensive checks throughout the event loop and transport stack: improved socket
8077
* kafka.net: Catch unhandled exceptions in IO thread (#2970)
8178
* kafka.net: Improve error handling on sock read/write (#2995)
8279
* kafka.net.transport: Close connection on socket write error (#2973)
83-
* kafka.net.transport: Fix double EVENT_WRITE register on socket (#2972)
84-
* kafka.net.transport: Handle OSError on sock when constructing __str__ (#2926)
85-
* kafka.net: Fix notify/wakeup race w/ pending flag (#2979)
86-
* kafka.net: Drop manager.poll(); call net.poll directly (#2948)
8780
* kafka.net: Check locks in _poll_once; add net.drain() (#2949)
8881

8982
## Protocol
9083

91-
The a new JSON-schema-based dynamic protocol generator now replaces the legacy hand-written protocol classes (moved to `kafka.protocol.old`).
84+
A new JSON-schema-based dynamic protocol generator now replaces the legacy hand-written protocol classes (moved to `kafka.protocol.old`).
9285

9386
### Dynamic Protocol Classes
94-
Protocol classes are now generated from the upstream Apache Kafka JSON schemas, with type stubs for IDE support and a registry-based lookup for headers and responses.
87+
Protocol classes are now generated from the upstream Apache Kafka JSON schemas.
9588
* Dynamic protocol classes using upstream json schemas (#2727, #2745, #2779, #2782, #2787, #2810)
9689
* Add .pyi type annotation stubs for generated protocol classes (#2784)
9790
* Migrate all internal usage to new protocol classes (#2764, #2765, #2766, #2767, #2768, #2772)
9891
* Refactor treatment of versioned ApiMessage classes (#2739)
9992
* Lookup request header -> response class via ResponseClassRegistry (#2730)
100-
* Combine broker_version/api_versions -> versions.py (#2913)
10193
* Manual json schema edits to restore fields dropped in Apache 4.0 (#2738)
10294

10395
### New Types and Messages
10496
* Add UUID protocol type (#2703)
10597
* Add UnsignedInt16 type
106-
* KIP-893: support nullable structs; add DescribeTopicPartitions (#2889)
107-
* Add DescribeQuorum protocol message
98+
* KIP-893: support nullable structs (#2889)
10899
* Replace kafka.structs BrokerMetadata/PartitionMetadata with MetadataResponse structs (#2794)
109100
* Add ConsumerProtocol data schemas (#2754)
110101
* Add StickyAssignorUserData json schema (#2755)
@@ -133,7 +124,7 @@ Broker version inference is consolidated into a single `BrokerVersionData` helpe
133124

134125
### BrokerVersionData
135126
* BrokerVersionData: consolidated version checks (#2795)
136-
* BrokerVersionData: infer up to 4.2 (#2835, #2836)
127+
* BrokerVersionData: infer up to 4.3 (#2835, #2836, #3032)
137128
* BrokerVersionData: support request min/max version (#2868)
138129
* BrokerVersionData: Fix IncompatibleBrokerVersion errors; add __str__ (#2804)
139130

@@ -163,7 +154,7 @@ All consumer network I/O now flows through the shared kafka.net IO thread; `cons
163154
* Consumer: send all requests from net io thread (#2980)
164155
* Consumer: simplify poll() with fetcher.fetch_records (#2960)
165156
* Consumer: drop poll loop optimizations for pending offset resets and rejoins (#2959)
166-
* Consumer: _update_fetch_positions -> _refresh_committed_offsets; dont poll in position() (#2958)
157+
* Consumer: `_update_fetch_positions` -> `_refresh_committed_offsets`; dont poll in position() (#2958)
167158
* Consumer: convert fetcher reset_offsets/send_list_offsets_requests to async def
168159
* Consumer: use new proto attrs in Fetcher (ListOffsets/Fetch) (#2923)
169160

@@ -203,26 +194,22 @@ All consumer network I/O now flows through the shared kafka.net IO thread; `cons
203194
KafkaProducer gains a sticky partitioner (KIP-480), enabled-by-default idempotence (KIP-679), tightened transaction handling, and a faster send/encode path.
204195

205196
### Transactions and Idempotence
206-
Producer epoch bumping, transaction-group metadata, and stricter ordering/error semantics bring the transactional producer in line with the Java client.
207197
* KIP-360 (pt1): Add transaction manager state and helper methods (#2852)
208198
* KIP-360 (pt2): Bump producer epoch with InitProducerIdRequest v3 on 2.5+ brokers (#2853)
209-
* KIP-447: Include group metadata in TxnOffsetCommitRequest (#2984)
210-
* KIP-447: Fixups to align with java client (#2988)
199+
* KIP-447: Include group metadata in TxnOffsetCommitRequest (#2984, #2988)
211200
* KIP-654: Txn Producer aborts with non-fatal TransactionAbortedError (#3010)
212201
* KAFKA-5793: Tighten up semantics of OutOfOrderSequenceNumber (pt1) (#2843)
213202
* Enable producer idempotence with max_in_flight_requests_per_connection > 1 (#2841)
214203
* Producer: Enforce guaranteed message order when idempotence_enabled (#2937)
215204
* Producer: treat InvalidProducerEpochError as ProducerFencedError (#2885)
216205
* Producer: Improve transaction manager retriable error handling (#2884)
217206
* Producer: Use new-style request construction in Txn Manager (#2983)
218-
* Reuse InitProducerIdHandler for idempotent producer
219207

220208
### Partitioning
221-
* KIP-480: StickyPartitioner for KafkaProducer (#2982)
222-
* KIP-480: Fixup StickyPartitioner edge-cases (#2989)
209+
* KIP-480: StickyPartitioner for KafkaProducer (#2982, #2989)
223210

224211
### Batching and Send Path
225-
Oversized batches now split-and-resend instead of failing, and the hot send path avoids redundant validation and buffer copies.
212+
Split-and-resend oversized batches instead of failing; avoid redundant validation and buffer copies on hot send-path.
226213
* KIP-126: Allow KafkaProducer to split and resend oversized batches (#2839)
227214
* Refactor producer.send / _wait_on_metadata for fast path (#2850)
228215
* Only ensure_valid_topic_name new topics on send (#2849)
@@ -235,18 +222,15 @@ Oversized batches now split-and-resend instead of failing, and the hot send path
235222

236223
## Admin Client
237224

238-
KafkaAdminClient has been split into focused mixin classes (cluster, topics, configs, groups, ACLs, log dirs, …) and its request-sending path has been converted to async coroutines running on the shared IO thread. Adds first-class support for several KIPs along with a number of new APIs.
225+
Split KafkaAdminClient into focused mixin classes (cluster, topics, configs, groups, ACLs, log dirs, etc), and convert request-sending path to async def methods that run on the kafka.net IO thread. Support for new KIPs using new protocol stack.
239226

240227
### Refactor and Async Migration
241-
The admin client is now async end-to-end, with cached coordinator lookups and a mixin layout that mirrors the broker's logical resource groups.
242-
* Admin: split into functional mixin classes (#2873)
228+
The admin client interface remains sync but wraps a fully-async internal api (does not support asyncio yet). Adds cached coordinator lookups and a mixin structure to separate logical resource groups.
229+
* Admin: split into functional mixin classes (#2873, #2877, #2882)
243230
* Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871)
244-
* Admin: refactor _send_request_to_controller error handling (#2751)
245-
* Admin mixin fixes and shuffling method ownership (#2877)
246-
* Admin: rename MetadataAdminMixin -> ClusterAdminMixin; move log dirs under cluster (#2882)
231+
* Admin: refactor `_send_request_to_controller` error handling (#2751)
247232

248233
### KIP Support
249-
* KIP-516: Initial topic id (uuid) support (#3031)
250234
* KIP-699: FindCoordinatorRequest v4 -- multi-group support (#3025)
251235
* KIP-709: OffsetFetch v8 -- use batch interface when available (#3024)
252236

@@ -261,7 +245,7 @@ The admin client is now async end-to-end, with cached coordinator lookups and a
261245
* Admin: Improve alter_configs w/ filters and missing keys (#2899)
262246

263247
### Topics and Partitions
264-
* Admin: Accept topic id (uuid) in _get_cluster_metadata and describe_topics
248+
* KIP-516: Support topic id (uuid) for describe topics (#3031)
265249
* Admin: re-use config processing for CreateTopicsResponse
266250
* Admin: wait_for_topics() and create_topics() wait_for_metadata option (#2856)
267251
* Admin: list_partition_offsets (#2894)
@@ -283,7 +267,7 @@ The admin client is now async end-to-end, with cached coordinator lookups and a
283267

284268
## CLI
285269

286-
The CLI gains shared parser machinery, SASL/SSL connection support across all subcommands, and several new admin subcommands (acls, configs alter, users).
270+
The CLI adds shared parser config, SASL/SSL connection support across all subcommands, and several new admin subcommands (acls, configs alter, users).
287271

288272
### Common Infrastructure
289273
* kafka.cli: common parser args; support sasl/ssl connections (#2887)
@@ -305,8 +289,8 @@ The CLI gains shared parser machinery, SASL/SSL connection support across all su
305289
Small quality-of-life additions to the public API surface.
306290

307291
* Support context manager interface for consumer/producer/admin (#2969)
308-
* Add OffsetSpec / IsolationLevel to kafka imports (#2898)
309292
* Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924)
293+
* Add OffsetSpec / IsolationLevel to kafka imports (#2898)
310294

311295
## Fixes
312296

@@ -344,11 +328,10 @@ A new in-memory `MockBroker` / `MockTransport` enables deterministic protocol-le
344328

345329
* Enable pylint workflow; disable/fix all outstanding errors (#2701)
346330
* Add coverage step to CI (#2786)
347-
* CI test matrix updated to Kafka 4.0-4.2 (#2836)
331+
* CI test matrix updated to Kafka 4.0-4.3 (#2836, #3032)
348332
* Enable manual workflow runs for Python Package
349333
* Add lint-unicode make target; include in make lint test (#2993)
350334
* pyproject updates: no more universal wheels; tested on py3.8+
351-
* Bump version to 3.0.0.dev
352335
* Bump actions/cache from 4 to 5 (#2702)
353336
* Bump actions/upload-artifact from 4 to 7 (#2789)
354337

0 commit comments

Comments
 (0)