Update changelog

Author: dpkp

CHANGES.md

@@ -1,37 +1,99 @@
 # 3.0.0 (Unreleased)
 
 Breaking Changes
-* Drop python2 support (#2699, #2774, #2837)
+* Drop python2 support (#2699, #2774, #2837, #2846)
 * Consumer: use assignor instances, not classes (#2775)
+* KIP-679: Update Producer defaults -- enable_idempotence=True, acks='all' (#3013)
+* KIP-735: Increase default consumer session_timeout_ms from 10s to 45s (#3030)
+* Admin: change response shapes to simple dicts (#2883)
+* Admin: groups apis renames; add offsets, members apis (#2892)
+* Admin: rename MetadataAdminMixin -> ClusterAdminMixin; move log dirs under cluster (#2882)
+* Admin: Deprecate NewTopics/NewPartitions in favor of simple dicts (#2869)
+* Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924)
+* KafkaError subclass Exception not RuntimeError (#2932)
+* Eliminate NoBrokersAvailableError (#2942)
+* KafkaProtocolError is not retriable (#2941)
+* Rename api_version_auto_timeout_ms -> bootstrap_timeout_ms; default 30s (#3028)
+* Remove kafka.client_async / kafka.conn legacy modules (#2918)
+
+Networking (kafka.net)
+* kafka.net: Refactored async networking modules (generator-based coroutines) (#2812)
+* kafka.net: SSL transport support (#2813)
+* kafka.net: SASL authentication support (#2814, #2886, #2994, #3026)
+* kafka.net: SOCKS5 proxy support (#2815)
+* kafka.net: HTTP CONNECT proxy support (RFC 7231 s4.3.6) (#2990)
+* kafka.net: Metrics tracking (#2834)
+* kafka.net: Per-request timeouts (#3008)
+* kafka.net: support socket_connection_setup_timeout_ms w/ exponential backoff (KIP-601) (#3027)
+* kafka.net: Network IO thread used by all clients (Admin/Consumer/Producer) (#2965, #2980)
+* kafka.net: Move io_thread and run() from manager to selector (#2961, #2968, #2963)
+* KafkaNetClient: drop-in replacement for KafkaClient using kafka.net (#2816)
+* KafkaNetSocket: interface class for socket and proxy connections (#2992)
+* Future.__await__ support (#2811)
+* Future: __slots__ and callback/errback inlining optimization (#2848)
+* Future: clear callbacks/errbacks when done to avoid reference cycles (#2891)
+* Pass task by value when adding call_soon callback to future (#2826)
+* Default SSLContext -> PROTOCOL_TLS_CLIENT; minimum version TLS 1.2 (#2807)
+* Enable TCP KeepAlive as default socket option (#2904)
+* WakeupNotifier primitive for cross-thread/task wakeups (#2925, #2933)
+* SASL: Prefer node hostname to IP address when building mechanisms (#3003)
+* kafka.net: Validate SASL/SCRAM iterations (#3026)
+* kafka.net.selector: Break scheduled heapq ties when tasks share scheduled_at (#3007)
+* kafka.net.selector: Monitor slow tasks with slow_task_threshold_secs (#2946)
+* kafka.net.selector: Use threading.Lock() to detect concurrent access to poll() (#2945)
+* kafka.net.selector: Track pending tasks to prevent gc before completion (#2950)
+* kafka.net.selector: Support reschedule(when, task); idempotent unschedule (#2939)
+* kafka.net: Raise RuntimeError on concurrent access to net.poll or wakeup() (#2938)
+* kafka.net: Raise RuntimeError on run/call_at/call_soon_threadsafe after closed (#2971)
+* kafka.net: Catch unhandled exceptions in IO thread (#2970)
+* kafka.net: Improve error handling on sock read/write (#2995)
+* kafka.net.transport: Close connection on socket write error (#2973)
+* kafka.net.transport: Fix double EVENT_WRITE register on socket (#2972)
+* kafka.net.transport: Handle OSError on sock when constructing __str__ (#2926)
+* kafka.net: Fix notify/wakeup race w/ pending flag (#2979)
+* kafka.net: Support get_connection timeout, use for check_version (#2943)
+* kafka.net: Drop manager.poll(); call net.poll directly (#2948)
+* kafka.net: Check locks in _poll_once; add net.drain() (#2949)
+* kafka.net.manager: Add call_soon(coro) and run(coro) for sync/async bridge (#2862)
+* kafka.net.manager: Bootstrap is sync/blocking (#2919)
+* kafka.net.manager: Close before stopping IO thread; fail pending bootstrap (#2934)
+* kafka.net: Raise UnknownBrokerIdError when connection fails because node_id is not in metadata (#2876)
+* kafka.net: Bootstrap if needed for metadata refresh (#2825)
+* connection: short-circuit send/recv when closed (#2967)
 
 Protocol
 * Dynamic protocol classes using upstream json schemas (#2727, #2745, #2779, #2782, #2787, #2810)
 * Add .pyi type annotation stubs for generated protocol classes (#2784)
 * Protocol encode/decode optimization with inline compile/exec (#2785)
 * Protocol benchmarks and profiling (make bench-protocol) (#2783)
 * Add UUID protocol type (#2703)
-* Fix TaggedFields encoding/decoding (#2725)
+* Add UnsignedInt16 type
+* KIP-893: support nullable structs; add DescribeTopicPartitions (#2889)
+* Add DescribeQuorum protocol message
+* Fix TaggedFields encoding/decoding (#2725, #2745, #2779)
 * Fix VarInt/VarLong encoding (#2706)
+* Fix CompactBytes encoding of struct data (#2782)
+* Fix compiled encode_into for None arrays (#2790)
+* Validate network frame size (#3019)
+* Validate DataContainer version (#2759)
+* DataContainer.to_dict() helper (#2758, #2872, #2879)
+* Refactor treatment of versioned ApiMessage classes (#2739)
+* Lookup request header -> response class via ResponseClassRegistry (#2730)
+* Store in-flight request headers only for protocol parser (#2723)
+* Combine broker_version/api_versions -> versions.py (#2913)
+* Manual json schema edits to restore fields dropped in Apache 4.0 (#2738)
 * Migrate all internal usage to new protocol classes (#2764, #2765, #2766, #2767, #2768, #2772)
 * Replace kafka.structs BrokerMetadata/PartitionMetadata with MetadataResponse structs (#2794)
 * Add ConsumerProtocol data schemas (#2754)
 * Add StickyAssignorUserData json schema (#2755)
-
-Networking
-* kafka.net: Refactored async networking modules (generator-based coroutines) (#2812)
-* kafka.net: SSL transport support (#2813)
-* kafka.net: SASL authentication support (#2814)
-* kafka.net: SOCKS5 proxy support (#2815)
-* kafka.net: HTTP CONNECT proxy support (RFC 7231 s4.3.6) (#2990)
-* kafka.net: Metrics tracking (#2834)
-* KafkaNetClient: drop-in replacement for KafkaClient using kafka.net (#2816)
-* Future.__await__ support (#2811)
-* Pass task by value when adding call_soon callback to future (#2826)
-* Default SSLContext -> PROTOCOL_TLS_CLIENT; minimum version TLS 1.2 (#2807)
+* Debug log send/recv bytes from protocol parser (#2707)
+* Adjust protocol debug logging; add KAFKA_PYTHON_PROTOCOL_DEBUG_LOG (#2719)
 
 Broker Version Check
 * BrokerVersionData: consolidated version checks (#2795)
 * BrokerVersionData: infer up to 4.2 (#2835, #2836)
+* BrokerVersionData: support request min/max version (#2868)
+* BrokerVersionData: Fix IncompatibleBrokerVersion errors; add __str__ (#2804)
 * Always send ApiVersionsRequest on connect (#2802)
 * Improve ApiVersionsRequest fallbacks (#2803, #2817, #2821)
 
@@ -40,23 +102,154 @@ Client Bootstrap
 * ClusterMetadata: bootstrap_brokers(), set_topics/add_topics/metadata_request (#2805, #2792, #2796, #2797)
 * Move bootstrap_brokers fallback from brokers() to least_loaded_node() (#2809)
 * KafkaClient.least_loaded_node: pass bootstrap_fallback=True if needed (#2830)
+* Consumer: add explicit bootstrap() method (#2975)
+* Admin: explicit bootstrap on __init__ (#2864)
+* Manage metadata refresh logic in ClusterMetadata via attached manager (#2920)
+* kafka.cluster: End refresh loop on close(); refresh loop catches KafkaError (#2935, #2936)
+* Respect metadata backoff in KafkaNetClient (#2854)
 
 Consumer
+* Consumer: use background thread for all network io; drop HeartbeatThread (#2965)
+* Consumer: send all requests from net io thread (#2980)
+* KIP-429: Incremental Cooperative Rebalance (#2999)
+* KIP-429: Add on_partitions_lost rebalance listener hook (#3016)
+* KIP-320: Detect and handle Log Truncation (#2978)
+* KIP-392: Rack-aware fetch from closest replica (#2986, #2996)
+* KIP-447: OffsetFetch v7 set require_stable flag if read_committed (#3023)
+* KIP-559: Bump JoinGroup v7 / SyncGroup v5; verify protocol_type/name (#2998)
+* KIP-595: Fetch v12 support; check for log truncation and leader updates in response (#3021)
+* KIP-602: Support client_dns_lookup in Consumer/Producer/Admin config (#3004)
+* KAFKA-7548: Retain fetched data for paused partitions (#2981)
+* KAFKA-9212: Leader Epoch unreliable before Metadata v9 (#2997)
+* Rebalance Listener: raise errors, block fetches on revoked, pre-revoke unsubscribed (#3018)
+* Consumer: call RebalanceListener on close (#3020)
+* Consumer: Retain subscription TopicPartitionState when possible (#3001)
+* Consumer: default request_timeout_ms 30s; use request-specific timeout for JoinGroup (#3011)
+* Consumer: drop max_version clamp on ListOffsetsRequest (#3012)
+* Consumer: handle retriable errors in offsets_for_times (#3022)
+* Consumer: Improve retriable offset fetch error handling (#2977)
+* Consumer: simplify poll() with fetcher.fetch_records (#2960)
+* Consumer: drop poll loop optimizations for pending offset resets and rejoins (#2959)
+* Consumer: _update_fetch_positions -> _refresh_committed_offsets; dont poll in position() (#2958)
+* Consumer: use new proto attrs in Fetcher (ListOffsets/Fetch) (#2923)
+* Consumer: convert fetcher reset_offsets/send_list_offsets_requests to async def
+* Preserve member id on IllegalGeneration error (#3017)
+* Retry metadata request if tracked topics have retriable errors (#2832)
 * Fix KeyError in KafkaConsumer.committed() (#2710)
 * Fix `Fetcher._fetch_offsets_by_times` retry handling (#2833)
-* Retry metadata request if tracked topics have retriable errors (#2832)
+
+Coordinator
+* Coordinator: convert join_group to async def (#2955)
+* Coordinator: convert offset APIs to async (#2951)
+* Coordinator: convert heartbeat loop to async task (#2929)
+* Coordinator: maybe_leave_group_async; LeaveGroupRequest no max-version (#2930)
+* Coordinator: new OffsetFetch/OffsetCommit construction; use manager.send (#2952)
+* Coordinator: new group proto construction; flexible versions (#2954)
+* Coordinator: Drop ready check before FindCoordinatorRequest (#2964)
+* Coordinator: default request_timeout_ms for close() and commit_offsets_sync() (#2974)
+* Coordinator: drop __del__ (#2931)
+* Dont use OffsetCommitRequest DEFAULT_RETENTION_TIME in coordinator (#2763)
 
 Producer
+* KIP-126: Allow KafkaProducer to split and resend oversized batches (#2839)
+* KIP-360 (pt1): Add transaction manager state and helper methods (#2852)
+* KIP-360 (pt2): Bump producer epoch with InitProducerIdRequest v3 on 2.5+ brokers (#2853)
+* KIP-447: Include group metadata in TxnOffsetCommitRequest (#2984)
+* KIP-447: Fixups to align with java client (#2988)
+* KIP-480: StickyPartitioner for KafkaProducer (#2982)
+* KIP-480: Fixup StickyPartitioner edge-cases (#2989)
+* KIP-654: Txn Producer aborts with non-fatal TransactionAbortedError (#3010)
+* KAFKA-5793: Tighten up semantics of OutOfOrderSequenceNumber (pt1) (#2843)
+* Enable producer idempotence with max_in_flight_requests_per_connection > 1 (#2841)
+* Producer: Enforce guaranteed message order when idempotence_enabled (#2937)
+* Producer: treat InvalidProducerEpochError as ProducerFencedError (#2885)
+* Producer: Improve transaction manager retriable error handling (#2884)
+* Producer: Use new-style request construction in Txn Manager (#2983)
+* Refactor producer.send / _wait_on_metadata for fast path (#2850)
+* Only ensure_valid_topic_name new topics on send (#2849)
+* Avoid unnecessary bytes/bytearray copies on send (#2845)
+* MemoryRecordsBuilder.build() returns bytearray buffer not bytes copy (#2842)
+* Drop PartitionResponse namedtuple; consolidate response and error handling (#2851)
+* Reuse InitProducerIdHandler for idempotent producer
 * Fixup Sender error class logging (#2828)
 
+Admin Client
+* KIP-516: Initial topic id (uuid) support (#3031)
+* KIP-699: FindCoordinatorRequest v4 -- multi-group support (#3025)
+* KIP-709: OffsetFetch v8 -- use batch interface when available (#3024)
+* Admin: split into functional mixin classes (#2873)
+* Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871)
+* Admin: describe_metadata_quorum (#2914)
+* Admin: alter_log_dirs (#2912)
+* Admin: cluster features describe/update (#2908)
+* Admin: cluster get_broker_version_data / api_versions (#2903)
+* Admin: list_config_resources (requires 4.1+) (#2900)
+* Admin: support incremental alter configs (#2901)
+* Admin: Improve alter_configs w/ filters and missing keys (#2899)
+* Admin: list_partition_offsets (#2894)
+* Admin: include latest offsets and lag in list_group_offsets; add reset_group_offsets (#2897)
+* Admin: support group state/type filters for list_groups (#2910)
+* Admin: add extended group reset options (#2911)
+* Admin: Expand partitions api support (#2890)
+* Admin: Cleanup alter_partition_reassignments (#3002)
+* Admin: re-use config processing for CreateTopicsResponse
+* Admin: Accept topic id (uuid) in _get_cluster_metadata and describe_topics
+* Admin: wait_for_topics() and create_topics() wait_for_metadata option (#2856)
+* Admin: retry delete_records / list_partition_offsets on NotLeaderForPartitionError (#2976)
+* Admin: Fix describe_log_dirs, accept optional topics/brokers (#2881)
+* Admin: refactor _send_request_to_controller error handling (#2751)
+* Admin mixin fixes and shuffling method ownership (#2877)
+* Admin: Dont return MemberToRemove as key in remove_group_members dict result (#2893)
+
+CLI
+* kafka.cli: common parser args; support sasl/ssl connections (#2887)
+* kafka.cli: common configuration for logging and connect kwargs (#2906)
+* admin cli: acls, users, configs alter (#2888)
+* admin cli: refactor admin group/command parsers; consumer/producer option groups (#2909)
+* admin cli: --id support for describe-topics
+* admin cli: close() after running command
+* admin cli: dont print stacktrace for BrokerResponseError or ValueError (#2895)
+* admin cli: catch AttributeError and print_help() (#2880)
+* admin cli: fix describe_configs (#2875)
+* Add --enable-logger/--disable-logger to cli options (#2798)
+* Set default cli log level => CRITICAL (#2760)
+
+Compatibility / Misc
+* Support context manager interface for consumer/producer/admin (#2969)
+* Add OffsetSpec / IsolationLevel to kafka imports (#2898)
+* Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924)
+
 Fixes
 * Fix zstd multi-frame decompression failure (#2717)
 * Use time.monotonic() instead of time.time() for elapsed time calculations (#2714)
+* Fix deprecated log.warn() -> log.warning()
+* REF: Switch to deque.copy() for Python 3 compatibility (#2712)
+
+Tests
+* MockBroker / MockTransport for deterministic protocol tests (#2861, #2902)
+* Organize test files into consumer/ producer/ admin/ directories (#2844)
+* Consolidate consumer integration tests (#2857)
+* Consolidate more pytest fixtures (admin/client) (#2921)
+* Integration fixtures use contextmanager (#2966)
+* Refactor coordinator test fixtures; prefer MockBroker to patched_coord (#2953)
+* Support SSL transports in KafkaFixture (#2806)
+* Simplify interacting with KafkaFixture (client factories, create_topics) (#2808)
+* Use producer/consumer/admin factories in producer_integration tests (#2829)
+* Set api_version for integration tests; fix test_group missing consumers (#2824)
+* Close local fixtures in tests (#2962)
+* Catch exceptions in fixture.open() -> close() (#2907)
+* Create topic on 0.8.2 broker to fix bootstrap (#2896)
+* Reduce timing flakiness in test_group (#3006)
+* Test timeouts: use pytest-timeout method=thread and add faulthandler
 
 Project Infra
 * Enable pylint workflow; disable/fix all outstanding errors (#2701)
 * Add coverage step to CI (#2786)
-* CI test matrix updated to Kafka 4.0-4.2
+* CI test matrix updated to Kafka 4.0-4.2 (#2836)
+* Enable manual workflow runs for Python Package
+* Add lint-unicode make target; include in make lint test (#2993)
+* pyproject updates: no more universal wheels; tested on py3.8+
+* Bump version to 3.0.0.dev
 * Bump actions/cache from 4 to 5 (#2702)
 * Bump actions/upload-artifact from 4 to 7 (#2789)