redpanda.adoc

What’s New in Redpanda

This topic includes new content added in version {page-component-version}. For a complete list of all product updates, see the Redpanda release notes. See also:

• redpanda-cloud:get-started:whats-new-cloud.adoc

• Redpanda Cloud vs Self-Managed feature compatibility

Crash recording for improved support

Redpanda now records detailed information about broker crashes to help streamline troubleshooting and reduce time to resolution. Crash reports include information such as a stack trace, exception details, the Redpanda broker version, and the timestamp of when the crash occurred. The recorded crash reports are now automatically collected as part of debug bundles, providing Redpanda customer support with more context to diagnose and resolve issues faster.

New health probes for broker restarts and upgrades

The Redpanda Admin API now includes new health probes to help you ensure safe broker restarts and upgrades. The pre_restart_probe endpoint identifies potential risks if a broker is restarted, and post_restart_probe indicates how much of its workloads a broker has reclaimed after the restart. See also:

• manage:cluster-maintenance/rolling-restart.adoc

• upgrade:rolling-upgrade.adoc

Redpanda Console v3.0.0

The Redpanda Console v3.0.0 release includes the following updates:

New features

Redpanda Console now supports unified authentication and authorization between Console and Redpanda, including user impersonation. This means you can authenticate to Redpanda using the same credentials you use for Redpanda Console.

See console:config/security/authentication.adoc for more information.

Breaking changes

• Authentication and authorization:

  • Renamed the login stanza to authentication.

  • Renamed login.jwtSecret to authentication.jwtSigningKey.

  • Removed the plain login provider.

  • OIDC group-based authorization is no longer supported.

  • Role bindings must now be configured in the authorization.roleBindings stanza (no longer stored in a separate file).

• Schema Registry:

  • Moved from under the kafka stanza to a top-level schemaRegistry stanza.

  • All authentication settings for Schema Registry are now defined under schemaRegistry.authentication.

• Admin API:

  • Authentication for the Redpanda Admin API is now defined under redpanda.adminApi.authentication.

• Serialization settings:

  • Moved kafka.protobuf, kafka.cbor, and kafka.messagePack to a new top-level serde stanza.

  • The kafka.protobuf.schemaRegistry setting is deprecated. Use the top-level schemaRegistry stanza instead.

• Connect:

  • Renamed the connect stanza to kafkaConnect to avoid ambiguity with Redpanda Connect.

• Console settings:

  • Moved console.maxDeserializationPayloadSize to serde.maxDeserializationPayloadSize.

Action required: migrate:console-v3.adoc.

Other changes

The admin panel has been removed from the Redpanda Console UI. To manage users, use the Security page. To generate debug bundles, use the link on the Cluster overview page. To upload a new license, use the link on the Cluster overview page or in the license expiration warning banner.

Iceberg improvements

Iceberg-enabled topics now support the following:

• Custom partitioning for improved query performance.

• Snapshot expiry.

• Dead-letter queue for invalid records.

• Schema evolution, with schema mutations implemented according to the Iceberg specification.

• For Avro and Protobuf data, structured Iceberg tables without the use of the Schema Registry wire format or SerDes. See manage:iceberg/choose-iceberg-mode.adoc for more information.

Protobuf normalization in Schema Registry

Redpanda now supports normalization of Protobuf schemas in the Schema Registry. You can normalize Avro, JSON, and Protobuf schemas both during registration and lookup. For more information, see the Schema Registry overview, and the Schema Registry API reference.

Protobuf well-known types in rpk

Support for Protobuf well-known types is available in rpk when encoding and decoding records using Schema Registry.

SASL/PLAIN authentication

You now can configure Kafka clients to authenticate using SASL/PLAIN with a single account using the same username and password. Unlike SASL/SCRAM, which uses a challenge response with hashed credentials, SASL/PLAIN transmits plaintext passwords. You enable SASL/PLAIN by appending PLAIN to the list of SASL mechanisms.

Pause and resume uploads

Redpanda now supports pausing and resuming uploads to object storage when running Tiered Storage, with no risk to data consistency or data loss. You can use the cloud_storage_enable_segment_uploads property to pause or resume uploads to help you troubleshoot any issues that occur in your cluster during uploads.

Trial license

All new Redpanda clusters automatically receive a trial license valid for 30 days. You can extend this trial for 30 days using the new rpk generate license command.

Metrics

The following metrics are new in this version:

Consumer lag gauges

Redpanda can now expose dedicated consumer lag gauges that eliminate the need to calculate lag manually. These metrics provide real-time insights into consumer group performance and help identify issues. The following metrics are available:

• redpanda_kafka_consumer_group_lag_max: Reports the maximum lag observed among all partitions for a consumer group. This metric helps pinpoint the partition with the greatest delay, indicating potential performance or configuration issues.

• redpanda_kafka_consumer_group_lag_sum: Aggregates the lag across all partitions, providing an overall view of data consumption delay for the consumer group.

See Monitor consumer group lag for more information.

Other metrics

• redpanda_rpc_received_bytes: Reports the number of bytes received from valid requests from the client.

• redpanda_rpc_sent_bytes: Reports the number of bytes sent to clients.

• redpanda_kafka_request_bytes_total: Reports the total number of bytes read from or written to the partitions of a topic.

• redpanda_cloud_storage_paused_archivers: Reports the number of paused archivers.

rpk commands

The following rpk commands are new in this version:

• rpk generate license

• rpk topic analyze

Cluster properties

The following cluster properties are new in this version:

Metrics

• enable_consumer_group_metrics: Enables detailed consumer group metrics collection.

• enable_host_metrics: Enables exporting of some host metrics like /proc/diskstats, /proc/snmp and /proc/net/netstat.

Iceberg integration

• iceberg_backlog_controller_p_coeff: Configures the coefficient for backlog control in Iceberg tables.

• iceberg_default_partition_spec: Sets the default partition specification for Iceberg tables.

• iceberg_disable_snapshot_tagging: Disables snapshot tagging in Iceberg.

• iceberg_invalid_record_action: Specifies the action for handling invalid records in Iceberg.

• iceberg_rest_catalog_authentication_mode: Defines the authentication mode for the Iceberg REST catalog.

• iceberg_rest_catalog_oauth2_server_uri: Specifies the OAuth2 server URI for the Iceberg REST catalog.

• iceberg_target_backlog_size: Sets the target backlog size for Iceberg.

• iceberg_target_lag_ms: Configures the target lag (in milliseconds) for Iceberg.

Log compaction

• log_compaction_adjacent_merge_self_compaction_count: Adjusts the number of self-compaction merges during log compaction.

• min_cleanable_dirty_ratio: Sets the minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition’s log is eligible for compaction in a compact topic.

Raft optimization

• raft_max_buffered_follower_append_entries_bytes_per_shard: Limits the maximum bytes buffered for follower append entries per shard.

• raft_max_inflight_follower_append_entries_requests_per_shard: Replaces the deprecated raft_max_concurrent_append_requests_per_follower to limit in-flight follower append requests per shard.

Tiered Storage

• cloud_storage_enable_remote_allow_gaps: Controls the eviction of locally stored log segments when Tiered Storage uploads are paused.

• cloud_storage_enable_segment_uploads: Controls the upload of log segments to Tiered Storage. If set to false, this property temporarily pauses all log segment uploads from the Redpanda cluster.

TLS configuration

• tls_certificate_name_format: Sets the format of the certificates’s distinguished name to use for mTLS principal mapping.

• tls_enable_renegotiation: Enables support for TLS renegotiation.

Throughput quota

• target_fetch_quota_byte_rate: Configures the fetch quota in bytes per second.

Topic configuration

• topic_partitions_memory_allocation_percent: Adjusts the percentage of memory allocated for topic partitions.

Scheduler improvements

• use_kafka_handler_scheduler_group: Enables the Kafka handler scheduler group.

• use_produce_scheduler_group: Enables the produce scheduler group.

Changes to the default configuration

• storage_read_readahead_count: Reduced default from 10 to 1 to optimize read throughput and minimize unaccounted memory usage, lowering the risk of OOM errors on local storage paths.

• topic_memory_per_partition: Decreased default from 4194304 to 204800

• topic_partitions_per_shard: Increased default from 1000 to 5000 to support larger partition counts per shard.

Client quota properties removed

The following client configuration properties were deprecated in version 24.2.1, and have been removed in this release:

• kafka_client_group_byte_rate_quota

• kafka_client_group_fetch_byte_rate_quota

• target_quota_byte_rate

• target_fetch_quota_byte_rate

• kafka_admin_topic_api_rate

Use rpk cluster quotas to manage client throughput limits based on the Kafka API.

Broker properties

• node_id_overrides: Overrides a broker ID and UUID at broker startup.

Topic properties

• min.cleanable.dirty.ratio: Sets the minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition’s log is eligible for compaction in a compact topic.