README.md

The scripts in this directory provide various examples of using Confluent's Python client for Kafka:

• adminapi.py: Various AdminClient operations.
• asyncio_example.py: AsyncIO webserver with Kafka producer.
• consumer.py: Read messages from a Kafka topic.
• producer.py: Read lines from stdin and send them to a Kafka topic.
• eos_transactions.py: Transactional producer with exactly once semantics (EOS).
• avro_producer.py: Produce Avro serialized data using AvroSerializer.
• avro_consumer.py: Read Avro serialized data using AvroDeserializer.
• json_producer.py: Produce JSON serialized data using JSONSerializer.
• json_consumer.py: Read JSON serialized data using JSONDeserializer.
• protobuf_producer.py: Produce Protobuf serialized data using ProtobufSerializer.
• protobuf_consumer.py: Read Protobuf serialized data using ProtobufDeserializer.
• sasl_producer.py: Demonstrates SASL Authentication.
• get_watermark_offsets.py: Consumer method for listing committed offsets and consumer lag for group and topics.
• oauth_producer.py: Demonstrates OAuth Authentication (client credentials).

Additional examples for Confluent Cloud:

• confluent_cloud.py: Produce messages to Confluent Cloud and then read them back again.
• oauth_oidc_ccloud_producer.py: Demonstrates OAuth/OIDC Authentication with Confluent Cloud (client credentials).
• oauth_oidc_ccloud_azure_imds_producer.py: Demonstrates OAuth/OIDC Authentication with Confluent Cloud (Azure IMDS metadata based authentication).
• confluentinc/examples: Integration with Confluent Cloud and Confluent Cloud Schema Registry

venv setup

It's usually a good idea to install Python dependencies in a virtual environment to avoid conflicts between projects.

To setup a venv with the latest release version of confluent-kafka and dependencies of all examples installed:

python3 -m venv venv_examples
source venv_examples/bin/activate
python3 -m pip install confluent_kafka
python3 -m pip  install -r requirements/requirements-examples.txt

To setup a venv that uses the current source tree version of confluent_kafka, you need to have a C compiler and librdkafka installed (from a package, or from source). Then:

python3 -m venv venv_examples
source venv_examples/bin/activate
python3 -m pip install .[examples]

When you're finished with the venv:

deactivate