Add Kafka to Kafka RAW example pipeline configuration

[Ashfaqbs]

This Apache Beam YAML pipeline demonstrates a basic Kafka-to-Kafka message mirroring use case. It reads raw byte messages from a source Kafka topic (i-topic) and writes them directly to a target Kafka topic (o-topic) without any transformation or decoding.

The pipeline is defined using Beam’s YAML DSL and leverages the ReadFromKafka and WriteToKafka transforms. It is configured to use the RAW data format, making it suitable for scenarios where the message structure is opaque or processing is deferred downstream.

This example serves as a minimal reference for validating Kafka I/O configurations and can be extended for more complex streaming dataflows. It is compatible with runners that support Kafka I/O, including Direct Runner and Flink Runner.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[robertwb]

Looks like this is failing because we are trying to test it (and it has no output asserts nor the requisite kafka topics set up). You could explicitly skip this particular example at https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/testing/examples_test.py#L310 (though it might be good to at least ensure that it parses).

The RAT failure is because explicit license headers are required on all files, see for example https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/wordcount_minimal.yaml that you could copy over. (A brief description of this pipeline would be good as well.)

[Ashfaqbs]

assign @robertwb

[Ashfaqbs]

Hi Robert

As discussed via email, this PR contributes a minimal Kafka-to-Kafka example in RAW byte format using Beam's YAML DSL. It's a simple pass-through pipeline that reads from an input Kafka topic and writes to an output topic without any transformation or decoding. This should serve as a helpful reference for users validating Kafka I/O integration with Beam YAML.

✅ Apache license header and description are included
✅ File is renamed to kafka-to-kafka_example.yaml to skip automated test execution (as the pipeline has no assertions and requires Kafka infra)
✅ All non-functional checks (RAT, Lint, Formatters, Docker, etc.) have passed
❌ The Python PreCommit test failures are unrelated and expected — they attempt to run YAMLs without assertable output

Let me know if any further changes are needed. Thanks again for your time and review!

[derrickaw]

assign set of reviewers

[github-actions]

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[github-actions]

Reminder, please take a look at this pr: @claudevdm

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @liferoad for label python.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[liferoad]

please fix the failed workflows.

[liferoad]

remind me after tests pass

[Ashfaqbs]

@liferoad , The Python PreCommit test failures are unrelated and expected — they attempt to run YAMLs without assertable output

[liferoad]

@liferoad , The Python PreCommit test failures are unrelated and expected — they attempt to run YAMLs without assertable output

Got it. Thanks.

[derrickaw]

Hi @Ashfaqbs,

1. I understand that failure is expected in this code situation, but we can't leave it like that and will need to address it in either of two ways - a) do what Robert suggested here with not testing this particular example or b) update this example to match something like. My preference is the latter option (you can use Charles's PR for a go by), so that all the examples that we have are tested.
2. We should also move your example to this location.

Thanks

[github-actions]

Reminder, please take a look at this pr: @liferoad

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @shunping for label python.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[derrickaw]

Unless there is something fundamentally wrong with the existing example, I would recommend that we keep the existing as is and then add another example that maybe you call it kafkaToKafka.yaml which essentially would be what you have now. Long as the examples work, more the better. Thanks.

[Ashfaqbs]

Hi @derrickaw ,

I’ve reverted the original example to its initial state and created a new file: kafkaToKafka.yaml, which includes my Beam Kafka-to-Kafka mirroring example.

[github-actions]

Reminder, please take a look at this pr: @shunping

[kennknowles]

Hello! To get this review moving, I recommend iterating on the changes until the tests are all green. Hopefully you can also reproduce the test failures locally, to make it easy.

[github-actions]

Reminder, please take a look at this pr: @shunping

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @liferoad for label python.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[github-actions]

Reminder, please take a look at this pr: @liferoad

[tvalentyn]

waiting on author

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.