Skip to content

Commit 73985e5

Browse files
authored
Clean up remaining references to Samza runner (#38326)
* Clean up remaining references to Samza runner * Fix compatibility matrix ordering * Revert javascript change --- ineffective
1 parent 4aedd4e commit 73985e5

12 files changed

Lines changed: 74 additions & 758 deletions

File tree

.agent/skills/adding-new-metadata/SKILL.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,6 @@ You must ensure that when a DoFn processes an element and outputs a new element,
8888
### Timers
8989
If metadata needs to survive timer firings (e.g., knowing an `@OnTimer` fired because of a system drain), it must be added to Timer data structures. This is a bit of uncharted area which was only implemented for CausedByDrain metadata that comes from backend, not from persisted metadata. In order to persist all WindowedValue metadata across timer, more work has to be done, below are some pointers:
9090
* `runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java` and implementations (e.g., `WindmillTimerInternals.java` in Dataflow).
91-
* `runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/KeyedTimerData.java` (or generic `TimerData`).
9291
* **Action:** Add the field to `TimerData`, next to `CausedByDrain`. Propagate it when setting the timer and expose it when the timer fires so it bubbles up.
9392
* Eventually, metadata from Timer lands in WindowedValue, so it can be exposed to users. Keep field names, types, and getters similar to WindowedValue as much as possible, as common interface may be introduced eventually.
9493

@@ -116,4 +115,4 @@ User needs to access the metadata in their `DoFn` (e.g., `@ProcessElement public
116115
9. [ ] Update `ReduceFnRunner` and `OutputAndTimeBoundedSplittableProcessElementInvoker` for complex transform propagation.
117116
10. [ ] If required by timers, update `TimerData` and `TimerInternals`.
118117
11. [ ] If exposed to the user, update `DoFnSignatures` and `ByteBuddyDoFnInvokerFactory`.
119-
12. [ ] Update other runners (Flink, Spark, Samza) to ensure they propagate the new `WindowedValue` fields correctly in their specific operators/runners.
118+
12. [ ] Update other runners (Flink, Spark) to ensure they propagate the new `WindowedValue` fields correctly in their specific operators/runners.

.agent/skills/runners/SKILL.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ Runners execute Beam pipelines on distributed processing backends. Each runner t
3434
| Dataflow | `runners/google-cloud-dataflow-java/` | Google Cloud Dataflow |
3535
| Flink | `runners/flink/` | Apache Flink |
3636
| Spark | `runners/spark/` | Apache Spark |
37-
| Samza | `runners/samza/` | Apache Samza |
3837
| Jet | `runners/jet/` | Hazelcast Jet |
3938
| Twister2 | `runners/twister2/` | Twister2 |
4039

.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,6 @@ categories:
7676
- "PostCommit Python ValidatesRunner Dataflow"
7777
- "PostCommit Python ValidatesRunner Spark"
7878
- "PostCommit Python ValidatesRunner Flink"
79-
- "PostCommit Python ValidatesRunner Samza"
8079
- "Build python source distribution and wheels"
8180
- "Java Tests"
8281
- "PostCommit Java"
@@ -113,7 +112,6 @@ categories:
113112
- "PreCommit Java Kafka IO Direct"
114113
- "PostCommit Java Examples Direct"
115114
- "PreCommit Java JDBC IO Direct"
116-
- "PostCommit Java ValidatesRunner Samza"
117115
- "PreCommit Java Mqtt IO Direct"
118116
- "PreCommit Java Kinesis IO Direct"
119117
- "PreCommit Java MongoDb IO Direct"
@@ -138,7 +136,6 @@ categories:
138136
- "PreCommit Java Thrift IO Direct"
139137
- "PreCommit Java Snowflake IO Direct"
140138
- "PreCommit Java Solr IO Direct"
141-
- "PostCommit Java PVR Samza"
142139
- "PreCommit Java Tika IO Direct"
143140
- "PostCommit Java SingleStoreIO IT"
144141
- "PostCommit Java ValidatesRunner Direct"
@@ -209,7 +206,6 @@ categories:
209206
- "PerformanceTests BigQueryIO Batch Java Json"
210207
- "PerformanceTests SQLBigQueryIO Batch Java"
211208
- "PerformanceTests XmlIOIT"
212-
- "PostCommit XVR Samza"
213209
- "PerformanceTests ManyFiles TextIOIT"
214210
- "PerformanceTests XmlIOIT HDFS"
215211
- "PerformanceTests ParquetIOIT"
@@ -291,8 +287,7 @@ categories:
291287
tests:
292288
- "PerformanceTests MongoDBIO IT"
293289
- "PreCommit GoPortable"
294-
- "PreCommit GoPrism"
295-
- "PostCommit Go VR Samza"
290+
- "PreCommit GoPrism"
296291
- "PostCommit Go Dataflow ARM"
297292
- "LoadTests Go CoGBK Dataflow Batch"
298293
- "LoadTests Go Combine Dataflow Batch"

learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md

Lines changed: 0 additions & 97 deletions
Original file line numberDiff line numberDiff line change
@@ -390,103 +390,6 @@ python -m apache_beam.examples.wordcount --input /path/to/inputfile \
390390
```
391391
{{end}}
392392

393-
{{if (eq .Sdk "java" "go")}}
394-
### Samza runner
395-
396-
The Apache Samza Runner can be used to execute Beam pipelines using Apache Samza. The Samza Runner executes Beam pipeline in a Samza application and can run locally. The application can further be built into a .tgz file, and deployed to a YARN cluster or Samza standalone cluster with Zookeeper.
397-
398-
The Samza Runner and Samza are suitable for large scale, stateful streaming jobs, and provide:
399-
400-
* First class support for local state (with RocksDB store). This allows fast state access for high frequency streaming jobs.
401-
* Fault-tolerance with support for incremental checkpointing of state instead of full snapshots. This enables Samza to scale to applications with very large state.
402-
* A fully asynchronous processing engine that makes remote calls efficient.
403-
* Flexible deployment model for running the applications in any hosting environment with Zookeeper.
404-
* Features like canaries, upgrades and rollbacks that support extremely large deployments with minimal downtime.
405-
406-
Additionally, you can read more about the Samza Runner [here](https://beam.apache.org/documentation/runners/samza/)
407-
408-
#### Run example
409-
{{end}}
410-
411-
{{if (eq .Sdk "go")}}
412-
413-
Need import:
414-
```
415-
"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/samza"
416-
```
417-
418-
It is necessary to give an endpoint where the runner is raised with `--endpoint`:
419-
```
420-
$ go install github.com/apache/beam/sdks/v2/go/examples/wordcount
421-
# As part of the initial setup, for non linux users - install package unix before run
422-
$ go get -u golang.org/x/sys/unix
423-
$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
424-
--output gs://<your-gcs-bucket>/counts \
425-
--runner samza \
426-
--project your-gcp-project \
427-
--region your-gcp-region \
428-
--temp_location gs://<your-gcs-bucket>/tmp/ \
429-
--staging_location gs://<your-gcs-bucket>/binaries/ \
430-
--worker_harness_container_image=apache/beam_go_sdk:latest \
431-
--endpoint=localhost:8081
432-
```
433-
{{end}}
434-
435-
{{if (eq .Sdk "java")}}
436-
You can specify your dependency on the Samza Runner by adding the following to your `pom.xml`:
437-
438-
```
439-
<dependency>
440-
<groupId>org.apache.beam</groupId>
441-
<artifactId>beam-runners-samza</artifactId>
442-
<version>2.42.0</version>
443-
<scope>runtime</scope>
444-
</dependency>
445-
446-
<!-- Samza dependencies -->
447-
<dependency>
448-
<groupId>org.apache.samza</groupId>
449-
<artifactId>samza-api</artifactId>
450-
<version>${samza.version}</version>
451-
</dependency>
452-
453-
<dependency>
454-
<groupId>org.apache.samza</groupId>
455-
<artifactId>samza-core_2.11</artifactId>
456-
<version>${samza.version}</version>
457-
</dependency>
458-
459-
<dependency>
460-
<groupId>org.apache.samza</groupId>
461-
<artifactId>samza-kafka_2.11</artifactId>
462-
<version>${samza.version}</version>
463-
<scope>runtime</scope>
464-
</dependency>
465-
466-
<dependency>
467-
<groupId>org.apache.samza</groupId>
468-
<artifactId>samza-kv_2.11</artifactId>
469-
<version>${samza.version}</version>
470-
<scope>runtime</scope>
471-
</dependency>
472-
473-
<dependency>
474-
<groupId>org.apache.samza</groupId>
475-
<artifactId>samza-kv-rocksdb_2.11</artifactId>
476-
<version>${samza.version}</version>
477-
<scope>runtime</scope>
478-
</dependency>
479-
```
480-
481-
Console:
482-
```
483-
$ mvn exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
484-
-Psamza-runner \
485-
-Dexec.args="--runner=SamzaRunner \
486-
--inputFile=/path/to/input \
487-
--output=/path/to/counts"
488-
```
489-
490393
### Nemo runner
491394

492395
The Apache Nemo Runner can be used to execute Beam pipelines using Apache Nemo. The Nemo Runner can optimize Beam pipelines with the Nemo compiler through various optimization passes and execute them in a distributed fashion using the Nemo runtime. You can also deploy a self-contained application for local mode or run using resource managers like YARN or Mesos.

release/src/main/scripts/jenkins_jobs.txt

Lines changed: 0 additions & 115 deletions
This file was deleted.

0 commit comments

Comments
 (0)