|
23 | 23 | * [Examples Catalog](#examples-catalog) |
24 | 24 | * [Wordcount](#wordcount) |
25 | 25 | * [Transforms](#transforms) |
26 | | - * [Element-wise](#element-wise) |
27 | 26 | * [Aggregation](#aggregation) |
| 27 | + * [Blueprints](#blueprints) |
| 28 | + * [Element-wise](#element-wise) |
| 29 | + * [IO](#io) |
| 30 | + * [ML](#ml) |
| 31 | + |
28 | 32 | <!-- TOC --> |
29 | 33 |
|
| 34 | +## Prerequistes |
| 35 | +Build this jar for running with the run command in the next stage: |
| 36 | +``` |
| 37 | +cd <path_to_beam_repo>/beam; ./gradlew sdks:java:io:google-cloud-platform:expansion-service:shadowJar |
| 38 | +``` |
| 39 | + |
| 40 | +## Example Run |
30 | 41 | This module contains a series of Beam YAML code samples that can be run using |
31 | 42 | the command: |
32 | 43 | ``` |
33 | 44 | python -m apache_beam.yaml.main --pipeline_spec_file=/path/to/example.yaml |
34 | 45 | ``` |
35 | 46 |
|
| 47 | +Depending on the yaml pipeline, the output may be emitted to standard output or |
| 48 | +a file located in the execution folder used. |
| 49 | + |
36 | 50 | ## Wordcount |
37 | 51 | A good starting place is the [Wordcount](wordcount_minimal.yaml) example under |
38 | 52 | the root example directory. |
39 | 53 | This example reads in a text file, splits the text on each word, groups by each |
40 | 54 | word, and counts the occurrence of each word. This is a classic example used in |
41 | 55 | the other SDK's and shows off many of the functionalities of Beam YAML. |
42 | 56 |
|
| 57 | +## Testing |
| 58 | +A test file is located in the testing folder that will execute all the example |
| 59 | +yamls and confirm the expected results. |
| 60 | +``` |
| 61 | +pytest -v testing/ |
| 62 | +
|
| 63 | +or |
| 64 | +
|
| 65 | +python -m unittest -v testing/examples_test.py |
| 66 | +``` |
| 67 | + |
43 | 68 | ## Transforms |
44 | 69 |
|
45 | 70 | Examples in this directory show off the various built-in transforms of the Beam |
46 | 71 | YAML framework. |
47 | 72 |
|
| 73 | +### Aggregation |
| 74 | +These examples leverage the built-in `Combine` transform for performing simple |
| 75 | +aggregations including sum, mean, count, etc. |
| 76 | + |
| 77 | +### Blueprints |
| 78 | +These examples leverage DF or other existing templates and convert them to yaml |
| 79 | +blueprints. |
| 80 | + |
48 | 81 | ### Element-wise |
49 | 82 | These examples leverage the built-in mapping transforms including `MapToFields`, |
50 | 83 | `Filter` and `Explode`. More information can be found about mapping transforms |
51 | 84 | [here](https://beam.apache.org/documentation/sdks/yaml-udf/). |
52 | 85 |
|
53 | | -### Aggregation |
54 | | -These examples leverage the built-in `Combine` transform for performing simple |
55 | | -aggregations including sum, mean, count, etc. |
| 86 | +### IO |
| 87 | +These examples leverage the built-in `Spanner_Read` and `Spanner_Write` |
| 88 | +transform for performing simple reads and writes from a spanner DB. |
| 89 | + |
| 90 | +### ML |
| 91 | +These examples leverage the built-in `Enrichment` transform for performing |
| 92 | +ML enrichments. |
56 | 93 |
|
57 | 94 | More information can be found about aggregation transforms |
58 | 95 | [here](https://beam.apache.org/documentation/sdks/yaml-combine/). |
0 commit comments