Skip to content

Commit 7952b19

Browse files
committed
Update running instructions on README
1 parent 8c7fe9e commit 7952b19

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

  • module5-batch-processing/pyspark-4.x

module5-batch-processing/pyspark-4.x/README.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,28 @@ pre-commit install
3737
docker compose -f ../compose.yaml up -d
3838
```
3939

40+
**5.** Spark Web UI
41+
- Spark Master Web UI can be accessed at [http://localhost:4040](http://localhost:4040)
42+
- Spark History Server can be accessed at [http://localhost:18080](http://localhost:18080)
43+
44+
45+
## Spark-submit Application
46+
47+
### Local (Spark Driver running on local machine)
48+
49+
With `--deploy-mode client` (default), the Spark Driver runs locally and doesn't pick up [spark-4.0-standalone.conf](../compose.spark-4.0-standalone.yaml), so the `--conf spark.hadoop.*` options must be set explicitly.
50+
51+
```shell
52+
spark-submit \
53+
--master spark://localhost:7077 \
54+
--packages "com.google.cloud.bigdataoss:gcs-connector:4.0.2" \
55+
--conf spark.eventLog.enabled=true \
56+
--conf spark.eventLog.dir=file://$(pwd)/../logs/ \
57+
--conf spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
58+
--conf spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
59+
fhv_zones_gcs.py
60+
```
61+
4062

4163
## Compatibility Matrix for GCS
4264

0 commit comments

Comments
 (0)