Skip to content

Commit b6c6421

Browse files
authored
[DOCS] Fix issues in the Compile Sedona source code page to make it easy to setup dev environment (#2011)
* Update to java 11 * export a SPARK_VERSION env var for convenience * Add instrs to install spark (missing before) * Change 'xxx' to '*' so the command runs if the reader copy-paste it * Fix numbering and make sure they're all on a separate line * Move brew install geo to a separate line * Bump python version to 3.8+ (PR #1958) * Add make checkinstall instruction to docs
1 parent d799f50 commit b6c6421

1 file changed

Lines changed: 34 additions & 13 deletions

File tree

docs/setup/compile.md

Lines changed: 34 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525

2626
Sedona Scala/Java code is a project with multiple modules. Each module is a Scala/Java mixed project which is managed by Apache Maven 3.
2727

28-
* Make sure your Linux/Mac machine has Java 1.8, Apache Maven 3.3.1+, and Python3.7+. The compilation of Sedona is not tested on Windows machines.
28+
* Make sure your Linux/Mac machine has Java 1.11, Apache Maven 3.3.1+, and Python3.8+. The compilation of Sedona is not tested on Windows machines.
2929

3030
To compile all modules, please make sure you are in the root folder of all modules. Then enter the following command in the terminal:
3131

@@ -77,16 +77,25 @@ Sedona uses GitHub Actions to automatically generate jars per commit. You can go
7777

7878
## Run Python test
7979

80-
1. Set up the environment variable SPARK_HOME and PYTHONPATH
80+
1) Set up the environment variable SPARK_HOME and PYTHONPATH
8181

8282
For example,
8383

8484
```
85-
export SPARK_HOME=$PWD/spark-3.0.1-bin-hadoop2.7
85+
export SPARK_VERSION=3.4.0
86+
export SPARK_HOME=$PWD/spark-${SPARK_VERSION}-bin-hadoop3
8687
export PYTHONPATH=$SPARK_HOME/python
8788
```
8889

89-
2. Put JAI jars to ==SPARK_HOME/jars/== folder.
90+
2) Install Spark if you haven't already
91+
92+
```
93+
wget https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz
94+
tar -xvzf spark-${SPARK_VERSION}-bin-hadoop3.tgz
95+
rm spark-${SPARK_VERSION}-bin-hadoop3.tgz
96+
```
97+
98+
3) Put JAI jars to ==SPARK_HOME/jars/== folder.
9099

91100
```
92101
export JAI_CORE_VERSION="1.1.3"
@@ -97,13 +106,13 @@ wget -P $SPARK_HOME/jars/ https://repo.osgeo.org/repository/release/javax/media/
97106
wget -P $SPARK_HOME/jars/ https://repo.osgeo.org/repository/release/javax/media/jai_imageio/${JAI_IMAGEIO_VERSION}/jai_imageio-${JAI_IMAGEIO_VERSION}.jar
98107
```
99108

100-
3. Compile the Sedona Scala and Java code with `-Dgeotools` and then copy the ==sedona-spark-shaded-{{ sedona.current_version }}.jar== to ==SPARK_HOME/jars/== folder.
109+
4) Compile the Sedona Scala and Java code with `-Dgeotools` and then copy the ==sedona-spark-shaded-{{ sedona.current_version }}.jar== to ==SPARK_HOME/jars/== folder.
101110

102111
```
103-
cp spark-shaded/target/sedona-spark-shaded-xxx.jar $SPARK_HOME/jars/
112+
cp spark-shaded/target/sedona-spark-shaded-*.jar $SPARK_HOME/jars/
104113
```
105114

106-
4. Install the following libraries
115+
5) Install the following libraries
107116

108117
```
109118
sudo apt-get -y install python3-pip python-dev libgeos-dev
@@ -113,25 +122,31 @@ sudo pip3 install -U virtualenvwrapper
113122
sudo pip3 install -U pipenv
114123
```
115124

116-
Homebrew can be used to install libgeos-dev in macOS: `brew install geos`
117-
5. Set up pipenv to the desired Python version: 3.7, 3.8, or 3.9
125+
Homebrew can be used to install libgeos-dev in macOS:
126+
127+
```
128+
brew install geos
129+
```
130+
131+
6) Set up pipenv to the desired Python version: 3.8, 3.9, or 3.10
118132

119133
```
120134
cd python
121-
pipenv --python 3.7
135+
pipenv --python 3.8
122136
```
123137

124-
6. Install the PySpark version and the other dependency
138+
7) Install the PySpark version and the other dependency
125139

126140
```
127141
cd python
128-
pipenv install pyspark
142+
pipenv install pyspark==${SPARK_VERSION}
129143
pipenv install --dev
130144
```
131145

132146
`pipenv install pyspark` installs the latest version of pyspark.
133147
In order to remain consistent with the installed spark version, use `pipenv install pyspark==<spark_version>`
134-
7. Run the Python tests
148+
149+
8) Run the Python tests
135150

136151
```
137152
cd python
@@ -201,3 +216,9 @@ If you just want to run one hook for example just run the `markdownlint` hook:
201216
`pre-commit run markdownlint --all-files`
202217

203218
We have a [Makefile](https://github.com/apache/sedona/blob/master/Makefile) in the repository root which has three pre-commit convenience commands.
219+
220+
For example, you can run the following to setup pre-commit to run before each commit
221+
222+
```
223+
make checkinstall
224+
```

0 commit comments

Comments
 (0)