We recommend Intellij IDEA with Scala plugin installed. Please make sure that the Project has the SDK set to a JDK 1.8.
The IDE usually has trouble understanding the complex project structure in Sedona.
You need to comment out the following lines in pom.xml at the root folder, as follows. ==Remember that you should NOT submit this change to Sedona.==
<!-- <parent>-->
<!-- <groupId>org.apache</groupId>-->
<!-- <artifactId>apache</artifactId>-->
<!-- <version>23</version>-->
<!-- <relativePath />-->
<!-- </parent>-->Make sure you reload the pom.xml or reload the maven project. The IDE will ask you to remove some modules. Please select yes.
In a terminal, go to the Sedona root folder. Run mvn clean install. All tests will take more than 15 minutes. To only build the project jars, run mvn clean install -DskipTests.
!!!Note
mvn clean install will compile Sedona with Spark 3.3 and Scala 2.12. If you have a different version of Spark in $SPARK_HOME, make sure to specify that using -Dspark command line arg.
For example, to compile sedona with Spark 3.4 and Scala 2.12, use: mvn clean install -Dspark=3.4 -Dscala=2.12
More details can be found on Compile Sedona
In the IDE, right-click a test case and run this test case.
When you run a test case written in Scala, the IDE might tell you that the "Path does not exist" as follows:
Go to Edit Configuration
Change the value of Working Directory to $MODULE_WORKING_DIR$.
Re-run the test case. Do NOT right-click the test case to re-run. Instead, click the button as shown in the figure below.
If you don't want to change the Working Directory configuration every time, you can change the default value of Working Directory
in the Run/Debug Configurations window. Click Edit configuration templates... and change the value
of Working Directory of ScalaTest to $MODULE_WORKING_DIR$.
Now newly created run configurations for ScalaTest will have the correct value set for Working Directory.
If you are using Java 11, you may encounter the following error when running tests:
/path/to/sedona/common/src/main/java/org/apache/sedona/common/geometrySerde/UnsafeGeometryBuffer.java
package sun.misc does not exist
sun.misc.Unsafe
You can fix this issue by disabling Use '--release' option for cross-compilation in the IDE settings.
If you want to test changes with different Spark/Scala versions, you can select the Spark and Scala profile in the Maven panel. Once you have selected the desired versions, reload the sedona-parent project. See picture below
!!!Note
The profile change won't update the module names in the IDE. Don't be misled if a module still has a -3.3-2.12 suffix in the name.
!!!Note Not all combinations of spark and scala versions are supported and so they will fail to compile.
We recommend PyCharm.
To run all Python test cases, follow steps mentioned here.
Once the environment is set up, you can run all tests using the following command in python directory:
cd python
uv run pytest -v testsTo run a particular Python test file, specify the path of the .py.
For example, to run all tests in test_function.py located in python/tests/sql/, use:
cd python
uv run pytest -v tests/sql/test_function.pyTo run a particular test in a particular .py test file, specify file_name::class_name::test_name to the pytest command.
For example, to run the test on ST_Contains function located in sql/test_predicate.py, use:
cd python
uv run pytest -v tests/sql/test_predicate.py::TestPredicate::test_st_containsThe following command will build the sdist and whl packages in the dist folder.
cd python
uv buildMore details to come.
We recommend RStudio