Skip to content

[SEDONA-738] Add moran i autocorrelation.#1975

Merged
jiayuasu merged 10 commits into
masterfrom
SEDONA-738-add-autocorrelation
Jul 17, 2025
Merged

[SEDONA-738] Add moran i autocorrelation.#1975
jiayuasu merged 10 commits into
masterfrom
SEDONA-738-add-autocorrelation

Conversation

@Imbruced
Copy link
Copy Markdown
Member

@Imbruced Imbruced commented Jun 4, 2025

Did you read the Contributor Guide?

  • Yes

Is this PR related to a ticket?

SEDONA-738

What changes were proposed in this PR?

MoranI index

How was this patch tested?

Unit tests

Did this PR include necessary documentation updates?

Yes

@Imbruced
Copy link
Copy Markdown
Member Author

Imbruced commented Jun 4, 2025

I need to add docs

@Imbruced
Copy link
Copy Markdown
Member Author

Imbruced commented Jun 4, 2025

and update the description of the MR

@github-actions github-actions Bot added the docs label Jun 5, 2025
@Imbruced Imbruced marked this pull request as ready for review June 6, 2025 18:48
@Imbruced Imbruced requested a review from jiayuasu as a code owner June 6, 2025 18:48
@james-willis
Copy link
Copy Markdown
Collaborator

Thanks for this Pawel.

I think we need an ST Function for this. Currently it is asymmetric with the other stats functions in this regard.

I also believe that we should deprecate these function call interfaces for geostats functions and tell users to use the ST functions instead. If we go this direction I would advise against making python and public scala functions for morans I (other than ST functions for the df interfaces of course)

@jiayuasu
Copy link
Copy Markdown
Member

jiayuasu commented Jun 7, 2025

@james-willis Is your SQL function framework able to support Pawel's Moran'I case as well? Maybe you can point him to the right place for this?

@james-willis
Copy link
Copy Markdown
Collaborator

Sure. We should be able to wrap the function in this PR with an implementation of a PhysicalFunction.

Here are the existing ones.

Here is the PR that added SQL support to geostats

@jiayuasu jiayuasu changed the title SEDONA-738 Add moran i autocorrelation. [SEDONA-738] Add moran i autocorrelation. Jun 20, 2025
@jiayuasu jiayuasu requested a review from Copilot July 17, 2025 15:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces Moran’s I spatial autocorrelation to the Scala, Java, and Python APIs, including implementations, wrappers, tests, and documentation.

  • Added Scala implementation and Java result container for Moran’s I
  • Exposed Moran’s I in Python with wrapper and updated weighting functions
  • Added unit tests (Scala and Python) and extended SQL API documentation

Reviewed Changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
spark/common/src/main/scala/org/apache/sedona/stats/autocorelation/Moran.scala Core Scala implementation of Moran’s I
spark/common/src/main/java/org/apache/sedona/stats/autocorrelation/MoranResult.java Java container class for result values
spark/common/src/main/scala/org/apache/sedona/stats/Weighting.scala Added addDistanceBandColumnPython bridge method
spark/common/src/test/scala/org/apache/sedona/stats/autocorellation/MoranTest.scala Scala tests for positive, negative, and zero cases
spark/common/src/test/scala/org/apache/sedona/stats/autocorellation/AutoCorrelationFixtures.scala Test fixtures for auto-correlation scenarios
python/sedona/spark/stats/autocorrelation/moran.py Python wrapper for Moran’s I
python/tests/stats/test_moran.py Python integration tests for Moran’s I
python/sedona/spark/stats/weighting.py Updated Python weighting functions to call wrapper
python/sedona/spark/stats/hotspot_detection/getis_ord.py Fixed DataFrame wrapping in g_local call
python/sedona/spark/register/java_libs.py Registered Moran in JVM libs enum
python/sedona/spark/stats/autocorrelation/init.py Initialized Python autocorrelation package
docs/api/stats/sql.md Documented MoranI algorithm and usage
Comments suppressed due to low confidence (3)

spark/common/src/test/scala/org/apache/sedona/stats/autocorellation/MoranTest.scala:1

  • The package name 'autocorellation' is misspelled and does not match the implementation package 'autocorrelation'. Please rename the directory and package declaration to 'autocorrelation' so the tests compile correctly.
/*

docs/api/stats/sql.md:142

  • [nitpick] Minor grammatical issue: remove 'the' before '1' to read 'close to 1', and consider rephrasing to improve clarity.
location and non-spatial attribute. When the value is close to the 1 it

docs/api/stats/sql.md:146

  • Grammar correction: change 'has' to 'have' since 'values' is plural.
correlation. Negative correlation means that close values has dissimilar values.

jiayuasu and others added 3 commits July 17, 2025 08:21
…ation/AutoCorrelationFixtures.scala

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@jiayuasu
Copy link
Copy Markdown
Member

@Imbruced I will merge this for now but the doc needs significant improvement. I believe we need some comprehensive end-to-end examples of each Stat function

@jiayuasu jiayuasu merged commit 04d313e into master Jul 17, 2025
35 checks passed
@james-willis
Copy link
Copy Markdown
Collaborator

Filed a ticket for ST function: #2115

Subham-KRLX pushed a commit to Subham-KRLX/sedona that referenced this pull request Jul 18, 2025
* SEDONA-738 Add moran i autocorrelation.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix scala 2.13 issue

* Update spark/common/src/test/scala/org/apache/sedona/stats/autocorellation/AutoCorrelationFixtures.scala

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix typos

* Update doc

---------

Co-authored-by: Jia Yu <jiayu@apache.org>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@jiayuasu jiayuasu deleted the SEDONA-738-add-autocorrelation branch September 8, 2025 07:19
Kontinuation pushed a commit to Kontinuation/sedona that referenced this pull request Jan 21, 2026
* SEDONA-738 Add moran i autocorrelation.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix unit tests.

* SEDONA-738 Fix scala 2.13 issue

* Update spark/common/src/test/scala/org/apache/sedona/stats/autocorellation/AutoCorrelationFixtures.scala

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix typos

* Update doc

---------

Co-authored-by: Jia Yu <jiayu@apache.org>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants