FINERACT-1420: Improve idempotency fallback using deterministic key generation

[elnafateh]

Description

This PR improves the existing idempotency mechanism by enhancing the fallback behavior when no idempotency key is provided by the client.

Currently, when a request is sent without an idempotency key, the system generates a random key, which breaks retry safety in real-world scenarios (e.g. network failures). Retried requests are treated as new operations, leading to potential duplicate processing.

This change introduces deterministic idempotency key generation based on a canonical representation of the request payload combined with a time window.

Key Points

• Does NOT replace the existing idempotency mechanism
• Retains DB-level uniqueness constraint as final safeguard
• Ensures consistent retry behavior without requiring client-provided keys
• Maintains backward compatibility (no migration required)

Trade-offs

Semantically identical requests within the same time window may be deduplicated unintentionally. This is a design trade-off to ensure retry safety.

Scope

This change aligns with FINERACT-1420 by improving API fault tolerance and preventing duplicate request execution.

This contribution is part of my preparation for Google Summer of Code and aims to improve reliability in Fineract's command processing pipeline.

Checklist

Please make sure these boxes are checked before submitting your pull request - thanks!

• [x ] Write the commit message as per our guidelines
• [ x] Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.
• [ x] Create/update unit or integration tests for verifying the changes made.
• [ x] Follow our coding conventions.
• Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes
• [ x] This PR must not be a "code dump". Large changes can be made in a branch, with assistance. Ask for help on the developer mailing list.

Your assigned reviewer(s) will follow our guidelines for code reviews.

[vidakovic]

Also: 2 commits... please squash into 1 according to rules. No reference to a ticket in the PR title... please follow the rules (there are not many).

[vidakovic]

I see nothing related to the "DB-level uniqueness constraint"... there is nothing mentioning the existing idempotency solution, possible side effects with this implementation, how we should migrate from one to the other, do we have to migrate at all? Looks incomplete.

[elnafateh]

Also: 2 commits... please squash into 1 according to rules. No reference to a ticket in the PR title... please follow the rules (there are not many).

I’ve made required update and then quash the commits into a single one and I’ve also added the ticket number.

[elnafateh]

I see nothing related to the "DB-level uniqueness constraint"... there is nothing mentioning the existing idempotency solution, possible side effects with this implementation, how we should migrate from one to the other, do we have to migrate at all? Looks incomplete.

Fair point — the PR description needs to be clearer.

This change does not replace the existing idempotency mechanism but enhances it by providing a fallback when no idempotency key is supplied.

• The DB-level uniqueness constraint remains unchanged and continues to act as the final safeguard.
• No migration is required since this is backward compatible.
• One trade-off is that semantically identical requests within the time window may be deduplicated unintentionally, which is a design consideration.

I will update the PR description to reflect these aspects clearly.

[elnafateh]

@vidakovic @adamsaghy @IOhacker I have rewritten the unit tests, and also extended the integration tests to reflect the change in behavior of generating the idempotency key from random to deterministic. I have run the tests locally and and add backward compatibility, docker,db and they are all passed . Kindly rerun the tests again.

[adamsaghy]

@vidakovic @adamsaghy @IOhacker I have rewritten the unit tests, and also extended the integration tests to reflect the change in behavior of generating the idempotency key from random to deterministic. I have run the tests locally and and add backward compatibility, docker,db and they are all passed . Kindly rerun the tests again.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':rat'.

> A failure occurred while executing org.nosphere.apache.rat.RatWork
   > Apache Rat audit failure - 1 unapproved license
     	See file:///home/runner/work/fineract/fineract/build/reports/rat/index.html

Seems you missed tobadd the license header to the new files

[adamsaghy]

Please run:
./gradlew spotlessApply spotbugsMain spotbugsTest checkstyleMain checkstyleTest
./gradlew --no-daemon build -x test -x cucumber -x doc

Before any PR or changes, please always run these two commands and make sure there is green build!

[elnafateh]

@vidakovic @adamsaghy @IOhacker I have rewritten the unit tests, and also extended the integration tests to reflect the change in behavior of generating the idempotency key from random to deterministic. I have run the tests locally and and add backward compatibility, docker,db and they are all passed . Kindly rerun the tests again.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':rat'.

> A failure occurred while executing org.nosphere.apache.rat.RatWork
   > Apache Rat audit failure - 1 unapproved license
     	See file:///home/runner/work/fineract/fineract/build/reports/rat/index.html

Seems you missed tobadd the license header to the new files

that has been fixed, am currently testing. thanks for the guide

[elnafateh]

Please run: ./gradlew spotlessApply spotbugsMain spotbugsTest checkstyleMain checkstyleTest ./gradlew --no-daemon build -x test -x cucumber -x doc

Before any PR or changes, please always run these two commands and make sure there is green build!

@adamsaghy ok noted, i really appropriate the guide

[elnafateh]

@adamsaghy I have run those tests and the build was all green locally

[IOhacker]

Please rebase El El jue, 26 de mar de 2026 a la(s) 4:39 p.m., elnafateh < ***@***.***> escribió:

…

*elnafateh* left a comment (apache/fineract#5674) <#5674?email_source=notifications&email_token=ALD2ZAQQEMEVS433BXFKMCD4SWWQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJTHA3TONZQGE32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4138777017> @adamsaghy <https://github.com/adamsaghy> I have run those tests and the build was all green locally, but here it build failed the way as previous — Reply to this email directly, view it on GitHub <#5674?email_source=notifications&email_token=ALD2ZAQQEMEVS433BXFKMCD4SWWQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJTHA3TONZQGE32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4138777017>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD2ZAXPUMT6QFNMLHNU4XT4SWWQLAVCNFSM6AAAAACW2WA2WCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCMZYG43TOMBRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[elnafateh]

Please rebase El El jue, 26 de mar de 2026 a la(s) 4:39 p.m., elnafateh < @.> escribió:
…
elnafateh left a comment (apache/fineract#5674) <#5674?email_source=notifications&email_token=ALD2ZAQQEMEVS433BXFKMCD4SWWQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJTHA3TONZQGE32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4138777017> @adamsaghy https://github.com/adamsaghy I have run those tests and the build was all green locally, but here it build failed the way as previous — Reply to this email directly, view it on GitHub <#5674?email_source=notifications&email_token=ALD2ZAQQEMEVS433BXFKMCD4SWWQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJTHA3TONZQGE32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4138777017>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALD2ZAXPUMT6QFNMLHNU4XT4SWWQLAVCNFSM6AAAAACW2WA2WCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCMZYG43TOMBRG4 . You are receiving this because you were mentioned.Message ID: @.>

it's done

[elnafateh]

am working on fixing the failed tests..

[adamsaghy]

@elnafateh Please review the failing checks!

[elnafateh]

@elnafateh Please review the failing checks!

I will fix them

[elnafateh]

@adamsaghy all tests are in green

[adamsaghy]

@elnafateh Can you help me understand a little better why would it be an issue to retry the request with:

• same idempotency key
• new idempotency key?

If something is retried, it means nothing was committed, so no harm no foul. Am I missing something?

[elnafateh]

@adamsaghy The core problem is about external client retries without providing an idempotency key.
When a client doesn't provide an idempotency key and the system generates a random UUID per request, this leads to double spending

however by generating deterministic key from the payload and context, the same logical operation always produce the same key within time window — even without the client explicitly providing the key

[adamsaghy]

@adamsaghy The core problem is about external client retries without providing an idempotency key. When a client doesn't provide an idempotency key and the system generates a random UUID per request, this leads to double spending

however by generating deterministic key from the payload and context, the same logical operation always produce the same key within time window — even without the client explicitly providing the key

Would you mind to raise this topic on the fineract dev mail list?

I dont really see an immediate problem, but would be happy to hear out the thoughts of others! ;)

[elnafateh]

Would you mind to raise this topic on the fineract dev mail list?

I dont really see an immediate problem, but would be happy to hear out the thoughts of others! ;)

@adamsaghy to shed more light on the problem, the concern is around cases where the request actually does get committed, but the client doesn’t receive the response (e.g., network timeout). In that situation, a retry without an idempotency key can be treated as a new operation if a random UUID is generated, which may lead to duplicate transactions.

To get broader input, I’ll raise this on the dev mailing list as you suggested so we can validate whether this is a real concern and agree on the expected behavior.