diff --git a/docs/source/submission.md b/docs/source/submission.md index 0c9aee3cbe..2c90d2e84d 100644 --- a/docs/source/submission.md +++ b/docs/source/submission.md @@ -126,6 +126,11 @@ Upon receiving a message from `submission_task_queue` with a binding key of `sub * Finally, the temporary computation directory allocated for this submission is removed. +## Submission Invariants + +The submission workflow enforces a set of invariants to ensure challenge integrity. +See [Submission Workflow Invariants](submission_workflow_invariants.md) for details. + ### Notes * REST API with url pattern `jobs:challenge_submission`. Here _jobs_ is application namespace and _challenge_submission_ is instance namespace. You can read more about [url namespace](https://docs.djangoproject.com/en/1.11/topics/http/urls/#url-namespaces) diff --git a/docs/source/submission_workflow_invariants.md b/docs/source/submission_workflow_invariants.md new file mode 100644 index 0000000000..f3e6dede46 --- /dev/null +++ b/docs/source/submission_workflow_invariants.md @@ -0,0 +1,40 @@ +# Submission Workflow Invariants + +The submission API in EvalAI enforces a set of invariants to ensure challenge integrity and prevent invalid submissions. +These rules are enforced at the API layer and are fundamental to how submissions are handled across challenges. + +This document describes those invariants and the reasoning behind them. + +--- + +## Core Invariants + +The submission workflow must satisfy the following rules: + +- Submissions must be rejected if the challenge or challenge phase is inactive. +- Only participant teams that are registered for a challenge are allowed to submit. +- Submission requests must be authenticated before workflow invariants are evaluated. + +These represent a subset of the fundamental invariants enforced by the submission workflow and apply regardless of client behavior. + +--- + +## Why These Invariants Matter + +These rules ensure that only valid submissions enter the system. +By enforcing them early in the request lifecycle, EvalAI prevents inconsistent state and avoids downstream issues in scoring, evaluation, and leaderboards. + +Making these constraints explicit also helps contributors reason about expected behavior and avoid introducing regressions. + +--- + +## Testing Approach + +These invariants are intended to be validated using API-level workflow tests rather than unit tests of internal functions. + +Each workflow test: +- Satisfies only the required preconditions +- Violates exactly one invariant +- Asserts both the API response and the absence of side effects (for example, ensuring that no `Submission` object is created) + +This approach helps keep tests stable while clearly documenting the guarantees provided by the submission workflow.