Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/source/submission.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,11 @@ Upon receiving a message from `submission_task_queue` with a binding key of `sub

* Finally, the temporary computation directory allocated for this submission is removed.

## Submission Invariants

The submission workflow enforces a set of invariants to ensure challenge integrity.
See [Submission Workflow Invariants](submission_workflow_invariants.md) for details.

### Notes

* REST API with url pattern `jobs:challenge_submission`. Here _jobs_ is application namespace and _challenge_submission_ is instance namespace. You can read more about [url namespace](https://docs.djangoproject.com/en/1.11/topics/http/urls/#url-namespaces)
40 changes: 40 additions & 0 deletions docs/source/submission_workflow_invariants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Submission Workflow Invariants

The submission API in EvalAI enforces a set of invariants to ensure challenge integrity and prevent invalid submissions.
These rules are enforced at the API layer and are fundamental to how submissions are handled across challenges.

This document describes those invariants and the reasoning behind them.

---

## Core Invariants

The submission workflow must satisfy the following rules:

- Submissions must be rejected if the challenge or challenge phase is inactive.
- Only participant teams that are registered for a challenge are allowed to submit.
- Submission requests must be authenticated before workflow invariants are evaluated.

These represent a subset of the fundamental invariants enforced by the submission workflow and apply regardless of client behavior.

---

## Why These Invariants Matter

These rules ensure that only valid submissions enter the system.
By enforcing them early in the request lifecycle, EvalAI prevents inconsistent state and avoids downstream issues in scoring, evaluation, and leaderboards.

Making these constraints explicit also helps contributors reason about expected behavior and avoid introducing regressions.

---

## Testing Approach

These invariants are intended to be validated using API-level workflow tests rather than unit tests of internal functions.

Each workflow test:
- Satisfies only the required preconditions
- Violates exactly one invariant
- Asserts both the API response and the absence of side effects (for example, ensuring that no `Submission` object is created)

This approach helps keep tests stable while clearly documenting the guarantees provided by the submission workflow.