FC-73 Add EXTERNAL_GRADER_SCORE_SUBMITTED signal to broadcast scoring events (Feature: Add event emission for external grader scores)

docs/source/decisions/0003-implementation-of-views-for-external-grader-integration.rst

@@ -0,0 +1,143 @@
+3. Implementation of XQueue Compatible Views for External Grader Integration
+############################################################################
+
+Status
+******
+
+**Provisional** *2025-02-21*
+
+Implemented by https://github.com/openedx/edx-submissions/pull/284
+
+Context
+*******
+
+Following the creation of ExternalGraderDetail (ADR 1) and SubmissionFile (ADR 2) models, we need to implement the API
+endpoints that will allow external graders (xqueue-watcher) to interact with the system. The current XQueue implementation
+provides three critical endpoints that need to be replicated:
+
+1. Authentication Service:
+   - Secure login mechanism for external graders
+   - Session management
+   - CSRF handling for specific endpoints
+
+2. Submission Retrieval (get_submission):
+   - Queue-based submission distribution
+   - Status tracking and locking mechanism
+   - File information packaging for graders
+
+3. Result Processing (put_result):
+   - Score validation and processing
+   - Status updates
+   - Error handling and retry mechanisms
+
+The current XQueue implementation has these services spread across multiple systems, requiring complex HTTP communication
+and session management. The existing workflow:
+
+1. Authentication Flow:
+   - Basic username/password authentication
+   - Session-based token management
+   - Manual CSRF handling for specific endpoints
+
+2. Submission Processing:
+   - Manual queue status checks
+   - Complex state transitions
+   - Synchronous HTTP-based file retrieval
+
+3. Result Handling:
+   - Direct database updates
+   - Limited error recovery
+   - Complex retry logic
+
+Decision
+********
+
+We will implement a unified service approach that brings together all the functionality needed for external grader integration. This approach will:
+
+1. Simplify Authentication:
+   - Create a more straightforward login process for external graders
+   - Make session handling more reliable
+   - Ensure security while reducing technical complexity
+
+2. Improve Submission Handling:
+   - Create a more efficient way to distribute submissions to graders
+   - Track the status of submissions more reliably
+   - Provide all necessary information to graders in a consistent format
+
+3. Enhance Results Processing:
+   - Process scores more reliably
+   - Handle errors gracefully
+   - Provide better feedback to both graders and the platform
+
+The solution will be built as a consolidated service that external graders can interact with through standard REST API endpoints. This will make integration easier for external services while providing better monitoring and control for the platform.
+
+Key Benefits:
+
+1. External graders will have a single, consistent interface
+2. The platform will have better visibility into the grading process
+3. Error handling will be improved across the entire workflow
+4. Security will be enhanced through better authentication practices
+
+This approach connects directly with the previously created data models for external graders and submission files, providing a complete end-to-end solution for the grading workflow.
+
+Consequences
+************
+
+Positive:
+---------
+
+1. Architecture:
+   - Consolidated service endpoints
+   - Clean separation of concerns
+   - Improved error handling
+   - Better session management
+
+2. Security:
+   - Robust authentication
+   - Secure file handling
+   - Protected state transitions
+
+3. Operations:
+   - Simplified deployment
+   - Better monitoring capabilities
+   - Improved error visibility
+   - Automatic retry handling
+
+Negative:
+---------
+
+1. Complexity:
+   - More complex session management
+   - Additional state validation required
+   - Complex transaction handling
+
+2. Performance:
+   - Additional database operations
+   - Session verification overhead
+
+3. Migration:
+   - Changes required in external graders
+   - New deployment procedures needed
+
+References
+**********
+
+Implementation References:
+
+* XQueue ViewSet Implementation: Link to PR
+* External Grader Integration Guide: Link to documentation
+
+Related ADRs:
+
+* ADR 1: Creation of ExternalGraderDetail Model
+* ADR 2: File Handling Implementation
+
+Documentation:
+
+* XQueue API Specification
+* External Grader Integration Guide
+* Session Management Documentation
+
+Architecture Guidelines:
+
+* Django REST Framework Best Practices
+* Open edX API Guidelines

docs/source/decisions/0004-transaction-handling-in-get_submission

@@ -0,0 +1,33 @@
+Improving Transaction Handling in Get Submission
+================================================
+
+Status
+------
+
+**Provisional** *2025-03-19*
+
+Context
+-------
+
+We identified a concurrency issue in the submission processing flow. Multiple instances of the xqueue watcher were processing the same transaction simultaneously, causing duplicate handling of submissions. Additionally, submissions remained stuck in the "pulled" state indefinitely when errors or unexpected interruptions occurred during processing.
+
+Decision
+--------
+
+We introduced a robust locking mechanism at the database level to prevent concurrent processing of the same submission. Specifically, we employed database row-level locking (`select_for_update(nowait=True)`) on submissions to ensure only one xqueue watcher instance can process a transaction at a time. Additionally, we implemented a timeout-based fallback, marking submissions in the "pulled" state as available again if they exceed a defined timeout period (5 minutes).
+
+Consequences
+------------
+
+Positive
+~~~~~~~~
+
+- Ensures data consistency by preventing duplicate transaction processing.
+- Implements automatic recovery for submissions stuck in the "pulled" state, improving overall system reliability.
+- Minimal impact on system performance due to efficient database-level locking.
+
+Negative
+~~~~~~~~
+
+- Slightly increased complexity in transaction handling logic.
+- Potential minor latency increase when contention occurs due to database locks.

requirements/base.txt

@@ -1,32 +1,16 @@
 #
-# This file is autogenerated by pip-compile with Python 3.12
+# This file is autogenerated by pip-compile with Python 3.11
 # by the following command:
 #
 #    make upgrade
 #
 asgiref==3.8.1
-    # via django
 django==4.2.21
-    # via
-    #   -c /home/runner/work/edx-submissions/edx-submissions/requirements/common_constraints.txt
-    #   -r requirements/base.in
-    #   django-model-utils
-    #   djangorestframework
-    #   edx-django-release-util
-    #   jsonfield
 django-model-utils==5.0.0
-    # via -r requirements/base.in
 djangorestframework==3.16.0
-    # via -r requirements/base.in
 edx-django-release-util==1.5.0
-    # via -r requirements/base.in
 jsonfield==3.1.0
-    # via -r requirements/base.in
 pytz==2025.2
-    # via -r requirements/base.in
 pyyaml==6.0.2
-    # via edx-django-release-util
 six==1.17.0
-    # via edx-django-release-util
 sqlparse==0.5.3
-    # via django

requirements/ci.txt

@@ -1,69 +1,25 @@
 #
-# This file is autogenerated by pip-compile with Python 3.12
+# This file is autogenerated by pip-compile with Python 3.11
 # by the following command:
 #
 #    make upgrade
 #
 cachetools==5.5.2
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
 certifi==2025.4.26
-    # via requests
 chardet==5.2.0
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
 charset-normalizer==3.4.2
-    # via requests
 colorama==0.4.6
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
 coverage[toml]==7.8.0
-    # via coveralls
 coveralls==4.0.1
-    # via -r requirements/ci.in
 distlib==0.3.9
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   virtualenv
 docopt==0.6.2
-    # via coveralls
 filelock==3.18.0
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
-    #   virtualenv
 idna==3.10
-    # via requests
 packaging==25.0
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   pyproject-api
-    #   tox
 platformdirs==4.3.8
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
-    #   virtualenv
 pluggy==1.5.0
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
 pyproject-api==1.9.1
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox
 requests==2.32.3
-    # via coveralls
 tox==4.25.0
-    # via -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
 urllib3==2.2.3
-    # via
-    #   -c /home/runner/work/edx-submissions/edx-submissions/requirements/common_constraints.txt
-    #   requests
 virtualenv==20.31.2
-    # via
-    #   -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
-    #   tox

requirements/common_constraints.txt

@@ -3,6 +3,16 @@
 # See BOM-2721 for more details.
 # Below is the copied and edited version of common_constraints
 
+# This is a temporary solution to override the real common_constraints.txt
+# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
+# See BOM-2721 for more details.
+# Below is the copied and edited version of common_constraints
+
+# This is a temporary solution to override the real common_constraints.txt
+# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
+# See BOM-2721 for more details.
+# Below is the copied and edited version of common_constraints
+
 # A central location for most common version constraints
 # (across edx repos) for pip-installation.
 #