feat: implement submission file workflow with comprehensive testing

- Create Submission file model
- Add classes to handle file errors
- Develop SubmissionFileProcessor for create files in DB
- Add TestSubmissionFileProcessor with extensive test coverage
  - Test file processing with different input types (bytes, file objects)
  - Implement error handling tests for IO errors and invalid files
  - Verify complete file processing workflow
  - Validation tests for file addition in create_external_grader_detail

- Document architectural decisions with ADR
- Add test queue folder in .gitignore

Author: leoaulasneo98

.gitignore

@@ -63,3 +63,4 @@ test.txt.tmp
 
 # VSCode
 .vscode
+test_queue/*

docs/source/decisions/0002-add-submission-file-model.rst

@@ -0,0 +1,77 @@
+2. File Handling Implementation for Submission System
+#####################################################
+
+Status
+******
+
+**Provisional** *2025-02-14*
+
+Implemented by https://github.com/openedx/edx-submissions/pull/286
+
+Context
+*******
+
+As part of the XQueue migration effort detailed in ADR 0001, we need to implement a file handling system within edx-submissions. Currently, XQueue manages file submissions through a tightly coupled approach.
+
+### Current Limitations
+
+1. **Inadequate File Management**: XQueue's approach relies on JSON strings in character fields, with size constraints and manual URL manipulation for file handling.
+
+2. **Process Inefficiencies**: The current system uses synchronous HTTP for file retrieval, lacks proper validation, and tightly couples submission processing with file handling.
+
+3. **Integration Challenges**: External graders depend on specific URL formats with HTTP-based retrieval, embedding file information directly in submission payloads.
+
+Decision
+********
+
+We will implement a dedicated file management system for the assessment submission process, focusing on workflow and educational needs:
+
+1. **Centralized Storage**: Create a unified repository for student-submitted files, ensuring they are properly associated with their assessments and accessible throughout the grading process.
+
+2. **Streamlined Workflow**: Design a clear process where files are automatically processed during submission creation, securely stored, and efficiently delivered to grading systems.
+
+3. **Consistent Experience**: Maintain compatibility with existing systems to ensure a smooth transition, allowing instructors and external graders to access files without changes to their established workflows.
+
+Consequences
+************
+
+Positive:
+---------
+
+1. **Architecture**: Clean separation of concerns, improved maintainability, better error handling, optimized database access
+
+2. **Integration**: Seamless xqueue-watcher compatibility, support for existing workflows, minimal client code changes
+
+3. **Operations**: Robust file validation, improved tracking, better error visibility, simplified lifecycle management
+
+Negative:
+---------
+
+1. **Technical**: Additional database structures
+
+2. **Migration**: Temporary system complexity, additional testing needs
+
+3. **Performance**: File processing overhead
+
+References
+**********
+
+Current System Documentation:
+   * XQueue Repository: https://github.com/openedx/xqueue
+   * XQueue Watcher Repository: https://github.com/openedx/xqueue-watcher
+
+Related Repositories:
+   * edx-submissions: https://github.com/openedx/edx-submissions
+   * edx-platform: https://github.com/openedx/edx-platform
+   * XQueue Repository: https://github.com/openedx/xqueue
+
+Related Documentation:
+   * ADR 0001: Creation of ExternalGraderDetail Model for XQueue Migration
+
+Future Event Integration:
+   * Open edX Events Framework: https://github.com/openedx/openedx-events
+   * Event Bus Documentation: https://openedx.atlassian.net/wiki/spaces/AC/pages/124125264/Event+Bus
+
+Related Architecture Documents:
+   * Open edX Architecture Guidelines: https://openedx.atlassian.net/wiki/spaces/AC/pages/124125264/Architecture+Guidelines
+

submissions/api.py

@@ -10,11 +10,13 @@
 
 from django.conf import settings
 from django.core.cache import cache
+from django.core.files.base import ContentFile
 from django.db import DatabaseError, IntegrityError, transaction
 
 # SubmissionError imported so that code importing this api has access
 from submissions.errors import (  # pylint: disable=unused-import
     ExternalGraderQueueEmptyError,
+    FileProcessingError,
     SubmissionError,
     SubmissionInternalError,
     SubmissionNotFoundError,
@@ -28,6 +30,7 @@
     ScoreSummary,
     StudentItem,
     Submission,
+    SubmissionFile,
     score_reset,
     score_set
 )
@@ -49,7 +52,6 @@
 TOP_SUBMISSIONS_CACHE_TIMEOUT = 300
 
 
-# pylint: disable=unused-argument
 def create_external_grader_detail(student_item_dict,
                                   answer,
                                   queue_name: str,
@@ -93,9 +95,13 @@ def create_external_grader_detail(student_item_dict,
             submission_uuid=submission_uuid,
             queue_name=queue_name,
             grader_file_name=grader_file_name,
-            points_possible=points_possible,
+            points_possible=points_possible)
+
+        files_dict = external_grader_additional_data.get('files')
+        if files_dict:
+            file_processor = SubmissionFileProcessor(instance)
+            file_processor.process_files(files_dict)
 
-        )
         return instance
 
     except DatabaseError as error:
@@ -106,6 +112,101 @@ def create_external_grader_detail(student_item_dict,
         raise SubmissionInternalError(error_message) from error
 
 
+class SubmissionFileProcessor:
+    """
+    Process file operations for submissions
+    """
+
+    def __init__(self, external_grader):
+        self.external_grader = external_grader
+
+    def process_files(self, files_dict):
+        """
+        Process uploaded files from an Open edX environment and store them as SubmissionFile objects.
+
+        This method handles various file object types that might be received from Open edX, including:
+        - Bytes objects
+        - Any object with a 'read' method (including Native Open edX FileObjForWebobFiles objects)
+
+
+        The method performs the following operations:
+        1. Validates each file object typeƒ
+        2. Reads content from file-like objects
+        3. Converts byte content to ContentFile objects
+        4. Creates SubmissionFile records in the database
+        5. Returns URLs in xqueue-compatible format
+
+        Args:
+            files_dict (dict): Dictionary mapping filenames to file objects.
+                              Format: {filename: file_object, ...}
+
+        Returns:
+            dict: Dictionary mapping original filenames to xqueue URLs.
+                  Format: {filename: "/queue_name/uuid", ...}
+
+        Raises:
+            FileProcessingError: If there's an error reading from a file object,
+                                including I/O errors or Unicode decoding errors.
+
+        Example:
+            >>> external_grader = ExternalGraderDetail.create_from_uuid(
+            submission_uuid=submission_uuid,
+            queue_name=queue_name,
+            grader_file_name=grader_file_name,
+            points_possible=points_possible)
+            >>> file_processor = SubmissionFileProcessor(external_grader)
+            >>> files = {'assignment.py': file_obj}
+            >>> urls = file_processor.process_files(files)
+            >>> print(urls)
+            {'assignment.py': '/my_queue/550e8400-e29b-41d4-a716-446655440000'}
+        """
+        files_urls = {}
+        for filename, file_obj in files_dict.items():
+            # Validate file object type
+            if not (isinstance(file_obj, bytes) or hasattr(file_obj, 'read')):
+                actual_type = type(file_obj).__name__
+                logger.error(f"Invalid file object type for {filename}: got {actual_type}")
+                raise TypeError(
+                    f"Invalid file object type for {filename}: expected bytes or an object with 'read' method, "
+                    f"got {actual_type}")
+
+            # Handle file-like objects by reading their content
+            # This returns a bytes object that will be converted to ContentFile later
+            if hasattr(file_obj, 'read'):
+                try:
+                    file_obj = file_obj.read()  # read() returns a bytes object
+                except (IOError, OSError) as e:
+                    logger.error(f"Error reading file {filename}: {e}")
+                    raise FileProcessingError(f"Error reading file {filename}: {e}") from e
+                except UnicodeDecodeError as e:
+                    logger.error(f"Error decoding file {filename}: {e}")
+                    raise FileProcessingError(f"Error decoding file {filename}: {e}") from e
+
+            # Convert bytes to ContentFile
+            # The read() method from file-like objects returns bytes, which we handle here
+            if isinstance(file_obj, bytes):
+                file_obj = ContentFile(file_obj, name=filename)
+
+            # Create a SubmissionFile record for storage and retrieval
+            submission_file = SubmissionFile.objects.create(
+                external_grader=self.external_grader,
+                file=file_obj,
+                original_filename=filename
+            )
+            files_urls[filename] = submission_file.xqueue_url
+
+        return files_urls
+
+    def get_files_for_grader(self):
+        """
+        Returns files in format expected by xqueue-watcher
+        """
+        return {
+            file.original_filename: file.file.url
+            for file in self.external_grader.files.all()
+        }
+
+
 def create_submission(
     student_item_dict,
     answer,

submissions/errors.py

@@ -40,6 +40,17 @@ class ExternalGraderQueueEmptyError(SubmissionError):
     """
 
 
+class FileProcessingError(SubmissionError):
+    """
+    Exception raised when there's an error reading or processing a file.
+
+    This exception is raised when file operations fail, such as:
+    - I/O errors when reading file content
+    - OS errors during file operations
+    - Unicode decoding errors when processing file content
+    """
+
+
 class SubmissionRequestError(SubmissionError):
     """
     This error is raised when there was a request-specific error

submissions/migrations/0005_submissionfile.py

@@ -0,0 +1,32 @@
+# Generated by Django 4.2.19 on 2025-04-21 13:20
+
+from django.db import migrations, models
+import django.db.models.deletion
+import django.utils.timezone
+import submissions.models
+import uuid
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('submissions', '0004_externalgraderdetail'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='SubmissionFile',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('uuid', models.UUIDField(default=uuid.uuid4, editable=False)),
+                ('file', models.FileField(max_length=512, upload_to=submissions.models.submission_file_path)),
+                ('original_filename', models.CharField(max_length=255)),
+                ('created_at', models.DateTimeField(default=django.utils.timezone.now)),
+                ('external_grader', models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL,
+                                                      related_name='files', to='submissions.externalgraderdetail')),
+            ],
+            options={
+                'indexes': [models.Index(fields=['external_grader', 'uuid'], name='submissions_externa_ff8089_idx')],
+            },
+        ),
+    ]

submissions/models.py

@@ -10,6 +10,7 @@
 """
 
 import logging
+import os
 from datetime import timedelta
 from uuid import uuid4
 
@@ -693,3 +694,59 @@ def update_status(self, new_status):
     def create_from_uuid(cls, submission_uuid, **kwargs):
         submission = Submission.objects.get(uuid=submission_uuid)
         return cls.objects.create(submission=submission, **kwargs)
+
+
+def submission_file_path(instance, _):
+    """
+    Generate file path for submission files.
+    Format: queue_name/uuid
+    The filename is replaced with the UUID to ensure uniqueness without preserving extension.
+    """
+    return os.path.join(
+        instance.external_grader.queue_name,
+        f"{instance.uuid}"
+    )
+
+
+class SubmissionFile(models.Model):
+    """
+    Model to handle files associated with submissions
+    """
+    uuid = models.UUIDField(default=uuid4, editable=False)  # legacy S3 key
+    external_grader = models.ForeignKey(
+        'submissions.ExternalGraderDetail',
+        on_delete=models.SET_NULL,
+        related_name='files',
+        null=True,
+    )
+    file = models.FileField(
+        upload_to=submission_file_path,
+        max_length=512
+    )
+    original_filename = models.CharField(max_length=255)  # This is necessary to send file name to xqueue-watcher
+    created_at = models.DateTimeField(default=now)
+
+    class Meta:
+        indexes = [
+            models.Index(fields=['external_grader', 'uuid']),
+        ]
+
+    @property
+    def xqueue_url(self):
+        """
+        Returns a URL in the XQueue-compatible format: /queue_name/uuid
+
+        This format is used for file references in both the legacy XQueue system
+        and the new integrated standard. It maintains backward compatibility
+        while supporting the migration from the external XQueue API to the
+        integrated Open edX solution.
+
+        The URL follows the pattern: /{queue_name}/{submission_uuid}
+        where:
+        - queue_name: identifies the external grader queue
+        - uuid: uniquely identifies this submission (legacy S3 key)
+
+        Returns:
+            str: Formatted URL path following XQueue conventions
+        """
+        return f"/{self.external_grader.queue_name}/{self.uuid}"