Skip to content

Commit 0836a82

Browse files
leoaulasneo98leoaulasneo98
committed
feat: implement submission file workflow with comprehensive testing
- Create Submission file model - Add custom storage to use ORA bucket - Develop SubmissionFileProcessor for create files in DB - Add TestSubmissionFileProcessor with extensive test coverage - Verify complete file processing workflow - Validation tests for file addition in create_external_grader_detail - Document architectural decisions with ADR - Add test queue folder in .gitignore
1 parent 0c351ce commit 0836a82

19 files changed

Lines changed: 928 additions & 216 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,3 +63,4 @@ test.txt.tmp
6363

6464
# VSCode
6565
.vscode
66+
test_queue/*

‎docs/source/decisions/0002-add-submission-file-model.rst‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
2. File Handling Implementation for Submission System
2+
#####################################################
3+
4+
Status
5+
******
6+
7+
**Provisional** *2025-02-14*
8+
9+
Implemented by https://github.com/openedx/edx-submissions/pull/286
10+
11+
Context
12+
*******
13+
14+
As part of the XQueue migration effort detailed in ADR 0001, we need to implement a file handling system within edx-submissions. Currently, XQueue manages file submissions through a tightly coupled approach.
15+
16+
### Current Limitations
17+
18+
1. **Inadequate File Management**: XQueue's approach relies on JSON strings in character fields, with size constraints and manual URL manipulation for file handling.
19+
20+
2. **Process Inefficiencies**: The current system uses synchronous HTTP for file retrieval, lacks proper validation, and tightly couples submission processing with file handling.
21+
22+
3. **Integration Challenges**: External graders depend on specific URL formats with HTTP-based retrieval, embedding file information directly in submission payloads.
23+
24+
Decision
25+
********
26+
27+
We will implement a dedicated file management system for the assessment submission process, focusing on workflow and educational needs:
28+
29+
1. **Centralized Storage**: Create a unified repository for student-submitted files, ensuring they are properly associated with their assessments and accessible throughout the grading process.
30+
31+
2. **Streamlined Workflow**: Design a clear process where files are automatically processed during submission creation, securely stored, and efficiently delivered to grading systems.
32+
33+
3. **Consistent Experience**: Maintain compatibility with existing systems to ensure a smooth transition, allowing instructors and external graders to access files without changes to their established workflows.
34+
35+
Consequences
36+
************
37+
38+
Positive:
39+
---------
40+
41+
1. **Architecture**: Clean separation of concerns, improved maintainability, better error handling, optimized database access
42+
43+
2. **Integration**: Seamless xqueue-watcher compatibility, support for existing workflows, minimal client code changes
44+
45+
3. **Operations**: Robust file validation, improved tracking, better error visibility, simplified lifecycle management
46+
47+
Negative:
48+
---------
49+
50+
1. **Technical**: Additional database structures
51+
52+
2. **Migration**: Temporary system complexity, additional testing needs
53+
54+
3. **Performance**: File processing overhead
55+
56+
References
57+
**********
58+
59+
Current System Documentation:
60+
* XQueue Repository: https://github.com/openedx/xqueue
61+
* XQueue Watcher Repository: https://github.com/openedx/xqueue-watcher
62+
63+
Related Repositories:
64+
* edx-submissions: https://github.com/openedx/edx-submissions
65+
* edx-platform: https://github.com/openedx/edx-platform
66+
* XQueue Repository: https://github.com/openedx/xqueue
67+
68+
Related Documentation:
69+
* ADR 0001: Creation of ExternalGraderDetail Model for XQueue Migration
70+
71+
Future Event Integration:
72+
* Open edX Events Framework: https://github.com/openedx/openedx-events
73+
* Event Bus Documentation: https://openedx.atlassian.net/wiki/spaces/AC/pages/124125264/Event+Bus
74+
75+
Related Architecture Documents:
76+
* Open edX Architecture Guidelines: https://openedx.atlassian.net/wiki/spaces/AC/pages/124125264/Architecture+Guidelines
77+

‎requirements/base.in‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,5 @@ jsonfield
88
pytz
99
djangorestframework
1010
edx-django-release-util
11+
django-storages
12+
boto3

‎requirements/base.txt‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,55 @@
11
#
2-
# This file is autogenerated by pip-compile with Python 3.12
2+
# This file is autogenerated by pip-compile with Python 3.11
33
# by the following command:
44
#
55
# make upgrade
66
#
77
asgiref==3.8.1
88
# via django
9+
boto3==1.38.9
10+
# via -r requirements/base.in
11+
botocore==1.38.9
12+
# via
13+
# boto3
14+
# s3transfer
915
django==4.2.20
1016
# via
11-
# -c /home/runner/work/edx-submissions/edx-submissions/requirements/common_constraints.txt
17+
# -c /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/common_constraints.txt
1218
# -r requirements/base.in
1319
# django-model-utils
20+
# django-storages
1421
# djangorestframework
1522
# edx-django-release-util
1623
# jsonfield
1724
django-model-utils==5.0.0
1825
# via -r requirements/base.in
26+
django-storages==1.14.6
27+
# via -r requirements/base.in
1928
djangorestframework==3.16.0
2029
# via -r requirements/base.in
2130
edx-django-release-util==1.5.0
2231
# via -r requirements/base.in
32+
jmespath==1.0.1
33+
# via
34+
# boto3
35+
# botocore
2336
jsonfield==3.1.0
2437
# via -r requirements/base.in
38+
python-dateutil==2.9.0.post0
39+
# via botocore
2540
pytz==2025.2
2641
# via -r requirements/base.in
2742
pyyaml==6.0.2
2843
# via edx-django-release-util
44+
s3transfer==0.12.0
45+
# via boto3
2946
six==1.17.0
30-
# via edx-django-release-util
47+
# via
48+
# edx-django-release-util
49+
# python-dateutil
3150
sqlparse==0.5.3
3251
# via django
52+
urllib3==2.2.3
53+
# via
54+
# -c /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/common_constraints.txt
55+
# botocore

‎requirements/ci.in‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,5 @@
33
-r tox.txt
44

55
coveralls
6+
django-storages
7+
boto3

‎requirements/ci.txt‎

Lines changed: 40 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,96 @@
11
#
2-
# This file is autogenerated by pip-compile with Python 3.12
2+
# This file is autogenerated by pip-compile with Python 3.11
33
# by the following command:
44
#
55
# make upgrade
66
#
7+
asgiref==3.8.1
8+
# via django
9+
boto3==1.38.9
10+
# via -r requirements/ci.in
11+
botocore==1.38.9
12+
# via
13+
# boto3
14+
# s3transfer
715
cachetools==5.5.2
816
# via
9-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
17+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
1018
# tox
1119
certifi==2025.4.26
1220
# via requests
1321
chardet==5.2.0
1422
# via
15-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
23+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
1624
# tox
1725
charset-normalizer==3.4.1
1826
# via requests
1927
colorama==0.4.6
2028
# via
21-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
29+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
2230
# tox
2331
coverage[toml]==7.8.0
2432
# via coveralls
2533
coveralls==4.0.1
2634
# via -r requirements/ci.in
2735
distlib==0.3.9
2836
# via
29-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
37+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
3038
# virtualenv
39+
django==4.2.20
40+
# via
41+
# -c /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/common_constraints.txt
42+
# django-storages
43+
django-storages==1.14.6
44+
# via -r requirements/ci.in
3145
docopt==0.6.2
3246
# via coveralls
3347
filelock==3.18.0
3448
# via
35-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
49+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
3650
# tox
3751
# virtualenv
3852
idna==3.10
3953
# via requests
54+
jmespath==1.0.1
55+
# via
56+
# boto3
57+
# botocore
4058
packaging==25.0
4159
# via
42-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
60+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
4361
# pyproject-api
4462
# tox
4563
platformdirs==4.3.7
4664
# via
47-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
65+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
4866
# tox
4967
# virtualenv
5068
pluggy==1.5.0
5169
# via
52-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
70+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
5371
# tox
5472
pyproject-api==1.9.0
5573
# via
56-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
74+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
5775
# tox
76+
python-dateutil==2.9.0.post0
77+
# via botocore
5878
requests==2.32.3
5979
# via coveralls
80+
s3transfer==0.12.0
81+
# via boto3
82+
six==1.17.0
83+
# via python-dateutil
84+
sqlparse==0.5.3
85+
# via django
6086
tox==4.25.0
61-
# via -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
87+
# via -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
6288
urllib3==2.2.3
6389
# via
64-
# -c /home/runner/work/edx-submissions/edx-submissions/requirements/common_constraints.txt
90+
# -c /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/common_constraints.txt
91+
# botocore
6592
# requests
6693
virtualenv==20.30.0
6794
# via
68-
# -r /home/runner/work/edx-submissions/edx-submissions/requirements/tox.txt
95+
# -r /Users/leonardoberoes/Documents/Aulasneo/Projects/edx-submissions/requirements/tox.txt
6996
# tox

‎requirements/common_constraints.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,31 @@
33
# See BOM-2721 for more details.
44
# Below is the copied and edited version of common_constraints
55

6+
# This is a temporary solution to override the real common_constraints.txt
7+
# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
8+
# See BOM-2721 for more details.
9+
# Below is the copied and edited version of common_constraints
10+
11+
# This is a temporary solution to override the real common_constraints.txt
12+
# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
13+
# See BOM-2721 for more details.
14+
# Below is the copied and edited version of common_constraints
15+
16+
# This is a temporary solution to override the real common_constraints.txt
17+
# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
18+
# See BOM-2721 for more details.
19+
# Below is the copied and edited version of common_constraints
20+
21+
# This is a temporary solution to override the real common_constraints.txt
22+
# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
23+
# See BOM-2721 for more details.
24+
# Below is the copied and edited version of common_constraints
25+
26+
# This is a temporary solution to override the real common_constraints.txt
27+
# In edx-lint, until the pyjwt constraint in edx-lint has been removed.
28+
# See BOM-2721 for more details.
29+
# Below is the copied and edited version of common_constraints
30+
631
# A central location for most common version constraints
732
# (across edx repos) for pip-installation.
833
#

0 commit comments

Comments
 (0)