Skip to content

Commit 22d7143

Browse files
ormsbeeormsbee
committed
feat: swtich LearningPackage restore to pydantic
This is a re-implementation of the restore part of backup_restore, with the goal of making it more robust and maintainable in the long term.
1 parent 91fc548 commit 22d7143

22 files changed

Lines changed: 1475 additions & 19 deletions

File tree

‎docs/decisions/0025-backup-restore.rst‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
25. Learning Package Serialization and Validation Approach
2+
==========================================================
3+
4+
Context
5+
-------
6+
7+
Content Libraries map 1:1 to LearningPackages and these need to be imported and exported as file archives. Initial support for this was released in Ulmo, but we wanted to revisit it to make it more robust during the Verawood timeline. This is part of that effort.
8+
9+
* Flexibility of Structure
10+
* Standardization of validation (JSON Schema)
11+
* Justify ZIP
12+
* Justify TOML
13+
* Max 100,000 items.
14+
* Use of fsspec as abstraction
15+
16+
Phases
17+
18+
Archive → Filesystem → Learning Package Doc + Resources → Input Models → LearningPackage
19+
20+
21+
Decision
22+
--------
23+
24+
Some key points:
25+
26+
1. We intentionally separate input and output formats because the output format
27+
will change over time, but the various input formats must continue to be
28+
supported. We don't inherit from one from the other because we don't *want*
29+
those changes to be automatically propogated--that breaks compatibility.
30+
2. We assemble into giant JSON in order to simplify validation and allow for
31+
more flexibility in structural representation. There's the archive layer and
32+
then the logical layer and then serialization into the database.
33+
34+
35+
Archive -> Model (validation) + Resources -> Database
36+
37+
38+
39+
Consequences
40+
------------
41+
42+
43+
44+
Rejected alternatives
45+
---------------------
46+

‎requirements/base.in‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,7 @@ rules<4.0 # Django extension for rules-based authorization check
1515
tomlkit # Parses and writes TOML configuration files
1616

1717
edx-organizations # Implemented the "Organization" model that CatalogCourse/CourseRun are keyed to
18+
19+
fsspec # Used by openedx_content's backup_restore to abstract zip access
20+
21+
pydantic[email] # Used by openedx_content's backup_restore for input validation

‎requirements/base.txt‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
#
77
amqp==5.3.1
88
# via kombu
9+
annotated-types==0.7.0
10+
# via pydantic
911
asgiref==3.11.1
1012
# via django
1113
attrs==26.1.0
@@ -67,7 +69,9 @@ djangorestframework==3.17.1
6769
# edx-drf-extensions
6870
# edx-organizations
6971
dnspython==2.8.0
70-
# via pymongo
72+
# via
73+
# email-validator
74+
# pymongo
7175
drf-jwt==1.19.2
7276
# via edx-drf-extensions
7377
edx-django-utils==8.0.1
@@ -82,8 +86,14 @@ edx-opaque-keys==4.0.0
8286
# edx-organizations
8387
edx-organizations==8.0.0
8488
# via -r requirements/base.in
89+
email-validator==2.3.0
90+
# via pydantic
91+
fsspec==2026.3.0
92+
# via -r requirements/base.in
8593
idna==3.11
86-
# via requests
94+
# via
95+
# email-validator
96+
# requests
8797
kombu==5.6.2
8898
# via celery
8999
packaging==26.0
@@ -96,6 +106,10 @@ psutil==7.2.2
96106
# via edx-django-utils
97107
pycparser==3.0
98108
# via cffi
109+
pydantic[email]==2.13.3
110+
# via -r requirements/base.in
111+
pydantic-core==2.46.3
112+
# via pydantic
99113
pyjwt[crypto]==2.12.1
100114
# via
101115
# drf-jwt
@@ -123,7 +137,13 @@ stevedore==5.7.0
123137
tomlkit==0.14.0
124138
# via -r requirements/base.in
125139
typing-extensions==4.15.0
126-
# via edx-opaque-keys
140+
# via
141+
# edx-opaque-keys
142+
# pydantic
143+
# pydantic-core
144+
# typing-inspection
145+
typing-inspection==0.4.2
146+
# via pydantic
127147
tzdata==2026.1
128148
# via kombu
129149
tzlocal==5.3.1

‎requirements/dev.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ amqp==5.3.1
88
# via
99
# -r requirements/quality.txt
1010
# kombu
11+
annotated-types==0.7.0
12+
# via
13+
# -r requirements/quality.txt
14+
# pydantic
1115
asgiref==3.11.1
1216
# via
1317
# -r requirements/quality.txt
@@ -164,6 +168,7 @@ djangorestframework-stubs==3.16.9
164168
dnspython==2.8.0
165169
# via
166170
# -r requirements/quality.txt
171+
# email-validator
167172
# pymongo
168173
docutils==0.22.4
169174
# via
@@ -192,6 +197,10 @@ edx-opaque-keys==4.0.0
192197
# edx-organizations
193198
edx-organizations==8.0.0
194199
# via -r requirements/quality.txt
200+
email-validator==2.3.0
201+
# via
202+
# -r requirements/quality.txt
203+
# pydantic
195204
filelock==3.25.2
196205
# via
197206
# -r requirements/ci.txt
@@ -200,6 +209,8 @@ filelock==3.25.2
200209
# virtualenv
201210
freezegun==1.5.5
202211
# via -r requirements/quality.txt
212+
fsspec==2026.3.0
213+
# via -r requirements/quality.txt
203214
grimp==3.14
204215
# via
205216
# -r requirements/quality.txt
@@ -211,6 +222,7 @@ id==1.6.1
211222
idna==3.11
212223
# via
213224
# -r requirements/quality.txt
225+
# email-validator
214226
# requests
215227
import-linter==2.11
216228
# via -r requirements/quality.txt
@@ -353,6 +365,12 @@ pycparser==3.0
353365
# via
354366
# -r requirements/quality.txt
355367
# cffi
368+
pydantic[email]==2.13.3
369+
# via -r requirements/quality.txt
370+
pydantic-core==2.46.3
371+
# via
372+
# -r requirements/quality.txt
373+
# pydantic
356374
pydocstyle==6.3.0
357375
# via -r requirements/quality.txt
358376
pygments==2.20.0
@@ -516,6 +534,13 @@ typing-extensions==4.15.0
516534
# grimp
517535
# import-linter
518536
# mypy
537+
# pydantic
538+
# pydantic-core
539+
# typing-inspection
540+
typing-inspection==0.4.2
541+
# via
542+
# -r requirements/quality.txt
543+
# pydantic
519544
tzdata==2026.1
520545
# via
521546
# -r requirements/quality.txt

‎requirements/doc.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@ amqp==5.3.1
1212
# via
1313
# -r requirements/test.txt
1414
# kombu
15+
annotated-types==0.7.0
16+
# via
17+
# -r requirements/test.txt
18+
# pydantic
1519
asgiref==3.11.1
1620
# via
1721
# -r requirements/test.txt
@@ -132,6 +136,7 @@ djangorestframework-stubs==3.16.9
132136
dnspython==2.8.0
133137
# via
134138
# -r requirements/test.txt
139+
# email-validator
135140
# pymongo
136141
doc8==2.0.0
137142
# via -r requirements/doc.in
@@ -161,15 +166,22 @@ edx-opaque-keys==4.0.0
161166
# edx-organizations
162167
edx-organizations==8.0.0
163168
# via -r requirements/test.txt
169+
email-validator==2.3.0
170+
# via
171+
# -r requirements/test.txt
172+
# pydantic
164173
freezegun==1.5.5
165174
# via -r requirements/test.txt
175+
fsspec==2026.3.0
176+
# via -r requirements/test.txt
166177
grimp==3.14
167178
# via
168179
# -r requirements/test.txt
169180
# import-linter
170181
idna==3.11
171182
# via
172183
# -r requirements/test.txt
184+
# email-validator
173185
# requests
174186
imagesize==2.0.0
175187
# via sphinx
@@ -249,6 +261,12 @@ pycparser==3.0
249261
# via
250262
# -r requirements/test.txt
251263
# cffi
264+
pydantic[email]==2.13.3
265+
# via -r requirements/test.txt
266+
pydantic-core==2.46.3
267+
# via
268+
# -r requirements/test.txt
269+
# pydantic
252270
pydata-sphinx-theme==0.16.1
253271
# via sphinx-book-theme
254272
pygments==2.20.0
@@ -381,7 +399,14 @@ typing-extensions==4.15.0
381399
# grimp
382400
# import-linter
383401
# mypy
402+
# pydantic
403+
# pydantic-core
384404
# pydata-sphinx-theme
405+
# typing-inspection
406+
typing-inspection==0.4.2
407+
# via
408+
# -r requirements/test.txt
409+
# pydantic
385410
tzdata==2026.1
386411
# via
387412
# -r requirements/test.txt

‎requirements/quality.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ amqp==5.3.1
88
# via
99
# -r requirements/test.txt
1010
# kombu
11+
annotated-types==0.7.0
12+
# via
13+
# -r requirements/test.txt
14+
# pydantic
1115
asgiref==3.11.1
1216
# via
1317
# -r requirements/test.txt
@@ -134,6 +138,7 @@ djangorestframework-stubs==3.16.9
134138
dnspython==2.8.0
135139
# via
136140
# -r requirements/test.txt
141+
# email-validator
137142
# pymongo
138143
docutils==0.22.4
139144
# via readme-renderer
@@ -158,8 +163,14 @@ edx-opaque-keys==4.0.0
158163
# edx-organizations
159164
edx-organizations==8.0.0
160165
# via -r requirements/test.txt
166+
email-validator==2.3.0
167+
# via
168+
# -r requirements/test.txt
169+
# pydantic
161170
freezegun==1.5.5
162171
# via -r requirements/test.txt
172+
fsspec==2026.3.0
173+
# via -r requirements/test.txt
163174
grimp==3.14
164175
# via
165176
# -r requirements/test.txt
@@ -169,6 +180,7 @@ id==1.6.1
169180
idna==3.11
170181
# via
171182
# -r requirements/test.txt
183+
# email-validator
172184
# requests
173185
import-linter==2.11
174186
# via -r requirements/test.txt
@@ -269,6 +281,12 @@ pycparser==3.0
269281
# via
270282
# -r requirements/test.txt
271283
# cffi
284+
pydantic[email]==2.13.3
285+
# via -r requirements/test.txt
286+
pydantic-core==2.46.3
287+
# via
288+
# -r requirements/test.txt
289+
# pydantic
272290
pydocstyle==6.3.0
273291
# via -r requirements/quality.in
274292
pygments==2.20.0
@@ -394,6 +412,13 @@ typing-extensions==4.15.0
394412
# grimp
395413
# import-linter
396414
# mypy
415+
# pydantic
416+
# pydantic-core
417+
# typing-inspection
418+
typing-inspection==0.4.2
419+
# via
420+
# -r requirements/test.txt
421+
# pydantic
397422
tzdata==2026.1
398423
# via
399424
# -r requirements/test.txt

‎requirements/test.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ amqp==5.3.1
88
# via
99
# -r requirements/base.txt
1010
# kombu
11+
annotated-types==0.7.0
12+
# via
13+
# -r requirements/base.txt
14+
# pydantic
1115
asgiref==3.11.1
1216
# via
1317
# -r requirements/base.txt
@@ -118,6 +122,7 @@ djangorestframework-stubs==3.16.9
118122
dnspython==2.8.0
119123
# via
120124
# -r requirements/base.txt
125+
# email-validator
121126
# pymongo
122127
drf-jwt==1.19.2
123128
# via
@@ -138,13 +143,20 @@ edx-opaque-keys==4.0.0
138143
# edx-organizations
139144
edx-organizations==8.0.0
140145
# via -r requirements/base.txt
146+
email-validator==2.3.0
147+
# via
148+
# -r requirements/base.txt
149+
# pydantic
141150
freezegun==1.5.5
142151
# via -r requirements/test.in
152+
fsspec==2026.3.0
153+
# via -r requirements/base.txt
143154
grimp==3.14
144155
# via import-linter
145156
idna==3.11
146157
# via
147158
# -r requirements/base.txt
159+
# email-validator
148160
# requests
149161
import-linter==2.11
150162
# via -r requirements/test.in
@@ -199,6 +211,12 @@ pycparser==3.0
199211
# via
200212
# -r requirements/base.txt
201213
# cffi
214+
pydantic[email]==2.13.3
215+
# via -r requirements/base.txt
216+
pydantic-core==2.46.3
217+
# via
218+
# -r requirements/base.txt
219+
# pydantic
202220
pygments==2.20.0
203221
# via
204222
# pytest
@@ -279,6 +297,13 @@ typing-extensions==4.15.0
279297
# grimp
280298
# import-linter
281299
# mypy
300+
# pydantic
301+
# pydantic-core
302+
# typing-inspection
303+
typing-inspection==0.4.2
304+
# via
305+
# -r requirements/base.txt
306+
# pydantic
282307
tzdata==2026.1
283308
# via
284309
# -r requirements/base.txt

0 commit comments

Comments
 (0)