feat!: Package and Entity keys are now opaque refs

The `{LearningPackage,PublishableEntity}.key` fields were always
meant to be externally-supplied refs whose format does not matter.
However, they were effectively being used as parseable psuedo-keys
for identifying containers and components, with more assumptions to their
structure being baked in over time, both within and outside openedx-core.

This commit renames them to `{package,entity}_ref`, and removes all
parsing of them from the API and from the internals (except for where
required for backcompat with Ulmo ZIP backups). Instead, the pairings
of (component_type,component_code) and (container_type,container_code)
are used to identify components and containers; their respective
entity_refs are derived from those *by conventional default* but also
are customizable by callers.

BREAKING CHANGE: Renamed key_field(...) -> ref_field(...) in openedx_django_lib.

BREAKING CHANGE: Renamed PublishableEntity.key -> PublishableEntityKey.entity_ref
BREAKING CHANGE: Renamed PublishableEntityMixin.key -> PublishableEntityMixin.entity_ref
BREAKING CHANGE: In create_publishable_entity(...), renamed param key -> entity_ref
BREAKING CHANGE: Renamed get_publishable_entity_by_key(...) ->
  get_publishable_entity_by_ref(...), and renamed param key -> entity_ref
BREAKING CHANGE: In get_entity_collections(...), renamed param entity_key -> entity_ref
BREAKING CHANGE: In create_component(...)/create_container(...):
  * Positional key argument removed
  * Optional entity_ref argument added
  * Defaults to defaults to "{namespace}:{type_name}:{component_code}" if unspecified
BREAKING CHANGE: Function get_or_create_component_type_by_entity_key() removed.
BREAKING CHANGE: Rename get_container_children_entities_keys(...) ->
  get_container_children_entity_refs(...)
BREAKING CHANGE: Renamed get_container_by_key(...) -> get_container_by_ref(..)
  and renamed param key -> entity_ref.
BREAKING CHANGE: Renamed get_container_children_entities_keys(...) ->
  get_container_children_entity_refs(...)

BRAEKING CHANGE: Renamed LearningPackage.key -> LearningPackage.package_ref.
BREAKING CHANGE: In create_learning_package(...), renamed param key -> package_ref
BREAKING CHANGE: In update_learning_package(...), renamed param key -> package_ref
BREAKING CHANGE: Renamed get_learning_package_by_key(...) ->
  get_learning_package_by_ref(...), and renamed param key -> package_ref
BREAKING CHANGE: In learning_package_exists(...), renamed param key -> package_ref

BREAKING CHANGE: In look_up_component_version_media(...):
  * Renamed param learning_package_key -> learning_package_ref
  * Renamed param component_key -> entity_ref

BREAKING CHANGE: In add_assets_to_component management command:
  * Renamed argument learning_package_key -> learning_package_ref
  * Renamed argument component_key -> entity_ref

BREAKING CHANGE In load_learning_package:
  * Renamed  param key -> package_ref
  * In return dict, within sub-dict ["lp_restored_data"]:
    * Renamed ["key"] -> ["package_ref"]
    * Renamed ["archive_lp_key"] -> ["archive_package_ref"]
    * Renamed ["archive_org_key"] -> ["archive_org_code"]
    * Renamed ["archive_slug"] -> ["archive_package_code"]
  * If archive_package_ref cannot be parsed into "{_prefix}:{org_code}:{package_code}", then
    archvie_org_code and archive_package_code will both be None. Callers should handle this case.
    (Previously, a ValueError would be raised by backup-restore code.)

Backup/Restore format changes (non-breaking):
* In [learning_package], key field is renamed -> package_ref.
* In [entity], key field is renamed -> entity_ref.
* For compatibility with Ulmo instances, both key and {package,entity}_ref
  will be written in all new archives.
* For compatibility with Ulmo archives, both key and {package,entity}_ref
  will be read, with priority given to the latter.

Part of: #322

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: kdmccormick

olx_importer/management/commands/load_components.py

@@ -61,33 +61,33 @@ def init_known_types(self):
 
     def add_arguments(self, parser):
         parser.add_argument("course_data_path", type=pathlib.Path)
-        parser.add_argument("learning_package_key", type=str)
+        parser.add_argument("learning_package_ref", type=str)
 
-    def handle(self, course_data_path, learning_package_key, **options):
+    def handle(self, course_data_path, learning_package_ref, **options):
         self.course_data_path = course_data_path
-        self.learning_package_key = learning_package_key
-        self.load_course_data(learning_package_key)
+        self.learning_package_ref = learning_package_ref
+        self.load_course_data(learning_package_ref)
 
     def get_course_title(self):
         course_type_dir = self.course_data_path / "course"
         course_xml_file = next(course_type_dir.glob("*.xml"))
         course_root = ET.parse(course_xml_file).getroot()
         return course_root.attrib.get("display_name", "Unknown Course")
 
-    def load_course_data(self, learning_package_key):
+    def load_course_data(self, learning_package_ref):
         print(f"Importing course from: {self.course_data_path}")
         now = datetime.now(timezone.utc)
         title = self.get_course_title()
 
-        if content_api.learning_package_exists(learning_package_key):
+        if content_api.learning_package_exists(learning_package_ref):
             raise CommandError(
-                f"{learning_package_key} already exists. "
+                f"{learning_package_ref} already exists. "
                 "This command currently only supports initial import."
             )
 
         with transaction.atomic():
             self.learning_package = content_api.create_learning_package(
-                learning_package_key, title, created=now,
+                learning_package_ref, title, created=now,
             )
             for block_type in SUPPORTED_TYPES:
                 self.import_block_type(block_type, now) #, publish_log_entry)

src/openedx_content/applets/backup_restore/api.py

@@ -5,26 +5,28 @@
 
 from django.contrib.auth.models import User as UserType  # pylint: disable=imported-auth-user
 
-from ..publishing.api import get_learning_package_by_key
+from ..publishing.api import get_learning_package_by_ref
 from .zipper import LearningPackageUnzipper, LearningPackageZipper
 
 
-def create_zip_file(lp_key: str, path: str, user: UserType | None = None, origin_server: str | None = None) -> None:
+def create_zip_file(
+        package_ref: str, path: str, user: UserType | None = None, origin_server: str | None = None
+) -> None:
     """
     Creates a dump zip file for the given learning package key at the given path.
     The zip file contains a TOML representation of the learning package and its contents.
 
-    Can throw a NotFoundError at get_learning_package_by_key
+    Can throw a NotFoundError at get_learning_package_by_ref
     """
-    learning_package = get_learning_package_by_key(lp_key)
+    learning_package = get_learning_package_by_ref(package_ref)
     LearningPackageZipper(learning_package, user, origin_server).create_zip(path)
 
 
-def load_learning_package(path: str, key: str | None = None, user: UserType | None = None) -> dict:
+def load_learning_package(path: str, package_ref: str | None = None, user: UserType | None = None) -> dict:
     """
     Loads a learning package from a zip file at the given path.
     Restores the learning package and its contents to the database.
     Returns a dictionary with the status of the operation and any errors encountered.
     """
     with zipfile.ZipFile(path, "r") as zipf:
-        return LearningPackageUnzipper(zipf, key, user).load()
+        return LearningPackageUnzipper(zipf, package_ref, user).load()

src/openedx_content/applets/backup_restore/serializers.py

@@ -12,15 +12,30 @@ class LearningPackageSerializer(serializers.Serializer):  # pylint: disable=abst
     """
     Serializer for learning packages.
 
+    Archives created in Verawood or later write ``package_ref``. Archives
+    created in Ulmo write ``key``. Both are accepted; ``package_ref`` takes
+    precedence.
+
     Note:
-        The `key` field is serialized, but it is generally not trustworthy for restoration.
-        During restore, a new key may be generated or overridden.
+        The ref/key field is serialized but is generally not trustworthy for
+        restoration. During restore, a new ref may be generated or overridden.
     """
+
     title = serializers.CharField(required=True)
-    key = serializers.CharField(required=True)
+    package_ref = serializers.CharField(required=False)
+    key = serializers.CharField(required=False)
     description = serializers.CharField(required=True, allow_blank=True)
     created = serializers.DateTimeField(required=True, default_timezone=timezone.utc)
 
+    def validate(self, attrs):
+        package_ref = attrs.pop("package_ref", None)
+        legacy_key = attrs.pop("key", None)
+        ref = package_ref or legacy_key
+        if not ref:
+            raise serializers.ValidationError("Either 'package_ref' or 'key' is required.")
+        attrs["package_ref"] = ref  # Normalise to 'package_ref' for create_learning_package.
+        return attrs
+
 
 class LearningPackageMetadataSerializer(serializers.Serializer):  # pylint: disable=abstract-method
     """
@@ -40,18 +55,33 @@ class LearningPackageMetadataSerializer(serializers.Serializer):  # pylint: disa
 class EntitySerializer(serializers.Serializer):  # pylint: disable=abstract-method
     """
     Serializer for publishable entities.
+
+    Archives created in Verawood or later write ``entity_ref``. Archives
+    created in Ulmo use ``key``. Both are accepted; ``entity_ref`` takes
+    precedence.
     """
+
     can_stand_alone = serializers.BooleanField(required=True)
-    key = serializers.CharField(required=True)
+    entity_ref = serializers.CharField(required=False)
+    key = serializers.CharField(required=False)
     created = serializers.DateTimeField(required=True, default_timezone=timezone.utc)
 
+    def validate(self, attrs):
+        entity_ref = attrs.pop("entity_ref", None)
+        legacy_key = attrs.pop("key", None)
+        ref = entity_ref or legacy_key
+        if not ref:
+            raise serializers.ValidationError("Either 'entity_ref' or 'key' is required.")
+        attrs["entity_ref"] = ref
+        return attrs
+
 
 class EntityVersionSerializer(serializers.Serializer):  # pylint: disable=abstract-method
     """
     Serializer for publishable entity versions.
     """
     title = serializers.CharField(required=True)
-    entity_key = serializers.CharField(required=True)
+    entity_ref = serializers.CharField(required=True)
     created = serializers.DateTimeField(required=True, default_timezone=timezone.utc)
     version_num = serializers.IntegerField(required=True)
 
@@ -78,6 +108,7 @@ def validate(self, attrs):
         ``"{namespace}:{type_name}:{component_code}"``, so we fall back to
         parsing that for backwards compatibility.
         """
+        super().validate(attrs)
         component_section = attrs.pop("component", None)
         if component_section:
             # Verawood+ format: component_type and component_code are explicit.
@@ -92,14 +123,21 @@ def validate(self, attrs):
                 ) from exc
             component_type_obj = components_api.get_or_create_component_type(namespace, type_name)
         else:
-            # Ulmo (legacy) format: parse the entity key.
-            entity_key = attrs["key"]
+            # Ulmo (legacy) format: parse the entity_ref (which ws normalized
+            # from "key" in super.validate()) assuming the format:
+            # (namespace, type_name, component_code). This parsing is
+            # intentionally only here — entity_ref must not be parsed anywhere
+            # else in the codebase. Verawood+ archives may not follow this
+            # convention.
+            entity_ref = attrs["entity_ref"]
             try:
-                component_type_obj, component_code = (
-                    components_api.get_or_create_component_type_by_entity_key(entity_key)
-                )
+                namespace, type_name, component_code = entity_ref.split(":", 2)
             except ValueError as exc:
-                raise serializers.ValidationError({"key": str(exc)})
+                raise serializers.ValidationError(
+                    {"key": f"Invalid entity key format: {entity_ref!r}. "
+                            "Expected '{namespace}:{type_name}:{component_code}'."}
+                ) from exc
+            component_type_obj = components_api.get_or_create_component_type(namespace, type_name)
         attrs["component_type"] = component_type_obj
         attrs["component_code"] = component_code
         return attrs
@@ -147,12 +185,13 @@ def validate(self, attrs):
         ``container_code`` field inside [entity.container]. Archives created
         in Ulmo do not, so we fall back to using the entity key.
         """
+        super().validate(attrs)
         container = attrs.pop("container")
         # It is safe to do this after validate_container
         container_type = next(k for k in container if k in ("section", "subsection", "unit"))
         attrs["container_type"] = container_type
-        # Verawood+: container_code is explicit. Ulmo: fall back to entity key.
-        attrs["container_code"] = container.get("container_code") or attrs["key"]
+        # Verawood+: container_code is explicit. Ulmo: fall back to entity_ref.
+        attrs["container_code"] = container.get("container_code") or attrs["entity_ref"]
         return attrs
 
 

src/openedx_content/applets/backup_restore/toml.py

@@ -43,7 +43,9 @@ def toml_learning_package(
     # Learning package main info
     section = tomlkit.table()
     section.add("title", learning_package.title)
-    section.add("key", learning_package.key)
+    # Write package_ref (Verawood+) and key (Ulmo back-compat).
+    section.add("package_ref", learning_package.package_ref)
+    section.add("key", learning_package.package_ref)
     section.add("description", learning_package.description)
     section.add("created", learning_package.created)
     section.add("updated", learning_package.updated)
@@ -89,8 +91,10 @@ def _get_toml_publishable_entity_table(
     """
     entity_table = tomlkit.table()
     entity_table.add("can_stand_alone", entity.can_stand_alone)
-    # Add key since the toml filename doesn't show the real key
-    entity_table.add("key", entity.key)
+    # Write entity_ref (Verawood+) and key (Ulmo back-compat) so that older
+    # restore code can still read archives produced after this rename.
+    entity_table.add("entity_ref", entity.entity_ref)
+    entity_table.add("key", entity.entity_ref)
     entity_table.add("created", entity.created)
 
     if not include_versions:
@@ -204,13 +208,13 @@ def toml_publishable_entity_version(version: PublishableEntityVersion) -> tomlki
     if hasattr(version, 'containerversion'):
         # If the version has a container version, add its children
         container_table = tomlkit.table()
-        children = containers_api.get_container_children_entities_keys(version.containerversion)
+        children = containers_api.get_container_children_entity_refs(version.containerversion)
         container_table.add("children", children)
         version_table.add("container", container_table)
     return version_table
 
 
-def toml_collection(collection: Collection, entity_keys: list[str]) -> str:
+def toml_collection(collection: Collection, entity_refs: list[str]) -> str:
     """
     Create a TOML representation of a collection.
 
@@ -228,7 +232,7 @@ def toml_collection(collection: Collection, entity_keys: list[str]) -> str:
     doc = tomlkit.document()
 
     entities_array = tomlkit.array()
-    entities_array.extend(entity_keys)
+    entities_array.extend(entity_refs)
     entities_array.multiline(True)
 
     collection_table = tomlkit.table()