update documentation for new encryption scheme

Author: phycodurus

docs/customization/encrypted_model_fields.rst

@@ -2,7 +2,8 @@ Encrypted Model Fields
 ======================
 
 If your ``custom_code`` or reusable app contains a model field storing
-user-specific sensitive data, you may want to encrypt that data at rest.
+user-specific sensitive data, TOM Toolkit provides a way to encrypt
+that data in in the database ("at rest").
 
 Examples of user-specific sensitive data include passwords or API keys
 for external services that your TOM uses on the user's behalf. TOM
@@ -16,18 +17,15 @@ How encryption works
 --------------------
 
 Encrypted fields are protected by a single Fernet cipher derived from
-``settings.SECRET_KEY`` using HKDF (RFC 5869) with a domain-separator
-label. There is no per-user key material and no additional environment
-variable to manage. For the operator-side concerns (rotating
+``settings.SECRET_KEY``. (For TOM administrator concerns (rotating
 ``SECRET_KEY`` without losing data, etc.) see
-:doc:`/deployment/encryption`.
+:doc:`/deployment/encryption`).
 
 .. note::
 
    Encryption protects data from passive database exposure. It does NOT
-   protect against a server administrator with access to ``SECRET_KEY``
-   — by design, since the same admin needs to be able to run the
-   ``rotate_encryption_key`` command. If you need user-level isolation
+   protect against a server administrator with access to the
+   ``settings.SECRET_KEY``. If you need user-level isolation
    from administrators, the toolkit's current scheme is not sufficient.
 
 Adding an encrypted field to a model
@@ -45,19 +43,21 @@ encryption on write and decryption on read.
 
 
     class MyAppProfile(models.Model):
-        user = models.OneToOneField(settings.AUTH_USER_MODEL,
-                                    on_delete=models.CASCADE)
+	# you probably have a user OneToOneField here
+
+	# this is the encryption part:
         _api_key_encrypted = models.BinaryField(null=True, blank=True)  # ciphertext (private)
         api_key = EncryptedProperty('_api_key_encrypted')               # descriptor (public)
 
 By convention, the ``BinaryField``'s name starts with an underscore —
-its only consumer is the :class:`EncryptedProperty` descriptor; plugin
-code should never read or write it directly.
+it is only referenced by the :class:`EncryptedProperty` descriptor; never
+read or write the ``BinaryField`` directly.
 
 Reading and writing the field
 -----------------------------
 
-Plain Python attribute access on the descriptor handles everything:
+Access the encrypted field just like a regular Python attribute.
+The ``EncryptedProperty`` descriptor handles everything:
 
 .. code-block:: python
 
@@ -67,23 +67,27 @@ Plain Python attribute access on the descriptor handles everything:
     # later, possibly in a different process / request:
     value = profile.api_key   # 'something-secret'
 
-On assignment, the descriptor calls
+    
+Some explanations
+-----------------
+
+That (above) is really all you need to know. However, here's what's
+going on "under the hood": On assignment, the descriptor calls
 :func:`tom_common.encryption.encrypt`, which builds a Fernet cipher
 from ``settings.SECRET_KEY`` and encrypts the value, then stores the
 ciphertext bytes in the underlying ``BinaryField``. On read, the
 descriptor calls :func:`tom_common.encryption.decrypt`, which
-transparently honours ``settings.SECRET_KEY_FALLBACKS`` (see
-:doc:`/deployment/encryption` for the rotation procedure).
+transparently honours ``settings.SECRET_KEY_FALLBACKS``. However,
+that's really just for admins to worry about (see
+:doc:`/deployment/encryption` for admin/deployment concerns).
 
 An empty string assignment clears the ciphertext (stores ``None`` in
 the ``BinaryField``). Reading an unset / empty field yields ``''``,
 not ``None`` — so consumers don't need to special-case the empty case.
 
-Some explanations
------------------
 
 :class:`EncryptedProperty` (`source <https://github.com/TOMToolkit/tom_base/blob/dev/tom_common/models.py>`__)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 A *property descriptor* implementing the Python descriptor protocol
 (``__get__``, ``__set__``, ``__set_name__``). It handles the details of

docs/deployment/deployment_tips.rst

@@ -1,8 +1,10 @@
 General Deployment Tips
 -----------------------
 
-When it comes to deploying your tom for general use, there are a few
-things you might want to consider.
+Your TOM is a Django project. First and foremost, you should be following
+`Django's deployment checklist <https://docs.djangoproject.com/en/6.0/howto/deployment/checklist/#deployment-checklist>`_.
+Subsequently, when it comes to deploying your TOM for general use, there are a
+few things you might want to consider.
 
 Choosing a database
 ~~~~~~~~~~~~~~~~~~~
@@ -71,4 +73,4 @@ If you provide the path to a file that does not exist, TOM Toolkit will still
 serve the default ``robots.txt`` file and log a warning message to that effect.
 
 Additional background on the ``robots.txt`` file can be found
-`here <https://en.wikipedia.org/wiki/Robots_exclusion_standard>`_.
+`here <https://en.wikipedia.org/wiki/Robots_exclusion_standard>`_.

docs/deployment/encryption.rst

@@ -1,26 +1,31 @@
-Encryption at Rest and the SECRET_KEY
+Encryption and the SECRET_KEY
 =====================================
 
-TOM Toolkit encrypts sensitive user data at rest (API keys, observatory
+This section is for TOM administrators and describes the relationship
+between the ``settings.SECRET_KEY`` and the way TOMToolkit encrypts
+sensitive user data. If you are a TOM developer looking for how to
+create an encrypted database field, your documentation is here:
+:doc:`/customization/encrypted_model_fields`
+
+------
+
+TOM Toolkit encrypts sensitive user data (API keys, observatory
 credentials, anything declared with :class:`EncryptedProperty`) using a
 single Fernet cipher derived from Django's ``settings.SECRET_KEY``.
-There is no additional environment variable to manage — your
-``SECRET_KEY`` is both your signing key (cookies, password-reset
-tokens, etc.) and the source of your encryption key.
-
-The derivation uses HKDF (RFC 5869) with a domain-separator label, so
-the encryption key is cryptographically independent of the way Django
-uses ``SECRET_KEY`` for HMAC signing. See
-:mod:`tom_common.encryption` for the implementation; for the plugin-
-developer-facing API see :doc:`/customization/encrypted_model_fields`.
+That means when an encrypted field is written to or read from the database,
+a cipher is created. The cipher can encrypt unencrypted plaintext (in) and decrypt
+encrypted ciphertext (out). Cipher creation requires an encryption key.
+TOMToolkit creates an encryption key that is based upon, but not identical
+to, Django's ``settings.SECRET_KEY``. That's how the ``SECRET_KEY`` is related
+to TOMToolkit's encryption. 
 
 Treat ``SECRET_KEY`` like an encryption key
 -------------------------------------------
 
 If you lose ``SECRET_KEY`` (and any active
 ``SECRET_KEY_FALLBACKS`` entries), every encrypted field becomes
 unrecoverable. Keep ``SECRET_KEY`` secret, never commit it, and back
-it up through whatever channel your other production secrets travel.
+it up through whatever channel your other production secrets use.
 
 The standard Django guidance applies — see the
 `Django deployment checklist
@@ -35,9 +40,16 @@ and the
    below will leave every previously-encrypted field unreadable. Follow
    the procedure exactly — don't just edit ``SECRET_KEY`` in your env.
 
+   Should 
+
 Graceful ``SECRET_KEY`` rotation
 --------------------------------
 
+Because TOMToolkit's encryption scheme depends on the value of ``settings.SECRET_KEY``,
+if you need to change your ``SECRET_KEY``, we must decrypt the encrypted data
+with a cipher derived from the old ``SECRET_KEY`` and re-encrypt it with a cipher derived
+from the new ``SECRET_KEY``. The following proceed explains the process in full.
+
 We use Django's built-in
 `SECRET_KEY_FALLBACKS <https://docs.djangoproject.com/en/stable/ref/settings/#secret-key-fallbacks>`_
 mechanism to rotate keys without an outage and without data loss. The
@@ -112,6 +124,15 @@ If you lose ``SECRET_KEY`` and have no backup:
 Treat ``SECRET_KEY`` backup with the same seriousness as your database
 backup.
 
+Key Derivation Function Implementation Details
+----------------------------------------------
+The derivation uses `HKDF <https://en.wikipedia.org/wiki/HKDF>`_
+(RFC 5869) with a domain-separator label, so the encryption key is
+cryptographically independent of the way Django
+uses ``SECRET_KEY`` for HMAC signing. See
+:mod:`tom_common.encryption` for the implementation.
+
+
 See also
 --------
 

docs/deployment/index.rst

@@ -19,4 +19,4 @@ accessible by you and your colleagues.
 
 :doc:`Using Amazon S3 to Store Data for a TOM <amazons3>` - Enable storing data on the cloud storage service Amazon S3 instead of your local disk.
 
-:doc:`Configuring the Master Encryption Key <encryption>` - How to set ``TOMTOOLKIT_DEK_ENCRYPTION_KEY`` across deploy environments and migrate off the public default.
+:doc:`Encryption and the SECRET_KEY <encryption>` - How Django's ``settings.SECRET_KEY`` relates to TOM Toolkit encryption.