1459 login problems intermittent

docs/customization/encrypted_model_fields.rst

@@ -1,107 +1,230 @@
 Encrypted Model Fields
 ======================
 
-If your ``custom_code`` or reusable app contains a Model field storing user-specific
-sensitive data, then you may want to encrypt that data.
+If your ``custom_code`` or reusable app needs to store sensitive data,
+TOM Toolkit provides a way to encrypt that data in the database.
 
-Examples of user-specific sensitive
-data include a password or API key for an external service that your TOM uses.
-For example, TOMToolkit Facility modules can use the mechanism described here to store,
-encrypted, user-specific credentials in a user profile model. Examples include the
-`tom_eso <https://github.com/TOMToolkit/tom_eso>`__ and the
-`tom_swift <https://github.com/TOMToolkit/tom_swift>`__ facility modules.
+Examples of this type of sensitive data include passwords or API keys
+for external services that your TOM stores on the user's behalf. TOM
+Toolkit's Facility modules, use the mechanism described
+here to store user-specific external-service credentials in a user
+profile model. Examples live in
+`tom_demoapp <https://github.com/TOMToolkit/tom_demoapp>`__ ,
+`tom_eso <https://github.com/TOMToolkit/tom_eso>`__ and
+`tom_swift <https://github.com/TOMToolkit/tom_swift>`__.
 
-As we explain below, TOMToolkit provides a *mix-in* class, a *property descriptor*, and
-utility functions to help encrypt user-specific sensitive data and access it when it's needed.
+Getting Started
+---------------
 
-.. note:: For sensitive data that is used by the TOM itself and is not user-specific,
-    we suggest that this data be stored outside the TOM and accessed through
-    environment variables.
+Here is a brief overview of the steps you'll need to create, display, and update and encrypted field.
+Each step is covered in its own section below.
 
-Creating and accessing an encrypted Model field
------------------------------------------------
+1. Add an ``EncryptedModelField`` to your model.
+2. List the field in your ``UpdateView``'s ``fields``.
+3. Pass the plaintext (from the model) to your profile-card template and
+   include the ``revealable_password_input.html`` partial.
 
-Creating an encrypted Model field
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If your Model has a field that should be encrypted, follow these steps:
+Adding an encrypted field to a model
+++++++++++++++++++++++++++++++++++++
 
-1. Import the mix-in class and property descriptor in your ``models.py``:
+Declare an :class:`~tom_common.encryption.EncryptedModelField` alongside
+your other model fields:
 
 .. code-block:: python
+    :caption: models.py
 
-    from tom_common.models import EncryptableModelMixin, EncryptedProperty
+    from django.db import models
+    from tom_common.encryption import EncryptedModelField
 
-2. Make your Model subclass a subclass of ``EcryptableModelMixin``. For example:
+
+    class MyAppProfile(models.Model):
+        # you probably have a user OneToOneField here as well
+        api_key = EncryptedModelField(null=True, blank=True)
+
+The underlying database column is a ``BinaryField`` (it holds
+the encrypted ciphertext as bytes). Django sees ``api_key`` as a normal
+named field. ``ModelForm``, the Django admin, DRF ``ModelSerializer``, etc
+all introspect it appropriately.
+
+Reading and writing the value in code
++++++++++++++++++++++++++++++++++++++
+
+Access the field like any other ``models.Field`` subclass:
 
 .. code-block:: python
 
-    class MyAppModel(EncryptableModelMixin, models.Model):
-        ...
+    # assignment
+    profile.api_key = 'something-secret'
+    profile.save()
+
+    # retrieval; later, possibly in a different process / request:
+    plaintext_value = profile.api_key   # 'something-secret'
+
+
+The field handles encryption on save and decryption on load.
+Assigning ``None`` or the empty string ``''`` clears the stored value
+(column stored as ``NULL``). Reading an unset value yields ``None``.
+For implementation details, see ``tom_common/encryption.py``. 
+
+Displaying the current value to your TOM users (read-only)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+To display an encrypted field value with a user-driven "reveal
+control" (i.e. the  "click the eye icon to see the value" pattern),
+use ``tom_common``'s ``revealable_password_input.html`` partial template.
 
-This gives your model access to a set of methods that will manage the encryption and
-decryption of your data into and out of the ``BinaryField`` that stores the encrypted data.
+The partial needs the **plaintext** as its ``value`` argument, and
+plaintext is available via direct attribute access on the model
+instance — e.g. ``profile.api_key``. Django introspection paths
+(``model_to_dict``, ``ModelSerializer``, ``dumpdata``, admin display)
+all go through ``EncryptedModelField.value_from_object``, which
+returns the ``REDACTED`` placeholder by design.
 
-3. Add the ``BinaryField`` that will store the encrypted data and the property descriptor
-through which the ``BinaryField`` will be accessed.
+In practice, when a profile card uses an inclusion tag (or a view's
+``get_context_data``) to provide fields to a template, the encrypted
+field has to be excluded from any auto-iteration over the model and
+passed in explicitly:
 
 .. code-block:: python
+    :caption: [function returning context to template]
 
-    _ciphertext_api_key = BinaryField(null=True, blank=True)  # encrypted data field (private)
-    api_key = EncryptedProperty('_ciphertext_api_key')  # descriptor that provides access (public)
+    ...
+    # exclude the encrypted field from the auto-iteration: model_to_dict
+    # would only return the REDACTED placeholder for it
+    excluded_fields = ['user', 'id', 'api_key']
+    profile_data = model_to_dict(profile, exclude=excluded_fields)
 
-By convention name of the ``BinaryField`` field should begin with and underscore
-(``_ciphertext_api_key`` in our example) because is it private to the Model class.
+    context = {
+        'profile_data': profile_data,  # dictionary without the excluded_fields
+        'api_key': profile.api_key,   # direct attribute access -> plaintext
+    }
+    return context
+
+Then in the template, render the encrypted field through the partial:
+
+.. code-block:: html+django
+    :caption: my_template.html
+
+    {% if api_key %}
+        {% include 'tom_common/partials/revealable_password_input.html' with value=api_key %}
+    {% else %}
+        (not set)
+    {% endif %}
+
+The partial renders a masked input of fixed length; the real value is
+only injected into the DOM when the user clicks the reveal icon.
+
+A worked example of this pattern lives in
+`tom_demoapp <https://github.com/TOMToolkit/tom_demoapp>`__'s
+``demo_extras.py`` and ``profile_demo.html``.
+
+Editing the value in an UpdateView
+++++++++++++++++++++++++++++++++++
+
+Include the ``EncryptedModelField`` in the ``fields`` list of your
+``ProfileUpdateView`` (or any other ``ModelForm``-based view) and the
+form renders the field automatically as a composite control with a masked
+password input joined to a "Clear" checkbox in a single input-group.
+
+.. code-block:: python
 
-Accessing encrypted data
-~~~~~~~~~~~~~~~~~~~~~~~~
-The following example shows how to get and set an encrypted field using the utility
-methods provided in ``tom_common.session_utils.py``:
+    class MyProfileUpdateView(UpdateView):
+        model = MyAppProfile
+        fields = ['api_key', 'display_name']
+
+How the control behaves on submit:
+
+- A typed value replaces the stored value.
+- An empty input leaves the stored value unchanged. This protects
+  the secret when a user edits an unrelated field on the same form
+  and saves without retyping.
+- Checking **Clear** with an empty input clears the stored value
+  (column becomes ``NULL``).
+- If a typed value and the **Clear** checkbox are submitted together,
+  the typed value wins and the checkbox is ignored — the more
+  conservative choice, since "don't lose what the user just typed" is
+  safer than the alternative.
+
+The stored value is never revealed by this
+toggle (or by anything else in the form), because the stored value
+is never rendered into the form's HTML to begin with.
+
+To bridge the resulting information gap (the form input is masked
+whether or not a value is stored), the input's placeholder is
+state-aware: it reads ``(A stored value is hidden) — type to replace``
+when a value is stored, and ``(not set) — type to add`` when nothing
+is stored. A hover tooltip on the control reinforces the same idea.
+The actual stored value is not communicated through either channel;
+to view it, follow the read-only display pattern above.
+
+How encryption works
+--------------------
+
+Encrypted fields are protected by a single Fernet cipher derived from
+``settings.SECRET_KEY``. For TOM administrator concerns (rotating
+``SECRET_KEY`` without losing data, etc.) see
+:doc:`/deployment/encryption`.
+
+.. note::
+
+   Encryption protects data from passive database exposure. It does NOT
+   protect against a server administrator with access to the
+   ``settings.SECRET_KEY``. If you need user-level isolation from
+   administrators, the toolkit's current scheme is not sufficient.
+
+What you cannot do with an EncryptedModelField
+----------------------------------------------
+
+**Filter on the value.** Fernet is non-deterministic — every
+``encrypt()`` produces a different ciphertext for the same plaintext,
+so database-level equality lookups can never match.
+:meth:`EncryptedModelField.get_lookup` raises ``FieldError`` rather
+than silently returning empty querysets:
 
 .. code-block:: python
 
-    from tom_common.session_utils import get_encrypted_field, set_encrypted_field
-    from tom_app_example.models import MyAppModel
-
-    profile: MyAppModel = user.myappmodel  # Model instance containing an encrypted field
-
-    # getter example
-    decrypted_api_key: str = get_encrypted_field(user, profile, 'api_key')
-
-    # setter example
-    new_api_key: str = 'something_secret'
-    set_encrypted_field(user, profile, 'api_key', new_api_key)
-
-Note here that the User instance (``user``) is used to access the ``EncryptableModelMixin``
-subclass and its encrypted data. The ``user`` property of the Model subclass containing the
-encrypted field (``MyAppModel`` in our example) is provided by the ``EncryptableModelMixin``.
-As such, the model *should not define a* ``user`` *property of its own*.
-
-Some Explanations
------------------
-
-EncryptableModelMixin (`source <https://github.com/TOMToolkit/tom_base/blob/069024f954e5540c1441c5186378de538f7d606f/tom_common/models.py#L100>`__)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The User's data is encrypted using (among other things) their password (i.e the
-password they use to login to your TOM). When the User changes their password,
-their encrypted data re-encrypted accordingly. The ``EncryptableModelMixin`` adds
-method for this to your otherwise normal Django model.
-
-EncryptedProperty (`source <https://github.com/TOMToolkit/tom_base/blob/069024f954e5540c1441c5186378de538f7d606f/tom_common/models.py#L39>`__)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-A *property descriptor* implements the Python descriptor protocol (``__get__``,
-``__set__``, etc). The ``EncryptedProperty`` property descriptor handles the details
-of decrypting the encrypted ``BinaryField`` on its way out of the database and
-encrypting it on the way in. It is invoked when the property is accessed
-(e.g. ``model_instance.api_key``).
-
-Session Utils (`example <https://github.com/TOMToolkit/tom_eso/blob/b74fe3b951ead6f6f332594724731d036944da47/tom_eso/eso.py#L209>`__)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The ``get_encrypted_field`` and ``set_encrypted_field`` functions implement
-boilerplate code for creating and destroying the cipher used to encrypt and
-decrypt the ``BinaryField``. *These methods must always be used to access any
-encrypted field*.
-
-
-The rest of the details are in the source code. If reading source code isn't your thing,
-please do feel free to get in touch and we'll be happy to answer any questions you may have.
+    MyAppProfile.objects.filter(api_key='foo')   # raises FieldError
+
+If you need equality search on the encrypted column, store a
+companion HMAC-derived hash column alongside it and query the hash.
+
+**Round-trip via dumpdata / loaddata.** Serialization paths emit a
+placeholder (``EncryptedModelField.REDACTED``, currently
+``'******** (encrypted, not shown)'``) rather than the plaintext.
+This keeps secrets out of fixture files, DRF API responses, and
+admin history by default. As a consequence, attempting to load a
+``dumpdata`` fixture back fails loudly when
+:meth:`~EncryptedModelField.to_python` encounters the placeholder.
+To migrate encrypted data between environments, copy the database
+row directly (the ciphertext survives) or write a one-off
+decrypt-and-re-encrypt script.
+
+Direct attribute access (``getattr(instance, field_name)``) bypasses
+the redaction and remains the only path to the plaintext — code that
+legitimately needs the secret reads the attribute directly.
+
+What happens on decryption failure
+----------------------------------
+
+If a stored ciphertext cannot be decrypted under any active key
+(``settings.SECRET_KEY`` plus any ``settings.SECRET_KEY_FALLBACKS``),
+:meth:`EncryptedModelField.from_db_value` raises
+``cryptography.fernet.InvalidToken`` at row-load time. The most
+common cause is a key removed from the rotation set before its data
+was re-encrypted under a new primary. See
+:doc:`/deployment/encryption` for the rotation procedure and the
+``rotate_encryption_key`` management command.
+
+API reference
+-------------
+
+:class:`~tom_common.encryption.EncryptedModelField` (`source <https://github.com/TOMToolkit/tom_base/blob/dev/tom_common/encryption.py>`__)
+    A ``models.BinaryField`` subclass that transparently encrypts on
+    save and decrypts on load. See the class docstring in
+    ``tom_common/encryption.py`` for the full method-level contract.
+
+:class:`~tom_common.encryption.EncryptedFormField` (`source <https://github.com/TOMToolkit/tom_base/blob/dev/tom_common/encryption.py>`__)
+    The form-side companion. Handles the masked-input UX and the
+    blank-submission-preserves-existing behavior. ``ModelForm`` picks
+    it up automatically via :meth:`EncryptedModelField.formfield`.

docs/deployment/deployment_tips.rst

@@ -1,8 +1,10 @@
 General Deployment Tips
 -----------------------
 
-When it comes to deploying your tom for general use, there are a few
-things you might want to consider.
+Your TOM is a Django project. First and foremost, you should be following
+`Django's deployment checklist <https://docs.djangoproject.com/en/stable/howto/deployment/checklist/#deployment-checklist>`__.
+Subsequently, when it comes to deploying your TOM for general use, there are a
+few things you might want to consider.
 
 Choosing a database
 ~~~~~~~~~~~~~~~~~~~
@@ -71,4 +73,4 @@ If you provide the path to a file that does not exist, TOM Toolkit will still
 serve the default ``robots.txt`` file and log a warning message to that effect.
 
 Additional background on the ``robots.txt`` file can be found
-`here <https://en.wikipedia.org/wiki/Robots_exclusion_standard>`_.
+`here <https://en.wikipedia.org/wiki/Robots_exclusion_standard>`_.