Skip to content

Commit 3e85ba4

Browse files
phycodurusphycodurus
committed
removes User specificity of data encryption key (DEK)
There is now one single DEK for all the encrypted fields and it is derived from the Django `settings.SECRET_KEY`. All changes in this commit simplify the encryption scheme implementation accordingly.
1 parent 2386711 commit 3e85ba4

16 files changed

Lines changed: 690 additions & 947 deletions

File tree

‎docs/customization/encrypted_model_fields.rst‎

Lines changed: 81 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -1,108 +1,102 @@
11
Encrypted Model Fields
22
======================
33

4-
If your ``custom_code`` or reusable app contains a Model field storing user-specific
5-
sensitive data, then you may want to encrypt that data.
4+
If your ``custom_code`` or reusable app contains a model field storing
5+
user-specific sensitive data, you may want to encrypt that data at rest.
6+
7+
Examples of user-specific sensitive data include passwords or API keys
8+
for external services that your TOM uses on the user's behalf. TOM
9+
Toolkit's Facility modules, for example, use the mechanism described
10+
here to store user-specific external-service credentials in a user
11+
profile model. Real examples live in
12+
`tom_eso <https://github.com/TOMToolkit/tom_eso>`__ and
13+
`tom_swift <https://github.com/TOMToolkit/tom_swift>`__.
14+
15+
How encryption works
16+
--------------------
17+
18+
Encrypted fields are protected by a single Fernet cipher derived from
19+
``settings.SECRET_KEY`` using HKDF (RFC 5869) with a domain-separator
20+
label. There is no per-user key material and no additional environment
21+
variable to manage. For the operator-side concerns (rotating
22+
``SECRET_KEY`` without losing data, etc.) see
23+
:doc:`/deployment/encryption`.
24+
25+
.. note::
26+
27+
Encryption protects data from passive database exposure. It does NOT
28+
protect against a server administrator with access to ``SECRET_KEY``
29+
— by design, since the same admin needs to be able to run the
30+
``rotate_encryption_key`` command. If you need user-level isolation
31+
from administrators, the toolkit's current scheme is not sufficient.
32+
33+
Adding an encrypted field to a model
34+
------------------------------------
35+
36+
Two pieces are needed on the model: a ``BinaryField`` for the
37+
ciphertext, and an :class:`EncryptedProperty` descriptor that handles
38+
encryption on write and decryption on read.
639

7-
Examples of user-specific sensitive
8-
data include a password or API key for an external service that your TOM uses.
9-
For example, TOMToolkit Facility modules can use the mechanism described here to store,
10-
encrypted, user-specific credentials in a user profile model. Examples include the
11-
`tom_eso <https://github.com/TOMToolkit/tom_eso>`__ and the
12-
`tom_swift <https://github.com/TOMToolkit/tom_swift>`__ facility modules.
13-
14-
As we explain below, TOMToolkit provides a *mix-in* class, a *property descriptor*, and
15-
utility functions to help encrypt user-specific sensitive data and access it when it's needed.
16-
17-
.. note:: For sensitive data that is used by the TOM itself and is not user-specific,
18-
we suggest that this data be stored outside the TOM and accessed through
19-
environment variables.
20-
21-
Creating and accessing an encrypted Model field
22-
-----------------------------------------------
40+
.. code-block:: python
2341
24-
Creating an encrypted Model field
25-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
42+
from django.conf import settings
43+
from django.db import models
44+
from tom_common.models import EncryptedProperty
2645
27-
If your Model has a field that should be encrypted, follow these steps:
2846
29-
1. Import the mix-in class and property descriptor in your ``models.py``:
47+
class MyAppProfile(models.Model):
48+
user = models.OneToOneField(settings.AUTH_USER_MODEL,
49+
on_delete=models.CASCADE)
50+
_api_key_encrypted = models.BinaryField(null=True, blank=True) # ciphertext (private)
51+
api_key = EncryptedProperty('_api_key_encrypted') # descriptor (public)
3052
31-
.. code-block:: python
53+
By convention, the ``BinaryField``'s name starts with an underscore —
54+
its only consumer is the :class:`EncryptedProperty` descriptor; plugin
55+
code should never read or write it directly.
3256

33-
from tom_common.models import EncryptableModelMixin, EncryptedProperty
57+
Reading and writing the field
58+
-----------------------------
3459

35-
2. Make your Model subclass a subclass of ``EcryptableModelMixin``. For example:
60+
Plain Python attribute access on the descriptor handles everything:
3661

3762
.. code-block:: python
3863
39-
class MyAppModel(EncryptableModelMixin, models.Model):
40-
...
64+
profile.api_key = 'something-secret'
65+
profile.save()
4166
42-
This gives your model access to a set of methods that will manage the encryption and
43-
decryption of your data into and out of the ``BinaryField`` that stores the encrypted data.
67+
# later, possibly in a different process / request:
68+
value = profile.api_key # 'something-secret'
4469
45-
3. Add the ``BinaryField`` that will store the encrypted data and the property descriptor
46-
through which the ``BinaryField`` will be accessed.
70+
On assignment, the descriptor calls
71+
:func:`tom_common.encryption.encrypt`, which builds a Fernet cipher
72+
from ``settings.SECRET_KEY`` and encrypts the value, then stores the
73+
ciphertext bytes in the underlying ``BinaryField``. On read, the
74+
descriptor calls :func:`tom_common.encryption.decrypt`, which
75+
transparently honours ``settings.SECRET_KEY_FALLBACKS`` (see
76+
:doc:`/deployment/encryption` for the rotation procedure).
4777

48-
.. code-block:: python
78+
An empty string assignment clears the ciphertext (stores ``None`` in
79+
the ``BinaryField``). Reading an unset / empty field yields ``''``,
80+
not ``None`` — so consumers don't need to special-case the empty case.
4981

50-
_ciphertext_api_key = BinaryField(null=True, blank=True) # encrypted data field (private)
51-
api_key = EncryptedProperty('_ciphertext_api_key') # descriptor that provides access (public)
82+
Some explanations
83+
-----------------
5284

53-
By convention name of the ``BinaryField`` field should begin with and underscore
54-
(``_ciphertext_api_key`` in our example) because is it private to the Model class.
85+
:class:`EncryptedProperty` (`source <https://github.com/TOMToolkit/tom_base/blob/dev/tom_common/models.py>`__)
86+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5587

56-
Accessing encrypted data
57-
~~~~~~~~~~~~~~~~~~~~~~~~
58-
The following example shows how to get and set an encrypted field using the utility
59-
methods provided in ``tom_common.session_utils.py``:
88+
A *property descriptor* implementing the Python descriptor protocol
89+
(``__get__``, ``__set__``, ``__set_name__``). It handles the details of
90+
decrypting the ciphertext ``BinaryField`` on its way out of the
91+
database and encrypting it on the way in. It is invoked whenever the
92+
property is accessed (e.g. ``profile.api_key`` reads;
93+
``profile.api_key = 'x'`` writes).
6094

61-
.. code-block:: python
62-
63-
from tom_common.session_utils import get_encrypted_field, set_encrypted_field
64-
from tom_app_example.models import MyAppModel
65-
66-
profile: MyAppModel = user.myappmodel # Model instance containing an encrypted field
67-
68-
# getter example
69-
decrypted_api_key: str = get_encrypted_field(user, profile, 'api_key')
70-
71-
# setter example
72-
new_api_key: str = 'something_secret'
73-
set_encrypted_field(user, profile, 'api_key', new_api_key)
74-
75-
Note here that the User instance (``user``) is used to access the ``EncryptableModelMixin``
76-
subclass and its encrypted data. The ``user`` property of the Model subclass containing the
77-
encrypted field (``MyAppModel`` in our example) is provided by the ``EncryptableModelMixin``.
78-
As such, the model *should not define a* ``user`` *property of its own*.
79-
80-
Some Explanations
81-
-----------------
95+
The descriptor reads the cipher from
96+
:func:`tom_common.encryption._get_cipher` on every access, so changes
97+
to ``settings.SECRET_KEY`` (via ``override_settings`` in tests, or a
98+
deployment-level rotation) take effect immediately on the next read.
8299

83-
EncryptableModelMixin
84-
~~~~~~~~~~~~~~~~~~~~~
85-
An abstract Django model mixin that provides a standardized ``user`` OneToOneField.
86-
Any model that stores encrypted data via ``EncryptedProperty`` should inherit from
87-
this mixin. The ``user`` field ties encrypted data to its owner, allowing the
88-
helper functions in ``session_utils`` to look up the user's Data Encryption Key
89-
(DEK) and build the cipher needed for encryption and decryption.
90-
91-
EncryptedProperty (`source <https://github.com/TOMToolkit/tom_base/blob/069024f954e5540c1441c5186378de538f7d606f/tom_common/models.py#L39>`__)
92-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
93-
A *property descriptor* implements the Python descriptor protocol (``__get__``,
94-
``__set__``, etc). The ``EncryptedProperty`` property descriptor handles the details
95-
of decrypting the encrypted ``BinaryField`` on its way out of the database and
96-
encrypting it on the way in. It is invoked when the property is accessed
97-
(e.g. ``model_instance.api_key``).
98-
99-
Session Utils (`example <https://github.com/TOMToolkit/tom_eso/blob/b74fe3b951ead6f6f332594724731d036944da47/tom_eso/eso.py#L209>`__)
100-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101-
The ``get_encrypted_field`` and ``set_encrypted_field`` functions implement
102-
boilerplate code for creating and destroying the cipher used to encrypt and
103-
decrypt the ``BinaryField``. *These methods must always be used to access any
104-
encrypted field*.
105-
106-
107-
The rest of the details are in the source code. If reading source code isn't your thing,
108-
please do feel free to get in touch and we'll be happy to answer any questions you may have.
100+
The rest of the details are in the source. If reading source isn't
101+
your thing, feel free to get in touch and we'll be happy to answer
102+
questions.

‎docs/deployment/encryption.rst‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
Encryption at Rest and the SECRET_KEY
2+
=====================================
3+
4+
TOM Toolkit encrypts sensitive user data at rest (API keys, observatory
5+
credentials, anything declared with :class:`EncryptedProperty`) using a
6+
single Fernet cipher derived from Django's ``settings.SECRET_KEY``.
7+
There is no additional environment variable to manage — your
8+
``SECRET_KEY`` is both your signing key (cookies, password-reset
9+
tokens, etc.) and the source of your encryption key.
10+
11+
The derivation uses HKDF (RFC 5869) with a domain-separator label, so
12+
the encryption key is cryptographically independent of the way Django
13+
uses ``SECRET_KEY`` for HMAC signing. See
14+
:mod:`tom_common.encryption` for the implementation; for the plugin-
15+
developer-facing API see :doc:`/customization/encrypted_model_fields`.
16+
17+
Treat ``SECRET_KEY`` like an encryption key
18+
-------------------------------------------
19+
20+
If you lose ``SECRET_KEY`` (and any active
21+
``SECRET_KEY_FALLBACKS`` entries), every encrypted field becomes
22+
unrecoverable. Keep ``SECRET_KEY`` secret, never commit it, and back
23+
it up through whatever channel your other production secrets travel.
24+
25+
The standard Django guidance applies — see the
26+
`Django deployment checklist
27+
<https://docs.djangoproject.com/en/stable/howto/deployment/checklist/>`_
28+
and the
29+
`SECRET_KEY documentation
30+
<https://docs.djangoproject.com/en/stable/ref/settings/#secret-key>`_.
31+
32+
.. warning::
33+
34+
Rotating ``SECRET_KEY`` without first running the rotation procedure
35+
below will leave every previously-encrypted field unreadable. Follow
36+
the procedure exactly — don't just edit ``SECRET_KEY`` in your env.
37+
38+
Graceful ``SECRET_KEY`` rotation
39+
--------------------------------
40+
41+
We use Django's built-in
42+
`SECRET_KEY_FALLBACKS <https://docs.djangoproject.com/en/stable/ref/settings/#secret-key-fallbacks>`_
43+
mechanism to rotate keys without an outage and without data loss. The
44+
encryption module's ``decrypt()`` tries the primary derived key first
45+
and then a derived key for each ``SECRET_KEY_FALLBACKS`` entry; the
46+
``encrypt()`` path always uses the primary. So once a new
47+
``SECRET_KEY`` is in place with the old key in fallbacks, *reads of
48+
existing encrypted data continue to work*, and *new writes use the new
49+
key*.
50+
51+
Scaffolded TOMs already include ``SECRET_KEY_FALLBACKS = []`` in their
52+
``settings.py`` (added by ``tom_setup``); if your TOM predates that
53+
template change, just add the line yourself.
54+
55+
The end-to-end procedure:
56+
57+
1. **Stage the old key as a fallback** and install a new
58+
``SECRET_KEY``. In ``settings.py``, move the existing
59+
``SECRET_KEY`` value into ``SECRET_KEY_FALLBACKS`` and set
60+
``SECRET_KEY`` to the new value::
61+
62+
SECRET_KEY = '<new key>'
63+
SECRET_KEY_FALLBACKS = ['<previously-current key>']
64+
65+
(Or via env vars, whatever pattern your deployment uses.)
66+
67+
2. **Restart the server.** All existing encrypted data is still
68+
readable (via the fallback). All new writes — including any
69+
re-encryption — use the new primary key. Django's HMAC signing
70+
machinery also honours the fallback, so existing signed cookies
71+
and password-reset tokens stay valid.
72+
73+
3. **Re-encrypt existing data forward** so it no longer depends on the
74+
fallback::
75+
76+
python manage.py rotate_encryption_key
77+
78+
The command walks every :class:`EncryptedProperty` field across
79+
``INSTALLED_APPS``, decrypts each value (transparently using either
80+
the primary or a fallback), and re-encrypts under the primary. After
81+
this, no value in the database requires the fallback to decrypt.
82+
83+
4. **Remove the fallback** from ``settings.py``::
84+
85+
SECRET_KEY = '<new key>'
86+
SECRET_KEY_FALLBACKS = []
87+
88+
Restart. You're now fully on the new key.
89+
90+
If anything goes wrong between steps 1 and 3, simply leave the fallback
91+
in place — the system stays functional indefinitely with both keys
92+
active. ``rotate_encryption_key`` is idempotent: running it again is
93+
always safe.
94+
95+
If the command reports per-row failures, those rows were encrypted under
96+
a key that is no longer in ``SECRET_KEY`` ∪ ``SECRET_KEY_FALLBACKS``
97+
(i.e., that key has been forgotten). Add it back if you can; otherwise
98+
the data on those rows is lost.
99+
100+
What if ``SECRET_KEY`` is lost?
101+
-------------------------------
102+
103+
If you lose ``SECRET_KEY`` and have no backup:
104+
105+
- Every :class:`EncryptedProperty` value (saved API keys, observatory
106+
credentials) becomes unrecoverable. The ciphertext is still in the
107+
database; the key needed to decrypt it is gone.
108+
- All Django signing-dependent state also breaks: outstanding password-
109+
reset tokens, signed URLs, persistent session cookies. Users will
110+
need to log in again and possibly re-enter any saved secrets.
111+
112+
Treat ``SECRET_KEY`` backup with the same seriousness as your database
113+
backup.
114+
115+
See also
116+
--------
117+
118+
- :doc:`/customization/encrypted_model_fields` — how plugin authors
119+
declare and use encrypted fields.
120+
- `Django deployment checklist
121+
<https://docs.djangoproject.com/en/stable/howto/deployment/checklist/>`_
122+
— the broader hardening you should do before going live.

‎docs/deployment/index.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Deploying your TOM Online
88
deployment_tips
99
deployment_heroku
1010
amazons3
11+
encryption
1112

1213
Once you’ve got a TOM up and running on your machine, you’ll probably want to deploy it somewhere so it is permanently
1314
accessible by you and your colleagues.
@@ -16,4 +17,6 @@ accessible by you and your colleagues.
1617

1718
:doc:`Deploy to Heroku <deployment_heroku>` - Heroku is a PaaS that allows you to publicly deploy your web applications without the need for managing the infrastructure yourself.
1819

19-
:doc:`Using Amazon S3 to Store Data for a TOM <amazons3>` - Enable storing data on the cloud storage service Amazon S3 instead of your local disk.
20+
:doc:`Using Amazon S3 to Store Data for a TOM <amazons3>` - Enable storing data on the cloud storage service Amazon S3 instead of your local disk.
21+
22+
:doc:`Configuring the Master Encryption Key <encryption>` - How to set ``TOMTOOLKIT_DEK_ENCRYPTION_KEY`` across deploy environments and migrate off the public default.

‎tom_base/settings.py‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,13 @@
2828
# SECURITY WARNING: keep the secret key used in production secret!
2929
SECRET_KEY = os.getenv('SECRET_KEY', 'testkey')
3030

31+
# Old SECRET_KEY values to keep accepting during a graceful rotation. See
32+
# docs/deployment/encryption.rst.
33+
SECRET_KEY_FALLBACKS = []
34+
3135
# SECURITY WARNING: don't run with debug turned on in production!
3236
DEBUG = True
3337

34-
# Encryption key for protecting sensitive user data (API keys, credentials) at rest.
35-
# This is a Fernet key — a 44-character URL-safe base64 string encoding 32 random bytes.
36-
# Treat this like SECRET_KEY. See the TOM Toolkit encryption documentation.
37-
TOMTOOLKIT_DEK_ENCRYPTION_KEY = os.getenv(
38-
'TOMTOOLKIT_DEK_ENCRYPTION_KEY',
39-
'UlUYyKsGzQVwjpTbvhtgCihKaj07H1voc-V4pmb7NN4=') # 44-char URL-safe base64 string
40-
4138
ALLOWED_HOSTS = ['']
4239

4340
# Application definition

0 commit comments

Comments
 (0)