docs: update sync process documentation

Author: f4z3r

docs/syncing/index.rst

@@ -5,7 +5,7 @@ Syncing
 Concepts
 --------
 
-The **store** holds serialized versions of syncable models. This includes both data that is on the current device and data synced from other devices.
+The **store** holds serialized versions of syncable models. This includes both data that is on the current device and data synced from other devices. The store is represented as a standard Django model, containing syncable models as JSON.
 
 The **outgoing buffer** and **incoming buffer** mirror the schema of the store. They also include a transfer session ID which used to identify sets of data that are being synced as a coherent group to other Morango instances.
 
@@ -15,11 +15,14 @@ Process
 
 Syncing is the actual exchange of data in a sync session. The general steps for syncing data are:
 
-1. **Serialization** - serializing data that is associated with Django models in the Application layer, and storing it in JSON format in a record in the Store
-2. **Queuing/Buffering** - storing serialized records and their modification history to a separate Buffers data structure
-3. **Transfer/chunking of data** - the actual transfer of data over a request/response cycle in chunks of 500 records at a time
-4. **Dequeuing** - merging the data received in the receiving buffers to the receiving store and record-max counter
-5. **Deserialization** - merging data from the receiving Store into the Django models in the Application layer
+1. **Serialization** - serializing data that is associated with Django models in the Application layer, and storing it in JSON format in a record in the Store. The serialized data in the store is versioned via a counter (described in `Counters <../counters#counters>`__).
+2. **Queuing/Buffering** - storing serialized records and their modification history to a separate Buffers data structure. This Django model only contains the changes to be synced with the other Morango instance. This is in contrast to the Store, which contains all data, regardless of what is getting transferred in this sync session.
+3. **Transfer/chunking of data** - the actual transfer of data over a request/response cycle in a set of chunked records. If both sides support it, the chunked records are compressed before being sent of the network. The actual transfer is done over HTTP.
+4. **Dequeuing** - merging the data received in the receiving buffers to the receiving store and record-max counter. During this step, the data from the incoming buffer is merged into the store on the receiving side. Merge conflicts in case of version splits can be solved automatically. As new data is written into the store, the dirty bit on that object is set to indicate that the data needs to be deserialized and pushed to the Application Layer.
+5. **Deserialization** - merging data from the receiving Store into the Django models in the Application layer. Data marked as stale in the Application Layer (where a newer version is available in the Store, on a record with the dirty bit set), the data in the store is deserialized from JSON into a Django model and integrated into the Application Layer.
+
+The individual steps of the syncing process are implemented in `morango/sync/operations.py <https://github.com/learningequality/morango/blob/HEAD/morango/sync/operations.py>`_. They are implemented as operations that are registered for every process step described above. A project using Morango can define their own operations and register them to be executed as part of an arbitrary step in the process via configuration options such as ``MORANGO_INITIALIZE_OPERATIONS``.
+
 
 In the illustration below, the application layer (on the right) is where app data resides as Django models, and the Morango layer (on the left) is where the Morango stores, counters, and buffers reside. *Instance A* (on the top) is sending data to *Instance B* (on the bottom). Application Django models in *Instance A* are serialized in JSON format and saved to the store. Data is queued in the buffers on *Instance A*, and then transmitted to the corresponding buffers on *Instance B*. The data is then integrated into the store and Django app models on *Instance B*.
 
@@ -53,6 +56,8 @@ Signals
 
 During the sync process, Morango fires a few different signals from ``signals`` in ``PullClient`` and ``PushClient``. These can be used to track the progress of the sync.
 
+The operations described in the previous section are triggered via such a signal, which has the operations attached to it. The ``SyncSignal`` definition can be found under `morango/sync/utils.py <https://github.com/learningequality/morango/blob/HEAD/morango/sync/utils.py>`_.
+
 There are four signal groups:
 
 - ``session``
@@ -66,6 +71,8 @@ Each signal group has 3 stages that can be fired:
 - ``in_progress``
 - ``completed``
 
+The ``SessionController`` is responsible to register the configured operations to the corresponding signal, and triggers the individual steps when its ``proceed_to`` function is called.
+
 For a push or pull sync lifecycle, the order of the fired signals would be as follows:
 
 1) Session started