feat(specification): add Content Format V3 and Payload Format V3 with bitmap field presence

Author: PromiseFru

docs/specification.md

@@ -6,10 +6,12 @@
   - [Content Format V0](#content-format-v0)
   - [Content Format V1](#content-format-v1)
   - [Content Format V2](#content-format-v2)
+  - [Content Format V3](#content-format-v3)
 - [Payload Format](#supported-payload-versions)
   - [Payload Format V0](#payload-format-v0)
   - [Payload Format V1](#payload-format-v1)
   - [Payload Format V2](#payload-format-v2)
+  - [Payload Format V3](#payload-format-v3)
 
 ## Content Format
 
@@ -182,13 +184,87 @@
    - **2 byte**: Length of `refresh_token` field (set to `0`).
    - **Variable**: Value of `from` field (considered the test ID).
 
+### Content Format V3
+
+> [!NOTE]
+>
+> For detailed instructions on encrypting the content format using the Double Ratchet algorithm, refer to the [smswithoutborders_lib_sig documentation](https://github.com/smswithoutborders/lib_signal_double_ratchet_python?tab=readme-ov-file#double-ratchet-implementations).
+
+> [!NOTE]
+>
+> This format uses a bitmap to indicate which fields are present, making it more efficient than previous versions.
+>
+> **All 2-byte length fields are encoded as unsigned little-endian.**
+
+#### Field Bitmap Definition
+
+The bitmap is a single byte where each bit represents the presence of a specific field:
+
+- **Bit 0 (0x01)**: `from` field
+- **Bit 1 (0x02)**: `to` field
+- **Bit 2 (0x04)**: `cc` field
+- **Bit 3 (0x08)**: `bcc` field
+- **Bit 4 (0x10)**: `subject` field
+- **Bit 5 (0x20)**: `body` field
+- **Bit 6 (0x40)**: `access_token` field
+- **Bit 7 (0x80)**: `refresh_token` field
+
+1. **Email format**: Binary-encoded fields with the following structure:
+
+   - **1 byte**: Field presence bitmap
+   - **1 byte**: Length of `from` field (if bit 0 is set)
+   - **2 bytes**: Length of `to` field (if bit 1 is set)
+   - **2 bytes**: Length of `cc` field (if bit 2 is set)
+   - **2 bytes**: Length of `bcc` field (if bit 3 is set)
+   - **1 byte**: Length of `subject` field (if bit 4 is set)
+   - **2 bytes**: Length of `body` field (if bit 5 is set)
+   - **2 bytes**: Length of `access_token` field (if bit 6 is set)
+   - **2 bytes**: Length of `refresh_token` field (if bit 7 is set)
+   - **Variable**: Value of `from` field (if present)
+   - **Variable**: Value of `to` field (if present)
+   - **Variable**: Value of `cc` field (if present)
+   - **Variable**: Value of `bcc` field (if present)
+   - **Variable**: Value of `subject` field (if present)
+   - **Variable**: Value of `body` field (if present)
+   - **Variable**: Value of `access_token` field (if present)
+   - **Variable**: Value of `refresh_token` field (if present)
+
+2. **Text format**: Binary-encoded fields with the following structure:
+
+   - **1 byte**: Field presence bitmap
+   - **1 byte**: Length of `from` field (if bit 0 is set)
+   - **2 bytes**: Length of `body` field (if bit 5 is set)
+   - **2 bytes**: Length of `access_token` field (if bit 6 is set)
+   - **2 bytes**: Length of `refresh_token` field (if bit 7 is set)
+   - **Variable**: Value of `from` field (if present)
+   - **Variable**: Value of `body` field (if present)
+   - **Variable**: Value of `access_token` field (if present)
+   - **Variable**: Value of `refresh_token` field (if present)
+
+3. **Message format**: Binary-encoded fields with the following structure:
+
+   - **1 byte**: Field presence bitmap
+   - **1 byte**: Length of `from` field (if bit 0 is set)
+   - **2 bytes**: Length of `to` field (if bit 1 is set)
+   - **2 bytes**: Length of `body` field (if bit 5 is set)
+   - **Variable**: Value of `from` field (if present)
+   - **Variable**: Value of `to` field (if present)
+   - **Variable**: Value of `body` field (if present)
+
+4. **Test format**: Binary-encoded fields with the following structure:
+
+   - **1 byte**: Field presence bitmap
+   - **1 byte**: Length of `from` field (if bit 0 is set)
+   - **Variable**: Value of `from` field (considered the test ID, if present)
+
 ## Supported Payload Versions
 
-| **Version**              | **Hexadecimal Value** | **Decimal Value** | **Description**                                             |
-| ------------------------ | --------------------- | ----------------- | ----------------------------------------------------------- |
-| [v0](#payload-format-v0) | `None`                | `None`            | No explicit version marker, backward-compatible formats.    |
-| [v1](#payload-format-v1) | `0x01`                | `1`               | Includes a version marker as the first byte of the payload. |
-| [v2](#payload-format-v1) | `0x02`                | `2`               | Includes a version marker as the first byte of the payload. |
+| **Version**              | **Hexadecimal Value** | **Decimal Value** | **Description**                                                                                                                                                          |
+| ------------------------ | --------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [v0](#payload-format-v0) | `None`                | `None`            | Legacy format with 4-byte ciphertext length and no version marker. Device ID has variable length without explicit length field.                                          |
+| [v1](#payload-format-v1) | `0x01`                | `1`               | Introduces version marker, 2-byte ciphertext length, explicit device ID length field, and language code support. Uses Content Format V1 with 1-byte token length fields. |
+| [v2](#payload-format-v2) | `0x02`                | `2`               | Same payload structure as V1 but uses Content Format V2 with 2-byte token length fields for better support of longer access/refresh tokens.                              |
+| [v3](#payload-format-v3) | `0x03`                | `3`               | Same payload structure as V1/V2 but uses Content Format V3 with bitmap-based field presence indication for more efficient encoding.                                      |
 
 ## Payload Format V0
 
@@ -320,3 +396,51 @@ payload = (
 encoded = base64.b64encode(payload).decode("utf-8")
 print(encoded)
 ```
+
+## Payload Format V3
+
+> [See available versions](#supported-payload-versions)
+
+### Message Payload
+
+- **Format**:
+  - **1 byte**: Version Marker. [See available versions](#supported-payload-versions).
+  - **2 bytes**: Ciphertext Length.
+  - **1 byte**: Device ID Length.
+  - **1 byte**: Platform shortcode.
+  - **Variable**: Ciphertext. (encrypted [Content Format V3](#content-format-v3)).
+  - **Variable**: Device ID.
+  - **2 bytes**: Language Code (ISO 639-1 format).
+
+> [!NOTE]
+>
+> For detailed instructions on using the Double Ratchet algorithm to create ciphertext, refer to the [smswithoutborders_lib_sig documentation](https://github.com/smswithoutborders/lib_signal_double_ratchet_python?tab=readme-ov-file#double-ratchet-implementations).
+
+#### Visual Representation:
+
+```plaintext
++----------------+-------------------+------------------+--------------------+-----------------+-----------------+---------------+
+| Version Marker | Ciphertext Length | Device ID Length | Platform shortcode | Ciphertext      | Device ID       | Language Code |
+| (1 byte)       | (2 bytes)         | (1 byte)         | (1 byte)           | (Variable size) | (Variable size) | (2 bytes)     |
++----------------+-------------------+------------------+--------------------+-----------------+-----------------+---------------+
+```
+
+```python
+version_marker = b'\x03'
+platform_shortcode = b'g'
+language_code = b'en'
+device_id = b'...'
+encrypted_content = b'...'
+
+payload = (
+   version_marker +
+   struct.pack("<H", len(encrypted_content)) +
+   bytes([len(device_id)]) +
+   platform_shortcode +
+   encrypted_content +
+   device_id +
+   language_code
+)
+encoded = base64.b64encode(payload).decode("utf-8")
+print(encoded)
+```