docs: Define JSTable on-disk format

Author: michaelmior

ROADMAP.md

@@ -27,7 +27,7 @@ This stage will implement logging of all operations in ArgusDB to allow for faul
 
 This stage will allow the data stored in memory to be dumped to disk to allow storage of large data sets, resulting in a two-level LSM tree.
 
-- [] Define an on-disk format for JSTables in specs/storage.md including both the schema and the data
+- [x] Define an on-disk format for JSTables in specs/storage.md including both the schema and the data
 - [] When documents are inserted past the in-memory threshold, write a new JSTable to disk and start a new in-memory structure
 
 # Stage 5 - LSM compaction

specs/storage.md

@@ -5,8 +5,22 @@ A JSTable is like an SSTable, but stores semi-structured JSON data with an assoc
 
 Each JSTable is stored in a single file.
 This file must contain a JSON Schema as well as all the associated documents.
-Documents are compressed according to the schema.
-For example, if a field must be an integer, the type information need not be stored with each document.
-If a field can have multiple possible types, then the type information must be stored with each document.
+
+The format is as follows:
+1.  The schema, serialized as JSON, followed by a newline.
+2.  A sequence of records, one per line. Each record is a JSON array `[id, document]`.
+
+Example:
+```
+{"type":"object","properties":{"a":{"type":["integer"]}}}
+["01H4J3J4J3J4J3J4J3J4J3J4J3", {"a": 1}]
+["01H4J3J4J3J4J3J4J3J4J3J4J4", {"a": 2}]
+```
+
+## Compression
+
+Documents are stored as full JSON objects. This means that no compression is currently applied.
+Future versions of the format might include schema-based compression. For example, if a field must be an integer, the type information need not be stored with each document. If a field can have multiple possible types, then the type information must be stored with each document.
+
 
 # Compaction