review, start docs

Author: myrrc

.clang-format

@@ -7,7 +7,7 @@ SortIncludes: false
 TabWidth: 4
 IndentWidth: 4
 ColumnLimit: 110
-AllowShortFunctionsOnASingleLine: false
+AllowShortFunctionsOnASingleLine: true
 ---
 AccessModifierOffset: -4
 AlignAfterOpenBracket: Align
@@ -25,7 +25,7 @@ DerivePointerAlignment: false
 IncludeBlocks: Regroup
 InsertBraces: true
 Language: Cpp
-NamespaceIndentation: Inner
+NamespaceIndentation: None
 PointerAlignment: Right
 SpaceBeforeCpp11BracedList: true
 SpaceBeforeCtorInitializerColon: true

vortex-cxx/.clang-tidy .clang-tidyvortex-cxx/.clang-tidy renamed to .clang-tidy

@@ -1,82 +1,224 @@
 C++ API
 =======
 
-The Vortex C++ API provides an idiomatic C++ wrapper around the Vortex C FFI, built using
-`cxx <https://cxx.rs/>`_. It currently supports reading and writing Vortex files and integrates
-with the Arrow C Data Interface via `nanoarrow <https://arrow.apache.org/nanoarrow/>`_.
-
-In the future we will expand the C++ API to cover Vortex's plugin and extension points. Please
-reach out if you are interested in extending Vortex from C++ so we can prioritize these features.
+Vortex C++ API allows you to read and write `.vortex` files directly or via
+Arrow compatibility layer `nanoarrow <https://arrow.apache.org/nanoarrow/>`_.
+The only dependency apart from nanoarrow are Vortex C bindings.
 
 .. note::
-    Both the C++ API and this documentation are a work in progress. The API surface may change
-    significantly. Please reach out if you are interested in using Vortex from C++ so we can
-    prioritize stabilization.
-
+   C++ API is a work in progress. Please reach out to us if you are interested
+   in using Vortex from C++ or you want a feature not covered yet e.g.
+   extension DataType support.
 
 Installation
 ------------
 
-The C++ bindings are built using CMake. Requirements:
+We don't provide prebuilt library files (yet) so you will need to build Vortex
+from source, and for that you will need:
+
+- C++20,
+- Rust toolchain,
+- and CMake 3.10.
+
+```
+git clone --depth 1 https://github.com/vortex-data/vortex
+cd vortex
+cargo build --release -p vortex-ffi
+cd vortex-cxx
+mkdir build
+cmake -Bbuild
+cmake --build build -j
+```
+
+This will produce a shared and a static library in `vortex/target/release/`
+which you can use directly or via
+
+```
+# static library
+target_link_libraries(target PRIVATE vortex_cxx nanoarrow)
+# shared library
+target_link_libraries(target PRIVATE vortex_cxx_shared nanoarrow_shared)
+```
+
+Have a look at `examples <https://github.com/vortex-data/vortex/tree/develop/vortex-cxx/examples>`_
+directory as well.
+
+Reading files
+-------------
 
-* CMake 3.22 or higher
-* C++20 compatible compiler
-* Rust toolchain (for building the underlying Rust library)
+TODO provide sample files.
 
-.. code-block:: bash
+Let's start with an example. Assuming you have Vortex files `people0`, `people1`,
+and `me` in a local folder, each containing two U64 columns "age" and "height",
+this is how you print all ages for specific heights:
 
-    cd vortex-cxx
-    mkdir build && cd build
-    cmake ..
-    make -j$(nproc)
+.. code-block:: cpp
 
+    using namespace vortex;
+    using namespace vortex::expr;
+    using namespace vortex::expr::ops;
+
+    const Session session;
+    const DataSource ds = DataSource::open(session, {"people*.vortex", "me.vortex"});
+    Scan scan = ds.scan({
+        .projection = col("age"),
+        .filter = col("height") >= lit<uint64_t>(180)
+    });
+
+    for (Partition& partition : scan.partitions()) {
+        for (Array& array : partition.batches()) {
+            PrimitiveView<uint64_t> age_view = batch.field("age").values<uint64_t>(session);
+            std::span<const uint64_t> age_values = age_view.values();
+            for (uint64_t value: age_values) {
+                std::cout << value << " ";
+            }
+        }
+    }
 
-Compatibility
--------------
+DataSource and Scan
+^^^^^^^^^^^^^^^^^^^
+
+First, you need to create a Vortex session which holds stuff like object store
+credentials and does extension bookkeeping (we'll get back to it later). Then
+you need to create a DataSource, which is a view over multiple files which may
+also be remote. You can specify globs for every item as it's shown.
+
+Once you have a DataSource, you can create Scans out of it. A Scan is a single
+traversal of a DataSource which projects columns and filters on them. Our Scan
+is consumed by following calls so it needs to be non-`const`. ScanOptions which
+are passed to Scan are a simple C++ aggregate so you can initialize any fields
+you want or avoid them altogether (`auto scan = ds.scan()`). Options have
+reasonable defaults: empty projection returns all fields, empty filter doesn't
+filter rows, and so on.
+
+Expressions
+^^^^^^^^^^^
+
+We want to return only the "age" column so we pass it a `col` Expression - it
+gets the root struct of the file (a `DataTypeVariant::Struct`) and then
+extracts the column we want by name. A filter works in a similar manner - we
+return an Expression which returns false for some "height" values, and the scan
+filters them out.
+
+Two additional things to look in `.filter`: first, we allow overloading
+Expression operators which produce Expressions themselves a-la Eigen. This is
+opt-in via `using namespace vortex::expr::ops`. If you prefer, you can use
+`eq(col("height"), lit<uint64_t>(180))` instead. Second thing is that we
+explicitly pass the type to the `lit` expression, which creates a literal
+constant. We don't do any type coersion in Expression enging, so if you were to
+write `lit(180)`, C++ would likely deduce the type to `uint8_t` and fail in
+runtime.
+
+Once we're done with the scan, we need to consume the data it provides. If
+you want, you can get the data in Arrow format, but now we want our own
+Partitions and Arrays.
+
+Partitions and Arrays
+^^^^^^^^^^^^^^^^^^^^^
+
+A Partition is an independent unit of work. Assuming your Vortex file
+processing will likely be multithreaded, each thread can pull Partitions from
+a Scan and handle them in parallel. For simplicity we don't cover it here, but
+look for the next sections! Each Partition produces Arrays which are batches of
+rows and columns. As we've requested "age", each Array will have one named
+column "age".
+
+.. note::
+   Arrays, DataTypes, and Expressions are reference-counted so copying them is
+   cheap
 
-The C++ bindings are supported on the following architectures:
+Once we have an Array, we can get access to its values. First, we need to
+extract "age" field. Then we want to get access to the raw bytes. However,
+Array likely references compressed data, so now we need to learn about
+canonicalization.
 
-* x86_64 Linux
-* ARM64 Linux
-* Apple Silicon macOS
+Canonicalization
+^^^^^^^^^^^^^^^^
 
-They support any Linux distribution with a GLIBC version >= 2.31. This includes
+Vortex files hold layers of compressed data. Each layer is a specific encoding
+(zstd, FSST, delta) on top of another layer. This is good for performance
+because we defer decompression and we can also pass compressed data directly to
+other systems. Say, if a reader knows how to deal with bitpacked integers but
+not RLE and we have `RLE(Bitpacked(U64))`, we can decompress just RLE and pass
+bitpacked array directly to the reader.
 
-* Amazon Linux 2022 or newer
-* Ubuntu 20.04 or newer
+In this example we want to remove all encodings and uncompress all data fully.
+This form is called a canonical Array, and the process is canonicalization.
 
+When we request `.values(session)`, we canonicalize the array and get a
+PrimitiveView because we already know the type of the column. As PrimitiveView
+holds uncompressed data, we can get raw `uint64_t` numbers by calling
+`.values()` on the view.
 
-Usage Example
+Writing files
 -------------
 
-Here's a basic example of reading a Vortex file into an Arrow array stream:
+Now let's write the files to be read by our previous example.
 
 .. code-block:: cpp
+    // Assume you have imported the namespaces
+    using dtype::Nullable;
+
+    const Session session;
+    const DataType dtype = dtype::struct({
+         {"age", dtype::uint8()},
+         {"height", dtype::uint16(Nullable)},
+    });
+
+    constexpr size_t SAMPLE_ROWS = 100;
+    std::vector<uint8_t> ages(SAMPLE_ROWS);
+    std::vector<uint16_t> heights(SAMPLE_ROWS);
+    for (size_t i = 0; i < SAMPLE_ROWS; ++i) {
+        ages[i] = static_cast<uint8_t>(i);
+        heights[i] = static_cast<uint16_t>((i + 1) % 200);
+    }
 
-    #include "vortex/file.hpp"
-    #include "vortex/scan.hpp"
+    Array array = make_struct({
+        {"age", Array::primitive<uint8_t>(ages)},
+        {"height", Array::primitive<uint16_t>(heights, AllValid)},
+    });
 
-    // Open a Vortex file and scan with a row limit
-    auto stream = vortex::VortexFile::Open("data.vortex")
-        .CreateScanBuilder()
-        .WithLimit(1000)
-        .IntoStream();
+    const Validity validity{ValidityType::Array, validity_array};
+    Array array2 = make_struct({
+        {"age", Array::primitive<uint8_t>(ages)},
+        {"height", Array::primitive<uint16_t>(heights, validity)},
+    });
 
-    // Consume the Arrow C Data stream
-    ArrowArray array;
-    while (stream.get_next(&stream, &array) == 0 && array.release != nullptr) {
-        // Process each batch...
-    }
+    FileWriter writer = FileWriter::open(session, "people0.vortex", dtype);
+    writer.push(array);
+    writer.push(array2);
+    writer.finish();
 
+DataType
+^^^^^^^^
+
+First, you want to create a schema. DataType is a logical representation of
+Vortex's type system. "Logical" as opposed to "physical" means DataType doesn't
+know what encodings the data uses. As another example, Arrow and Parquet's types
+are physical. There's a convenience function `struct_` which creates a
+`DataTypeVariant::Struct` with fields passed as a `intializer_list`.
+
+Each DataType has a notion of nullability: whether some items in the row can be
+invalid (Vortex uses "null" and "invalid" terms interchangeably). DataTypes are
+non-nullable by default, so "age" column is not nullable, but "height" is.
+
+Then we prepare individual columns and use `make_struct` to assemble them into
+Array, and here we see Validity.
+
+Validity
+^^^^^^^^
+
+DataType's Nullable flag tells us we may theoretically have invalid items in
+the Array, but to know this for sure we need Validity. "age" is non-Nullable
+so its Validity is NonNullable which means, again, we can't have invalid
+elements.
+
+"height" in first Array is AllValid which means in this Array we don't have
+invalid items.
 
-API Reference
--------------
 
-.. toctree::
-   :maxdepth: 2
+Multithreaded scans
+-------------------
 
-   dtypes
-   scalars
-   expr
-   file
-   scan
+Arrow roundtrip
+---------------

docs/api/cpp/dtypes.rst

@@ -1,4 +1,7 @@
 # Vortex C++ bindings
+
+For a usage guide, see docs/api/cpp/index.rst.
+
 ## Requirements
 
 - CMake 3.10+