docs: Add Python requirements file for data visualization dependencies

Author: lukefleed

README.md

@@ -11,26 +11,20 @@ A Rust library that provides space-efficient, in-memory representations for inte
 
 The library is designed to reduce the memory footprint of standard [`std::vec::Vec`] collections of integers while retaining performant access patterns.
 
-## Core Structures: `FixedVec` and `IntVec`
+## Core Structures: [`FixedVec`] and [`IntVec`]
 
 The library provides two distinct vector types, each based on a different encoding principle. Choosing the right one depends on your specific use case, performance requirements, and data characteristics.
 
-### `FixedVec`: Fixed-Width Encoding
+### [`FixedVec`]: Fixed-Width Encoding
 
 Implements a vector where every integer occupies the same, predetermined number of bits.
 
 *   **Key Features**:
-    *   **O(1) Random Access**: The memory location of any element is determined by a direct bit-offset calculation, resulting in minimal-overhead access. With low bit widths (e.g., 8, 16, 32), `FixedVec` can be 2-3x faster than `std::vec::Vec` for random access.
+    *   **O(1) Random Access**: The memory location of any element is determined by a direct bit-offset calculation, resulting in minimal-overhead access. With low bit widths (e.g., 8, 16, 32), [`FixedVec`] can be 2-3x faster than [`std::vec::Vec`] for random access.
     *   **Mutability**: Supports in-place modifications after creation through an API similar to [`std::vec::Vec`] (e.g., `push`, `set`, `pop`).
     *   **Atomic Operations**: Provides [`AtomicFixedVec`], a thread-safe variant that supports atomic read-modify-write operations for concurrent environments.
 
-*   **Use Cases**:
-    *   Data with a uniform or near-uniform distribution where all values fit within a known bit width.
-    *   Applications where low-latency random access is the primary performance requirement.
-    *   Scenarios requiring in-place vector modification or concurrent atomic updates.
-
-
-### `IntVec`: Variable-Width Encoding
+### [`IntVec`]: Variable-Width Encoding
 
 Implements a vector using variable-length instantaneous codes (e.g., Gamma, Delta, Rice, Zeta) to represent each integer.
 
@@ -39,32 +33,17 @@ Implements a vector using variable-length instantaneous codes (e.g., Gamma, Delt
     *   **Automatic Codec Selection**: Can analyze the data to select the most effective compression codec automatically.
     *   **Amortized O(1) Random Access**: Enables fast random access by sampling the bit positions of elements at a configurable interval (`k`).
 
-*   **Use Cases**:
-    *   Applications where minimizing memory usage is the primary goal.
-    *   Read-only datasets with skewed distributions.
-    *   Large-scale data processing where the vector is built once and read many times.
-
-### Summary
-
-| Feature             | [`FixedVec`]                                     | [`IntVec`]                                     |
-| :------------------ | :--------------------------------------------- | :------------------------------------------- |
-| **Encoding**        | Fixed-Width                                    | Variable-Width (Instantaneous Codes)         |
-| **Random Access**   | O(1) (direct computation)                      | O(k) (amortized via sampling)                |
-| **Mutability**      | Yes (`push`, `set`, etc.)                      | No (immutable after creation)                |
-| **Atomic Support**  | Yes ([`AtomicFixedVec`])                       | No (read-only concurrency)                   |
-
 [`FixedVec`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/fixed/struct.FixedVec.html
 [`IntVec`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/variable/struct.IntVec.html
 [`AtomicFixedVec`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/fixed/atomic/struct.AtomicFixedVec.html
 [`std::vec::Vec`]: https://doc.rust-lang.org/std/vec/struct.Vec.html
 
 ## Quick Start
 
-The following examples demonstrate the primary use cases for [`FixedVec`] and [`IntVec`]. All common types and traits are available through the [`prelude`].
+The following examples shows some use cases for [`FixedVec`] and [`IntVec`]. All common types and traits are available through the [`prelude`].
 
-### Example: `FixedVec` for Uniform Data and Mutable Access
+### Example: [`FixedVec`] for Uniform Data and Mutable Access
 
-Use [`FixedVec`] when data is uniformly distributed or when you require in-place modification or atomic operations.
 
 ```rust
 use compressed_intvec::prelude::*;
@@ -86,8 +65,6 @@ assert_eq!(vec.get(2), Some(35));
 
 ### Example: `IntVec` for High-Ratio Compression
 
-Use [`IntVec`] when minimizing memory usage is the priority and the data is read-only. It is most effective on data with skewed distributions.
-
 ```rust
 use compressed_intvec::prelude::*;
 
@@ -113,9 +90,9 @@ assert_eq!(intvec.get(3), Some(1000));
 [`LEIntVec`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/variable/type.LEIntVec.html
 [`VariableCodecSpec::Auto`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/variable/codec/enum.VariableCodecSpec.html#variant.Auto
 
-## Compression with `IntVec`
+## Compression with [`IntVec`]
 
-[`IntVec`] is the optimal choice when data that is not uniformly distributed and we want to minimize memory usage. It uses variable-length codes to represent integers, allowing for significant space savings compared to fixed-width encodings.
+[`IntVec`] is the optimal choice when data that is not uniformly distributed and we want to minimize memory usage. It uses variable-length codes to represent integers.
 
 The compression strategy is controlled by the [`VariableCodecSpec`] enum, passed to the builder.
 
@@ -126,19 +103,19 @@ For most use cases, the recommended strategy is [`VariableCodecSpec::Auto`], whi
 | `VariableCodecSpec` Variant | Description & Encoding Strategy | Optimal Data Distribution |
 | :--- | :--- | :--- |
 | **`Auto`** | **Recommended default.** Analyzes the data to choose the best variable-length code, balancing build time and compression ratio. | Agnostic; adapts to the input data. |
-| `Gamma` (γ) | A universal, parameter-free code. Encodes `n` using the unary code of ⌊log₂(*n*+1)⌋, followed by the remaining bits of `n`+1. | Implied distribution is ≈ 1/(2*x*²). Optimal for data skewed towards small non-negative integers. |
-| `Delta` (δ) | A universal, parameter-free code. Encodes `n` using the γ code of ⌊log₂(*n*+1)⌋, making it more efficient than γ for larger values. | Implied distribution is ≈ 1/(2*x*(log *x*)²). |
+| `Gamma` (γ) | A universal, parameter-free code. Encodes `n` using the unary code of log₂(*n*+1), followed by the remaining bits of `n`+1. | Implied distribution is ≈ 1/(2*x*²). Optimal for data skewed towards small non-negative integers. |
+| `Delta` (δ) | A universal, parameter-free code. Encodes `n` using the γ code of  log₂(*n*+1) , making it more efficient than γ for larger values. | Implied distribution is ≈ 1/(2*x*(log *x*)²). |
 | `Rice` | A fast, tunable version of Golomb codes where the parameter *b* must be a power of two. Encodes `n` by splitting it into a quotient (stored in unary) and a remainder (stored in binary). | Geometric distributions. |
 | `Golomb` | A tunable code, more general than Rice. Encodes `n` by splitting it into a quotient (stored in unary) and a remainder (stored using a minimal binary code). | Geometric distributions. Implied distribution is ≈ 1/*r*ˣ. |
-| `Zeta` (ζ) | A tunable code for power-law data. Encodes `n` based on ⌊⌊log₂(*n*+1)⌋/*k*⌋ in unary, followed by a minimal binary code for the remainder. | Power-law distributions (e.g., word frequencies, node degrees). Implied distribution is ≈ 1/*x*<sup>1+1/*k*</sup>. |
+| `Zeta` (ζ) | A tunable code for power-law data. Encodes `n` based on log₂(*n*+1)/*k* in unary, followed by a minimal binary code for the remainder. | Power-law distributions (e.g., word frequencies, node degrees). Implied distribution is ≈ 1/*x*<sup>1+1/*k*</sup>. |
 | `VByteLe`/`Be`| A byte-aligned code that uses a continuation bit to store integers in a variable number of bytes. Fast to decode. The big-endian variant is lexicographical. | Implied distribution is ≈ 1/*x*<sup>8/7</sup>. Good for general-purpose integer data. |
 | `Omega` (ω) | A universal, parameter-free code that recursively encodes the length of the binary representation of `n`. | Implied distribution is approximately 1/*x*. Compact for very large numbers. |
 | `Unary` | The simplest code. Encodes `n` as `n` zero-bits followed by a one-bit. | Geometric distributions with a very high probability of small values (e.g., boolean flags). |
 | `Explicit` | An escape hatch to use any code from the [`dsi-bitstream::codes::Codes`][dsi-bitstream-codes] enum. | Advanced use cases requiring specific, unlisted codes. |
 
 [dsi-bitstream-codes]: https://docs.rs/dsi-bitstream/latest/dsi_bitstream/codes/enum.Codes.html
 
-### Automatic Selection with `VariableCodecSpec::Auto`
+### Automatic Selection with [`VariableCodecSpec::Auto`]
 
 The `Auto` strategy removes the guesswork from codec selection. During the build phase, it analyzes the input data and selects the codec that offers the best compression ratio. This introduces a (non negligible) one-time cost for the analysis at construction time. Use the `Auto` codec when you want to create an [`IntVec`] once and read it many times, as the amortized cost of the analysis is negligible compared to the space savings and the performance of subsequent reads.
 
@@ -153,7 +130,7 @@ If you need to create multiple [`IntVec`] instances at run-time, consider using
 [dsi-bitstream-codes]: https://docs.rs/dsi-bitstream/latest/dsi_bitstream/codes/enum.Codes.html
 [`mem-dbg`]: https://docs.rs/mem-dbg/latest/mem_dbg/
 
-## `IntVec` Access Patterns
+## [`IntVec`] Access Patterns
 
 The access strategy for a compressed [`IntVec`] has significant performance implications. The library provides several methods, each optimized for a specific access pattern. Using the appropriate method is key to achieving high throughput.
 
@@ -162,7 +139,8 @@ The access strategy for a compressed [`IntVec`] has significant performance impl
 For retrieving a batch of elements from a slice of indices.
 
 *   **Mechanism**: This method sorts the provided indices to transform a random access pattern into a single, monotonic scan over the compressed data. This approach minimizes expensive bitstream seek operations and leverages data locality.
-*   **Use Case**: The preferred method for any batch lookup when all indices are known and can be stored in a slice.
+
+This should be your preferred method for any batch lookup when all indices are known and can be stored in a slice.
 
 ```rust
 use compressed_intvec::prelude::*;
@@ -171,7 +149,7 @@ use rand::Rng; // For random number generation
 let data: Vec<u64> = (0..10_000).collect();
 let intvec: LEIntVec = IntVec::builder()
     .codec(VariableCodecSpec::Delta)
-    .k(64)
+    .k(32)
     .build(&data)
     .unwrap();
 
@@ -189,7 +167,8 @@ assert_eq!(values, indices_to_get.iter().map(|&i| data[i]).collect::<Vec<_>>());
 For retrieving elements from a streaming iterator of indices.
 
 *   **Mechanism**: Processes indices on-the-fly using a stateful [`IntVecSeqReader`] internally, which is optimized for streams with sequential locality.
-*   **Use Case**: Use when indices cannot be collected into a slice, for example due to memory constraints.
+
+Use when indices cannot be collected into a slice, for example due to memory constraints.
 
 ```rust
 use compressed_intvec::prelude::*;
@@ -205,14 +184,15 @@ assert_eq!(values, vec![500, 501, 502, 503, 504]);
 
 ### Dynamic Lookups: `IntVecReader` and `IntVecSeqReader`
 
-For dynamic or interactive scenarios where lookup indices are not known in advance.
+There are interactive scenarios where lookup indices are not known in advance. The library provides two reader types to handle such cases:
 
 #### [`IntVecReader`]: Stateless Random Access
 
 A **stateless** reader for efficient, repeated random lookups.
 
 *   **Mechanism**: Amortizes the setup cost of the bitstream reader across multiple calls. Each `get` operation performs an independent seek from the nearest sample point.
-*   **Use Case**: Optimal for sparse, unpredictable access patterns where there is no locality between consecutive lookups.
+
+Optimal for sparse, unpredictable access patterns where there is no locality between consecutive lookups.
 
 ```rust
 use compressed_intvec::prelude::*;
@@ -231,7 +211,8 @@ assert_eq!(reader.get(10).unwrap(), Some(10));
 A **stateful** reader optimized for access patterns with sequential locality.
 
 *   **Mechanism**: Maintains an internal cursor. If a requested index is forward and within the same sample block, it decodes from the last position, avoiding a full seek.
-*   **Use Case**: Optimal for iterating through sorted or clustered indices where consecutive lookups are near each other.
+
+Optimal for iterating through sorted or clustered indices where consecutive lookups are near each other.
 
 ```rust
 use compressed_intvec::prelude::*;
@@ -252,15 +233,14 @@ assert_eq!(seq_reader.get(505).unwrap(), Some(505));
 [`IntVecReader`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/variable/reader/struct.IntVecReader.html
 [`IntVecSeqReader`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/variable/seq_reader/struct.IntVecSeqReader.html
 
-## Concurrency with `AtomicFixedVec`
+## Concurrency with [`AtomicFixedVec`]
 
-For concurrent applications, the library provides [`AtomicFixedVec`], a thread-safe variant of [`FixedVec`]. It supports atomic read-modify-write (RMW) operations, enabling safe and efficient manipulation of shared integer data across multiple threads.
+For concurrent applications, the library provides [`AtomicFixedVec`], a thread-safe variant of [`FixedVec`]. It supports atomic read-modify-write (RMW) operations, enabling safe and efficient manipulation of shared integer data across multiple threads. The atomicity guarantees depend on the element's bit width and its alignment within the underlying `u64` storage words.
 
-*   **Mechanism**: The atomicity guarantees depend on the element's bit width and its alignment within the underlying `u64` storage words.
-    *   **Lock-Free Path**: When an element is fully contained within a single `u64` word (common for power-of-two bit widths), operations are performed using lock-free atomic hardware instructions. Performance is optimal as no locks are involved.
-    *   **Locked Path**: When an element's bits span across the boundary of two `u64` words, operations are protected by a fine-grained mutex from a striped lock pool. This ensures atomicity for the two-word update without resorting to a global lock.
+*   **Lock-Free Path**: When an element is fully contained within a single `u64` word (guaranteed for power-of-two bit widths), operations are performed using lock-free atomic instructions. Performance here is optimal as no locks are involved.
+*   **Locked Path**: When an element's bits span across the boundary of two `u64` words (common for non-power-of-two bit widths), operations are protected by a fine-grained mutex from a striped lock pool. This ensures atomicity for the two-word update without resorting to a global lock.
 
-*   **Use Case**: Ideal for shared counters, parallel data processing, and any scenario requiring multiple threads to read from and write to the same integer vector concurrently. For write-heavy workloads, configuring the bit width to a power of two (e.g., 8, 16, 32) is recommended to ensure all operations remain on the lock-free path.
+Ideal for shared counters, parallel data processing, and any scenario requiring multiple threads to read from and write to the same integer vector concurrently. For write-heavy workloads, configuring the bit width to a power of two (e.g., 8, 16, 32) is recommended to ensure all operations remain on the lock-free path.
 
 ### Example: Concurrent Atomic Operations
 
@@ -323,7 +303,7 @@ fn main() {
     println!("Size of the uncompressed Vec<u64>:");
     data.mem_dbg(DbgFlags::HUMANIZE | DbgFlags::PERCENTAGE | DbgFlags::RUST_LAYOUT);
 
-    // Create an IntVec with a generic Gamma encoding.
+    // Create an IntVec with Gamma encoding.
     let gamma_intvec = LEIntVec::builder()
         .codec(VariableCodecSpec::Gamma)
         .build(&data)
@@ -355,12 +335,12 @@ fn main() {
 
 The output displays memory breakdown for a 1,000,000-element vector of `u64` integers, uniformly distributed between 0 and 2<sup>20</sup>
 
-- Standard `Vec<u64>`: 80.02 kB (8 bytes per element)
+- `Vec<u64>`: 80.02 kB (8 bytes per element)
 - [`IntVec`] with Gamma encoding: 47.10 kB (41% reduction)
 - [`IntVec`] with Auto codec selection: 28.33 kB (65% reduction, selected Zeta k=10)
 - [`FixedVec`] with minimal bit width: 25.06 kB (69% reduction, 20 bits per element)
 
-The memory analysis reveals the internal structure of each data type, including storage overhead for metadata, sampling structures, and encoding parameters.
+The memory analysis also shows the internal structure of each data type, including storage overhead for metadata, sampling structures, and encoding parameters.
 
 ```text
 Size of the uncompressed Vec<u64>:
@@ -405,23 +385,6 @@ Size of the FixedVec with minimal bit width:
     0  B   0.00% ╰╴_phantom
 ```
 
-## Cargo Features
-
-The library's functionality can be customized through the following Cargo features.
-
-### `parallel` (Default Feature)
-
-Enables parallel operations using the [Rayon] crate.Provides `par_iter` and `par_get_many` methods on [`FixedVec`] and [`IntVec`], as well as `par_iter` and `par_iter_mut` on [`AtomicFixedVec`].
-
-This feature is enabled by default, as it significantly improves performance for batch operations on large vectors.
-
-### `serde`
-
-Enables serialization and deserialization for [`FixedVec`], [`AtomicFixedVec`], and [`IntVec`] by implementing the `Serialize` and `Deserialize` traits from the [Serde] framework.
-
-This feature is not enabled by default.
-
-
 [Rayon]: https://docs.rs/rayon/latest/rayon/
 [Serde]: https://serde.rs/
 [`FixedVec`]: https://docs.rs/compressed-intvec/latest/compressed_intvec/fixed/struct.FixedVec.html
@@ -436,7 +399,7 @@ The library includes benchmarks for both [`FixedVec`] and [`IntVec`]. It also te
 cargo bench
 ```
 
-The benchmarks measure the performance of random access, batch access, iter_access, and memory usage for various data distributions and vector sizes. For a visual representation of the random_access performance, you can then run the following command:
+The benchmarks measure the performance of random access, batch access, iter_access, and memory usage for various data distributions and vector sizes. For a visual representation of the random access performance, run the following command:
 
 ```bash
 pip3 install -r python/requirements.txt

python/requirements.txt

@@ -0,0 +1,2 @@
+pandas==2.3.1
+plotly==6.2.0