Update version to 0.5.0 and add API reference documentation

Author: birchkwok

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "numpack"
-version = "0.4.5"
+version = "0.5.0"
 edition = "2021"
 
 [lib]

docs/02_core_operations.md

@@ -12,6 +12,7 @@ This document covers all core NumPack operations including save, load, replace,
 - [Drop Operations](#drop-operations)
 - [Random Access](#random-access)
 - [Metadata Operations](#metadata-operations)
+- [Array Operations](#array-operations)
 - [Stream Loading](#stream-loading)
 - [File Management](#file-management)
 
@@ -531,6 +532,65 @@ with NumPack("data.npk") as npk:
 
 ---
 
+## Array Operations
+
+### `clone(source_name, target_name)`
+
+Clone an existing array to a new array name.
+
+#### Parameters
+
+- `source_name` (str): Name of the source array to clone
+- `target_name` (str): Name for the cloned array
+
+#### Example
+
+```python
+with NumPack("data.npk") as npk:
+    # Save original data
+    npk.save({'original': np.random.rand(100, 50)})
+    
+    # Clone to new array
+    npk.clone('original', 'backup')
+    
+    # Modify the clone independently
+    backup = npk.load('backup')
+    backup *= 2.0
+    npk.save({'backup': backup})
+    
+    # Original is unchanged
+    original = npk.load('original')
+```
+
+#### Notes
+
+- The cloned array is independent of the original
+- Raises `KeyError` if source array doesn't exist
+- Raises `ValueError` if target array already exists
+
+### `get_io_stats()`
+
+Get I/O statistics for the NumPack instance.
+
+#### Returns
+
+- `Dict[str, Any]`: Dictionary containing backend statistics
+
+#### Example
+
+```python
+with NumPack("data.npk") as npk:
+    stats = npk.get_io_stats()
+    print(f"Backend: {stats['backend_type']}")
+```
+
+#### Notes
+
+- Currently returns basic backend information
+- Detailed per-call statistics may be added in future versions
+
+---
+
 ## Stream Loading
 
 ### `stream_load(array_name, buffer_size=None)`

docs/06_quick_reference.md

@@ -201,7 +201,7 @@ to_torch_file('input.npk', 'output.pt')
 to_parquet_file('input.npk', 'output.parquet')
 ```
 
-### Metadata
+### Metadata & Array Operations
 
 ```python
 # Get shape
@@ -216,11 +216,17 @@ exists = npk.has_array('array')
 # Get modification time
 timestamp = npk.get_modify_time('array')
 
+# Clone an array
+npk.clone('source_array', 'target_array')
+
 # Reset (clear all)
 npk.reset()
 
 # Compact after deletions
 npk.update('array')
+
+# Get I/O statistics
+stats = npk.get_io_stats()
 ```
 
 ### Batch Modes

docs/07_io_conversion.md

@@ -2,6 +2,23 @@
 
 NumPack provides comprehensive format conversion utilities for seamless integration with popular data frameworks.
 
+## Table of Contents
+
+- [Overview](#overview)
+- [PyTorch Conversion](#pytorch-conversion)
+- [PyArrow/Feather Conversion](#pyarrowfeather-conversion)
+- [Parquet Conversion](#parquet-conversion)
+- [SafeTensors Conversion](#safetensors-conversion)
+- [Other Formats](#other-formats)
+- [Text File Conversion](#text-file-conversion)
+- [Pandas Conversion](#pandas-conversion)
+- [S3 Cloud Storage](#s3-cloud-storage)
+- [Zero-Copy Utilities](#zero-copy-utilities)
+- [Supported Formats Summary](#supported-formats-summary)
+- [Best Practices](#best-practices)
+
+---
+
 ## Overview
 
 NumPack supports two types of conversions:
@@ -333,6 +350,140 @@ print("Model conversion pipeline complete!")
 
 ---
 
+## Text File Conversion
+
+### TXT Files
+
+```python
+from numpack.io import from_txt, to_txt
+
+# .txt → .npk (whitespace-delimited)
+from_txt('data.txt', 'output.npk', array_name='data', delimiter=None)
+
+# .npk → .txt
+to_txt('input.npk', 'output.txt', array_name='data', delimiter='\t')
+```
+
+**Parameters:**
+- `delimiter`: Field separator (default: whitespace)
+- `skip_header`: Number of header rows to skip
+- `dtype`: Target data type
+
+---
+
+## Pandas Conversion
+
+### DataFrame ↔ .npk
+
+```python
+from numpack.io import from_pandas, to_pandas
+import pandas as pd
+
+# DataFrame → .npk
+df = pd.DataFrame({'a': [1, 2, 3], 'b': [4.0, 5.0, 6.0]})
+from_pandas(df, 'output.npk', array_name='dataframe')
+
+# .npk → DataFrame
+df = to_pandas('input.npk', array_name='dataframe')
+print(df.columns)
+```
+
+**Notes:**
+- Numeric columns are converted to NumPy arrays
+- String columns may require special handling
+
+---
+
+## S3 Cloud Storage
+
+NumPack supports direct reading and writing to Amazon S3.
+
+### S3 ↔ .npk
+
+```python
+from numpack.io import from_s3, to_s3
+
+# Download from S3 and convert to .npk (uses default AWS credentials)
+from_s3('s3://my-bucket/data.npy', 'output.npk')
+
+# Public bucket access
+from_s3('s3://public-bucket/data.csv', 'output.npk', anon=True)
+
+# Upload .npk to S3
+to_s3('input.npk', 's3://my-bucket/output.parquet')
+
+# Specify output format
+to_s3('input.npk', 's3://my-bucket/output.csv', format='csv')
+```
+
+**Parameters:**
+- `s3_path`: S3 URI in the form `s3://bucket/path/to/file`
+- `format`: Input/output format (`'auto'`, `'numpy'`, `'csv'`, `'txt'`, `'parquet'`, `'feather'`, `'hdf5'`)
+- `**s3_kwargs`: Keyword arguments forwarded to `s3fs.S3FileSystem` (e.g., `anon=True` for public buckets)
+
+**Dependencies:** `s3fs`
+
+---
+
+## Zero-Copy Utilities
+
+NumPack provides zero-copy utilities for efficient data exchange with other libraries.
+
+### DLPack Protocol
+
+```python
+from numpack.io import to_dlpack, from_dlpack
+
+# NumPy → DLPack capsule
+arr = np.random.rand(100, 50)
+capsule = to_dlpack(arr)
+
+# DLPack capsule → NumPy
+arr_restored = from_dlpack(capsule)
+```
+
+### Arrow Zero-Copy
+
+```python
+from numpack.io import numpy_to_arrow_zero_copy, arrow_to_numpy_zero_copy
+
+# NumPy → Arrow (zero-copy)
+arr = np.random.rand(100, 50).astype(np.float32)
+arrow_arr = numpy_to_arrow_zero_copy(arr)
+
+# Arrow → NumPy (zero-copy)
+numpy_arr = arrow_to_numpy_zero_copy(arrow_arr)
+```
+
+### PyTorch Zero-Copy
+
+```python
+from numpack.io import numpy_to_torch_zero_copy, torch_to_numpy_zero_copy
+
+# NumPy → PyTorch (shared memory)
+arr = np.random.rand(100, 50).astype(np.float32)
+tensor = numpy_to_torch_zero_copy(arr)
+
+# PyTorch → NumPy (shared memory)
+numpy_arr = torch_to_numpy_zero_copy(tensor)
+```
+
+### ZeroCopyArray Wrapper
+
+```python
+from numpack.io import ZeroCopyArray, wrap_for_zero_copy
+
+# Wrap array for zero-copy operations
+arr = np.random.rand(100, 50)
+zc_arr = wrap_for_zero_copy(arr)
+
+# Access as different formats
+torch_tensor = zc_arr.to_torch()
+arrow_array = zc_arr.to_arrow()
+```
+
+---
+
 ## Supported Formats Summary
 
 | Format | Import | Export | Dependencies |
@@ -345,7 +496,9 @@ print("Model conversion pipeline complete!")
 | HDF5 (.h5) | ✅ | ✅ | `h5py` |
 | Zarr | ✅ | ✅ | `zarr` |
 | CSV | ✅ | ✅ | - |
+| TXT | ✅ | ✅ | - |
 | Pandas | ✅ | ✅ | `pandas` |
+| S3 | ✅ | ✅ | `boto3`, `s3fs` |
 
 ---
 

docs/README.md

@@ -15,6 +15,11 @@ Welcome to the NumPack API documentation! NumPack is a high-performance array st
 
 ### API Reference
 
+- **[API Reference (Detailed)](./api_reference/README.md)** ⭐ NEW
+  - Complete function-level documentation
+  - Parameters, return values, and examples
+  - Organized by module (Core, IO, Utils)
+
 - **[02. Core Operations](./02_core_operations.md)**
   - Complete API reference for all basic operations
   - `save()`, `load()`, `replace()`, `append()`, `drop()`
@@ -76,7 +81,6 @@ Welcome to the NumPack API documentation! NumPack is a high-performance array st
 | **API lookup** | [Core Operations](./02_core_operations.md) | Complete API reference |
 | **Performance optimization** | [Batch Processing](./03_batch_processing.md), [Performance Guide](./05_performance_guide.md) | 25-174x speedup |
 | **Large datasets** | [Advanced Features](./04_advanced_features.md) | Lazy loading, streaming |
-｜ 
 | **Quick answers** | [Quick Reference](./06_quick_reference.md) | Cheatsheet, common patterns |
 | **Format conversion** | [IO Conversion](./07_io_conversion.md) | PyTorch, Arrow, Parquet, SafeTensors |
 

docs/api_reference/README.md

@@ -0,0 +1,86 @@
+# NumPack API Reference
+
+Complete API documentation for all NumPack modules and functions.
+
+## Documentation Structure
+
+```
+api_reference/
+├── README.md                 # This file
+├── core/
+│   ├── numpack_class.md      # NumPack class
+│   ├── lazy_array.md         # LazyArray class
+│   └── batch_modes.md        # BatchModeContext & WritableBatchMode
+├── io/
+│   ├── README.md             # IO module overview
+│   ├── pytorch.md            # PyTorch conversion
+│   ├── arrow_feather.md      # PyArrow/Feather conversion
+│   ├── parquet.md            # Parquet conversion
+│   ├── safetensors.md        # SafeTensors conversion
+│   ├── numpy.md              # NumPy conversion
+│   ├── hdf5.md               # HDF5 conversion
+│   ├── zarr.md               # Zarr conversion
+│   ├── csv_txt.md            # CSV/TXT conversion
+│   ├── pandas.md             # Pandas conversion
+│   ├── s3.md                 # S3 cloud storage
+│   └── zero_copy.md          # Zero-copy utilities
+└── utils/
+    ├── package_io.md         # pack/unpack functions
+    └── utilities.md          # Utility functions
+```
+
+## Quick Navigation
+
+### Core Classes
+
+| Class | Description | Documentation |
+|-------|-------------|---------------|
+| `NumPack` | Main array storage class | [numpack_class.md](./core/numpack_class.md) |
+| `LazyArray` | Memory-mapped lazy loading | [lazy_array.md](./core/lazy_array.md) |
+| `BatchModeContext` | In-memory batch caching | [batch_modes.md](./core/batch_modes.md) |
+| `WritableBatchMode` | Zero-copy writable batch | [batch_modes.md](./core/batch_modes.md) |
+
+### IO Conversion Functions
+
+| Format | Import | Export | Documentation |
+|--------|--------|--------|---------------|
+| PyTorch | `from_torch`, `from_torch_file` | `to_torch`, `to_torch_file` | [pytorch.md](./io/pytorch.md) |
+| Arrow/Feather | `from_arrow`, `from_feather_file` | `to_arrow`, `to_feather_file` | [arrow_feather.md](./io/arrow_feather.md) |
+| Parquet | `from_parquet_table`, `from_parquet_file` | `to_parquet_table`, `to_parquet_file` | [parquet.md](./io/parquet.md) |
+| SafeTensors | `from_safetensors`, `from_safetensors_file` | `to_safetensors`, `to_safetensors_file` | [safetensors.md](./io/safetensors.md) |
+| NumPy | `from_numpy` | `to_numpy` | [numpy.md](./io/numpy.md) |
+| HDF5 | `from_hdf5` | `to_hdf5` | [hdf5.md](./io/hdf5.md) |
+| Zarr | `from_zarr` | `to_zarr` | [zarr.md](./io/zarr.md) |
+| CSV/TXT | `from_csv`, `from_txt` | `to_csv`, `to_txt` | [csv_txt.md](./io/csv_txt.md) |
+| Pandas | `from_pandas` | `to_pandas` | [pandas.md](./io/pandas.md) |
+| S3 | `from_s3` | `to_s3` | [s3.md](./io/s3.md) |
+
+### Utility Functions
+
+| Function | Description | Documentation |
+|----------|-------------|---------------|
+| `pack` | Package NumPack directory | [package_io.md](./utils/package_io.md) |
+| `unpack` | Extract NumPack package | [package_io.md](./utils/package_io.md) |
+| `get_package_info` | Get package metadata | [package_io.md](./utils/package_io.md) |
+| `get_backend_info` | Get backend information | [utilities.md](./utils/utilities.md) |
+
+## Import Patterns
+
+```python
+# Core class
+from numpack import NumPack, LazyArray
+
+# IO conversion functions
+from numpack.io import from_torch, to_torch
+from numpack.io import from_numpy, to_numpy
+
+# Package operations
+from numpack import pack, unpack, get_package_info
+
+# Backend info
+from numpack import get_backend_info
+```
+
+## Version
+
+This documentation is for NumPack version **0.5.0**.