Add godoc comments and improve documentation

- Add comprehensive godoc comments to all public types and functions
- Add documentation for key internal functions
- Replace custom nullWriter with io.Discard from stdlib
- Fix staticcheck warnings (unused err values, redundant return, unused field)
- Expand README.md with detailed usage examples, configuration options,
  API reference, and explanation of how the library works

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: MagicalTux

README.md

@@ -2,27 +2,216 @@
 [![GoDoc](https://godoc.org/github.com/KarpelesLab/smartremote?status.svg)](https://godoc.org/github.com/KarpelesLab/smartremote)
 [![Coverage Status](https://coveralls.io/repos/github/KarpelesLab/smartremote/badge.svg?branch=master)](https://coveralls.io/github/KarpelesLab/smartremote?branch=master)
 
-
 # SmartRemote
 
-NOTE: this is not a remote for your TV, just an easy way to access remote (http) files.
+SmartRemote is a Go library that provides seamless access to remote HTTP files with intelligent partial downloading and local caching. Rather than downloading entire files upfront, it allows you to open a URL and read from it like a regular file, automatically fetching only the needed portions on-demand.
+
+This is particularly useful for large files like ISOs, ZIPs, and other archives where you might only need to access specific sections (e.g., reading the central directory of a ZIP file without downloading the entire archive).
+
+## Features
+
+- **Lazy Loading**: Downloads only the blocks you actually read
+- **Resume Support**: Partial downloads can be saved and resumed via `.part` files
+- **Intelligent Seeking**: Handles seekable HTTP connections using Range requests
+- **Concurrent Downloads**: Manages multiple concurrent download clients with configurable limits
+- **Idle Background Downloading**: Automatically fills gaps in partial downloads when not actively reading
+- **Block-Based Tracking**: Uses efficient RoaringBitmap for tracking downloaded 64KB blocks
+- **Standard Interfaces**: Implements `io.Reader`, `io.ReaderAt`, and `io.Seeker`
+
+## Installation
+
+```bash
+go get github.com/KarpelesLab/smartremote
+```
+
+## Quick Start
+
+```go
+package main
+
+import (
+    "fmt"
+    "io"
+    "github.com/KarpelesLab/smartremote"
+)
+
+func main() {
+    // Open a remote file
+    f, err := smartremote.Open("https://example.com/largefile.zip")
+    if err != nil {
+        panic(err)
+    }
+    defer f.Close()
+
+    // Use f as a regular read-only file
+    // It will download parts as needed from the remote URL
+    buf := make([]byte, 1024)
+    n, err := f.Read(buf)
+    if err != nil && err != io.EOF {
+        panic(err)
+    }
+    fmt.Printf("Read %d bytes\n", n)
+}
+```
+
+## How It Works
+
+### Block-Based Downloads
 
-How to use:
+SmartRemote divides remote files into 64KB blocks. When you read from the file, only the blocks containing the requested data are downloaded. Downloaded blocks are:
 
-```Go
-	f, err := smartremote.Open("http://...")
-	if err != nil {
-		panic(err)
-	}
+1. Stored in a local temporary file
+2. Tracked using a RoaringBitmap for efficient status checking
+3. Persisted to a `.part` file so downloads can be resumed
 
-	// Use "f" as a regular readonly file, it'll download parts as needed from the remote url
+### HTTP Range Requests
+
+The library uses HTTP Range requests (status 206 Partial Content) to download specific byte ranges. If the server doesn't support Range requests, SmartRemote falls back to downloading the entire file.
+
+### Connection Pooling
+
+The `DownloadManager` maintains a pool of HTTP connections (default: 10) that are reused across requests. Idle connections are automatically cleaned up after 5 minutes.
+
+### Background Downloading
+
+When there are no active read requests, SmartRemote opportunistically downloads missing blocks in the background, progressively completing the file.
+
+## Advanced Usage
+
+### Custom DownloadManager
+
+Create a custom `DownloadManager` for more control:
+
+```go
+dm := smartremote.NewDownloadManager()
+dm.MaxConcurrent = 5           // Limit to 5 concurrent connections
+dm.MaxDataJump = 1024 * 1024   // Allow skipping up to 1MB when seeking
+dm.TmpDir = "/custom/tmp"      // Custom temp directory
+dm.Client = customHTTPClient   // Use a custom http.Client
+
+f, err := dm.Open("https://example.com/file.iso")
+if err != nil {
+    panic(err)
+}
+defer f.Close()
 ```
 
-This can be used with any kind of file as long as the server supports resume.
-If it doesn't then this will just download the whole file, and still work the
-same.
+### Specify Local Storage Path
+
+Store the downloaded file at a specific path:
+
+```go
+dm := smartremote.NewDownloadManager()
+f, err := dm.OpenTo("https://example.com/file.iso", "/path/to/local/file.iso")
+if err != nil {
+    panic(err)
+}
+defer f.Close()
+```
+
+### Simple ReaderAt Interface
+
+For simple use cases where you just need `io.ReaderAt`:
+
+```go
+dm := smartremote.NewDownloadManager()
+reader := dm.For("https://example.com/file.bin")
+
+buf := make([]byte, 100)
+n, err := reader.ReadAt(buf, 1000) // Read 100 bytes starting at offset 1000
+```
+
+### Force Complete Download
+
+Download the entire file:
+
+```go
+f, err := smartremote.Open("https://example.com/file.zip")
+if err != nil {
+    panic(err)
+}
+defer f.Close()
+
+// Download everything
+err = f.Complete()
+if err != nil {
+    panic(err)
+}
+```
+
+### Manual Progress Saving
+
+Manually trigger a save of download progress:
+
+```go
+f, err := smartremote.Open("https://example.com/file.zip")
+if err != nil {
+    panic(err)
+}
+
+// ... perform some reads ...
+
+// Save progress explicitly
+err = f.SavePart()
+if err != nil {
+    panic(err)
+}
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `MaxConcurrent` | `int` | 10 | Maximum number of concurrent HTTP connections |
+| `MaxDataJump` | `int64` | 512KB | Maximum bytes to read and discard when seeking forward (vs opening a new connection) |
+| `TmpDir` | `string` | `os.TempDir()` | Directory for temporary download files |
+| `Client` | `*http.Client` | `http.DefaultClient` | HTTP client for making requests |
+| `Logger` | `*log.Logger` | stderr | Logger for debug output |
+
+## API Reference
+
+### Package Functions
+
+- `Open(url string) (*File, error)` - Open a remote URL using the default manager
+
+### DownloadManager
+
+- `NewDownloadManager() *DownloadManager` - Create a new download manager
+- `Open(url string) (*File, error)` - Open a URL with auto-generated local path
+- `OpenTo(url, localPath string) (*File, error)` - Open a URL with specific local path
+- `For(url string) io.ReaderAt` - Get a simple ReaderAt for a URL
+
+### File
+
+- `Read(p []byte) (n int, err error)` - Read from current position
+- `ReadAt(p []byte, off int64) (int, error)` - Read from specific offset
+- `Seek(offset int64, whence int) (int64, error)` - Seek to position
+- `Close() error` - Close file and save progress
+- `GetSize() (int64, error)` - Get remote file size
+- `SetSize(size int64)` - Manually set file size
+- `Stat() (os.FileInfo, error)` - Get file info
+- `Complete() error` - Download entire file
+- `SavePart() error` - Manually save download progress
+
+## Resume Behavior
+
+When opening a URL:
+
+1. If the local file doesn't exist, a new download begins
+2. If the local file exists with a `.part` file, the download resumes from where it left off
+3. If the local file exists without a `.part` file, it's assumed to be complete
+
+On close:
+- If download is incomplete and progress was saved successfully, both files are kept for resume
+- If download is incomplete and progress save failed, the partial file is deleted
+- If download is complete, the `.part` file is removed
+
+## Requirements
+
+- Go 1.18 or later
+- Server must support HTTP Range requests for partial downloads (falls back to full download otherwise)
 
-# TODO
+## TODO
 
-* Add support for range invalidation (bad checksum causes re-download of affected area)
-* Refactor idle downloader for better performances
+- Add support for range invalidation (bad checksum causes re-download of affected area)
+- Refactor idle downloader for better performance

client.go

@@ -9,6 +9,9 @@ import (
 	"time"
 )
 
+// dlClient is an internal HTTP connection handler that manages individual
+// connections to remote URLs. It handles Range requests, connection reuse,
+// and idle background downloading of missing blocks.
 type dlClient struct {
 	dlm      *DownloadManager
 	url      string
@@ -23,6 +26,7 @@ type dlClient struct {
 	expire time.Time
 }
 
+// Close closes the HTTP connection and signals completion to the manager.
 func (dl *dlClient) Close() error {
 	dl.lk.Lock()
 	defer dl.lk.Unlock()
@@ -38,17 +42,19 @@ func (dl *dlClient) Close() error {
 	return nil
 }
 
+// dropDataCount reads and discards cnt bytes from the HTTP response body.
+// If a handler is set, it opportunistically saves complete blocks to disk.
 func (dl *dlClient) dropDataCount(cnt, startPos int64) error {
 	if dl.handler == nil {
-		_, err := io.CopyN(nullWriter{}, dl.reader.Body, cnt)
+		_, err := io.CopyN(io.Discard, dl.reader.Body, cnt)
 		return err
 	}
 
 	// download data in buffers
 	sz := dl.handler.getBlockSize()
 	if sz <= 0 || cnt < sz {
 		// doesn't want data?
-		_, err := io.CopyN(nullWriter{}, dl.reader.Body, cnt)
+		_, err := io.CopyN(io.Discard, dl.reader.Body, cnt)
 		return err
 	}
 
@@ -57,7 +63,7 @@ func (dl *dlClient) dropDataCount(cnt, startPos int64) error {
 	for cnt > 0 {
 		if cnt < sz {
 			// can't download enough so that it's worth it
-			_, err := io.CopyN(nullWriter{}, dl.reader.Body, cnt)
+			_, err := io.CopyN(io.Discard, dl.reader.Body, cnt)
 			return err
 		}
 
@@ -71,7 +77,7 @@ func (dl *dlClient) dropDataCount(cnt, startPos int64) error {
 		err = dl.handler.ingestData(buf, startPos)
 		if err != nil {
 			// give up
-			_, err := io.CopyN(nullWriter{}, dl.reader.Body, cnt)
+			_, err := io.CopyN(io.Discard, dl.reader.Body, cnt)
 			return err
 		}
 
@@ -81,6 +87,8 @@ func (dl *dlClient) dropDataCount(cnt, startPos int64) error {
 	return nil
 }
 
+// ReadAt reads data from the remote URL at the specified offset into p.
+// It reuses existing HTTP connections when possible, or opens new ones as needed.
 func (dl *dlClient) ReadAt(p []byte, off int64) (int, error) {
 	dl.lk.Lock()
 	defer dl.lk.Unlock()
@@ -110,10 +118,13 @@ func (dl *dlClient) ReadAt(p []byte, off int64) (int, error) {
 		}
 	}
 
-	// instanciate a new reader if needed
+	// instantiate a new reader if needed
 	if dl.reader == nil {
 		// spawn a new reader
 		req, err := http.NewRequest("GET", dl.url, nil)
+		if err != nil {
+			return 0, err
+		}
 
 		if off != 0 {
 			req.Header.Set("Range", fmt.Sprintf("bytes=%d-", off))
@@ -147,6 +158,8 @@ func (dl *dlClient) ReadAt(p []byte, off int64) (int, error) {
 	return n, err
 }
 
+// idleTaskRun is called during idle periods to download missing blocks
+// in the background. It runs in a separate goroutine.
 func (dl *dlClient) idleTaskRun() {
 	// this is run in a separate process
 
@@ -207,6 +220,10 @@ func (dl *dlClient) idleTaskRun() {
 
 	// spawn a new reader
 	req, err := http.NewRequest("GET", dl.url, nil)
+	if err != nil {
+		dl.dlm.logf("idle: failed to create request: %s", err)
+		return
+	}
 
 	if off != 0 {
 		req.Header.Set("Range", fmt.Sprintf("bytes=%d-", off))
@@ -250,5 +267,4 @@ func (dl *dlClient) idleTaskRun() {
 		dl.dlm.logf("idle write failed: %s", err)
 		dl.failure = true
 	}
-	return
 }

download.go

@@ -6,6 +6,9 @@ import (
 	"io"
 )
 
+// downloadFull downloads the entire file when partial downloads are not
+// supported by the server. This is a fallback for servers that don't
+// support HTTP Range requests.
 func (f *File) downloadFull() error {
 	// download the file fully
 	// (we are in a lock)
@@ -45,6 +48,9 @@ func (f *File) downloadFull() error {
 	return nil
 }
 
+// needBlocks ensures that all blocks from start to end (inclusive) are
+// downloaded and available locally. It skips blocks that are already
+// downloaded and saves progress after downloading.
 func (f *File) needBlocks(start, end uint32) error {
 	// ensure listed blocks exist and are downloaded
 	// need to be called with lock acquired

factory.go

@@ -10,6 +10,8 @@ import (
 	"github.com/RoaringBitmap/roaring"
 )
 
+// DefaultBlockSize is the size in bytes of each download block (64KB).
+// Downloaded data is tracked and stored in blocks of this size.
 const DefaultBlockSize = 65536
 
 // Open a given URL and return a file pointer that will run partial downloads
@@ -42,6 +44,9 @@ func (dlm *DownloadManager) Open(u string) (*File, error) {
 	return dlm.OpenTo(u, localPath)
 }
 
+// OpenTo opens a given URL and stores downloaded data at the specified local
+// path. If the file already exists with a .part file, the download will resume.
+// If the file exists without a .part file, it is assumed to be complete.
 func (dlm *DownloadManager) OpenTo(u, localPath string) (*File, error) {
 	// generate hash (again if called with Open)
 	hash := sha256.Sum256([]byte(u))

file.go

@@ -7,12 +7,17 @@ import (
 	"github.com/RoaringBitmap/roaring"
 )
 
+// File represents a remote file that can be accessed locally through partial
+// downloads. It implements io.Reader, io.ReaderAt, and io.Seeker interfaces,
+// allowing transparent access to remote HTTP content as if it were a local file.
+// Downloaded data is cached locally in blocks, and only the required portions
+// are fetched on demand. Partial download progress can be persisted to disk
+// and resumed later.
 type File struct {
 	path string // local path on disk
 	url  string // url
 	hash [32]byte
 
-	offset  int64 // offset in url
 	size    int64 // size of url
 	hasSize bool  // is size valid
 	pos     int64 // read position in file