SQLR-43 — Go edge/IoT event collector (concurrent writes showcase) (#150)

A Go HTTP collector that buffers telemetry from many concurrent
producers into a local .sqlrite file while a background uploader
goroutine drains it to a pluggable sink — both writing the same
database via BEGIN CONCURRENT (Phase 11 MVCC). The fifth and final
app under the SQLR-38 example-apps umbrella.

What it exercises:
- Go SDK (cgo via sqlrite-ffi) + the process-level sibling-handle
  registry, so the HTTP path and the uploader each hold their own
  BEGIN CONCURRENT transaction against one engine.
- Row-level conflict detection + the canonical retry-on-ErrBusy loop.
- A durable-buffer-for-unreliable-networks shape: survives reboots,
  stays queryable on-device, optional read-only sqlrite-mcp sidecar.

Honest, measured framing: on the v0 engine, BEGIN CONCURRENT is ~0.9x
the throughput of a single mutex and ~2x worse tail latency (global
engine mutex + per-tx table clone + per-commit O(N) B-tree rebuild —
all documented v0 limitations). The win demonstrated here is the
capability + correctness (independent in-process writers, zero dropped
events under load), not raw speed. The README ships all three measured
tables (throughput / latency-under-contention / disjoint-row) and
explains why, with reproduce commands.

Designs around the verified v0 engine sharp edges, each found by
testing: no Go-SDK param binding (inlined+escaped SQL via a single
chokepoint); CREATE TABLE IF NOT EXISTS not honored + sqlrite_master
not queryable (SELECT-probe for fresh-vs-reopen); CREATE INDEX
rejected under MVCC (DDL before the journal_mode switch); the 4 KiB
MVCC commit-record cap (payloads bounded at ingest so any single row
commits, plus an adaptive checkpoint writer that halves the chunk on a
cap error down to one-per-commit); and AUTOINCREMENT collisions under
MVCC (app-assigned ids seeded off MAX(id)).

Includes: backpressure (503) + /healthz + /stats, a load generator
that asserts no drops, Dockerfile + compose (with the MCP sidecar
profile), a Makefile, unit/integration tests across all packages, and
a go-collector CI job (Linux + macOS) against the in-repo engine.

Also: examples index + sqlritedb.com Examples card + a cross-link from
docs/concurrent-writes.md.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: joaoh82

.github/workflows/ci.yml

@@ -296,6 +296,43 @@ jobs:
         working-directory: sdk/go
         run: go test -v ./...
 
+  # ---------------------------------------------------------------------------
+  # Go edge/IoT collector example (SQLR-43). Builds + tests the example
+  # app against the in-repo libsqlrite_c, so an engine/SDK API change
+  # that breaks the example is caught here. Same {ubuntu, macos} matrix
+  # and cgo wiring as the go-sdk job.
+  go-collector:
+    name: go-collector (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.21'
+          cache-dependency-path: examples/go-collector/go.mod
+
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          shared-key: go-collector-${{ matrix.os }}
+
+      - name: Build libsqlrite_c
+        # The Go driver's `#cgo LDFLAGS` references target/release.
+        run: cargo build --release -p sqlrite-ffi
+
+      - name: go vet ./...
+        working-directory: examples/go-collector
+        run: go vet ./...
+
+      - name: go test ./...
+        working-directory: examples/go-collector
+        run: go test -v ./...
+
   # ---------------------------------------------------------------------------
   # WASM: only the build + size check (no tests — the WASM module's
   # behavior is covered by the other SDKs' test suites; WASM just

docs/concurrent-writes.md

@@ -332,6 +332,7 @@ Decoders accept v1..=v3. A v2 reader on a v3 WAL emits a clean "unsupported WAL
 - [`docs/design-decisions.md`](design-decisions.md) §12a–§12h — the design notes accumulated across Phase 11 sub-phases.
 - [`docs/roadmap.md`](roadmap.md#phase-11--concurrent-writes-via-mvcc--begin-concurrent-sqlr-22-in-flight--see-concurrent-writes-planmd) — phase-by-phase shipped vs deferred status.
 - [`examples/rust/concurrent_writers.rs`](../examples/rust/concurrent_writers.rs) — runnable retry-loop example.
+- [`examples/go-collector/`](../examples/go-collector/) — a Go edge/IoT collector that drives `BEGIN CONCURRENT` from two writers (HTTP path + background uploader) against one file, with **measured** throughput/latency numbers that honestly show the v0 cost of the limitations above (per-tx clone + per-commit rebuild). The README is a good worked example of designing an app around the v0 sharp edges (4 KiB commit cap, no param binding, app-assigned ids under MVCC).
 
 External:
 

examples/README.md

@@ -26,6 +26,7 @@ Beyond the per-SDK quick-start tours above, the [SQLR-38 umbrella](../docs/roadm
 | Chat with your notes (MCP)       | Node.js | Markdown → SQLRite hybrid retrieval, served to Claude Desktop via `sqlrite-mcp --read-only` | [`nodejs-notes/`](nodejs-notes/) |
 | Local-first journaling           | Tauri 2 + Svelte 5 | Markdown daily notes in one `.sqlrite` file, BM25 full-text search, "ask my journal" natural-language SQL | [`desktop-journal/`](desktop-journal/) |
 | Browser SQL playground           | WASM | The full engine in WebAssembly — SQL editor, sample datasets, HNSW vector search, all in a browser tab. Live at [sqlritedb.com/playground](https://sqlritedb.com/playground) | [`wasm-playground/`](wasm-playground/) |
+| Edge / IoT event collector       | Go | HTTP collector + background uploader writing one `.sqlrite` buffer concurrently via `BEGIN CONCURRENT`; durable-buffer-for-unreliable-networks shape, with measured throughput | [`go-collector/`](go-collector/) |
 
 ## Running the Rust quickstart
 

examples/go-collector/.dockerignore

@@ -0,0 +1,10 @@
+# The Docker build context is the repo root (see Dockerfile header).
+# Keep the context lean: the Rust stage rebuilds target/ anyway, and
+# node_modules / git history bloat the upload.
+**/target
+**/node_modules
+.git
+**/*.sqlrite
+**/*.sqlrite-wal
+web/.next
+desktop/src-tauri/target

examples/go-collector/Dockerfile

@@ -0,0 +1,50 @@
+# Multi-stage build for the SQLRite Go edge/IoT collector (SQLR-43).
+#
+# Build context MUST be the repository root, because the Go module
+# links the engine's C library (built from `sqlrite-ffi`) and depends
+# on `sdk/go` via a replace directive:
+#
+#   docker build -f examples/go-collector/Dockerfile -t sqlrite-collector .
+#
+# cgo + cross-compilation note: this image is built for the host's
+# architecture. cgo binaries can't be cross-compiled with the plain Go
+# toolchain, so produce per-arch images with `docker buildx --platform`
+# (the Rust and Go stages both honor the target platform). See the
+# README's "Build & distribution matrix" section.
+
+# ---- Stage 1: build libsqlrite_c.so from the engine source ----
+FROM rust:1-bookworm AS rust
+WORKDIR /src
+COPY . .
+RUN cargo build --release -p sqlrite-ffi
+
+# ---- Stage 2: build the Go binaries against that library ----
+FROM golang:1.22-bookworm AS go
+ENV CGO_ENABLED=1
+WORKDIR /src
+# Bring the whole tree over (we need sdk/go, sqlrite-ffi/include, and
+# the freshly-built target/release/libsqlrite_c.so from the rust stage).
+COPY --from=rust /src /src
+WORKDIR /src/examples/go-collector
+RUN go build -trimpath -o /out/collector ./cmd/collector \
+ && go build -trimpath -o /out/loadgen ./cmd/loadgen
+
+# ---- Stage 3: slim runtime ----
+FROM debian:bookworm-slim
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends ca-certificates \
+ && rm -rf /var/lib/apt/lists/* \
+ && useradd --create-home --uid 10001 collector
+# Install the shared library where the dynamic linker will find it.
+# The Go binary was linked with an rpath pointing at the build tree;
+# that path is absent here, so cgo falls through to the ldconfig cache.
+COPY --from=rust /src/target/release/libsqlrite_c.so /usr/local/lib/
+RUN ldconfig
+COPY --from=go /out/collector /usr/local/bin/collector
+COPY --from=go /out/loadgen /usr/local/bin/loadgen
+USER collector
+WORKDIR /data
+VOLUME ["/data"]
+EXPOSE 8080
+ENTRYPOINT ["collector"]
+CMD ["-db", "/data/events.sqlrite", "-addr", ":8080"]

examples/go-collector/Makefile

@@ -0,0 +1,46 @@
+# Convenience targets for the SQLRite Go edge/IoT collector.
+#
+# Everything depends on the engine's C library. `make lib` builds it
+# from the repo root; the other targets assume it exists.
+
+REPO_ROOT := ../..
+
+.PHONY: lib build test vet run loadtest bench docker clean
+
+## lib: build the SQLRite cdylib (libsqlrite_c) the cgo driver links.
+lib:
+	cd $(REPO_ROOT) && cargo build --release -p sqlrite-ffi
+
+## build: compile the collector + loadgen binaries.
+build: lib
+	go build -o bin/collector ./cmd/collector
+	go build -o bin/loadgen   ./cmd/loadgen
+
+## test: run the Go test suite (needs lib).
+test: lib
+	go test ./... -count=1
+
+## vet: type-check all packages and tests.
+vet: lib
+	go vet ./...
+
+## run: start the collector on :8080 with a local DB file.
+run: build
+	./bin/collector -db events.sqlrite -addr :8080
+
+## loadtest: fire concurrent producers at a running collector.
+loadtest: build
+	./bin/loadgen -target http://localhost:8080 -workers 64 -duration 30s
+
+## bench: in-process throughput matrix (concurrent vs serialized, ±index).
+bench: build
+	./bin/loadgen -bench -workers 32 -duration 30s
+
+## docker: build the container image (context = repo root).
+docker:
+	cd $(REPO_ROOT) && docker build -f examples/go-collector/Dockerfile -t sqlrite-collector .
+
+## clean: remove build artifacts and local DB files.
+clean:
+	rm -rf bin
+	rm -f events.sqlrite events.sqlrite-wal