Skip to content

Commit 8bccbeb

Browse files
committed
docs(go): flesh out store + io + process landing pages
Replaces the "page in flight" stubs on the three core/go packages that got the Result API + Windows compile path treatment this week. go/store - Two-layer story: SQLite KV + DuckDB workspace - Service Register pattern with Result-returning constructors - Scoped namespaces + quota config - Canonical scoring tables + Parquet export - Windows compile status (CGO required, mingw or wails-cross) go/io - Full Medium interface listing with every method - Three backends: sandboxed local, in-memory, mock - io.Copy example for backend-to-backend transfer - Service registration + io/* action dispatch - Windows compile note — core.Lstat + portable linkInfo go/process - One-shot exec via Start / StartWithOptions - Long-lived Daemon with restart policy + log ring buffer - HealthServer for HTTP introspection - Service + Registry pattern for multi-process apps - Cross-platform helper triplet table: processSignal, processKillGroup, processSignalGroup, applyProcessGroup, exitWasSignaled — Unix vs Windows behaviour mapped explicitly Cross-links the three pages to each other + to result/ so the navigation flows naturally from primitive to package to consumer. Co-Authored-By: Virgil <virgil@lethean.io>
1 parent ca1cb80 commit 8bccbeb

3 files changed

Lines changed: 305 additions & 12 deletions

File tree

src/content/docs/go/io.mdx

Lines changed: 94 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,10 @@ head:
1212
content: 'dappco.re/go/io https://github.com/dappcore/go-io https://github.com/dappcore/go-io/tree/main{/dir} https://github.com/dappcore/go-io/blob/main{/dir}/{file}#L{line}'
1313
---
1414

15-
Universal IO — Medium transport, datanode, sigil, sqlite, S3. Built on the [`core/go`](/go/) primitives.
15+
One file-system interface, several backends. The `Medium` contract is what
16+
every other [`core/go`](/go/) package speaks when it needs to read, write,
17+
list, or watch a path — and the path is the same regardless of whether the
18+
storage is local disk, an in-memory mock, or a sandboxed root.
1619

1720
## Install
1821

@@ -26,7 +29,94 @@ go get dappco.re/go/io@latest
2629
import "dappco.re/go/io"
2730
```
2831

29-
## Status
32+
## The Medium contract
3033

31-
Page in flight — content fills in as the package converges. Source of truth
32-
today is the README and inline docstrings in the repo.
34+
```go
35+
type Medium interface {
36+
Read(path string) (string, error)
37+
Write(path, content string) error
38+
WriteMode(path, content string, mode fs.FileMode) error
39+
EnsureDir(path string) error
40+
IsFile(path string) bool
41+
Delete(path string) error
42+
DeleteAll(path string) error
43+
Rename(oldPath, newPath string) error
44+
List(path string) ([]fs.DirEntry, error)
45+
Stat(path string) (fs.FileInfo, error)
46+
Open(path string) (fs.File, error)
47+
Create(path string) (io.WriteCloser, error)
48+
}
49+
```
50+
51+
Every method takes a string path. Permissions, locking, encoding choices —
52+
all live behind the backend. A consumer writes against `Medium` once and
53+
swaps backends at construction without code changes.
54+
55+
## Backends
56+
57+
| Backend | Construct | Use case |
58+
|---|---|---|
59+
| **Sandboxed local** | `io.NewSandboxed(root)` | Disk-backed with a chroot — paths can never escape `root` |
60+
| **In-memory** | `io.NewMemoryMedium()` | Ephemeral state, fixtures, fast unit tests |
61+
| **Mock** | `io.NewMockMedium()` | Tests that need to assert specific call patterns |
62+
63+
```go
64+
disk, r := io.NewSandboxed("/srv/app")
65+
if !r.OK { return r }
66+
67+
_ = disk.Write("config/app.yaml", "port: 8080")
68+
content, _ := disk.Read("config/app.yaml")
69+
70+
// Same code, different backend
71+
mem := io.NewMemoryMedium()
72+
_ = mem.Write("config/app.yaml", "port: 8080")
73+
```
74+
75+
## Copy across backends
76+
77+
`io.Copy` moves payloads between any two Mediums with the same semantics
78+
regardless of where the bytes physically live:
79+
80+
```go
81+
local, _ := io.NewSandboxed("/srv/app")
82+
backup, _ := io.NewSandboxed("/srv/backup")
83+
84+
_ = io.Copy(local, "data/report.json", backup, "daily/report.json")
85+
```
86+
87+
A unit test can mount a `MemoryMedium` in place of either side and the
88+
production code path stays unchanged.
89+
90+
## Service registration
91+
92+
The package follows the `core/go` Service pattern — register once on a
93+
`*core.Core` and every consumer with the Core handle can dispatch IO
94+
actions through `c.Action("io/...")`:
95+
96+
```go
97+
c := core.New(core.Options{})
98+
if r := io.Register(c); !r.OK { return r }
99+
100+
io.RegisterActions(c) // Adds io/read, io/write, io/list, etc.
101+
```
102+
103+
## Cross-platform compile
104+
105+
The local backend is portable across Linux, macOS, and Windows. Symbolic
106+
link handling — historically Unix-only via `syscall.Stat_t` — is now
107+
abstracted behind `core.Lstat` plus a portable `linkInfo` helper so the
108+
same file builds clean for `GOOS=windows GOARCH=amd64 CGO_ENABLED=0`.
109+
See `medium_link.go` for the platform-neutral implementation that ships
110+
the Windows compile path.
111+
112+
## Sibling packages
113+
114+
- [`go/store`](/go/store/) — KV + workspace persistence; uses Medium when
115+
the underlying store lives off-disk
116+
- [`go/process`](/go/process/) — process supervisor that watches files
117+
through Medium
118+
- [`go/scm`](/go/scm/) — repo-sync actions that fan out across Mediums
119+
120+
## Source
121+
122+
[`github.com/dappcore/go-io`](https://github.com/dappcore/go-io) — full source, issues, and releases.

src/content/docs/go/process.mdx

Lines changed: 109 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,10 @@ head:
1212
content: 'dappco.re/go/process https://github.com/dappcore/go-process https://github.com/dappcore/go-process/tree/main{/dir} https://github.com/dappcore/go-process/blob/main{/dir}/{file}#L{line}'
1313
---
1414

15-
Process orchestration — exec, daemon, lifecycle, status. Built on the [`core/go`](/go/) primitives.
15+
Process supervisor primitives — start, stop, signal, wait, restart. The
16+
same call shape that runs a one-shot exec also drives long-lived daemons
17+
with health endpoints. Built on [`core/go`](/go/) so every operation
18+
returns a [`Result`](/go/result/).
1619

1720
## Install
1821

@@ -26,7 +29,109 @@ go get dappco.re/go/process@latest
2629
import "dappco.re/go/process"
2730
```
2831

29-
## Status
32+
## One-shot exec
3033

31-
Page in flight — content fills in as the package converges. Source of truth
32-
today is the README and inline docstrings in the repo.
34+
```go
35+
r := process.Start(ctx, "git", "rev-parse", "HEAD")
36+
if !r.OK { return r }
37+
sha := r.Value.(string)
38+
```
39+
40+
For more control — environment, working directory, stdout/stderr capture
41+
— pass `RunOptions`:
42+
43+
```go
44+
r := process.StartWithOptions(ctx, process.RunOptions{
45+
Command: "ffmpeg",
46+
Args: []string{"-i", "in.mov", "out.mp4"},
47+
Env: map[string]string{"PATH": "/usr/local/bin:/usr/bin"},
48+
Dir: "/tmp/encode",
49+
Capture: true,
50+
})
51+
if !r.OK { return r }
52+
out := r.Value.(*process.ExecResult)
53+
fmt.Println(out.Stdout)
54+
```
55+
56+
## Long-lived daemon
57+
58+
`Daemon` wraps a command with a supervisor loop — restart on exit, ring
59+
buffer for the last N stdout/stderr lines, optional health endpoint:
60+
61+
```go
62+
d := process.NewDaemon(process.DaemonOptions{
63+
Name: "indexer",
64+
Command: "/usr/local/bin/indexer",
65+
Args: []string{"--watch", "/srv/data"},
66+
RestartPolicy: process.RestartAlways,
67+
RestartBackoff: 5 * time.Second,
68+
LogBufferSize: 1024,
69+
})
70+
71+
if r := d.Start(ctx); !r.OK { return r }
72+
defer d.Stop(ctx)
73+
74+
// Tail the recent logs at any time
75+
lines := d.RecentLogs(50)
76+
```
77+
78+
For HTTP introspection:
79+
80+
```go
81+
hs := process.NewHealthServer(":7100")
82+
hs.Attach(d)
83+
go hs.Listen(ctx)
84+
85+
// GET /health → JSON daemon status
86+
// GET /logs → recent ring-buffer lines
87+
```
88+
89+
## Service + Registry
90+
91+
A `Registry` tracks every Daemon a Core registered so multi-process
92+
applications get a single view of who's alive. The Service factory plugs
93+
the registry directly into the Core lifecycle:
94+
95+
```go
96+
c := core.New(core.Options{})
97+
98+
svc := process.NewService(process.Options{
99+
Dir: "/var/run/myapp",
100+
})
101+
102+
if r := svc(c); !r.OK { return r }
103+
104+
// Discover registered daemons
105+
running := process.NewRegistry("/var/run/myapp").List()
106+
```
107+
108+
## Cross-platform compile
109+
110+
Unix-only `syscall.Kill`, `syscall.SIGKILL`, `SysProcAttr{Setpgid: true}`,
111+
and `WaitStatus.Signaled()` have all moved behind a small platform-helper
112+
triplet — `platform.go` declares the shared signatures, `platform_unix.go`
113+
keeps the original POSIX semantics, and `platform_windows.go` provides
114+
the equivalent or best-effort behaviour:
115+
116+
| Helper | Unix | Windows |
117+
|---|---|---|
118+
| `processSignal(pid, sig)` | `syscall.Kill(pid, sig)` | `os.FindProcess(pid).Signal(sig)` |
119+
| `processKillGroup(pid)` | `syscall.Kill(-pid, SIGKILL)` | best-effort leader-only kill |
120+
| `processSignalGroup(pid, sig)` | `syscall.Kill(-pid, sig)` | `Ok(nil)` stub — no Win32 equivalent |
121+
| `applyProcessGroup(cmd)` | `SysProcAttr{Setpgid: true}` | no-op |
122+
| `exitWasSignaled(state)` | `WaitStatus.Signaled()` | `false` |
123+
124+
The Service compiles clean for `GOOS=windows GOARCH=amd64` and the smoke
125+
tests cover both targets. Full process-group semantics on Windows (Job
126+
Objects) are Phase 2; today's stubs are sufficient for tray-lifecycle
127+
and supervisor-restart cases.
128+
129+
## Sibling packages
130+
131+
- [`go/io`](/go/io/) — file-system Medium for daemon log persistence
132+
- [`go/store`](/go/store/) — persisted daemon registry state
133+
- [`go/log`](/go/log/) — structured logging that daemons emit through
134+
135+
## Source
136+
137+
[`github.com/dappcore/go-process`](https://github.com/dappcore/go-process) — full source, issues, and releases.

src/content/docs/go/store.mdx

Lines changed: 102 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,10 @@ head:
1212
content: 'dappco.re/go/store https://github.com/dappcore/go-store https://github.com/dappcore/go-store/tree/main{/dir} https://github.com/dappcore/go-store/blob/main{/dir}/{file}#L{line}'
1313
---
1414

15-
Persistent KV — SQLite + DuckDB, workspace buffer. Built on the [`core/go`](/go/) primitives.
15+
Two-layer persistence built on the [`core/go`](/go/) primitives. SQLite for
16+
durable key-value state, DuckDB for analytical and workspace queries — both
17+
fronted by a single Service so callers get one entry point and one
18+
[`Result`](/go/result/) shape.
1619

1720
## Install
1821

@@ -26,7 +29,102 @@ go get dappco.re/go/store@latest
2629
import "dappco.re/go/store"
2730
```
2831

29-
## Status
32+
## Two backends, one Service
3033

31-
Page in flight — content fills in as the package converges. Source of truth
32-
today is the README and inline docstrings in the repo.
34+
| Layer | Backed by | Best for |
35+
|--------|-----------|----------|
36+
| **SQLite KV** | `modernc.org/sqlite` (pure Go) | Durable per-key state — config, sessions, cache, scoped namespaces with quotas |
37+
| **DuckDB workspace** | `marcboeker/go-duckdb` (CGO) | Analytical reads, Parquet import/export, scoring tables, ad-hoc joins over local data |
38+
39+
The Service Register pattern is the canonical entry — register once on a
40+
`*core.Core` and the store is available through `c.Action("store/...")`.
41+
42+
```go
43+
c := core.New(core.Options{})
44+
if r := store.Register(c); !r.OK { return r }
45+
46+
// Now any code with the *core.Core can reach the store.
47+
```
48+
49+
For per-call construction, the constructors all return `core.Result`:
50+
51+
```go
52+
// SQLite KV at ~/.local/share/app/state.db
53+
r := store.New("~/.local/share/app/state.db")
54+
if !r.OK { return r }
55+
db := r.Value.(*store.Store)
56+
57+
// DuckDB analytics workspace
58+
r := store.OpenDuckDB("workspace.duckdb")
59+
if !r.OK { return r }
60+
ws := r.Value.(*store.DuckDB)
61+
```
62+
63+
## Scoped namespaces
64+
65+
Cluster a slice of the KV under a namespace so one application can hand
66+
sub-namespaces to its modules without leaking keys:
67+
68+
```go
69+
sessions := store.NewScoped(db, "sessions")
70+
sessions.Set("user-42", payload) // stored at sessions/user-42 internally
71+
72+
// With quota
73+
sub, r := store.NewScopedWithQuota(db, "uploads", store.QuotaConfig{
74+
MaxKeys: 1000,
75+
MaxBytes: 50 << 20, // 50 MiB
76+
})
77+
if !r.OK { return r }
78+
```
79+
80+
## Analytical layer — DuckDB
81+
82+
The DuckDB side ships canonical table names + helpers for the most common
83+
patterns:
84+
85+
```go
86+
const (
87+
TableCheckpointScores = "checkpoint_scores"
88+
TableProbeResults = "probe_results"
89+
)
90+
91+
_ = ws.EnsureScoringTables()
92+
ws.Exec(core.Sprintf("SELECT * FROM %s WHERE score > ?", TableCheckpointScores), 0.8)
93+
```
94+
95+
Parquet round-trip helpers cover the most common ETL shapes:
96+
97+
```go
98+
r := store.ExportParquet(ws, "scores.parquet", "checkpoint_scores")
99+
if !r.OK { return r }
100+
rows := r.Value.(int)
101+
102+
// Sharded export — splits by column value, writes one Parquet per shard.
103+
store.ExportSplitParquet(ws, "shards/", "probe_results", "model_id")
104+
```
105+
106+
## Cross-platform compile
107+
108+
SQLite is pure Go and compiles on every target with no special flags. The
109+
DuckDB layer is CGO and follows the same pattern as the rest of the
110+
[`core/go`](/go/) ecosystem — Windows support is live for `amd64` with a
111+
C toolchain (native `mingw` or the Wails `wails-cross` Docker image with
112+
Zig). The `windows/arm64` target requires the same toolchain via
113+
`CGO_ENABLED=1`.
114+
115+
`marcboeker/go-duckdb` is the active driver path; `mapping`/`bindings`
116+
sub-modules are kept transitively. See [RFC.md](https://github.com/dappcore/go-store/blob/dev/RFC.md)
117+
for the current Windows compile checklist and consumer-side build flags.
118+
119+
## Sibling packages
120+
121+
- [`go/io`](/go/io/) — universal `Medium` transport that store consumes
122+
when persistence lives on remote backends (S3, cube)
123+
- [`go/process`](/go/process/) — supervisor primitives that own the
124+
process running the store
125+
- [`go/orm`](/go/orm/) — the typed query layer that fronts both backends
126+
when consumers want models instead of raw SQL
127+
128+
## Source
129+
130+
[`github.com/dappcore/go-store`](https://github.com/dappcore/go-store) — full source, issues, and releases.

0 commit comments

Comments
 (0)