Skip to content

artifact-system.md

Latest commit

 

History

History
206 lines (148 loc) · 10.2 KB

artifact-system.md

File metadata and controls

206 lines (148 loc) · 10.2 KB
Raw

Artifact System

Go sidecar service managing per-business bare Git repos for versioned artifact storage. Multiple Ghost agents for the same business all commit to one shared repo. BizApp reads history, diffs, and file trees via the Artifact System HTTP API.

Architecture

Ghost Engine (pure control plane)
        |
        | HTTP tool calls
        |
Toolbox-Devtools (repo-git: clone/commit/push via file://)
        |
        | file:// protocol
        |
Shared PVC (/mnt/repos/)
        |
Artifact System Sidecar (Go, :2369)
        |
        | HTTP API
        |
BizApp Backend → Frontend

Three containers participate:

Container /mnt/workspace /mnt/repos Role
Ghost -- -- Pure control plane; delegates all workspace/repo ops to devtools via HTTP
Toolbox-Devtools rw (PVC) rw (PVC) Runs repo-git and coding agents; owns workspace and repo access
Artifact System -- rw (PVC) Manages bare repos, serves read API, optional GCS backup

The devtools toolbox handles the full git lifecycle: clone, edit, commit, push. Ghost never touches /mnt/workspace or /mnt/repos directly — it sends tool calls to the toolbox-server over HTTP.

Repository Storage

Bare repos live on a PVC mounted at /mnt/repos/, one directory per business:

/mnt/repos/
├── templates/
│   └── default-business.git/
├── hysp/                  ← business_slug = "hysp"
├── acme-corp/
├── megastore/
└── ...

repo_id = business_slug. Multiple agents for the same business all clone and push to /mnt/repos/{repo_id}/. Each agent's working copy is at /mnt/workspace/{agent_id}/artifact-repo.

Bare Repo Permissions

Bare repos are created by the artifact-system (running as root) but pushed to by per-agent sandbox users in the devtools container. Both containers share the agents group with a fixed GID of 1500.

When the artifact-system creates or clones a repo, it:

  1. Sets core.sharedRepository = group in the bare repo's git config — this tells git to create new objects with group-write permissions
  2. Recursively sets group ownership to agents (GID 1500) and group-write + setgid permissions on all directories

On startup, the artifact-system also fixes permissions on all existing repos to handle repos created before the permission model was introduced.

The matching agents group exists in both Dockerfiles:

  • toolbox/artifact-server/Dockerfile: groupadd --gid 1500 agents
  • toolbox/devtools/Dockerfile: groupadd --gid 1500 agents

Per-agent users created by the toolbox-server belong to the agents group, so they can push objects into any bare repo.

Scalability

  • 10k repos (~5 MB avg): ~50 GB
  • 100k repos: ~500 GB (SSD storage class recommended)

Repository File Tree

Each business repo follows a template-defined structure:

site/
├── header.md
├── homepage.md
├── about.md
├── footer.md
├── logo.png
└── favicon.png
branding/
├── guidelines.md
└── colors.json
legal/
├── privacy_policy.md
└── terms_of_service.md
metadata/
└── business.yaml

Template System

Templates live in architect/templates/. On startup the Artifact System commits each template directory into a bare repo at /mnt/repos/templates/{name}.git. New business repos are cloned from the template.

Git Workflow

  1. Init: BizApp calls POST /api/v1/repos with repo_id (= business_slug) → Artifact System clones template into /mnt/repos/{repo_id}/.
  2. Checkout: Agent calls repo-git checkout <repo_id> → clones file:///mnt/repos/{repo_id}/ into $AGENT_WORKSPACE/artifact-repo (i.e. /workspace/artifact-repo inside the sandbox). Multiple agents for the same business all clone the same bare repo.
  3. Work: Agent delegates to cursor-agent or claude-agent pointing at the working copy. Tools write files normally.
  4. Commit: Agent calls repo-git commit-push --message "..." --author "..." → stages all, commits, pushes. If another agent pushed first, it rebases before retrying.
  5. Read: BizApp queries the Artifact System API for tree, log, diff, file content.

repo-git Tool

repo-git is a bash script at toolbox/devtools/bin/repo-git that wraps git operations for the sandboxed environment. It runs inside the devtools sandbox under the per-agent user.

Command Description
repo-git checkout <repo_id> Clone or pull bare repo into $AGENT_WORKSPACE/artifact-repo
repo-git commit-push --message <msg> --author <name> Stage all, commit, push (rebase on conflict)
repo-git status Show working copy status
repo-git init <repo_id> [--template <name>] Create repo via artifact API (rarely used by agents)

Git in the sandbox

Agent workspaces and bare repos are on PVCs. Bare repos at /mnt/repos/ are owned by root but the sandbox runs commands under per-agent UIDs, triggering git's safe.directory ownership check. Pushing also requires write access to the bare repo's objects/ directory, handled by the agents group permissions described in Bare Repo Permissions.

Three mitigations work together to make git reliable in the sandbox:

1. GIT_CONFIG_COUNT environment variables (sandboxEnv)

The sandbox sets core.fileMode=false and core.sharedRepository=umask via GIT_CONFIG_COUNT environment variables. This prevents git from attempting to write core.filemode to .git/config during clone — the env value already matches what git detects, so the config write (and its lockfile fchmod) is skipped entirely.

2. safe.directory in system gitconfig (Dockerfile)

The Dockerfile runs git config --system --add safe.directory '*' to write the setting into /etc/gitconfig. Git intentionally ignores safe.directory from environment variables and -c flags for security reasons — it can only be set in system or global gitconfig. This allows per-agent users to access bare repos owned by root at /mnt/repos/.

3. Group-writable bare repos (artifact-system)

Bare repos are created by the artifact-system as root, but agents push as per-agent UIDs. The artifact-system sets core.sharedRepository=group and makes repos group-writable for the agents group (GID 1500). Without this, git push fails with remote unpack failed: unable to create temporary object directory.

Testing

The test script at scripts/tests/test-repo-git.sh validates the full repo lifecycle — create repo via artifact API, checkout, mutate, commit-push, verify push — against running toolbox-devtools and artifact-system containers. The toolbox --remote URL and artifact --artifact-api URL must point at containers that share the same /mnt/repos volume (e.g. make test-tool passes --remote http://localhost:9091 with --artifact-api http://localhost:2370 for docker/ci-tool-test.yml).

scripts/tests/test-repo-reseed.sh checks that the bare template templates/default-business.git matches the source tree under architect/templates/default-business (use RESEED_TEMPLATES=true so startup rebuilds the template from the mount). Run via make test-tool after docker/ci-tool-test.yml is up.

Agent ↔ Repo Mapping

Agents learn which repo_id to use via shared knowledge (injected by BizApp when creating the agent). The Artifact System itself is repo_id-agnostic — it just stores and serves bare repos.

HTTP API

Base URL: http://artifact-system:2369/api/v1

Lifecycle

Method Path Description
POST /repos Init repo from template (repo_id + optional template)
DELETE /repos/{repoID} Delete repo
GET /repos/{repoID}/status HEAD sha, last commit date, file count

Read

Method Path Description
GET /repos/{repoID}/tree File tree at ref (default HEAD)
GET /repos/{repoID}/files/{path} Raw file content
GET /repos/{repoID}/log Commit history (paginated)
GET /repos/{repoID}/diff Diff between two commits
GET /repos/{repoID}/commits/{sha} Single commit detail

Write

Method Path Description
POST /repos/{repoID}/rollback Revert to a target commit
POST /repos/{repoID}/commit Programmatic commit (for UI edits)

Configuration

Env var Default Description
REPOS_DIR /mnt/repos Base directory for bare repos
TEMPLATES_DIR /app/templates Source template files
RESEED_TEMPLATES (off) If true/1/yes/on, on each startup remove each templates/{name}.git under REPOS_DIR and recreate it from TEMPLATES_DIR/{name} so the bare template tracks the mounted source (local dev or GCS). Default keeps the one-time seed behavior.
LISTEN_ADDR :2369 HTTP listen address
GCS_BACKUP_BUCKET (disabled) GCS bucket for periodic backups
GCS_BACKUP_INTERVAL 1h Backup frequency

Alternative: External GitHub Repos

For working with external GitHub repositories (public or private), the github-repo toolbox tool clones repos directly from GitHub into $AGENT_WORKSPACE/repos/<name>/ over HTTPS or SSH. This is an alternative to the artifact system for use cases like open-source contributions, multi-repo projects, or any code not hosted on the platform. Both repo-git (artifact) and github-repo (external) can be enabled simultaneously via the archetype config.md tools list. See Tools — GitHub Repos for details.

Related Docs

Code Layout

cmd/artifact-server/main.go
internal/artifact/
├── server.go      # HTTP server + router
├── handler.go     # API handlers
├── repo.go        # Git operations (go-git)
├── storage.go     # Flat directory management (per repo_id)
├── template.go    # Template loading + repo init
├── gitops.go      # Ghost working-copy helpers (clone/push via git CLI)
└── backup.go      # Optional GCS backup