feat(web-ui): backend image-composition service#737
Conversation
Add internal/api package and `serve` command backing the Basic tab: - GET /manifest — serves combinations + dropdown labels (embedded manifest.yaml seeded with Retail Edge / Robotics / Fed Aero demo combinations) - POST /templates/compose — manifest lookup → template YAML + summary (summary computed via config.LoadAndMergeTemplate) - Build-path handlers (POST /builds via os/exec, SSE logs, artifacts) wired and compiling; end-to-end build verification pending on a build-capable host - Vendor generic-handheld-os-template.yml from the edge-node-infrastructure-blueprint repo so the Fed Aero combination is buildable locally Read path verified via curl: manifest lists 3 combinations; compose resolves the robotics template (41 packages, 2 partitions) and returns 400 on no-match.
ICT builds require root (chroot/mounts) and are run with sudo on build hosts. - Add --sudo serve flag: builds invoke `sudo -n <ict> build ...` (non-interactive so the server never blocks on a password prompt; needs a passwordless sudoers rule for the ICT binary) - Add --work-dir serve flag (default webui-workspace) for per-build isolated work/output dirs, avoiding the root-owned ./workspace - Build still runs from repo root so ICT resolves config/osv/... correctly Read path and build-path plumbing (spawn, live SSE log streaming, status tracking, artifacts) verified via curl. A successful root build to completion must be verified on a host with (passwordless) sudo.
Expand the --sudo flag help to instruct operators to grant a passwordless sudoers rule scoped to the ICT binary only, not blanket sudo. Build args are server-controlled (manifest-derived template path + generated work dir) and passed as argv (no shell), so the scoped rule has no user-controlled input.
Builds run as root under sudo and write outputs into root-owned subdirs, so the non-root server's filesystem scan couldn't traverse them (artifacts came back null despite a successful build). Parse the authoritative artifact list from ICT's own "Generated Artifacts" output block instead — name, absolute path, and type (image/sbom) — with the work-dir scan kept as a fallback. This also catches outputs the old matcher missed (.vhdx, spdx_manifest_*.json). Also serialize an empty artifact list as [] rather than null in the SSE complete event. Add a parser unit test built from real build output.
extractPath took the first "/" in the line, which matched the logger's "display/display.go:80" source prefix instead of the real path — artifact paths came back as "/display.go:80\t/home/...". Take the last tab-separated field (the log message) and require a leading "/" instead. Strengthen the parser test to assert paths are clean absolute paths (no logger prefix, no tabs) ending in the artifact name — the old behavior now fails it.
The manifest is compiled into the binary via //go:embed, so edits required a rebuild to take effect. Add a --manifest flag: when set, the server reads the manifest file from disk at startup (live-editable, restart to apply); when empty it falls back to the embedded copy, preserving the single-binary default.
- Fix data race: SSE and artifacts handlers read build status/artifacts/errMsg without the mutex while runBuild writes them. Introduce a locked snapshot() returning an immutable result, and a single locked finish() for terminal state; readers use snapshot(). Set-once fields documented as lock-free. - Handle the previously-ignored cmd.StdoutPipe() error (errcheck). Verified: go build, go vet, go test -race all clean.
Map the fed-aero combination to ubuntu24-x86_64-minimal-raw.yml (a template that exists) until a purpose-built Fed Aero template is available.
gofmt normalizes the parseArtifacts doc-comment example (bullet+tab was reformatted to a list item). Apply gofmt to satisfy the Go Lint gate; the parser behavior is unchanged (it still matches the • bullet in ICT output).
Add targeted unit tests to clear the CI coverage gate: manifest load/lookup, manifest + compose + artifacts handlers (success and error paths), build tracker + snapshot/finish state, buildCommand + template resolution, middleware (CORS preflight, panic recovery, writeJSON/writeError), SSE event formatting + log streaming for a finished build, server New/routes, and discoverArtifacts. Raises internal/api coverage ~10%% -> ~78%%.
golangci-lint (errcheck) also lints _test.go files. Check the previously unchecked os.WriteFile/os.MkdirAll/json.Unmarshal returns in api_test.go.
There was a problem hiding this comment.
Pull request overview
This PR introduces a new internal/api HTTP backend and a image-composer-tool serve subcommand to expose key image-composition workflows (manifest discovery, template composition, build execution, SSE log streaming, and artifact reporting) over a REST API, aligning with the Web UI contract established in #727.
Changes:
- Adds an
internal/apiserver with endpoints for manifest, compose, builds, SSE logs, and artifacts. - Executes end-to-end ICT builds per request (optionally via
sudo -n) and tracks per-build state/logs in-memory. - Adds unit tests for manifest parsing, compose/build handlers, SSE formatting, and artifact parsing/discovery; bumps coverage threshold slightly.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/api/sse.go | SSE log streaming endpoint + event formatting helper |
| internal/api/server.go | Server config, construction, and HTTP server startup |
| internal/api/router.go | Route registration on Go ServeMux patterns |
| internal/api/middleware.go | CORS, request logging, panic recovery, JSON/error helpers |
| internal/api/manifest.go | Manifest model, embedded loading, template lookup |
| internal/api/handlers_read.go | GET /manifest and POST /templates/compose handlers |
| internal/api/handlers_artifacts.go | GET /builds/{id}/artifacts handler |
| internal/api/data/manifest.yaml | Embedded manifest mapping UI selections to templates |
| internal/api/builds.go | Build lifecycle tracking, command execution, artifact parsing/discovery |
| internal/api/builds_test.go | Tests for artifact parsing from real-ish log output |
| internal/api/api_test.go | Broad handler/unit tests for manifest/compose/build state/middleware/SSE |
| image-templates/generic-handheld-os-template.yml | Adds a new (very large) Ubuntu-based template |
| cmd/image-composer-tool/serve.go | New serve subcommand wiring CLI flags to API server config |
| cmd/image-composer-tool/main.go | Registers the new serve subcommand |
| .coverage-threshold | Increases repo coverage threshold from 68.4 to 68.5 |
Security: - Path-traversal guard: safeTemplatePath() rejects absolute paths and ../ traversal so a bad/edited manifest can't read files outside TemplatesDir (compose + build resolution). - Bind 127.0.0.1 by default via new serve --host flag; 0.0.0.0 is opt-in (this API can trigger privileged builds). - CORS restricted to localhost origins (echoes Origin) instead of "*". - Per-build work dir created 0700 (was 0755); inline template written 0600. - Remove baked password hash from generic-handheld template; use empty password + placeholder comment per repo convention. Correctness: - compose returns 422 when the matched template fails to load/validate instead of a misleading 200 with a zero summary. - resolveBuildTemplate distinguishes client (400) from server (500) errors via errBadBuildRequest sentinel. - runBuild checks scanner.Err() so a truncated log stream fails the build rather than being marked successful. - findTemplate: an omitted SKU resolves only when unambiguous (no order-dependent guess across SKUs). Docs: - Manifest spelling Blue Print -> Blueprint. - Comment why os/exec is used directly rather than internal/utils/shell (need to stream stdout + hold the process for cancellation). Tests: CORS localhost behavior, safeTemplatePath traversal rejection, ambiguous-SKU resolution, compose invalid-template 422. Coverage ~78.6%.
Point the fed-aero combination at generic-handheld-os-template.yml (the vendored handheld blueprint) instead of the generic ubuntu24-x86_64-minimal-raw.yml, so the combination resolves to its intended purpose-built template.
|
|
||
| func executeServe(cmd *cobra.Command, args []string) error { | ||
| srv, err := api.New(api.Config{ | ||
| Addr: serveHost + ":" + servePort, |
There was a problem hiding this comment.
This concatenation will not work for IPV6 hosts.
| b.finish(statusFailed, nil, err.Error()) | ||
| return | ||
| } | ||
| cmd.Stderr = cmd.Stdout // merge streams |
There was a problem hiding this comment.
use more cleaner approach of merging the stdout and stderr streams. There is a potential race condition to access the same buffer by both the streams but with different priorities. preferred is cmd.CombinedOutput.
Merge Checklist
All boxes should be checked before merging the PR
Description
Adds an
internal/apipackage and animage-composer-tool servesubcommand that drives a real ICT image build end-to-end over REST, so a client can compose and build without the CLI. Implements the read + build path of the OpenAPI contract added in #727 (merged): manifest, compose, build, SSE logs, artifacts.Endpoints:
GET /api/v1/manifest-0 supported combinations + display labels (from an embedded,--manifest-overridable manifest.yaml)POST /api/v1/templates/compose- selection → resolved pre-authored template + summary (viaconfig.LoadAndMergeTemplate)POST /api/v1/builds- build via ICT invocation (sudo -nscoped rule; ICT needs root for chroot/mounts); per-build isolated--work-dirGET /api/v1/builds/{id}/logs- live build logs over SSE; terminalcomplete/erroreventGET /api/v1/builds/{id}/artifacts- image + SBOM paths, parsed from ICT outputNotable: build state is mutex-guarded (race-free); artifacts parsed from ICT's authoritative output (immune to root-owned output dirs); localhost bind by default.
Scope note: this is the backend only. The React Basic-mode UI will be in next story (separate PR). The OpenAPI spec it implements landed in #727.
Any Newly Introduced Dependencies
None. Reuses existing deps (
google/uuid,cobra,zap,yaml) and internal packages (config.LoadAndMergeTemplate).How Has This Been Tested?
Automated:
go build ./...andgo vet ./internal/api/...clean;gofmtclean.go test -race ./internal/api/...clean;internal/apiunit-test coverage ~78% (manifest, compose/manifest/artifacts handlers, build tracker + state, middleware, SSE).Manual E2E on a build-capable host (curl):
Prerequisites
Start the server (leave running)
1. Manifest: supported combinations + labels
2. Compose: selection → resolved template + summary
3. Start a build: returns a build ID
4. Stream logs (SSE; blocks until done, minutes for a real build)
5. Artifacts: image + SBOM paths
6.
--manifestlive-edit (no rebuild): editinternal/api/data/manifest.yaml, restart withserve --manifest internal/api/data/manifest.yaml, changes apply.Prerequisite recap: builds require the scoped, passwordless sudoers rule shown above; without it
POST /buildsfails at the sudo step.