Skip to content

feat(web-ui): backend image-composition service#737

Open
arodage wants to merge 15 commits into
mainfrom
feat/web-ui-backend-service
Open

feat(web-ui): backend image-composition service#737
arodage wants to merge 15 commits into
mainfrom
feat/web-ui-backend-service

Conversation

@arodage

@arodage arodage commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Merge Checklist

All boxes should be checked before merging the PR

  • The changes in the PR have been built and tested
  • Documentation has been updated to reflect the changes (or no doc update needed)
  • Ready to merge

Description

Adds an internal/api package and an image-composer-tool serve subcommand that drives a real ICT image build end-to-end over REST, so a client can compose and build without the CLI. Implements the read + build path of the OpenAPI contract added in #727 (merged): manifest, compose, build, SSE logs, artifacts.

Endpoints:

  • GET /api/v1/manifest -0 supported combinations + display labels (from an embedded, --manifest-overridable manifest.yaml)
  • POST /api/v1/templates/compose - selection → resolved pre-authored template + summary (via config.LoadAndMergeTemplate)
  • POST /api/v1/builds - build via ICT invocation (sudo -n scoped rule; ICT needs root for chroot/mounts); per-build isolated --work-dir
  • GET /api/v1/builds/{id}/logs - live build logs over SSE; terminal complete/error event
  • GET /api/v1/builds/{id}/artifacts - image + SBOM paths, parsed from ICT output

Notable: build state is mutex-guarded (race-free); artifacts parsed from ICT's authoritative output (immune to root-owned output dirs); localhost bind by default.

Scope note: this is the backend only. The React Basic-mode UI will be in next story (separate PR). The OpenAPI spec it implements landed in #727.

Any Newly Introduced Dependencies

None. Reuses existing deps (google/uuid, cobra, zap, yaml) and internal packages (config.LoadAndMergeTemplate).

How Has This Been Tested?

Automated:

  • go build ./... and go vet ./internal/api/... clean; gofmt clean.
  • go test -race ./internal/api/... clean; internal/api unit-test coverage ~78% (manifest, compose/manifest/artifacts handlers, build tracker + state, middleware, SSE).

Manual E2E on a build-capable host (curl):

Prerequisites

go build -o ./image-composer-tool ./cmd/image-composer-tool/
# Scoped passwordless sudoers rule (builds need root for chroot/mounts):
echo "$(whoami) ALL=(root) NOPASSWD: $(pwd)/image-composer-tool build *" \
  | sudo tee /etc/sudoers.d/ict-webui && sudo chmod 440 /etc/sudoers.d/ict-webui

Start the server (leave running)

./image-composer-tool serve --sudo --ict-binary "$(pwd)/image-composer-tool"
# -> "ICT web UI API listening on :8080"

1. Manifest: supported combinations + labels

curl -s localhost:8080/api/v1/manifest | python3 -m json.tool
# Expect: combinations[] (retail/robotics/fed-aero) + verticals/skus/platforms/targets

2. Compose: selection → resolved template + summary

curl -s -X POST localhost:8080/api/v1/templates/compose \
  -H 'Content-Type: application/json' \
  -d '{"vertical":"robotics","sku":"robotics-jazzy-ubuntu24","platform":"wcl","os":"ubuntu24","imageType":"iso"}' \
  | python3 -m json.tool
# Expect: template filename, YAML, summary{packageCount, partitionCount, ...}
# Negative: an unsupported combo -> HTTP 400 NO_MATCH

3. Start a build: returns a build ID

BID=$(curl -s -X POST localhost:8080/api/v1/builds \
  -H 'Content-Type: application/json' \
  -d '{"compose":{"vertical":"fed-aero","sku":"generic-handheld-blueprint","platform":"ptl","os":"ubuntu24","imageType":"raw"}}' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["buildId"])')
echo "buildId=$BID"

4. Stream logs (SSE; blocks until done, minutes for a real build)

curl -sN localhost:8080/api/v1/builds/$BID/logs
# Expect: event: log lines streaming live, ending with
#   event: complete  data: {"status":"success","artifacts":[...]}

5. Artifacts: image + SBOM paths

curl -s localhost:8080/api/v1/builds/$BID/artifacts | python3 -m json.tool
# Expect: status "success" + artifacts[] with the .raw.gz image and spdx SBOM (clean absolute paths)

6. --manifest live-edit (no rebuild): edit internal/api/data/manifest.yaml, restart with serve --manifest internal/api/data/manifest.yaml, changes apply.

Prerequisite recap: builds require the scoped, passwordless sudoers rule shown above; without it POST /builds fails at the sudo step.

arodage added 9 commits July 8, 2026 19:13
Add internal/api package and `serve` command backing the Basic tab:
- GET /manifest — serves combinations + dropdown labels (embedded manifest.yaml
  seeded with Retail Edge / Robotics / Fed Aero demo combinations)
- POST /templates/compose — manifest lookup → template YAML + summary
  (summary computed via config.LoadAndMergeTemplate)
- Build-path handlers (POST /builds via os/exec, SSE logs, artifacts) wired and
  compiling; end-to-end build verification pending on a build-capable host
- Vendor generic-handheld-os-template.yml from the edge-node-infrastructure-blueprint
  repo so the Fed Aero combination is buildable locally

Read path verified via curl: manifest lists 3 combinations; compose resolves the
robotics template (41 packages, 2 partitions) and returns 400 on no-match.
ICT builds require root (chroot/mounts) and are run with sudo on build hosts.
- Add --sudo serve flag: builds invoke `sudo -n <ict> build ...` (non-interactive
  so the server never blocks on a password prompt; needs a passwordless sudoers
  rule for the ICT binary)
- Add --work-dir serve flag (default webui-workspace) for per-build isolated
  work/output dirs, avoiding the root-owned ./workspace
- Build still runs from repo root so ICT resolves config/osv/... correctly

Read path and build-path plumbing (spawn, live SSE log streaming, status
tracking, artifacts) verified via curl. A successful root build to completion
must be verified on a host with (passwordless) sudo.
Expand the --sudo flag help to instruct operators to grant a passwordless
sudoers rule scoped to the ICT binary only, not blanket sudo. Build args are
server-controlled (manifest-derived template path + generated work dir) and
passed as argv (no shell), so the scoped rule has no user-controlled input.
Builds run as root under sudo and write outputs into root-owned subdirs, so the
non-root server's filesystem scan couldn't traverse them (artifacts came back
null despite a successful build). Parse the authoritative artifact list from
ICT's own "Generated Artifacts" output block instead — name, absolute path, and
type (image/sbom) — with the work-dir scan kept as a fallback. This also catches
outputs the old matcher missed (.vhdx, spdx_manifest_*.json).

Also serialize an empty artifact list as [] rather than null in the SSE
complete event. Add a parser unit test built from real build output.
extractPath took the first "/" in the line, which matched the logger's
"display/display.go:80" source prefix instead of the real path — artifact paths
came back as "/display.go:80\t/home/...". Take the last tab-separated field
(the log message) and require a leading "/" instead.

Strengthen the parser test to assert paths are clean absolute paths (no logger
prefix, no tabs) ending in the artifact name — the old behavior now fails it.
The manifest is compiled into the binary via //go:embed, so edits required a
rebuild to take effect. Add a --manifest flag: when set, the server reads the
manifest file from disk at startup (live-editable, restart to apply); when
empty it falls back to the embedded copy, preserving the single-binary default.
- Fix data race: SSE and artifacts handlers read build status/artifacts/errMsg
  without the mutex while runBuild writes them. Introduce a locked snapshot()
  returning an immutable result, and a single locked finish() for terminal
  state; readers use snapshot(). Set-once fields documented as lock-free.
- Handle the previously-ignored cmd.StdoutPipe() error (errcheck).

Verified: go build, go vet, go test -race all clean.
Map the fed-aero combination to ubuntu24-x86_64-minimal-raw.yml (a template
that exists) until a purpose-built Fed Aero template is available.
gofmt normalizes the parseArtifacts doc-comment example (bullet+tab was
reformatted to a list item). Apply gofmt to satisfy the Go Lint gate; the
parser behavior is unchanged (it still matches the • bullet in ICT output).
@arodage arodage changed the title feat(web-ui): backend image-composition service (Story 1) feat(web-ui): backend image-composition service Jul 8, 2026
arodage and others added 4 commits July 8, 2026 20:37
Add targeted unit tests to clear the CI coverage gate: manifest load/lookup,
manifest + compose + artifacts handlers (success and error paths), build
tracker + snapshot/finish state, buildCommand + template resolution,
middleware (CORS preflight, panic recovery, writeJSON/writeError), SSE event
formatting + log streaming for a finished build, server New/routes, and
discoverArtifacts. Raises internal/api coverage ~10%% -> ~78%%.
golangci-lint (errcheck) also lints _test.go files. Check the previously
unchecked os.WriteFile/os.MkdirAll/json.Unmarshal returns in api_test.go.
@arodage arodage marked this pull request as ready for review July 9, 2026 20:04
@arodage arodage requested a review from a team as a code owner July 9, 2026 20:04
Copilot AI review requested due to automatic review settings July 9, 2026 20:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new internal/api HTTP backend and a image-composer-tool serve subcommand to expose key image-composition workflows (manifest discovery, template composition, build execution, SSE log streaming, and artifact reporting) over a REST API, aligning with the Web UI contract established in #727.

Changes:

  • Adds an internal/api server with endpoints for manifest, compose, builds, SSE logs, and artifacts.
  • Executes end-to-end ICT builds per request (optionally via sudo -n) and tracks per-build state/logs in-memory.
  • Adds unit tests for manifest parsing, compose/build handlers, SSE formatting, and artifact parsing/discovery; bumps coverage threshold slightly.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
internal/api/sse.go SSE log streaming endpoint + event formatting helper
internal/api/server.go Server config, construction, and HTTP server startup
internal/api/router.go Route registration on Go ServeMux patterns
internal/api/middleware.go CORS, request logging, panic recovery, JSON/error helpers
internal/api/manifest.go Manifest model, embedded loading, template lookup
internal/api/handlers_read.go GET /manifest and POST /templates/compose handlers
internal/api/handlers_artifacts.go GET /builds/{id}/artifacts handler
internal/api/data/manifest.yaml Embedded manifest mapping UI selections to templates
internal/api/builds.go Build lifecycle tracking, command execution, artifact parsing/discovery
internal/api/builds_test.go Tests for artifact parsing from real-ish log output
internal/api/api_test.go Broad handler/unit tests for manifest/compose/build state/middleware/SSE
image-templates/generic-handheld-os-template.yml Adds a new (very large) Ubuntu-based template
cmd/image-composer-tool/serve.go New serve subcommand wiring CLI flags to API server config
cmd/image-composer-tool/main.go Registers the new serve subcommand
.coverage-threshold Increases repo coverage threshold from 68.4 to 68.5

Comment thread internal/api/handlers_read.go Outdated
Comment thread internal/api/handlers_read.go Outdated
Comment thread internal/api/builds.go
Comment thread internal/api/builds.go
Comment thread internal/api/builds.go
Comment thread cmd/image-composer-tool/serve.go
Comment thread internal/api/manifest.go Outdated
Comment thread internal/api/data/manifest.yaml Outdated
Comment thread image-templates/generic-handheld-os-template.yml
Comment thread image-templates/generic-handheld-os-template.yml
arodage added 2 commits July 9, 2026 22:01
Security:
- Path-traversal guard: safeTemplatePath() rejects absolute paths and ../
  traversal so a bad/edited manifest can't read files outside TemplatesDir
  (compose + build resolution).
- Bind 127.0.0.1 by default via new serve --host flag; 0.0.0.0 is opt-in
  (this API can trigger privileged builds).
- CORS restricted to localhost origins (echoes Origin) instead of "*".
- Per-build work dir created 0700 (was 0755); inline template written 0600.
- Remove baked password hash from generic-handheld template; use empty
  password + placeholder comment per repo convention.

Correctness:
- compose returns 422 when the matched template fails to load/validate
  instead of a misleading 200 with a zero summary.
- resolveBuildTemplate distinguishes client (400) from server (500) errors
  via errBadBuildRequest sentinel.
- runBuild checks scanner.Err() so a truncated log stream fails the build
  rather than being marked successful.
- findTemplate: an omitted SKU resolves only when unambiguous (no
  order-dependent guess across SKUs).

Docs:
- Manifest spelling Blue Print -> Blueprint.
- Comment why os/exec is used directly rather than internal/utils/shell
  (need to stream stdout + hold the process for cancellation).

Tests: CORS localhost behavior, safeTemplatePath traversal rejection,
ambiguous-SKU resolution, compose invalid-template 422. Coverage ~78.6%.
Point the fed-aero combination at generic-handheld-os-template.yml (the vendored
handheld blueprint) instead of the generic ubuntu24-x86_64-minimal-raw.yml, so
the combination resolves to its intended purpose-built template.

func executeServe(cmd *cobra.Command, args []string) error {
srv, err := api.New(api.Config{
Addr: serveHost + ":" + servePort,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This concatenation will not work for IPV6 hosts.

Comment thread internal/api/builds.go
b.finish(statusFailed, nil, err.Error())
return
}
cmd.Stderr = cmd.Stdout // merge streams

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use more cleaner approach of merging the stdout and stderr streams. There is a potential race condition to access the same buffer by both the streams but with different priorities. preferred is cmd.CombinedOutput.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants