Skip to content

local emulator image optimization #1318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BilalG1 wants to merge 4 commits into from
Closed

local emulator image optimization #1318

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7d9e156
local emulator image opt
BilalG1 Apr 9, 2026
d724eb2
emulator fixes
BilalG1 Apr 10, 2026
cd087c5
Merge branch 'dev' into local-emulator-image-optimization
BilalG1 Apr 10, 2026
784f17c
emulator: fail-fast on provision errors, diagnose smoke test failures
BilalG1 Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions docker/local-emulator/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,21 @@ RUN cp $(which qstash) /qstash-binary 2>/dev/null || \
{ echo "ERROR: qstash binary not found" >&2; exit 1; }


# ── Strip / compress service binaries (parallel stages) ──────────────────────

FROM debian:trixie-slim AS upx-compress
RUN apt-get update && apt-get install -y --no-install-recommends upx-ucl binutils && \
rm -rf /var/lib/apt/lists/*
COPY --from=clickhouse-bin /usr/bin/clickhouse /out/clickhouse
COPY --from=svix-bin /usr/local/bin/svix-server /out/svix-server
COPY --from=minio-bin /usr/bin/minio /out/minio
COPY --from=mc-bin /usr/bin/mc /out/mc
COPY --from=qstash-bin /qstash-binary /out/qstash
RUN chmod u+w /out/* && \
strip --strip-all /out/clickhouse /out/minio /out/svix-server /out/mc /out/qstash && \
upx -9 /out/minio /out/svix-server /out/mc /out/qstash


# ── Final image ───────────────────────────────────────────────────────────────

FROM debian:trixie-slim
Expand Down Expand Up @@ -139,20 +154,20 @@ COPY --from=node-base /usr/local/bin/node /usr/local/bin/node
# Inbucket
COPY --from=inbucket-bin /opt/inbucket /opt/inbucket

# Svix
COPY --from=svix-bin /usr/local/bin/svix-server /usr/local/bin/svix-server
# Svix (UPX-compressed)
COPY --from=upx-compress /out/svix-server /usr/local/bin/svix-server

# ClickHouse
COPY --from=clickhouse-bin /usr/bin/clickhouse /usr/bin/clickhouse
# ClickHouse (stripped only)
COPY --from=upx-compress /out/clickhouse /usr/bin/clickhouse
RUN ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-server && \
ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-client

# MinIO
COPY --from=minio-bin /usr/bin/minio /usr/local/bin/minio
COPY --from=mc-bin /usr/bin/mc /usr/local/bin/mc
# MinIO (UPX-compressed)
COPY --from=upx-compress /out/minio /usr/local/bin/minio
COPY --from=upx-compress /out/mc /usr/local/bin/mc

# QStash
COPY --from=qstash-bin --chmod=755 /qstash-binary /usr/local/bin/qstash
# QStash (UPX-compressed)
COPY --from=upx-compress --chmod=755 /out/qstash /usr/local/bin/qstash

# App
WORKDIR /app
Expand All @@ -164,6 +179,10 @@ COPY --from=builder /app/apps/backend/node_modules ./apps/backend/node_modules
COPY --from=builder /app/apps/dashboard/.next/standalone ./
COPY --from=builder /app/apps/dashboard/.next/static ./apps/dashboard/.next/static
COPY --from=builder /app/apps/dashboard/public ./apps/dashboard/public
# Save the standalone-traced node_modules (runtime deps only) before the full
# migration-pruner copy overwrites it. The slim-docker-image step in the QEMU
# build restores this after migrations are baked in.
RUN cp -a /app/node_modules /app/node_modules.standalone 2>/dev/null || mkdir -p /app/node_modules.standalone

@greptile-apps greptile-apps Bot Apr 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Silent cp failure creates an empty node_modules.standalone

The 2>/dev/null || mkdir -p /app/node_modules.standalone fallback means that if the cp -a silently fails (e.g. disk full, source missing), the build continues with an empty node_modules.standalone. When slim-docker-image later swaps it in, the container would have no node_modules and fail at runtime — likely caught only by the smoke test. Consider removing the suppression and fallback to surface the failure fast:

Suggested change
RUN cp -a /app/node_modules /app/node_modules.standalone 2>/dev/null || mkdir -p /app/node_modules.standalone
RUN cp -a /app/node_modules /app/node_modules.standalone
Prompt To Fix With AI 
This is a comment left during a code review.
Path: docker/local-emulator/Dockerfile
Line: 188

Comment:
**Silent `cp` failure creates an empty `node_modules.standalone`**

The `2>/dev/null || mkdir -p /app/node_modules.standalone` fallback means that if the `cp -a` silently fails (e.g. disk full, source missing), the build continues with an empty `node_modules.standalone`. When `slim-docker-image` later swaps it in, the container would have no node_modules and fail at runtime — likely caught only by the smoke test. Consider removing the suppression and fallback to surface the failure fast:

```suggestion
RUN cp -a /app/node_modules /app/node_modules.standalone
```

How can I resolve this? If you propose a fix, please make it concise.

COPY --from=migration-pruner /pruned-node_modules ./node_modules
COPY --from=builder /app/packages ./packages

Expand Down
102 changes: 91 additions & 11 deletions docker/local-emulator/qemu/build-image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,46 @@ prepare_bundle_artifacts() {
printf "%s" "$current_ids" > "$bundle_meta"
}

contains_provision_marker() {
local provision_log="$1"
local serial_log="$2"
local marker="$3"

if [ -f "$provision_log" ] && grep -Fqx "$marker" "$provision_log" 2>/dev/null; then
return 0
fi

if [ -f "$serial_log" ] && LC_ALL=C strings -a "$serial_log" 2>/dev/null | grep -Fqx "$marker" 2>/dev/null; then
return 0
fi

return 1
}

line_count() {
local file="$1"
local count=0

if [ -f "$file" ]; then
count="$(wc -l < "$file" | tr -d '[:space:]')" || count=0
fi

case "$count" in
''|*[!0-9]*) count=0 ;;
esac

printf '%s\n' "$count"
}

persist_provision_logs() {
local arch="$1"
local serial_log="$2"
local provision_log="$3"

cp "$serial_log" "$IMAGE_DIR/provision-emulator-${arch}.log" 2>/dev/null || true
cp "$provision_log" "$IMAGE_DIR/provision-emulator-${arch}.progress.log" 2>/dev/null || true
}

build_one() {
local arch="$1"
local base_img="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2"
Expand All @@ -192,8 +232,11 @@ build_one() {
local bundle_iso="$tmp_dir/bundle.iso"
local bundle_dir="$tmp_dir/bundle"
local serial_log="$tmp_dir/serial.log"
local provision_log="$tmp_dir/provision.log"
local pidfile="$tmp_dir/qemu.pid"
local qemu_base pid elapsed
local qemu_base pid elapsed total_build_lines
local last_build_lines=0
local guest_exited=false
local start_time=$SECONDS

cp "$base_img" "$tmp_img"
Expand All @@ -209,21 +252,24 @@ build_one() {

mkdir -p "$bundle_dir"
cp "$bundle_tgz" "$bundle_dir/img.tgz"
cp "$BUILD_ENV_FILE" "$bundle_dir/build.env"
make_iso_from_dir "$bundle_iso" "STACKBUNDLE" "$bundle_dir"

: > "$serial_log"
: > "$provision_log"
qemu_base="$(qemu_cmd_prefix_for_arch "$arch")"

# shellcheck disable=SC2086
$qemu_base \
-boot order=c \
-m "$RAM" \
-smp "$CPUS" \
-drive "file=$tmp_img,format=qcow2,if=virtio" \
-drive "file=$tmp_img,format=qcow2,if=virtio,discard=on,detect-zeroes=unmap" \
-drive "file=$seed_iso,format=raw,if=virtio,readonly=on" \
-drive "file=$bundle_iso,format=raw,if=virtio,readonly=on" \
-netdev user,id=net0 \
-device virtio-net-pci,netdev=net0 \
-virtfs "local,path=$tmp_dir,mount_tag=hostfs,security_model=none" \
-serial "file:$serial_log" \
-display none \
-daemonize \
Expand All @@ -232,23 +278,55 @@ build_one() {
pid="$(cat "$pidfile")"
elapsed=0
while [ "$elapsed" -lt "$PROVISION_TIMEOUT" ]; do
if grep -q "STACK_CLOUD_INIT_DONE" "$serial_log" 2>/dev/null; then
if contains_provision_marker "$provision_log" "$serial_log" "STACK_CLOUD_INIT_DONE"; then
break
fi

if [ -f "$provision_log" ]; then
total_build_lines="$(line_count "$provision_log")"
if [ "$total_build_lines" -gt "$last_build_lines" ]; then
echo ""
sed -n "$((last_build_lines + 1)),${total_build_lines}p" "$provision_log" 2>/dev/null | while IFS= read -r msg; do
if [ "$msg" = "STACK_CLOUD_INIT_DONE" ]; then
continue
fi
printf " [%3ds] %s\n" "$elapsed" "$msg"
done
last_build_lines="$total_build_lines"
fi
fi

if ! kill -0 "$pid" 2>/dev/null; then
guest_exited=true
break
fi

sleep 5
elapsed=$((SECONDS - start_time))
printf "\r [%3ds / %ds] provisioning emulator..." "$elapsed" "$PROVISION_TIMEOUT"
done
echo ""

if ! grep -q "STACK_CLOUD_INIT_DONE" "$serial_log" 2>/dev/null; then
err "Provisioning timed out for emulator (${arch})"
tail -50 "$serial_log" >&2 || true
if ! contains_provision_marker "$provision_log" "$serial_log" "STACK_CLOUD_INIT_DONE"; then
if [ "$guest_exited" = true ]; then
err "Provisioning exited before completion for emulator (${arch})"
else
err "Provisioning timed out for emulator (${arch})"
fi

if [ -s "$provision_log" ]; then
tail -50 "$provision_log" >&2 || true
else
LC_ALL=C strings -a "$serial_log" 2>/dev/null | tail -50 >&2 || tail -50 "$serial_log" >&2 || true
fi

if kill -0 "$pid" 2>/dev/null; then
kill "$pid" 2>/dev/null || true
sleep 1
kill -9 "$pid" 2>/dev/null || true
fi

persist_provision_logs "$arch" "$serial_log" "$provision_log"
rm -rf "$tmp_dir"
exit 1
fi
Expand All @@ -266,19 +344,21 @@ build_one() {
kill -9 "$pid" 2>/dev/null || true
fi

cp "$tmp_img" "$final_img"
cp "$serial_log" "$IMAGE_DIR/provision-emulator-${arch}.log"
rm -rf "$tmp_dir"
persist_provision_logs "$arch" "$serial_log" "$provision_log"

log "Compressing final image (this may take several minutes)..."
qemu-img convert -p -O qcow2 -c "$final_img" "$final_img.tmp"
mv "$final_img.tmp" "$final_img"
qemu-img convert -p -O qcow2 -c "$tmp_img" "$final_img"
rm -rf "$tmp_dir"
Comment on lines 358 to +360

@coderabbitai coderabbitai Bot Apr 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Check the file size and read lines around 358-360
wc -l docker/local-emulator/qemu/build-image.sh

Repository: stack-auth/stack-auth

Length of output: 110

🏁 Script executed:

# Read the relevant section
sed -n '350,375p' docker/local-emulator/qemu/build-image.sh

Repository: stack-auth/stack-auth

Length of output: 951

🏁 Script executed:

# Search for node invocations and check_deps function
rg -n "node " docker/local-emulator/qemu/build-image.sh | head -20

Repository: stack-auth/stack-auth

Length of output: 137

🏁 Script executed:

# Find the check_deps function to see if node is validated
rg -n "check_deps\|qemu-img\|node" docker/local-emulator/qemu/build-image.sh | head -30

Repository: stack-auth/stack-auth

Length of output: 47

🏁 Script executed:

# Search for check_deps function more broadly
rg -n "check_deps|declare -a|qemu-img" docker/local-emulator/qemu/build-image.sh

Repository: stack-auth/stack-auth

Length of output: 274

🏁 Script executed:

# Get the full check for dependencies/requirements
sed -n '1,100p' docker/local-emulator/qemu/build-image.sh | rg -n "check|qemu|curl|docker|gzip|node"

Repository: stack-auth/stack-auth

Length of output: 453

🏁 Script executed:

# Search for any validation of dependencies in the entire file
rg -B2 -A5 "command -v|which " docker/local-emulator/qemu/build-image.sh

Repository: stack-auth/stack-auth

Length of output: 610

Avoid writing the final qcow2 in place.

Line 359 converts directly into "$final_img". If qemu-img convert fails or is interrupted, an existing image can be left partially overwritten, which makes reruns or package steps pick up a corrupt artifact. Use a temporary file with atomic rename to ensure failure doesn't corrupt the final artifact.

Additionally, node is invoked at line 368 but is not validated in the check_deps() function (line 50), creating an incomplete dependency check.

Suggested fix 
-  qemu-img convert -p -O qcow2 -c "$tmp_img" "$final_img"
-  rm -rf "$tmp_dir"
+  local tmp_final="${final_img}.tmp"
+  rm -f "$tmp_final"
+  qemu-img convert -p -O qcow2 -c "$tmp_img" "$tmp_final"
+  mv "$tmp_final" "$final_img"
+  rm -rf "$tmp_dir"

Also add node to the dependency check loop at line 50.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
log "Compressing final image (this may take several minutes)..."
qemu-img convert -p -O qcow2 -c "$final_img" "$final_img.tmp"
mv "$final_img.tmp" "$final_img"
qemu-img convert -p -O qcow2 -c "$tmp_img" "$final_img"
rm -rf "$tmp_dir"
log "Compressing final image (this may take several minutes)..."
local tmp_final="${final_img}.tmp"
rm -f "$tmp_final"
qemu-img convert -p -O qcow2 -c "$tmp_img" "$tmp_final"
mv "$tmp_final" "$final_img"
rm -rf "$tmp_dir"
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docker/local-emulator/qemu/build-image.sh` around lines 358 - 360, The
qemu-img convert currently writes directly to "$final_img" which can leave a
partially overwritten corrupt file on failure; change the workflow around the
qemu-img convert invocation (the block using tmp_img and final_img) to write the
output to a temporary file (e.g., "$final_img.tmp" or similar) and then
atomically rename/move it to "$final_img" only after convert succeeds, and still
ensure tmp_dir cleanup (rm -rf "$tmp_dir") happens appropriately on both success
and failure; additionally, update the dependency check in the check_deps
function to include "node" in the tooling list so the script validates node is
installed before invoking it later.


local size
size="$(du -h "$final_img" | cut -f1)"
log "━━━ Emulator image ready: $final_img (${size}) ━━━"
}

log "Generating emulator build env file..."
node "$REPO_ROOT/docker/local-emulator/generate-env-development.mjs"
BUILD_ENV_FILE="$REPO_ROOT/docker/local-emulator/.env.development"
Comment on lines +367 to +369

@coderabbitai coderabbitai Bot Apr 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

# First, let's examine the script file around lines 367-369
cat -n docker/local-emulator/qemu/build-image.sh | sed -n '360,375p'

Repository: stack-auth/stack-auth

Length of output: 718

🏁 Script executed:

# Now let's find the check_deps() function
rg -A 15 "check_deps\(\)" docker/local-emulator/qemu/build-image.sh

Repository: stack-auth/stack-auth

Length of output: 576

🏁 Script executed:

# Let's also search for all occurrences of "node" in the script to see if it's used elsewhere
rg -n "node" docker/local-emulator/qemu/build-image.sh

Repository: stack-auth/stack-auth

Length of output: 137

Add node to the upfront dependency check.

This script unconditionally calls node at line 368, but check_deps() doesn't verify it's available. Adding it to the existing dependency check will catch missing node early instead of failing at runtime.

Suggested fix 
-  for cmd in qemu-img curl docker gzip; do
+  for cmd in qemu-img curl docker gzip node; do
     command -v "$cmd" >/dev/null 2>&1 || missing+=("$cmd")
   done
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docker/local-emulator/qemu/build-image.sh` around lines 367 - 369, The script
calls node to run generate-env-development.mjs but check_deps() doesn’t include
node, so add "node" to the dependency list used by check_deps() (the same place
where other tools like docker/awk are listed) so that check_deps() fails early
if node is missing; update the dependency array or call to check_deps() that
feeds into the check_deps function and keep the BUILD_ENV_FILE and the node
"$REPO_ROOT/docker/local-emulator/generate-env-development.mjs" invocation
unchanged.


for arch in "${TARGET_ARCHS[@]}"; do
local_base="$IMAGE_DIR/debian-${DEBIAN_VERSION}-base-${arch}.qcow2"
download_cloud_image "$arch" "$local_base"
Expand Down
Loading
Loading