Skip to content

Commit 791e367

Browse files
authored
Various changes to speed up local testing on arm hosts (#975)
* Various changes to speed up local testing on arm hosts * Review fixes
1 parent 28887a7 commit 791e367

10 files changed

Lines changed: 344 additions & 67 deletions

File tree

server/README.md

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,38 @@ During the testing local servers will be run. That's why it necessary to have pr
118118
* **test** - run for integration testing (see *IntegrationTest.java*)
119119
* **auth-test** - run for authentication testing (see *AuthenticationTest.java*)
120120

121+
### Speeding up local runs (Apple Silicon)
122+
123+
A full `server:test` builds every target across every Defold version and, on an arm64 Mac, runs
124+
those builders emulated. A few opt-in flags cut that down for local iteration. They are all
125+
no-ops in CI, so they do not change what CI runs.
126+
127+
* **Enable Rosetta first.** In Docker Desktop → Settings → General → Virtual Machine Options, use
128+
*Apple Virtualization framework* with *Use Rosetta for x86_64/amd64 emulation on Apple Silicon*.
129+
This is a prerequisite, not a tip: the Android and `x86_64-linux` builders can never be arm64
130+
(their toolchains are x86_64-only), so they always run emulated.
131+
132+
* `-PtargetPlatforms=<list>` — build only these targets, and boot only the builders they need.
133+
Values: `x86_64-linux`, `armv7-android`, `arm64-android`, `js-web`, `wasm-web`, `x86_64-win32`.
134+
* `-PdefoldVersions=<list>` — which Defold SDK versions to test. `latest`, `ci` (first + last, what
135+
CI uses), `all`, or a comma list like `1.12.3,1.11.1`.
136+
* `-PreuseStack=true` — leave the docker stack running after the tests and reuse it next time. The
137+
stack is recreated automatically when `server/app/extender.jar` is rebuilt, so you never test
138+
stale code. Stop it by hand with `EXTENDER_KEEP_STACK=0 docker compose -p extender-test down`.
139+
* `-PbuildSleepTimeout=500` — poll async build status every 500ms instead of the client default
140+
5000ms. Every build waits at least one full interval before its first status check, so with fast
141+
(Rosetta) builders the default turns each ~5s build into a flat 10s.
142+
143+
```sh
144+
# one target, one version - 1 config and 2 containers instead of 36 configs and 8 containers
145+
./gradlew server:test --tests '*IntegrationTest*' -PtargetPlatforms=x86_64-linux -PdefoldVersions=latest \
146+
-PreuseStack=true -PbuildSleepTimeout=500
147+
```
148+
149+
Downloaded Defold SDKs are kept in an external docker volume (`extender-sdk-cache`) between runs, so
150+
they are fetched once instead of on every run. This is on by default locally and off in CI; set
151+
`EXTENDER_DEV_CACHE=0` to opt out. Remove the cache with `docker volume rm extender-sdk-cache`.
152+
121153
## How to debug running instance
122154
To enable remote JVM debug need to add following additional options to entrypoint section in `./server/docker/common-services.yml` for `common_builder` service
123155
```
@@ -189,4 +221,4 @@ Keys in `extender.remote-builder.platform` should be formed in the following way
189221
5. Frontend instance sends a build request to the found server url.
190222

191223
# Testing notes
192-
When runs integration tests on Macos at arm chips - check docker engine configuration. It's better to use `Virtual Machine option` -> `Apple Virtualization framework` with checked `Use Rosetta for x86_64/amd64 emulation on Apple Silicon` checkbox.
224+
When runs integration tests on Macos at arm chips - check docker engine configuration. It's better to use `Virtual Machine option` -> `Apple Virtualization framework` with checked `Use Rosetta for x86_64/amd64 emulation on Apple Silicon` checkbox. See [Speeding up local runs](#speeding-up-local-runs-apple-silicon) for the flags that make a local run practical.

server/build.gradle

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,22 @@ test {
218218
excludeTags project.getProperty("excludeTags")
219219
}
220220
}
221+
// Opt-in knobs to shrink a local integration test run. All absent in CI, where they are no-ops.
222+
// -PtargetPlatforms=x86_64-linux,js-web only build these targets, and only boot the
223+
// builders they need (see docker-compose.yml profiles)
224+
// -PdefoldVersions=all|ci|latest|1.12.3 which Defold SDK versions to test against
225+
// -PreuseStack=true leave the docker stack running between runs
226+
["targetPlatforms", "defoldVersions", "reuseStack"].each { name ->
227+
if (project.hasProperty(name)) {
228+
systemProperty "extender.test.${name}", project.getProperty(name)
229+
}
230+
}
231+
// -PbuildSleepTimeout=500 poll job status every 500ms instead of the client
232+
// default 5000ms - each async build waits at least one
233+
// full interval before its first status check
234+
if (project.hasProperty("buildSleepTimeout")) {
235+
systemProperty "com.defold.extender.client.build-sleep-timeout", project.getProperty("buildSleepTimeout")
236+
}
221237
finalizedBy jacocoTestReport
222238
}
223239

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Local-developer override: keep downloaded Defold SDKs between test runs.
2+
#
3+
# Without this, /var/extender/sdk lives in the container's writable layer, so every
4+
# `docker compose down` throws the SDKs away and the next run re-downloads and re-unzips
5+
# one per (builder x Defold version) - by far the largest cost of a local test run.
6+
#
7+
# Enabled by server/scripts/start-test-server.sh unless EXTENDER_DEV_CACHE=0, and never
8+
# in CI. Layered with `-f`, so docker-compose.yml stays exactly what CI sees.
9+
#
10+
# Only the builders mount the cache: the frontend is FRONTEND_ONLY and never downloads an
11+
# SDK (only AsyncBuilder does), and keeping it off the shared volume keeps its @PreDestroy
12+
# cache cleanup away from everyone else's SDKs.
13+
14+
volumes:
15+
# External, so the `extender-test` and `extender-test-auth` compose projects share one
16+
# cache instead of each getting a project-prefixed copy. Created by start-test-server.sh.
17+
extender-sdk-cache:
18+
external: true
19+
20+
x-sdk-cache: &sdk-cache
21+
volumes:
22+
- extender-sdk-cache:/var/extender/sdk
23+
environment:
24+
# DefoldSdkService deletes the whole cache in @PreDestroy. Compose only grants a 10s
25+
# stop grace, so a multi-GB delete gets SIGKILLed part-way through - which is harmless
26+
# for a throwaway writable layer but corrupts a persistent volume, because getRemoteSdk
27+
# treats "directory exists" as "already verified".
28+
- EXTENDER_SDK_CACHECLEARONEXIT=false
29+
# The builders now share one cache directory, so raise the eviction bound above the
30+
# default 10 to keep them from evicting each other's SDKs.
31+
- EXTENDER_SDK_CACHESIZE=32
32+
33+
services:
34+
linux-integration-test:
35+
<<: *sdk-cache
36+
android_ndk25-integration-test:
37+
<<: *sdk-cache
38+
android_ndk25_sdk36-integration-test:
39+
<<: *sdk-cache
40+
emscripten_406-integration-test:
41+
<<: *sdk-cache
42+
win_2022-integration-test:
43+
<<: *sdk-cache
44+
win_2022_144435207-integration-test:
45+
<<: *sdk-cache
46+
win_2026_145136231-integration-test:
47+
<<: *sdk-cache

server/docker/docker-compose.yml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,16 @@ services:
205205
- xbox-251002
206206

207207
# services for integration testing
208+
#
209+
# "test" starts every builder and is what CI uses. The test-<target> profiles let a local
210+
# run boot only the builders it needs: IntegrationTest derives them from -PtargetPlatforms.
211+
# They are purely additive - "test" still selects all of these services.
212+
# test-linux -> x86_64-linux
213+
# test-android -> armv7-android, arm64-android
214+
# test-web -> js-web, wasm-web
215+
# test-windows -> x86_64-win32
216+
# Both android builders, and all three winsdk builders, stay grouped together: which one
217+
# serves a request is decided by that Defold version's platform.sdks.json, not by the test.
208218
frontend-integration-test:
209219
image: europe-west1-docker.pkg.dev/extender-426409/extender-public-registry/extender-base-env:latest
210220
extends:
@@ -217,6 +227,10 @@ services:
217227
- "9000:9000"
218228
profiles:
219229
- test
230+
- test-linux
231+
- test-android
232+
- test-web
233+
- test-windows
220234
networks:
221235
default:
222236
aliases:
@@ -229,6 +243,7 @@ services:
229243
profiles:
230244
- test
231245
- auth-test
246+
- test-linux
232247
platform: linux/amd64
233248
networks:
234249
default:
@@ -243,6 +258,7 @@ services:
243258
profiles:
244259
- test
245260
- auth-test
261+
- test-android
246262
networks:
247263
default:
248264
aliases:
@@ -256,6 +272,7 @@ services:
256272
profiles:
257273
- test
258274
- auth-test
275+
- test-android
259276
networks:
260277
default:
261278
aliases:
@@ -267,6 +284,7 @@ services:
267284
service: test_remote_builder
268285
profiles:
269286
- test
287+
- test-web
270288
networks:
271289
default:
272290
aliases:
@@ -278,6 +296,7 @@ services:
278296
service: test_remote_builder
279297
profiles:
280298
- test
299+
- test-windows
281300
networks:
282301
default:
283302
aliases:
@@ -289,6 +308,7 @@ services:
289308
service: test_remote_builder
290309
profiles:
291310
- test
311+
- test-windows
292312
networks:
293313
default:
294314
aliases:
@@ -300,6 +320,7 @@ services:
300320
service: test_remote_builder
301321
profiles:
302322
- test
323+
- test-windows
303324
networks:
304325
default:
305326
aliases:

server/scripts/start-test-server.sh

Lines changed: 84 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -18,89 +18,109 @@ if [ "$PORT" == "" ]; then
1818
PORT=9000
1919
fi
2020

21+
# Local developer speedups, both off in CI (GITHUB_ACTION is set by Actions).
22+
# EXTENDER_DEV_CACHE=1 keep the downloaded Defold SDKs in a named volume between runs
23+
# EXTENDER_KEEP_STACK=1 leave the containers running afterwards (see stop-test-server.sh)
24+
if [ "$GITHUB_ACTION" != "" ]; then
25+
EXTENDER_DEV_CACHE=0
26+
fi
27+
if [ "$EXTENDER_DEV_CACHE" == "" ]; then
28+
EXTENDER_DEV_CACHE=1
29+
fi
30+
if [ "$EXTENDER_KEEP_STACK" == "" ]; then
31+
EXTENDER_KEEP_STACK=0
32+
fi
33+
2134
echo "Using RUN_ENV: ${RUN_ENV}"
2235
echo "Using compose profile: ${COMPOSE_PROFILE}"
2336
echo "Start application: ${APPLICATION}"
2437
echo "Using PORT: ${PORT}"
2538

2639
URL=http://localhost:${PORT}
2740

41+
COMPOSE_FILES=(-f "${DIR}/../docker/docker-compose.yml")
42+
if [ "$EXTENDER_DEV_CACHE" == "1" ]; then
43+
# The volume is external so that the 'extender-test' and 'extender-test-auth'
44+
# projects share one SDK cache instead of getting a project-prefixed copy each.
45+
docker volume create extender-sdk-cache > /dev/null
46+
COMPOSE_FILES+=(-f "${DIR}/../docker/docker-compose.dev-cache.yml")
47+
fi
48+
49+
# COMPOSE_PROFILE may name several profiles, e.g. "test-linux test-web"
50+
PROFILE_ARGS=()
51+
for profile in $COMPOSE_PROFILE; do
52+
PROFILE_ARGS+=(--profile "$profile")
53+
done
54+
55+
COMPOSE=(docker compose -p "$APPLICATION" "${COMPOSE_FILES[@]}" "${PROFILE_ARGS[@]}")
56+
2857
function check_server() {
2958
if curl -s --head --request GET ${URL} | grep "200 OK" > /dev/null; then
3059
echo "ERROR: ${URL} is already occupied!"
3160
exit 1
3261
fi
3362
}
3463

35-
# fail early
36-
check_server
64+
# A container keeps running whatever extender.jar it loaded at startup, so a stack we
65+
# are reusing must be recreated once the jar is rebuilt - otherwise the tests would
66+
# silently exercise stale code.
67+
function stack_is_current() {
68+
local jar="${DIR}/../app/extender.jar"
69+
[ -f "$jar" ] || return 0
70+
71+
local ids
72+
ids=$("${COMPOSE[@]}" ps -q 2>/dev/null)
73+
[ -n "$ids" ] || return 1
74+
75+
local jar_mtime
76+
# GNU stat must come first: BSD-style "stat -f %m" fails on GNU but only after printing
77+
# the whole filesystem status to stdout, which would corrupt jar_mtime. "stat -c" on BSD
78+
# fails cleanly with nothing on stdout.
79+
jar_mtime=$(stat -c %Y "$jar" 2>/dev/null || stat -f %m "$jar")
80+
81+
local started started_epoch
82+
for id in $ids; do
83+
# StartedAt is UTC (e.g. 2026-07-14T22:40:26.4Z); parse it as UTC on both BSD and GNU date.
84+
started=$(docker inspect -f '{{.State.StartedAt}}' "$id")
85+
started_epoch=$(date -u -j -f "%Y-%m-%dT%H:%M:%S" "${started:0:19}" +%s 2>/dev/null \
86+
|| date -u -d "$started" +%s)
87+
if [ "$jar_mtime" -gt "$started_epoch" ]; then
88+
return 1
89+
fi
90+
done
91+
return 0
92+
}
93+
94+
# fail early - but a stack we intend to reuse is expected to answer on ${URL}
95+
if [ "$EXTENDER_KEEP_STACK" != "1" ]; then
96+
check_server
97+
fi
3798

3899
# For CI to be able to work with the test files
39100
if [ "$GITHUB_ACTION" != "" ]; then
40101
chmod -R a+xrw ${DIR}/../test-data || true
41102
fi
42103

43-
docker compose -p $APPLICATION -f ${DIR}/../docker/docker-compose.yml --profile $COMPOSE_PROFILE up -d
44-
45-
# Retry configuration
46-
max_retries=10
47-
retry_interval=15
48-
49-
check_containers_health() {
50-
all_healthy=true
51-
52-
for container in $(docker ps -q); do
53-
health_field=$(docker inspect --format='{{.State.Health}}' "$container")
54-
if [ "$health_field" == "<nil>" ]; then
55-
echo "Container has no health status. Skipped."
56-
continue
57-
fi
58-
name=$(docker inspect --format='{{.Name}}' "$container" | sed 's/\///')
59-
app_name=$(docker inspect "$container" | jq -r '.[0].Config.Labels["com.docker.compose.project"]')
60-
health_status=$(docker inspect --format='{{.State.Health.Status}}' "$container")
61-
62-
if [ "$app_name" != "$APPLICATION" ]; then
63-
echo "Skip health check of unrelated container."
64-
fi
65-
66-
# If health status is empty, container doesn't have a health check defined
67-
if [ -z "$health_status" ]; then
68-
health_status="no health check"
69-
continue
70-
fi
71-
72-
echo "$name: $health_status"
73-
74-
# Check if the container is not healthy
75-
if [ "$health_status" != "healthy" ]; then
76-
all_healthy=false
77-
fi
78-
done
79-
80-
# Return whether all containers are healthy
81-
$all_healthy && return 0 || return 1
82-
}
104+
UP_ARGS=(up -d --wait --wait-timeout 300)
105+
if [ "$EXTENDER_KEEP_STACK" == "1" ] && ! stack_is_current; then
106+
echo "extender.jar is newer than the running containers - recreating them"
107+
UP_ARGS+=(--force-recreate)
108+
fi
83109

84-
# Main loop to retry until all containers are healthy or retries run out
85-
for (( i=1; i<=$max_retries; i++ )); do
86-
echo "Attempt $i of $max_retries: Checking container health..."
87-
88-
if check_containers_health; then
89-
echo "All containers are healthy!"
90-
CREATED_SERVICES=$(docker compose -p $APPLICATION ps -a --services)
91-
for service in $CREATED_SERVICES; do
92-
echo "Copy test sdk to $service"
93-
docker compose -p $APPLICATION cp "${DIR}/../test-data/sdk/a" "$service:/var/extender/sdk"
94-
docker compose -p $APPLICATION exec -u root $service chown -R extender:extender /var/extender/sdk/a
95-
done
96-
97-
exit 0
98-
else
99-
echo "Some containers are not healthy yet. Retrying in $retry_interval seconds..."
100-
sleep $retry_interval
101-
fi
110+
"${COMPOSE[@]}" "${UP_ARGS[@]}"
111+
112+
for service in $("${COMPOSE[@]}" ps --services); do
113+
# base-env creates /var/extender and /var/extender/cache but not /var/extender/sdk,
114+
# so with EXTENDER_DEV_CACHE the volume mounts root-owned and the extender user
115+
# (uid 2222) cannot write to it. Harmless when there is no volume.
116+
"${COMPOSE[@]}" exec -T -u root "$service" mkdir -p /var/extender/sdk
117+
"${COMPOSE[@]}" exec -T -u root "$service" chown extender:extender /var/extender/sdk
118+
119+
# "a" is the version-controlled test sdk. It survives in the cache volume, and both
120+
# evictCache() and destroy() in DefoldSdkService deliberately exempt it.
121+
if ! "${COMPOSE[@]}" exec -T "$service" test -d /var/extender/sdk/a; then
122+
echo "Copy test sdk to $service"
123+
"${COMPOSE[@]}" cp "${DIR}/../test-data/sdk/a" "$service:/var/extender/sdk"
124+
"${COMPOSE[@]}" exec -T -u root "$service" chown -R extender:extender /var/extender/sdk/a
125+
fi
102126
done
103-
104-
# If we reach this point, some containers did not become healthy within the retry limit
105-
echo "Some containers did not become healthy after $max_retries retries."
106-
exit 1

server/scripts/stop-test-server.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,11 @@ if [ "$APPLICATION" == "" ]; then
44
APPLICATION="extender-test"
55
fi
66

7+
if [ "$EXTENDER_KEEP_STACK" == "1" ]; then
8+
echo "stop-test-server.sh: Leaving ${APPLICATION} running (EXTENDER_KEEP_STACK=1)"
9+
exit 0
10+
fi
11+
712
echo "stop-test-server.sh: Stopping ${APPLICATION}:"
813

914
docker compose -p $APPLICATION down

0 commit comments

Comments
 (0)