Skip to content

Commit 4986779

Browse files
committed
ops(ci): upload full demo cluster log on Jepsen failure
The previous "Dump demo cluster log on failure" step ran `tail -n 500 /tmp/elastickv-demo.log` which, for a 3-minute workload, captures only the startup lines and discards the window where the actual anomaly happens. Investigation of the scheduled run 26198185540 (which surfaced a real :duplicate-elements anomaly) was blocked because the GH UI showed only server-startup logs, not the start_ts / commit_ts / raft term / write-conflict / lock-resolver events that would identify the offending code path. Change: - Inline dump now prints head + tail (200 / 1000 lines for the single-process demo, 200 / 500 per node for the 3-node demo) so the GH UI still shows enough at-a-glance. - Full log uploaded as a 14-day artifact (`elastickv-demo-log` for the scheduled workflow, `elastickv-demo-logs` with three files for the per-push workflow). 14 days mirrors the existing Jepsen-store artifact retention so logs and history.txt can be correlated for the same run. Self-review: 1. Data loss — N/A. 2. Concurrency — N/A. 3. Performance — only fires on failure; artifact upload is a single tar-gz of a multi-MB log, negligible CI time. 4. Data consistency — N/A. 5. Test coverage — workflow change; no Go-side test surface. Verification = next failing scheduled run produces a downloadable artifact.
1 parent c9a7870 commit 4986779

2 files changed

Lines changed: 49 additions & 1 deletion

File tree

.github/workflows/jepsen-test-scheduled.yml

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,28 @@ jobs:
229229
--host 127.0.0.1
230230
- name: Dump demo cluster log on failure
231231
if: failure()
232-
run: tail -n 500 /tmp/elastickv-demo.log || true
232+
# The previous `tail -n 500` truncated a 3-minute workload's
233+
# log down to startup-only lines, making it impossible to
234+
# correlate a Jepsen anomaly with the server-side state
235+
# (start_ts, commit_ts, raft term, write conflicts, lock-
236+
# resolver events). Print head + tail inline so the GH UI
237+
# still shows the most recent activity at-a-glance, then
238+
# upload the full log as an artifact for offline analysis.
239+
run: |
240+
echo "=== first 200 lines (startup) ==="
241+
head -n 200 /tmp/elastickv-demo.log || true
242+
echo "=== last 1000 lines (most recent activity) ==="
243+
tail -n 1000 /tmp/elastickv-demo.log || true
244+
echo "=== full log line count ==="
245+
wc -l /tmp/elastickv-demo.log || true
246+
- name: Upload demo cluster log on failure
247+
if: failure()
248+
uses: actions/upload-artifact@v4
249+
with:
250+
name: elastickv-demo-log
251+
path: /tmp/elastickv-demo.log
252+
retention-days: 14
253+
if-no-files-found: warn
233254
- name: Stop demo cluster
234255
if: always()
235256
run: |

.github/workflows/jepsen-test.yml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,33 @@ jobs:
194194
--partition-count 4 --group-count 6 \
195195
--drain-time 15 \
196196
--sqs-ports 63501,63502,63503 --host 127.0.0.1
197+
- name: Dump demo cluster logs on failure
198+
if: failure()
199+
# Same rationale as jepsen-test-scheduled.yml: inline summary
200+
# for the GH UI + full per-node logs uploaded as artifact so
201+
# any Jepsen anomaly can be traced to the server-side ts/raft
202+
# state without re-running the workflow.
203+
run: |
204+
for node in n1 n2 n3; do
205+
log=/tmp/elastickv-demo-${node}.log
206+
echo "=== ${node}: first 200 lines (startup) ==="
207+
head -n 200 "$log" || true
208+
echo "=== ${node}: last 500 lines (most recent activity) ==="
209+
tail -n 500 "$log" || true
210+
echo "=== ${node}: full log line count ==="
211+
wc -l "$log" || true
212+
done
213+
- name: Upload demo cluster logs on failure
214+
if: failure()
215+
uses: actions/upload-artifact@v4
216+
with:
217+
name: elastickv-demo-logs
218+
path: |
219+
/tmp/elastickv-demo-n1.log
220+
/tmp/elastickv-demo-n2.log
221+
/tmp/elastickv-demo-n3.log
222+
retention-days: 14
223+
if-no-files-found: warn
197224
- name: Stop demo cluster
198225
if: always()
199226
run: |

0 commit comments

Comments
 (0)