vmop: add VMOP_KILL_THREAD and VMOP_KILL_CHILD (#65)

* vmop: fix killed-thread escape races and add VMOP_KILL_THREAD/KILL_CHILD

Fixes the v68/opt pi-test hang (and the underlying correctness hole it
exposed) and wires up a proper cross-thread kill API.

## Root cause

A thread marked for death (KILL|VMWORK set by kill_thread_locked) could
still be on its way into a blocking syscall.  While it spun for the BKL
it was still RUNNING, so the per-state kill action had already fired.
Once it took the BKL the gate in futex_classic/futex_pi checked
VMWORK&&IE — not VMWORK alone.  Kill leaves IE alone, so with IE clear
the thread sailed past, set STATUS_BLOCKED, and parked in a wait ring.
Nobody reaped it; num_cpus never drained; the booter (no timer fallback)
slept forever.

A second bug: after h2_vmkill(VMOP_KILL_VM_SELF) some siblings died
mid-cleanup holding pthread_exit_lock.  The main thread's pthread_exit
tried to acquire it and blocked forever (num_cpus stuck at 1) because
the original kill loop skipped the caller with "if (target == me)
continue".

## Changes

vmop.ref.c / vmop.h / h2_vm.h
  - Add VMOP_KILL_THREAD and VMOP_KILL_CHILD operations.  KILL_THREAD
    sets KILL+VMWORK on a target and forces it out of every state
    (VMWAIT, RUNNING, INTBLOCKED, BLOCKED, READY) ignoring guest IE.
    KILL_CHILD iterates a child VM's contexts[] under one BKL section.
  - Remove the "if (target == me) continue" skip in the kill_vm loop so
    the caller also receives KILL|VMWORK+IPI; the IPI fires before any
    mutex can be acquired after vmkill returns.

futex_classic / futex_pi (ref + v4opt.S)
  - Change the VMWORK gate from VMWORK&&IE to VMWORK alone, matching
    what H2K_switch already does.  Gate calls vm_do_work_withlock (BKL
    held), which handles KILL (never returns) or clears VMWORK and
    returns -1 so the caller retries.

popup.ref.c / intpool.ref.c
  - Add the same VMWORK gate (these had no check at all).
  - popup.v4opt.S: add allocframe(#0) to make the function non-leaf,
    add vmstatus load + bitsset after k0lock, jump to fail label that
    calls vm_do_work_withlock; change jumpr r31 → dealloc_return at all
    early-return paths.
  - Note: h2_intwait (trap 11) routes to H2K_popup_wait; covered here.

vmwork.ref.c
  - Remove the IE guard around the VMWORK clear.  The interrupt
    controller retains the pending interrupt, so clearing VMWORK with IE
    disabled cannot lose work — it is re-discovered when the guest
    re-enables IE.

pthread_thread.ref.c
  - Add h2_thread_stop_trap in the last==true branch of pthread_exit as
    defense-in-depth (now redundant given the kill-loop fix above, but
    kept as a hard guarantee).

specs and docs
  - vmdefs.spec: document VMWORK contract ("reconsider before blocking").
  - vmwork.spec: drop "and interrupts are enabled" from the clear rule.
  - futex_classic.spec, futex_pi.spec, popup.spec, intpool.spec: state
    the spurious-failure / retry contract for all block primitives.

tests
  - futex/test/tests/vmwork: new directed unit test; cases B (VMWORK
    alone, no IE) and C (VMWORK|KILL) fail pre-fix, pass post-fix for
    both ref and opt builds.
  - vmwork/test: assert VMWORK is cleared after every non-KILL call,
    including the IE-disabled path.
  - vmop/test: updated assertions for new caller-kill behavior;
    KILL_THREAD and KILL_CHILD coverage.
  - popup/test, intpool/test: cases B and C mirroring the futex pattern.
  - libs/posix/pthread/test_h2: new pthread test suite covering join,
    detach, barriers, condvars, rwlocks, semaphores, TLS, and stuck-thread
    scenarios that exercise the vmkill path end-to-end.

Verified: 1224 passed, 0 failed across all 9 variants
(v68/v73/v81 x opt/ref/opt_si).  Previously v68/opt and v81/opt_si
pi-test hung on every run.

Signed-off-by: Erich Plondke <erich@qti.qualcomm.com>

* vmwork: remove H2K_vm_do_work wrapper; callers acquire BKL directly

H2K_vm_do_work was a thin BKL acquire/call-withlock/release wrapper
with a single call site (vmipi.ref.c).  Inline the lock pair there
and delete the wrapper, leaving H2K_vm_do_work_withlock as the sole
public entry point.

- vmipi.ref.c: inline BKL_LOCK/UNLOCK around _withlock call; drop
  duplicate #include <hw.h>
- vmipi/test: stub H2K_vm_do_work_withlock instead of H2K_vm_do_work
- vmwork.ref.c: remove wrapper; drop now-unused #include <hw.h>
- vmwork.h: remove H2K_vm_do_work declaration
- vmwork.spec: remove H2K_vm_do_work section; update description

1224/1224 tests pass across 9 variants.

Signed-off-by: Erich Plondke <erich@qti.qualcomm.com>

* popup,vmop: fix popup_cancel <<#2 typo and add dispatcher/cancel coverage tests

popup.v4opt.S: H2K_popup_cancel used <<#2 (4-byte stride) when indexing
inthandlers[], which is an array of 8-byte H2K_inthandler_t entries.
For any intnum != 0 the wrong slot was cleared (e.g. cancelling intnum=2
cleared intnum=1's handler) and odd intnums produced a misaligned 64-bit
store in the opt build.  Fix to <<#3 to match H2K_popup_int and
H2K_popup_wait which already use the correct stride.

vmop/test: add dispatcher coverage -- OOB op codes (including -1, 100,
and VMOP_MAX) must return -1, and VMOP_KILL_THREAD / VMOP_KILL_VM must
dispatch correctly through H2K_vmop().  Declare H2K_vmop() in vmop.h so
the test can call the dispatcher directly.

popup/test: add H2K_popup_cancel coverage -- simulate the INTBLOCKED
state left by popup_wait (handler registered, r00=intnum) and verify
that cancel clears the handler slot and resets r00 to -1.

Signed-off-by: Erich Plondke <erich@qti.qualcomm.com>

* build: fix two bugs that broke make cov

Makefile.inc.test: try_gmon_out is PHONY so it re-runs on every cov
invocation, but hexagon-gmon-merge refuses to overwrite an existing
gmon.out (exits 255).  If a previous run left gmon.out behind (e.g.
after a partial cov run following testall), subsequent cov runs would
fail for every affected test.  Fix by removing gmon.out at the top of
try_gmon_out before the merge.

makefile: the build recipe wrote "v$(ARCHV) $@ ..." to the ver file,
so a plain "make build" produced "v81 build ..." instead of "v81 opt ...".
merge_cov.py expects the second token to be "opt" or "ref" and exits
with an error otherwise, preventing the merged cov.txt/cov.rpt from
being generated.  Fix by using $(TARGET) instead of $@.

Signed-off-by: Erich Plondke <erich@qti.qualcomm.com>

* build: don't symlink cov.txt from source tree into cov build dir

The cov.txt rule symlinked all non-Makefile.inc files from the source
directory into the per-test build directory.  If a stale cov.txt
happened to exist in the source tree (e.g. from a previous cov run that
wrote through an earlier symlink), it would be linked in, causing the
build-dir cov.txt target to write back through it into the source tree.
On the next run, try_gmon_out would find gmon.out already present (from
the previous testall run) and hexagon-gmon-merge would exit 255 with
"gmon.out: File exists."

Fix by excluding cov.txt and cov-*.txt from the symlink step, matching
how generated_tests.dat and scenarios.h are already excluded.

Signed-off-by: Erich Plondke <erich@qti.qualcomm.com>

---------

Signed-off-by: Erich Plondke <erich@qti.qualcomm.com>

Author: eplondke

kernel/event/intpool/intpool.ref.c

@@ -13,6 +13,7 @@
 #include <readylist.h>
 #include <switch.h>
 #include <dosched.h>
+#include <vmwork.h>
 
 /*
  * EJP: FIXME: Should there be a single intpool for a vmblock? 
@@ -72,6 +73,11 @@ int H2K_intpool_wait(u32_t int_ack_num, H2K_thread_context *me)
 		H2K_intcontrol_enable(int_ack_num);
 	}
 	BKL_LOCK(&H2K_bkl);
+	if (me->vmstatus & H2K_VMSTATUS_VMWORK) {
+		H2K_vm_do_work_withlock(me);
+		BKL_UNLOCK(&H2K_bkl);
+		return -1;
+	}
 	if (unlikely(vmblock->intpool_anypending)) {
 		if ((intno = get_pending_interrupt(vmblock)) >= 0) {
 			BKL_UNLOCK(&H2K_bkl);

kernel/event/intpool/intpool.spec

@@ -51,16 +51,22 @@ Note that this functionality is very similar to H2K_popup_int
 H2K_intpool_int
 ---------------
 
-.. c:function:: H2K_intpool_wait(u32_t int_ack_num, H2K_thread_context *me)
+.. c:function:: int H2K_intpool_wait(u32_t int_ack_num, H2K_thread_context *me)
 
 	:param int_ack_num: Interrupt to enable, or -1 for no interrupt
 	:param me: Context for the current thread
+	:returns: the interrupt number on wake, -1 on failure (including spurious VMWORK bail)
 
 Description
 ~~~~~~~~~~~
 
 This function optionally enables an interrupt, and then waits for an interrupt to arrive.
 
+Per the VMWORK contract (see :doc:`../../util/vmdefs/vmdefs`), -1 may
+be returned spuriously when VMWORK is observed after the BKL is
+acquired.  Callers must treat -1 as "did not block -- re-check the
+wait condition and retry."
+
 Functionality
 ~~~~~~~~~~~~~
 

kernel/event/intpool/test/test.c

@@ -19,6 +19,7 @@
 #include <checker_ready.h>
 #include <setjmp.h>
 #include <globals.h>
+#include <vmdefs.h>
 
 void FAIL(const char *str)
 {
@@ -27,6 +28,14 @@ void FAIL(const char *str)
 	exit(1);
 }
 
+static int TH_saw_do_work;
+
+int H2K_vm_do_work_withlock(H2K_thread_context *me)
+{
+	TH_saw_do_work = 1;
+	return -1;
+}
+
 H2K_thread_context *TB_in;
 
 u32_t TB_saw_dosched = 0;
@@ -358,6 +367,28 @@ int main()
 		if ((fakeint[i/32+0x200/sizeof(u32_t)-1] & (1<<(i&0x1f))) == 0) FAIL("not acked");
 	}
 
+	/* Case B: VMWORK alone -- gate must fire and return -1 without INTBLOCKED */
+	H2K_runlist_push(&a);
+	a.status = H2K_STATUS_RUNNING;
+	a.vmstatus = H2K_VMSTATUS_VMWORK;
+	TH_saw_do_work = 0;
+	if (TH_call_intpool_wait(0, &a) >= 0) FAIL("B: intpool_wait should fail with VMWORK set");
+	if (!TH_saw_do_work) FAIL("B: VMWORK alone must call vm_do_work");
+	if (a.status == H2K_STATUS_INTBLOCKED) FAIL("B: must not set INTBLOCKED");
+	a.vmstatus = 0;
+	H2K_runlist_remove(&a);
+
+	/* Case C: VMWORK|KILL -- killed thread racing into intpool_wait */
+	H2K_runlist_push(&a);
+	a.status = H2K_STATUS_RUNNING;
+	a.vmstatus = H2K_VMSTATUS_VMWORK | H2K_VMSTATUS_KILL;
+	TH_saw_do_work = 0;
+	if (TH_call_intpool_wait(0, &a) >= 0) FAIL("C: intpool_wait should fail with VMWORK|KILL");
+	if (!TH_saw_do_work) FAIL("C: VMWORK|KILL must call vm_do_work");
+	if (a.status == H2K_STATUS_INTBLOCKED) FAIL("C: must not set INTBLOCKED");
+	a.vmstatus = 0;
+	H2K_runlist_remove(&a);
+
 	puts("TEST PASSED\n");
 	return 0;
 }

kernel/event/popup/popup.ref.c

@@ -15,6 +15,7 @@
 #include <switch.h>
 #include <max.h>
 #include <intcontrol.h>
+#include <vmwork.h>
 
 void H2K_popup_int(u32_t intnum, H2K_thread_context *me, u32_t hwtnum, H2K_thread_context *woken)
 {
@@ -50,6 +51,11 @@ int H2K_popup_wait(u32_t intnum, H2K_thread_context *me)
 	if (intnum == L2_CORE_INTERRUPT) return -1;
 
 	BKL_LOCK(&H2K_bkl);
+	if (me->vmstatus & H2K_VMSTATUS_VMWORK) {
+		H2K_vm_do_work_withlock(me);
+		BKL_UNLOCK(&H2K_bkl);
+		return -1;
+	}
 	if (H2K_gp->inthandlers[intnum].param != NULL) {
 		BKL_UNLOCK(&H2K_bkl);
 		return -1;

kernel/event/popup/popup.spec

@@ -45,16 +45,22 @@ interrupt disabled, we allow it to pend while the popup thread executes.
 H2K_popup_int
 -------------
 
-.. c:function:: H2K_popup_wait(u32_t intno, H2K_thread_context *me)
+.. c:function:: int H2K_popup_wait(u32_t intno, H2K_thread_context *me)
 
 	:param intno: Interrupt number
 	:param me: Context for the current thread
+	:returns: the interrupt number on wake, -1 on failure (including spurious VMWORK bail or cancel)
 
 Description
 ~~~~~~~~~~~
 
 This function blocks a thread to waiting for an interrupt.
 
+Per the VMWORK contract (see :doc:`../../util/vmdefs/vmdefs`), -1 may
+be returned spuriously when VMWORK is observed after the BKL is
+acquired.  Callers must treat -1 as "did not block -- re-check the
+wait condition and retry."
+
 Functionality
 ~~~~~~~~~~~~~
 

kernel/event/popup/popup.v4opt.S

@@ -137,14 +137,27 @@ FUNC_START H2K_popup_wait .text.int.popup .falign
 #define ACKBASE r12
 #define MASK r13
 #define RETURN_PAIR r15:14
+	{
+		allocframe(#0)
+		HWTNUM = memub(ME+#CONTEXT_hthread)
+	}
 	{
 		p0 = cmp.gtu(INTNO,#MAX_INTERRUPTS-1)
 		if (p0.new) jump:nt 9f
 		p1 = cmp.eq(INTNO, #L2_CORE_INTERRUPT)
 		if (p1.new) jump:nt 9f
-		HWTNUM = memub(ME+#CONTEXT_hthread)
 	}
 	k0lock
+#define TMP INTHANDLERS_ADDR
+	{
+		TMP = memub(ME+#CONTEXT_vmstatus)
+		MASK = #H2K_VMSTATUS_VMWORK
+	}
+	{
+		p0 = bitsset(TMP,MASK)
+		if (p0.new) jump:nt .Lpopup_wait_vmwork
+	}
+#undef TMP
 	{
 		PARAM_HANDLER = combine(ME,##H2K_popup_int)
 		INTHANDLERS_ADDR = add(H2K_GP,#KG_inthandlers)
@@ -199,7 +212,17 @@ FUNC_START H2K_popup_wait .text.int.popup .falign
 9:
 	{
 		r0 = #-1
-		jumpr r31
+		dealloc_return
+	}
+.Lpopup_wait_vmwork:
+	{
+		r0 = ME
+		call H2K_vm_do_work_withlock
+	}
+	k0unlock
+	{
+		r0 = #-1
+		dealloc_return
 	}
 #undef INTNO
 #undef ME
@@ -230,7 +253,7 @@ FUNC_START H2K_popup_cancel .text.int.popup
 		LLZERO = #0
 	}
 	{
-		memd(INTHANDLERS+INTNO<<#2) = LLZERO
+		memd(INTHANDLERS+INTNO<<#3) = LLZERO
 		memw(DEST+#CONTEXT_r00) = #-1
 		jumpr r31
 	}

kernel/event/popup/test/test.c

@@ -18,6 +18,7 @@
 #include <checker_ready.h>
 #include <setjmp.h>
 #include <globals.h>
+#include <vmdefs.h>
 
 void FAIL(const char *str)
 {
@@ -26,6 +27,14 @@ void FAIL(const char *str)
 	exit(1);
 }
 
+static int TH_saw_do_work;
+
+int H2K_vm_do_work_withlock(H2K_thread_context *me)
+{
+	TH_saw_do_work = 1;
+	return -1;
+}
+
 H2K_thread_context *TB_in;
 
 u32_t TB_saw_dosched = 0;
@@ -312,6 +321,54 @@ int main()
 		TH_check_old(&a);
 		TH_clear_ready(&a);
 	}
+	/* Case B: VMWORK alone -- gate must fire and return -1 without INTBLOCKED */
+	H2K_runlist_push(&a);
+	a.status = H2K_STATUS_RUNNING;
+	a.vmstatus = H2K_VMSTATUS_VMWORK;
+	TH_saw_do_work = 0;
+	if (TH_call_popup_wait(0, &a) >= 0) FAIL("B: popup_wait should fail with VMWORK set");
+	if (!TH_saw_do_work) FAIL("B: VMWORK alone must call vm_do_work");
+	if (a.status == H2K_STATUS_INTBLOCKED) FAIL("B: must not set INTBLOCKED");
+	a.vmstatus = 0;
+	H2K_runlist_remove(&a);
+
+	/* Case C: VMWORK|KILL -- killed thread racing into popup_wait */
+	H2K_runlist_push(&a);
+	a.status = H2K_STATUS_RUNNING;
+	a.vmstatus = H2K_VMSTATUS_VMWORK | H2K_VMSTATUS_KILL;
+	TH_saw_do_work = 0;
+	if (TH_call_popup_wait(0, &a) >= 0) FAIL("C: popup_wait should fail with VMWORK|KILL");
+	if (!TH_saw_do_work) FAIL("C: VMWORK|KILL must call vm_do_work");
+	if (a.status == H2K_STATUS_INTBLOCKED) FAIL("C: must not set INTBLOCKED");
+	a.vmstatus = 0;
+	H2K_runlist_remove(&a);
+
+	/* popup_cancel: thread was waiting on interrupt i; cancel must clear the
+	 * handler slot and reset r00 to -1. */
+	{
+		int i;
+		for (i = 0; i < MAX_INTERRUPTS; i++) {
+#if ARCHV >= 4
+			if (i == L2_CORE_INTERRUPT) continue;
+#endif
+			TH_clear_popups();
+			H2K_runlist_push(&a);
+			/* Simulate the state popup_wait leaves: INTBLOCKED, r00=intnum, handler set */
+			a.status = H2K_STATUS_INTBLOCKED;
+			a.r0100 = i;
+			H2K_gp->inthandlers[i].handler = H2K_popup_int;
+			H2K_gp->inthandlers[i].param = &a;
+
+			BKL_LOCK();
+			H2K_popup_cancel(&a);
+			BKL_UNLOCK();
+
+			if (H2K_gp->inthandlers[i].raw != 0) FAIL("cancel: handler not cleared");
+			if (a.r00 != (u32_t)-1) FAIL("cancel: r00 not reset");
+			H2K_runlist_remove(&a);
+		}
+	}
+
 	puts("TEST PASSED\n");
 	return 0;
 }

kernel/futex/futex/test/tests/vmwork/test.c

@@ -18,13 +18,21 @@ void FAIL(const char *str)
 	exit(1);
 }
 
+int n_fails = 0;
+void CHECK_FAIL(const char *str)
+{
+	printf("FAIL: %s\n", str);
+	n_fails++;
+}
+#define CHECK(cond, msg) do { if (!(cond)) CHECK_FAIL(msg); } while (0)
+
 H2K_thread_context a;
 
 int TH_saw_do_work;
 int TH_work_return;
 int TH_expected_safemem;
 
-int H2K_vm_do_work(H2K_thread_context *me)
+int H2K_vm_do_work_withlock(H2K_thread_context *me)
 {
 	if (me != &a) FAIL("bad context");
 	TH_saw_do_work = 1;
@@ -43,28 +51,55 @@ u32_t H2K_safemem_check_and_lock(void *va, u32_t perms, pa_t *out, H2K_thread_co
 int main()
 {
 	u32_t i = 0;
+
+	/* Case A: VMWORK|IE -- the original gate fires today (and post-fix).
+	 * Expected to pass both pre- and post-fix. */
 	a.vmstatus = H2K_VMSTATUS_VMWORK | H2K_VMSTATUS_IE;
 
 	TH_saw_do_work = 0;
-	if (H2K_futex_wait(&i,i,&a) != -1) FAIL("return val");
-	if (TH_saw_do_work == 0) FAIL("no vmwork");
+	CHECK(H2K_futex_wait(&i,i,&a) == -1, "A: wait return val");
+	CHECK(TH_saw_do_work != 0, "A: wait did not call vm_do_work");
 
 	TH_saw_do_work = 0;
-	if (H2K_futex_lock_pi(&i,&a) != -1) FAIL("pi return val");
-	if (TH_saw_do_work == 0) FAIL("no vmwork");
+	CHECK(H2K_futex_lock_pi(&i,&a) == -1, "A: lock_pi return val");
+	CHECK(TH_saw_do_work != 0, "A: lock_pi did not call vm_do_work");
 
+	/* Case B: VMWORK alone -- the no-block rule from vmdefs.spec.  Pre-fix
+	 * the gate is VMWORK&&IE so falls through to safemem (allowed via
+	 * TH_expected_safemem) and returns -1 via the safemem-fail path with
+	 * vm_do_work never called.  Post-fix the gate fires on VMWORK alone. */
 	a.vmstatus = H2K_VMSTATUS_VMWORK;
 
 	TH_saw_do_work = 0;
 	TH_expected_safemem = 1;
-	if (H2K_futex_wait(&i,i,&a) != -1) FAIL("return val");
-	if (TH_saw_do_work != 0) FAIL("vmwork");
+	CHECK(H2K_futex_wait(&i,i,&a) == -1, "B: wait return val");
+	CHECK(TH_saw_do_work != 0, "B: VMWORK alone must gate (wait)");
+
+	TH_saw_do_work = 0;
+	TH_expected_safemem = 1;
+	CHECK(H2K_futex_lock_pi(&i,&a) == -1, "B: lock_pi return val");
+	CHECK(TH_saw_do_work != 0, "B: VMWORK alone must gate (lock_pi)");
+
+	/* Case C: VMWORK|KILL -- the killed-thread escape from VMKILL_race.md.
+	 * kill_thread_locked sets KILL|VMWORK and the victim is on its way into
+	 * a blocking syscall.  Pre-fix same failure mode as case B (gate is
+	 * VMWORK&&IE; KILL isn't IE). */
+	a.vmstatus = H2K_VMSTATUS_VMWORK | H2K_VMSTATUS_KILL;
+
+	TH_saw_do_work = 0;
+	TH_expected_safemem = 1;
+	CHECK(H2K_futex_wait(&i,i,&a) == -1, "C: wait return val");
+	CHECK(TH_saw_do_work != 0, "C: VMWORK|KILL must gate (wait)");
 
 	TH_saw_do_work = 0;
 	TH_expected_safemem = 1;
-	if (H2K_futex_lock_pi(&i,&a) != -1) FAIL("pi return val");
-	if (TH_saw_do_work != 0) FAIL("vmwork");
+	CHECK(H2K_futex_lock_pi(&i,&a) == -1, "C: lock_pi return val");
+	CHECK(TH_saw_do_work != 0, "C: VMWORK|KILL must gate (lock_pi)");
 
+	if (n_fails != 0) {
+		puts("FAIL");
+		exit(1);
+	}
 	puts("TEST PASSED\n");
 	return 0;
 }

kernel/futex/futex_classic/futex_classic.ref.c

@@ -24,9 +24,9 @@ s32_t H2K_futex_wait(u32_t *lock, u32_t val, H2K_thread_context *me)
 	u32_t readval;
 	pa_t pa;
 	BKL_LOCK();
-	if ((me->vmstatus & H2K_VMSTATUS_VMWORK) && (me->vmstatus & H2K_VMSTATUS_IE)) {
+	if (me->vmstatus & H2K_VMSTATUS_VMWORK) {
+		H2K_vm_do_work_withlock(me);
 		BKL_UNLOCK();
-		H2K_vm_do_work(me);
 		return -1;
 	}
 	if (!H2K_safemem_check_and_lock(lock,SAFEMEM_R,&pa,me)) {

kernel/futex/futex_classic/futex_classic.spec

@@ -24,6 +24,11 @@ If the value has changed, the kernel returns -1.
 
 If the thread went to sleep, the kernel returns 0.
 
+Per the VMWORK contract (see :doc:`../../util/vmdefs/vmdefs`), -1 may
+also be returned spuriously when VMWORK is observed after the BKL is
+acquired.  Callers must treat -1 as "did not block -- re-check the
+wait condition and retry."
+
 Functionality
 ~~~~~~~~~~~~~