sched: switch to BESTWAIT hardware interrupt steering, retire software lowprio

Replace the software reschedule-interrupt steering with the hardware
BESTWAIT/SCHEDCFG comparator (V85 spec Rev AB, section 6.5). The hardware
raises RESCHED_INT whenever any HW thread's STID.PRIO is strictly worse than
the armed BESTWAIT value and then resets BESTWAIT to 0x1FF, so the kernel no
longer needs to designate a victim thread in software.

TARGET=ref passes on archs 68/73/81. TARGET=opt is intentionally NOT converted
yet -- the opt *.Svariants still implement the old software model and will be
rewritten in a follow-up after this ref change is reviewed.

Enable hardware steering
-------------------------
* boot.ref.S: set imask = 0 on all threads so every HW thread is a qualified
  RESCHED target. Previously imask was closed (-1) and software opened exactly
  one victim's RESCHED bit via lowprio_imask().
* setup.ref.c: SCHEDCFG.EN | INTNO(RESCHED_INT) arms the feature at boot.

BESTWAIT arming moved into the readylist API
---------------------------------------------
Arming the comparator is now the readylist's job. New *_arm variants
(H2K_ready_append_arm / _insert_arm / _remove_arm) call H2K_set_bestwait(
H2K_ready_best_prio()); the plain variants do not touch BESTWAIT. Callsites
choose based on whether they are a wake/sleep transition or a mid-reschedule
step:

  arm  : intpool, popup, futex_common (wakes that go straight to H2K_switch),
         create, vmop kill paths, vmint deliver, resched, and getbest.
  noarm: yield and futex_pi raise -- these are immediately followed by
         H2K_dosched on the same HW thread, which re-arms via getbest.

resched() must arm: the hardware fire that delivered the interrupt already
reset BESTWAIT to 0x1FF, so re-arming immediately closes the disarmed window
before dosched's getbest/cluster-search runs. Voluntary paths (yield/hwconfig)
did not fire and still hold a valid armed value, so they can defer to dosched.

Delete the software steering subsystem
---------------------------------------
* Removed sched/lowprio/ entirely (lowprio_imask/highprio_imask/priomask
  designated-victim machinery and H2K_lowprio_notify).
* Removed the priomask block from dosched.ref.c and the H2K_lowprio_notify
  call site; deleted check_sanity.ref.c (H2K_check_sanity / _unlock). Callers
  now BKL_UNLOCK and return directly (futex_classic, futex_pi, create).
* switch.ref.S: sleep path no longer writes priomask; opens imask fully;
  switch-in path returns via k0unlock/jumpr r31 instead of jumping to the
  deleted check_sanity_unlock.

Retire the software runlist mirror
-----------------------------------
* Removed data/runlist/ and checkers/checker_runlist/. The runlist[]/
  runlist_prios[] mirror and the wip_dummy_runlist* timing shim only fed
  H2K_runlist_worst_prio() for software steering; nothing reads it once
  steering is in hardware.
* Every H2K_runlist_push/_remove replaced by a direct
  status = H2K_STATUS_RUNNING (dosched, intpool, popup, setup boot path).

futex_pi priority sync
----------------------
PI boost/restore now writes dest->prio and set_thread_stid_prio() directly
(no runlist mirror). unlock_pi restores me->base_prio in software and hardware
STID up front, before releasing, so the BESTWAIT comparator reads the correct
priority and cannot hide the unblocked holder behind a stale boosted value.

Spec-mandated synchronization
-----------------------------
switch.ref.S adds isync after STID writes (V85 spec: "After updating the STID,
an isync instruction must be executed to ensure that the new STID value is
fully observed"). This is a correctness requirement, not removable overhead --
and the change is still faster overall (see below).

CLUSTER_SCHED guards
--------------------
Convert every `#ifdef CLUSTER_SCHED` to `#if CLUSTER_SCHED` so the single
`#define CLUSTER_SCHED 0/1` in h2_common_defs.h actually gates the cluster/coproc
code. Behavior is identical to the previously-tested build: the cluster paths
were already runtime-gated by `cluster`sched == 0`, so compiling them out
changes nothing observable. H2K_ready_head reduces to returning the ready-list
head; getbest's sleep branch reduces to `return NULL`.

Performance (futex/pi TOTAL TEST TIME, pcycles)
-----------------------------------------------
  ARCHV   before (SW)   after (HW)   delta
  68      1,813,302     1,731,697    -81,605  (-4.5%)
  73      1,833,006     1,765,404    -67,602  (-3.7%)
  81      1,968,943     1,889,073    -79,870  (-4.1%)

Faster on every arch despite the added spec-mandated isyncs, from removing
per-schedule software work: the runlist_worst_prio() walk, priomask
bookkeeping, H2K_lowprio_notify, and the mirror writes.

Follow-ups (not in this change)
-------------------------------
* Rewrite the opt *.Svariants to the pure-BESTWAIT model; TARGET=opt failures
  are expected until then.
* Research whether wait_mask is still needed for cluster scheduling; if not,
  drop it and its write sites in resched/intpool/popup/switch.ref.S/setup.

Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

Author: zbelinsk

kernel/CMakeLists.txt

@@ -16,7 +16,6 @@ set(
   "data/readylist/readylist.ref.c"
   "data/intconfig/intconfig.ref.c"
   "data/stacks/stacks.ref.S"
-  "data/runlist/runlist.ref.c"
   "data/globals/globals.ref.c"
   "data/context/context.v${ARCHV}opt.S"
   "data/vm/vm.ref.c"

kernel/bugs

@@ -37,17 +37,3 @@ Other specing / assertions
 	* For every data element in a struct, spec where it should be updated
 	* Add collection of assertions that we can check on function entry/exit
 	* Add structure verifier that can make sure structures are sane
-
-
-
-runlist valid states
-====================
-
-An empty runlist is apparently not a valid state; should probably
-document this in the spec and maybe make an assertion.
-
-
-
-
-
-

kernel/build/offsets/offsets.ref.c

@@ -119,7 +119,7 @@ int main(int argc, char **argv)
 	PRINT_KG_OFFSET(hthreads_mask);
 	PRINT_KG_OFFSET(hthreads);
 
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 	PRINT_KG_OFFSET(cluster_clusters);
 	PRINT_KG_OFFSET(cluster_hthreads);
 	PRINT_KG_OFFSET(cluster_mask);

kernel/checkers/checker_runlist/checker_runlist.check.c

@@ -97,7 +97,7 @@ typedef struct {
 	pa_t gpio_reg;
 #endif
 
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 	u32_t cluster_clusters;   // number of clusters
 	u32_t cluster_hthreads;   // hardware threads per cluster
 	u32_t cluster_mask[4];    // bitmask of threads in cluster
@@ -146,8 +146,6 @@ typedef struct {
 #endif
 	H2K_thread_context *runlist[MAX_HTHREADS];
 	s16_t runlist_prios[(MAX_HTHREADS+7)/8*8] __attribute__((aligned(8)));
-	H2K_thread_context *wip_dummy_runlist[MAX_HTHREADS];
-	s16_t wip_dummy_runlist_prios[(MAX_HTHREADS+7)/8*8] __attribute__((aligned(8)));
 	H2K_vmblock_t *vmblocks[H2K_ID_MAX_VMS];
 	u32_t phys_offset;
 	u32_t build_id;
@@ -218,7 +216,7 @@ static inline H2K_kg_t PURITY *H2K_gp_llvm()
 #undef PURITY
 #endif
 
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 
 static inline u32_t H2K_hthread_cluster(u32_t hthread) {
 	return (hthread / H2K_gp->cluster_hthreads);

kernel/checkers/checker_runlist/checker_runlist.h

@@ -69,7 +69,7 @@ void H2K_kg_init(u32_t phys_offset, u32_t multicore_shift, u32_t devpage_offset,
 		have_hvx = (H2K_cfg_table(CFG_TABLE_COPROC_TYPE) & CFG_TABLE_COPROC_TYPE_HVX_MASK) != 0;
 		have_silver = (H2K_cfg_table(CFG_TABLE_COPROC_TYPE) & CFG_TABLE_COPROC_TYPE_SILVER_MASK) != 0;
 		H2K_kg.coproc_contexts = (have_hvx || have_silver ? H2K_cfg_table(CFG_TABLE_COPROC_CONTEXTS) : 0);
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 		/* FIXME: need a cfg_table entry for this */
 		H2K_kg.cluster_clusters = (u32_t)(Q6_R_popcount_P(H2K_cfg_table(CFG_TABLE_HTHREADS_MASK)) > 8 ? 4 : 2); // hack
 		H2K_kg.cluster_hthreads = (u32_t)(Q6_R_popcount_P(H2K_cfg_table(CFG_TABLE_HTHREADS_MASK)) / H2K_kg.cluster_clusters);
@@ -144,7 +144,7 @@ void H2K_kg_init(u32_t phys_offset, u32_t multicore_shift, u32_t devpage_offset,
 		H2K_kg.hmx_units = (H2K_cfg_table(CFG_TABLE_HMX_INT8_RATE) != 0);  // exists?
 		H2K_kg.info_boot_flags.boot_have_hmx = (H2K_kg.hmx_units > 0);
 #endif
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 		H2K_kg.coproc_max_save = ((H2K_kg.coproc_contexts + H2K_kg.hmx_units) / H2K_kg.cluster_clusters) + (((H2K_kg.coproc_contexts + H2K_kg.hmx_units) % H2K_kg.cluster_clusters) != 0);
 		H2K_kg.coproc_max_save = (H2K_kg.coproc_max < CLUSTER_SCHED_MIN_COPROCS ? CLUSTER_SCHED_MIN_COPROCS : H2K_kg.coproc_max);
 #endif	
@@ -230,7 +230,7 @@ void H2K_kg_init(u32_t phys_offset, u32_t multicore_shift, u32_t devpage_offset,
 
 }
 
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 void H2K_cluster_config(void) {
 	u32_t i;
 

kernel/checkers/checker_runlist/make.inc

@@ -13,7 +13,6 @@
 #include <globals.h>
 #include <hw.h>
 #include <log.h>
-#include <runlist.h>
 
 /* Get the best ready priority */
 static inline u32_t H2K_ready_best_prio()
@@ -86,7 +85,29 @@ static inline void H2K_ready_remove(H2K_thread_context *thread)
 	if (H2K_gp->ready[prio] == NULL) H2K_ready_clear_prio(prio);
 }
 
-#ifdef CLUSTER_SCHED
+static inline void H2K_ready_append_arm(H2K_thread_context *thread)
+{
+	H2K_ready_append(thread);
+	H2K_set_bestwait(H2K_ready_best_prio());
+}
+
+/* Take the thread and place it in the ready structure,
+ * the first thread to be scheduled at its priority */
+static inline void H2K_ready_insert_arm(H2K_thread_context *thread)
+{
+	H2K_ready_insert(thread);
+	H2K_set_bestwait(H2K_ready_best_prio());
+}
+
+/* Remove a specific thread from the ready list */
+/* The caller guarantees that the thread is actually in the ready list correctly */
+static inline void H2K_ready_remove_arm(H2K_thread_context *thread)
+{
+	H2K_ready_remove(thread);
+	H2K_set_bestwait(H2K_ready_best_prio());
+}
+
+#if CLUSTER_SCHED
 static inline void H2K_update_coprocs(u32_t hthread, u32_t hthread_xe, u32_t hthread_xe2, u32_t hthread_xe3,u32_t head_xe, u32_t head_xe2, u32_t head_xe3) {
 	xex(hthread, head_xe, head_xe2, head_xe3, hthread_xe, hthread_xe2, hthread_xe3);
 	if (hthread_xe) {
@@ -123,7 +144,7 @@ static inline void H2K_update_coprocs(u32_t hthread, u32_t hthread_xe, u32_t hth
 static inline H2K_thread_context *H2K_ready_head(u32_t prio, u32_t hthread) {
 	H2K_thread_context *head = H2K_gp->ready[prio];
 
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 	if ((!H2K_gp->cluster_sched) || H2K_gp->coproc_max == -1) {
 		return head;
 	}
@@ -223,7 +244,7 @@ static inline H2K_thread_context *H2K_ready_getbest(u32_t hthread)
 	H2K_log("hthread %d  getbest\n", hthread);
 	prio = H2K_ready_best_prio();
 	if (prio >= MAX_PRIOS) {  // !H2K_ready_any_valid(), go to sleep
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 		if (!H2K_gp->cluster_sched) {
 			return NULL;
 		}
@@ -246,7 +267,7 @@ static inline H2K_thread_context *H2K_ready_getbest(u32_t hthread)
 
 	ret = H2K_ready_head(prio, hthread);
 	if (ret != NULL) {
-		H2K_ready_remove(ret);
+		H2K_ready_remove_arm(ret);
 	}
 	return ret;
 }

kernel/data/globals/globals.h

@@ -90,7 +90,7 @@ H2K_thread_context *H2K_ready_getbest_TB()
 	return H2K_ready_getbest(CURRENT_HTHREAD);
 }
 
-#ifdef CLUSTER_SCHED
+#if CLUSTER_SCHED
 void H2K_ready_REG_SSR_XE_CLEAR_TB()
 {
 	H2K_set_ssr(H2K_get_ssr() & ~SSR_XE_BIT_MASK);
@@ -230,7 +230,7 @@ __asm__ __volatile(GLOBAL_REG_STR " = %0 " : : "r"(&H2K_kg));
 	if (H2K_ready_getbest_TB() != &c) FAIL("ready_best_prio failed (c) ");
 	if (H2K_ready_getbest_TB() != NULL) FAIL("ready_best_prio failed (empty) ");
 
-# ifdef CLUSTER_SCHED
+# if CLUSTER_SCHED
 
 	u32_t hthreadmask = H2K_cfg_table(CFG_TABLE_HTHREADS_MASK);
 	u32_t hthreads = (u32_t)Q6_R_popcount_P(hthreadmask);