[WIP] Explore BESTWAIT hardware steering for scheduler optimization by zbelinsk · Pull Request #57 · qualcomm/hexagon-hypervisor

zbelinsk · 2026-06-21T20:12:02Z

This commit explores using the BESTWAIT register for hardware-assisted scheduling decisions in the reference implementation. The goal is to investigate whether BESTWAIT can improve scheduler performance or enable different implementation strategies.

Bryan Bayerdorffer (bryanb-h2) · 2026-06-29T15:54:05Z

The STID register (per-thread state) is not accessible from other threads

This prevents reading the effective priority of other threads

Blocks the core comparison logic needed for proper BESTWAIT usage

Might be a good idea, but I don't understand this stuff from the summary:

The STID register (per-thread state) is not accessible from other threads
This prevents reading the effective priority of other threads
Blocks the core comparison logic needed for proper BESTWAIT usage

zbelinsk · 2026-06-29T17:06:33Z

e STID register (per-thread state) i

The STID register (per-thread state) is not accessible from other threads

This prevents reading the effective priority of other threads

Blocks the core comparison logic needed for proper BESTWAIT usage

Might be a good idea, but I don't understand this stuff from the summary:

The STID register (per-thread state) is not accessible from other threads

This prevents reading the effective priority of other threads

Blocks the core comparison logic needed for proper BESTWAIT usage

In the kernel sched code, we are keeping scheduling going, by doing two things:

Unmasking least prioritized hthread.
Sending the interrupt.

Today we have the software runlist array, for achieving the described functionallity- we query it. If we want to offload it to hardware, we need to query somehow- "who is the least prioritized"- which can't be done, since we are not able to access STID.PRIO of hthread Y from hthread X for reading, tho we can set that by using set_thread_stid_prio setter I have added in this pr.

Using BESTWAIT in the current state, while we need to update also the software runlist- produces no benefit (packets counting wise, check_sanity.opt is ~10 packets already).

zbelinsk · 2026-07-08T17:18:57Z

Was thinking about this again, here are my conclusions:

Current state:

The BESTWAIT register currently works as follows:

Receives the lowest ready thread priority value in the lowest 9 bits.
Outputs an indication of "did I fire?" in the lowest 9 bits. The value is 0x1FF if fired, which is bigger than the worst possible priority (0x0FF).
We know for sure that if we send 0x1FF in the lowest 9 bits of BESTWAIT, it will not fire.

What can be added:

BESTWAIT is already a comparator that iterates over all HW threads. This means it can return not only "did I fire?" but also "did I fire, and who is the lowest-priority HW thread?", since the comparison across all HW threads is already being performed.

With that being said, if I want to fire an interrupt for reschedule, the content of the new proposed BESTWAIT would be (all described inputs and outputs are BESTWAIT reg):

input: [XXXXXXXX...XXX | lowest ready prio]
output: if fired -> [highest priority running hw thread | 0x1FF]
if not fired -> [highest priority running hw thread | lowest ready prio]

If I only want to retrieve the highest-priority running thread (without sending an interrupt):

input: [XXXXXXXX...XXX | 0x1FF]
output: always -> [highest priority running | 0x1FF]

Remark that 0x1FF means "cannot interrupt anyone", it is effectively higher than the highest possible priority (0x0FF).

This way, we reuse existing hardware in order to query the lowest running hw thread number at any given time.

Small limitation:

Since a HW thread priority is 8 bits long, a waiting thread is represented as priority 255 (0xFF, i.e. -1).

Consider the case where we have:

A sleeping HW thread (-1 == 0xFF == 255)
A ready thread with priority 255

In this case, we would not fire an interrupt because both values are identical.

To overcome this, we can continue using H2K->wait_mask (which already exists in the current implementation, no further implementation needed), allowing us to distinguish which HW threads are actually waiting.

Big win, offloading from software to hardware:

In check_sanity we keep the implementation proposed in this pr- but making H2K_lowprio_notify() to receive also the highest priority hw thread, since it can be queried from BESTWAIT post interrupt firing.
Also, in dosched, the line: if ((H2K_gp->wait_mask == 0) && (new->prio IS_WORSE_THAN H2K_runlist_worst_prio()))....
Will not be software iterating runlist any more- we can receive it from the new proposed BESTWAIT: input [XXXXXXXX...XXX | 0x1FF] => output: [highest priority running | 0x1FF].

These are the only call sites across the code in which H2K_runlist_* API is used for anything that is not insert to or removing from runlist. Arguably, with this approach we could avoid having the H2K_runlist at all, since handled in hardware.

Erich Plondke (eplondke)

This is looking pretty good! Probably the easy thing for now is to update BESTWAIT at check_sanity time, but then eventually move it to the time where we update the readylist.

There are probably lots of scheduler-related tests that check for how we maintain the IMASK stuff, as we make it simpler a lot of those tests will have to change.

I do think you should add a new document specifically about how we're changing the scheduler, and especially highlight that with the new interrupt logic:

If all the threads have their IMASK enabled for interrupts, the hardware delivers the interrupt to the hardware thread with the worst STID.PRIO field
The hardware will compare the STID.PRIO field for all hardware threads and the BESTWAIT register, and if there is a hardware thread that is worse than the BESTWAIT register it will automatically raise the reschedule interrupt (which would then be steered to the worst hardware thread as above)
This should greatly simplify dosched and check_sanity

Erich Plondke (eplondke) · 2026-07-09T19:12:35Z

 		H2K_runlist_set_thread_prio(dest, prio);
+		/* Sync hardware STID.PRIO so BESTWAIT sees the boosted priority
+		 * immediately -- avoids spurious preemption of the holder. */
+		set_thread_stid_prio(dest->hthread, prio);


This should become the "setprio" instruction after the inline after the change, I think?

Erich Plondke (eplondke) · 2026-07-09T19:13:08Z

+	 * comparator reads STID.PRIO; without the hardware sync the stale
+	 * boosted value hides me from the comparator causing a priority inversion. */
 	H2K_runlist_set_thread_prio(me, me->base_prio);
+	set_thread_stid_prio(me->hthread, me->base_prio);


This could also be the "setprio" instruction or re-writing our whole STID register

Erich Plondke (eplondke) · 2026-07-09T19:14:16Z

@@ -0,0 +1,208 @@
+/*


If we're getting rid of maintaining IMASK a lot of these tests will need to change. The good news is that hopefully they will get simpler?

Erich Plondke (eplondke) · 2026-07-09T19:16:48Z

@@ -11,18 +11,25 @@
 #include <readylist.h>


We can write BESTWAIT here, or we can do it when we update the ready list (that would be more exact!)
Let's assume we update BESTWAIT when we update the ready list.

No need to update lowprio interrupt masks; interrupt steering is in hardware

No need to raise reschedule interrupt, reschedule interrupt raised by hardware

No need to check wait mask / etc

check_sanity becomes empty, check_sanity_unlock becomes just the unlock of the BKL.

Erich Plondke (eplondke) · 2026-07-09T19:18:18Z

 	return imask;
 }

+/* Write STID.PRIO for a named hardware thread (cross-thread, Monitor only).


Erich Plondke (eplondke) · 2026-07-09T19:18:43Z


 	asm volatile ("dmcfgwr(%0, %1);" : : "r" (index), "r" (data));
 }
+


looks good.

Erich Plondke (eplondke) · 2026-07-09T19:18:54Z

 This function retrieves clears the contents of the IPEND status register.


+H2K_get_bestwait


Thank you for your documentation!

This commit explores using the BESTWAIT register for hardware-assisted scheduling decisions in the reference implementation. The goal is to investigate whether BESTWAIT can improve scheduler performance or enable different implementation strategies. ## What was attempted - Added BESTWAIT register support to hw.h/hw.spec - Implemented test infrastructure (H2K_bestwait test suite) - Modified check_sanity.ref.c to explore BESTWAIT-based scheduling - Updated futex_pi and setup reference implementations - Attempted to implement opt version of check_sanity with BESTWAIT BESTWAIT is a global priority register that: - Holds the priority of the highest-priority task waiting to run - Triggers a reschedule interrupt when ANY thread's effective priority becomes worse than the BESTWAIT value - Does NOT deliver interrupts to specific threads; it's a global signal ## Critical blockers preventing full implementation 1. **Cannot access per-thread STID.PRIO from software** - The STID register (per-thread state) is not accessible from other threads - This prevents reading the effective priority of other threads - Blocks the core comparison logic needed for proper BESTWAIT usage 2. **Resched function cannot implement priority handoff** - The H2K_dosched() function needs to compare thread priorities to determine which thread should handle the reschedule interrupt - Without access to STID.PRIO, we cannot implement the required logic - This is the blocker for steering interrupts correctly ## Performance implications - Current check_sanity.ref implementation helps performance - Attempted opt version also shows NO improvement over current implementation - The reference implementation remains exploratory ## Why only in ref? The opt version was attempted but provides no performance benefit. The ref version is kept to document the exploration and architectural constraints for future scheduler work. ## Next steps - Investigate if STID.PRIO can be exposed via privileged interface - Consider alternative scheduling implementation accommodating the bestwait reg usage Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

Replace the software low-priority steering (priomask / lowprio_imask / H2K_runlist_worst_prio) with the hardware BESTWAIT + SCHEDCFG[8] priority comparator. check_sanity now arms BESTWAIT with the best ready priority and lets hardware post RESCHED to the worst-priority (STID.PRIO) hardware thread; a software resched_int() backstop covers the 255==255 case the comparator cannot fire on- will be removed in the upcoming changes. Boot opens IMASK (imask=0) so every thread is a steering candidate, and setup drops H2K_lowprio_init. runlist_worst_prio/_hthread, lowprio_notify/raise, highprio/lowprio_imask, and the priomask bookkeeping in dosched/check_sanity are removed. The runlist[]/runlist_prios[] mirror arrays are no longer read by the scheduler. MAX_PRIO is split into WAITING_PRIO (255, the idle/sentinel priority) and MAX_READY_PRIO (254, the cap for runnable threads) to match the comparator's strict-greater semantics- will be applied in upcoming changes. runlist and lowprio unit tests are disabled in the testlist accordingly. Known scheduler-callsite races (documented in-tree, not yet root-caused): - dosched go-to-sleep: change_imask(hthread,0) is issued before H2K_switch. It is logically redundant with the imask=0 the switch sleep path already does, but removing it loses wakeups -- the thread must become a valid RESCHED steering target (IMASK[RESCHED]=0) early enough for a producer's set_bestwait to steer to it. Without it the futex/pi test hangs. - dosched H2K_runlist_push(new), and the H2K_runlist_remove(me) in yield and resched: these now write to throwaway wip_dummy_runlist[] arrays that nothing reads. The writes are retained only because removing them hangs the futex_pi test: the extra stores shift interrupt-arrival timing on the scheduler hot path enough to keep a wakeup/steering window from being lost. This is a timing shim over a real lost-wakeup race in the ready-append -> set_bestwait -> steer sequence, not a data dependency; every other callsite is fine without them. - The H2K_runlist_remove(me) calls in yield.ref.c and resched.ref.c are the same case: dropping their dummy-array writes also hangs the futex_pi test, for the same timing reason as the dosched push above. - The whole H2K_runlist mechanism (push/remove/set_thread_prio and the runlist[]/runlist_prios[] arrays) could be removed entirely across all callsites once this timing sensitivity is fixed: the arrays are write-only now (nothing reads them), so they carry no data dependency -- only the incidental instruction-timing that currently masks the race. v81/ref: 135 tests pass, including kernel/futex/futex/test/tests/pi. Other test_variants havent been tested. Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

Signed-off-by: zbelinsk <zbelinsk@qti.qualcomm.com>

Erich Plondke (eplondke)

When we go to idle do you change STID.PRIO to 255 before you go to sleep? I think that's in "switch".

Erich Plondke (eplondke) · 2026-07-14T01:43:26Z

 /* Check whether a thread at a given priority is ready */
 static inline u32_t H2K_ready_prio_valid(u32_t prio)
 {
-	if (prio > MAX_PRIO) return 0;


recommend keeping MAX_PRIO and setting it to 254 or smaller.

Erich Plondke (eplondke) · 2026-07-14T01:45:29Z

 		}
 	}
 	/* FIXME: check pending intpool interrupts */
-	H2K_runlist_remove(me);


Getting rid of the runlist entirely is a little scary... very cool if we could do that though!

Erich Plondke (eplondke) · 2026-07-14T01:47:34Z

@@ -348,10 +348,11 @@ boot_continuation:
 	SETCONST(H2K_GP,H2K_kg)

 	/* Set up rising edge triggered */


probably want to fix the comments.

Erich Plondke (eplondke) · 2026-07-14T01:50:06Z

 	DPRINTF("SET_PRIO_TRAPMASK\n\n");
 	/* SET_PRIO_TRAPMASK bad prio */
-	ret = H2K_trap_config(CONFIG_VMBLOCK_INIT, vm, SET_PRIO_TRAPMASK, MAX_PRIO + 1, 0, NULL);
+	ret = H2K_trap_config(CONFIG_VMBLOCK_INIT, vm, SET_PRIO_TRAPMASK, WAITING_PRIO + 1, 0, NULL);


again I'd keep MAX_PRIO but change it to 254.

Erich Plondke (eplondke) · 2026-07-14T03:23:48Z

We want IMASK to stay clear for all threads with this change. You might want to look for all instances of setting IMASK directly (with IMASK = R) or indirectly (with setimask)... and of course you can get a break point in --interactive mode in the simulator and see if any threads have their IMASK set. If we are setting any IMASK bit to nonzero the hardware won't steer the interrupt to the right place (it will go to any unmasked thread instead of the worst priority one).

…e lowprio Replace the software reschedule-interrupt steering with the hardware BESTWAIT/SCHEDCFG comparator (V85 spec Rev AB, section 6.5). The hardware raises RESCHED_INT whenever any HW thread's STID.PRIO is strictly worse than the armed BESTWAIT value and then resets BESTWAIT to 0x1FF, so the kernel no longer needs to designate a victim thread in software. TARGET=ref passes on archs 68/73/81. TARGET=opt is intentionally NOT converted yet -- the opt *.Svariants still implement the old software model and will be rewritten in a follow-up after this ref change is reviewed. Enable hardware steering ------------------------- * boot.ref.S: set imask = 0 on all threads so every HW thread is a qualified RESCHED target. Previously imask was closed (-1) and software opened exactly one victim's RESCHED bit via lowprio_imask(). * setup.ref.c: SCHEDCFG.EN | INTNO(RESCHED_INT) arms the feature at boot. BESTWAIT arming moved into the readylist API --------------------------------------------- Arming the comparator is now the readylist's job. New *_arm variants (H2K_ready_append_arm / _insert_arm / _remove_arm) call H2K_set_bestwait( H2K_ready_best_prio()); the plain variants do not touch BESTWAIT. Callsites choose based on whether they are a wake/sleep transition or a mid-reschedule step: arm : intpool, popup, futex_common (wakes that go straight to H2K_switch), create, vmop kill paths, vmint deliver, resched, and getbest. noarm: yield and futex_pi raise -- these are immediately followed by H2K_dosched on the same HW thread, which re-arms via getbest. resched() must arm: the hardware fire that delivered the interrupt already reset BESTWAIT to 0x1FF, so re-arming immediately closes the disarmed window before dosched's getbest/cluster-search runs. Voluntary paths (yield/hwconfig) did not fire and still hold a valid armed value, so they can defer to dosched. Delete the software steering subsystem --------------------------------------- * Removed sched/lowprio/ entirely (lowprio_imask/highprio_imask/priomask designated-victim machinery and H2K_lowprio_notify). * Removed the priomask block from dosched.ref.c and the H2K_lowprio_notify call site; deleted check_sanity.ref.c (H2K_check_sanity / _unlock). Callers now BKL_UNLOCK and return directly (futex_classic, futex_pi, create). * switch.ref.S: sleep path no longer writes priomask; opens imask fully; switch-in path returns via k0unlock/jumpr r31 instead of jumping to the deleted check_sanity_unlock. Retire the software runlist mirror ----------------------------------- * Removed data/runlist/ and checkers/checker_runlist/. The runlist[]/ runlist_prios[] mirror and the wip_dummy_runlist* timing shim only fed H2K_runlist_worst_prio() for software steering; nothing reads it once steering is in hardware. * Every H2K_runlist_push/_remove replaced by a direct status = H2K_STATUS_RUNNING (dosched, intpool, popup, setup boot path). futex_pi priority sync ---------------------- PI boost/restore now writes dest->prio and set_thread_stid_prio() directly (no runlist mirror). unlock_pi restores me->base_prio in software and hardware STID up front, before releasing, so the BESTWAIT comparator reads the correct priority and cannot hide the unblocked holder behind a stale boosted value. Spec-mandated synchronization ----------------------------- switch.ref.S adds isync after STID writes (V85 spec: "After updating the STID, an isync instruction must be executed to ensure that the new STID value is fully observed"). This is a correctness requirement, not removable overhead -- and the change is still faster overall (see below). CLUSTER_SCHED guards -------------------- Convert every `#ifdef CLUSTER_SCHED` to `#if CLUSTER_SCHED` so the single `#define CLUSTER_SCHED 0/1` in h2_common_defs.h actually gates the cluster/coproc code. Behavior is identical to the previously-tested build: the cluster paths were already runtime-gated by `cluster`sched == 0`, so compiling them out changes nothing observable. H2K_ready_head reduces to returning the ready-list head; getbest's sleep branch reduces to `return NULL`. Performance (futex/pi TOTAL TEST TIME, pcycles) ----------------------------------------------- ARCHV before (SW) after (HW) delta 68 1,813,302 1,731,697 -81,605 (-4.5%) 73 1,833,006 1,765,404 -67,602 (-3.7%) 81 1,968,943 1,889,073 -79,870 (-4.1%) Faster on every arch despite the added spec-mandated isyncs, from removing per-schedule software work: the runlist_worst_prio() walk, priomask bookkeeping, H2K_lowprio_notify, and the mirror writes. Follow-ups (not in this change) ------------------------------- * Rewrite the opt *.Svariants to the pure-BESTWAIT model; TARGET=opt failures are expected until then. * Research whether wait_mask is still needed for cluster scheduling; if not, drop it and its write sites in resched/intpool/popup/switch.ref.S/setup. Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

zbelinsk · 2026-07-20T07:49:01Z

This is looking pretty good! Probably the easy thing for now is to update BESTWAIT at check_sanity time, but then eventually move it to the time where we update the readylist.

There are probably lots of scheduler-related tests that check for how we maintain the IMASK stuff, as we make it simpler a lot of those tests will have to change.

I do think you should add a new document specifically about how we're changing the scheduler, and especially highlight that with the new interrupt logic:

If all the threads have their IMASK enabled for interrupts, the hardware delivers the interrupt to the hardware thread with the worst STID.PRIO field

The hardware will compare the STID.PRIO field for all hardware threads and the BESTWAIT register, and if there is a hardware thread that is worse than the BESTWAIT register it will automatically raise the reschedule interrupt (which would then be steered to the worst hardware thread as above)

This should greatly simplify dosched and check_sanity

I have edited the readylist.spec to outline the behavior of the new scheduling mechanism.
Changes were made in ref version, potentially will be implemented in opt after review.

Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

github-actions Bot added the untested Mark untested PRs label Jun 21, 2026

zbelinsk requested a review from Bryan Bayerdorffer (bryanb-h2) June 21, 2026 20:13

zbelinsk marked this pull request as draft June 22, 2026 07:03

Erich Plondke (eplondke) reviewed Jul 9, 2026

View reviewed changes

zbelinsk force-pushed the bestwait-hw-steering branch from 3db7bc1 to 4239f0c Compare July 12, 2026 05:54

zbelinsk added 2 commits July 13, 2026 15:56

Update dosched.ref.c

5a88e2c

Signed-off-by: zbelinsk <zbelinsk@qti.qualcomm.com>

Erich Plondke (eplondke) reviewed Jul 14, 2026

View reviewed changes

zbelinsk force-pushed the bestwait-hw-steering branch from 0a919f6 to a6d1ca0 Compare July 20, 2026 06:55

zbelinsk added 2 commits July 20, 2026 00:40

document and organize files

79acae5

Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

readylist.spec: use consistent tilde underlines

4e5af1a

Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>

.

aa65415

Signed-off-by: Zeev Belinsky <zbelinsk@qti.qualcomm.com>


		asm volatile ("dmcfgwr(%0, %1);" : : "r" (index), "r" (data));
		}

		This function retrieves clears the contents of the IPEND status register.


		H2K_get_bestwait

		@@ -348,10 +348,11 @@ boot_continuation:
		SETCONST(H2K_GP,H2K_kg)

		/* Set up rising edge triggered */

Uh oh!

Conversation

zbelinsk commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bryan Bayerdorffer (bryanb-h2) commented Jun 29, 2026

Uh oh!

zbelinsk commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zbelinsk commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Erich Plondke (eplondke) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erich Plondke (eplondke) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erich Plondke (eplondke) commented Jul 14, 2026

Uh oh!

zbelinsk commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zbelinsk commented Jun 21, 2026 •

edited

Loading

zbelinsk commented Jun 29, 2026 •

edited

Loading

zbelinsk commented Jul 8, 2026 •

edited

Loading