[DRAFT] Priority-based scheduler for federated applications

[fra-p]

This draft pull request is the work @edwardalee and I have been doing to introduce priority scheduling of reactions with deadline for federated applications with the reactor-uc runtime. It is still a draft, because testing is incomplete and the syntax for some customization options is to be discussed (more on this in a dedicated section). However, I am creating this pull request to get feedback on the priority assignment rule and the scheduler design that would certainly help improve the scheduler. Feel free to comment with thoughts and suggestions!

Introduction

The reactor-uc runtime features a single OS thread that schedules the reactions in the reaction queue. In federated applications, each federate has its own runtime and its own thread. When the federates run on the same machine, the threads compete for the CPUs of the platform, and a scheduling algorithm handles contention.

Federates may have different urgency of execution depending on what reaction they are executing at a given time. In fact, reactions can be assigned with a deadline. To guarantee that deadlines are met, a real-time thread scheduler is needed to prioritize the completion of reactions with more urgent deadlines.

This PR implements a priority-based scheduler in reactor-uc that assigns a priority value to a reaction as a function of its deadline. The rule is based on the deadline monotonic (DM) principle, but extended to support platforms like Linux that feature a finite number of assignable priority values.

Scheduler design

Control flow

The main thread of each federate serves one reaction at a time. The thread picks the first ready reaction in the reaction queue and sets its priority to a value that depends on the deadline of the picked reaction. The OS thread scheduler will use that priority value to schedule the thread together with the threads of the other federates on the same machine.

Once a federate is done with the reactions in the reaction queue and waits until its physical time catches up with the next tag, it sets its priority to a value that is higher than any possible main thread executing reactions. This is needed to handle the case of asynchronous events (e.g., physical connections) and guarantees that the thread is woken up as soon as the event arrives.

Federates rely on TCP connections to communicate, and the connections are handled by the TCP thread. Similarly to the previous case, the priority of the TCP thread is higher than any sleeping thread (and thus, any thread executing reactions), to be sure that the messages are received without delays. However, this might be a problem with large messages.

Priority function

The priority value for a thread executing a reaction with deadline $d$ is determined by the priority function $prio(d)$, which maps the deadline range [$d_{min}, d_{max}$] to the priority range [${prio}{min}, {prio}{max}$]. $d_{min}$ and $d_{max}$ are the minimum and maximum deadlines found in a given federated application, while ${prio}{min}$ and ${prio}{max}$ are the minimum and maximum scheduling priorities that can be assigned to threads scheduled with a given OS thread scheduler. The function is decreasing exponential to subsume the deadline monotonic rule: tighter deadlines are mapped to higher priority values. This function is tailored to the specific federated application and generated by the code generator.

The analytical expression of the priority function is:

$prio(d) = round(K e^{- \alpha * d} + P)$.

$\alpha$ is the parameter that controls the speed of decay of the exponential function. $K$ and $P$ are computed such that:

• $prio(d_{min}) = {prio}_{max}$;
• $prio(d_{max}) = {prio}_{min}$.

Since the priority space is finite, collisions may occur: two very close deadline values may be mapped to the same priority value. The priority function allows to control in what part of the deadline range collisions are more likely to occur by tuning the $\alpha$ parameter. A big $\alpha$ value increases the speed of decay of the function, which means that most of the priority space is used to map small deadlines. This is useful when most of the deadlines of the federated application are tight and we do not want collisions to occur with these. Collisions occurring with looser deadlines are likely to cause less harm. A small $\alpha$ value decreases the speed of decay, and the priority function performs a priority mapping that tends to a linear conversion. $\alpha$ is automatically selected in the range [$\alpha_{min}$, $\alpha_{max}$] (empirically derived) by the code generator as a function of the median deadline of the federated application, to make the priority function better fit its deadline distribution.

Implementation details

The priority scheduler is a sublcass of the dynamic scheduler and is called PriorityScheduler. It only works with platforms having threading capabilities that are modeled by the ThreadedPlatform class.

The only supported platform at the moment is Linux. Both SCHED_FIFO and SCHED_RR can be selected as underlying thread scheduling algorithms. The POSIX thread API is used to control the thread scheduler.

Federate's main threads can be assigned with priority values in the range [2, 97]. 1 is reserved to reactions with no deadline; 98 is the sleeping thread's priority; 99 is the TCP thread's priority.

The PriorityScheduler_run_timestep function sets the priority of the current thread right before executing the reaction body by calling the priority function.

Example

Tasks.lf.txt
Attached is a simple example of a federated real-time application in which deadline violations occur when it is not executed with the priority scheduler. It is made of two periodic tasks both triggered by a 50 ms timer, one of which has a tighter deadline and a longer execution time than the other. The more urgent task starts at a slightly greater logical time, which means that when its timer fires, the loose-deadline task is already executing. This test case shows the ability of the priority scheduler to preempt the loose-deadline task to execute the more urgent one. With the default scheduler, preemption does not happen and the urgent task misses its deadline.

Future work

Currently, the priority scheduler is enabled when either SCHED_FIFO or SCHED_RR are selected as Linux scheduling policy. The selection is done with the thread-policy target property that takes as values "rt-fifo", "rt-rr" and "normal" (for the default non-real-time Linux scheduler). Similarly, the cores target property restricts the number of cores that the application can use. In the future, these two target properties will become configuration properties specified in the main reactor with the '@' sign. The most appropriate syntax is yet to be defined.

A more thorough testing is in the works to make sure all corner cases are handled correctly.

[github-actions]

Benchmark results after merging this PR:

Benchmark results

Performance:

PingPongUc:
Best Time: 615.334 msec
Worst Time: 727.115 msec
Median Time: 677.175 msec

PingPongC:
Best Time: 176.757 msec
Worst Time: 201.395 msec
Median Time: 177.573 msec

ReactionLatencyUc:
Best latency: 7124 nsec
Median latency: 66397 nsec
Worst latency: 8953177 nsec

ReactionLatencyC:
Best latency: 15249 nsec
Median latency: 64599 nsec
Worst latency: 8493852 nsec

Memory usage:

PingPongUc:
text data bss dec hex filename
40868 800 9064 50732 c62c bin/PingPongUc

PingPongC:
text data bss dec hex filename
48098 894 384 49376 c0e0 bin/PingPongC

ReactionLatencyUc:
text data bss dec hex filename
26557 776 3112 30445 76ed bin/ReactionLatencyUc

ReactionLatencyC:
text data bss dec hex filename
43640 862 384 44886 af56 bin/ReactionLatencyC

[edwardalee]

I've only been through part of this PR, but I found some issues, so I'm submitting a partial review in case these comments are useful. Most of the comments are insignificant... minor suggestions for documentation. But one is potentially a bug.

[edwardalee]

Suggested change

	* @brief Set the global maximum wait time for remote input ports for this federate.
	* @brief Set the global maximum wait time for this federate to assume remote input ports are absent.

[edwardalee]

Redundant comment is not needed. @brief is sufficient.

[edwardalee]

Since this is the user-facing function, perhaps it needs documentation?

[edwardalee]

Suggested change

	LF_SCHED_FAIR, // Non real-time scheduling policy. Corresponds to SCHED_OTHER
	LF_SCHED_TIMESLICE, // Real-time, time-slicing priority-based policy. Corresponds to SCHED_RR.
	LF_SCHED_PRIORITY, // Real-time, priority-only based scheduling. Corresponds to SCHED_FIFO.
	/** Non real-time scheduling policy. Corresponds to SCHED_OTHER in Linux. */
	LF_SCHED_FAIR,
	/** Real-time, time-slicing priority-based policy. Corresponds to SCHED_RR in Linux. */
	LF_SCHED_TIMESLICE,
	/** Real-time, priority-only based scheduling. Corresponds to SCHED_FIFO in Linux. */
	LF_SCHED_PRIORITY

[edwardalee]

What does the default 0 mean? The default and its meaning should be documented.

[edwardalee]

Suggested change

	* Collects all deadlines from federates, including nested reactors.
	* Collect all deadlines from federates, including nested reactors.

[edwardalee]

Would it make more sense for the return value to be a Set rather than a List? A list has semantics to the ordering and has duplicate entries. Same for all related methods below.

[edwardalee]

This doesn't look quite right to me. If delay is a parameter reference that is not overridden in the instantiation, then this will return 0 because delayExpr will be null. But the right return value is the default value of the parameter, which can be obtained using getDefaultAsTimeValue.

Also, rhs.expr could itself be a ParameterReference, in which case, the parameter referred to belongs to the parent Instantiation. I'm not sure how to get that parent Instantiation, however. I don't think it's even possible given the arguments. I think that instead of the instance argument, this needs to be an instances argument with type List<Instantiation>. This list needs to include all the Instantiations that contain this Instantiation. That list would need to be constructed in the collectDeadlinesFromNestedReactors below, which does the recursive visiting of instantiations.

Also, this looks to me like a function that should be provided in ASTUtils. I'm kind of surprised it isn't. I guess that is because there is an implementation of the same functionality in ReactorInstance.getTimeValue(Expression). reactor-uc is not using ReactorInstance or any of that infrastructure, however.

[edwardalee]

Should this be a Set rather than a list?

[edwardalee]

I guess if you need the median deadline, then collecting the deadlines as a Set won't work. You need to know how many times a deadline appears to get the median.

[fra-p]

@edwardalee I fixed the bug with nested parameters and implemented most of your minor fixes. Thanks for your review!

[tanneberger]

Firstly: Very Good Work! 🎉 👍 🔧

I left a lot of comments (sorry :D)

Couple of more big picture comments (it's okay if you hate me XD):

• I would like to remove target properties from uC => turn them into annotations, this would allow us to have federates running on embedded (zephyr/riot) talking to a LinuxRT federate.

• The code generator is right now is the ugliest part of uC: Put the Priority stuff into a PriortyScheduler.kt and lets add a good integration

• I would recommend we add a PosixRT platform that implements the ThreadedPlatform, we still want that @edwardalee can run uC on his Macbook where there is no LinuxRT scheduler :D

[tanneberger]

Why do we need the max_wait here? Because the max_wait is normally stored inside FederatedInputConnections and can be configured per (Bundle/Connection)

[tanneberger]

This would require that all POSIX platforms are RT. I don't think this is wise, for example this won't work on MacOS. I would say we create posix and a posixRT platform.

[tanneberger]

You dont need to declare the functions here, because the functions are only called through the function pointers stored in the objects. (Declaration is not needed)

[tanneberger]

Yes the convention is <Class Name>_<method_name>

[tanneberger]

same here you can remove this.

[tanneberger]

the _ before the function name made clear that these functions where internal to this file.

[tanneberger]

For what do we need the FederatedInputConnection name? Except printing it :D

[tanneberger]

and maybe put into an internal function for reuse ;D

[tanneberger]

why this move?

[tanneberger]

we made some changes in #288 to the scheduler especially when it comes to PollingNetwork channels.

[github-actions]

Coverage after merging priority into main will be

50.86%

Coverage Report

File	Stmts	Branches	Funcs	Lines	Uncovered Lines
src
action.c	70.29%	65.63%	62.50%	72.45%	117–118, 131–132, 135–138, 141–142, 144–147, 149, 153, 156–159, 25, 42, 42, 44–46, 50, 52, 59–60, 67–69, 69, 69–72
builtin_triggers.c	90.91%	70%	100%	96.77%	14, 18, 40, 43
clock_synchronization.c	0%	0%	0%	0%	100–103, 105–106, 106, 106–109, 114–116, 116, 116–118, 120–122, 122, 122–124, 124, 124, 124, 124–126, 129, 13, 130, 132, 135, 137–139, 14, 141, 143, 143, 143–145, 147–149, 149, 149–151, 154, 156, 158, 16, 160–161, 161, 161–163, 163, 163, 166, 166, 166–167, 17, 170–175, 175, 175–176, 18, 180, 182–188, 188, 188–189, 19, 192, 195–199, 20, 200, 200, 200–201, 203, 205, 208, 210–214, 214, 214–215, 215, 215–224, 224, 224–225, 227, 23, 23, 23, 230–232, 234–236, 238, 238, 238–239, 24, 241, 244–245, 247–249, 25, 250–254, 254, 254–256, 26, 26, 26, 261–264, 264, 264–266, 268–269, 27, 273–275, 277–278, 278, 278–279, 279, 279–287, 287, 287–288, 290, 294, 297–298, 30, 301–310, 310, 310–311, 314, 317–320, 323, 323, 323, 323, 323, 323, 323, 323, 323–329, 33, 330–339, 34, 340–344, 347–348, 35, 350, 354–362, 364, 367, 367, 367–368, 372–376, 379, 379, 379–380, 383, 385, 40, 42, 44, 46, 49, 49, 49–51, 51, 51–52, 55, 58, 62, 64–66, 66, 66, 69, 69, 69–70, 73–79, 79, 79–80, 83, 89–90, 93, 95, 95, 95–97
connection.c	78.34%	53.19%	100%	88.12%	10, 10, 105, 111, 124–125, 139, 14, 14, 140, 146, 148–149, 151, 16–17, 21–22, 22, 22–23, 25, 27–28, 34, 48, 48, 48–49, 55, 61–63, 98
environment.c	94.12%	83.33%	100%	100%	16, 29
event.c	100%	100%	100%	100%
federated.c	6.05%	3.19%	8.33%	7.21%	10, 101–102, 104–107, 11, 110, 112–118, 12, 120, 120, 120–121, 123, 123, 123, 123, 123–124, 124, 124–126, 128, 128, 128–129, 13, 130–131, 133, 136, 136, 136–138, 14, 142–143, 146–148, 148, 148–149, 149, 149, 15, 150–151, 153, 156, 158, 160–165, 168–169, 17, 17, 17, 170, 172, 172, 172, 174–177, 179, 18, 18, 18, 181, 181, 181–182, 184–185, 188–189, 19, 19, 19, 196–198, 198, 198–199, 20, 20, 20, 201–202, 204, 206, 206, 206–209, 21, 21, 21, 210, 210, 210, 210, 210, 210, 210–220, 220, 220–221, 223–229, 23, 230, 232–235, 239, 242, 242, 242–244, 246, 248, 25, 250–252, 252, 252, 252, 252–259, 26, 261–264, 264, 264–265, 267, 27, 270–273, 28, 281–283, 296–297, 297, 297–298, 298, 298–299, 299, 299, 30, 30, 30, 300, 300, 300–301, 301, 301–302, 302, 302–303, 303, 303, 305, 305, 305–306, 306, 306–307, 307, 307–308, 308, 308–309, 309, 309, 311, 32–33, 33, 33–34, 36, 38, 40, 40, 40–41, 45, 47, 51, 54–56, 58, 58, 58, 58, 58–59, 59, 59–60, 65, 65, 65–67, 71–72, 75–76, 78, 8, 80, 82, 84, 86, 88–89, 9, 90–97, 99
logging.c	88.52%	83.33%	100%	89.36%	25, 38–40, 47, 60–61
network_channel.c	69.23%	62.50%	100%	70.59%	40, 40, 40, 45–48, 57
physical_clock.c	89.61%	83.33%	71.43%	93.10%	26, 41, 57–58, 81, 83
port.c	75%	42.86%	100%	91.67%	10, 10, 10, 15, 15, 15–16, 22, 26, 31, 31–33, 33, 33–34, 46, 46, 46–47
queues.c	86.05%	79.03%	100%	87.94%	100–106, 129–131, 133, 140–141, 146, 162, 167, 173, 34, 34, 38–44, 69–70, 96, 96
reaction.c	71.31%	56.52%	100%	78.87%	15, 21, 28–30, 30, 30–32, 32, 32–33, 43, 46, 53–54, 54, 54–56, 56, 56–57, 72, 88–90, 90, 90–93, 93, 93–94
reactor.c	69.33%	51.52%	100%	82.28%	10, 101–102, 14–19, 22, 28, 30, 32–37, 37, 37–38, 38, 38, 43, 55, 58–59, 59, 59–60, 60, 60–61, 63, 77–78, 81–82,