Benchmarking RK0: 2.7 Million Equals

Quoth the raven: because so.

Last year, I don’t really remember the fiction writer; I think it was Edgar Allan Poe, or his raven, who published a report comparing RTOSes’ “performance.” Assessing third-party work already requires caution. Assessing without giving a chance to reproduce needs a reason, I guess. The benchmarks were ThreadX’s own methodology. The results are useful, although ‘performance’ can tell little, because that would mean talking about throughput. If we are doing real-time, we want determinism. From these markers, we can see that, under a specific scenario, a mechanism will behave ‘the same way’ at a ‘given rate’.

In real time, we want computation results with a bounded upper response time; the tests help to correlate results with deterministic behaviour for each mechanism. The application is called ‘Benchmark’. Here is the devil.

RK0 under Thread-Metric on a 72 MHz Cortex-M3

Real-time kernels are boring

A timer tick arrives, does the minimum necessary work, and gets out of the way.

So I finally put RK0 through the complete Thread-Metric RTOS benchmark suite on real hardware: an STM32F103RB, Cortex-M3 at 72 MHz.

Thread-Metric consists of eight workloads covering basic processing, cooperative and preemptive scheduling, interrupt processing, interrupt-driven preemption, message queues, synchronisation and memory allocation.

The canonical test interval is 30 seconds. I used the specified 10 ms periodic tick.

The result was pleasantly boring.

BenchmarkTransactions in 30 sApprox. rate
Basic processing124,0034,133/s
Cooperative scheduling4,800,725160,024/s
Preemptive scheduling2,706,68890,223/s
Interrupt processing13,580,995452,700/s
Interrupt preemption2,070,37469,012/s
Message processing5,384,998179,500/s
Synchronisation17,137,994571,266/s
Memory allocation27,334,191911,140/s

All eight tests passed. Zero benchmark errors.

And the scheduling tests are the interesting part because they shed light on design decisions.

Five tasks, one scheduling relationship

The cooperative test runs five equal-priority tasks which repeatedly relinquish execution to one another.

RK0 finished with:

960145 960145 960145 960145 960145
max_delta = 0

The preemptive test is rather nastier.

Five tasks have successively higher priorities. The lowest-priority task wakes the next one, which immediately preempts it and wakes the next, until the highest-priority task runs. The chain then unwinds as each task suspends itself.

RK0 completed 2,706,688 task executions in 30 seconds:

average = 541337
max_delta = 1
errors = 0

Thread-Metric explicitly permits a difference of one because the reporting task can sample the system part-way through a traversal.

In other words, after millions of state transitions, the scheduling relationship is still exactly where the benchmark expects it to be.

This does not prove that RK0 is “deterministic”.

It also means that failing this check does not magically prove that another RTOS is “non-deterministic”.

Quoth the raven: A self-check is not a theorem prover.

It shows that RK0 repeatedly preserves the execution contract expected by this workload.

That is already useful: the numbers match the architecture

RK0 uses fixed-priority preemptive scheduling with an O(1) priority-selection mechanism. There is no scheduler walk proportional to the task count and there is no time slicing.

The ready structure is priority-indexed and backed by a bitmap.

So the scheduler does not ask:

Which of these N tasks should I run?

It asks:

What is the highest-priority READY level?

and obtains the answer directly.

The benchmark numbers are consistent with that design.

On this Cortex-M3, the cooperative workload consumes an average CPU budget of about 450 cycles per benchmark execution, while the preemptive workload is about 798 cycles per reported execution.

Those are not context-switch latency measurements. They include the surrounding benchmark and kernel machinery.

For comparison, separate RK0 profiling on an 80 MHz Cortex-M4 has measured the core context-switch path at roughly 125 cycles, around 1.6 µs.

The numbers fit together rather nicely.

SysTick should be boring too

The timer interrupt is another place where an RTOS can quietly burn CPU forever.

And unlike timer insertion, SysTick is not optional.

At a 1 ms tick it arrives:

1000 times/second
60,000 times/minute
3,600,000 times/hour

whether the application is doing anything interesting or not.

RK0 therefore deliberately keeps the ordinary tick path small.

Its timeout mechanism uses delta-ordered timers. In the common case, the tick updates time and touches the head of the timeout structure. It does not walk every sleeping task.

More importantly, after all timeout housekeeping has completed, RK0 now asks the only scheduling question that matters:

Did this tick make a task READY that can actually preempt the current RUNNING task?

If not, it returns directly to the interrupted task.

No pointless PendSV. No save-and-restore exercise merely to discover that the same task should continue running.

This is one reason I prefer paying some bounded insertion cost when a timer is created rather than moving complexity into an interrupt that fires forever.

Timing wheels can make insertion attractive, but range, resolution, bucket population and hierarchical cascading all have costs somewhere.

RK0’s choice is simpler:

Spend occasional work when somebody requests a timer. Keep the periodic interrupt cheap.

And the 1 kHz experiment was amusing

I also ran the preemptive workload with a 1 ms tick rather than 10 ms.

That means roughly 30,000 SysTick interrupts during the 30-second benchmark instead of 3,000.

The result:

10 ms tick: 2,706,775
1 ms tick: 2,700,770

Ten times the timer interrupt rate cost only about 0.22% of preemption throughput.

And with the 1 ms tick the five counters ended at:

540154 540154 540154 540154 540154

Zero skew.

Again, this is not a mathematical proof of temporal determinism.

However, it is a fairly good demonstration that frequently interrupting RK0 does not make its scheduler behave differently on each return.

What Thread-Metric does — and does not — tell us

This matters because the original comparison generated considerably more confidence than its methodology justified. Predictably, people argued over which preferred RTOS had “won” or “lost”, while comparatively little attention went to what the benchmark could actually establish.

Thread-Metric originated in the ThreadX ecosystem. Its API and workloads reflect a particular RTOS model.

That does not make it useless. Quite the opposite: the tests are small, understandable, open and contain useful integrity checks.

But a benchmark should be allowed to say what it measured.

A passing scheduling test means the implementation preserved the benchmark’s scheduling relationship. The counters are observations, not a causal explanation. If they drift, the next question is which mechanism caused the drift. If you haven’t identified the mechanism producing the deviation, calling the scheduler “non-deterministic” isn’t engineering-backed. It is something you say, an unreasonable observation – pretty much like a ‘your mum’ joke.

Real-time behaviour is notoriously context-dependent. That is why we normally investigate the mechanism before naming the property. To be clear, in that report, I know the source code of two systems reasonably well. And the results did not surprise me. The conclusions around the result and its non-reproducibility… wow.

The actual RK0 result

All said: For this STM32F103RB run:

Cortex-M3
72 MHz
10 ms tick
30 seconds/test
8 / 8 PASS
0 benchmark errors
4.8 million cooperative scheduling events
2.7 million preemptive scheduling events
13.6 million interrupt transactions
5.4 million message transactions
17.1 million semaphore transactions
27.3 million allocation transactions

RK0 remains what it was intended to be: small, explicit, fixed-priority and aggressively uninterested in surprising its user.

The important takeaway is not that one number was larger than another. The processing baseline remained stable while the kernel mechanisms produced repeatable, internally consistent results under the benchmark workload.

Real-time software should be boring. RK0 is getting quite good at it.

The results show that the tested workloads execute repeatably under the stated hardware and build configuration, and the combination of kernel services they exercise. It is impressive how, on small MCUs, image layout affects the execution count of the same workload. Inline a function or make it a macro – you will see some numbers increasing, others decreasing. You know you are fine when this impact is negligible, and dispersion is low. This is empirical, but very reasonable, evidence of deterministic service behaviour. It supports RK0’s design choices.


Meanwhile, the current embedded mindset leans toward virtual-machine application-design lore, and some claims/methods/packages/ACRONYMS cause more fuss than the evidence supports.

Leave a Reply

Discover more from RK0 - Embedded Real-Time Kernel '0’

Subscribe now to keep reading and get access to the full archive.

Continue reading