Fix README typos, heading hierarchy, and formatting.

Maxim Egorushkin · Maxim Egorushkin · commit 43ca2e8d45bb · 2026-03-19T19:04:29.000Z
diff --git a/README.md b/README.md
@@ -48,35 +48,39 @@ Ultra-low-latency applications need just that and nothing more. The minimalism p
 
 ## Role Models
 Several other well established and popular thread-safe containers are used for reference in the [benchmarks][1]:
-* `std::mutex` - a fixed size ring-buffer with `std::mutex`.
-* `pthread_spinlock` - a fixed size ring-buffer with `pthread_spinlock_t`.
-* `boost::lockfree::spsc_queue` - a wait-free single-producer-single-consumer queue from Boost library.
-* `boost::lockfree::queue` - a lock-free multiple-producer-multiple-consumer queue from Boost library.
-* `moodycamel::ConcurrentQueue` - a lock-free multiple-producer-multiple-consumer queue used in non-blocking mode. This queue is designed to maximize throughput at the expense of latency and eschewing the global time order of elements pushed into one queue by different threads. It is not equivalent to other queues benchmarked here in this respect.
-* `moodycamel::ReaderWriterQueue` - a lock-free single-producer-single-consumer queue used in non-blocking mode.
-* `xenium::michael_scott_queue` - a lock-free multi-producer-multi-consumer queue proposed by [Michael and Scott](http://www.cs.rochester.edu/~scott/papers/1996_PODC_queues.pdf) (this queue is similar to `boost::lockfree::queue` which is also based on the same proposal).
-* `xenium::ramalhete_queue` - a lock-free multi-producer-multi-consumer queue proposed by [Ramalhete and Correia](http://concurrencyfreaks.blogspot.com/2016/11/faaarrayqueue-mpmc-lock-free-queue-part.html).
-* `xenium::vyukov_bounded_queue` - a bounded multi-producer-multi-consumer queue based on the version proposed by [Vyukov](https://groups.google.com/forum/#!topic/lock-free/-bqYlfbQmH0).
-* `tbb::spin_mutex` - a locked fixed size ring-buffer with `tbb::spin_mutex` from Intel Threading Building Blocks.
-* `tbb::concurrent_bounded_queue` - eponymous queue used in non-blocking mode from Intel Threading Building Blocks.
-
-# Using the library
+
+| Queue | Type | Description |
+|-------|------|-------------|
+| `std::mutex` | MPMC | A fixed size ring-buffer with `std::mutex`. |
+| `pthread_spinlock` | MPMC | A fixed size ring-buffer with `pthread_spinlock_t`. |
+| `boost::lockfree::spsc_queue` | SPSC | A wait-free queue from Boost library. |
+| `boost::lockfree::queue` | MPMC | A lock-free queue from Boost library. |
+| `moodycamel::ConcurrentQueue` | MPMC | A lock-free queue used in non-blocking mode. Designed to maximize throughput at the expense of latency, eschewing global time order. Not equivalent to other queues benchmarked here in this respect. |
+| `moodycamel::ReaderWriterQueue` | SPSC | A lock-free queue used in non-blocking mode. |
+| `xenium::michael_scott_queue` | MPMC | A lock-free queue proposed by [Michael and Scott](http://www.cs.rochester.edu/~scott/papers/1996_PODC_queues.pdf) (similar to `boost::lockfree::queue` which is also based on the same proposal). |
+| `xenium::ramalhete_queue` | MPMC | A lock-free queue proposed by [Ramalhete and Correia](http://concurrencyfreaks.blogspot.com/2016/11/faaarrayqueue-mpmc-lock-free-queue-part.html). |
+| `xenium::vyukov_bounded_queue` | MPMC | A bounded queue based on the version proposed by [Vyukov](https://groups.google.com/forum/#!topic/lock-free/-bqYlfbQmH0). |
+| `tbb::spin_mutex` | MPMC | A locked fixed size ring-buffer with `tbb::spin_mutex` from Intel Threading Building Blocks. |
+| `tbb::concurrent_bounded_queue` | MPMC | Eponymous queue used in non-blocking mode from Intel Threading Building Blocks. |
+
+## Using the library
 The containers provided are header-only class templates, no building/installing is necessary.
 
-## Install from GitHub
+### Install from GitHub
 1. Clone the project:
-```
-git clone https://github.com/max0x7ba/atomic_queue.git
-```
+   ```bash
+   git clone https://github.com/max0x7ba/atomic_queue.git
+   ```
 2. Add `atomic_queue/include` directory (use full path) to the include paths of your build system.
 3. `#include <atomic_queue/atomic_queue.h>` in your C++ source.
+
 If you use CMake, these can be simplified as follows:
 ```cmake
 add_subdirectory(atomic_queue)
 target_link_libraries(main PRIVATE atomic_queue::atomic_queue)
 ```
 
-## Using cmake FetchContent
+### Using cmake FetchContent
 You can also use CMake's FetchContent.
 ```cmake
 include(FetchContent)
@@ -89,7 +93,7 @@ FetchContent_MakeAvailable(atomic_queue)
 target_link_libraries(main PRIVATE atomic_queue::atomic_queue)
 ```
 
-## Install using vcpkg
+### Install using vcpkg
 ```
 vcpkg install atomic-queue
 ```
@@ -99,16 +103,23 @@ find_package(atomic_queue CONFIG REQUIRED)
 target_link_libraries(main PRIVATE atomic_queue::atomic_queue)
 ```
 
-## Install using conan
+### Install using conan
 Follow the official tutorial on [how to consume conan packages](https://docs.conan.io/2/tutorial/consuming_packages.html).
 Details specific to this library are available in [ConanCenter](https://conan.io/center/recipes/atomic_queue).
 
-## Benchmark build and run instructions
-The containers provided are header-only class templates that require only `#include <atomic_queue/atomic_queue.h>`, no building/installing is necessary.
-
-Building is necessary to run the tests and benchmarks.
+### Test build and run instructions
+Building is necessary to run the tests and benchmarks. The tests require the Boost.Test library (e.g. `libboost-test-dev` on Debian/Ubuntu).
 
+```bash
+git clone https://github.com/max0x7ba/atomic_queue.git
+cd atomic_queue
+make -r -j4 BUILD=debug run_tests
 ```
+
+### Benchmark build and run instructions
+The benchmarks require several third-party libraries to be cloned as sibling directories, as well as Intel TBB.
+
+```bash
 git clone https://github.com/cameron314/concurrentqueue.git
 git clone https://github.com/cameron314/readerwriterqueue.git
 git clone https://github.com/mpoeter/xenium.git
@@ -119,30 +130,30 @@ make -r -j4 run_benchmarks
 
 The benchmark also requires Intel TBB library to be available. It assumes that it is installed in `/usr/local/include` and `/usr/local/lib`. If it is installed elsewhere you may like to modify `cppflags.tbb` and `ldlibs.tbb` in `Makefile`.
 
-# Library contents
-## Available queues
+## Library contents
+### Available queues
 * `AtomicQueue` - a fixed size ring-buffer for atomic elements.
 * `OptimistAtomicQueue` - a faster fixed size ring-buffer for atomic elements which busy-waits when empty or full. It is `AtomicQueue` used with `push`/`pop` instead of `try_push`/`try_pop`.
 * `AtomicQueue2` - a fixed size ring-buffer for non-atomic elements.
 * `OptimistAtomicQueue2` - a faster fixed size ring-buffer for non-atomic elements which busy-waits when empty or full. It is `AtomicQueue2` used with `push`/`pop` instead of `try_push`/`try_pop`.
 
 These containers have corresponding `AtomicQueueB`, `OptimistAtomicQueueB`, `AtomicQueueB2`, `OptimistAtomicQueueB2` versions where the buffer size is specified as an argument to the constructor.
 
-Totally ordered mode is supported. In this mode consumers receive messages in the same FIFO order the messages were posted. This mode is supported for `push` and `pop` functions, but for not the `try_` versions. On Intel x86 the totally ordered mode has 0 cost, as of 2019.
+Totally ordered mode is supported. In this mode consumers receive messages in the same FIFO order the messages were posted. This mode is supported for `push` and `pop` functions, but not for the `try_` versions. On Intel x86 the totally ordered mode has 0 cost, as of 2019.
 
 Single-producer-single-consumer mode is supported. In this mode, no expensive atomic read-modify-write CPU instructions are necessary, only the cheapest atomic loads and stores. That improves queue throughput significantly.
 
-Move-only queue element types are fully supported. For example, a queue of `std::unique_ptr<T>` elements would be `AtomicQueue2B<std::unique_ptr<T>>` or `AtomicQueue2<std::unique_ptr<T>, CAPACITY>`.
+Move-only queue element types are fully supported. For example, a queue of `std::unique_ptr<T>` elements would be `AtomicQueueB2<std::unique_ptr<T>>` or `AtomicQueue2<std::unique_ptr<T>, CAPACITY>`.
 
-## Queue schematics
+### Queue schematics
 
 ```
 queue-end                 queue-front
 [newest-element, ..., oldest-element]
 push()                          pop()
 ```
 
-## Queue API
+### Queue API
 The queue class templates provide the following member functions:
 * `try_push` - Appends an element to the end of the queue. Returns `false` when the queue is full.
 * `try_pop` - Removes an element from the front of the queue. Returns `false` when the queue is empty.
@@ -161,14 +172,14 @@ Note that _optimism_ is a choice of a queue modification operation control flow,
 
 See [example.cc](src/example.cc) for a usage example.
 
-# Implementation Notes
-## Memory order of non-atomic loads and stores
+## Implementation Notes
+### Memory order of non-atomic loads and stores
 `push` and `try_push` operations _synchronize-with_ (as defined in [`std::memory_order`][17]) with any subsequent `pop` or `try_pop` operation of the same queue object. Meaning that:
 * No non-atomic load/store gets reordered past `push`/`try_push`, which is a `memory_order::release` operation. Same memory order as that of `std::mutex::unlock`.
 * No non-atomic load/store gets reordered prior to `pop`/`try_pop`, which is a `memory_order::acquire` operation. Same memory order as that of `std::mutex::lock`.
 * The effects of a producer thread's non-atomic stores followed by `push`/`try_push` of an element into a queue become visible in the consumer's thread which `pop`/`try_pop` that particular element.
 
-## Ring-buffer capacity
+### Ring-buffer capacity
 The available queues here use a ring-buffer array for storing elements. The capacity of the queue is fixed at compile time or construction time.
 
 In a production multiple-producer-multiple-consumer scenario the ring-buffer capacity should be set to the maximum expected queue size. When the ring-buffer gets full it means that the consumers cannot consume the elements fast enough. A fix for that is any of:
@@ -186,22 +197,22 @@ The containers use `unsigned` type for size and internal indexes. On x86-64 plat
 
 While the atomic queues can be used with any moveable element types (including `std::unique_ptr`), for best throughput and latency the queue elements should be cheap to copy and lock-free (e.g. `unsigned` or `T*`), so that `push` and `pop` operations complete fastest.
 
-## Lock-free guarantees
+### Lock-free guarantees
 *Conceptually*, a `push` or `pop` operation does two atomic steps:
 
 1. Atomically and exclusively claims the queue slot index to store/load an element to/from. That's producers incrementing `head` index, consumers incrementing `tail` index. Each slot is accessed by one producer and one consumer threads only.
 2. Atomically store/load the element into/from the slot. Producer storing into a slot changes its state to be non-`NIL`, consumer loading from a slot changes its state to be `NIL`. The slot is a spinlock for its one producer and one consumer threads.
 
 These queues anticipate that a thread doing `push` or `pop` may complete step 1 and then be preempted before completing step 2.
 
-An algorithm is *lock-free* if there is guaranteed system-wide progress. These queue guarantee system-wide progress by the following properties:
+An algorithm is *lock-free* if there is guaranteed system-wide progress. These queues guarantee system-wide progress by the following properties:
 
 * Each `push` is independent of any preceding `push`. An incomplete (preempted) `push` by one producer thread doesn't affect `push` of any other thread.
 * Each `pop` is independent of any preceding `pop`. An incomplete (preempted) `pop` by one consumer thread doesn't affect `pop` of any other thread.
 * An incomplete (preempted) `push` from one producer thread affects only one consumer thread `pop`ing an element from this particular queue slot. All other threads `pop`s are unaffected.
 * An incomplete (preempted) `pop` from one consumer thread affects only one producer thread `push`ing an element into this particular queue slot while expecting it to have been consumed long time ago, in the rather unlikely scenario that producers have wrapped around the entire ring-buffer while this consumer hasn't completed its `pop`. All other threads `push`s and `pop`s are unaffected.
 
-## Preemption
+### Preemption
 Linux task scheduler thread preemption is something no user-space process should be able to affect or escape, otherwise any/every malicious application would exploit that.
 
 Still, there are a few things one can do to minimize preemption of one's mission critical application threads:
@@ -216,14 +227,14 @@ People often propose limiting busy-waiting with a subsequent call to `std::this_
 
 [In Linux, there is mutex type `PTHREAD_MUTEX_ADAPTIVE_NP`][9] which busy-waits a locked mutex for a number of iterations and then makes a blocking syscall into the kernel to deschedule the waiting thread. In the benchmarks it was the worst performer and I couldn't find a way to make it perform better, and that's the reason it is not included in the benchmarks.
 
-C++20 introduced blocking `std::atomic::wait` which uses Linux futex for atomic compare-and-block operation, similar to `PTHREAD_MUTEX_ADAPTIVE_NP`, with hard-coded spin-count limits. And `thread_yield` calls, which Linus Torvalds above explains is only applicable for real-time threads with a single run-queue for them. A queue implemenation with `std::atomic::wait` is due to be benchmarked, with its performance expected to be similar to that of `PTHREAD_MUTEX_ADAPTIVE_NP`, but I'd love to be pleasantly surpised.
+C++20 introduced blocking `std::atomic::wait` which uses Linux futex for atomic compare-and-block operation, similar to `PTHREAD_MUTEX_ADAPTIVE_NP`, with hard-coded spin-count limits. And `thread_yield` calls, which Linus Torvalds above explains is only applicable for real-time threads with a single run-queue for them. A queue implementation with `std::atomic::wait` is due to be benchmarked, with its performance expected to be similar to that of `PTHREAD_MUTEX_ADAPTIVE_NP`, but I'd love to be pleasantly surprised.
 
 On Intel CPUs one could use [the 4 debug control registers][6] to monitor the spinlock memory region for write access and wait on it using `select` (and its friends) or `sigwait` (see [`perf_event_open`][7] and [`uapi/linux/hw_breakpoint.h`][8] for more details). A spinlock waiter could suspend itself with `select` or `sigwait` until the spinlock state has been updated. But there are only 4 of these registers, so that such a solution wouldn't scale.
 
-# Benchmarks
+## Benchmarks
 [View throughput and latency benchmarks charts][1].
 
-## Methodology
+### Methodology
 There are a few OS behaviours that complicate benchmarking:
 * CPU scheduler can place threads on different CPU cores each run. To avoid that the threads are pinned to specific CPU cores.
 * CPU scheduler can preempt threads. To avoid that real-time `SCHED_FIFO` priority 50 is used to disable scheduler time quantum expiry and make the threads non-preemptable by lower priority processes/threads.
@@ -234,30 +245,30 @@ Benchmark performance of single-producer-single-consumer queues `boost::lockfree
 
 I only have access to a few x86-64 machines. If you have access to different hardware feel free to submit the output file of `scripts/run-benchmarks.sh` and I will include your results into the benchmarks page.
 
-### Huge pages
+#### Huge pages
 When huge pages are available the benchmarks use 1x1GB or 16x2MB huge pages for the queues to minimise TLB misses. To enable huge pages do one of:
-```
+```bash
 sudo hugeadm --pool-pages-min 1GB:1
 sudo hugeadm --pool-pages-min 2MB:16
 ```
 Alternatively, you may like to enable [transparent hugepages][15] in your system and use a hugepage-aware allocator, such as [tcmalloc][14].
 
-### Real-time thread throttling
+#### Real-time thread throttling
 By default, Linux scheduler throttles real-time threads from consuming 100% of CPU and that is detrimental to benchmarking. Full details can be found in [Real-Time group scheduling][2]. To disable real-time thread throttling do:
-```
+```bash
 echo -1 | sudo tee /proc/sys/kernel/sched_rt_runtime_us >/dev/null
 ```
 
-## Throughput and scalability benchmark
+### Throughput and scalability benchmark
 N producer threads push a 4-byte integer into one same queue, N consumer threads pop the integers from the queue. All producers posts 1,000,000 messages in total. Total time to send and receive all the messages is measured. The benchmark is run for from 1 producer and 1 consumer up to `(total-number-of-cpus / 2)` producers/consumers to measure the scalability of different queues.
 
-## Ping-pong benchmark
+### Ping-pong benchmark
 One thread posts an integer to another thread through one queue and waits for a reply from another queue (2 queues in total). The benchmarks measures the total time of 100,000 ping-pongs, best of 10 runs. Contention is minimal here (1-producer-1-consumer, 1 element in the queue) to be able to achieve and measure the lowest latency. Reports the average round-trip time.
 
-# Contributing
+## Contributing
 Contributions are more than welcome. `.editorconfig` and `.clang-format` can be used to automatically match code formatting.
 
-# Reading material
+## Reading material
 Some books on the subject of multi-threaded programming I found quite instructive:
 
 * _Programming with POSIX Threads_ by David R. Butenhof.
@@ -278,7 +289,6 @@ Copyright (c) 2019 Maxim Egorushkin. MIT License. See the full licence in file L
 [9]: https://stackoverflow.com/a/25168942/412080
 [10]: https://en.cppreference.com/w/cpp/atomic/atomic/is_lock_free
 [11]: https://en.cppreference.com/w/cpp/language/type
-[12]: https://en.cppreference.com/w/cpp/types/is_arithmetic
 [13]: https://en.cppreference.com/w/cpp/error/assert
 [14]: https://google.github.io/tcmalloc/temeraire.html
 [15]: https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html
