Direct dispatch soak¶
A development note. It says what the dispatch soak harness
(src/script/dispatch_soak_bench_tests.rs) measures, how to run it, and what it
found when pointed at the direct awaitable dispatch of #864 to #866, which #870
backed out.
Nothing here is operator documentation. It is the measurement record behind a design decision, kept next to the code so the next attempt starts from numbers.
Why a soak and not the microbench¶
Direct dispatch stepped an async def handler that awaits only siphon's own
awaitables on the Python executor, with no asyncio task. A per-handler
microbench put it at 3 to 5 times less CPU per call. An end-to-end A/B on
CPU-limited nodes then showed the interpreter attach wait peaking 10 to 40
times higher, the executor pool full with 22 to 25 jobs in flight where it had
2 to 7, ten times the pending coroutines, and a quarter less goodput.
The microbench could not have seen that. Its awaitables mostly finished before the handler suspended, its heap was empty, a case ran for 3 seconds, nothing timed an attach, the handlers did no work between awaits, and it had every CPU of the machine.
The soak changes each of those: every await suspends on a log-normal delay, arrivals are open loop, 30 % of the load is sync handlers that block released, the script keeps a live heap that changes, and a thread of its own times an attach every millisecond.
Running it¶
Build the test binary on free-threaded Python and run the one ignored test by name:
PYO3_PYTHON=python3.14t cargo test --release --lib --no-run
taskset -c 0-5 target/release/deps/siphon-<hash> --ignored --nocapture \
--test-threads=1 script::dispatch_soak_bench_tests::dispatch_soak
Everything is set by SOAK_* environment variables, listed at the top of the
file. The run that matters is the one under a CPU quota:
SOAK_RATE=300 SOAK_WORK=1100 SOAK_HEAP_MB=0 SOAK_SECONDS=120 \
systemd-run --user --scope -p CPUQuota=100% taskset -c 0-5 \
target/release/deps/siphon-<hash> --ignored --nocapture --test-threads=1 \
script::dispatch_soak_bench_tests::dispatch_soak
To compare two commits, cherry-pick the harness commit onto each and build each into its own binary. Two worktrees sharing one target directory build the same artifact name, and cargo will hand back the first tree's binary for the second unless the crate is touched in between, so copy each binary out before building the next.
What was measured¶
Free-threaded CPython 3.14.2, release build, 6 tokio workers, 6 asyncio
drivers, executor sized by resolve_sizing with a queue of 1024. "main" is
the tree after #870. "direct" is the tree at #866 with the harness
cherry-picked. One machine, shared with other work, so single figures move by
10 to 20 % between repeats. The outcomes below are not that kind of
difference.
Columns: the attach wait of the pinned 1 ms probe and of the unpinned 100 ms
probe (the one siphon's own liveness probe takes), collector pauses from
gc.callbacks (an upper bound, see the CPython notes), the executor's
in-flight mean and final pool size, pending coroutines, goodput (answered
within 2 s), requests shed, and CPU per answered request.
With all six CPUs: no difference¶
300 MB heap, no work between awaits, the executor at 12 threads growing to 48.
| run | attach p99 ms | attach max ms | unpinned p99 ms | unpinned max ms | gc n | gc p99 ms | gc max ms | in-flight | pool | pending | goodput /s | shed | CPU us/req |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 150/s main | 0.001 | 302.3 | 0.69 | 252.3 | 4 | 532 | 532 | 1.4 | 39 | 4 | 149 | 0 | 665 |
| 150/s direct | 0.002 | 239.3 | 0.88 | 239.4 | 3 | 437 | 437 | 1.7 | 45 | 4 | 150 | 0 | 540 |
| 300/s main | 0.002 | 347.7 | 0.58 | 328.8 | 7 | 659 | 659 | 2.8 | 48 | 8 | 300 | 0 | 545 |
| 300/s direct | 0.002 | 304.8 | 0.56 | 271.0 | 7 | 557 | 557 | 3.4 | 48 | 8 | 301 | 0 | 393 |
| 600/s main | 0.000 | 262.2 | 0.59 | 234.8 | 3 | 512 | 512 | 5.2 | 48 | 15 | 599 | 0 | 267 |
| 600/s direct | 0.002 | 292.1 | 0.64 | 245.1 | 3 | 557 | 557 | 5.4 | 48 | 15 | 597 | 0 | 230 |
Here direct dispatch gives the same service for 14 to 28 % less CPU. That is the direction the microbench showed and a small part of its 3 to 5 times: a request here is more than its awaits. The attach wait peaks at the length of a full collection of the 300 MB heap in both, because that is the only long stop-the-world either has.
Under a one-CPU quota: main carries the load, direct can tip over¶
CPUQuota=100%, no heap to speak of, SOAK_WORK=1100 (about 1.2 ms of Python
per request spread over its steps). The quota sizes the executor at 4 threads
growing to 32. Runs of 120 s; runs 3 and 4 are 90 s and the 600 per second
pair 60 s.
| run | attach p99 ms | attach max ms | unpinned p99 ms | unpinned max ms | gc n | gc p99 ms | gc max ms | in-flight | pool | pending | goodput /s | shed | CPU us/req |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 150/s main | 0.003 | 12.6 | 0.65 | 1.3 | 2 | 28 | 28 | 1.1 | 8 | 4 | 149 | 0 | 1798 |
| 150/s direct | 0.003 | 11.0 | 0.62 | 4.3 | 2 | 21 | 21 | 1.6 | 8 | 4 | 152 | 0 | 1623 |
| 300/s main | 0.001 | 19.6 | 0.61 | 9.4 | 4 | 44 | 44 | 2.6 | 12 | 8 | 300 | 0 | 1571 |
| 300/s direct | 0.004 | 2989.0 | 1.31 | 605.9 | 5 | 5820 | 5820 | 32.0 | 32 | 1166 | 131 | 744 | 3390 |
| 300/s main, run 2 | 0.003 | 16.3 | 0.62 | 2.6 | 4 | 44 | 44 | 3.1 | 16 | 8 | 300 | 0 | 1772 |
| 300/s direct, run 2 | 0.005 | 192.8 | 3.58 | 102.3 | 4 | 891 | 891 | 31.8 | 32 | 1381 | 119 | 663 | 3384 |
| 300/s main, run 3 | 0.002 | 16.7 | 0.60 | 1.5 | 3 | 45 | 45 | 2.7 | 13 | 8 | 298 | 0 | 1811 |
| 300/s direct, run 3 | 0.003 | 17.0 | 0.62 | 16.8 | 3 | 42 | 42 | 2.8 | 19 | 8 | 300 | 0 | 1976 |
| 300/s direct, run 4 | 0.002 | 14.0 | 0.61 | 5.7 | 3 | 39 | 39 | 3.0 | 14 | 8 | 299 | 0 | 1757 |
| 600/s main | 0.005 | 137.5 | 38.14 | 128.8 | 6 | 2297 | 2297 | 16.9 | 32 | 13746 | 180 | 0 | 1660 |
| 600/s direct | 0.004 | 213.1 | 3.26 | 109.4 | 3 | 1181 | 1181 | 31.8 | 32 | 1168 | 82 | 19722 | 3686 |
| 300/s main, 300 MB heap | 0.004 | 439.1 | 1.21 | 381.1 | 5 | 1011 | 1011 | 7.5 | 32 | 11232 | 89 | 0 | 3365 |
| 300/s direct, 300 MB heap | 0.005 | 238.0 | 0.30 | 228.0 | 2 | 714 | 714 | 32.0 | 32 | 1890 | 12 | 18803 | 6893 |
At 150 per second both are healthy and alike. At 300 per second main uses about half the quota and answers everything, in every run. Direct either does the same, or tips into a second state and stays in it: every executor thread busy, a queue wait above a second, over a thousand coroutines pending, requests shed, twice the CPU per request, collector pauses of seconds where main has 44 ms, and the attach wait peaking 12 and 150 times higher on the pinned probe in the first two pairs (39 and 64 times on the unpinned one). The p99 of the pinned probe does not move in any run: the stalls are a handful of long ones, not many short ones.
It tipped in the first two runs and not in the next two. Counting the control runs of the next section, the unchanged direct tree tipped in 4 of 8 runs at this load, and main in none. So the ratio is not a steady property of direct dispatch. It is what the collapsed state looks like, and at this load a run of a minute or two enters it about half the time.
At 600 per second both are past the quota, and they fail differently: main sheds nothing and queues without bound on its drivers, direct fills the executor queue and sheds. With a 300 MB heap under the same quota (last two rows) the load is past what main carries too, so that pair says nothing about the two paths.
One variable at a time¶
All at 300 per second under the one-CPU quota, 90 s (the first two unchanged runs 120 s), on the direct tree with one change each. Throwaway patches, not in this repository. A run counts as collapsed when its mean in-flight is above 16.
| variant | runs | collapsed | in-flight | attach max ms | goodput /s | CPU us/req |
|---|---|---|---|---|---|---|
| main | 3 | 0 | 2.6 to 3.1 | 16 to 20 | 298 to 300 | 1571 to 1811 |
| direct, unchanged | 8 | 4 | 2.7 to 32.0 | 14 to 2989 | 89 to 302 | 1757 to 3583 |
| 1: continuations on their own lane (4, 12 or 32 threads) | 5 | 0 | 2.5 to 3.0 | 10 to 19 | 298 to 301 | 1523 to 1997 |
| 2: each coroutine pinned to one lane thread (4 or 12) | 2 | 0 | 2.5 to 2.8 | 10 to 10 | 299 to 302 | 1384 to 1415 |
| 3: executor max 8 | 4 | 0 | 2.7 to 3.0 | 11 to 20 | 300 to 300 | 1512 to 1860 |
| 3: executor core 12 instead of 4 | 3 | 1 | 2.7 to 45.0 | 16 to 111 | 181 to 301 | 1843 to 3332 |
| 4: attached teardown | 3 | 1 | 2.9 to 31.9 | 12 to 84 | 134 to 302 | 1608 to 3343 |
| 6: no asyncio-using handler | 1 | 1 | 31.8 | 97 | 76 | 3883 |
Every run, for the detail:
| run | attach p99 ms | attach max ms | unpinned p99 ms | unpinned max ms | gc n | gc p99 ms | gc max ms | in-flight | pool | pending | goodput /s | shed | CPU us/req |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| unchanged | 0.005 | 119.5 | 2.10 | 93.2 | 4 | 4207 | 4207 | 27.2 | 32 | 1070 | 134 | 262 | 3132 |
| unchanged | 0.003 | 14.4 | 0.55 | 1.1 | 3 | 42 | 42 | 3.0 | 17 | 8 | 302 | 0 | 2068 |
| unchanged | 0.002 | 18.0 | 0.59 | 1.1 | 3 | 35 | 35 | 2.7 | 12 | 8 | 296 | 0 | 1817 |
| unchanged | 0.005 | 189.3 | 3.27 | 106.8 | 3 | 1406 | 1406 | 32.0 | 32 | 1426 | 89 | 2214 | 3583 |
| 1: lane, 4 threads | 0.001 | 10.0 | 0.83 | 5.2 | 3 | 25 | 25 | 2.8 | 12 | 8 | 301 | 0 | 1523 |
| 1: lane, 4 threads | 0.002 | 11.7 | 0.69 | 1.2 | 3 | 27 | 27 | 2.7 | 12 | 8 | 298 | 0 | 1659 |
| 1: lane, 12 threads | 0.001 | 15.8 | 0.72 | 7.7 | 3 | 27 | 27 | 2.8 | 14 | 8 | 300 | 0 | 1532 |
| 1: lane, 32 threads | 0.003 | 15.3 | 0.61 | 1.8 | 3 | 31 | 31 | 3.0 | 10 | 8 | 299 | 0 | 1895 |
| 1: lane, 32 threads | 0.002 | 19.4 | 0.62 | 1.5 | 3 | 37 | 37 | 2.5 | 10 | 8 | 299 | 0 | 1997 |
| 2: pinned, 12 threads | 0.001 | 10.0 | 0.77 | 0.9 | 3 | 28 | 28 | 2.8 | 14 | 8 | 299 | 0 | 1415 |
| 2: pinned, 4 threads | 0.001 | 10.3 | 0.65 | 2.4 | 3 | 37 | 37 | 2.5 | 10 | 8 | 302 | 0 | 1384 |
| 3: executor max 8 | 0.002 | 14.3 | 0.60 | 1.2 | 3 | 33 | 33 | 2.7 | 8 | 8 | 300 | 0 | 1512 |
| 3: executor max 8 | 0.003 | 19.7 | 0.73 | 2.1 | 3 | 34 | 34 | 2.8 | 8 | 8 | 300 | 0 | 1860 |
| 3: executor max 8 | 0.002 | 15.0 | 0.48 | 0.7 | 3 | 32 | 32 | 2.7 | 8 | 8 | 300 | 0 | 1742 |
| 3: executor max 8 | 0.002 | 11.2 | 0.62 | 2.4 | 3 | 32 | 32 | 3.0 | 8 | 8 | 300 | 0 | 1636 |
| 3: executor core 12 | 0.002 | 16.8 | 0.59 | 4.6 | 3 | 38 | 38 | 2.8 | 18 | 8 | 297 | 0 | 1866 |
| 3: executor core 12 | 0.005 | 110.7 | 2.21 | 112.0 | 3 | 4701 | 4701 | 45.0 | 48 | 1099 | 181 | 0 | 3332 |
| 3: executor core 12 | 0.002 | 16.2 | 0.60 | 3.8 | 3 | 48 | 48 | 2.7 | 13 | 8 | 301 | 0 | 1843 |
| 4: attached teardown | 0.001 | 15.2 | 0.80 | 0.9 | 3 | 26 | 26 | 2.9 | 19 | 8 | 301 | 0 | 1608 |
| 4: attached teardown | 0.001 | 12.5 | 0.61 | 7.0 | 3 | 36 | 36 | 3.2 | 14 | 8 | 302 | 0 | 1647 |
| 4: attached teardown | 0.004 | 83.9 | 0.62 | 16.6 | 3 | 116 | 116 | 31.9 | 32 | 1026 | 134 | 99 | 3343 |
| 5: asyncio-only handlers, main | 0.000 | 11.4 | 0.61 | 0.8 | 3 | 29 | 29 | 2.6 | 13 | 7 | 297 | 0 | 1098 |
| 5: asyncio-only handlers, direct | 0.001 | 13.8 | 0.62 | 1.3 | 3 | 36 | 36 | 2.7 | 11 | 7 | 301 | 0 | 1303 |
| 5: asyncio-only handlers, direct, stock loop class | 0.001 | 13.0 | 0.65 | 2.2 | 3 | 33 | 33 | 2.9 | 16 | 7 | 300 | 0 | 1321 |
| 6: no asyncio-using handler | 0.005 | 96.9 | 0.64 | 81.1 | 3 | 498 | 498 | 31.8 | 32 | 803 | 76 | 4030 | 3883 |
Three more pairs away from that operating point:
| run | attach p99 ms | attach max ms | unpinned p99 ms | unpinned max ms | gc n | gc p99 ms | gc max ms | in-flight | pool | pending | goodput /s | shed | CPU us/req |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 7: 500 MB heap, 300/s, main | 0.004 | 454.0 | 0.64 | 360.9 | 3 | 1195 | 1195 | 8.1 | 32 | 9072 | 89 | 0 | 3348 |
| 7: 500 MB heap, 300/s, direct | 0.003 | 192.2 | 0.60 | 190.2 | 1 | 709 | 709 | 32.0 | 32 | 1537 | 0 | 16613 | 8551 |
| 7: 500 MB heap, 150/s, main | 0.003 | 414.5 | 0.80 | 385.0 | 3 | 1310 | 1310 | 2.1 | 32 | 110 | 123 | 0 | 4419 |
| 7: 500 MB heap, 150/s, direct | 0.004 | 400.4 | 0.62 | 362.2 | 3 | 1196 | 1196 | 26.2 | 32 | 850 | 69 | 0 | 5972 |
| no quota, executor core 4, main | 0.002 | 13.8 | 0.61 | 0.9 | 3 | 42 | 42 | 2.6 | 10 | 8 | 300 | 0 | 1802 |
| no quota, executor core 4, direct | 0.003 | 20.3 | 0.63 | 1.4 | 3 | 32 | 32 | 2.7 | 12 | 8 | 295 | 0 | 1872 |
Reading it¶
With half the unchanged runs healthy, one healthy run of a variant says little. What follows leans on the counts: if a change did nothing, 7 healthy runs in a row would turn up about once in a hundred tries, and 4 about once in sixteen.
Continuations in the shared executor queue are what lets it tip. With continuations on threads of their own, direct dispatch did not collapse in 7 runs out of 7, whether the lane had 4, 12 or 32 threads and whether or not a coroutine kept to one of them. In the shared queue a continuation waits behind sync handlers that hold a worker for 5 to 50 ms and behind new arrivals, on a pool that starts at 4 threads. On main a continuation never enters that queue: it runs on an asyncio driver.
The pool growing far past the CPUs is what keeps it down. With the
executor capped at 8 threads and the queue still shared, it did not collapse
in 4 runs out of 4. Starting the pool at 12 instead of 4 did not prevent it
(1 of 3 collapsed, at 45 in flight of 48), because the pool can still grow.
The executor adds threads while there is a backlog and never takes one away,
so one bad moment leaves 32 threads running handler Python on one CPU's worth
of quota, and from there the CPU per request doubles and the backlog cannot
drain. main cannot get there: a suspended handler's Python runs on the
asyncio drivers, and there are only script.async_pool_size of those.
Why that many runnable attached threads is so costly was measured apart from siphon, in plain free-threaded Python on six CPUs: one stop-the-world took about 30 microseconds with 2 busy threads, between 0.03 and 0.7 ms with 6, and 13 to 28 ms with 24. A stop-the-world has to wait for every attached thread to reach a safe point, and a thread that is runnable but not running reaches none. Under a quota a throttled thread is exactly that, for up to a period. The collector pauses above say the same thing: 44 ms on main, 0.1 to 5.8 s on collapsed direct, for the same small heap.
Thread hopping and teardown: no effect shown. Pinning each coroutine's continuations to one thread did no better than a shared lane, which already did not collapse. Letting go of a finished coroutine's Python objects attached, on the thread that ran its last step, instead of from a released thread through pyo3's deferred-decref pool, collapsed in 1 run of 3. That is not distinguishable from the unchanged tree.
A stop-the-world that is not the collector's was not needed. Without any asyncio-using handler direct dispatch collapsed the same way. The cross-thread task stop-the-world is real (next section), but with a handler remembered as an asyncio user after its first call it happens once per handler, not once per call.
The driver loop subclass costs nothing measurable. With asyncio-only handlers nothing is driven directly and all three builds are healthy. The direct tree costs the same CPU with its loop subclass as with the stock loop class, about a fifth more than main either way, in single runs.
Heap size sets the floor of the attach wait, not the collapse. Both trees stop for a full collection, and that pause grows with the live heap: about 10 to 20 ms with no heap, 240 to 350 ms with 300 MB on six CPUs, 400 ms with 500 MB under the quota. With a large heap main's own peak is therefore as high as direct's and the attach wait no longer tells them apart, while in-flight, pending coroutines and goodput still do (the 500 MB pair at 150 per second).
Without the quota it did not happen. The same load with the same 4-thread pool and all six CPUs was healthy on both trees, in one run each.
CPython notes¶
Read in the 3.14.8 sources; the same code is in 3.14.2, which the runs used.
gc.callbacksbrackets the whole collection. InPython/gc_free_threading.c,gc_collect_maincalls the callbacks with"start"beforegc_collect_internaland with"stop"after it. Inside, the world is stopped twice (refcount merge, marking, finding the unreachable set, clearing weak references; then, after finalizers have run with the world started, resurrection handling and free lists), and the garbage is deleted with the world running. So a callback timing is an upper bound on the stop-the-world, and on a starved process it also counts the time the collecting thread spent waiting for a CPU.- A task that finishes on another thread than it was made on stops the
world. In
Modules/_asynciomodule.c,Task.__init__records the creating thread, andunregister_taskstops the world to unlink the task when the calling thread is a different one.future_schedule_callbackscalls it when a task finishes, andTaskObj_dealloccalls it again when the task is freed. Measured: about 1 microsecond per task finished on its own thread, 25 to 670 microseconds on another one with 2 to 6 busy threads. Dropping the last reference to a finished task on another thread did not show that cost in the same test (0.02 to 0.03 microseconds per task). The likely reason is that the object is handed back to its owning thread to free, which was not checked in the source. _swap_current_tasktakes any object. It checks that the given loop is the calling thread's running loop and stores the object. Nothing requires anasyncio.Task.current_task(loop)for another thread's loop stops the world, to walk the threads. The fast path is the calling thread's own running loop.- When a collection starts. The young count has to pass both the threshold and a quarter of the live objects at the last collection, and then either the process has grown by a tenth since, or 40 thresholds' worth of new objects have been deferred. A large live heap therefore collects rarely, and each collection is a full one.
Request and Diameter handlers on the runner¶
The redesign steps a coroutine on its asyncio driver's runner, never on the
executor. With request handlers routed through it (SOAK_DIRECT=1), the mix
was moved onto the request path and given a gathering handler:
call=10,request=40,request_gather=15,sync=20,gather=5,wrapped=5,wrapped_asyncio=5.
request_gather is tasks.gather of a script coroutine and one awaitable.
One binary, the switch off and on in turn, four runs each, 300/s.
Six CPUs, 300 MB heap, 120 s:
| Switch | Attach max ms | Unpinned max ms | GC max ms | In-flight max | Pending max | Goodput /s | Late | CPU us/request |
|---|---|---|---|---|---|---|---|---|
| off | 244 to 265 | 184 to 198 | 454 to 485 | 8 to 48 | 39 to 74 | 298 to 299 | 0 | 449 to 494 |
| on | 252 to 276 | 169 to 246 | 464 to 505 | 5 to 15 | 49 to 69 | 300 to 302 | 0 | 364 to 403 |
One-CPU quota, no heap, SOAK_WORK=1100, 90 s:
| Switch | Attach max ms | Unpinned max ms | GC max ms | In-flight max | Pool at end | Pending max | Goodput /s | Late | CPU us/request |
|---|---|---|---|---|---|---|---|---|---|
| off, 3 runs | 10 to 17 | 0.9 to 1.5 | 31 to 32 | 4 to 6 | 10 to 13 | 21 to 22 | 300 to 302 | 0 | 1710 to 1792 |
| off, 1 run | 85 | 609 | 1208 | 20 | 32 | 8424 | 67 | 21241 | 3301 |
| on, 4 runs | 11 to 15 | 0.9 to 1.2 | 27 to 33 | 5 to 7 | 8 to 16 | 20 to 21 | 299 to 301 | 0 | 1594 to 1698 |
No run failed or shed a request and the runners refused none. With the switch on, about 25,000 handlers per run (19,000 under the quota) finished without an asyncio task. The attach wait with the 300 MB heap is the collector scanning that heap, the same in both arms. The one collapse was a run with the switch off: the executor grew to its ceiling and never caught up, which is the failure the quota run exists to show, on the asyncio path this time. Four runs an arm do not say how often either arm collapses, only that the switch did not in these.
Limits of this record¶
- One machine, not a quiet one, and runs of a minute or two. The collapse is an event that a run has or does not have, so the variant counts are small samples. Where a row is a single run, it says so.
- The quota run sits at half of what main carries on this machine when the machine has cores to spare. With every core busy elsewhere (load average above 40) a short main run used the whole quota and fell behind too, so the pairs are only comparable when taken under similar load. The four unchanged direct runs that collapsed started at a load average of 4 to 11, and three of the four that stayed healthy at 25 to 51. Eight runs do not settle whether a busy machine makes the collapse less likely (a process that cannot burn its quota in a burst is throttled less), so that is an observation, not a finding.
- The attach wait separates only once direct dispatch has collapsed. Below that point the two trees have the same attach wait, and the harness shows no steady 10 to 40 times ratio on a healthy node.
- The quota is applied from outside (
systemd-run). The harness prints the quota it runs under but does not set one. - The session heap and the work between awaits are a model. A script that holds a lock or calls a blocking API between awaits will sit elsewhere on the same curve.