Before and after, every route
Same lab pod, same data, interleaved arms. Before is the first watch-list build (batch.js), which already contains the per-item → batch change; after is everything combined (round7.js, changes #1–#18). The per-item baseline for the watch-list was 36.5 ms of CPU per page.
| Route | Encoding | CPU ms/req before | CPU ms/req after | Change | req/s before | req/s after | p99 ms before | p99 ms after | Errors | Runs |
|---|---|---|---|---|---|---|---|---|---|---|
| Watch-list page | compressed | 3.44 | 2.22 | −35% | 372 | 623 | 94 | 63 | 0 | 3+3 |
| Watch-list page | uncompressed | 2.99 | 1.55 | −48% | 402 | 746 | 89 | 50 | 0 | 3+3 |
| Single title | compressed | 3.92 | 3.02 | −23% | 364 | 496 | 95 | 81 | 0 | 3+3 |
| Single title | uncompressed | 2.92 | 1.96 | −33% | 406 | 574 | 82 | 69 | 0 | 3+3 |
| Watch-history display | compressed | 8.87 | 3.55 | −60% | 211 | 432 | 166 | 93 | 0 | 3+3 |
| Watch-history display | uncompressed | 4.00 | 2.41 | −40% | 306 | 494 | 108 | 74 | 0 | 3+3 |
| Watch-history gates | compressed | 2.42 | 1.68 | −31% | 520 | 796 | 67 | 46 | 0 | 3+3 |
| Watch-history gates | uncompressed | 1.95 | 1.25 | −36% | 579 | 894 | 58 | 41 | 0 | 3+3 |
| contentData | compressed | 2.38 | 1.03 | −57% | 475 | 1,097 | 74 | 30 | 0 | 3+3 |
| contentData | uncompressed | 2.34 | 0.96 | −59% | 481 | 1,133 | 70 | 26 | 0 | 3+3 |
| Page routes | compressed | 200 | 66.41 | −67% | 8 | 23 | 1,554 | 453 | 0 | 3+3 |
Intermediate build (contentData stage) vs start
| Route | Encoding | CPU ms/req start | CPU ms/req mid | Change | req/s start | req/s mid | p99 ms start | p99 ms mid | Errors | Runs |
|---|---|---|---|---|---|---|---|---|---|---|
| Watch-list page | compressed | 3.12 | 2.07 | −34% | 413 | 657 | 85 | 58 | 0 | 2+2 |
| Watch-list page | uncompressed | 2.66 | 1.39 | −48% | 447 | 805 | 86 | 45 | 0 | 2+2 |
| Single title | compressed | 3.96 | 3.54 | −11% | 379 | 446 | 98 | 85 | 0 | 2+2 |
| Single title | uncompressed | 2.67 | 2.11 | −21% | 450 | 544 | 77 | 57 | 0 | 2+2 |
| Watch-history display | compressed | 8.14 | 6.80 | −17% | 235 | 282 | 151 | 129 | 0 | 2+2 |
| Watch-history display | uncompressed | 3.47 | 2.85 | −18% | 351 | 427 | 96 | 83 | 0 | 2+2 |
| Watch-history gates | compressed | 1.98 | 1.60 | −20% | 651 | 850 | 55 | 56 | 0 | 2+2 |
| Watch-history gates | uncompressed | 1.56 | 1.15 | −26% | 714 | 872 | 51 | 62 | 0 | 2+2 |
| contentData | compressed | 1.93 | 0.87 | −55% | 581 | 1,257 | 56 | 25 | 0 | 2+2 |
| contentData | uncompressed | 1.87 | 0.81 | −56% | 592 | 1,288 | 49 | 19 | 0 | 2+2 |
| Page routes | compressed | 218 | 216 | −1% | 7 | 7 | 1,562 | 1,599 | 0 | 2+2 |
Where the time goes now
CPU profile of each route on the final Express build (all changes through #19; page routes profiled on #18), as a share of main-thread busy time. Idle is time the event loop waited on I/O. Categories: app our code, framework Nest/Express/validation, json parse/stringify of bodies, mongo/bson driver and decode, compression zstd cache decode + response compression, logging pino + PII sanitizer + OTel log export, telemetry spans, node runtime buffers/async context/V8.
| Route | Main thread busy | app | framework | json | mongo/bson | compression | device-detect | logging | telemetry | http/net | hash (etag/jwt) | gc | node runtime | valkey | other |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| contentData | 80% | 9% | 15% | 0% | 20% | 0% | 0% | 8% | 10% | 7% | 1% | 4% | 19% | 0% | 2% |
| Entitlement | 80% | 5% | 12% | 1% | 7% | 2% | 31% | 3% | 5% | 7% | 1% | 4% | 11% | 0% | 7% |
| Single title | 80% | 18% | 15% | 6% | 7% | 5% | 0% | 5% | 6% | 5% | 2% | 5% | 19% | 1% | 2% |
| Watch-history display | 80% | 8% | 13% | 10% | 22% | 1% | 0% | 4% | 7% | 5% | 2% | 5% | 17% | 0% | 2% |
| Watch-history gates | 79% | 10% | 18% | 1% | 18% | 2% | 0% | 6% | 10% | 6% | 1% | 4% | 18% | 0% | 3% |
| Watch-list page | 80% | 18% | 19% | 2% | 5% | 5% | 0% | 7% | 8% | 6% | 1% | 4% | 17% | 2% | 3% |
| Page routes | 82% | 21% | 4% | 21% | 3% | 12% | 0% | 2% | 2% | 2% | 3% | 8% | 14% | 1% | 2% |
What observability costs
The same build with OpenTelemetry tracing and log export switched off by env (APM_TRACES_ENABLED, APM_LOGS_ENABLED), 3 interleaved rounds per configuration. Every difference is within run-to-run spread: exporting traces and logs costs no measurable CPU. An earlier 2-round pass suggested −11%; its baseline runs spread 2.19 vs 1.69 ms, and the 3-round rerun did not reproduce it. In the profile, the export-only frames (OTLP encoding, metric attribute hashing) add up to under 2% of busy time, below what a 3-round A/B can resolve; the rest of the telemetry and logging slices is context propagation and instrumentation hooks that run either way.
| Route (uncompressed) | Telemetry | CPU ms/req (median) | vs today | Runs | req/s |
|---|---|---|---|---|---|
| Watch-list page | Traces + logs on (today) | 1.64 | — | 1.62, 1.70, 1.64 | 725 |
| Watch-list page | Traces off | 1.69 | +3% | 1.66, 1.69, 1.70 | 699 |
| Watch-list page | Traces + logs off | 1.66 | +1% | 1.67, 1.65, 1.66 | 705 |
| Single title | Traces + logs on (today) | 2.31 | — | 2.20, 2.44, 2.31 | 507 |
| Single title | Traces off | 2.19 | −5% | 2.19, 2.23, 2.19 | 526 |
| Single title | Traces + logs off | 2.30 | −0% | 2.42, 2.22, 2.30 | 510 |
| contentData | Traces + logs on (today) | 0.90 | — | 0.87, 0.90, 0.96 | 1,209 |
| contentData | Traces off | 0.95 | +6% | 0.88, 0.95, 0.96 | 1,142 |
| contentData | Traces + logs off | 0.94 | +4% | 1.01, 0.91, 0.94 | 1,152 |
NestJS NodeAdapter on the final code
Same Nest 12.3.3 dependencies and same code, only the HTTP adapter differs (Express vs @nestjs/platform-node, with the compression, route-name, query-rewrite and ETag shims from the port). 3 interleaved rounds.
| Route (uncompressed) | Express CPU ms/req | NodeAdapter | Change | Verdict | req/s Express | req/s NodeAdapter | Errors |
|---|---|---|---|---|---|---|---|
| Watch-list page | 1.63 | 1.51 | −7% | beyond spread | 739 | 785 | 0 |
| Single title | 2.61 | 2.48 | −5% | beyond spread | 502 | 535 | 0 |
| Watch-history display | 2.30 | 2.24 | −3% | within spread | 514 | 528 | 0 |
| Watch-history gates | 1.24 | 1.17 | −6% | within spread | 895 | 969 | 0 |
| contentData | 0.89 | 0.81 | −9% | within spread | 1,194 | 1,322 | 0 |
Under overload (single thread, 64–256 connections, 2 rounds) neither adapter returns an error, and NodeAdapter keeps higher throughput and lower p99 at every step:
| Route | Connections | req/s Express | req/s NodeAdapter | Change | p99 Express | p99 NodeAdapter | Errors |
|---|---|---|---|---|---|---|---|
| Watch-list page | 64 | 767 | 822 | +7% | 175 | 164 | 0 |
| Watch-list page | 128 | 830 | 884 | +7% | 288 | 264 | 0 |
| Watch-list page | 256 | 854 | 930 | +9% | 523 | 476 | 0 |
| Single title | 64 | 527 | 558 | +6% | 185 | 174 | 0 |
| Single title | 128 | 538 | 566 | +5% | 360 | 330 | 0 |
| Single title | 256 | 524 | 569 | +9% | 746 | 696 | 0 |
How far one pod goes before it saturates
Each route was stepped from 4 to 128 connections (20 s per step) until throughput stopped rising (checked from 32 connections; an earlier pass stopped I/O-bound routes too early at 8 and was rerun). The probe reports per-thread event-loop utilisation (share of time the loop is busy) and event-loop delay (how late timers fire).
| Route | 1 thread: max req/s | at conns | cores | ELU | 2 threads: max req/s | cores | Gain | 1-thread limit | 2-thread limit |
|---|---|---|---|---|---|---|---|---|---|
| Watch-list page (uncompressed) | 749 | 32 | 1.20 | 0.81 | 1,519 | 1.98 | 2.03× | event loop (one thread) | pod CPU limit (2 cores) |
| Single title (uncompressed) | 535 | 32 | 1.20 | 0.83 | 1,019 | 1.99 | 1.90× | event loop (one thread) | pod CPU limit (2 cores) |
| Single title (compressed) | 468 | 8 | 1.50 | 0.82 | 699 | 1.97 | 1.50× | event loop (one thread) | pod CPU limit (2 cores) |
| Watch-history display (uncompressed) | 499 | 32 | 1.21 | 0.82 | 864 | 1.91 | 1.73× | event loop (one thread) | pod CPU limit (2 cores) |
| Watch-history gates (uncompressed) | 863 | 32 | 1.15 | 0.82 | 1,366 | 1.71 | 1.58× | event loop (one thread) | I/O wait (Mongo/Valkey/upstream) |
| contentData (uncompressed) | 1,396 | 32 | 1.12 | 0.83 | 2,239 | 1.91 | 1.60× | event loop (one thread) | pod CPU limit (2 cores) |
| Page routes (compressed) | 10 | 4 | 1.47 | 0.77 | 14 | 1.98 | 1.38× | event loop (one thread) | pod CPU limit (2 cores) |
| Entitlement (compressed) | 248 | 64 | 1.15 | 0.82 | 332 | 1.74 | 1.34× | event loop (one thread) | I/O wait (Mongo/Valkey/upstream) |
What the sweep shows
On one thread a pod plateaus at about 1.2 of its 2 cores with the event loop ~80% busy: the single JS thread is the bottleneck, and extra connections only add latency. On two worker threads (SO_REUSEPORT, one process) the same pod reaches its 2-core limit; peak throughput rises 1.3–2.0×: most on CPU-bound uncompressed routes, least where compression already used the spare core or requests wait on Mongo and upstream services. The limit then is the container's CPU quota, not Node.
Watch-list page (uncompressed) — max 749 req/s on 1 thread, 1,519 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 606 | 5.4 | 21 | 1.19 | 0.81 | 20 | 0 |
| 1 | 8 | 679 | 10 | 31 | 1.19 | 0.80 | 20 | 0 |
| 1 | 16 | 718 | 19.4 | 58 | 1.19 | 0.82 | 25 | 0 |
| 1 | 32 | 749 | 36.8 | 96 | 1.20 | 0.81 | 33 | 0 |
| 1 | 64 | 741 | 70.9 | 181 | 1.25 | 0.81 | 71 | 0 |
| 2 | 4 | 1,137 | 2.6 | 16 | 1.89 | 0.71, 0.70 | 30, 30 | 0 |
| 2 | 8 | 1,389 | 4.4 | 26 | 1.99 | 0.80, 0.77 | 33, 27 | 0 |
| 2 | 16 | 1,519 | 8.3 | 44 | 1.98 | 0.82, 0.81 | 34, 30 | 0 |
| 2 | 32 | 1,519 | 17.6 | 76 | 1.99 | 0.81, 0.81 | 46, 36 | 0 |
Single title (uncompressed) — max 535 req/s on 1 thread, 1,019 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 488 | 7.1 | 22 | 1.16 | 0.79 | 18 | 0 |
| 1 | 8 | 508 | 14.3 | 43 | 1.19 | 0.82 | 22 | 0 |
| 1 | 16 | 535 | 27.9 | 72 | 1.18 | 0.78 | 28 | 0 |
| 1 | 32 | 535 | 56.7 | 113 | 1.20 | 0.83 | 45 | 0 |
| 2 | 4 | 751 | 4.4 | 21 | 1.71 | 0.46, 0.80 | 25, 22 | 0 |
| 2 | 8 | 888 | 8 | 45 | 1.95 | 0.66, 0.79 | 39, 40 | 0 |
| 2 | 16 | 977 | 14.1 | 70 | 1.99 | 0.82, 0.81 | 47, 47 | 0 |
| 2 | 32 | 1,019 | 27.9 | 96 | 1.99 | 0.79, 0.79 | 49, 50 | 0 |
| 2 | 64 | 997 | 60.4 | 168 | 1.99 | 0.83, 0.82 | 76, 62 | 0 |
Single title (compressed) — max 468 req/s on 1 thread, 699 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 430 | 7.8 | 51 | 1.37 | 0.78 | 22 | 0 |
| 1 | 8 | 468 | 15.5 | 56 | 1.50 | 0.82 | 26 | 0 |
| 1 | 16 | 451 | 31.6 | 90 | 1.59 | 0.82 | 41 | 0 |
| 2 | 4 | 624 | 5.2 | 39 | 1.85 | 0.43, 0.69 | 27, 28 | 0 |
| 2 | 8 | 699 | 10.9 | 54 | 1.97 | 0.83, 0.41 | 36, 32 | 0 |
| 2 | 16 | 668 | 18.3 | 85 | 2.00 | 0.80, 0.78 | 49, 47 | 0 |
Watch-history display (uncompressed) — max 499 req/s on 1 thread, 864 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 460 | 7.9 | 21 | 1.14 | 0.79 | 18 | 0 |
| 1 | 8 | 479 | 15.3 | 53 | 1.18 | 0.80 | 23 | 0 |
| 1 | 16 | 495 | 31.1 | 75 | 1.19 | 0.82 | 25 | 0 |
| 1 | 32 | 499 | 62.5 | 118 | 1.21 | 0.82 | 35 | 0 |
| 2 | 4 | 598 | 5.6 | 26 | 1.34 | 0.28, 0.67 | 18, 21 | 0 |
| 2 | 8 | 642 | 8.3 | 60 | 1.46 | 0.40, 0.69 | 25, 28 | 0 |
| 2 | 16 | 797 | 13.6 | 79 | 1.77 | 0.64, 0.76 | 38, 43 | 0 |
| 2 | 32 | 864 | 29.6 | 102 | 1.91 | 0.80, 0.78 | 52, 48 | 0 |
| 2 | 64 | 805 | 80.1 | 173 | 1.93 | 0.78, 0.76 | 59, 58 | 0 |
Watch-history gates (uncompressed) — max 863 req/s on 1 thread, 1,366 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 722 | 4.4 | 15 | 1.01 | 0.73 | 16 | 0 |
| 1 | 8 | 837 | 7.9 | 23 | 1.14 | 0.83 | 17 | 0 |
| 1 | 16 | 860 | 17.2 | 41 | 1.15 | 0.83 | 20 | 0 |
| 1 | 32 | 863 | 35 | 78 | 1.15 | 0.82 | 25 | 0 |
| 2 | 4 | 863 | 3.6 | 13 | 1.24 | 0.61, 0.27 | 14, 13 | 0 |
| 2 | 8 | 1,291 | 4.9 | 24 | 1.64 | 0.79, 0.42 | 20, 22 | 0 |
| 2 | 16 | 1,351 | 6.9 | 60 | 1.70 | 0.69, 0.63 | 30, 30 | 0 |
| 2 | 32 | 1,366 | 13.3 | 86 | 1.71 | 0.62, 0.69 | 33, 35 | 0 |
contentData (uncompressed) — max 1,396 req/s on 1 thread, 2,239 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 1,260 | 3 | 6 | 1.12 | 0.80 | 11 | 0 |
| 1 | 8 | 1,337 | 5.7 | 11 | 1.10 | 0.80 | 13 | 0 |
| 1 | 16 | 1,361 | 11.3 | 22 | 1.12 | 0.82 | 15 | 0 |
| 1 | 32 | 1,396 | 22.6 | 45 | 1.12 | 0.83 | 20 | 0 |
| 2 | 4 | 1,592 | 2.4 | 7 | 1.40 | 0.52, 0.52 | 12, 12 | 0 |
| 2 | 8 | 1,698 | 3.4 | 51 | 1.44 | 0.41, 0.67 | 14, 12 | 0 |
| 2 | 16 | 1,781 | 5.1 | 70 | 1.56 | 0.68, 0.57 | 20, 19 | 0 |
| 2 | 32 | 2,196 | 10.4 | 64 | 1.82 | 0.73, 0.71 | 21, 19 | 0 |
| 2 | 64 | 2,239 | 23.9 | 83 | 1.91 | 0.79, 0.79 | 28, 36 | 0 |
Page routes (compressed) — max 10 req/s on 1 thread, 14 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 10 | 371 | 1,001 | 1.47 | 0.77 | 176 | 0 |
| 1 | 8 | 10 | 819 | 1,522 | 1.55 | 0.82 | 210 | 0 |
| 2 | 4 | 14 | 244 | 733 | 1.98 | 0.76, 0.77 | 194, 197 | 0 |
| 2 | 8 | 14 | 499 | 2,102 | 1.99 | 0.82, 0.84 | 233, 337 | 0 |
Entitlement (compressed) — max 248 req/s on 1 thread, 332 on 2
| Threads | Connections | req/s | p50 ms | p99 ms | CPU cores | Event-loop utilisation (per thread) | Event-loop delay p99 ms | Errors |
|---|---|---|---|---|---|---|---|---|
| 1 | 4 | 214 | 18.9 | 54 | 1.10 | 0.77 | 16 | 0 |
| 1 | 8 | 233 | 34.4 | 88 | 1.13 | 0.80 | 20 | 0 |
| 1 | 16 | 234 | 65 | 161 | 1.14 | 0.81 | 26 | 0 |
| 1 | 32 | 244 | 128 | 283 | 1.18 | 0.82 | 38 | 0 |
| 1 | 64 | 248 | 281 | 582 | 1.15 | 0.82 | 69 | 0 |
| 2 | 4 | 188 | 12.9 | 174 | 1.17 | 0.43, 0.43 | 14, 14 | 0 |
| 2 | 8 | 280 | 22 | 92 | 1.55 | 0.59, 0.59 | 18, 19 | 0 |
| 2 | 16 | 302 | 43.5 | 174 | 1.66 | 0.57, 0.70 | 24, 29 | 0 |
| 2 | 32 | 326 | 87.3 | 288 | 1.75 | 0.68, 0.67 | 35, 34 | 0 |
| 2 | 64 | 332 | 196 | 527 | 1.74 | 0.74, 0.63 | 59, 44 | 0 |
One thread vs two worker threads
Same build and pod limit; 32 connections; median of 2 rounds per arm. Two worker_threads each run the unmodified app and listen on :3002 with reusePort, so the kernel spreads connections. Per-thread CPU ticks were equal (1,419 vs 1,423) in a spot check.
| Route | Encoding | req/s 1 thread | req/s 2 threads | Change | cores 1t | cores 2t | CPU ms/req 1t | CPU ms/req 2t | p99 1t | p99 2t | Errors |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Watch-list page | compressed | 581 | 805 | +39% | 1.36 | 2.00 | 2.34 | 2.48 | 110 | 110 | 0 |
| Watch-list page | uncompressed | 706 | 1,087 | +54% | 1.20 | 2.00 | 1.70 | 1.83 | 95 | 92 | 0 |
| Single title | uncompressed | 528 | 808 | +53% | 1.20 | 2.00 | 2.27 | 2.48 | 111 | 112 | 0 |
| Single title | compressed | 388 | 504 | +30% | 1.73 | 2.00 | 4.45 | 3.96 | 145 | 179 | 0 |
| Watch-history display | compressed | 329 | 471 | +43% | 1.77 | 2.00 | 5.39 | 4.24 | 183 | 183 | 0 |
| Watch-history display | uncompressed | 469 | 751 | +60% | 1.19 | 2.00 | 2.54 | 2.67 | 124 | 122 | 0 |
| contentData | compressed | 1,046 | 1,723 | +65% | 1.13 | 2.00 | 1.08 | 1.16 | 62 | 61 | 0 |
| contentData | uncompressed | 1,080 | 1,815 | +68% | 1.10 | 1.94 | 1.02 | 1.08 | 52 | 62 | 0 |