Content CPU · results

Results and capacity

Before/after on every route, where the CPU goes now, and how many requests a 2-CPU pod serves before it saturates on one thread vs two.

Measured by Claude (agent) in the HOI-4097 UAT lab

Before and after, every route

Same lab pod, same data, interleaved arms. Before is the first watch-list build (batch.js), which already contains the per-item → batch change; after is everything combined (round7.js, changes #1–#18). The per-item baseline for the watch-list was 36.5 ms of CPU per page.

RouteEncodingCPU ms/req beforeCPU ms/req afterChangereq/s beforereq/s afterp99 ms beforep99 ms afterErrorsRuns
Watch-list pagecompressed3.442.22−35%372623946303+3
Watch-list pageuncompressed2.991.55−48%402746895003+3
Single titlecompressed3.923.02−23%364496958103+3
Single titleuncompressed2.921.96−33%406574826903+3
Watch-history displaycompressed8.873.55−60%2114321669303+3
Watch-history displayuncompressed4.002.41−40%3064941087403+3
Watch-history gatescompressed2.421.68−31%520796674603+3
Watch-history gatesuncompressed1.951.25−36%579894584103+3
contentDatacompressed2.381.03−57%4751,097743003+3
contentDatauncompressed2.340.96−59%4811,133702603+3
Page routescompressed20066.41−67%8231,55445303+3
Intermediate build (contentData stage) vs start
RouteEncodingCPU ms/req startCPU ms/req midChangereq/s startreq/s midp99 ms startp99 ms midErrorsRuns
Watch-list pagecompressed3.122.07−34%413657855802+2
Watch-list pageuncompressed2.661.39−48%447805864502+2
Single titlecompressed3.963.54−11%379446988502+2
Single titleuncompressed2.672.11−21%450544775702+2
Watch-history displaycompressed8.146.80−17%23528215112902+2
Watch-history displayuncompressed3.472.85−18%351427968302+2
Watch-history gatescompressed1.981.60−20%651850555602+2
Watch-history gatesuncompressed1.561.15−26%714872516202+2
contentDatacompressed1.930.87−55%5811,257562502+2
contentDatauncompressed1.870.81−56%5921,288491902+2
Page routescompressed218216−1%771,5621,59902+2

Where the time goes now

CPU profile of each route on the final Express build (all changes through #19; page routes profiled on #18), as a share of main-thread busy time. Idle is time the event loop waited on I/O. Categories: app our code, framework Nest/Express/validation, json parse/stringify of bodies, mongo/bson driver and decode, compression zstd cache decode + response compression, logging pino + PII sanitizer + OTel log export, telemetry spans, node runtime buffers/async context/V8.

RouteMain thread busyappframeworkjsonmongo/bsoncompressiondevice-detectloggingtelemetryhttp/nethash (etag/jwt)gcnode runtimevalkeyother
contentData80%9%15%0%20%0%0%8%10%7%1%4%19%0%2%
Entitlement80%5%12%1%7%2%31%3%5%7%1%4%11%0%7%
Single title80%18%15%6%7%5%0%5%6%5%2%5%19%1%2%
Watch-history display80%8%13%10%22%1%0%4%7%5%2%5%17%0%2%
Watch-history gates79%10%18%1%18%2%0%6%10%6%1%4%18%0%3%
Watch-list page80%18%19%2%5%5%0%7%8%6%1%4%17%2%3%
Page routes82%21%4%21%3%12%0%2%2%2%3%8%14%1%2%

What observability costs

The same build with OpenTelemetry tracing and log export switched off by env (APM_TRACES_ENABLED, APM_LOGS_ENABLED), 3 interleaved rounds per configuration. Every difference is within run-to-run spread: exporting traces and logs costs no measurable CPU. An earlier 2-round pass suggested −11%; its baseline runs spread 2.19 vs 1.69 ms, and the 3-round rerun did not reproduce it. In the profile, the export-only frames (OTLP encoding, metric attribute hashing) add up to under 2% of busy time, below what a 3-round A/B can resolve; the rest of the telemetry and logging slices is context propagation and instrumentation hooks that run either way.

Route (uncompressed)TelemetryCPU ms/req (median)vs todayRunsreq/s
Watch-list pageTraces + logs on (today)1.64—1.62, 1.70, 1.64725
Watch-list pageTraces off1.69+3%1.66, 1.69, 1.70699
Watch-list pageTraces + logs off1.66+1%1.67, 1.65, 1.66705
Single titleTraces + logs on (today)2.31—2.20, 2.44, 2.31507
Single titleTraces off2.19−5%2.19, 2.23, 2.19526
Single titleTraces + logs off2.30−0%2.42, 2.22, 2.30510
contentDataTraces + logs on (today)0.90—0.87, 0.90, 0.961,209
contentDataTraces off0.95+6%0.88, 0.95, 0.961,142
contentDataTraces + logs off0.94+4%1.01, 0.91, 0.941,152

NestJS NodeAdapter on the final code

Same Nest 12.3.3 dependencies and same code, only the HTTP adapter differs (Express vs @nestjs/platform-node, with the compression, route-name, query-rewrite and ETag shims from the port). 3 interleaved rounds.

Route (uncompressed)Express CPU ms/reqNodeAdapterChangeVerdictreq/s Expressreq/s NodeAdapterErrors
Watch-list page1.631.51−7%beyond spread7397850
Single title2.612.48−5%beyond spread5025350
Watch-history display2.302.24−3%within spread5145280
Watch-history gates1.241.17−6%within spread8959690
contentData0.890.81−9%within spread1,1941,3220

Under overload (single thread, 64–256 connections, 2 rounds) neither adapter returns an error, and NodeAdapter keeps higher throughput and lower p99 at every step:

RouteConnectionsreq/s Expressreq/s NodeAdapterChangep99 Expressp99 NodeAdapterErrors
Watch-list page64767822+7%1751640
Watch-list page128830884+7%2882640
Watch-list page256854930+9%5234760
Single title64527558+6%1851740
Single title128538566+5%3603300
Single title256524569+9%7466960

How far one pod goes before it saturates

Each route was stepped from 4 to 128 connections (20 s per step) until throughput stopped rising (checked from 32 connections; an earlier pass stopped I/O-bound routes too early at 8 and was rerun). The probe reports per-thread event-loop utilisation (share of time the loop is busy) and event-loop delay (how late timers fire).

Route1 thread: max req/sat connscoresELU2 threads: max req/scoresGain1-thread limit2-thread limit
Watch-list page (uncompressed)749321.200.811,5191.982.03×event loop (one thread)pod CPU limit (2 cores)
Single title (uncompressed)535321.200.831,0191.991.90×event loop (one thread)pod CPU limit (2 cores)
Single title (compressed)46881.500.826991.971.50×event loop (one thread)pod CPU limit (2 cores)
Watch-history display (uncompressed)499321.210.828641.911.73×event loop (one thread)pod CPU limit (2 cores)
Watch-history gates (uncompressed)863321.150.821,3661.711.58×event loop (one thread)I/O wait (Mongo/Valkey/upstream)
contentData (uncompressed)1,396321.120.832,2391.911.60×event loop (one thread)pod CPU limit (2 cores)
Page routes (compressed)1041.470.77141.981.38×event loop (one thread)pod CPU limit (2 cores)
Entitlement (compressed)248641.150.823321.741.34×event loop (one thread)I/O wait (Mongo/Valkey/upstream)

What the sweep shows

On one thread a pod plateaus at about 1.2 of its 2 cores with the event loop ~80% busy: the single JS thread is the bottleneck, and extra connections only add latency. On two worker threads (SO_REUSEPORT, one process) the same pod reaches its 2-core limit; peak throughput rises 1.3–2.0×: most on CPU-bound uncompressed routes, least where compression already used the spare core or requests wait on Mongo and upstream services. The limit then is the container's CPU quota, not Node.

Watch-list page (uncompressed) — max 749 req/s on 1 thread, 1,519 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
146065.4211.190.81200
1867910311.190.80200
11671819.4581.190.82250
13274936.8961.200.81330
16474170.91811.250.81710
241,1372.6161.890.71, 0.7030, 300
281,3894.4261.990.80, 0.7733, 270
2161,5198.3441.980.82, 0.8134, 300
2321,51917.6761.990.81, 0.8146, 360
Single title (uncompressed) — max 535 req/s on 1 thread, 1,019 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
144887.1221.160.79180
1850814.3431.190.82220
11653527.9721.180.78280
13253556.71131.200.83450
247514.4211.710.46, 0.8025, 220
288888451.950.66, 0.7939, 400
21697714.1701.990.82, 0.8147, 470
2321,01927.9961.990.79, 0.7949, 500
26499760.41681.990.83, 0.8276, 620
Single title (compressed) — max 468 req/s on 1 thread, 699 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
144307.8511.370.78220
1846815.5561.500.82260
11645131.6901.590.82410
246245.2391.850.43, 0.6927, 280
2869910.9541.970.83, 0.4136, 320
21666818.3852.000.80, 0.7849, 470
Watch-history display (uncompressed) — max 499 req/s on 1 thread, 864 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
144607.9211.140.79180
1847915.3531.180.80230
11649531.1751.190.82250
13249962.51181.210.82350
245985.6261.340.28, 0.6718, 210
286428.3601.460.40, 0.6925, 280
21679713.6791.770.64, 0.7638, 430
23286429.61021.910.80, 0.7852, 480
26480580.11731.930.78, 0.7659, 580
Watch-history gates (uncompressed) — max 863 req/s on 1 thread, 1,366 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
147224.4151.010.73160
188377.9231.140.83170
11686017.2411.150.83200
13286335781.150.82250
248633.6131.240.61, 0.2714, 130
281,2914.9241.640.79, 0.4220, 220
2161,3516.9601.700.69, 0.6330, 300
2321,36613.3861.710.62, 0.6933, 350
contentData (uncompressed) — max 1,396 req/s on 1 thread, 2,239 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
141,260361.120.80110
181,3375.7111.100.80130
1161,36111.3221.120.82150
1321,39622.6451.120.83200
241,5922.471.400.52, 0.5212, 120
281,6983.4511.440.41, 0.6714, 120
2161,7815.1701.560.68, 0.5720, 190
2322,19610.4641.820.73, 0.7121, 190
2642,23923.9831.910.79, 0.7928, 360
Page routes (compressed) — max 10 req/s on 1 thread, 14 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
14103711,0011.470.771760
18108191,5221.550.822100
24142447331.980.76, 0.77194, 1970
28144992,1021.990.82, 0.84233, 3370
Entitlement (compressed) — max 248 req/s on 1 thread, 332 on 2
ThreadsConnectionsreq/sp50 msp99 msCPU coresEvent-loop utilisation (per thread)Event-loop delay p99 msErrors
1421418.9541.100.77160
1823334.4881.130.80200
116234651611.140.81260
1322441282831.180.82380
1642482815821.150.82690
2418812.91741.170.43, 0.4314, 140
2828022921.550.59, 0.5918, 190
21630243.51741.660.57, 0.7024, 290
23232687.32881.750.68, 0.6735, 340
2643321965271.740.74, 0.6359, 440

One thread vs two worker threads

Same build and pod limit; 32 connections; median of 2 rounds per arm. Two worker_threads each run the unmodified app and listen on :3002 with reusePort, so the kernel spreads connections. Per-thread CPU ticks were equal (1,419 vs 1,423) in a spot check.

RouteEncodingreq/s 1 threadreq/s 2 threadsChangecores 1tcores 2tCPU ms/req 1tCPU ms/req 2tp99 1tp99 2tErrors
Watch-list pagecompressed581805+39%1.362.002.342.481101100
Watch-list pageuncompressed7061,087+54%1.202.001.701.8395920
Single titleuncompressed528808+53%1.202.002.272.481111120
Single titlecompressed388504+30%1.732.004.453.961451790
Watch-history displaycompressed329471+43%1.772.005.394.241831830
Watch-history displayuncompressed469751+60%1.192.002.542.671241220
contentDatacompressed1,0461,723+65%1.132.001.081.1662610
contentDatauncompressed1,0801,815+68%1.101.941.021.0852620