The lab in one picture
lab-<lane> pod→ own m8g.xlarge (prod instance type), limit 2 CPU / 2Gi like prod, Node 26, real UAT Mongo + Valkey lab-loadgen-<lane>→ separate node, so the load source never steals the app's CPU bundle hot-swap→ copy main.js (2 MB) in, restart node: ~3 s instead of a 6-minute image build
Four lanes ran in parallel (pages, single title, watch-history, entitlement), each with its own lab pod, its own load generator and its own git worktree. Proven changes were then combined into one build and remeasured on every route.
Measuring CPU: the container counter, not the profiler
Each run reads usage_usec from /sys/fs/cgroup/cpu.stat before and after: CPU ms per request = Δusage_usec ÷ successful requests. This counts every thread in the container (event loop, garbage collector, libuv's thread pool where compression runs). The CPU profiler only sees the main thread, so the compression win was only visible through this counter.
The load driver
- Closed loop: N keep-alive connections, each sending its next request as soon as the last one finishes.
- Scenarios from prod shapes: watch-list pages sized p50 3 / p90 11 / p99 54, 11.7% not-found ids, prod platform split, prod user-agent mix, real UAT content ids.
- Same sequence for both arms: requests are shuffled with a fixed seed.
- Honest counting: only 2xx plus statuses a scenario declares (404/401) count as success; anything else is an error.
- Warm-up: 10–15 s unscored so the JIT and caches are warm (a 5 s warm-up once gave a misleading cold result).
- Generator check: the driver reports its own CPU, to prove the load source is not the bottleneck.
// scen/drive.js (excerpt): closed loop over keep-alive connections
async function worker() { while (process.hrtime.bigint() < tEnd) await one(); }
// a response counts only if 2xx or explicitly expected by the scenario
if ((c >= 200 && c < 300) || (r.expect || []).includes(c)) ok++; else bad++;A/B proof format
Arms alternate A, B, A, B, A, B on the same pod (drift hits both equally). Each arm reports the median of CPU ms/request, requests/s, p50/p99 and errors, with the individual runs printed so the spread is visible. A change is accepted only if its gain is larger than that spread, with 0 errors.
Where the time goes: CPU profiles
- Start: write
--cpu-prof --cpu-prof-interval=500into the pod's flags file and restart node (V8 samples the stack every 500 µs). - Load: drive the scenario for 20–25 s.
- Stop: Nest ignores SIGINT/SIGTERM, so V8 never wrote a profile. A 1-line preload makes SIGUSR2 call
process.exit(), which flushes the.cpuprofile. - Summarise: self time by package and function, inclusive time, and a category breakdown (framework, serialisation, Mongo, logging, telemetry, GC, idle…). Native regex frames are charged to the JS caller that ran them.
Output checks for every change
Each change was checked by calling the old and new build with the same requests and comparing what the client receives (for watch-history, a script replays analytics' exact mapping of the response). Results are reported as items compared and mismatches.
- Side-by-side capture
- Pages vary between two runs of the *unchanged* code (shuffled recommendations, cache-state image variants). The pages lane ran old and new in one pod on two ports and fetched each URL from both at the same moment; a difference only counts if it repeats.
- A checker bug, caught
- One false mismatch was a Bengali character split across network chunks in the checker itself. Fixed with
res.setEncoding('utf8'). - Library equivalence
- The device-detector patch was compared on 40,186 user agents from the library's own upstream test fixtures: 0 differences.
Limits
- UAT data differs from prod (about half the series), so results are old-vs-new ratios on the same data, not absolute prod numbers.
- The profiler adds a few percent; CPU numbers always come from separate runs without it.
- Lanes share UAT's Mongo and Valkey. Valkey CPU was checked during the parallel runs (no throttling in the sampled window).
- One event loop caps a pod at about 1.35 of its 2 CPUs; see the capacity page.
Lab files
- Harness
.orca/drops/HOI-4097-l4/lab/lab.sh,mklab.py,scen/drive.js,allroutes.sh,mt-ab.sh,sweep.sh- Profiling
exit-on-usr2.js,topfn.py,buckets.py,elu-probe.js- Results
results/*.jsonl(every run),LEDGER.md(accepted changes)