Content CPU · ledger

Experiments ledger

All 49 experiments in one table: what was tried, how it was measured, the reading, and why it was kept or dropped. Then what to try next.

Measured by Claude (agent) in the HOI-4097 UAT lab
49one row each
Experiments run
19output-checked
Kept
25noise, slower or loses output
Rejected
2Node 26 / V8 14.6
Not possible today

Every idea that was built or probed, in the order it was tried. Kept means an interleaved A/B beat run-to-run spread (or was lower in every paired round) and the output was identical. Readings are CPU ms per request unless noted. Details and code for kept items are on the changes page; threads and Watt are on the runtime page.

#AreaIdeaHow measuredReadingVerdict
1watch-listOne batch call per page instead of one per itemprod-shaped mix, 3 rounds36.5 → 3.69 ms CPU/pageKept (#1)
2watch-listCard-sized movie cache (videocard:<lang>)3 rounds3.58 → 2.52 msKept (#2)
3all internalInternal calls send Accept-Encoding: identity3 rounds, 4 routessingle 3.35 → 2.15 ms; display 6.74 → 2.84 msKept (#3)
4entitlementOne DeviceDetector per process, one detection per request3 rounds9.08 → 5.23 msKept (#4)
5entitlementnode-device-detector patch: regex memo + 64-brand prefilterdetect() on prod UAs; 40,186 fixtures7.2 → 1.9 ms per detect, 0 diffsKept (#5, needs patch-package)
6entitlementAnonymous-token key built once3 rounds5.58 → 5.01 msKept (#6)
7single titleGeo check projects one field3 rounds4.67 → 4.10 msKept (#7)
8contentDataProjected reads instead of full documents3 rounds2.05 → 0.95 msKept (#8)
9displaySeasons only when an episode lacks images (two-phase)3 rounds; 377,730 consumer rows8.26 → 4.62 ms compressedKept (#9)
10pagesRead each title once per page3 rounds236 → 163 msKept (#10)
11pagesFilter walk only touches what it changes3 rounds163 → 156 msKept (#11)
12allDrop log lines the collector discards3 roundssingle 3.01 → 2.43 msKept (#12)
13all validatedRemember class-validator container lookups3 roundswatch-list 3.65 → 3.56 msKept (#13)
14allLog sanitizer fast pre-test300,138 inputs; 3 rounds9.6 → 1.8 µs/messageKept (#14)
15allSkip context building for disabled log levelswith #14/#16—Kept (#15)
16seriesDRM filter skips episodes without renditionswith #14/#15—Kept (#16)
17pagesSeries tray projection (seriestray:<lang>)3 rounds; 23 pages byte-identical ×2153.2 → 98.7 msKept (#17)
18pagesMovie tray projection (videotray:<lang>)3 rounds; 23 pages byte-identical ×280.95 → 68.38 msKept (#18)
19contentData, gatesReuse Mongo Db handle per client6 paired rounds; 450 requests identicalcontentData −6% (6/6), gates −2.9% (6/6)Kept (#19)
20allExpress ETag off3 rounds4.05 vs 4.03 msRejected: noise, loses header
21allOpenTelemetry traces + logs export off3 rounds × 3 routesall within spreadRejected: no cost to cut
22allMetrics SDK off / single HTTP semconv3 roundswithin spreadRejected
23allTrace sampling 10%3 roundsno gainRejected
24allioredis spans off3 rounds−3%Rejected: loses Valkey visibility
25gatesMongo batchSize = number of ids5 roundsruns overlapRejected
26MongoOTel statement serializer truncationprofile< 2% of busyRejected
27pagesTrim series episode fields in traysfield auditrenditions/captions needed on 515 episodesRejected: changes output
28entitlementStart subscription call with the content readcode readneeds monetisation fields firstRejected
29runtimeNode 22 vs 263 rounds26 is 1.25–1.5× fasterProd already on 26
30allLOG_LEVEL=warn3 roundswithin spreadSuperseded by #12
31pagessimdjson instead of JSON.parse37 real cached values0.64 vs 2.45 ms (3.8× slower)Rejected
32display/gatesBSON enableUtf8Validation:falsemicrobench + 3 roundsdecode −17%; route −3.7/−2.5/+1.2% (spread)Rejected: noise
33pagesDrop zstd buffer copiesmicrobench + 3 roundsdecode −8%; pages +1.0% (spread)Rejected: noise
34pagesOne zstd call per batch (concat frames, split by size)microbench + 6 roundsdecode 2× faster; pages −0.7%, watch-list −3.2% (spread)Rejected: noise
35pagesReusable zstd decoder (reset() / decompressZstdSync)Node 26.13 probe'zlib binding closed'; API not exportedImpossible today
36pagesPre-serialised tray items via JSON.rawJSONNode 26 / V8 14.6 probethrows on objects (primitives only)Impossible
37pagesfast-json-stringify schema serializer1 MB real page2.67 vs 1.83 ms (46% slower), output differsRejected
38pagesCheaper ETag hash1 MB bodysha1 0.62, sha256 0.59, md5 1.69 msNo cheaper built-in
39allTextDecoder instead of Buffer.toStringmicrobench−3% of decode (~0.1% of route)Rejected: too small
40displayBSON promoteValues:false / raw1,539 real docs10.9 µs/doc default; promoteValues slowerRejected
41pagesSemi-space 16 / 64 / min16-max643 rounds, flags verified+0.7% / ±0% / +0.5%Rejected: noise
42pages--no-concurrent-marking3 rounds, flags verified−3.7% CPU (spread), req/s 23 → 21Rejected
43pages--minor-ms young-gen collector3 rounds, flags verified+3.9% (real)Rejected: slower
44pages, single, watch-list--single-threaded-gc3 rounds eachCPU −5.5/−7/−11%, but req/s −13/−16/−3% and p99 +50%Rejected on 1 thread; retest with threads
45entitlementDetector prefilter chunk 16 instead of 64microbench + 4-round route A/Bmicrobench 1.72 → 1.17 ms; route +4.5% (real, slower)Rejected
46contentData, gatesRouter: skip decodeURIComponent without %4 paired roundsgates −1.5% (4/4), contentData −3.0% (3/4)Rejected: inside spread
47runtimeNestJS NodeAdapter (Nest 12.3.3)3 rounds + overload 64–256−5 to −7% CPU; +5–9% req/s under overloadReal: owner decision
48runtime2 worker threads (our launcher)3 rounds × 6 routes+23% to +65% req/s on same 2 CPUsReal: owner decision
49runtimePlatformatic Watt 3.77, 2 workers3 rounds × 6 routes; stall test+11% to +52% req/s; 5–20% more CPU/req than our launcherWorks; see runtime page

What to try next

Everything left is either a runtime change or needs an owner decision. Code-level slices were all measured (rows 20–46).

IdeaExpected gain (source)What it needs
Two worker threads per pod (our launcher or Watt)+23% to +65% req/s per pod at the same 2-CPU limit (measured)Launcher in prod image, per-thread Mongo/Valkey pools, UAT soak; decide Watt vs ours (runtime page)
Merge NodeAdapter−5 to −7% CPU/req; better overload behaviour (measured)Branch experiment/HOI-4097-node-adapter-v2, Nest 12.0.3 → 12.3.3
Ship the device-detector patchdetect 7.2 → 1.9 ms on prod UAs (measured)Add patch-package + postinstall to Hoichoi.Backend
Per-user-agent detection cacheRemoves most of entitlement's remaining 31% (prod sees ~15 distinct UAs)Needs an exception to the no-in-pod-cache rule
Retest --single-threaded-gc with 2 threads−5 to −11% CPU/req measured on 1 threadOnly pays when the event loop is no longer the bottleneck
Per-worker ports behind a balancer (Watt perWorkerIncrement)Booking.com: fixed up to 40% thread imbalanceWe measured 1.43× on pages under Watt's shared port; needs an L7 balancer in the pod or mesh