49one row each
Experiments run
19output-checked
Kept
25noise, slower or loses output
Rejected
2Node 26 / V8 14.6
Not possible today
Every idea that was built or probed, in the order it was tried. Kept means an interleaved A/B beat run-to-run spread (or was lower in every paired round) and the output was identical. Readings are CPU ms per request unless noted. Details and code for kept items are on the changes page; threads and Watt are on the runtime page.
| # | Area | Idea | How measured | Reading | Verdict |
|---|---|---|---|---|---|
| 1 | watch-list | One batch call per page instead of one per item | prod-shaped mix, 3 rounds | 36.5 → 3.69 ms CPU/page | Kept (#1) |
| 2 | watch-list | Card-sized movie cache (videocard:<lang>) | 3 rounds | 3.58 → 2.52 ms | Kept (#2) |
| 3 | all internal | Internal calls send Accept-Encoding: identity | 3 rounds, 4 routes | single 3.35 → 2.15 ms; display 6.74 → 2.84 ms | Kept (#3) |
| 4 | entitlement | One DeviceDetector per process, one detection per request | 3 rounds | 9.08 → 5.23 ms | Kept (#4) |
| 5 | entitlement | node-device-detector patch: regex memo + 64-brand prefilter | detect() on prod UAs; 40,186 fixtures | 7.2 → 1.9 ms per detect, 0 diffs | Kept (#5, needs patch-package) |
| 6 | entitlement | Anonymous-token key built once | 3 rounds | 5.58 → 5.01 ms | Kept (#6) |
| 7 | single title | Geo check projects one field | 3 rounds | 4.67 → 4.10 ms | Kept (#7) |
| 8 | contentData | Projected reads instead of full documents | 3 rounds | 2.05 → 0.95 ms | Kept (#8) |
| 9 | display | Seasons only when an episode lacks images (two-phase) | 3 rounds; 377,730 consumer rows | 8.26 → 4.62 ms compressed | Kept (#9) |
| 10 | pages | Read each title once per page | 3 rounds | 236 → 163 ms | Kept (#10) |
| 11 | pages | Filter walk only touches what it changes | 3 rounds | 163 → 156 ms | Kept (#11) |
| 12 | all | Drop log lines the collector discards | 3 rounds | single 3.01 → 2.43 ms | Kept (#12) |
| 13 | all validated | Remember class-validator container lookups | 3 rounds | watch-list 3.65 → 3.56 ms | Kept (#13) |
| 14 | all | Log sanitizer fast pre-test | 300,138 inputs; 3 rounds | 9.6 → 1.8 µs/message | Kept (#14) |
| 15 | all | Skip context building for disabled log levels | with #14/#16 | — | Kept (#15) |
| 16 | series | DRM filter skips episodes without renditions | with #14/#15 | — | Kept (#16) |
| 17 | pages | Series tray projection (seriestray:<lang>) | 3 rounds; 23 pages byte-identical ×2 | 153.2 → 98.7 ms | Kept (#17) |
| 18 | pages | Movie tray projection (videotray:<lang>) | 3 rounds; 23 pages byte-identical ×2 | 80.95 → 68.38 ms | Kept (#18) |
| 19 | contentData, gates | Reuse Mongo Db handle per client | 6 paired rounds; 450 requests identical | contentData −6% (6/6), gates −2.9% (6/6) | Kept (#19) |
| 20 | all | Express ETag off | 3 rounds | 4.05 vs 4.03 ms | Rejected: noise, loses header |
| 21 | all | OpenTelemetry traces + logs export off | 3 rounds × 3 routes | all within spread | Rejected: no cost to cut |
| 22 | all | Metrics SDK off / single HTTP semconv | 3 rounds | within spread | Rejected |
| 23 | all | Trace sampling 10% | 3 rounds | no gain | Rejected |
| 24 | all | ioredis spans off | 3 rounds | −3% | Rejected: loses Valkey visibility |
| 25 | gates | Mongo batchSize = number of ids | 5 rounds | runs overlap | Rejected |
| 26 | Mongo | OTel statement serializer truncation | profile | < 2% of busy | Rejected |
| 27 | pages | Trim series episode fields in trays | field audit | renditions/captions needed on 515 episodes | Rejected: changes output |
| 28 | entitlement | Start subscription call with the content read | code read | needs monetisation fields first | Rejected |
| 29 | runtime | Node 22 vs 26 | 3 rounds | 26 is 1.25–1.5× faster | Prod already on 26 |
| 30 | all | LOG_LEVEL=warn | 3 rounds | within spread | Superseded by #12 |
| 31 | pages | simdjson instead of JSON.parse | 37 real cached values | 0.64 vs 2.45 ms (3.8× slower) | Rejected |
| 32 | display/gates | BSON enableUtf8Validation:false | microbench + 3 rounds | decode −17%; route −3.7/−2.5/+1.2% (spread) | Rejected: noise |
| 33 | pages | Drop zstd buffer copies | microbench + 3 rounds | decode −8%; pages +1.0% (spread) | Rejected: noise |
| 34 | pages | One zstd call per batch (concat frames, split by size) | microbench + 6 rounds | decode 2× faster; pages −0.7%, watch-list −3.2% (spread) | Rejected: noise |
| 35 | pages | Reusable zstd decoder (reset() / decompressZstdSync) | Node 26.13 probe | 'zlib binding closed'; API not exported | Impossible today |
| 36 | pages | Pre-serialised tray items via JSON.rawJSON | Node 26 / V8 14.6 probe | throws on objects (primitives only) | Impossible |
| 37 | pages | fast-json-stringify schema serializer | 1 MB real page | 2.67 vs 1.83 ms (46% slower), output differs | Rejected |
| 38 | pages | Cheaper ETag hash | 1 MB body | sha1 0.62, sha256 0.59, md5 1.69 ms | No cheaper built-in |
| 39 | all | TextDecoder instead of Buffer.toString | microbench | −3% of decode (~0.1% of route) | Rejected: too small |
| 40 | display | BSON promoteValues:false / raw | 1,539 real docs | 10.9 µs/doc default; promoteValues slower | Rejected |
| 41 | pages | Semi-space 16 / 64 / min16-max64 | 3 rounds, flags verified | +0.7% / ±0% / +0.5% | Rejected: noise |
| 42 | pages | --no-concurrent-marking | 3 rounds, flags verified | −3.7% CPU (spread), req/s 23 → 21 | Rejected |
| 43 | pages | --minor-ms young-gen collector | 3 rounds, flags verified | +3.9% (real) | Rejected: slower |
| 44 | pages, single, watch-list | --single-threaded-gc | 3 rounds each | CPU −5.5/−7/−11%, but req/s −13/−16/−3% and p99 +50% | Rejected on 1 thread; retest with threads |
| 45 | entitlement | Detector prefilter chunk 16 instead of 64 | microbench + 4-round route A/B | microbench 1.72 → 1.17 ms; route +4.5% (real, slower) | Rejected |
| 46 | contentData, gates | Router: skip decodeURIComponent without % | 4 paired rounds | gates −1.5% (4/4), contentData −3.0% (3/4) | Rejected: inside spread |
| 47 | runtime | NestJS NodeAdapter (Nest 12.3.3) | 3 rounds + overload 64–256 | −5 to −7% CPU; +5–9% req/s under overload | Real: owner decision |
| 48 | runtime | 2 worker threads (our launcher) | 3 rounds × 6 routes | +23% to +65% req/s on same 2 CPUs | Real: owner decision |
| 49 | runtime | Platformatic Watt 3.77, 2 workers | 3 rounds × 6 routes; stall test | +11% to +52% req/s; 5–20% more CPU/req than our launcher | Works; see runtime page |
What to try next
Everything left is either a runtime change or needs an owner decision. Code-level slices were all measured (rows 20–46).
| Idea | Expected gain (source) | What it needs |
|---|---|---|
| Two worker threads per pod (our launcher or Watt) | +23% to +65% req/s per pod at the same 2-CPU limit (measured) | Launcher in prod image, per-thread Mongo/Valkey pools, UAT soak; decide Watt vs ours (runtime page) |
| Merge NodeAdapter | −5 to −7% CPU/req; better overload behaviour (measured) | Branch experiment/HOI-4097-node-adapter-v2, Nest 12.0.3 → 12.3.3 |
| Ship the device-detector patch | detect 7.2 → 1.9 ms on prod UAs (measured) | Add patch-package + postinstall to Hoichoi.Backend |
| Per-user-agent detection cache | Removes most of entitlement's remaining 31% (prod sees ~15 distinct UAs) | Needs an exception to the no-in-pod-cache rule |
Retest --single-threaded-gc with 2 threads | −5 to −11% CPU/req measured on 1 thread | Only pays when the event loop is no longer the bottleneck |
Per-worker ports behind a balancer (Watt perWorkerIncrement) | Booking.com: fixed up to 40% thread imbalance | We measured 1.43× on pages under Watt's shared port; needs an L7 balancer in the pod or mesh |