{"window":"2026-08-18 → 2026-09-18 (30–31 d depending on source)","spend":[{"account":"026090548158","label":"PROD (EKS hoichoi-prod, ap-south-1)","usd_30d":4208.85,"in_scope":true},{"account":"211125449373","label":"UAT (EKS hoichoi-uat, ap-south-1)","usd_30d":1415.14,"in_scope":true},{"account":"651706761110","label":"CDN (hoichoicdn.com video CloudFront, 18 dists) — 99.8 % CloudFront","usd_30d":235464.85,"in_scope":true},{"account":"629400601464","label":"Sooper / API-CDN (11 dists behind Cloudflare)","usd_30d":1211.0,"in_scope":true},{"account":"854551250067","label":"Logline — separate ECS/Aurora product, leave on AWS (no cross-account role into prod/UAT)","usd_30d":1261.11,"in_scope":false},{"account":"302295340241","label":"dev — retire; 10 buckets with Principal * policies, 2 public-write","usd_30d":20.95,"in_scope":false},{"account":"unidentified SVF/FAST vendor","label":"External vendor account holding 4 foreign buckets; retire the 7 dependent dists","usd_30d":null,"in_scope":false},{"account":"062484260092","label":"CleverTap export account (inferred from policy Sid) — only third-party AWS identity writing into our estate (s3:PutObject* on UAT hoichoi-clevertap-export-eng, +9.3 k objects / +5 GB per day)","usd_30d":null,"in_scope":false}],"stack":[{"layer":"Edge / DNS / WAF / Workers / R2 / Tunnels / WARP","where":"Cloudflare account `hoichoi` (10 zones, 4 Pro)","detail":"Front door for all but raw video; 129 Workers scripts, 3 R2 buckets, 5 tunnels, 7 Access apps. Image Resizing on for hoichoicdn.com (3,081 req/31 d, a CI canary). Thumbor is env-only portable."},{"layer":"Video CDN (3.1 PB/month)","where":"CloudFront in AWS acct 651706761110 (18 dists)","detail":"hoichoicdn.com apex = 99.15 % of bytes; DNS-only CNAME, Cloudflare blind to it."},{"layer":"API CDN (two-layer cache)","where":"CloudFront in acct 629400601464 (11 dists) behind Cloudflare","detail":"prod-contents-api.hoichoi.dev etc. → eks-prod-alb.hoichoi.dev; serves 10.4 TB/mo to viewers but pulls only ≤ 0.75 TB/mo from the ALB (MinTTL 86400 cache)."},{"layer":"Compute","where":"EKS hoichoi-prod (026090548158) + hoichoi-uat (211125449373), both k8s 1.35, ap-south-1, 100 % arm64 Graviton","detail":"eksctl-created (prod), Karpenter 1.14 + 3 static MNGs each; no PriorityClass / ResourceQuota / LimitRange anywhere (8 non-kube-system pods run system-cluster-critical)."},{"layer":"Ingress","where":"1 ALB per cluster (UAT +1 chatwoot ALB +1 data NLB) via aws-load-balancer-controller","detail":"33 listener rules prod; 96 % of traffic is host-less path routing; prod ALB carries plain HTTP/1.1 to pods only (no WebSocket, gRPC, SSE); readiness probe = TG health check."},{"layer":"Data plane","where":"In-cluster on EBS gp3","detail":"MongoDB PSMDB x3 (prod wire TLS off, no NetworkPolicy), ClickHouse x2 (1 TiB ea), Redpanda x3, Valkey/Typesense/Keeper x3, Temporal, Debezium x9, PostHog 46 pods. Only RDS PG 15 + S3 are AWS-managed."},{"layer":"Object storage","where":"S3","detail":"prod 45 buckets / ~25.4 TB / 95.8 M objects; media origin 13.8 TB (+1.03 TB, +1.0 M objects/mo gross); CH cold 1.16 TB (99.7 % noncurrent); backups 5.8 TB; 0 event notifications first-party."},{"layer":"Secrets","where":"SSM Parameter Store (474 prod / 414 UAT) → ESO ClusterSecretStore aws-ssm","detail":"40 / 30 ExternalSecrets; prod ESO's purpose-built role has never been assumed (node-role by accident); hoichoi-llm-gateway is the one prod namespace with no ExternalSecret (hand-created 6-key secret)."},{"layer":"Registry","where":"ECR","detail":"prod 66 repos (21 first-party + 40 PTC mirrors + 5 other), UAT 67; pull-through caches for docker-hub / ghcr / quay / k8s."},{"layer":"Observability","where":"SigNoz self-hosted (two-tier OTel collectors), Grafana, Kite; CloudWatch only for AWS-side metrics","detail":"71 GiB/day ingest. External OTLP ingest = tunnel otlp.prod.hoichoi.dev: 51.5 TB/mo inbound, 976 M req/mo, unauthenticated, ≥ 81 % from installed KMP apps + browsers."},{"layer":"Serverless","where":"1 live Lambda behind API GW oh4nleqg1k, plus EventBridge","detail":"Payment webhook relay; the raw execute-api URL is hard-coded in every Android build ≥ 3.1.23, and the Lambda rejects 18/18 SSLCommerz IPNs with 400. MediaConvert retired Apr 2026."},{"layer":"Media pipeline","where":"hc-transcode k8s Jobs","detail":"1 pod = 1 × 32-vCPU node, 90 Gi ephemeral request on a 120 Gi root, launched by kubectl from an operator laptop — no Temporal, no CMS, no controller."},{"layer":"IaC","where":"helmfile (27 releases) + kustomize overlays + Karpenter CRs in git","detail":"Only accurate IaC; nothing below k8s is under live Terraform. ClickHouse SQL RBAC (12 users / 1 role / 323 grants), 2 Kafka-table overrides, Redpanda topic configs and cms.v_user_360 are live-only."},{"layer":"CI/CD","where":"GitHub Actions on ubuntu-24.04-arm","detail":"Static IAM keys (0 OIDC), push ECR, kubectl apply filtered kustomize. The in-cluster ARC Android-KVM lane is a manual Maestro suite, dormant since 2026-09-01 (2/45 green, ~$11/mo), not a PR gate."}],"metrics":[{"group":"compute","metric":"Nodes (instantaneous)","prod":"6 static + 8–11 Karpenter (typ. 15–17), all arm64","uat":"18 (16 OD + 2 spot), all arm64","cdn_acct":null,"note":"Never give prod nodes as one integer."},{"group":"compute","metric":"vCPU / RAM present","prod":"snapshot 64 vCPU / ~362 GiB; 30-d mean 16.4 nodes ≈ 73 vCPU / ~410 GiB steady pools","uat":"allocatable 37.4 cores / 128.6 GiB","cdn_acct":null,"note":null},{"group":"compute","metric":"30-d usage p95 (steady pools)","prod":"25.2 cores / 185 GiB (peak30m 52 / 227)","uat":"≈ 6.6 cores busy of 42 vCPU; memory ~75 % committed","cdn_acct":null,"note":null},{"group":"compute","metric":"Spot share","prod":"29 % of EC2 $ (10 % of instance-hours); apps pool 100 % spot, 753 node replacements / 30 d","uat":"2 spot nodes only","cdn_acct":null,"note":null},{"group":"compute","metric":"IAM roles (RoleLastUsed, measured)","prod":"108: 9 app/data roles to rebuild (all 9 used in-window), 11 EKS plumbing, 10 other AWS-native, 11 external/Org trusts, 32 ECS/MSP dead, 35 service-linked","uat":"95: 4 app/data live (mongodb-backup-role never used), 6 EKS plumbing, 25 other, 11 external, 26 dead, 23 SL","cdn_acct":"NOT MEASURED (iam:ListRoles denied)","note":"Sooper/API-CDN acct 629400601464: NOT MEASURED. Identity metric, filed under compute for grouping."},{"group":"storage","metric":"PVCs / EBS","prod":"44 PVCs / 5,012 GiB gp3; EBS total 66–69 vols / 6,185–6,505 GiB (superseding 64 vols / 6,065 GiB; two captures 2 h apart on 2026-09-18 differed by +3 vols / +320 GiB); 0 true orphan volumes; only stateful non-PVC disks are 5 stopped legacy EC2 roots (413 GiB, all RETIRE except video-processing, restarted 2026-09-18). Node roots: Karpenter apps/analytics-db/clevertap-sandbox 60 Gi (git says 30 Gi — drift), batch-jobs 200 Gi, media-jobs 120 Gi, ci-android-kvm 200 Gi, cloudflared-egress 20 Gi, managed NGs 30 Gi","uat":"41 Bound + 1 Pending / 1,110 GiB (EBS 77 vols / 2,390 GiB, 24 orphan = 552 GiB)","cdn_acct":null,"note":"The claimed 'available 400 GB orphan' is the Bound clevertap-sandbox CHI PVC, in-use again at 15:21 IST."},{"group":"storage","metric":"S3","prod":"45 buckets, 23,771 GiB (= 25,406.9 GB) / 95.8 M objects; 11 versioned, 11 with lifecycle (23 rules, prefix-filtered only), 7 CORS, 3 public-policy (1 intended), 0 replication / object-lock / KMS","uat":"32 buckets (superseding 32–33), 4.09 TB (3,727 GiB Std + 135 IA); 5 versioned, 5 lifecycle, 6 CORS, 1 public + static-website","cdn_acct":"7 buckets, ≈ 18.7 TB CloudFront logs (CE-inferred)","note":"Sooper/API-CDN acct 629400601464: 1 bucket, 134.4 GB."},{"group":"storage","metric":"Cluster-attached S3 cost (ch-cold, redpanda-tiered, 2 backup buckets)","prod":"≈ $350 window / ≈ $307 forward (superseding the ≈ $200 inference) = storage $247 (10.1 TB window mean, 8.17 TB steady) + requests $103 measured (38.5 M GET + 17.3 M PUT) + ≤ $27 inferred backup ops","uat":null,"cdn_acct":null,"note":null},{"group":"edge","metric":"CloudFront origin fetch 30-d","prod":"241,371 GiB origin→CloudFront (superseding 238,118 GiB) = S3 240,618 GiB (video) + ALB→CloudFront 749 GiB (content-API + Thumbor); $0 only while CloudFront is the puller (≈ $20 K/mo at DTO list price if a non-AWS CDN pulls the same way without a shield)","uat":"0.75 GB","cdn_acct":"2,928,681 GiB = 2.79 PiB = 3.145 PB to viewers (4.93 B req)","note":"Sooper/API-CDN acct 629400601464: 9,692 GB (≈ 111 M req), $1,176 (DTO $1,045 + requests $131)."},{"group":"edge","metric":"Cloudflare edge (all zones, 31 d)","prod":"≈ 2.80 B requests / 22.9 TB: hoichoi.dev 2.39 B / 13.1 TB (⚠ CONFLICT: [06 §4] 12.2 TB vs [03 R2-01.2] 13.15 TB; canonical 13.1 TB) at 1.2 % req cache-hit; hoichoicdn.com 268 M / 9.47 TB in 18 d (97.2 % hit); hoichoi.tv 140 M / 1.16 TB. Uncached from AWS origins ≈ 10.35 TB / 31 d (79 % = content-API JSON)","uat":null,"cdn_acct":null,"note":"Table gives this as a single all-zones figure, not per account."},{"group":"edge","metric":"Cloudflare Workers","prod":"55.55 M invocations (hoichoi-web 54.7 M)","uat":null,"cdn_acct":null,"note":"Table gives this as a single all-zones figure, not per account."},{"group":"edge","metric":"Tunnel bytes (31 d)","prod":"cloudflared-otlp 51.5 TB in / 976 M req (52.7 KB/req; Aug ≈ 2.2 TB/day, Sep ≈ 1.2 TB/day); cloudflared 1.6 TB (≈ 97 % PostHog ingest); cloudflared-private-net 35 GB","uat":null,"cdn_acct":null,"note":"Table gives this as a single figure, not per account."},{"group":"network","metric":"Internet egress (non-CDN), measured split","prod":"2,517 GiB · $229.72 (superseding 2,487 GB · $227.14) = EC2 1,606 GB (tunnel responses ≈ 1,331 + NAT 275) + ELB 852 GB + ECR/S3 59 GB; Sep run-rate ≈ 1.8–2.0 TB/mo (tunnel half halved from 09-02)","uat":"74 GB · $7.12","cdn_acct":null,"note":null},{"group":"network","metric":"Internet inbound (free)","prod":"56.0 TB: EC2 52.7 TB (OTLP tunnel ingest), ELB 2.25 TB, S3 1.08 TB","uat":null,"cdn_acct":null,"note":"Verify the target does not meter ingress."},{"group":"network","metric":"Cross-AZ (Regional-Bytes)","prod":"28,771 GB · $287.70","uat":"6,626 GB · $66.26","cdn_acct":null,"note":null},{"group":"network","metric":"NAT","prod":"844 GB processed · 3 GW · $170.56 (275 GB out / 640 GB in; tunnel nodes bypass NAT via public IPs)","uat":"116.7 GB · 1 GW · $47.64","cdn_acct":null,"note":null},{"group":"network","metric":"ALB","prod":"414.9 M req / 3.68 TB; peak-hour 1.84 M req / 18.6 GB = 0.041 Gbps, peak 5-min 0.060 Gbps, 81 k new TLS conn/h","uat":"31.2 M req / 58.9 GB","cdn_acct":null,"note":"Sizing driver is RPS + conn/s, not bytes."},{"group":"network","metric":"Cluster east-west (pod traffic)","prod":"≈ 470 TB / 31 d, dominated by monitoring↔egress telemetry (110 TB), data replication (88 TB), apps (65 TB); media pool only 2.8 TB","uat":"NOT MEASURED","cdn_acct":null,"note":null},{"group":"data","metric":"Redpanda Kafka bytes","prod":"159 GB/day produced, 285 GB/day consumed (98 % = 3 PostHog Cloud Topics; clickhouse_events_json re-read 4×); wire ≈ 209 GB/day in / 420 out; S3 side ≈ 587 GB/day PUT / 220 GET","uat":null,"cdn_acct":null,"note":"broker-0 (1c) is the smallest broker, so the old ×3 inference was 4.2× low on produce."},{"group":"data","metric":"RDS","prod":"1 × db.t4g.large PG 15.17, 400 GB gp3 (12 k IOPS), ~341 GB used, single-AZ, 1-day backups, $196.97","uat":"1 × db.m6g.large, 200 GB gp2, 52 GB used, publicly accessible, $194.13","cdn_acct":null,"note":null}],"shaping_facts":[{"n":1,"fact":"97 % of AWS spend is video egress: 3.1 PB/month, 80.5 % India, Kolkata PoP alone 32.8 % of bytes, mean 13.5 Gbps / median peak-hour 21 Gbps / max 36 Gbps. Everything else combined is ≈ $7 K/month."},{"n":2,"fact":"The platform is already 'self-hosted on k8s': only RDS + S3 + edge are AWS-managed, so helmfile/kustomize port ~unchanged. But L4 is declarative only for Kubernetes objects — ClickHouse SQL RBAC (12 users, 1 role, 323 grants), 2 Kafka-table setting overrides, Redpanda topic configs and 3 view bodies exist only on the live cluster and must be exported before any rebuild."},{"n":3,"fact":"Nothing below Kubernetes is under live IaC; the tofu tree was never merged and its S3 state contains 950+ plaintext secret values. Target IaC is authored fresh from this inventory."},{"n":4,"fact":"The whole fleet is arm64, but only 16 prod / 12 UAT first-party images are arm64-only; every third-party image is multi-arch. An amd64 target = 10 CI lines + one rebuild cycle, not a port."},{"n":5,"fact":"Karpenter has no provider for Akamai/Linode (knowledge-cutoff claim, NOT re-verified). Hard parts: the spot-first apps pool and scale-to-zero media-jobs (46 x 32-vCPU cap, 484-core peak, >=120 GiB boot disk with >=~106 GiB usable ephemeral/node, >=120 s reclaim notice as SIGTERM). The amd64 nested-KVM ci-android pool is not a blocker: one manual consumer, dormant since 2026-09-01, ~$11/mo."},{"n":6,"fact":"Workload identity = 14 Pod Identity + 4 IRSA (prod) → exactly 9 app/data roles measured in use (all 9 assumed in-window) + 4 in UAT → 13 static key pairs scoped to 13 buckets + 2 Athena rewrites on a non-AWS target. Prod ESO actually authenticates via the node role by accident (its role was never assumed); ~70 roles per account are deletable today."},{"n":7,"fact":"chi-analytics ClickHouse is the storage + network red flag: p95 3,000 IOPS (at cap 8 % of minutes), peak 566 MiB/s, 3.0 Gbps NIC bursts to S3 (80 % of its 3.75 Gbps baseline) on cold-tier scan; its 968 GiB cold tier has no backup and a single-PVC metadata SPOF. No prod node exceeded baseline NIC in 31 d, but analytics-db 3.0, data 1.04 and egress 0.62 Gbps each way would saturate a 1 Gbps NIC."},{"n":8,"fact":"No restore drill has ever been run for Mongo (PBM) or ClickHouse against prod data; the migration would be the first."},{"n":9,"fact":"Cross-AZ engineering (1c pinning, two-tier collectors, Cloud Topics, rack awareness) exists only to dodge $288/month; single-DC makes it dead weight but loses AZ fault domains. The 15 zone pins (ap-south-1c) and 3 zone podAntiAffinity CRs (PSMDB, Keeper, CHI) block scheduling on a single-zone or differently-labelled target; the 26 zone DoNotSchedule spreads (minDomains unset) silently no-op."},{"n":10,"fact":"Partner-facing static IPs: 3 NAT EIPs (probably) allow-listed at 8 partners. The PSP webhook is a raw execute-api hostname with no DNS lever: 7 PSP consoles + 1 Mongo field (pg_config.sslcommerzconfig.ssl_ipn_url) + 1 Android release, not 8 consoles. That release carries 100 % of the 7-d active base (508 k users), ~85 % in 30 d, ~10 % tail at 90 d. Already dead (Lambda 400s the IPNs)."},{"n":11,"fact":"Origin egress without the CloudFront hop is bounded 2.5-3.3 TB/mo (with a Cloudflare cache rule on /contents/api/v1/* reproducing CloudFront's 86,400 s TTL) to 10-10.5 TB/mo (without): one cache rule is worth 7-8 TB/mo. In a dual-CDN window, S3 to a non-AWS CDN is ~$20 K/mo at DTO list without a shield (~$9.6 K with); the 14.8 TB library copy ~$1.5 K one-off unless the DTO waiver lands first."},{"n":12,"fact":"The OTLP tunnel is the largest byte flow after video: 51.5 TB/mo inbound from installed KMP apps and browsers (hostname compiled into the binaries, no auth, no WAF), free on AWS. The target must not meter ingress and must answer otlp.prod.hoichoi.dev from minute one (zero client change if the tunnel model is kept). Size for August (>=3.3 TB/day, >=50 M req/day), not the Sep mean."},{"n":13,"fact":"Across 77 prod+UAT buckets the only live object-store features are prefix-filtered lifecycle (23 rules prod / 10 UAT: expiry + AbortIncompleteMultipartUpload, 0 tag- or size-based), CORS on 7+6 buckets, and versioning only for tofu state; zero replication, object-lock, KMS, metrics or events. Two non-universal must-haves: object tagging (R2 lacks it) and thumbor's HEAD-before-GET doubling ops."},{"n":14,"fact":"Hoichoi holds no Akamai account, CP code, NetStorage group or property today: the 'NetStorage touchpoint' was a one-off 2024 inbound ViewLift-to-S3 migration identity (key unused since 2024-10-20), Backend uploadToNetStorage is dead code with a literal 'cp-code', and the akamaized.net hosts in partner-feed code are partner-owned and NXDOMAIN. Akamai starts from zero, with onboarding lead time."}]}