Hardening a ~3-billion-events/month client-side threat-detection pipeline on AWS

Kinesis distributes work by partition key — and the platform’s key correlated with customer project, so a single hot project (spikes of ~0.5 billion events/month) could drive one shard to its limits while its neighbors idled. Hot shards meant throttling at peak precisely when a security pipeline most needs to be ingesting, and the pipeline’s entitlement rejections happened after ingestion — the stream doing paid work on traffic the platform would discard. For a detection product whose customers rely on freshness, the streaming tier had to hold near-real-time delivery under bursty, skewed, adversarial traffic shapes.

VeUP ran discovery on the live streams and produced a costed target pattern. The core fix is key de-skew: a partition-key scheme that spreads any single project’s burst across shards, so stream capacity is a pool rather than a lottery, paired with edge throttling at the API tier so out-of-entitlement traffic never reaches the stream at all. The two-stream topology (raw ingest, enriched) is retained — it cleanly separates the ingest contract from consumer-facing enrichment — with the ~15–20 Lambda consumers scaled against de-skewed shards, and Amazon Kinesis Data Firehose continuing to archive every raw event to S3 as the replay path that makes downstream mistakes recoverable. Success criteria were set as measurable stream health: near-zero throttled or dropped events at peak, sub-60–90-second end-to-end freshness, and per-ARN cost telemetry on the streaming tier so shard capacity tracks value, not skew.