Hardening a ~3-billion-events/month client-side threat-detection pipeline on AWS

The pipeline’s consumers wrote aggregates into OpenSearch with frequent in-place updates during each document’s first hour of life — and in OpenSearch, an update is a delete plus a reindex. At this volume that pattern becomes write-amplification: the cluster doing multiples of the logically necessary work, with segment churn driving both cost and pressure on query performance. Meanwhile index growth without a lifecycle policy meant yesterday’s hot data aged on expensive storage. The tier had to keep serving sub-real-time security analytics — freshness is the product — while its unit economics were rebuilt.

VeUP scoped the target pattern around a simple principle: write once, aggregate deliberately, age out automatically. Events append immutably instead of updating in place; the customer-facing hourly aggregates become materialized views computed from the append-only stream, so the cluster’s write load tracks event volume rather than update churn. Index lifecycle management tiers indices as they age — hot for the query-heavy first window, then progressively cheaper storage and eventual deletion — aligned with the Firehose → Amazon S3 raw-event archive that guarantees any index can be rebuilt by replay. The de-skewed Kinesis distribution and edge throttling upstream shape the write traffic the cluster receives, and per-tier FinOps telemetry makes the OpenSearch cost curve a measured quantity. Success criteria: hourly rollups plus sub-60–90-second freshness on the live path, at a materially lower cost per ingested event.