BrontoBytes runs 4-5 TB/day per customer on AWS Lambda + Kinesis Firehose

Kafka semantics matter to a log platform — ordered, replayable, producer-friendly ingestion is what customers’ shippers and agents expect to write to. But Kafka operated casually at terabyte scale becomes its own incident source: authentication between producers, brokers, and consumers must be right (the platform’s MSK IAM authentication had open questions that needed resolution with AWS Support), and without a deliberate flow-control posture, one tenant’s log storm becomes every tenant’s latency. The review had to leave the Kafka lane both trustworthy and boring — secure by configuration, predictable under burst, and cheap enough to carry multi-TB/day indefinitely.

VeUP’s modernization treated the managed-Kafka path as production infrastructure to harden, not replumb. The MSK IAM-authentication posture was worked through to resolution with AWS Support, landing the cluster on IAM-based auth — access to topics governed by the same identity plane as the rest of the AWS estate, with no parallel credential system to operate. Flow control was engineered end to end: producer and broker behavior under tenant bursts examined against the platform’s real traffic shapes, so back-pressure degrades a noisy tenant’s lane instead of the shared plane, with the parallel Firehose path’s buffering rigor applied consistently across both ingestion lanes. Downstream, the S3 log store and the GraalVM-Lambda burst-search tier consume from the hardened plane unchanged, and the engagement’s FinOps program — targeting 30–40% off per-search compute cost without sacrificing P95 latency — prices the whole flow, Kafka included, as one measured system.