Data & Analytics Competency · Streaming Log-Analytics + FinOps

BrontoBytes runs 4-5 TB/day per customer on AWS Lambda + Kinesis Firehose

Well-Architected ReviewAdvisoryHigh-risk-issue remediation roadmapResilience-gap assessment (RTO/RPO)Blast-radius & tenancy isolation reviewCost-optimization deep diveStreaming ingest & data-lake designServerless analytics on S3Workload placement & runtime selectionCommitment & RI optimizationRightsizing & instance-family modernizationUnit-cost optimizationMulti-tenant isolationFoundational Technical Review readinessMarketplace listing launchManaged billing & resellStanding cost-optimization mechanism

4–5 TB

per day, per customer, in production

30–40%

per-search cost-reduction target, latency-guarded

Managed

ingestion modernized onto Kinesis Firehose

AWS LambdaAmazon Kinesis FirehoseAmazon S3Amazon MSK

For BrontoBytes — a multi-tenant log-analytics SaaS ingesting ~4-5 TB/day per customer — VeUP delivered a Well-Architected modernization: Kinesis Firehose ingestion into an S3 log store, hundreds of GraalVM Lambdas for burst search, and a FinOps target of a 30-40% per-search Lambda cost cut.

The challenge

The customer operates a multi-tenant log-visualization-and-analytics SaaS that ingests, stores, and queries customer log data in real time at ~4-5 TB/day per customer at scale. Its ingestion ran on a Lambda log-forwarder path that coupled cost and reliability to a function fleet and lacked managed buffering; burst search over the data is spiky and embarrassingly parallel, so per-search Lambda cost was a growing margin driver; and a Well-Architected gap analysis was needed to clear a Foundational Technical Review (FTR).

The solution

A Well-Architected modernization of a TB/day log-analytics pipeline: Amazon Kinesis Data Firehose ingestion (buffered, flow-controlled) into an Amazon S3 log store (per-tenant partitioning), over which hundreds of parallel AWS Lambda functions — GraalVM-compiled Java in container images to minimize cold starts at high fan-out — run burst searches feeding real-time visualization, with Amazon MSK (managed Kafka, IAM-auth resolved with AWS Support) in the ingest path and a proposed Amazon Bedrock serving layer for AI-driven log analytics. FinOps layered a hybrid baseline (EC2-hosted Lambda + Savings Plans) plus on-demand peak design and a Graviton re-evaluation. Every big fork went the cost-defensible way: S3 over a warehouse, Lambda fan-out over an always-on cluster, Firehose over the old log-forwarder.

Architecture

The ingestion-and-search estate — from the pre-modernization Lambda log-forwarder path to the production Kinesis Firehose + Amazon MSK ingestion, the Amazon S3 log store, and the Lambda burst-search fan-out.

The production estate on AWS: Amazon MSK (IAM-hardened) and Amazon Kinesis Data Firehose (5 KB flow-controlled) landing into an Amazon S3 log store, an AWS Lambda burst-search fleet (GraalVM/Graviton, EC2-hosted baseline + on-demand peak), AWS Glue/Amazon Athena analytics, a proposed Amazon Bedrock AI-observability layer, VPC endpoints and NAT gateway, an IAM/KMS/TLS/security-groups rail, and a CloudWatch/FinOps observability rail.

The estate as it runs today — Firehose and MSK land the logs in S3, and a Lambda fleet fans out to search them on demand.

The previous estate: an AWS VPC with public-subnet load balancers, a Lambda log-forwarder ingestion path with no managed buffering, and a Kafka/MSK path with an unresolved IAM-authentication issue, plus undefined RPO/RTO and no fault isolation. — The starting point — a Lambda log-forwarder with no managed buffering, and no defined RPO/RTO.

Production outcomes

KPI	Result
Production outcomes	Ingestion moved off the Lambda log-forwarder path onto Amazon Kinesis Data Firehose — buffered, flow-controlled, and billing-increment-aware at multi-TB/day. The Well-Architected roadmap closes four high-risk items: fault isolation, undefined RPO/RTO, and two cost-optimization findings. And the FinOps target is a 30-40% cut in per-search Lambda cost with no P95-latency regression.
Engagement window	The engagement opened with an Art-of-the-Possible session in March 2026; the Well-Architected review followed in April, and the modernization work is ongoing.
Cost / TCO posture	Per-search unit economics is the platform-margin lever: a 30-40% cut in per-search Lambda cost, held to a P95-latency guardrail, via a hybrid EC2-hosted-Lambda + Savings Plans baseline, on-demand peak design, and a Graviton re-evaluation. Firehose flow-control aligns records to the 5 KB billing increment so ingestion cost stays honest at TB/day.
Lessons & continuation	At TB/day, S3 over a warehouse and Lambda fan-out over an always-on cluster are the cost-defensible choices (pay-per-search vs idle capacity); Firehose ingestion must be flow-controlled to the 5 KB billing increment or cost inflates; the per-search cost-down needs a P95-latency guardrail so margin gains can't degrade the product; the managed-Kafka/MSK IAM-auth path needed AWS Support to harden.

AWS services in production

Amazon Kinesis Data FirehoseAmazon S3AWS LambdaAmazon MSKAmazon Bedrock (proposed)AWS Cost ExplorerAWS Savings PlansAWS Well-Architected Tool

Runtime

GraalVMAWS Graviton