Sofvie launches a production AI data agent 28 days early on Amazon Bedrock
Amazon Bedrock
AWS Fargate
Amazon CloudWatchUnder the ML lens, Sofvie’s natural-language analytics feature is a supervised-output NLP problem — intent understanding plus structured-output generation — and VeUP engineered it with full ML-lifecycle discipline: a customer-sourced golden test set, a deterministic AST-based correctness oracle, an adversarial robustness corpus, and explicit A/B model selection across Amazon Nova Lite 2, Amazon Nova Pro, and Anthropic Claude on Amazon Bedrock.
The challenge
Turning natural language into correct structured queries over a production schema is a classical applied-NLP problem — and the failure mode that kills these systems in production is unmeasured quality: no labeled corpus, no correctness gate, no way to tell whether a model or prompt change made the system better or worse. Sofvie’s platform serves safety-critical industrial analytics to multiple client tenants, so a wrong answer is not a cosmetic bug. The engagement needed the two things demo-grade NL-to-SQL systems never have: a deterministic definition of “correct output,” and a regression corpus that grows with real usage.
The solution
VeUP built the workload as a foundation-model-grounded ML system with the full evaluation stack. A labeled golden set of 100+ representative question pairs — originally an NL-to-SQL corpus, carried forward as a tool-call selection accuracy suite when the architecture pivoted to MCP orchestration — is the regression gate for every change. Output correctness is judged by a deterministic oracle: AST-level validation of every generated query before it can execute. Robustness is measured against an adversarial prompt-injection corpus run in CI. Model selection was treated as an explicit ML decision, A/B-compared on the golden set — Amazon Nova Lite 2 as the production workhorse, Amazon Nova Pro for harder questions, and Anthropic Claude on Bedrock qualified as the alternate. Fine-tuning was evaluated and deliberately rejected on corpus-size economics in favor of grounding and evaluation rigor. In-product thumbs feedback closes the loop, growing ground truth from live usage, with per-tenant evaluation and Amazon CloudWatch instrumenting quality KPIs from go-live.
Production outcomes
| KPI | Result |
|---|---|
| Evaluation discipline | 100+ labeled golden-set pairs scored on tool-call selection accuracy as the standing regression suite; deterministic AST validation as the supervised-output correctness gate; adversarial prompt-injection corpus in CI. |
| Model selection | Amazon Nova Lite 2 selected on measured golden-set performance, Nova Pro engaged for harder questions, Anthropic Claude on Bedrock held as the qualified alternate — a reversible, evidence-based routing decision rather than a default. |
| Production delivery | Production cutover four weeks ahead of the contracted date with deterministic output-safety gates live from day one; continuous ML-quality KPIs instrumented in CloudWatch, with realized values staying in Sofvie’s own AWS account. |
| Lessons & continuation | A deterministic correctness oracle beats subjective grading for structured-output NLP; golden sets survive architecture pivots and compound in value; fine-tuning is a cost decision, not a reflex — grounding plus evaluation rigor carried the quality bar here. |
Amazon RDS for MariaDB
AWS IAM