Executive Summary
Nexus Labs runs an enterprise sales-ops automation agent product that chains multiple AI models and internal tools per user task. They attempted to build a fine-tuning dataset from 41,000 production agent traces but found nearly half the data unusable due to retries, fallbacks, and incorrect tool call results masked as successes. By deploying the Bifrost gateway to unify and enrich trace metadata—capturing actual provider, fallback chains, and metrics—they reduced corrupted traces from 17% to under 3%. This enabled filtering for single-provider, single-model traces without retries, improving dataset quality for fine-tuning. However, Bifrost does not validate tool result correctness, so post-hoc schema validation remains necessary.
ResultCorrupted trace rate dropped from 17% to under 3%, enabling use of 29% of raw traces as high-quality training data instead of relying on noisy user thumbs-up signals.
Implementation ComplexityMedium effort
Best forEnterprise software - sales operations automation / AI engineering team building fine-tuning datasets / Bifrost gateway