In this post, we’ll explore what’s new in MLflow v3.10, walk you through getting started with SageMaker AI MLflow Apps, and how to leverage these enhancements to build generative AI applications.

MLflow 3.10 introduces a set of targeted improvements to the MLflow ecosystem that extend the tracing and observability capabilities established in MLflow 3.0, with a particular focus on generative AI application development and agentic workflows. On the generative AI front, this release delivers improved tracing for complex multi-turn workflows, tighter integration with popular LLM frameworks and libraries, and streamlined logging for generative AI interactions and invocations. Evaluation receives a substantial upgrade through the mlflow.genai.evaluation() API, which provides a programmatic interface for systematically measuring and maintaining generative AI quality across the development-to-production lifecycle with built-in metrics covering relevance, faithfulness, correctness, and safety—all of which integrate seamlessly with SageMaker AI workflows.

Observability improvements include more granular trace filtering and search, richer metadata capture for debugging and root-cause analysis, and pre-built performance dashboards that surface workload level metrics—latency distributions, request counts, quality scores, and token usage—at a glance without manual chart configuration, giving teams running production workloads clear visibility into operational costs while MLflow workspaces provide a structured way to organize MLflow artifacts across teams and projects, as shown below.

These improvements coupled with SageMaker AI provide an enterprise-grade generative AI infrastructure, making it straightforward to track experiments, monitor generative AI performance, and maintain governance across AI applications at scale.

For new users, creating a SageMaker AI MLflow App is straightforward through the SageMaker Studio console, AWS CLI, or API. The default configuration automatically provisions MLflow 3.10, giving you immediate access to all the latest capabilities.