AI & Models
Microsoft launches ASSERT for AI behavior testing
Microsoft has released ASSERT, an open-source framework that appears to help developers evaluate AI system behavior for specific product and service requirements.
On Tuesday, Microsoft took the wraps off ASSERT, an open-source framework designed to evaluate application-specific AI behavior. The tool, whose name stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing, aims to simplify the testing process for developers. It allows them to turn high-level, natural-language descriptions of goals, policies, or intended behaviors into structured, scored tests. This release comes as companies and developers face what appears to be a new, specific need to ensure AI systems behave as intended for their specific products or services.
ASSERT operates by taking plain-language descriptions of an AI model’s expected behavior and policies, converting them into a structured set of acceptable and unacceptable behaviors. The framework then generates problem scenarios and test cases, runs them against the target system, and scores the results. To help developers inspect where failures happen, the tool can record the paths the AI system takes, including intermediate actions and tool calls. Developers can also provide system context, tools, and constraints to further customize what the evaluations cover. For example, a developer could specify that a document research AI agent should not send emails to people outside the company, and ASSERT will use those rules to generate test cases to check compliance on an ongoing basis.
According to Microsoft, the framework fills a gap that broader, more general evaluations cannot address when AI models must behave in a manner shaped by an application’s specific context, policies, and tools. Sarah Bird, the chief product officer of Responsible AI at Microsoft, emphasized the importance of this tailored approach. “One of the things we’ve learned is that evaluations are absolutely critical to making good decisions,” Bird said. She noted that without understanding the behavior of an AI system, organizations struggle to know if it meets their standards, adding that building a trustworthy system requires evaluating many more dimensions that are application-specific. Bird stated that ASSERT can be used to evaluate systems during development, after deployment, and for continuous monitoring.
The release of ASSERT aligns with a gradual, broader shift in the artificial intelligence industry toward repeatable testing and regression checks as models grow more capable. Microsoft’s framework joins a growing ecosystem of evaluation tools and benchmarks. These include Stanford University’s HELM (Holistic Evaluation of Language Models) benchmark, MLCommons’ AILuminate (an AI evaluation benchmark), and METR (an AI evaluation group), which have been rolling out benchmarks to measure how models behave under different conditions.
Why it matters
The framework addresses a critical gap in AI development: while general benchmarks exist, they often fail to account for the specific context, policies, and tools that define how an AI application should behave in production.