How should evaluators test AI systems (as opposed to models)?

Overview

A large share of evaluation research has focused on the model layer. However, users interact with AI through AI systems (i.e. applications built on top of models), making them the primary interface through which both the features and risks are experienced. It is therefore important to invest in and advance the evaluation of AI systems, not just models. Trust in AI systems is crucial to confident adoption and diffusion of the technology across industries.

At the International Network for Advanced AI Measurement, Evaluation and Science Directors’ meeting in San Diego in December 2025, there was extensive discussion on “what makes a good evaluation.” Discussions highlighted that it is essential to contextualise evaluations to a system’s specific deployment setting (e.g. domain, use case, audience) and the tests must be representative of the system’s purpose and reflect real user interactions. This blog explores these considerations and outlines what is needed to evaluate AI systems for safety, reliability and capability. It also covers open research questions and future directions in this space.

This blogpost is most relevant for third-party or in-house evaluators testing AI systems deployed in specific contexts. For consistency, we refer to these practitioners as “evaluators,” whilst “evaluation techniques” refers to the graders and evaluation technology they use.

How are AI systems different from models and how do risks manifest for them?

The AI model (often a Large Language Model in today’s age) is the core engine, while the AI system (or application) is the complete product built around it. AI systems layer in components like retrieval-augmented generation (RAG) for knowledge grounding, guardrails like input/output filters for safety and compliance, and in more sophisticated deployments: skills/tools, autonomous planning, and networks of collaborating agents. The result is a spectrum ranging from a simple Q&A chatbot to a highly autonomous agent (or multi-agent setup) capable of tackling complex tasks with minimal human oversight.

Due to this spectrum, risks and testing needs are also highly contextual: shaped by the domain, use case, capabilities, deployment context and user base, amongst other factors. For instance, for a general-purpose model, hallucination may come up in the form of a fabricated historical fact. In a customer-service chatbot, this risk could manifest as fabricating a refund policy. With more autonomous deployments, failure modes multiply in the form of erroneous actions with tangible impact: processing the wrong refund, falsely claiming an action was taken, or exceeding its permitted boundaries. Similarly, for harmful content, at the model level, it may entail producing content that promotes violence or crime – but as the system becomes more specific, it is more about encompassing organisational policies and sector-specific regulations across industries like finance, healthcare, and legal services.

How should these AI systems be tested?

While testing fundamentals may be shared between model and system testing, the key difference is again contextualisation.

Models are often assessed through standardised benchmarks such as MMLU-Pro, GPQA Diamond, and SWE-Bench Verified, sometimes via public leaderboards. However, strong leaderboard performance does not guarantee that a model (or the system built on top of it) is fit for deployment, as it may still fail once embedded with domain-specific data, guardrails, retrieval components, or multi-turn workflows.

For this reason, standard leaderboard benchmarks are often insufficient for AI systems. They may serve as a starting point, especially in early evaluation stages, but rarely provide comprehensive coverage or translate directly to real-world deployment.

Singapore’s Infocomm Media Development Authority’s Starter Kit for Testing LLM-based Applications for Safety and Reliability (‘Starter Kit’) details some of these principles and outlines a clear process for testing AI systems. We present an adapted version here: 

• Identifying relevant risks and calibrating extent of testing

Evaluators should identify the relevant risks to test for the system, calibrate the extent of testing required (and associated thresholds) to ensure that the system is fit-for-purpose and reliable.

Common content-based risks include hallucination and inaccurate content, undesirable content, biased recommendations, and data leakage, which can be triggered by routine (or benign) use as well as adversarial inputs and prompt injections. Agentic systems take this a step further with the possibility of erroneous or unauthorised actions and privilege escalations, as explored in the Model Governance Framework for Agentic AI. There is also the risk of tools returning unexpected or malformed outputs, or becoming unavailable entirely.

To identify the risks that apply, and the required testing rigour, it is useful to ask questions like the following to assess the likelihood and impact of different risks:  

• What are the capabilities and features of the system? What are associated failure modes?
• What safeguards or guardrails have been deployed?
  • What are the risks that are not addressed by current guardrails?
  • What are the risks that emerge if guardrails fail?
• What is the impact of each failure mode? Are there organisational or regulatory implications?

• Constructing a realistic testing environment

Setting up the testing environment correctly is as important as the tests themselves. Evaluators should configure it to reflect real production conditions. While this is a constantly evolving space, some good practices include the following:

• Realistic access: The evaluator should access the system the same way end users would, such as via API, and replicate all relevant tools, services, guardrails, and input/output filters.
• Memory and session state: The environment should reflect any memory layers or session management mechanisms, including historical interactions that may influence system behaviour.
• Skills, tools, servers and external service configuration: For systems with external tool integrations, evaluators should ensure that the testing environment accurately replicates how the system connects to and communicates with those tools. For example, tool outputs or server protocols (e.g. MCP) should be proxied/reproduced faithfully in the test environment.
• Sandboxing: Evaluators should ensure that the environment is sufficiently isolated so that tests do not affect live systems or connected services in unanticipated ways.     

The Singapore and Korea AISI explored this in their bilateral testing exercise on agentic systems.  

• Running evals: Constructing test datasets, metrics and evaluation techniques

Dataset

A well-designed test dataset satisfies three key criteria.

• Representative of purpose: Prompts or test tasks should reflect the system’s actual use case. For a medical documentation system, for instance, use clinician-patient summaries rather than multiple-choice questions.
• Sufficient coverage: Evaluators should ensure comprehensive coverage across all relevant topics, failure modes, and difficulty levels. Higher-risk systems may require more in-depth testing and deliberate efforts to include edge cases. A practical approach involves mapping all categories and variations upfront, then systematically selecting a suitable number of prompts or test cases across each category to ensure representativeness and coverage in a structured manner. Starting with a small sample to surface early issues before scaling up is also recommended, as this allows for iterative refinement.
• Realistic interaction modes: Evaluations should reflect how users actually engage with the system, including different prompt formats, implicit versus explicit queries, benign and malicious inputs, and varying user personas.

Evaluation Techniques

The evaluation and metrics should cover both the output/outcome of the AI system and the manner of achieving this outcome. For instance, for an AI agent, outcome metrics could include accuracy, data leakage rates, attack success rate, while trajectory-level metrics could include tool-use correctness, partial task completion rates, etc.  

Metrics should align across the following:

• Test objectives: The metric should directly capture the quality being evaluated – e.g. accuracy vs leakage rates.
• Data format: For structured tasks like multiple-choice, accuracy is a natural fit. For free-text outputs, options such as semantic similarity, ROUGE, or BLEU may be more appropriate depending on whether meaning or exact phrasing matters.
• Organisational priorities: Metrics should reflect business or policy goals. In a hiring tool, evaluators might prioritise recall to avoid missing deserving candidates; in a content moderation tool, precision may matter more.
• Practical requirements: A system may perform well on effectiveness metrics but still be unsuitable for real-world use if it is prohibitively resource-intensive. Hence, it is also important to consider metrics like latency and response time, token or compute usage, cost per evaluation or per successful task.

The choice of metrics choice also depends on the system’s architecture, autonomy, and design.In agentic settings, simple pass/fail scoring is often insufficient. Evaluators should consider partial success metrics such as task completion rates or phase-wise completion rates, which better capture multi-step performance. It is equally important to measure consistency between what an agent does and what it claims to have done. However, the manner of checking these steps can vary.

For agents with pre-defined workflows, fidelity to the exact intended sequence matters, and hence metrics such as tool call sequence accuracy are appropriate. However, the same metric applied to a more open-ended agent may be misleading, as the agent may reach the correct goal via a different path. In such cases, open-ended milestone-based metrics like tool use accuracy, downstream state correctness (e.g. whether a file was updated correctly, regardless of the tool used) are more meaningful. Anthropic’s pieces on building effective AI agents and demystifying evals for AI agents cover this range well.

To evaluate effectively, it is essential to develop robust references to evaluate against. These can take the form of:

1. Ground-truth (e.g. a golden answer or “expected answer” to compare against)
2. Evaluation criteria which reflect expected outcomes, behaviour or milestones (e.g. the response should not encourage crime, the email content should not include any personal email IDs, the agent should have called the email tool, etc.)

The technology or technique itself depends on the context. Human evaluation remains the most reliable option but does not scale easily, making automated approaches necessary in practice. Rule-based evaluators apply fixed, deterministic checks and work well for objective criteria, but struggle with outputs requiring interpretation. LLM-as-a-Judge offers greater flexibility for nuanced, open-ended responses at scale, though it can be inconsistent across runs and may carry the same biases as the system being tested; fine-tuned variants offer greater precision but require meaningful investment and do not generalise well across tasks.

In practice, combining approaches tends to work best, e.g.rule-based checks for objective criteria and LLM-based judges for more subjective dimensions. Regardless, developing clear evaluation criteria and assessment milestones is always a helpful strategy.

Analysing results and trajectories to inform improvements and mitigations

Analysing test results helps establish whether AI systems can fit the purpose that they are designed for. Evaluators may structure their analysis in two complementary layers:

• Quantitative analysis: Slicing test results by dimensions, such as test case category (e.g. violent crime versus IP violations) or capability type (e.g. instruction-following versus domain accuracy), reveals where systems perform well or poorly. Assessment should therefore extend beyond pass/fail thresholds to get a more realistic and nuanced understanding of deployment readiness.
• Qualitative analysis: Once failure patterns are identified, qualitative review helps answer the why. Examining system logs, conversation transcripts, and other behavioural signals can surface the underlying causes of failures and inform targeted mitigations. Log analysis is particularly valuable for multi-turn and agentic systems, where failures may only become apparent across a sequence of steps. The piece: Seven simple steps for log analysis in AI systems provides useful strategies.

Together, quantitative and qualitative analysis give evaluators a fuller picture of what works versus not.

Establishing fit-for-purpose thresholds A commonly encountered challenge in evaluations is setting meaningful pass/fail thresholds, especially in the absence of a universal standard (given how contextual AI systems can be). Two approaches can help evaluators establish thresholds that are practical and meaningful:   No/low-tolerance failure modes: Rather than seeking a universal threshold, evaluators should identify scenarios that are simply unacceptable regardless of overall system performance. For these cases, if there are failure cases spotted during testing, they must be addressed (and re-tested) until resolved.   Comparative thresholds: Where absolute thresholds are hard to define, meaningful comparisons can serve as a useful reference point, such as benchmarking against prior versions of the system, equivalent human performance, or similar systems in the same domain.   When AI systems underperform, component-level testing (described in detail in the Starter Kit), and log analysis reveal where to improve. For instance, identifying whether data leakage occurs in tool use or at output stage would help determine whether to fix tool use safeguards, output filters, system prompts or all.

Finally, post-deployment monitoring of AI systems remains essential, as behaviour may shift with evolving integrations, interaction patterns, and external environments. The pre-deployment evaluation process can contribute to this by establishing evaluation suites (e.g. regression sets) and informing reassessment frequency and escalation triggers for ongoing monitoring.

Case Studies The Global AI Assurance Sandbox and Pilot demonstrate practical applications of these testing principles for real-world use cases.

What does this mean for evaluators?

Evidently, testing AI systems is meaningfully different from testing AI models and demands a broader set of capabilities. Evaluators would need to adapt and build capabilities across the following dimensions:

Building the Right Technical Capabilities

• Creating meaningful and representative test data is harder than it looks – hence the ability to generate high-quality synthetic test data at scale is an important one for evaluators. This often requires starting with a high-quality seed dataset and expanding it through extensions, variations, and perturbations. Validating this approach with real experts and comparing against real user logs, user personas, domain language, and known edge cases is essential to ensure quality and relevance. Having said that, in some cases, baseline or early-stage testing may still benefit from publicly available benchmarks, and finding the right balance in the use of context-specific system testing with more general or comparable evaluation approaches is also an important consideration for evaluators.

• Automated-grader quality is a big challenge. Evaluators should develop the capability to deploy multiple evaluation types on the same system simultaneously based on the requirements. Where LLM-as-a-judge is used, evaluators should stress-test its reliability, use granular and well-defined criteria, and not treat it as a default. Rule-based or human evaluation may be more appropriate depending on the context.
• Connectors and integrations – often there would be a need to connect to specific AI systems and require the suitable connectors and integrations.

Accessing Systems and Handling Data Responsibly

• Access and confidentiality requirements: Most evaluators will access AI systems via API, but some deployments, particularly in sensitive or regulated sectors, may require bespoke connectors or on-premises arrangements. Evaluators should be prepared to navigate operator confidentiality constraints and any hosting requirements that may shape their choice of cloud services.
• Data handling: Evaluators must also handle proprietary data with care, adopting data minimisation practices from the outset, including anonymising data early and ensuring safe management of any sensitive information encountered during testing.

Domain Expertise and Human Judgement

• Human Subject Matter Experts (SMEs) remain essential: Many AI systems operate in specialised domains where technical evaluation alone is insufficient. Evaluators should either develop independent domain expertise or have access to SMEs who can assess whether outputs are accurate, appropriate, and contextually sound. Human SMEs are particularly important for annotation, grading, and validating results in areas where automated methods fall short.

Regulatory Awareness

• Understanding legal context: Evaluating AI systems often requires understanding the legal/regulatory context in which they operate. For example, evaluating a hiring application requires familiarity with relevant fairness laws.

Working Within Resource Constraints

• Not all evaluators will have access to large teams or significant compute. When resources are limited, starting small is a practical and effective approach. A focused sample of test cases can surface early issues and guide prioritisation before scaling up. Evaluators should also consider fine-tuning evaluation tools and processes to the specific context of the system being tested, rather than relying on general-purpose approaches.

INESIA’s piece on “What should evaluators prioritise when resources are limited?” Covers this well. The Global AI Assurance Sandbox and Pilot also serve as useful references for how third-party evaluators can navigate these challenges.

What’s next / evidence gaps

System-level evaluation is still maturing, and several important questions remain open.

• Sector-specific benchmarks: General-purpose frameworks are insufficient for domains like healthcare, legal, or financial services, where risk tolerances and regulatory requirements differ significantly. Developing rigorous, sector-specific benchmarks remains an open challenge.
• Comparability of results: System test results are highly context-dependent and difficult to compare across deployments. Shared reporting standards that enable meaningful comparison without oversimplifying context are needed.
• Log and transcript analysis: Multi-turn and agentic systems increasingly require log analysis as a core evaluation method, but doing this effectively, reliably, at scale and within resource constraints… is tough.

Conclusion

Evaluating AI systems demands more than adapting model-level methods. To do it well, evaluators must build capabilities across several fronts: generating high-quality, representative test data; deploying the right mix of rule-based, LLM-based, and human evaluation methods; developing domain expertise or accessing subject matter experts; and handling operational considerations and sensitive data responsibly. They must also stay attuned to the regulatory contexts in which the systems they test operate.

Many hard questions remain open, from sector-specific benchmarks to the evaluation of increasingly autonomous agents, and answering them will require sustained international collaboration. Ultimately, the rigour of AI evaluation shapes the foundation of AI trust. The NAAIMES Network, as an international platform with wide representation and expertise across geographies, is uniquely placed to advance the sciences in this important area.