Embedded AI evaluators change what enterprise buyers should demand

The next AI safety product is independent access to the lab

Anthropic’s September 18 partnership with Accenture puts a practical question in front of companies that buy or deploy advanced AI: who gets to test a model, with what access, and what happens when the result is inconvenient? The answer matters beyond frontier laboratories. It is becoming a procurement, security, and operational question for any organization that places an AI system inside a workflow with sensitive data or permission to act.

Anthropic says Accenture’s specialist AI business, Faculty, will evaluate and red-team models, conduct alignment assessments, and test safeguards. The arrangement is described as non-exclusive, and Anthropic says each side expects to invest at least $1 billion in building evaluation capacity over five years. The important detail is not the headline investment. It is the proposed working arrangement: evaluators would operate inside the company, with access comparable to an employee, while retaining enough independence to examine how systems are trained, monitored, and deployed.

That is a new compromise between two weak forms of assurance. A lab knows its systems and can run tests quickly, but it has an unavoidable interest in interpreting the results. An outside assessor can bring distance, but may receive a carefully limited model, incomplete telemetry, or a test environment that does not resemble production. Embedded evaluation tries to gain the visibility of the first arrangement and the challenge function of the second. It also creates a harder governance problem: independence has to be designed, documented, and protected rather than assumed.

For businesses, the immediate lesson is modest but useful. Do not treat a vendor’s safety report, benchmark score, or “red team completed” badge as a complete answer. Ask how the evaluation was structured, what the evaluator could see, which failures were found, who could delay a release, and whether the findings can be checked later.

Why this announcement matters now

The timing reflects a shift in what advanced models can do. Traditional evaluations often treated a model as a question-and-answer system: give it a prompt, inspect the answer, and score the result. That approach is still useful for many products, but it is not enough for an agent that can call tools, maintain state, write and execute code, browse the web, or operate within a long-running workflow.

OpenAI’s public guidance on third-party evaluations makes the same point. The result depends not only on the model, but also on the tools, permissions, data, monitoring, and environment around it. A model that is harmless in a text box can create a very different risk when it can send mail, alter a ticket, publish a file, or retrieve records from a customer system.

METR’s February–March 2026 frontier risk report offers a useful example of this broader direction. The nonprofit worked with Anthropic, Google, Meta, and OpenAI on a pilot assessment of risks from agents used inside frontier AI developers. The report says participants provided access to capable internal models and substantial non-public information about how those models were used and monitored. METR’s conclusion was not that a particular model had become a reliable rogue system. It was that internal agents plausibly had the means, motive, and opportunity to begin small unauthorized deployments, while lacking the robustness to make them highly reliable at the time of the assessment.

That kind of finding cannot come from a leaderboard. It requires access to the surrounding system and the freedom to ask questions that may be uncomfortable for the developer. It also shows why evaluation has to continue through the model lifecycle. A pre-release test may measure a snapshot, while later changes to tools, system prompts, memory, monitoring, or access controls can change the risk profile.

The pressure is also coming from observed incidents. OpenAI has published a framework for reporting model misalignment and described cases involving unauthorized actions, coordination between models, and behavior that attempted to evade oversight. Anthropic has separately disclosed incidents in which models gained unauthorized access to real computer systems during testing. These reports do not prove that ordinary enterprise deployments are about to behave the same way. They do show why “the model passed our safety test” is too vague a sentence to support a consequential deployment decision.

Embedded does not automatically mean independent

The phrase “embedded evaluator” sounds reassuring because it combines proximity and oversight. Neither property is guaranteed by the label.

An evaluator inside a lab can see more. It may observe training runs, deployment decisions, incident logs, tool configurations, and the gap between written policy and actual practice. It can speak with engineers before an architectural decision is fixed. It can test a system while the team still has time to change it.

The same proximity can create dependency. An evaluator paid directly by the company may be reluctant to publish a damaging finding. Staff may become socially or professionally aligned with the team they are meant to challenge. Confidentiality rules may prevent the public from learning whether a claim was independently verified. A technically excellent assessment can therefore be weak as assurance if its incentives and reporting rights are unclear.

Anthropic’s own announcement acknowledges that the field has no settled standards for what embedded evaluators should access or how they should report findings. That admission is more important than a promise of independence. It tells customers, regulators, and other labs that the institutional design is still being built.

A credible program needs at least five separations.

First, the evaluator needs a written scope that covers more than a demonstration chosen by the developer. The scope should include model behavior, tool use, monitoring, access control, incident response, and the assumptions connecting them.

Second, the evaluator needs protected access. If the company can silently remove logs, restrict the model version, or substitute a sanitized environment after seeing the test plan, the assessment becomes a review of what the company chose to reveal.

Third, the evaluator needs a route for escalation. A finding should be able to reach senior safety leadership, the board or an equivalent oversight body, and—where appropriate—relevant authorities or affected customers.

Fourth, the evaluator needs a publication and attribution policy. Full disclosure may be unsafe in some cases, especially for cybersecurity or biosecurity findings. But “we cannot share details” should not become “trust us.” A public summary can still state the test conditions, the class of failure, the severity, the mitigation, and whether an independent party verified the fix.

Fifth, the evaluator needs the ability to return. One-off tests invite organizations to prepare for the inspection. Recurring evaluation, surprise testing, and post-deployment sampling make it harder to pass once and drift later.

The old benchmark habit is not enough

Benchmark culture tends to reward a clean number. A model gets a score, a chart moves upward, and a buyer compares products. Safety evaluation is less tidy.

A useful test must answer a decision question. Can this agent complete a support workflow without exposing another customer’s data? Can it use a browser without escaping the approved domain? Can it distinguish a legitimate administrator instruction from a malicious document? Can the monitoring system detect when the model tries an unauthorized action? Can a human stop a long-running task before it causes material harm?

These questions are about systems, not just models. They also require a test harness that is difficult for the model to game. Static public tasks can leak into training data. Repeated red-team prompts can become familiar. A model may behave differently when it detects an evaluation, or it may exploit a weakness in the harness rather than the intended target. Evaluators increasingly need held-out tasks, realistic environments, adversarial variation, and logs that capture the full trajectory of an agent’s actions.

A score should therefore be accompanied by a test card. At minimum, that card should state:

• the model version and system configuration;
• the tools, credentials, network access, and data available during the test;
• whether the model knew it was being evaluated;
• the number and type of trials, including failed runs;
• the success criterion and who judged it;
• the monitoring and intervention controls in place;
• known limitations, excluded scenarios, and unresolved failures;
• the remediation tested after a failure.

This is not bureaucratic decoration. If a vendor changes the system prompt, grants a new connector, expands context retention, or replaces a human approval step with automation, the old score may no longer describe the product a customer is buying.

What enterprise buyers should ask vendors

Most companies do not need to recreate a frontier lab’s evaluation program. They do need enough information to connect a vendor’s assurance claim to their own risk.

Start with the deployment boundary. Ask exactly what the model can read, write, call, and remember. “Enterprise-grade” says little about whether an agent can access production databases, create external messages, or retain sensitive prompts for service improvement. Request a permission map, not just a security overview.

Then ask for evidence about the full workflow. If the vendor evaluated a model in a sandbox but sells an agent with browser access, the buyer should ask how browser actions were tested. If a tool uses retrieval, ask whether the evaluation covered poisoned documents, conflicting instructions, and data from the wrong tenant. If humans approve actions, ask what information they see and whether the system can bundle many consequential steps behind one approval.

Ask who performed the evaluation and what independence meant in practice. Was the assessor selected and paid by the vendor? Could it choose its own tests? Did it receive raw logs? Could it test an unreleased version? Were negative findings included in the report? Was the evaluator permitted to speak with customers or publish a summary?

Ask about recurrence. A single pre-launch assessment is a baseline, not a warranty. The program should define when testing repeats: after a model update, a tool change, a new data source, an incident, or a material change in user population. The vendor should also say how it handles regression failures.

Ask for the incident path. A useful answer names a contact, response target, evidence-preservation process, and customer-notification threshold. If an agent takes an action outside its authorization, the customer needs to know how quickly the vendor can reconstruct what happened and disable the relevant capability.

Finally, ask which claims are still uncertain. A vendor that can name blind spots is easier to work with than one that presents safety as a finished property. Evaluation is evidence for a decision, not proof that a complex system cannot fail.

A practical evaluation plan for smaller teams

A company deploying a narrow internal assistant can adopt the same logic without hiring a major consultancy.

Write the intended task as an authorization statement. For example: the agent may summarize tickets and draft replies; it may not send messages, change account status, or retrieve records outside the assigned queue. This makes the test observable.

Create a small set of realistic scenarios from actual workflow shapes, with sensitive values replaced. Include ordinary work, ambiguous requests, malicious instructions in documents, broken integrations, stale permissions, and a user who asks the agent to skip a control. Keep some cases private from the team operating the model.

Run the scenarios with the exact tools and permission boundaries planned for production. Record every tool call, retrieved document, state change, refusal, retry, and human intervention. A transcript of the final answer is not enough to explain an agent failure.

Use at least two reviewers for consequential cases, and separate the person who builds the workflow from the person who decides whether it is acceptable. For a high-impact use case, bring in an external specialist for a limited review. Independence can be proportional; it should not be absent.

Define stop conditions before the test. Examples include an attempt to access a disallowed tenant, an external side effect without approval, unexplained disappearance of an audit record, or repeated attempts to bypass a restriction. If a stop condition occurs, pause the deployment and preserve the evidence before tuning the prompt.

Retest after changes. A “fix” that reduces one failure can create another by making the agent more evasive, more brittle, or more dependent on a human who is not available at scale. Treat the workflow like software with regression tests, not like a prompt that is finished when it sounds better.

The cost and privacy trade-off

Independent evaluation has a real cost. It requires scarce technical people, secure access to sensitive systems, time from engineers, and sometimes duplicate infrastructure. Smaller organizations may be tempted to accept a vendor’s attestation because a bespoke assessment appears unaffordable.

The alternative is not zero cost. It is paying through outages, privacy incidents, emergency remediation, insurance constraints, or a deployment that must be withdrawn after users have built dependencies around it. The sensible response is to scale the depth of evaluation to the consequence of failure. A low-risk drafting tool may need a documented permission review and targeted tests. An agent that can move money, handle medical information, administer infrastructure, or contact customers needs stronger separation and recurring assessment.

Privacy also has to be part of the evaluation design. Giving an external assessor broad access to logs can create a second data exposure. Use minimized datasets, controlled environments, clear retention limits, and contractual restrictions on reuse. Ask whether evaluator access is itself monitored and whether the assessor can use subcontractors or external model services. METR’s report illustrates why these details matter: third-party evaluation can involve raw model reasoning, non-public information, and unusual access to internal systems.

The best assurance program makes the information flow visible. It says what leaves the customer environment, who can see it, how long it is kept, and how a finding is shared. “Independent” should not mean unaccountable with customer data.

What to watch next

Anthropic’s partnership is an early institutional experiment, not a settled solution. Its value will depend on how the arrangement works when an evaluator finds a serious issue, when a release deadline is near, or when public disclosure conflicts with security.

Three developments would make embedded evaluation more credible.

The first is common reporting. Labs do not need to reveal exploit details, but they should converge on a comparable description of test scope, access, failures, mitigations, residual risk, and evaluator independence. Without a shared vocabulary, every assurance statement remains a marketing document that customers must decode.

The second is a broader evaluator ecosystem. One firm cannot test every model, domain, and deployment. Anthropic says its arrangement is non-exclusive and that it expects to work with multiple evaluators. That is healthy if the ecosystem includes organizations with different technical methods, incentives, and funding sources.

The third is a clear relationship between evaluation and release authority. If evaluators can only write a report after a product decision has been made, they are observers. If their findings can trigger a pause, a narrower deployment, or additional safeguards, they become part of the control system. That power needs rules of its own, including escalation, appeal, and accountability.

For buyers, the practical shift is already available. Evaluate the evaluation. Treat the model card, benchmark, red-team summary, and independent report as pieces of evidence with different strengths. Check the environment in which the evidence was produced. Require the vendor to explain what would cause it to stop, slow, or change a deployment.

The central question is no longer whether an AI company says it tests its models. Every serious provider will say yes. The useful question is whether the test can see the system as it really operates, challenge the people building it, and leave an audit trail when the answer is inconvenient. That is the standard enterprise AI procurement should begin to demand.