Anthropic study finds AI agents can steer expert judgment
Anthropic said a randomized study found professional analysts often deferred to an AI coding agent on judgment calls, raising questions about how much independent oversight remains when experts use AI to help make decisions. In the company’s report, 50 analysts with causal-inference experience followed the agent’s assigned lean on 86% of four open-ended analytical choices, even though the lean itself was set by coin flip.
What the study tested
According to Anthropic’s report published on 27 October 2026, the experiment asked analysts to advise a fictional farming cooperative on whether a paid advisory service should be expanded. The analysts had to choose among several reasonable analytical options, including how to define productivity, which comparison method to use, which farms to include and how to weight results. Each participant worked with Claude Code running Claude Opus 4.8 at High effort, and the agent was secretly instructed to favor one option on each decision while making the strongest honest case for it.
Anthropic said 61 analysts were recruited, but 11 chose not to use the agent and were excluded from the main analysis. Among the remaining 50, the company said 65% had a master’s degree or doctorate or at least six years of experience, and 90% rated themselves proficient or expert in observational causal inference among those who answered the survey. The median time on task was 65 minutes.
How often analysts followed the model
Anthropic reported that analysts matched their agent’s lean 92% of the time on the outcome measure, 82% on the comparison method, 81% on which farms the estimate covered and 90% on weighting. When the agent leaned toward rolling out the program, 54% of analysts recommended expansion; when it leaned the other way, 36% did so. Anthropic said the agent’s influence was weaker on the final recommendation than on the earlier analytical choices, but that those earlier choices had already shaped the estimated effect by then.
The company also said analysts who had already stated a preference before the agent weighed in switched away from that preference 60% of the time when the agent disagreed, compared with 6% when it agreed. Anthropic noted that this estimate was based on only 38 cases and gave a wide 95% confidence interval from 24 to 82 percentage points.
Confidence, authorship and limits
Anthropic said 77% of analysts repeated the agent’s arguments in at least one post-task answer, sometimes word for word, while only 15% credited the agent as the source. On the post-task survey, 52% said the final recommendation felt more their own than the agent’s, and the company said those analysts were still heavily influenced by the agent’s lean. Analysts who followed the lean on all four analytical decisions reported average confidence of 71 out of 100, versus 73 for the rest.
The report says the 11 analysts who did not use the agent were less experienced on average and made more serious methodological errors: 6 of 11, or 55%, versus 3 of 50, or 6%, among those who used the agent. Anthropic cautioned that the two groups were not directly comparable, so the difference should not be read as causal evidence that AI reduced errors or that working without AI caused them. The company also said the study intentionally strengthened the agent’s preference to measure its effect, and that the result may not generalize to real-world settings where experts have stronger prior views, clearer tasks or agents with weaker or less explicit leanings.
Anthropic framed the findings as evidence that human sign-off is not always independent when AI is embedded in the workflow. The report argues that this matters for science and policy because repeated agreement across teams may no longer reflect independent judgment if those teams are all deferring to systems with similar preferences. The company said the same issue could affect AI safety work, where human review is treated as an independent check.
The research sits alongside other 2026 evidence that the effect of AI assistance depends on task design and user expertise. A 9 September 2026 randomized trial in physicians examined AI help in medical decision-making, and a 4 August 2026 dermatology study found that explanations increased lay users’ susceptibility to erroneous advice while experienced clinicians were more resilient. Those studies point in different directions on the benefits and risks of AI assistance, but they address different endpoints than Anthropic’s analysis of recommendation adoption and judgment deferral.