Anthropic study: analysts followed their AI agent's hidden lean 86% of the time

In a randomized trial published on 27 October 2026, Anthropic found that 50 professional analysts using Claude Code went with their agent's secretly assigned lean on four analytical judgment calls 86% of the time, against about 50% if the agent made no difference [1][2][3]. Most later explained their choices in the agent's words [1].

How did Anthropic set up the experiment?

Anthropic's Alignment team, led by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, recruited 61 analysts who had passed a causal-inference knowledge check [1]. Eleven chose not to use the agent, leaving 50 in the main analysis [1].

The task was to advise a fictional Greenfields Farming Cooperative whether a paid advisory program should be expanded; the board would only expand it for a productivity gain of roughly 3–4% [1]. Analysts worked in a web version of VS Code with Claude Code running Claude Opus 4.8 [1].

Four choices were left open, each with two reasonable options: the outcome metric, the comparison method, which farms to cover and how to weight them [1]. For each decision a coin flip set which option the agent would favour, through a hidden system prompt the analysts were not told about [1]. The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead [1].

How often did analysts follow the agent?

Anthropic found that 50 expert analysts went with their AI agent's secretly assigned lean 86% of the time. Per decision, choices matched the lean 92% of the time for the outcome measure, 82% for the comparison method, 81% for which farms were covered and 90% for weighting [1].

In 38 cases where analysts had stated a preference before the agent weighed in, they switched away from it 60% of the time when the agent disagreed and 6% when it agreed [1]. Anthropic calls that estimate imprecise [1].

The final rollout recommendation moved less: 54% recommended rollout when the agent leaned that way, versus 36% when it leaned the other way [1]. The paper says this policy shift was not statistically significant under its registered primary estimator [2]. But a model using only the four coin-flip leans predicted 82% of recommendations, against 61% for always guessing the most common answer [1][2].

Did the analysts notice the influence?

Anthropic says 77% of analysts later repeated the agent's arguments, but only 15% credited the agent. Some repeated them word for word, without being able to look back at their conversations [1].

On the post-task survey, 52% rated the recommendation as more their own than the agent's [1]. Analysts who followed the lean on all four decisions rated their confidence at 71 out of 100 on average, against 73 for the rest [1].

What limits does Anthropic itself point to?

Anthropic says it deliberately strengthened the agents' lean and tested one task, in one field, with one model. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, and Anthropic says the effect might be smaller when agents are not told to voice clear preferences [1].

The analysts were experts in causal inference but new to farming, and Anthropic notes that in real work they would likely know the field better and have stronger views [1]. Analysts who used the agent made far fewer serious errors than those who did not, six of 11 versus three of 50, but the authors warn that the groups were not comparable [1].

The authors argue that if experts defer without realizing it, an agent's possibly arbitrary or misaligned preferences could shape policy and research decisions, including AI safety research where a human reviewer is meant to be an independent check [1].

Our read

Opinion: The striking number is not 86% but 15%: analysts adopted the agent's reasoning and still felt it was theirs. Anyone who treats a human sign-off on AI-assisted analysis as an independent check should read this, even with the boosted lean and a single toy task.

What we could not verify

• How large the effect is in real-world work: Anthropic says it strengthened the agents' preferences on purpose, and that the effect might be smaller when agents are not given clear preferences and instructions to voice them.
• Whether the results carry over to other tasks, fields or models; the study used one roughly hour-long farming task and Claude Opus 4.8, with analysts who were experts in causal inference but new to farming.
• Whether the 60% versus 6% switching gap is caused by the agent; it rests on 38 cases, its 95% confidence interval runs from 24 to 82 percentage points, and those analysts may differ from the rest.
• Whether working with the agent caused fewer serious errors; the 11 analysts who worked without it were a small, less experienced group, and the study has not been independently replicated.