One of AI’s most attractive promises is speed. A system can work through information, produce options and generate recommendations in a fraction of the time a person would need. The difficulty begins when our ability to produce possible answers grows faster than our ability to decide which answers deserve to be trusted.
OpenAI’s recent mathematics release offers an unusually vivid example. Its public repository contains hundreds of AI-generated mathematical manuscripts, not all of them accompanied by formal Lean verification; OpenAI explicitly warns that unformalised results may contain errors. The interesting problem is therefore no longer simply whether AI can produce sophisticated work. It is how much of that work humans can realistically verify.
Something similar is beginning to happen in organisations. AI can generate reports, forecasts, scenarios and recommendations almost without friction. Yet somebody still has to understand what they mean, decide how much confidence to place in them and take responsibility for what happens next.
That becomes particularly interesting once the AI is not merely faster, but actually better.
When accuracy is not enough
Take medical screening. In a narrowly defined diagnostic task, suppose an AI system identifies abnormalities more accurately than the average clinician and produces fewer false negatives overall. If the evidence is strong enough, insisting that a human must always make the call simply because humans are human becomes difficult to justify.
Now imagine that the system gives one patient an all-clear and is wrong.
The fact that human doctors also make mistakes does not resolve the issue. No diagnostic process is perfect, and a system that makes fewer errors overall may prevent more harm than one based solely on human judgement. The uncomfortable question is what happens in the individual case when automation fails.
Who explains the decision to the patient? Who decided that the system was reliable enough to act without a second opinion? Who determined which error rate was acceptable?
Even a rule such as “human review is required below 95 per cent confidence” simply moves the judgement elsewhere. Someone still chose 95 per cent.
That is why accuracy, although enormously important, cannot be the only criterion by which we decide how much authority to give AI.
Prediction, recommendation and decision
It helps to distinguish three things that are often bundled together.
An AI can predict that a patient has a very high probability of a particular outcome. From that prediction, it may recommend a course of action. The eventual decision, however, may involve considerations that cannot be settled by probability alone.
Imagine a system predicting with 97 per cent confidence that a seriously ill patient is unlikely to survive another six months. That estimate may be highly useful. It still does not tell us whether another treatment should be attempted, how much suffering is acceptable for a small chance of additional time, or how the patient values that time.
The same problem appears in business decisions. An AI could rank employees for redundancy according to performance, salary, skills, future potential and dozens of other variables. The calculation might be excellent. The more difficult question is why those variables were chosen, how they were weighted and what conception of fairness lies behind them.
Optimisation always optimises for something. The choice of that “something” is already a judgement.
This is where the claim that AI has the “better answer” starts to become more complicated. Better at predicting an outcome is not necessarily the same as better at deciding what ought to happen.
Where autonomy makes sense
None of this means that a human should be inserted into every automated process.
If software notices that a warehouse is running short of standard screws and automatically places another order within pre-agreed limits, requiring a manager to approve every transaction would add little value. The objective is clear, errors are relatively easy to detect and correct, and the consequences are limited.
The case becomes less straightforward as consequences grow more serious.
Rather than searching for one universal rule, it may be more useful to ask:
How serious would the consequences of an error be?
Can the decision be reversed?
Is the objective itself relatively uncontroversial?
How much uncertainty remains?
Does the decision involve competing values rather than only measurable outcomes?
Can a mistake be detected before the damage becomes irreversible?
An automated stock replenishment decision sits at one end of that spectrum. Deciding whether a firefighter should enter a burning building, whether a patient receives a high-risk treatment or which employee loses a job sits much further towards the other.
In those situations AI may still provide valuable information. It may calculate structural risk in the building, estimate medical probabilities or identify patterns in workforce data that people would miss. That does not automatically answer who should have the final authority.
The human in the loop can also become the bottleneck
“Keep a human in the loop” sounds like a reassuring solution. In practice, it can mean very different things.
If a person must independently repeat every analysis before accepting an AI recommendation, much of the promised increase in speed disappears. Human verification becomes the bottleneck.
Yet the opposite arrangement is not particularly reassuring either: a manager receives an AI recommendation, scans it for thirty seconds and clicks approve. Technically, there was a human in the loop. Whether meaningful judgement took place is another matter.
This problem is not theoretical. Experimental research has found that people can over-rely on AI-generated advice, even following it when it conflicts with contextual information available to them. Other recent work shows that people’s confidence and their ability to judge when AI advice is useful do not always align particularly well.
The interesting design problem is therefore not simply whether a human appears somewhere in the process. It is what that human is expected to contribute.
AI shapes judgement before we notice it
There is another reason why a final human approval may be less independent than it appears.
AI can influence a decision long before it recommends an answer.
I notice this even when using AI as a sparring partner. The questions it asks determine which parts of an issue I examine more closely. Its wording can make one argument appear stronger than another. What it challenges influences my reasoning, and so does what it leaves unchallenged.
That is not necessarily manipulation. Human colleagues, advisers and facilitators influence conversations in exactly the same way. The difference is that AI often arrives wrapped in the language of analysis, which can make its framing feel more neutral than it really is.
Research on human reliance on AI points in the same general direction. People do not respond to AI advice solely according to its objective accuracy; attitudes towards AI, trust and confidence affect how much weight they give it. In one 2026 study, participants with more positive attitudes towards AI performed worse at distinguishing real from synthetic faces when they received AI guidance.
This suggests that transparency may need to mean more than disclosing, at the end of a process, that AI was used. We may also need to understand how it shaped the options, assumptions and framing that produced the decision in the first place.
Who is responsible when the system is wrong?
For me, this is one of the hardest parts of the discussion.
When a person makes a consequential decision, we can at least ask that person to explain it. Their reasoning may have been flawed. They may have been biased. Human accountability is certainly no guarantee of a good decision. But there is somebody to question.
As AI takes on more of the analysis, recommendation and eventually perhaps the decision itself, responsibility becomes distributed. The developer built the system. The organisation chose to deploy it. Someone selected the data. Someone determined the acceptable risk threshold. A manager may have approved the process without reviewing the individual case at all.
So perhaps the real question is not simply, “Who made this decision?”
It may be:
Who designed the conditions under which this decision could be delegated?
That is a more demanding version of accountability because responsibility does not disappear when the human leaves the immediate decision loop. It moves upstream.
And then there is the longer-term problem
Even if we design sensible rules for delegation, another question remains.
Human oversight assumes that the person overseeing the system has enough experience and judgement to recognise when something is wrong. Where will that judgement come from if AI increasingly performs the analysis that once helped people develop it?
We already know that relying on AI advice can distort decisions when people trust it too much. There is also growing concern in the research literature about the longer-term cognitive effects of overdependence, although the evidence is still developing and should not be overstated.
The issue is easy to see in practical terms. Someone who has spent years making difficult decisions, getting some of them wrong and learning from the consequences may notice something that a novice would miss. If future professionals spend much of their careers reviewing AI-generated options rather than building those judgements themselves, we may eventually ask them to supervise systems using capabilities they had fewer opportunities to acquire.
That is not an argument for preserving inefficient work for its own sake. We did not keep doing long division by hand because calculators might make us rusty.
Judgement may be different, though, because it is precisely the capability we expect humans to contribute when automation reaches its limits.
And that leaves me with a question for which I do not yet have a satisfying answer:
If AI increasingly does the analysis, frames the options and proposes the decision, how do humans retain the judgment needed to challenge it?
