Your team says AI saves them time. Here's how to find out if it does

Who this is for: you've been asked to make AI pay in your organisation, in a team, a process or a department. By the end you'll know how to find out whether it's actually saving time and money, with a six-step plan for one process, and you'll know which parts of that plan the evidence backs and which are just common practice.

Skip it if you use a chat assistant on your own work and want tips for it. This is about changing how a process runs when other people work in it.

On the length: it's a long read. The argument and the plan are in the first third. The rest answers four questions my last piece left open, then goes through the claims you'll hear most and how little sits behind them.

Ask a team using AI whether it saves them time and most will say yes. In a 2025 trial, experienced software developers said the same, after AI tools had made them slower. If what your people say is all you have to go on, you don't know yet.

In May 2026 the two biggest AI labs both went into the services business. OpenAI launched a company to put its models to work inside other firms, with "more than $4 billion of initial investment". Anthropic announced a services venture with Blackstone, Hellman & Friedman and Goldman Sachs. OpenAI's chief operating officer, Brad Lightcap, was quoted giving the reason: customers "need help going from pilot to production".

The same year, a survey asked nearly 6,000 executives in the US, the UK, Germany and Australia what AI had done for them. Nine in ten saw no effect on their productivity or employment over the previous three years (Yotzov, Bloom and colleagues, a 2026 working paper for the US National Bureau of Economic Research).

So the models do what they say. Getting them to change how an organisation runs is where it stalls. I wanted to know what the evidence says works at that step. So I ran the question through the research process I use for everything. It pulls the primary sources, grades each claim by how good its evidence is, and keeps the authors' own caveats with the claim.

Building the AI turns out to be the easy part. What the evidence backs is the set of checks around it. Measure the process before you start. Run the new way against a comparison. Test whether the people reviewing it still catch mistakes. Carry on measuring after it goes live. Most of that evidence doesn't come from AI at all. It comes from hospital trials, airport screening and the rules banks already follow for testing their models.

"Deploying AI" covers three different things, and most headline numbers mix them up. One person using a chat assistant on their own work is the best measured, with proper trials behind it. A firm saying it "uses AI" is what surveys count, and the definitions keep moving: the US Census Bureau changed its question in November 2025, so its figures from before and after can't be compared. This piece is about the thing in between. One process, redesigned so that some of its steps are done by AI. That's the job of the engineers the labs send into companies, and it has the thinnest direct evidence of the three.

Three findings that should change how you run it

People think AI sped them up when it slowed them down. In a randomised trial by METR, a research group, 16 experienced open-source developers took 19% longer to finish tasks with AI tools. Afterwards they still believed AI had sped them up by 20%. METR now marks that study as out of date, and its 2026 follow-up with newer tools was inconclusive. What survives is the lesson about measurement. What people report about their own speed isn't evidence of their speed.

The lesson that survives the update is about measurement: what people report is not what the clock shows.

That matters, because self-report is the easiest number to collect. The UK government's cross-department Copilot experiment, written up in June 2025, reported 26 minutes saved a day. That number is what users said, with no comparison group. When the Department for Business and Trade evaluated the same tool in August 2025, it "did not find evidence that time savings have led to improved productivity".

The gains aren't reaching pay or profit yet. Danish researchers linked 25,000 workers in exposed jobs to national pay records and ruled out any effect on earnings above 2% (Humlum and Vestergaard, revised March 2026). In a trial across 7,137 workers, Microsoft's own researchers found that Copilot users spent about two fewer hours a week on email, with no detectable change in the mix of work they did (Dillon and colleagues). Across these studies, time saved on a task didn't turn into more output, pay or profit. Whether deliberately redesigning the work fixes that hasn't been tested.

The human check is the weakest control, not the strongest. "Keep a human in the loop" is the standard reassurance. A study of 41 government policies requiring human oversight of algorithms found that people "defer to automated systems", and that the requirements "provide a false sense of security" (Green, 2022). It's the question that matters most, so I come back to it below with the numbers.

Six moves for one process

This is the method that holds up against both kinds of evidence: the narrow trials of AI on single tasks, and the older literature on controls from medicine, aviation and banking. It's for one process with an outcome you can count. It isn't evidenced for "AI transformation" across a whole firm, where no randomised study exists.

Moves 1, 2, 5 and 6 are the checks. That is where the evidence is.

1. Qualify. Pick one process with an owner and a number you can count. Before you start, write down the result that would stop the project. Research on runaway IT projects found that clear success and failure criteria are among the things that let a failing project get stopped (Keil and Robey, 1999).

2. Baseline. Measure how today's process runs: how long a case takes, how often work gets corrected or sent back, what each case costs. Start with what the systems record, then interview and watch people for what the records miss. Put an end date on it. The UK government's service manual says a discovery phase is typically "around 4 to 8 weeks". That's guidance: my research found no study of how long this step should take.

3. Sort every step, in this order: cut it, simplify it, write a rule for it, give it to AI, or keep a person. A review of 63 studies found that plain rules suit work that's high-volume, standardised, low on exceptions and digital (Wewerka and Reichert, 2020). AI suits only work inside the range it handles well. In a trial with 758 consultants, AI raised quality and speed on tasks inside that range and cut correctness on tasks outside it (Dell'Acqua and colleagues, 2026). A person stays where a mistake is expensive or can't be undone, and where the law requires one. That last rule is sensible practice. I found no study that tests it.

4. Build at the point of decision. The output arrives inside the work, at the moment someone decides, without them going to look for it. The evidence for this is good, and it's more specific than "use the tools people already have". More on that below.

5. Prove it against a comparison. Run the new process beside the old one, then widen it in stages. Plant known errors to measure whether reviewers still catch them. Ask for a reason whenever someone overrides the AI.

6. Keep measuring after it goes live. This is the step the audits found missing. In May 2025 the US tax authority's own inspector general found that the IRS had evaluated its AI for picking returns to audit only before deployment, with no monitoring afterwards. And expect a dip first: across firms, the measured return to AI has fallen before it rose.

Here it is as a plan you can copy for one process.

SIX MOVES FOR ONE PROCESS

1  QUALIFY          One process. One owner. A number you can count.
                    Write down the result that stops the project.

2  BASELINE         Measure today: time per case, rework, cost per
                    case, share that goes straight through.
                    Records first, then people. Set an end date.

3  SORT             Every step, asked in this order:
                    CUT  SIMPLIFY  RULE  AI  PERSON
                    A person keeps anything expensive or hard
                    to undo.

4  BUILD            The output arrives where the decision is made,
                    without anyone having to look for it.
                    Ask for a reason when someone overrides it.

5  PROVE            Run new beside old, against a comparison.
                    Widen in stages. Plant known errors and track
                    each reviewer's catch rate. A falling catch
                    rate is a stop signal. Count review time
                    as a cost.

6  KEEP MEASURING   At 3 and 6 months, read the direction and
                    look for harm. Judge the payoff later, not on
                    one quarter. Name who stops it, and on what
                    evidence.

The four questions, answered

In my piece on forward deployed engineers, the engineers AI firms send to work inside client companies, I took apart one firm's method. Vasuman Moza of Varick Agents set it out in an interview: find the real process, sort every step, build inside the tools people already use, and measure before and after. It left four questions open. Here's what the evidence says on each.

Does the person approving still catch the mistakes?

Often not, and experience doesn't protect them. In one study, 27 radiologists read mammograms with an AI's suggestion on screen (Dratsch and colleagues, 2023). When the AI was right, the very experienced readers were right 82.3% of the time. When it was wrong, they were right 45.5% of the time. The AI was simulated, there was no group reading without it, and only five of the readers were very experienced. Those three caveats are usually dropped when the study is quoted.

Accuracy held when the AI was right and fell when it was wrong, at every level of experience.

Rarity makes it worse. In another study, six radiologists looked for cancers planted in scans (Evans, Birdwell and Wolfe, 2013). When cancers were as rare as in normal screening, they missed 30%. When the same cancers were common, they missed 12%. An AI that's right nearly all the time turns the person checking it into someone hunting for something rare.

Training doesn't fix it. A review of the research on people working with automated aids concluded that automation bias "cannot be prevented by training or instructions" (Parasuraman and Manzey, 2010).

What does help is measuring the reviewer. Airport security already does this. EU rules, kept in UK law, require screening machines to project fictional threat images onto real bags. The rate is adjustable. Each screener's results are stored for 12 months, and the screener is told when they miss one. A 2025 study found these scores predicted how screeners did in covert tests, where real prohibited items were smuggled through (Buser and colleagues, a single study). That shows the scores are a valid measure. Whether planting images keeps screeners alert is a separate claim, and nothing I found tests it.

Three other things help, on weaker evidence. Requiring a reason to override the advice was strongly associated with decision support working, across 162 clinical trials (Roshanov and colleagues, 2013). Pilots who felt accountable for the outcome checked more and made fewer errors (Mosier, Skitka and colleagues, 1996, a correlation). And showing information rather than a bare recommendation helps, according to a review of 74 clinical studies (Goddard and colleagues, 2012).

Volume works against all of it. At one firm, after AI coding tools arrived, the number of code changes submitted rose 3.1 times, while the number of reviewers rose 1.5 times (He and colleagues, a July 2026 preprint). The share of changes getting a substantive human review fell from 39% to 21%. If review capacity doesn't grow with the work, the check thins out without anyone deciding it should.

Is building inside the tools people already use enough?

Half right. A review of 70 trials of clinical decision support found it worked in 75% of trials when it arrived automatically in the clinician's workflow, and in none when they had to go and look for it (Kawamoto and colleagues, 2005). So delivery at the moment of decision matters.

But being inside the existing system isn't the same thing. A larger review covered 162 trials (Roshanov and colleagues, 2013). Where the advice was built into electronic charting or ordering systems, it was less likely to show it worked, not more. The review is observational, so this is an association, not proof. What carried the effect was how and when the advice reached the person deciding, not where it was hosted.

Moza's reason for building inside existing tools still stands: staff don't want to move to a new system. That's about getting people to use it. Whether it works depends on the delivery.

How long before the before-and-after means anything?

Longer than most plans allow. Deloitte surveyed firms across Europe and the Middle East and got 1,854 responses. By the respondents' own account, a typical AI use case took two to four years to earn a satisfactory return. Only 6% paid back within a year. The MIT report behind the "95% of AI pilots fail" headline concedes in its own limitations section that its six-month window "may be insufficient". And in US Census manufacturing data, the measured productivity return to AI follows a curve that dips before it rises, over roughly four to five years (McElheran and colleagues, 2025).

So read three and six months for direction and for harm, against a comparison. A before-and-after with no comparison can't tell your change apart from a market that was rising anyway. Don't call success or failure on one quarter.

Does a firm that works this way get better results?

I found no test of it. No study in my research compares the ways of deploying AI on outcomes: engineers sent in by the vendor, consultancies, in-house teams, or staff left to get on with it. Every claim that one beats another is either self-reported or comes from the people selling it.

What exists is cases, and they split cleanly.

The ones that worked were measured. The audited ones had no measure to fail against.

Where someone measured against a comparison, the gains were real and modest. A customer-support assistant at a US software firm raised resolutions an hour by 14% (Brynjolfsson, Li and Raymond, 2025). Novices gained 34%. Experts gained little. At UCLA Health, in a trial with 238 doctors, the best AI note-taker saved about 41 seconds a note against about 18 seconds in the comparison group, according to UCLA's release on the study. On one online retail platform, seven AI uses were tested at random and four raised sales, by up to 16.3% (Zhang and colleagues, a preprint partly written by the platform's own staff).

Where auditors and evaluators looked, the measuring was missing. The IRS had no process to show its audit-selection AI beat the old method. Australia's National Audit Office found that 74% of the Australian Taxation Office's 43 models in production had no completed data-ethics assessment, and no monitoring in production. An independent evaluation of the NHS AI Diagnostic Fund found the rollout "took longer than anticipated" and measured no clinical outcome.

That isn't proof that measuring causes success. Firms that measure may be different in other ways. But it's the same pattern my paper on why AI projects fail found in its two success cases, now in three more organisations, in three countries.

Your bank already does this

Moves five and six are the ones the audits found missing. Banks covered by the UK's model risk rules are already expected to run them, on models that inform business decisions, AI included.

The Prudential Regulation Authority's model risk rules for banks (supervisory statement SS1/23) expect "comparisons of the model output with the output of available challenger models", "parallel outcomes analysis", and ongoing monitoring of vendor model performance. They require a named senior manager to own model risk. A revised version took effect in April 2026, and I haven't checked what changed. In the US, the Federal Reserve's guidance (SR 11-7, from 2011) asks for "effective challenge" by "objective, informed parties".

Banks are already expected to run these checks on their models, under rules and guidance on both sides of the Atlantic.

I've sat on the approving side. From 2023 I was a member of a bank's committee that approved AI and cloud initiatives for development and deployment. Each discipline signed off its own area: information security, infrastructure, risk, legal, compliance and finance. Most of what came to it was about data in the cloud, hosting, resilience and data moving offshore. I didn't design that process. I was a member of it.

The labs' adverts show where the gap sits. Anthropic advertises two roles. Its forward deployed engineers build "production applications with Claude models" inside customer systems. A separate Technical Deployment Lead owns the delivery plan and "value measurement and ROI", and doesn't write production code. None of the engineering adverts in my research mentions a challenge function : the people whose job is to try to prove the model wrong. US bank guidance has asked for it since at least 2011. Whether buyers outside banking will pay for it is a question the research can't answer.

Repeated most, supported least

You'll hear these in nearly every pitch. Here's what sits behind them.

"70% of change programmes fail." This started as a reengineering estimate in Hammer and Champy's 1993 book, reported in secondary sources as their "unscientific estimate". Hammer was later quoted saying it had been "transmogrified and distorted into a normative statement". A review of the sources found "no valid and reliable empirical evidence" for the 70% (Hughes, 2011).
"95% of AI pilots fail." It comes from the MIT NANDA report, whose own six-month window it says may understate success. My failure paper takes it apart.
"Forward deployed engineer job postings are up 800%." The trail ends at a Financial Times report citing a job-listings site, and my research could open neither the report nor the data.
"Klarna's AI does the work of 700 agents." That's a company press claim, and it isn't in Klarna's filing to list its shares. Its chief executive was later reported saying service quality fell, and that it was hiring people again.
"Put a human in the loop and it's safe." See the radiologists above. The EU AI Act names automation bias as a risk in its own article on human oversight.
"Embed it in the existing system and adoption follows." Half true, as the clinical trials above show. Delivery at the moment of decision is supported. Embedding alone isn't.

Where this breaks

The method is assembled. Moves four to six come from clinical decision support, airport screening and bank model risk, not from trials of AI deployment, because almost none exist. Carrying them across is reasonable. It's still an inference.

It doesn't fit firm-wide platform bets, where the unit isn't a process. It struggles with low-volume work that's mostly judgement, where there's nothing to count. And it may not hold for AI agents acting across many systems at speed, where separating doing from approving at every step might not be possible.

The fair objection from anyone running a programme is cost. A comparison means running two processes for a while. Planting errors and tracking reviewers is work that has to go in the budget. That's true, and the checks aren't free. But their cost is known before you start. The cost of a deployment that wasn't measured turns up later, in an audit report, as it did at the IRS and the Australian Taxation Office. And none of this is meant for one person using a chat assistant on their own work. It's for a process other people depend on.

Two things would prove the emphasis wrong. A randomised study across firms showing that deployments with no baseline, no comparison and no measured review pay off as often as those with them. Or official data from 2026 and 2027 showing firm-level gains from staff simply picking the tools up, with no redesign.

Until then, if you do one move from this piece, do the sixth. It's the one the IRS skipped.

Sources

How this was produced: a research run on 13 October 2026 that pulled primary sources and graded each claim by the strength of its evidence. The full evidence file, with every source's tier and caveats, is in my research notes. The main sources:

OpenAI, Deployment Company announcement, 11 May 2026; Goldman Sachs Asset Management, release on the Anthropic services venture, 4 May 2026. Brad Lightcap's quote is from press reporting of the launch.
Yotzov, Barrero, Bloom, Davis and colleagues, NBER working paper w34836, 2026.
METR, randomised controlled trial of experienced open-source developers, July 2025: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ (and METR's February 2026 update).
UK Government Digital Service, cross-government Copilot experiment findings, June 2025; Department for Business and Trade, Microsoft 365 Copilot evaluation, August 2025.
Humlum and Vestergaard, working paper revised March 2026; Dillon and colleagues (Microsoft), arXiv 2504.11436.
Green, "The flaws of policies requiring human oversight of government algorithms", 2022.
US Census Bureau, business AI use survey, 2025-26.
Keil and Robey, 1999; GOV.UK Service Manual, discovery phase; Wewerka and Reichert, 2020; Dell'Acqua and colleagues, Organization Science, 2026.
Dratsch and colleagues, Radiology, 2023; Evans, Birdwell and Wolfe, 2013; Parasuraman and Manzey, 2010; EU Regulation 2015/1998, Annex, Division 12; Buser and colleagues, 2025; Mosier, Skitka and colleagues, 1996; Goddard and colleagues, 2012; He and colleagues, arXiv 2607.01904, 2026.
Kawamoto and colleagues, BMJ, 2005; Roshanov and colleagues, BMJ, 2013.
Deloitte, EMEA AI survey, 22 October 2025; MIT NANDA report, limitations section; McElheran, Yang, Kroff and Brynjolfsson, US Census Bureau working paper CES-25-27, 2025. Deloitte and NANDA are set out in my paper: https://jamiewatters.work/journey/why-ai-projects-fail
Brynjolfsson, Li and Raymond, Quarterly Journal of Economics, 2025; UCLA Health release on the ambient AI scribe trial (NEJM AI, 2025); Zhang and colleagues, arXiv 2510.12049.
Treasury Inspector General for Tax Administration, report 2025-308-022, May 2025; Australian National Audit Office, governance of AI at the Australian Taxation Office; NIHR evaluation of the NHS AI Diagnostic Fund, 2025.
Prudential Regulation Authority, SS1/23, Model risk management principles for banks: https://www.bankofengland.co.uk/prudential-regulation/publication/2023/may/model-risk-management-principles-for-banks-ss ; Federal Reserve, SR 11-7, 2011.
Anthropic job adverts for Forward Deployed Engineer and Technical Deployment Lead, October 2026.
Hughes, 2011, on the 70% change-failure figure; Klarna's F-1 filing; EU AI Act, Article 14.