Telling AI to act as your customers makes its answer worse: 52% of shoppers would buy, it said. Real sales: 33%.

Ask an AI to pretend to be your customer and it will like your idea more than your customers do. In a pre-registered test against a supermarket's real purchase records, a panel of AI shoppers predicted that 52% of households would buy from a product category in the next 28 days. 33% did (BehaviorLens). Asked to predict coupon redemptions, the same panel said 32%. The tills said 14%.

I built a product on the opposite bet.

In January 2026 I wrote the plan for PlebTest: AI customer personas that interview you about your idea, push back, and hand down a verdict of kill, pivot or build. It was for founders who dread cold calls, which is me. The landing page promised "brutally honest feedback without picking up the phone". The plan said 1,000 paying founders and $250,000 a year by the end of 2026, with 60% of free trials converting to paid and break-even at three customers (the planning documents are in the public repo). I had checked none of it with a single real customer. A tool for validating ideas, itself never validated. The README says so now, in its first line.

What I built

The plan on the left, the record on the right. Both are in the repo.

The engine worked. By 10 February the develop branch had 62 of the 78 tasks for the first release done: sign-in, Stripe billing with pro-rated upgrades, three interview modes and a report that handed down the verdict (commit history). The part I was proudest of was the anti-sycophancy design. Every panel of personas was 40% high-sceptic, 40% medium, 20% low. A persona had to raise at least two objections before it was allowed to show interest, and if it failed to, the prompt was rerun with "you haven't challenged this enough". There were three pushback presets, the Cheerleader, the Pragmatist and the Critic, and a golden set of test ideas with a scorecard counting objections per session so I'd notice the engine going soft (architecture notes).

Every one of those is a dial. None of them is a reference point. I could make the fake customers harsher or kinder. I had no way of knowing which setting matched real customers, because real customers were the thing the product existed to avoid.

The plan had a task for announcing it. It stayed unticked. The last commit to the product was 10 February. I parked it in May, and in July I retired the whole model of selling software like this on my own, PlebTest with it.

What the studies say

Same models, same questions. The persona is the part that hurts.

Two pre-registered studies from this year put numbers on the thing I'd been tuning by feel.

The first, BehaviorLens (Meryem Sakin, 2026), gave language-model personas a household's shopping history and asked whether it would buy in each of five categories over the next 28 days. That ran across 1,456 household-periods, and every answer was checked against what the household bought. Persona panel: 52% predicted, 33% observed. Coupons: 32% predicted, 14% redeemed. The models were also asked to rank nine coupon campaigns by how many people would redeem them. A plain statistical model built on the same data matched the real order at 0.55. A language model given the structured data and no persona scored 0.13. The persona panel scored minus 0.10. The author's own caveats: nine campaigns is nine data points, and the dataset is public, so the models may have seen it in training.

The second (Alexandre Maiorano, 2026) used 399 headline A/B tests from the Upworthy archive, where real readers on real traffic picked the winner by clicking. Asking the model which headline would win, with no persona, matched the real ranking at 0.361 and picked the winner 49.2% of the time. A panel of personas scored 0.084 and 34.6%. His explanation: asked plainly, the model draws on an accurate sense of what people in general do. Told to play a particular person, it adds bias and noise.

Read those together and my product inverts. I'd built the persona layer as the feature. The studies say it's the fault.

Why the sceptic dial could not fix it

Each step after the first made the answer different. None made it checkable.

My instinct, and the instinct of every tool in this market, was that the trouble with AI customers is that they're too nice, so the fix is to make them meaner. The BehaviorLens result agrees the panel was too nice. But a dial that makes personas raise more objections does not move the panel towards the real answer. It moves it somewhere, by an amount I chose, with nothing to check it against.

The third paper says what a check would take. Tigre and Souto (2026) treat the question properly: when can you trust a synthetic consumer panel, when can you correct it, and when should you abandon it? Their correction method needs a small sample of real responses, 50 to 300 people, to calibrate the synthetic ones against (arXiv 2609.13148). So the panel is fixable in proportion to the real customer data you have to fix it with. A founder who uses an AI panel to avoid talking to 50 people has exactly the data needed to calibrate it: none.

Simile, which sells simulated users for marketing and product research, raised $200 million at a $2 billion valuation on 30 July 2026, five months after a $100 million round (TechCrunch). I don't think that contradicts the studies. A large company has real transactions to calibrate against and a research team to do it. The product I built was for the one buyer who has nothing to calibrate with. That buyer is the one the method cannot serve.

What a fake customer is still good for

Keep the first three. The fourth was the product.

I'd use one again for three things, and never for the fourth.

Finding the objection you haven't thought of. My own sample card had a persona called Sarah saying "I already use Rover. Unless you're half the price, I'm not switching." As a reminder that the question exists, that's free and instant. Whether real dog owners would say it is a separate question the persona can't answer.
Rehearsing the conversation. If you're going to talk to real people and the prospect scares you, a few rounds with a rude persona takes the edge off. That's practice, not evidence.
Asking the model plainly. The Maiorano result says the no-persona question beat the persona panel. "Would people buy this, and why not?" asked straight is the better use of the same tokens.

The fourth is the verdict. Kill, pivot or build was the product's whole promise, and it's the one output the studies say a persona panel gets wrong in a known direction. The evidence I've cited is one retailer, one headline archive and one calibration method, so it's narrow. If someone publishes a pre-registered test where persona panels beat the plain question on real purchase data, I'll change my mind and say so here.

The repo is public under an MIT licence. The engine runs. The verdict screen still says kill, pivot or build in large type. The studies say to read everything on that screen except the verdict.

Sources

Sakin, M. (2026). BehaviorLens: a pre-registered evaluation of LLM personas against held-out dunnhumby retail transactions. https://github.com/meryemsakin/BehaviorLens
Maiorano, A. C. (2026). Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation. arXiv:2609.25010. https://arxiv.org/abs/2609.25010
Tigre, R. and Gobato Souto, H. (2026). When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels. arXiv:2609.13148. https://arxiv.org/abs/2609.13148
TechCrunch, 30 July 2026. Synthetic-user startup Simile raises $200M at $2B valuation 5 months after $100M Series A. https://techcrunch.com/?p=3148353
PlebTest repository, planning documents, architecture and commit history. https://github.com/TheWayWithin/PlebTest