We Built the Validation Engine I Wish I'd Had

I've been building PlebTest in public for the past few weeks, and I just shipped something that fundamentally changes how founders validate ideas: a complete AI-powered validation system that actually pushes back on you.

Here's what we shipped since our last update.

The Core Loop: From Idea to Verdict

The heart of PlebTest is now complete. You start by describing your startup idea — just a few sentences about what you want to build. Our AI generates a structured proposal with the problem statement, solution hypothesis, and key assumptions that need testing.

Then you define your ICP (Ideal Customer Profile). Who are you building for? What are their pain points? What psychographic traits matter? This feeds into persona generation, where AI creates realistic customer personalities — but here's the twist: they're skewed toward skeptics. 40% high skepticism, 40% medium, 20% low. Because cheerleaders won't help you learn what's actually broken.

Anti-Sycophancy: The Feature That Makes This Work

LLMs want to be helpful. Too helpful. They'll agree with anything if you don't constrain them properly.

We built an anti-sycophancy system with three calibrated presets:

• Cheerleader: Supportive but still raises 2 real objections
• Pragmatist: Balanced, focused on execution reality
• Critic: Actively looks for flaws, requires 3+ objections minimum

This isn't just prompt engineering. We run a regression test suite with 12 golden cases to ensure personas stay honest. If they start getting too nice, we catch it.

Two Ways to Test: Interactive or Spectator

Interactive Mode lets you chat directly with AI personas in real-time. They push back using Mom Test principles — asking about past behavior, not hypothetical futures. The responses stream token-by-token, so it feels like a real conversation.

Spectator Mode is for when you want to watch a skilled interviewer do it for you. An AI interviewer conducts the conversation while you observe. Sometimes the best way to learn is to see how someone else probes for real signals.

From Sessions to Signals

After each conversation, our AI analyzes the transcript and extracts key signals: did the need get validated, did the solution resonate, what were the real objections, and any commitment signals.

When all your sessions finish, you get a verdict: Kill, Pivot, or Build. The scoring is weighted — need validation (35%), solution resonance (30%), commitment signals (25%), and anti-sycophancy quality (10%). You also get a confidence level based on sample size, agreement rate, and conversation quality.

The report aggregates everything: problem validation summary, solution validation summary, top objections, strongest signals, and actionable next steps. Toggle it public and share it with anyone.

The Infrastructure That Makes It Reliable

Tests run asynchronously using a background job system so your browser doesn't hang. Every API route has rate limiting. Every session has a token budget to control costs. We use SSE for real-time streaming so conversations feel natural.

And we built Quick Fire right into the landing page — get an instant risk score for any idea, no signup required. Try before you commit.

What's Next

This is the validation engine I wish I'd had before wasting months on ideas nobody wanted. It won't replace talking to real customers — nothing does — but it'll help you figure out which ideas are worth that effort.

If you're building something and dread cold calls as much as I do, join the waitlist at plebtest.com. We're letting people in soon.

— Jamie