AI Features That Earn Retention, Not Just Demos

We love AI where it removes a real step a user would otherwise struggle with.

We defer AI where it adds a demo but not a reason to return.

Does this AI help a stranger finish the job — or just make the demo longer? The answer shows up in one number: completion rate for the core job, measured before and after the AI ships.

AI DECISION — DOES IT EARN RETENTION?

We ship AI only when it improves completion or removes friction we can measure.

✕ Hype trap

AI demo that adds steps. Looks clever, completion drops.

Defer

✓ Earns retention

AI removes a real step or error. Measured completion ↑.

Ship

— No change

AI that doesn’t change the job. Adds cost, not value.

Skip

⚠ Risky

AI helps some, hurts others. Needs instrumented test.

Test first

high friction ↑low retention → high retention

We instrument completion rate before/after. If it doesn’t move, the AI doesn’t ship.

Answer first: does this AI change whether the job gets finished?

If an AI feature shortens the path to a clear outcome — fewer fields, fewer decisions, fewer errors — it has an honest slot in an MVP.

If it adds a generation step the user must check, edit, and re-do, it is not an MVP feature. It is a follow-up iteration that needs its own instrumented test.

We learned this the same way we learned to cut scope in our MVP checklist: viability is trust, and trust is measured by whether a stranger can finish without a sales call.

The decision matrix we use

Ask two questions:

1. Does it increase the rate at which strangers complete the core job?
2. Does it increase friction, cost, or the need to double-check?

	Low friction	High friction
Increases completion	Ship — AI removes a real step. Measure before/after.	Test first — help for some, hurt for others. Needs instrumented slice.
No increase	Skip — AI doesn’t change the job. Adds cost, not value.	Hype trap — defer — demo that adds steps and lowers completion.

We put every proposed AI feature in one quadrant before we estimate it. The quadrant decides the roadmap, not the demo schedule.

When AI earns a slot — concrete earns

AI earns a slot when one of these is true and measurable:

• A field users often get wrong is auto-filled inside a typed boundary — so mistakes don’t become dispute threads. Example: parsing an invoice total into balanceInCents with explicit pending vs reconciled, as we do in Ankik.
• A step that takes minutes drops to seconds without adding a new decision — so more strangers finish without help. Example: suggesting the next status update for a field visit — the user taps accept or edits, not composes from scratch.
• Error handling becomes obvious — so the user knows what is safe to retry without calling support. Example: explaining why a save failed in plain language tied to retry safety, as shown in failure states.

In each case we measure: did more strangers finish the job without help after the AI shipped? If yes, we keep it. If not, we remove it — even if the demo was liked in a review.

When AI is hype

AI is hype when:

• It generates a draft the user must carefully verify every time, adding a second job (checking the draft) to the first.
• It creates content for a job that wasn’t asked for — a summary page when the user actually needed a status update they can trust.
• It is added because “every MVP needs AI” rather than because a specific friction was measured.

We prefer what we call boring structure first: explicit flows, typed boundaries, and the five-state checklist. AI is a material inside that structure, not the product.

For a practical map of hype vs correct trade-offs, read cheap vs correct AI integration.

How we test AI in a Build Sprint

We treat an AI slice like any other hypothesis in how we ship in 28 days:

1. Define the job and the friction in one sentence.
2. Ship the non-AI path that already lets a stranger finish.
3. Add the AI as a flagged variant on the same path.
4. Measure completion rate and time-to-complete for both variants.
5. Keep the AI only if the variant wins on real use.

The cost of an AI feature is not just the model bill. It is the support load when the generation is wrong in a way the user cannot spot quickly. That is why we pair AI with careful pricing of scope — a cheap model call that creates a support queue is not cheap.

Scoping an MVP where AI might help? Put Each AI Idea Through the Ship / Skip Matrix — we instrument the winner so you keep what moves completion rate, not what demoed well.