Atiq Israk has taken AI features from demo to production on Navbot (85% inquiry automation), AssetIQ (264% revenue growth), and enterprise systems at Toyota/Navana ($40M saved). This guide covers eval gates that prevent demo-grade AI from reaching customers.

Key Takeaways

• Demos optimize for the happy path; production requires pass/fail thresholds on a labeled set before any customer sees output.
• Eval gates should block release, not just inform it—if accuracy on golden examples is below threshold, you fix scope or retrieval first.
• Post-launch, re-run the golden set on every model or prompt change to catch quality drift.

Why Do AI Demos Lie?

Demos cherry-pick inputs, use short context windows, and hide latency. Production hits messy PDFs, noisy scans, and users who type in Bangla-English mix. The gap between "works in the room" and "works at 2 a.m. for a franchise operator" is where reputations die.

Eval gates exist to make that gap visible before launch—not in a post-mortem.

What Is an Eval Gate?

An eval gate is a ship/no-ship checkpoint tied to measurable quality on a fixed test set. Minimum bar:

1. Golden set — 50+ real inputs labeled with expected outcomes.
2. Pass rate threshold — e.g., 92% accuracy on intent classification, <2% unsafe outputs.
3. Cost/latency ceiling — p95 under X ms, cost per success under Y.
4. Fallback defined — what happens when the model fails (human handoff, cached answer, graceful degrade).

This extends the framework in evals as the new PRD from document to release process.

How Did Navbot Use Eval Gates?

Navbot targeted restaurant inquiry automation. We did not launch "general chat." We launched three high-frequency intents with a golden set built from real tickets. Until response quality cleared the rubric on those intents, we kept human review in the loop. Result: 85% automation without the brand-risk of wrong answers on reservations or hours.

How Do Eval Gates Connect to Revenue?

Every gate should trace to a business number from Find the Money:

• AssetIQ: stock accuracy eval → fewer stockouts → revenue.
• Navbot: resolution eval → handle time → labor cost.
• Enterprise workflow AI: error rate eval → rework hours → $40M-scale savings on Toyota systems work.

If the eval passes but the metric does not move, your labels or scope are wrong—not the gate.

What Tools Run Eval Gates?

Start simple: spreadsheet + script + CI job. Scale to Braintrust, LangSmith, or custom harness in your repo. The tool matters less than the habit: no merge to production without golden-set results attached to the PR.

How Do You Roll Out After the Gate?

Phased exposure beats big bang:

1. Internal dogfood with eval logging.
2. 5% traffic with automatic rollback on eval regression.
3. Full rollout when business KPI and eval bar hold for two weeks.

Explore: case studies · frameworks · scope AI for revenue