## Sources

1. [The PM’s Playbook for Shipping AI Features That Actually Work in Production](https://www.oreilly.com/radar/the-pms-playbook-for-shipping-ai-features-that-actually-work-in-production/)

---

### **The PM’s Playbook for Shipping AI Features That Actually Work in Production – By Gaurav Savla**

The core argument of this playbook is that the transition from a successful AI prototype to a production-ready feature is fraught with unique challenges—the **"Demo to Production Death Valley"**—and requires a level of engineering discipline distinct from traditional software development [1, 2]. While prototypes often perform perfectly in controlled settings, production environments surface issues like high latency, hallucinations on edge cases, and nondeterministic behavior that can undermine user trust [2].

**Main Arguments and Key Takeaways:**

*   **The Latency Budget:** AI features carry a **"latency tax"** that must be managed through specific budgets based on interaction type [3].
    *   **Synchronous interactions** (where users wait) should resolve in **under 1 second** [4].
    *   **Progressive interactions** (streaming) should provide the first token in **under 500 milliseconds** and finish within 5 seconds [4].
    *   **Asynchronous interactions** can take up to **20 seconds** if accompanied by a progress indicator [4].
    *   Teams must measure **p90 latency** rather than just p50, as 10% of users may otherwise face wait times exceeding 15 seconds [3].
    *   The entire pipeline—including **preprocessing, inference, and postprocessing**—must be optimized, not just the model inference itself [5].
*   **A Hierarchy of Fallbacks:** Because AI fails in creative and unpredictable ways, PMs should design a **hierarchy of fallbacks** to ensure the user experience degrades gracefully rather than breaking completely [6].
    *   **Model fallback:** Switch to a simpler, faster, and more reliable model if the primary fails [6].
    *   **Cache fallback:** Serve a previously stored response for similar queries [6].
    *   **Template fallback:** Use prewritten templates when generation fails [6].
    *   **Graceful omission:** If a high-quality response cannot be generated, the best option may be to **hide the feature entirely** [6].
*   **The Four-Layer Quality Pyramid:** Unlike binary "work/no-work" traditional software, AI quality is continuous and subjective [7].
    *   **Level 1: Safety (Nonnegotiable):** Automated classifiers must ensure 100% of outputs are free of harmful content, PII, or false facts [7].
    *   **Level 2: Factual Correctness:** Domain-specific suites evaluate if code compiles or if grammar is stylistically appropriate [8].
    *   **Level 3: Usefulness:** Measured through user-centric metrics like **acceptance rate, edit distance, and repeat usage** [8].
    *   **Level 4: Delight:** An experimental layer that captures the "gut feel" of users, which is essential for long-term adoption [9].
*   **A/B Testing Challenges:** Traditional A/B testing frameworks struggle with AI because outputs are **nondeterministic**, introducing high variance [9].
    *   AI experiments often require **three to five times the sample size** of traditional tests to reach statistical significance [10].
    *   Raw engagement metrics can be misleading; PMs should use compound metrics like **"engaged interactions where quality score exceeds threshold"** [10].
    *   Short-term tests may be skewed by **novelty bumps** or long learning curves, so longer experiment durations and Bayesian methods are recommended [11].
*   **Monitoring for Model Drift:** AI quality can rot over time due to three types of drift [12].
    *   **Data drift:** Changes in user behavior or cultural slang over time [12].
    *   **Provider drift:** Third-party API changes (e.g., GPT-4 performance shifts); pinning model versions is the recommended mitigation [12].
    *   **Evaluation drift:** The metrics and criteria used at launch may become inadequate as usage patterns evolve [13].
*   **Evaluation Frameworks:** Effective quality control requires balancing **automated, human, and "model-as-judge"** evaluations [14, 15].
    *   **Automated evaluation** uses "golden datasets" of 500–2,000 examples for speed and velocity [14].
    *   **Human evaluation** remains necessary to catch nuance but is expensive ($15–$30 per example) and slow [15].
    *   **Model as judge** is a middle ground where one model evaluates another's output, which is effective but must still be validated against human judgment [15].
*   **Production-Grade Prompt Engineering:** In a production environment, prompts must be treated as **code** [16].
    *   They require **version control, parameterization** (avoiding hardcoding context), and rigorous **regression testing** against 200–500 test cases [16, 17].
    *   New prompt versions should undergo **canary deployments**, being monitored for at least 72 hours before they are considered stable [17].

**Important Details:**
*   **Cold Starts:** Requests can be **10 times slower** when a model first loads into memory, which disproportionately affects engaged users during peak bursty traffic [4].
*   **Perception of Speed:** Using **streaming** to push tokens as they are generated can make a four-second response feel much faster than one that appears all at once [5].
*   **Human Evaluator Management:** To minimize bias and fatigue, human evaluation sessions should be capped at **two hours** with rotating evaluators [15].
*   **Proactive Communication:** When AI capabilities are degraded, communicating this to the user (e.g., "AI suggestions are temporarily limited") builds more trust than silently delivering poor-quality results [16].
*   **Final Implementation Advice:** These production systems cannot be "bolted on" later; they must be built as **core infrastructure** from the start to avoid feature failure [18].