Zillow's AI ROI Secret: Why You Must Measure Before Building AI Systems

Zillow's AI ROI Secret: Why Measurement Must Come First

At VB Transform 2026, Zillow's SVP of Engineering Toby Roberts shared a counterintuitive truth that challenges how most organizations approach artificial intelligence implementation: AI ROI numbers only hold up if you measure before you build. This insight, delivered alongside insights from Glean's leadership, reveals a fundamental gap in how enterprises validate their AI investments.

The Real Estate AI Challenge: Context Across Conversations

Zillow's case study illustrates why one-size-fits-all AI solutions fail in complex business environments. Real estate transactions aren't single interactions—they're extended customer journeys spanning months or even years. A homebuyer might start with a phone screening, transition to conversations with a loan officer, and eventually work with a real estate agent. Each touchpoint requires understanding the complete context of previous interactions.

This fragmented journey creates the fundamental problem that simple chatbots cannot solve. A single chatbot could never carry that thread of context across multiple channels, team members, and time periods. This is where sophisticated AI architecture becomes essential—and where most organizations fail to measure effectiveness.

Why Traditional AI Implementation Fails

The conventional approach to deploying AI tools follows this pattern:

• Leadership identifies a business problem
• Teams select an AI solution
• Implementation begins
• ROI measurement happens afterward (if at all)

This backwards approach explains why so many AI projects disappoint stakeholders. Without baseline metrics established before deployment, teams cannot accurately measure whether AI actually improved efficiency, reduced costs, or enhanced customer satisfaction. They're essentially flying blind.

The Pre-Build Measurement Framework

Roberts' recommendation inverts this process. Organizations should establish their measurement framework before building or deploying any AI system. This means:

• Identifying specific, quantifiable metrics relevant to your use case
• Recording baseline performance without AI intervention
• Establishing clear success thresholds
• Planning how to measure ongoing performance post-deployment

For Zillow's case, this might mean measuring how frequently context is lost between customer touchpoints, how long resolution times take without unified AI systems, or customer satisfaction scores across different interaction channels. Only with these baseline numbers can teams accurately measure whether their AI architecture actually delivers value.

What This Means for AI Tool Users

This insight has profound implications for anyone evaluating or implementing AI tools:

• Ask vendors for pre-implementation metrics: If a tool provider can't help you establish baseline measurements, be skeptical about their ROI claims
• Demand transparency: Any AI tool worth its cost should come with a framework for measuring its impact on your specific workflows
• Plan for measurement: Budget time and resources to document baseline performance before deployment
• Build context awareness: For complex customer journeys, ensure your AI system architecture can maintain context across multiple interactions and channels

The Broader AI Landscape Shift

This conversation signals a maturation in how enterprises approach AI adoption. The days of deploying tools based on hype are ending. Instead, organizations are demanding rigorous, evidence-based approaches to AI implementation. Tools that can't prove their value through measurable outcomes will struggle to gain enterprise adoption.

The engineering leaders at companies like Zillow are setting a new standard: show me the data before, during, and after. This shift from hope-based to data-driven AI implementation benefits everyone in the AI tools ecosystem by raising accountability and ensuring investments actually deliver results.

The Takeaway

Whether you're implementing enterprise AI tools or evaluating new platforms, remember Zillow's core principle: measure before you build. This simple discipline transforms AI from a speculative investment into a strategic business tool with proven, measurable returns. The organizations that adopt this approach first will establish themselves as leaders in responsible, ROI-focused AI adoption.