From Metrics to Meaning: AI's ROI

Last year, it was reported that 95% of GenAI pilots fail. This doesn’t come as a surprise given that every wave of innovative technology is met by the same response. Organizations rush to adopt new tools for fear of getting left behind. But in their desperation to unlock potential, these initiatives rarely evolve beyond the experimentation phase.

From AR to voice assistants, and now to GenAI, organizations are quick to invest their faith (and resources) before defining where the technology actually creates value, and then rely on surface-level metrics to prove it’s working.

Businesses need to first think about how they’re measuring ROI, because only then can they make sure they’re part of the 5% that have something to measure in the first place.

From Reactive to Proactive

Most businesses evaluate AI pilots based on usage and engagement rates. These metrics are easy to track, but don’t equate to value or show purchase behavior. The results are also often underwhelming, causing companies to abandon projects prematurely. 

To really create and measure ROI, businesses need to go beyond generic use cases and deploy AI where it addresses specific friction points. For example, checkout flows. Customers facing card failures or complicated purchases are at high risk of abandonment and often require additional support. 

Apply has been working with several retail clients to design deeply integrated AI experiences that reimagine the entire shopping flow. For one client who offers complex, higher-value orders, we’re developing an AI solution that proactively assembles a cart with everything the customer needs to complete their project. For another, we’re creating an AI assistant that offers detailed advice and context-aware product recommendations for highly specific scenarios.  

These AI-driven solutions also align with evolving search behavior. With over 50% of Google results now presented as AI-generated summaries, users increasingly expect concise answers rather than traditional search lists. 

Does AI Measure Up?

One of the most effective ways to evaluate whether an AI experience is producing meaningful results is to A/B test the actual business impact — an approach that is no different to the way teams traditionally review new features or strategies. Teams can then compare results from the new AI-driven experience to those from the standard search page, focusing on conversion rates, not just usage or dwell time.

Moreover, businesses must include AI higher up in the funnel, integrating into search and browsing functions, rather than just tacking it on at the end with a chatbot that’s all too easy to ignore in the corner.

Here are some questions to evaluate the effectiveness of an AI tool:

1. Is it embedded into search discovery?

2. Does it provide proactive advice, not just reactive support?

3. Is it connected to the purchase journey?

4. Does it remove friction, or does it add additional steps?

It’s Sink or Swim Time

AI offers huge potential for businesses. But if businesses want to see real ROI, they need to give AI the chance to prove its value.

They must be brave enough to put AI in the deep end, embedding it at high-volume friction points. Keeping AI in the shallow end, such as with basic chatbots, demonstrates the technology works, but it doesn’t put anything meaningful at stake or provide real proof of value.

True success comes when AI is measured in critical workflows, connected to outcomes that matter. Only by doing this can organizations move beyond surface-level wins and see the tangible returns that set the top 5% apart.

Want to go deeper on measuring what actually matters?

The ACx Report explores how agentic intelligence is reshaping the way the world's best brands connect, convert, and compete — including a full chapter on the new metrics that replace vanity tracking with outcomes that drive real business value.