Quantitative metrics: Why they’re only half the story

Tracking quantitative metrics is crucial, but they don’t tell the whole story alone. Learn why and follow these three rules to stay on track.

Share this

Quantitative metrics.

These babies are meant to shed light on your most successful (and not-so-successful) business processes.

For example, you can see at a glance when churn rate drops and ROI spikes. Great, right?

Sort of.

It’s easy to get obsessed with these numbers. But here’s the thing: numbers alone can’t tell you the whole story, and you shouldn’t let them dictate important decisions.

In this post, we’ll cover:

  1. A quick primer on quantitative metrics
  2. The problem with quantitative metrics
  3. How you might be relying on measurable metrics a little too much
  4. Why looking at the numbers alone is risky when making business decisions
  5. How to make quantitative metrics work with other data to tell the whole story

Let’s get started.

A primer on quantitative metrics

Quantitative metrics are objective, measurable numerical data. Basically, they’re a relatively easy way to track business performance.

Revenue up? Great. Retention down? Bad.

Most companies collect more data than they know what to do with. They’ve got numbers flying around for product adoption, sales velocity, throughput on projects, and a whole lot more.

Different teams also tend to focus on or take charge of different types of metrics.

Here are some examples of quantitative metrics product teams use:

  • Return on investment (ROI) tracks which channels have the most impact on sales
  • Revenue tracks how much money your business makes
  • Monthly recurring revenue (MRR) tracks how much subscription and SaaS companies make
  • Customer lifetime value (CLV) tracks how much you earn from the average customer
  • Trial-to-paid conversion rate tracks the percentage of users who become paying customers
  • Product adoption rate tracks the number of people who use a specific feature
  • Net Promoter Score (NPS) tracks customer loyalty
  • Customer satisfaction (CSAT) score tracks customer or user happiness levels
  • Customer acquisition cost (CAC) tracks the sales and marketing efforts you’ve spent on a new customer

The metrics you choose usually correlate to an objective.

Let’s say you want to increase usage of a new feature on your artwork app. Artists can now use augmented reality (AR) anchors to keep projected tracing images steadier on the canvas.

The objective: Increase new AR mode adoption.

The metric: Feature usage rate (e.g. the number of active users who click on the feature and spend at least 60 seconds there).

The strategy: You send out an email newsletter highlighting how cool AR mode is, plus a video on how to use it.

A lot of marketing’s blood, sweat, and tears went into that video. (You shed a couple yourself.) So, was it worth it?

You’ll only know if you compare AR mode’s usage rate since the email newsletter went out to your baseline (or base rate) before you sent it.

The result: After a couple of weeks, usage has increased from your 5% benchmark to 16%. Success!

The problem with quantitative metrics

Let’s look at another scenario.

What about if after those two weeks, feature usage rate hasn’t increased much at all?

People don’t care about the budget that went into developing AR mode, the testing hours, and endless disagreements over video soundtrack and email copy.

They care about the value it brings them. And in this case, they’re just not using the feature significantly more than they were before.

Quantitative metrics will eventually tell you all this has happened with a few digits. But it can’t tell you why.

And therein lies the problem.

Many teams use quantitative metrics like these for performance measurement. You collect numbers. You analyze patterns. You tear your hair out as you try to make assumptions or predictions.

You report back to your team or company leaders and use these findings to justify strategies and decision-making.

Sometimes, you decide to double down, invest more budget, and roll out a social media campaign around the feature.

But how can you be sure adoption isn’t stagnating for another reason?

Based on the numbers alone? You can’t.

And assuming too often (like this) can actually derail progress and eat into profits.

Are you over-relying on quantitative data?

Numerical data alone can be misleading.

In fact, there’s a well-known historical scenario that proves it. It’s called “The McNamara fallacy.”

Relying solely on metrics in complex situations can cause you to lose sight of the bigger picture

Source: Geckoboard

It’s named after the US Secretary of Defense who relied solely on numerical data and statistics to manage strategic decisions during the Vietnam War.

McNamara neglected anything that he couldn’t measure or run through formulas. As a result, he ignored all other variables and focused on one metric (“enemy body count”) to dictate success.

This method was not only ineffective, it was despicable.

And the tragic events that unfolded now warn against relying solely on quantitative data in any industry.

The lesson? 👉 Insights that you can’t measure are still important. 👈

These are called qualitative insights. And using specific methods, you learn the “why” behind the numbers.

Qualitative methods include:

  1. Open-ended survey questions
  2. Interviews
  3. Focus groups
  4. User testing sessions
  5. Unstructured customer feedback

For example, you might set a goal to reduce user complaints about system bugs. You want these to drop by 20% in time for the next review cycle.

To your delight, in three months you notice that complaints about bugs are down by 50%!

But all is not well.

After interviewing customers, you learn that this isn’t because you’ve been fixing bugs left, right, and center. Instead, support tickets are missing.

To make matters worse, your users are still upset about bugs. And now you haven’t responded, they’ve taken to forums and subreddits like an angry mob.

Without this explanation from your qualitative interviews, you would have missed the chance to get in front of the issue and mitigate fallout.

Used well, good ‘ole fashioned numbers can tell you what's performing and what isn't.

But in partnership with qualitative data? Quantitative metrics can give you leading indicators to fix what's broken or optimize what's working to help it work even better. When you transform metrics into a story, decision making and innovation are accelerated.

Venture Partner and Co-founder, Alvin Foo, captures it beautifully:

Top 3 quantitative metrics rules for success

Gartner predicts that by 2025, the most popular way to present and analyze analytics will be through data stories.

Follow these three rules to set your quantitative metrics up for success.

1. Choose the right metrics for your product lifecycle phase

The phase of your product depends on your current priority and where you are in the product lifecycle.

We can break this down into three stages:

  1. Discovery: Creating methodologies based on what matters to customers
  2. Delivery: Planning strategies and producing valuable solutions
  3. Scaling: Choosing what to scale, pivot, or drop based on what you deliver

Some also use venture capitalist Dave McClure’s AARRR framework to determine their stage;

  1. Acquisition: Tracking the marketing strategy and effort it takes to recruit potential customers
  2. Activation: Tracking “aha moments” when customers realize the value of your product
  3. Retention: Tracking how customers engage with your product and where you can improve it to keep them on board
  4. Revenue: Tracking how all of these actions impact your bottom line
  5. Referral: Tracking who tells others about your company and why

Why are these called “pirate metrics?” Because they AARRR 🏴‍☠️

New business or startup? You might track metrics like customer acquisition cost (CAC) or product adoption rate.

Need to know why you’re losing customers? You could be in the Discovery, Retention, or Referral phases. Here, your performance measures may be retention rate or Net Promoter Score (NPS).

You can also use these phases to determine your unique product metrics.

Let’s say you run a monthly wine subscription service.

You notice that people who fill out your wine preference quiz order their first box. So, one of your unique activation metrics will be the number of people who complete the quiz before signing up.

You also see that those who buy at least three consecutive monthly boxes of wine become long-term customers. So, one of your unique retention metrics will be the number of customers who buy three boxes in a row.

The list of performance metrics you choose to measure can get longer and longer. And even seasoned product managers can make the mistake of trying to track too many at once.

So, stick with those that relate to your current phase and leave the rest for another day.

2. Ask questions to fill out that data

A tale as old as time: important discoveries come from asking interesting questions.

Take Netflix as an example. The product team might look at the 30% who didn’t finish the first season of “Stranger Things” and try to pinpoint why.

The team may ask questions like:

  • “How long did they watch before giving up?”
  • “Which episode was the common drop-off point?”
  • “What did they watch after?”

Each answer gives them quantitative data they can supplement with qualitative insights.

So, ask questions about your own key metrics.

For example, your NPS score may be low. This raises the question, “Why are customers reluctant to recommend our product?”

This query can lead you to conduct customer success surveys and track CSAT metrics. And these insights can lead you to more relevant product metrics.

Here are some more examples:

Once you discover insights to these questions, ask more.

It’s a simple process. But many teams miss out due to a lack of resources or time. Or, they simply trust that their KPIs or key metrics alone contain the truth.

Don’t let the same happen to you.

Your product should always be in a constant cycle of discovery, iteration and optimization. Because these golden nuggets of insights can completely shift your focus to a transformative outcome.

3. Present them intuitively to make collaboration fast and flexible

Now, you need to communicate and present your data in way your team can easily digest and understand. Or, perhaps relay it to other team members or stakeholders for their input.

Sure, you could use a mix of Excel and PowerPoint or send an email with a few stats. But why not use an interactive document instead?

Decipad is unique because it allows you to communicate numbers, data and metrics with context.

Here’s what a product update could look like in Decipad: 

Learn how to forget spreadsheets and master product storytelling in our guide to product reporting.

And, here is what an investment case could look like prospective:

This document breaks down a discounted cash flow valuation of Apple (AAPL):

These data-first memos, reports and models are simple to build, adjust, and understand. (Even if you’re not a math or technical whizz.)

Decipad is power by a natural language so you can present data in human terms.

For example, “Market Value” = Enterprise Value - Net Debt

You can also connect directly to your data sources to ensure accurate and real-time calculations. No more scrambling to update your report with the latest data before your big presentation!

You can customize each of our templates when creating your own notebook. Or start from scratch and use Decipad in a way no one else has yet.

Either way, let it supplement fast, frictionless decision-making.

Root your quantitative metrics in context

By incorporating both quantitative and qualitative metrics, you can accelerate understanding and the pace of progress.

That’s why we built Decipad.

Any team member, customer, or stakeholder can easily explore the relationship between your numbers and their real business context.

And the faster you get permission for your new strategy, pitch or proposal the faster you can implement it and optimize your customer experience.

That way. everyone stays engaged, and better outcomes happen.

Related reads