0 com

Measuring meaningful metrics

All metrics are shortcuts. When we're faced with uncertainty, we use metrics to break our problem down into simpler, tangible pieces that we can understand.

Metrics are simple proxies that allow us to transform difficult questions into empirical, demonstrable ones.

When we're faced with a difficult question, we often answer an easier one instead, usually without even noticing this subtle substitution. This is what economists call an availability heuristic — a mental shortcut where we use what we already know, rather than complete information, when making a decision.

History is filled with examples of people who used a similar approach to solve complex problems. One of the best examples of this is the story of the map of the London Underground.

The London Underground officially opened in 1863. Despite being such a revolutionary way for moving people around the city, the Underground Electric Railways Company still had to convince people to actually use it.



The map was a perfect representation of the city. It was precise and detailed, and different colors were used to indicate the different lines, with circles to highlight the various stops.

However, Londoners found it too complex and confusing.

Over the next 10 years, a variety of geographical maps were produced. They changed sizes, lines widths, colors, fonts, legends over and over again.



In, 1931 the operators took a chance on a new, "radical" map designed by a former Underground Electric Railways Company draughtsman and printed a few copies. His name was Harry Beck and this was his map.



Beck's Tube Map

Beck came up with a completely new design with:


  1. Only 45- and 90-degree angles.
  2. No geographical references: No streets, no parks, no churches.
  3. Stops were represented with equal distances.
Beck's map was completely different from the actual world, which has all kinds of angles, streets, parks and churches, and where distances matter. While in theory, he introduced a few misleading rules, in practice he removed the clutter, he dramatically reduced the signal to noise ratio and he made the underground more accessible and understandable to humans. Quite simply, it worked.

Today, the London Underground map is by far the most famous map in the world. Years later, it became the standard for every underground in the world.

The danger of me-too thinking and the Survivorship Bias

In the era of SaaS and digital products, our strongest convictions are powerfully swayed by what other businesses are doing. Their core metrics become our core metrics, their best practices become our best practices, and their benchmarks for success become ours.

This way of thinking causes many businesses to become victims of survival bias. This happens when we focus too narrowly on those who have been successful and overlook the companies and entrepreneurs who didn't quite make it. This can lead us to draw a number of false conclusions about the reason for their success.

We see this happening when businesses with different products, different value chains, and different core values level out on the same metrics and, even worse, optimize for those same metrics. These bad habits make businesses think of their products in the exact same way, and it pushes the proliferation of "me-too" strategies.

Entering a Post-Click Era

We know that the number of lines of code is a bad proxy metric for measuring the quality of the output of a software engineer. We know that the number of hours spent at the office is a bad proxy metric for productivity. And while we are fully aware that certain proxy metrics are wrong, we're often not willing to do the extra work required to go beyond standard industry practices. Here are a few examples of benchmarks that are no longer effective:

1. Pageviews

According to Wikipedia, a pageview is a request to load a single HTML file (web page) of an internet site. The web page loads some analytic tracking code and sends an event to a back-end server. This metric is thought to be a good estimation of your page's impressions (aka views).

This has worked pretty well for the last 10 years, but think about you browse the internet today. If you're like most people, you always have gmail.com open in your browser and you just switch tabs to check for new email without reloading the whole page. You probably do the same thing on Twitter, Facebook and with lots of other products.

When this behavior stops being a marginal case and becomes the way everyone uses the internet, pageviews become a misleading and unreliable indicator.

As suggested by Google engineer, Phil Walton here, instead of tracking how many times a page is loaded, we should track how many times it was viewed using the Page Visibility API.

2. Engaged users

The more users actively use your product, the more they should care about it. Right? Engagement is one of the best predictors of success. For years, this simple equation has been considered the bedrock of every successful software product.While this is still valid for a lot of consumer products, it's not valid for a wide range of SaaS products where the software works in full autonomy.

The product (1) understands the problem, (2) works out a solution and (3) outputs a result. These products don't need human interaction at any level of their value-chain and therefore, optimizing for "engagement" doesn't make sense. Engagement is not the silver bullet metric for every SaaS. Don't take this for granted. I extensively discussed this idea in a recent post: The Next Generation of SaaS Won't Optimize for User Engagement.

3. Time spent & bounce rate

High time spent is good. High bounce rate is bad. We're used to the idea of time spent as a proxy metric for "interesting content" and high bounce rate a proxy metric for "bad content".

That's so untrue. Just because people spend time on your article page doesn't necessarily mean that they were actually interested in what you wrote. For website content, the ratio of unique views versus "read" is a much better metric for success.

It's no surprise that Pete Davies and Medium rejected the idea of time spent on page and replaced it with Total Time Reading (TTR).

We measure every user interaction with every post. Most of this is done by periodically recording scroll positions. We pipe this data into our data warehouse, where offline processing aggregates the time spent reading (or our best guess of it): we infer when a reader started reading, when they paused, and when they stopped altogether. The methodology allows us to correct for periods of inactivity (such as having a post open in a different tab, walking the dog, or checking your phone).
The aggregate Total Time Reading (TTR) is a metric that helps us understand how the Medium platform is doing as a whole. We can slice that number in lots of ways (logged-in vs. logged-out, new posts vs. old, etc.).

4. Open rate and click rate for notifications

High open rates mean they want to hear from you. High click rates mean that what you sent them is relevant.

Suppose you sell online courses and you're sending triggered notifications to users who didn't complete a course. At this point, looking at open rates and click rates doesn't really matter as your ultimate goal is having users complete what they started.

The ratio of users who got the notification versus users who completed the course would be a much better metric for success.

Notifications are good when they're able to deliver real value to the user. You can't know the actual value delivered but you can select the indicators that are an actually better predictor of success.

#5. Refunds
At the end of the day, businesses optimize for increasing revenues. This leads most businesses to see refunds as an obstacle to revenue. The fewer refunds you have, the better.

When every other retailer was trying to drive down returns, Zappos chose to optimize the process, making the return not just a normal part of shopping, but a delight.

While looking at the wrong proxy metrics can distract organizations from the real benchmarks that do matter, things can get seriously worse when they optimize for those bad metrics. This is where adopting the wrong metrics (and optimizing for them) can seriously affect your growth and negatively impact your customers. British economist, Charles Goodhart explained this in the Goodhart's law:

When a measure becomes a target, it ceases to be a good measure.

As a result, teams start focusing on lifting engagement by sending nonsense notifications that users don't really care about. They use emoji in email subject-lines because they have higher open rates, they hide no-refund policies in the service terms or in FAQs their users will never read. In time, this leads to a death spiral of misplaced attention; the longer you focus and optimize irrelevant metrics, the further and further away you move from the metrics that do matter for your business.

Before you know it, you will find yourself trying to desperately justify your new content strategy, your new product decisions, your new customer support line. You will eventually realize that your business hasn't improved because you've spent too long looking at the wrong benchmarks for success.

The Need for Holistic Thinkers

When Harry Beck created his Tube map, he really did just one difficult thing: He was able to understand what users needed. He realized that a more detailed map placed a higher cognitive load on the user, making it more difficult for them to make a decision.

Instead, he created a map that didn't look like a map at all. We need people who are able to go beyond the prevailing wisdom and common standards. When they pick up their core metrics, they need to be able to think about their organizations, their products and their strategies in a holistic way.

We need people who truly understand customer behavior and who are great at optimizing for the value delivered rather than magician good at optimizing for short-term value. We need companies who truly understand that marketing, product and customer support go hand-in-hand as one big organism and who are focused on constantly improving their customer experience rather than merely lifting up industry-approved metrics.

0 com

The Next Gen SaaS won’t optimize for user engagement

A few weeks ago Hiten Shah explained in a new interesting post why the most successful SaaS companies of the future will focus on usage, just like Facebook. In the write-up, he goes very deep into his explanation bringing examples of world-class SaaS companies like Trello, Slack, and Dropbox that are all building their strategies around this consumer-oriented product approach.

He predicts that this is how the next generation of SaaS will look like.

While I was reading Hiten's post, I immediately recalled a frugal email conversation I had last month with Patrick Campbell, at Price Intelligently.

Patrick briefly introduced me to the definition of what he calls "anti-active usage" products.

While at first, this might sound very counterintuitive, it's actually the natural evolution of most of the SaaS products that we know today.

#Defining anti-active usage SaaS products
Harnessing the world of software in a single statement is very hard. Tom Tunguz explained in a post on the blog his vision about software in a simple way.

Software world divides into systems of record and workflow applications.

Systems of record unify data from different sources under a single view. Common applications of Systems of record are CRM and ERP.

Here's how the value chain for systems of record looks like:



Workflow applications enable workers to do work. These products represent a huge portion of the products that we use in our daily work life.

Here's the value chain for workflow applications:



Systems of records and workflow applications have one thing in common, at their core level they need some human interactions.

The paradigm under which you have to actively use something to do a given task or to reach a certain goal is the bedrock of most of the SaaS products out there.

Anti-active usage products flip this model — you don't necessarily need to use the product to get something done because the product (1) understands the problem, (2) works out a solution and (3) outputs a result. Anti-active usage products don't need human interactions at any level of their value-chain.

We can expect in 10 years from now, a good part of today's SaaS product flocking to this new category:



There are many reasons I see why anti-active SaaS products might come in the next 10 years:

1. The scarcity of time

We can build a solid business strategy around things that are stable in time. This is why we create businesses on things that don't change. Time is the scarcest resource and your employees' time is one of the most important assets of your business. Products that don't impact by any means on your team but yet they are able to generate relevant outcomes, can change the rules of the game.

2. The tragedy of the commons

The tragedy of the commons is an economic theory of a situation within a shared resource system where individual users acting independently according to their own self-interest behave contrary to the common good of all users by depleting or spoiling that resource through their collective action. The "individual users" in this instance are the SaaS vendors who are all trying to optimize for engagement/product usage. The reliance on this model is not only unsustainable but is demonstrably damaging the environment.

3. SaaS switching cost

The tool explosion I've been talking about for a while made companies more flexible, but it also made people waste time jumping back and forth between apps just to accomplish a given task. Not to mention the lack of context they need to do good work. This SaaS tools explosion broke your work into tiny pieces and scattered it across a dozen apps — making it almost impossible to feel on top of things. Anti-active usage products slash the switching costs between products and tools in our workday and centralize the metrics of their output in a single dashboard. These products allow you and your team to concentrate more on a strategic and on a tactic level.

4. AI as core product value

What we're going through right now, is a discovery moment where companies use Artificial Intelligence to optimize their core value to serve better customer experience. Amazon, Google, Netflix are all doing this, but AI is not the core value of their products. Amazon is still an e-commerce store, Netflix is still a video entertainment company, and Google is still a media company. We can expect AI from being an attachment that optimizes the core value, the ultimate core value.

Let's clarify a few areas where products can be more suitable for this transition.

We can expect anti-active usage products mainly falling into these three categories:

Automated products — anticipate fraudulent payment attacks before they happen, schedule meetings by email
Operational products — flag errors in legal documents, buy/sell stocks, make a cross-sell offer
Single decision taker — give a discount or approve a credit
On the other side, products that require a deep level of human interactions, or empathy are probably still not ready for this transition. Same for all those workflow applications that require some tactical or strategic thinking when using them.

Let's dig in a bit and take the real example of Social Media Scheduler SaaS product:

Traditional workflow for an active user:


  1. Each Monday your user logs in →
  2. She enters texts and links for the next week →
  3. She sets a specific time window (according to data, previous experience, best practices etc..) for the posts to go out →
  4. She logs in the app to perform additional tweaks on the fly (on average, let's say 10 to 20 times in any given week) →
  5. She ultimately logs in to monitor KPIs and make sure he's gaining results →

In your ideal world, this what happens. Each Monday your user logs in, she performs some actions in the product, she periodically makes some tweaks and she keeps her eyes on some of the metrics your product pulled out. The more users you have like this, the better.

On the contrary, when your sales and your revenue outrun usage and adoption something bad is happening. Because you know that when customers realize that they are paying for something they are not using, it creates a perfect storm for churn.

With Anti-Active Usage products, this model doesn't work. They don't require your users to be engaged at any level with the product.

This is how a possible workflow would look like:


  1. → The product automatically collects an interesting piece of content that's worth sharing according to your preference
  2. → The product schedules those posts in a time window using previous data, audience location, best practices, etc..
  3. → The product provides an API that displays on a third-party business intelligence product all the outcomes (engagement level, new conversion our audience reach out, etc..)

The end-user doesn't need to do anything in the product. The SaaS doesn't need the intervention of the user during all its value chain. The product is able to work in complete autonomy and yet, it's able to deliver outcomes.

Beyond Product Usage

At its core level every product wants to be able to find an answer to the following question:

Am I relevant to my users?

Pretty much every week I see new articles that try to give their interpretation on the topic. One of the good read I saw has been written last week by Josh Elman: The Only Metric That Matters.

Long story short, we answered the question "Do my users care about my product?" with another answer "Are people using your product?".

The vast majority of SaaS products relied for years on the idea that if your users keep coming back to your product to perform a certain action, then they are finding something valuable in what you offer.

This simple equation has been considered for years the bedrock of every successful software product.

Engagement is one of the best predictors of success.

On the other side, if your users are not coming back to your product, they land .. and leave, then you have some growth problem. You user don't stick, and if your users aren't sticky, churn will be high, and you won't have engagement.

In reality, when we try to find to answer the question "Do my users care about my product?" we are trying to measure something that we can't know with certainty.

So we pick up a metric to approximate the actual underlying user behavior. This is what Sriram Krishnan defines in an interesting essay a proxy metric.

The process we ran into when we defined product-usage as the best proxy metric is called reification and it's one of the principles of the well-known Gestalt effect.

Reification is the constructive or generative aspect of perception, by which the experienced percept contains more explicit spatial information than the sensory stimulus on which it is based.



For instance, you perceived a triangle in picture A, though no triangle is there. In pictures B and D the eye recognizes disparate shapes as "belonging" to a single shape, in C a complete three-dimensional sphere is seen, where in actuality no such thing is drawn.

Reification allows you to infer the abstract (unknown) by treating the illusionary contours (known) of a visual system as "real" contours.



We do this natural approximation all the time when we pick up a metric — no exception when we are trying to infer if our users find our product valuable by looking at their usage.



Anti-active usage products break this rule and revert the paradigm by pushing us to think way beyond the idea of retention in terms of product engagement.

When your product works in full autonomy and yet, it's able to deliver outcomes, user engagement is not what you want to optimize for. So while usage and engagement are all good proxy metrics for most of the consumer-facing products, for anti-active usage SaaS products, these are fallacious metrics. In these cases finding the right proxy-metrics is harder, often very counterintuitive and it requires you to think in a holistic way.

Conclusions

In the next 10 years, a lot of SaaS products will flock from traditional systems of records and workflow applications to the anti-active usage products category. A relevant portion of today's SaaS companies might not even have the log-in button and for those products, usage and engagement might not be the only metrics that matter.