Why it’s difficult to select a single north star product metric.
Thousands of words have been spilled discussing metrics, but what actually makes understanding, decomposing, and selecting a product metric difficult?
There are probably thousands of articles discussing goal setting and product metrics. For many product managers, this proliferation of content has created the opposite effect: confusion.
Exhibit 1: Case in point
VS
Now, I’m not going to try to resolve this debate. But I do want to elaborate on why there are lots of differing opinions when it comes to selecting metrics. Let’s understand why selecting product metrics is hard.
Problem 1: Do you understand the difference between a goal and a metric?
The misunderstanding between goals and metrics is so common. It’s easy to confuse the two because they are intertwined. But if you don’t understand the difference between a goal and a metric, you’ll almost certainly have a difficult time selecting a product metric.
Goal definition: A goal is a desired outcome, objective, vision, or destination. It can be general or specific.
Metric definition: A metric is an quantifable measurement. In our usage, measuring things can help us determine if we’re close to reaching our goal.
In summary:
Ø A goal tell us where we WANT to end up. We may not reach our goal, but we have a desired destination.
Ø We then measure with a metrics if we’re getting close or not.
Brainstorm metrics from a goal using an analogy
I’ll use a simple product: me. As a human, I have a general goal of living a healthy life. So, I ask myself; how do I know if I’ve reached this desired outcome? Answering my own question, I brainstorm a few possible signs such as:
Stress free
Active life
Loving family and friends
Healthy diet
Here, my list is essentially different ways I might categorize measurements that’ll help me determine if I’m “living a healthy lifestyle”. They also hint at different strategies I could execute to reach my goal. Let’s suppose I decide to select “Healthy diet”. How would I specifically measure if my diet is healthy? Perhaps I could count my caloric intake, a specific metric.
Problem 2: When we don’t have a goal and use a metric as the goal.
In my prior analogy, I started with a goal: living a healthy lifestyle. Yet all too often, we don’t know where we want to go. In real-life, unplanned trips can be fun experiences. This happens with products too, aka no-objective hackathons. Yet not having a goal when you’re tasked with selecting a product metric often results in the metric becoming the goal. The problem with this is you might end up at a destination you don’t like. I hope you’re lucky.
This issue is commonly experienced by internal, non-revenue generating products (e.g., a knowledge repository software). Partly, this is because internal products are heavily influenced by internal users and select stakeholders who hold power. Partly it’s also because internal, non-revenue generating products are often managed by inexperienced product managers because it’s less complicated and the stakes for mistakes are lower for the company’s survival. Without a clear goal of where to take the product, PMs for such products often gravitate towards popular metrics such as active users espoused by more popular, customer-facing products. But does it make sense to increase active users? Is the end destination of having everyone at the company using the tool once a day the desired outcome?
Problem 3: When the metric causes us to deviate from our goal unnoticed
Even if you start with a goal instead of a metric, you can still run a problem if you don’t meticulously document your thinking. Let’s continue with my living a healthy life goal.
Having selected caloric intake as a metric, I decide to set a daily target of 1700 calories. This leads me to think about losing weight. Losing or maintaining a certain weight probably would make me healthier. Perhaps I should target a specific weight. Suddenly, without realizing it, I’m already thinking about setting a weight target of 150lbs/68kg. And in that train of thought, I have mentally shifted my goal: “living a healthy life” (goal) > caloric intake (metric) > “lose weight” (new goal) > pounds/kilograms (new metric).
This interplay between thinking about goals and metrics, which causes us to rethink our goals is what makes goals and metrics confusing. It occurs so readily, especially when we are just thinking sequentially. And if you glance up, recall that a “healthy diet” is only one approach to “living a healthy life”. We might have gone down an entire rabbit hole and not considered the other ideas (e.g., stress-free living). Here, I can’t stress enough the importance of writing down our thinking about goals and metrics, including how the metrics are connected to the goals. I’ve recently migrated away from written documentation to tree diagrams as the first step, but I’ll have to work on another post to explain. Meanwhile, this Reforge article’s point about breaking down metrics into drives is what I’m doing.
Problem 4: Not treating general versus specific goals differently.
The fourth reason I believe selecting product metrics is confusing is because we can write our goals either too general or too specific. Both create issues when it comes to identifying metrics to measure, but in different ways.
Again, back to my goal of “live a healthy life”. Because the goal I picked is general, there were multiple paths to reach our destination. Hence “healthy diet”, “loving family and friends”, etc. This issue exists if you pick a general goal for your product such as “grow revenue”. You might identify multiple different paths or drivers such as:
grow users
retain more users
upsell users
create new product
The problem isn’t that there are multiple paths when we pick a general goal. The problem is people want to believe and argue there is only one correct path: “the most optimal path” (i.e., best). This “pick one”, prioritizing the correct path, leads to a lot of unnecessary animosity and contention among people, which turns selecting metric into picking which group of people (i.e, tribe) you want to join.
When in reality, there is no “absolute” best and all the best are only considered best in hindsight. This is why it’s important for general goals to consider all metrics and not argue you’ve picked the “best” north star metric, but the metric for this moment in the company and product’s lifecycle. If you’ve considered all metrics, you can keep the goal “living a healthy life”, but shift the metric (from healthy diet via caloric intake to stress free via heart rate variability) things change. Again, I’ll have to write another post that’ll help determine when and how to shift metrics, but keep your goal.
On the other hand, a goal can be too specific. For example, I could have written that my goal is to “live a healthy life by going out with a friend for dinner once per week.” Here, the goal includes the metric “number of dinners with friends” and includes a target of “once per week”. While this might seem like a good, clear goal, it presents another problem. Recall that we brainstormed different ideas on how to reach “living a healthy life.” Yet this goal has a specific metric. When goals include specific metrics, it means that if the wrong metric is selected when writing the goal, people often conclude the goal is wrong or bad if the metric is wrong because of association. What’s worse, because the goal includes a metric, it shuts down our thinking about alternative metrics. Thus, if the metric “number of dinners with friends” is wrong, we don’t easily have an alternative.
In summary: picking metrics is hard because:
we confuse goals and metrics,
without a goal, we gravitate to metrics as a goal substitute,
we get lost when iterating on goals and metrics and documenting thinking on this topic is difficult, and
we confuse general versus specific goals and don’t treat them differently
Like this topic? Forward to a friend or colleague. Have a thought? Leave a comment
Additional Reading
The Difference Between Goals, Strategies, Metrics, OKRs, KPIs, and KRIs (I could nitpick a few parts I might slightly disagree, but a good quick reference if you’re ever confused).