How to Measure and Improve Activation

The following is an excerpt from our eBook, “The Heap Guide to Activation.”

It’s the second in a series of posts that discuss why activation is a critical metric for product teams and what hypothesis-driven product teams can do to measure and increase activation. Read part one here.

In part one of our blog series, we explored why activation matters and the role it plays in your growth metrics. In part two, we’re going to discuss how you can measure and improve activation.

The right way to measure activation

Before you can improve activation, you have to first understand how you should be measuring it. We’ve found that many product teams start by measuring activation one way, only to learn later on that the measurement they’ve been using doesn’t quite do what they want it to.

For example, many product teams measure activation by looking at a report or dashboard similar to the graph below.

In this case, we’re looking at a simple graph of users who had a visit (also referred to as a session) within the past month. (These users are also known as monthly active users, or “MAU.”) Product teams might also look at daily or weekly visits to measure these users as well.

A recent Heap poll found that over 80% of respondents measure activation this way. However, the problem with this metric is that visits don’t actually signify that your customer is getting value out of your product.

For example, think about the following scenario for a gaming app. In an effort to drive more daily visits, your gaming app institutes a daily rewards system. Because of this, users start coming in every day to redeem their treasure chest, coins, or extra points — and cause daily activation charts like the one previously shown to spike.

At first, everyone on your team is happy and high-fiving. But there’s a problem: these users are coming just to collect their prize. They don’t even play the game. What’s worse is that after a few weeks, they stop coming back.

So, the spikes that you saw? They didn’t actually do anything for the business in the end. This is just one example, but it does highlight the problem of equating visits with value.

Another example: consider a SaaS tool whose users log in everyday because they have to, but they hate the experience. This might be okay for a while, but as that contract comes up for renewal, you could be in for a surprise when the account decides to churn.

If there are flaws in measuring activation this way, why do so many people do it? To a certain extent, because this is how it’s always been done — this is what legacy tools have taught us to do. When all you have is page view and session data, it makes sense that we would look at these and assume they captured value.

Let’s explore a better approach to measuring activation.

The role of behavioral data in activation

For decades, companies have segmented their customers using demographic data. Though details change, these tend to follow a familiar formula. Men over 5’10” who wear Asics tennis shoes and drink domestic beer tend to vote for a certain political candidate. Single women between 30-35 who went to Northeastern colleges and currently have no children prefer their kombucha in a clear bottle. Engineers at companies with 3000+ people prefer a specific SaaS product.

Certainly, knowing who is likely to buy a certain flavor of mouthwash can help marketing efforts. But in most digital products these demographic data points tend to be less useful for predicting if that group will get value from your product.

The general theory behind demographic segmentation is that knowing which group a user comes from can help predict what they’ll do in your product. In digital products, however — especially B2B SaaS products — demographic data usually ends up telling you scant little about activation, conversion, retention, or feature interaction. This is because demographic data is broad, not granular, and by nature applies heuristics across a wide group of people.

Analytics tools have advanced in recent years, and there are now tools that allow product teams to look at behavioral metrics. Behavioral metrics are more about the specific actions people take in your product. Behavioral segmentation turns the demographic model on its head. When you take a behavioral approach, you track what people actually do in your product.

You see what behaviors tend to correlate with other behaviors. You can segment user groups with whatever degree of granularity is most effective. You can see what sorts of activities predict future activities. And so on.

Once you see how people behave in your product, you can then see how much any given behavior correlates with the metric you’re trying to track (in this case: activation).

Let’s examine how you can use behavioral data to better measure activation.

A five-step framework for measuring activation

How do you start measuring activation beyond just active visits to your site? Equally important, how can you discover what your activation metric truly should be? At Heap, we recommend using this five-step framework.

1. Start tracking user behavior.

The first step in this process is to simply start tracking user behavior. If you are a Heap user, you’re already ahead of the game because you’ve already collected this data with Heap’s auto-capture feature.

If you aren’t tracking user behavior, start as soon as possible. Why? Because your customers’ behavioral data includes the biggest leading indicators of how your users find value in your product.

2. Define “active.”

Once you have this data, the next step is to come up with hypotheses about what your activation metric is. Get a cross-functional team together that includes members from your sales team, customer success team, marketing team, and anyone with a good understanding of your audience and what they want to get from your product.

As a team, throw out ideas on what that “a-ha!” moment might be for your end users. In general, an “a-ha” moment will be the moment your customer realizes that he or she is getting value.

For example, when we look at some well-known activation metrics, such as adding seven friends in 10 days or following 30 users in your first week, they make sense: You’re not going to want to log in to Facebook every day if there isn’t activity for you to view and people to interact with. Twitter wouldn’t be very interesting if there weren’t a stream of topics or thoughts that you were interested in.

Another way to think about it: If Facebook measured activation solely on the frequency of user logins, it wouldn’t represent the ongoing value their users are getting.

Sure, a new user could log in daily for a few weeks, but if they don’t take any action to connect with the larger community, they will quickly lose interest and stop returning to the site. Think about the key moments that might happen early in the customer’s journey and come up with a list of at least five behaviors or actions that you think could qualify as an “a-ha!” moment.

3. Use data to validate your activation metric.

The next step is analyzing the impact of your activation metrics. For example, what sort of impact should we analyze? To the business? On other metrics? Are you saying that you want to see how well that metric tracks other key metrics (like conversion and retention)? Or you want to see how well that tracks with revenue? Both?

Simply coming up with guesses isn’t enough for data-driven organizations. Product managers need to then turn them into hypotheses, analyze them, and look at them in a behavioral analytics tool.

For example, do users that perform those actions actually retain at a greater rate? Look at the behaviors that retained users perform. You also want to look at whether or not your churned users also performed that action. Once you find the metric with the greatest variance, you can define that as your activation metric.

4. Make hypotheses about what might improve activation, and then run experiments to test them.

Take action! At Heap, we believe the best way to increase activation is to come up with lots of hypotheses (use our Book of Questions for inspiration!), then run experiments and measure the results.

Lots of product managers tend to make gut decisions about what to change, and don’t hold themselves accountable if their hypotheses turn out wrong. We think it’s great if things turn out wrong because it helps you understand what didn’t work — and why! So, once you’ve identified what this activation metric is for your organization, the experiments can really begin. Now you’ll want to actually run experiments aimed at getting more active users.

Come up with experiments and strategies that you can try to get more users to perform these behaviors, sooner. Sometimes customers move through the different growth stages naturally. More often they don’t. That’s ok — those moments of failure are opportunities for you to intervene!

How can you re-engage past users who were never truly activated? Is there an opportunity to nudge your customers along with a new user experience, in-app guides, or a new user email campaign?

5. Repeat!

The last step in this cycle is to simply revisit it periodically. As your business and customer base grows, you’ll want to re-evaluate your activation metric over time to make sure that you’re driving the right behaviors for the right group of users.

For example, an early startup might find that an activity such as scheduling a report is an indication of activation. As the company grows and its product evolves, it may later find that usage of a different feature set is a better activation indicator.

Interested in learning more about how to improve activation in your business? Download “The Heap Guide to Activation” to learn why activation is a critical metric for product teams and what hypothesis-driven product teams can do to measure and increase activation.