How Heap Measures Activation

The following is an excerpt from our eBook, “The Heap Guide to Activation.” 

This is the third post in a series of blogs that discuss why activation is a critical metric for product teams and what hypothesis-driven product teams can do to measure and increase activation

Data drives our own business decisions at Heap, notably within our product and customer success organizations. Heap’s net retention is one of our top company metrics, and many of our business decisions (including how we approach activation!) are focused on improving it.

Improving our net retention depends directly on our ability to drive customer value in the onboarding phase. Activation is centered on getting our customers onboarded successfully with a high-touch onboarding process that includes key customer-driven success milestones, such as running queries, saving reports, and using our group-by feature. 

Heap's journey to activation

Finding our activation metric

At Heap, we found two metrics that correlated strongly with activation: “Saving Reports” and “Group By.”

Saving Reports

We think of our value as delivering actionable insights to our customers. There’s no easy way to measure that we’ve delivered this, but we ideated that if you save a report, it’s a good indicator that you found a data point or insight meaningful enough to revisit it later. 

After spending time with our metrics, we realized that when someone saves a report in their first session, they are 2x more likely to be long-term users of Heap. No other activation event/definition had this same effect.

With this information in hand, we looked more closely at how users execute queries and which queries were then saved as reports. We needed to better understand the relationship between executing a query and becoming a long-term user. Was this a correlation or a causal relation between the two?

Through our experiments, we found that we had an opportunity to not only help our users execute more queries, but also help make the query meaningful enough for users to reference later on by saving it as a report.

This was an indication that if we can up-level the questions our users ask in Heap, they will have a higher retention rate, and therefore a higher activation rate in Heap.

Group By

The same logic applied to our second metric, “Group by,” a query functionality that helps users get a deeper understanding of something, such as count of sessions, that is grouped by device type.

This metric told us that when a Heap user uses the “Group by” feature when running a query, their engagement rates tend to be 2x those of users who don’t use “Group by.”

In addition to running a query, we found that both saving a report and doing a “Group by” were more focused definitions of active users in Heap. With this information, we could then analyze different ways we could help deepen the questions that our customers ask in the tool and ultimately drive higher activation rates.

Using data to address customer needs

Along with the data we pulled from Heap, we also looked at qualitative data from our customer base, including their reasons for churning, to learn about some of the issues that customers were having. This gave us key insight into areas we could focus attention on during the onboarding process. 

For example, some of the common questions customers and churned customers asked were:

  • How can Heap be less of a blank slate? 
  • What questions can I ask, and how do I ask them in Heap?
  • How can I share analyses in a dashboard with my coworkers?
  • How can I easily define events in Heap?

By uncovering this information, our Customer Success team could address these potential issues early on with our customers, and provide them with the resources, tools, and support they needed to not feel stuck.

For example, to help customers understand the kinds of questions they can ask in Heap, we updated our in-app walk-through to guide users through the process of running a query and saving a report. We also made sure our onboarding sessions covered this material and even designed a new feature called “Suggested Reports” to help address some of those pain points.

How Heap uses onboarding to improve activation

The onboarding process plays a significant role in improving activation rates. Onboarding is the process by which new customers become familiar with your product’s features, functionality, and overall business value. When it’s done well, a successful onboarding process helps new users uncover the value proposition of your product more quickly. 

It’s for these reasons that we looked to our own onboarding process at Heap to help our customer services team activate customers. We created a product-health metric to help our customer success team further gauge whether or not a customer was truly activated and ready to leave the onboarding phase. 

In addition to the activation metric we outlined in the previous section, we wanted to keep a pulse on how active our users are in the product during the onboarding phase. This gives our customer success team an idea of which customers might need more support using the tool and running queries or saving reports. 

We combined our Heap data from the UI with usage data from our product warehouse to create health metrics. In doing so, we decided to evaluate customer health using color-coded activation metrics:

Customer success health metrics

The indicators help signal to our Customer Success team how to adjust their onboarding strategy to help customers query more events. They also serve as a red flag for customers who are not adopting the product and enable our customer success managers to take corrective action to improve customers’ health score. 

We also use these metrics to quickly identify at-risk customers early on in their life cycle rather than waiting until renewal to discover that they never saw value or are under-utilizing the tool. Account managers who have yellow or red customers in the early stages of their life cycle might offer additional training, point their customers to helpful webinars, or identify new champions within the account. 

They might also spend more time reaching out to uncover what the blockers are. Customers who are green are often sent more content about up-leveling their usage through more advanced features or use cases.

That’s all for activation

When it comes to how you measure a customer’s interaction with your product, activation is the one metric that can help you understand what — exactly — makes new users feel connected to your product so that they become loyal customers.

In this blog series, we’ve outlined the role that activation plays in growth metrics, offer methods to help you define, measure, and improve activation, and share some of the lessons we’ve learned at Heap during our own journey.

If you have questions or thoughts about any of the material in this blog series, or simply want to know more about how Heap’s approach to product management could be useful for you, we encourage you to get in touch with us!