Building a Retention Strategy, Part 3: How to test the strength of your ideas with an Impact Calculator

If you’re concerned about developing an effective retention strategy, this series is for you. In this and future posts, we’ll walk through the key ways to use data to make sure you’re building a smart, scalable retention motion. Most will also include worksheets you can download and use with your team.

In Part 1, we showed you how to establish Active Usage Metrics for the value of your product equate to value for your business.

In Part 2, we showed you how to connect these metrics to your ultimate goal—revenue—by designing a metrics tree.

-------------------------

Today, our next step is to figure out what kinds of interventions will make the biggest difference. In the past, you would have just guessed at this randomly. But there are smarter ways to improve your accuracy! At Heap, we use this Impact Calculator. It’s a simple tool for quickly understanding the potential impact of any project so you can compare the value of different initiatives you’re considering.

PRO TIP: This is also a great exercise for making sure you’re tracking all your metrics, both general and local.

As a PM, you have difficult choices to make!

These days, everything is about business impact. When faced with the prospect of moving the needle on important topics like revenue and retention, PMs are typically overwhelmed (and even strong-armed) with suggestions and requests on what paths they should pursue. But you only have so much time, energy and budget at your disposal, so it’s critical to know how to evaluate the best choices in order to invest their team's time and energy.

Our goal for you: To have hypotheses at the ready about which interventions will move the needle most effectively and efficiently for the greatest ROI.

There are two main questions that you need to ask:

1) “If we make a specific change, how many users will be exposed to it?”

2) “How much will that change impact our users’ behavior?”

Imagine the advantages if you could answer these questions before taking on a project.

• What approaches would seem most profitable?

• What adventurous new steps might you take?

• What old tactics would you abandon?

• What new ideas might light up in your team’s imaginations?

Meet your new favorite tool: The Impact Calculator

Very simply, impact calculators assess the potential rewards and consequences of different actions and scenarios. Decision makers can now make informed choices about optimizing resources by understanding the risks and benefits associated with various decisions.

In our application, the Impact Calculator evaluates your ideas for changes in a user flow by measuring their effect on a higher-level KPI. You start by analyzing the current baseline measurements for your flow, and then predict how your proposed changes will impact those baseline measurements. The calculator does this by tracking how the two most important functions of the change will affect the baseline metrics.

There are many ways to move a metric, but the success of any initiative at improving a KPI ultimately depends on these key factors—Scope and Magnitude. 

Scope = Scale of the audience

An improvement that impacts 1,000 users will move the overall KPI more than a similar improvement that only impacts 100 users. Makes sense, right? In general, the larger the audience, the better.

Magnitude = Scale of the action

An improvement that impacts the user experience more deeply and meaningfully will make a larger impact. It sounds obvious, but every PM has encountered proposals for changes that minimally affect the user experience, but are presented as something that will make a significant impact, like color changes and button placements. Will that be true?

The Impact Calculator has three basic components:

1. Baseline metrics for the overall experience.

2. Baseline metrics for the part of the experience you wish to change.

3. Estimated guess for the new rate.

How to use it

Note: looking for a more detailed walkthrough? We have a full post on that.

1. Start with your metrics tree. What have you identified as most important?

2. Now let’s look at either the core metrics (breadth, depth, etc) or the active usage metrics for your product. What you choose will depend on the size of your product, your product area, etc.

3. Brainstorm potential ways to move these metrics! There are lots of things to think about here. They can be tiny and incremental or a whole new approach. You don’t have to assume that minor change = minor impact. Remember, that’s what the calculator is for!

Just like your other brainstorming efforts, the sky's the limit! Go nuts here.

4. Now you have a list of great ideas. It’s time to take the most promising ones and run them through the calculator!  

NOTE: You will be doing this more than once! The calculator is simply a way to call attention to your most viable ideas and weed out your least viable approaches.

An example

Let's look at a hypothetical situation and see how the team might use an Impact Calculator to determine which intervention they should attempt.

You start with an overall retention rate of 80%. You have 10,000 customers.

Instead of changing notifications in the product to nudge people to use a certain feature, you’re considering sending an email campaign to a select group of people.

Let’s imagine that 500 people will receive the email campaign, and 1000 will see the nudge in the product. 

• Of those 500, the overall retention rate is 40%

• Of those 1000, the retention rate is 60%

(You should know this from your Heap data, your Marketo data, or wherever you pull data from.)

For the last step of the Impact Calculator, you’ll need to make an educated guess.

So you make these hypotheses:

• With the group of 500, you can raise the retention rate from 40% to 55%

• With the group of 1000, you can raise the retention rate from 60% to 65%.

You can base this assumption on prior experiments and your product knowledge. (When in doubt, it’s best to be conservative with your estimate.) You can also try different values to see how sensitive your outcome is to the assumptions you are making.

Using the calculator, you can see the overall increase is 0.45% vs 0.2%. It makes more sense to implement the email campaign.

Some caveats

As a good PM, you know there are an infinite amount of changes you can make to a digital experience. So don’t just settle on your first idea. Come up with many approaches, make new estimates, and see which idea performs best. Accuracy through volume is a good approach here!  

You’re always making estimates of what you think the change in retention rate will be. Feel free to be flexible with your numbers and see how they compute.

You may receive a directional signal instead of a definitive answer. But that in itself can be super useful! 

The overarching goal is just to be smart about what you’re doing. Build the habit of paying attention to your markers, and notice what they mean.

Use the Impact Calculator to measure the difference in performance between your ideas so you can make the best investment in terms of not only capital but your team’s time, energy, focus and motivation.

Collect your measurements in advance, make hypotheses, and see what works! And have fun with this and push your boundaries, because at this stage there’s no risk to getting anything wrong.

In our upcoming Retention Strategy Posts, we’ll cover prioritizing your book of business, and orchestrating your scaled CS strategies.