How we built a health score with Catalyst that predicts renewals with 95%+ Accuracy

The problem

Health models allow revenue leaders to communicate easily with other teams internally and confidently articulate their revenue forecast. If the health score doesn’t assist these leaders in accurately predicting revenue, it can cast doubt on the leader, the CS team, and the strategy in place. 

But there are two challenges for teams that want to maintain a consistent and accurate measure of health: The difficulty in selecting health factors for a health score combined with the temptation to incorporate constant changes."

As post-sales teams discover new information, they might also be tempted to constantly change their health scores to keep up with the evolving world around them. This constant change presents a challenge down the road when looking to attribute true impact to specific factors in their score.

Moreover, the individuals that are asked to use the health score as their guide can get lost in the dust if factors within the score are constantly changing. They are being asked to learn and adopt a new approach with each change – sometimes without supporting data on why the change happened. 

Play intro

At Heap, we organize our post-sales teams under our Customer Success organization. This comprises Professional Services, Solutions Consultants, Pre-Sales Engineers, Customer Success Managers, Education, Scale, Adoption, and Support. The health score is a north star to the Customer Success organization at Heap. 

Lane Hart, the Senior Director of Customer Strategy and Operations at Heap, has been responsible for the accuracy of the health score at Heap for years and has built multiple iterations of the model. He knows the importance of having a reliable health score to inform the team, drive the value of Heap across their customer base, and influence revenue forecasting. 

“The goal with the health score isn’t to tell the CSMs how to do their jobs because that would be unfulfilling (and annoying), but it’s a good way to highlight their book of business in a way that shows them areas where they can drive the most impact.”

Previous to Lane’s ownership of the health score, anyone working accounts could update the health score by selecting either red (at risk), yellow (neutral), or green (healthy). To inform their decision, our post-sales teams usually review the usage and adoption data within our product, Heap, as it houses behavioral data. But this process was ad-hoc, inconsistent, and ultimately reliant on human capital. 

With so many available metrics, Lane and his team quickly learned that the teams weren’t all looking at the same data when rating health. In addition, an individual’s sentiment was too subjective – some were pessimists while others were overly optimistic, which influenced the health score they associated with an account.  

The results? The account health was not reliable.

Lane knew that if we wanted our health score to be reliable and impactful, we needed to build a model that surfaced the right data at the right time so the account owners could take the right action. In this Play, Lane details the steps we took to operationalize our predictive health score by powering it with data.

How we ran the play

When Lane and his team started to build a predictive health score model, they knew they wanted to leverage the data they had at hand and ensure their score used leading indicators of success as the foundation. They wanted CSMs to have timely information to mitigate risks and boost account expansion.

To build the workflow, Lane and his team used Heap, Salesforce, and Catalyst. Here’s the step-by-step breakdown of the Play: 

Step 1: Segment your customers to deliver tailored experiences

Our team learned a few years ago that not every customer needed the same level of service. 

Customers with larger digital footprints had a higher amount of sessions, larger teams, and more data being requested by team members. To provide an appropriate experience, we provided a strategic resource to these teams.

On the other hand, customers that had smaller digital footprints had fewer sessions and smaller teams. These teams didn’t require as much high-touch support. 

Clear customer segmentation is an important step in building a predictive health score as what your customer needs, outcomes, and the playbooks you’ll run won’t always be a one-size-fits-all approach. 

Thanks to these segments, we were able to create multiple health profiles that meet our customers' needs so we can allocate our resources accordingly. 

Step 2: Set up an informed health score based on those experiences

Because we are a product analytics company, our post-sales team is fortunate to have access to a lot of customer data! However, having access to so much data doesn’t always mean the right data is being used in a health score. 

In 2020 we implemented a new Adoption team. That team worked in partnership with our Data Science team to identify 18 metrics to test our renewal data. After analyzing renewal data against these 18 metrics, the teams were able to see that the likelihood of renewal was strongly correlated to the number of queries a customer was running.

We learned that the more Product Managers run queries, the more likely they were to answer their own questions, ultimately associating an increased use of Heap with delivering more value.

In contrast, Product Managers that were not running queries often were likely getting those answers elsewhere and not driving much value out of Heap directly. We further dug into this metric to look at frequency and discovered that the monthly frequency of running queries resulted in the retention of users and the growth of accounts.

With data-backed metrics, our Customer Success team at Heap built a health score combining three tools — Heap, Salesforce, and Catalyst.

In this health score, we organized factors into two buckets: adoption and relationship.

Step 3: Collect the data on health score performance

Though it’s incredibly tempting to make changes on the fly, Lane can’t stress enough how important it is for CS teams to allow a set amount of time to pass before making any changes to a health score.

At Heap, our CS and Data Science teams committed to a six-month period of unaltered data collection for their health score. In this period, they may review any trends as the data accumulates, but no changes can be made to the health score.

Step 4: Review the performance data to identify trends

After a six-month period had lapsed, Lane, along with Michelle Mazzotta, our Customer Success Operations Manager, partnered with our Data Science team to analyze the findings.

The objective of their analysis was to identify additional leading indicators that can help the Customer Success team better mitigate churn and drive account expansion all at once. They also reviewed the impact of the existing indicators to ensure they were still strongly correlated to renewal predictability.

Our Data Science team was brought in to help with the analysis of the data, looking for patterns and trends, and hypothesizing on new vectors that could be added to the health score. Lane and Michelle did some statistical testing themselves using R, the Statistical Computing Programming Language, but our Data Science team helped to validate it.

As mentioned in step 2, the first iteration of the health score included two buckets: adoption and relationship.

The adoption bucket included the product usage data (i.e. queries) and how much of the platform the user consumed. However, after reviewing the performance of this bucket, we learned that we needed to add a new bucket called “consumption.”

Ultimately, customers who had purchased a larger plan than needed had lower-than-expected adoption scores, which skewed our results. To account for this, the consumption bucket measures how well the user is doing against what they purchased. Since Catalyst allows teams to weigh each bucket of the health score accordingly, our consumption bucket accounts for a small percentage of the overall health score.

The relationship bucket was initially based on manual inputs from our Customer Success Managers (CSM), specifically around the last time they talked to the economic buyer or the champion of the account. From the analysis by Lane and his team, it was found that this dependency on the individual led to outdated data and, therefore, a less reliable health score.

With their next iteration, they committed to automating that information. They used Snowflake to aggregate data from activities in Salesforce. We would look at whether a contact was tagged as an economic buyer or champion and when the last inbound communication was received from them.

In addition to the data analysis conducted, Lane and Michelle also identified some learnings related to the health score itself, which they used to update their new version of the score.

Their learnings included:

• To create more trust in the score, show logical groupings and weights that align with the team’s POV (not everything is weighted equally!)

• Preference metrics that are tighter in a timeframe help ensure they're showing the most up-to-date data (i.e. monthly instead quarterly)

• Each part of the health score has a specific action a CSM can take to improve it (and each action aligns with customer value so we don't unintentionally create annoying interventions like 'touch bases')

• We need to validate that the metrics are actually leading indicators of renewal and/or growth and lean on our team for feedback

• It's best to maintain separate metrics for health score and "CSM Sentiment," and present them together

• Data can become outdated if you don't reduce the onus on the customer-facing team (i.e. decrease the number of manual inputs)‍

With their findings in hand, it was finally time to put pen to paper and revise the health score.

Step 5: Update the health score to reflect the findings

As we've said, we run our health score model in Catalyst at Heap. Using the data connecter between Salesforce and Heap, we can push data directly to Catalyst (without the help of engineering).

Once we had our data in Catalyst, Lane and team were able to update our health profile to include the new bucket they defined, “consumption,” and update our existing buckets for “adoption” and “relationship” with the new data points (e.g., “Last Inbound Executive Sponsor Email”).

With the right data points added to Catalyst, updating our health score model was a quick plug-and-play in a single settings page to do the following:

• Create a new indicator bucket

• Update data points across adoption, relationships, and consumption

Once saved, our teams were able to start seeing updates to the health score within their accounts so that they could quickly begin actioning with little delay.

We combined our findings and learnings to develop the latest version of the health score:

Step 6: Update the team's operating cadence around health scores

“As with anything you do in a system, it’s only as good as the people and the process around that tool. You can’t just create a health score and then expect health to improve,” says Lane.

Lane and his team have refined the operating cadences that rely on our health score several times in order to drive efficiency and action.

Here are the two iterations of our operating cadence: 

Version 1: Initially, CSMs and managers reviewed accounts every week during 1:1s, using the health score. However, this was time-consuming, especially for CSMs managing 50-60 accounts.

Version 2: A cross-functional team meeting was established to review different parts of the customer journey. The audience includes heads from CS, Professional Services, Solutions Consulting, Pre-sales Engineering, Education, Scaled Adoption, and Support.

The bi-weekly call is divided into four sections: 

• Part 1: Address the needs of new accounts that need to be resourced

• Part 2: Review accounts that are going through implementation to see how they’re tracking, as well as surface any needs from our post-sales team

• Part 3: Highlight success stories for implemented accounts that are on track to achieve their first win within Heap

• Part 4: Find solutions for how to secure renewal and expansions for mature customers as well as review accounts that have been with Heap for a while and are either on/off track to renew or expand.

The cross-functional team leverages health scores not only to help determine the scope of customers discussed during the meeting, but also to advise on next steps. They use the health scores to understand why accounts are off track. If the account owner’s sentiment or health score is not green, they assign specific "Get Well" tasks to each account based on the performance against the included indicators to make progress toward improving overall health.

This latest version ensures all post-sale teams are aligned on the plan of action to drive renewals and expansions. Account owners leave with clear next steps that lead to results.

Step 7: Use the health score to forecast renewals

Lane and his team incorporated the health score into their medium and long-term planning to forecast renewals. 

Every 6 months, there is a forecasting exercise done using a point-in-time look at the health score. This helps our finance and leadership teams negotiate targets for not only Customer Success, but our entire business. Lane uses the numeric health score (a number between 1-10 to 1 decimal) in a model to predict renewal rates for the next 6 months. If the health score is anything but green with a renewal in the next 6 months, it is a red flag that requires focus.‍

Impact of the Play

The impact of this Play has been threefold for us: 

• Accuracy: The predictive health score helped us predict renewals with 95%+ accuracy. This means that our revenue leaders can confidently articulate our revenue forecast, which is critical for the success of the business, defining future strategies, and communicating the overall impact of the post-sales team.

• Time-savings: Our post-sales team no longer needs to rely on heavy manual work around health scores. We eliminated the need to update one-off fields and perform ad-hoc analysis. As a result of the predictive health score, our CSMs reallocated 5+ hours per week back to customer-facing work by no longer needing to analyze data manually.

• Confidence: Our post-sales team now feels confident in suggestions offered by prescriptive playbooks. This is an important impact, as it means they can be more proactive in providing guidance to customers and can save time by relying on the prescriptive playbooks.

Now it’s your turn!

Discover how to replicate this Play within your business by checking out the full post from Catalyst.