Using Data to Design Powerful Products

In Episode 2 of The Analyst, we speak with Bill Farrell, Director of Product at HelloSign, about how to use data—both quantitative and qualitative—in product development, how to prioritize your roadmap, build your product team, and more.

Receive new episodes instantly by subscribing to The Analyst:
iTunes | Google Play Music | Stitcher

Episode Transcript

Sarah Siwak: In a couple of sentences could you tell us what you do?

Bill Farrell: I’m the Director of Product at HelloSign. I head up project management and design there.

How did you get into product development and design? What do you enjoy most about it?

I got into it accidentally. I was going to school, in college, to be a computer engineer. My last year of school I had time to take on two internships. One was this very technical internship for the Navy where I was analyzing sonar data on submarines. And I hated it. They holed me up in a lab and I didn’t get to talk to anyone. It was just heads-down programming.

My other internship was at a company called Amie Street, which was a startup out of Providence, Rhode Island. So I had these two really kind of very different experiences. The computer engineering government job and also a startup. Really fell in love with working for startups. After I left Amie Street, I went on to go to a company called It was a social network that was focused on finding twenty-somethings and thirty-somethings things to do in their city. I started out as QA engineer, so still in a technical role.

Over time, I found myself really interested in providing feedback and also I became very opinionated on what we should be building. Luckily, I had a founder that saw product sensibility in me and asked me to move full-time into a product manager role. To be quite honest, at that point, this is about ten years ago, I didn’t know what a product manager was or does. But he recognized something in me and said, “I want you to do this full-time and we’ll find another QA engineer.”

Then the rest was history. I’ve been doing product management exclusively for the last ten years.

Could you tell us what HelloSign is, some popular use cases for it, and what makes it unique relative to some competitors?

HelloSign is an e-signature product. You use HelloSign to send, sign, and manage all of your legally binding documents. If you’ve ever filled out a lease electronically or purchased a house or applied for something with a legally binding contract, you’ve used some type of e-signature product. HelloSign differs in the sense that we’ve really built a product for developers to integrate legally binding e-signature into their own products and workflows and that’s one of our biggest differentiators. We’re built from the ground up to be embedded into your product or your service, so it’s like Stripe for legally binding contracts.

When you’re creating the product, how do you understand how customers use HelloSign, or any other product? Do you like to adhere to a specific practice for researching your target users, like persona development, or the jobs-to-be-done framework? What does that look like?

One of our product team values is to constantly be talking to customers. We do that in a variety of different ways, including setting up focus groups. I highly recommend creating a customer focus group that you can use at any point to give you feedback. These are people who love your product, are interested in providing feedback, and want to guide the direction of your product.

From there, you start to see trends in the way that people use the product. We do have personas at HelloSign. We’ve built that over multiple years of getting to know our customers and identifying similarities. I highly recommend using your product personas when you’re doing future development. Who is this for and what are they going to benefit from it? It really helps you get into the mind of your customer and build a product that they’re going to find valuable.

On that topic of product development, what did product development and measurement look like before you joined and how do you and the team approach it now?

Way before I joined, our CEO and co-founder were doing product, which is a very common situation for small startups. He realized that he needed someone to own it and made one a product-centric sales person HelloSign’s first product manager. He did a great job, so as they were experiencing hyper-growth they realized, “Okay. We need to bring in someone a little bit more seasoned to own the function here at HelloSign.” That’s when they interviewed for my position.

Getting in there and making sure that we were measuring things was one of the first things I did as Director of Product. We wanted to have a data-informed product group and that’s kind of how I’ve built the team since then.

What is the relationship between product and analytics? Is there a separate analytics team, or is the function distributed across marketing and product? What does that setup look like?

I look for product managers that are data-driven, so I expect that each one of our product managers does a very good amount of their own analysis. I want them to be responsible for the success of their product and also how it’s performing against the KPIs, so we do a lot of that in the product group. We also do have a BI team and they’re more focused on analytics and metrics at the company level. We also have someone who focuses on analytics for the marketing team. I encourage having a product team that’s really self-sufficient when it comes to metrics and getting answers to questions through data.

What are some ways that the product team measures its impact when it comes to feature releases? Is there a set way that the team quantifies its goals?

We use the OKR framework, Objectives and Key Results. I personally really love this framework. What happens is we’ll set up a main objective for the quarter, like, this quarter we want to focus on free trials, or this quarter we want to focus on moving this metric. We set a big business objective and identify what’s called key results and those are basically the KPIs, the performance indicators for whether or not we’re achieving our objective. That’s what management specifies for the team, and we let the product managers come in and figure out the best way to move that metric.

As far as measuring the impact that product is having, we’re measuring the key results, and our performance against those key results more than anything. More so than, say, release velocity. I personally don’t care as much about how many things we’re shipping, I want to ship things that are having a real impact on things that we have specified as our priority.

When it comes to your product roadmap, how does the team prioritize what you spend time on? Is it informed by data or some other mix of factors?

It’s kind of a balancing act of all the various demands on resources. Those that heavily influence our roadmap boils down to a handful of things. We have our long-term product strategic goals, like what do we want the product to look like in a year, two years? Those are just our strategic goals for the product.

One thing that we really like to track as a SaaS business are sales dealbreakers and the lost MRR associated with features that you don’t have. Anytime we lose a deal, we keep track of what that deal was worth to us and what features did we not have that made it so the customer couldn’t go with us. That’s one way to be data-informed, that’s just not like, “How many page hits did I have?” or “How many people completed this funnel?” It’s instead, “how many customers and how much revenue did we actually lose by not having this feature?”

We like to track sales dealbreakers and associated lost MRR. How many customers and how much revenue did we actually lose by not having this feature?

You can rest assured that features we don’t have that we were losing a lot of customers over tend to make it into the roadmap. That’s a way that we can continually make sure that we have product-market fit, that we’re building something that our target customers found valuable and want to pay for. It’s just a really powerful tactic for building for a SaaS product.

What are some effective and not-so-effective arrangements you’ve seen when it comes to creating product teams? If there were a product dream team that you could build, what would that look like?

My advice for setting PMs up for success is to have very clear areas of ownership with very clear objectives and metrics for those areas of ownership. When you have several PMs that kind of own a bunch of things, then no one really owns anything.

When you have several PMs that kind of own a bunch of things, then no one really owns anything.

We’ve adopted what we’re calling “pods” where there’s a PM and there’s anywhere from two to four engineers associated with that pod. They own a particular product area. They have objectives and metrics that they’re looking to increase or improve and they work with the same engineers on their product area. It really instills a sense of ownership not only for the product manager but the devs who are building it out as well. I think it’s more efficient because the PM tends to not need to convince the dev team why we’re building something because they’ve been along for the ride and they understand the business value behind it.

By making it so that the PM is working with the same group of engineers and they have very clear areas of ownership and objectives and metrics that they’re looking to improve, that’s how you build a really high-functioning product dev team. If you’re hiring the right product managers, they’ll really appreciate that autonomy. They’ll really appreciate having the flexibility to try things that they think will move the needle for the particular metric that they’re looking to improve. The devs feel very invested in what they’re working on. They don’t feel like stories are just being thrown over at the fence at them.

I think the challenge for startups is hitting some kind of critical mass where you can dedicate certain people to certain product areas. It takes a little while to get to the point where you have enough, say, front-end engineers to dedicate one to each pod. I get that. It takes a little while to get there, but once you can dedicate resources like that, I think it’s really effective.

Do you have any advice for companies that are starting a product organization from scratch?

My advice to CEOs out there that just took their seed round or some funding is to hire the most senior product person that you can afford. A lot of founders make the mistake of bringing in someone junior because they’re cheaper. The issue with that is you still tend to, as the CEO maybe who’s product-driven, you still tend to be hands-on, and you’re not being a CEO.

By bringing in someone more senior, whether it be a senior PM, a director, a VP, someone with more experience who knows how to build out some of the processes and structures to have a well functioning product development team—that’s going to be a great investment. Long story short, hire the most senior person that you can afford and empower that person to start building out some of the processes that will help you scale.

What are a few different ways that you would say that data—qualitative or quantitative—can play a role in product development and design?

There’s traditional analytics where you define what the most important funnels are within your product and you track and measure those and find out where the biggest drop-offs are. That’s a really straightforward product manager-type way to look at data to figure out where to focus your resources and time.

There are creative ways to help your product team be data-informed that aren’t just, “Okay, well, how many clicks did that button get?”

Also, I think the sales dealbreaker thing that we talked about earlier, particularly for SaaS businesses, is a way that product managers don’t tend to use data that I find to be really helpful. There are creative ways to help your product team be data-informed that aren’t just, “Okay, well, how many clicks did that button get?”

There are creative ways to help your product team be data-informed that aren’t just, “Okay, well, how many button clicks did that button get?”

We try to use a wide variety of different tactics. For design, one tool that’s more qualitative that we really like is UserTesting, which makes usability testing very low-cost. As much as I’m talking about data today, I really recommend that product people are doing ample usability testing as well. Data’s great, but you also want to watch people using your product. You want to hear their reactions and what they’re confused by, and that’s something you don’t necessarily get by just looking at data.

UserTesting makes it very low-cost to conduct multiple usability studies at any given time. When we release a new feature, we put up a test, and we’ll get twenty videos of people interacting with our new feature. We do that early on so usually that usability testing uncovers something that we can improve or make more clear.

Do you have any questions that people should be asking when they start to approach onboarding, or strategies for creating an onboarding experience in a SaaS product?

Your onboarding experience is all about time to value. First off, ask yourself, “What’s the one or two actions that a user completes within your product that makes it click for them?” That’s when they realize why your product exists and what its value is. If you don’t know the answer to that question, that’s certainly a problem, and you should do some customer development and figure out when these customers get their “aha!” moment that yours is a really great, valuable product.

Assuming you know the triggers for a user gaining value from your product, you’re going to want to productize that. Just to use HelloSign as an example, maybe a user discovers value when they send their first signature request. That’s that moment in time where they say, “Okay. I get what HelloSign does. I get why this is valuable,” and they’ve gone through the process and know how to use it and how they can use it for their lease that’s coming up.

Your onboarding should be focused on getting the user in and making it as simple as possible to complete that action that they deem valuable. You want to get them in and show them how to extract value from your product, and you want your onboarding process to make that really simple. That’s something you should be using like funnel analysis to identify where it’s breaking down.

If you weren’t able to get them to extract value from your product on their first session, there’s a very high likelihood that they’ll never come back again. One way you can combat that would be having a really effective drip campaign where after a day or two you reach out to them with some tips or a reminder, but most importantly gets them back to the site so you have a second shot at showing them the value of your product.

Given your experience in the field, how would you say that product management or development or design are changing versus when you started?

The tools that are available in 2016 are incredible. They provide tremendous insight into what your customers are doing, and that data wasn’t accessible when I was starting out, like easily being able to measure everything, conduct usability testing, stack rank all of the various initiatives that you could be working on. Usability has emerged as a key component in building great products that people love.

Product management as a field used to be engineers that started taking on a more business role, and it’s evolved as a blossoming industry. It’s just a great time to be a product manager.