Join us every Wednesday for a weekly live demo of Heap

How the Most Innovative SaaS Companies Use Data Science

What does analytics look like at a SaaS company? How is it different from a consumer company, and what’s the best way to ensure that data practices can scale, while maintaining data integrity?

Join us in this special episode of The Analyst from a SaaS data panel discussion held recently at Heap. Heap’s CEO and Co-Founder Matin Movassate chats with:

  • Bilal Mahmood, Founder, Bolt and former PM of Data Engineering at Optimizely
  • Ville Tuulos, Head of Data, AdRoll
  • Maurice Mongeon, Data and Analytics Lead, App Annie


Short on time? Consider this food for thoughtfrom our panelists:

  1. Data quality can be better addressed by thinking about its component problems: fragmented data, duplicate data, and incorrect data.
  2. Users? Accounts? The primary key that you perform analysis on changes the entire architecture of your data—how it’s stored, the types of analysis you’re doing, and how downstream reports are built.
  3. High-level aggregations can be misleading. Instead, seek out granular informationto draw insights from data that you can actually act on.

Receive new episodes instantly by subscribing to The Analyst:

iTunes | Google Play Music | Stitcher

Matin Movassate: I’m Matin, founder/CEO of Heap, and in case you don’t know what Heap does, we build an analytics tool that makes it really easy to automatically capture all your user data and get insights out of it. We’re going to talk about some of the most notable SaaS companies. We have people here from App Annie, AdRoll, and Optimizely (and now founding his own company). And we’re going to talk about: what does analytics look like at a company that has recurring business and recurring revenue? Hopefully we can have a lot of concrete stories and takeaways here for everyone.

Could you all introduce yourselves?

Bilal Mahmood: I’m Bilal. I used to be the PM of Data Engineering at Optimizely, which involved both the ETL infrastructure, the BI, and data science, across their respective functions. I was there for a couple years, and then recently started my own company, Bolt, which is trying to basically put myself out of a job, or what we were trying to do at Optimizely before.

Maurice Mongeon: Hey, I’m Maurice. I’m the Data and Analytics Lead at App Annie. I’ve been working there for two years. I’m a co-founding member of the team. For those who don’t know what App Annie does, we provide mobile and download analytics on every app in the App Store. So we provide market estimates, MAU, download revenue, advertising revenue, lots of different metrics around all the apps.

Ville Tuulos: Hi all, I’m Ville Tuulos, I work at AdRoll. My title is Head of Data. AdRoll is a performance marketing platform, so if you ever want to run a display advertising campaign, or maybe do something with email, AdRoll is the place to go. We do social, email, and web advertising. I’ve been at that role for three and a half years, seeing the company grow from a medium-sized start-up to a global company, so that’s been pretty exciting.

Matin: Let’s dive into analytics at your companies. What did it look like before you joined? What was broken?

Bilal: When I joined Optimizely, we had a situation where the problem I had going in was primarily data quality. We had a lot of analysts, we had good data scientists, we had good data engineers, but we were in this state where we had a bunch of data that wasn’t really usable.

Within “data quality,” you have fragmented data, you have duplicate data, and then you have incorrect data.

That basically came down to three different problems, and I think they’re pretty transferable to most other companies as well. You have fragmented data, you have duplicate data, and then you have incorrect data.

  • Fragmented data was because our tech stack matched our org chart: marketing was using one thing, sales was using another thing, engineering was using another thing. All different parts of the user lifecycle were in different places.
  • Duplicate data happened to be because the data was fragmented between different systems. Sometimes two different locations had the same attribute that people were tracking, like you had an address in Marketo, you had an address in Salesforce, and you had an address in NetSuite. Because of that, which one do you choose?
  • Partial dataor incorrect data was data that was missing, or was wrong, and that was often because someone had incorrectly entered it in an original source system.

A lot of what we were doing originally in the first couple months I was there was just documentation, systemizing processes to align those expectations across time, and that enabled the data scientists and the analysts to have a cleaner schema to work with and actually do the job that they needed to do.

Maurice: When I joined, it was really department-driven as well. We had the sales ops team running Salesforce, we had marketing running Marketo, and we had web analytics kind of run by product/marketing. At the time it was Google Analytics, and my previous manager was a founding member. He said, “Alright, this is ridiculous, we can’t really dig into our users, and we’ve got to get Kissmetrics going at least, right?” It’s a start.

Now we’re Heap customers, just to preface here. We’re really getting everything standardized, starting to create some clean datasets. We used to have this beautiful table in Salesforce we called “Salesforce Cleaned Leads and Contacts.” It’s really just cleaning up all the dupes initially until you can figure out the root source of all the problems.

Some of the first big things we tackled was finding all the inputs, ingestion of data, from all the different sources, and then starting to meet the initial reporting needs.

Ville: I can totally echo the story here. My version is even more extreme, it wasn’t just broken, it was nonexistent. Some background—my old co-founder, Yuri, and I used to run a small analytics startup, before we went to AdRoll, called Bitdeli. What Bitdeli was doing at the time was making analytics more approachable, of course like many other companies these days, but we started in 2011, so it was kind of ahead of its time. The idea was that, especially, smaller companies who didn’t have resources to build their own infrastructure should be able to still do more than what you can do with off-the-shelf tools like Kissmetrics or Google Analytics and so forth.

Anyway, that company was all about analytics and targeting smaller companies, and obviously I wouldn’t be here, at least not wearing the AdRoll hoodie, if that had been successful.

When we moved to AdRoll, I asked the CEO, “What should we be doing here?” And he said, it would be nice to have some kind of a business intelligence, some kind of analytics on how the company’s doing. And at the same time he was showing the revenue chart, and it was totally going like a hockey stick. That was one of the most convincing reasons for me to join AdRoll in the first place—the company was growing like crazy, and there was nothing. I was thinking, look, if there’s a company that can do so much money with so little, imagine what it could be doing if it actually had [data infrastructure].

This was kind of a pretty hard lesson for me to learn personally, that honestly… I mean, this is maybe my first controversial statement today, but you don’t need analytics if you have an excellent product-market fit. If you have founded a magical money-making machine, you don’t need any analytics. Actually, you don’t even need any kind of financial department. The money just goes to the account.

Now people know more, they can ask more questions. The money chart is still growing, but it has been an interesting journey.

Bilal: I’d echo that at Optimizely. I joined and the company had a ridiculous run rate, and it clearly wasn’t necessary nor sufficient to have this analytics infrastructure to get to that point, and it still worked.

Matin: I don’t know if that’s applicable to most people here, that they have hockey stick growth, but at some point that doesn’t happen anymore, and you need to think about analytics. You mentioned actually, Bilal, three things: the data was fragmented, there was duplicate data, there was partial data. And Mo, you mentioned Salesforce, Marketo, web data. I guess this is more of a question more for Ville and Maurice now. At your current roles, what are those different data sources that you need to bring together, and how do you do so?

Maurice: Sure. Really, we bring it together through just ETLs. Everything feels custom in the end, right? You have logic that’s on the front end of all those data sources like Marketo, Salesforce triggers, things like that, as well as, you’re not just ingesting it, you’re still doing stuff with it. You have to make updates on your own end and our internal warehouse into our product databases, and it all still ends up being processed in the end in your own way, because that’s kind of how we’ve developed our company and developed all of our objects.

So particularly, we use Luigi internally as well as Pentaho in the past. I really love using Luigi for ETLs. It’s super easy to use and it’s scaled to our needs, at least. We use Luigi for ETLs into Postgres, which is our primary data warehouse. Now we’re moving into Redshift eventually as well.

But to get back to your question, you have to get a certain mastery of all the data sources. We have ops teams that run each department’s products, or their parts of the dataset that they use. But really, there’s a lot more happening behind the scene with all of it together. You need a team of analysts, data engineers, that have intimate knowledge of everything.

Ville: We use Amazon Web Services for everything. I’m actually a big fan of using Amazon History, the storage service, it’s a big data lake. The other thing about that role is that we produce tons of data internally, so we produce today about maybe a hundred billion events every day. How AdRoll works is that we participate in so-called real-time ad auctions, and every time we bid on an ad, every time you surf somewhere on the web, we get requested to maybe show an ad to someone, and we collect all the data and analyze all the data, so it produces a huge amount of data internally.

And then in addition to that, we collect data from all our customers, who want to run performance marketing campaigns. And then on top of that, of course we are a SaaS company, and we are starting to use Heap more and more, since the whole idea is that we have this philosophy of collecting everything. We want to collect everything internally about all our systems, but also of course about our website usage. How people use our dashboards, how people use our marketing sites.

And then there’s a third component, which is our internal sales, and that would be of course Salesforce, so altogether if you think all the marketing tools: we have Marketo, we actually are using a number of different tools, Appcues, Intercom, and so forth for the marketing side, and collecting that data in one place. And then on the other hand we have all the internal systems pushing data to Amazon History. And then we have Salesforce, and we are trying to pull data from Salesforce, which is always painful.

And then of course, like he pointed out as well, the idea is that then you have to start actually massaging the data, and making sure that there are no duplicates and that there is no fragmentation and so forth, which is a big thing. Actually, we’re also using Luigi, so we have actually a couple of blog articles, if you go to the AdRoll tech blog, about how we are using Luigi to process large amounts of data. That really works well for us as well.

Matin: For SaaS businesses, having a complete data set is important, because it’s all about that recurring user journey, understanding their entire lifecycle as a user. Bilal, question for you, in your current role, in your current company, you work with a bunch of different data sets, trying to get insights out of it. What is specific to SaaS? What sort of analysis, what sort of insights should companies in this room try to get from their dataset?

Bilal: I’ll back up a little bit, I guess I haven’t talked a little bit about what we’re doing now. With Bolt, my company, we’re essentially building AI to predict consumer behavior. As our first product, we’re building a predictive marketing layer to ingest data from your web analytics stuff like Heap, payment and email solutions to help predict consumer outcomes for your users. So we’re focused on predicting individual user outcomes, which can be keyed to an email address.

I’d say one of the biggest differences between, say, a SaaS company, and who my current customers are, which are B2C companies, is what is the primary key that you are trying to do analysis on? For most companies it’s a user, but for a B2B company, it’s your accounts, it’s your companies. And so in your data structure it can be account name, it can be account ID, it can be a range of things.

One of the biggest differences between a SaaS company and B2C company is the primary key that you do analysis on—a user vs. an account. Every kind of downstream effect that differentiates B2B analytics from B2C stems from that core concept.

I think that’s really what changes the entire architecture of how your data is stored, the types of analysis you’re doing and whatnot. And every kind of downstream effect that differentiates a B2B analytics from B2C stems, I think, from that core concept.

Because you’re doing account-based analytics, there’s different problems that emerge because of that. There are aspects of how the number of observations that you have for any analysis are a lot smaller than, say, a B2C company. Most companies that are B2B only have, say, maximum thousands of accounts, whereas people who are B2C (shopping, ecommerce, media), have millions of users. So when you’re trying to do analysis, and you’re doing segmentation, you lose statistical significance very fast when you’re segmenting accounts because there’s only thousands of them.

Second, I think, also, is that the latency of what you’re trying to predict is extremely long. B2C companies, whether you’re gaming or whether you’re media, you’re looking for actions that are happening within days or maybe a week, whereas B2B, if you look at where the metrics that you’re trying to operate on, it’s MRR, churn, and expansion. Those happen in quarterly or yearly cycles, those same metrics (revenue, conversion, and churn), happen for most B2C companies on a daily, if not weekly, basis.

So, the metrics and the latency by which those metrics are measured is very different between those respective companies. It creates problems for B2B companies, it’s harder to do predictive analytics, it’s harder to do analytics in general, because there’s less to slice and dice without increasing variance for losing significance in that sense.

Matin: That’s a really good point about account-based analysis, and that’s something we want to make a lot easier in Heap as well. Zooming out, we talked about the data sets, how to bring them together. Let’s talk a little bit about how people use analytics at the companies. Tell us about an interesting story where there was a really big business decision based on data. Tell us what that looked like, what the impact was. End to end, what did that process look like?

Ville: AdRoll serves all kinds of customers all around the world, so we have literally customers selling frozen dog food in Germany, and then actually at the same time we’re trying to serve some of the largest B2B companies in the U.S., so there are many stories.

One thing that I personally think is kind of a nice example of something that I think might be the way business overall will be done in the future. This is about the new product that we launched about one and a half years ago. It’s called AdRoll Prospecting. I was part of the team building the product. The product tries to find you new leads, new potential customers, so obviously a very important thing. As you pointed out, the problem is of course how you get new leads, how you get new customers works very differently for different customers. If you’re selling frozen dog food, it’s a very different thing than if you’re Heap, for instance.

Also one thing about that is we have a large sales force, we do have lots of inbound stuff, people come to AdRoll organically, but also we have a large sales force going outbound and selling AdRoll products. So now of course it’s a super important for our products is that they really have to work, since people really measure the success at AdRoll by looking at the performance numbers. And now the problem is that if you have a product that doesn’t work for everybody but works for some, of course it doesn’t instill confidence with the salespeople, which is totally understandable. And now this is a problem, so what do you do?

It’s a fact that we can’t make one thing that works for everybody, so we can actually predict for whom it’s going to work. My specific example is from Europe. The obvious first thing is, let’s build a dashboard. So we build a dashboard that shows, okay, where are the biggest potential leads? The dashboard shows you should be going and selling to UK and Germany and France and Spain and, well, those are the largest countries in Europe, so it doesn’t help really anything.

The next idea is that the problem is that it’s way too high-level. It’s high-level in the product point of view. It’s also too high-level in being actionable. It doesn’t really help you with anything. What we actually ended up doing, since how the product works is that our customers basically put their data in this big data pool, and then we can predict using the data pool who’s most likely to be interested in your stuff.

What we did is we ran an absolutely massive machine learning job in Amazon that basically created these different hypotheticals: what if this guy, what if this account opted in? What if this account started using the product, what would happen? And we did this for basically every single account that we knew in Europe, and we basically did lead scoring by account, estimating what’s really the effect. The effect is not only to the single customer, but there’s also this transitive effect to all other customers. If this customer joined, what would happen to everyone else? It’s really, really hard to understand by looking at the dashboard or anything like that.

The funny thing is that the end result of that wasn’t a big dashboard or anything, nothing too complicated. What it was is basically one score, one lead score, that we then pushed to our internal system, where there are calls to action to our sales people. And we told them, “Look, if you got this account, the product would work for them and it would be extremely beneficial for everybody else as well.”

Fight against high-level aggregations, since those can be misleading. What you really want to have is that granular information, which tends to be way more understandable and actionable.

We were able to do this at the very granular level, and I think overall, that’s my experience at AdRoll. I feel that my role these days is really to fight against aggregates, fight against averages. Oftentimes, fight against any kind of high-level aggregations, since those can be so misleading. And oftentimes what you really want to have is that granular information, whether it’s an account or a single user, and that really makes a big difference. That tends to be way more understandable and way more actionable than anything at the super high levels.

I think that this is really an interesting example of something that combines really sophisticated machine learning, really fancy stuff on the back-end side, but then the end result was something that was extremely small, extremely actionable to people who are actually ending up making money through the business. So we had a big impact in the end.

Maurice: Our company is B2B; we focus on accounts, mostly. That’s how we make all of our revenue, is by selling to mid-market enterprise. But we also have this free product. It creates this mix of users.

If you’re an app developer and you wanted to keep an eye on all your downloads and revenues, all your user metrics through Google Analytics, all in one dashboard, you could do that and sign up today for free on App Annie. The trade-off is we’re taking your data and putting it into our data science model, and that app economy.

I worked on a project recently, taking a look at our user base and trying to find people who are moving up and down the ranks. We want to capture someone’s attention the moment that they’re hitting critical mass within the app economy. We want to be reaching out to them right when they are exploding, they’re starting to get funding, they could be the next King, the next Clash Royale app or something.

So we’ll reach out to them through this free way, through the analytics platform. We also want to sell something to them, so we have to use a ton of data enrichment tools to sift through all the leads, as well as look at our own internal data. We have daily estimates, the ranks change daily, apps get featured, and that’ll just explode an app’s growth. So we’re trying to attract them from two different angles in some ways, if you’re looking at some of the top players.

Our PMs and our sales people came up to us, and they provide incentives to get companies to also connect. Our PMs had this great idea: why not try to get someone at that company to connect without selling them something? And that’s where Heap came in.

We did some analysis inside Heap’s GUI as well as SQL, and we were able to identify a bunch of these key events that led the users down a path of connecting. The series of actions that have led to a super high-value conversion—we’ve been able to look at it intently with Heap.

Bilal: I always feel that the way I or a team can have the biggest impact with data is surfacing self-evident truths, those where in retrospect your customer success or sales team will say, “Yes, I already knew that.” It means you did the right job.

One example of that was understanding, at Optimizely, our maturity matrix, our maturity curve for how our customer evolves over time. We were a very data-driven company, but in a qualitative sense then. We had processes and operations built around using data to make decisions, but it was often very much around qualitative assessments about qualitative data that we’d collected rather than actual things we’d logged.

So one thing we’d often been missing is we didn’t really know how often someone was actually using the product. Even in our value-based metric, which was impressions, or monthly unique visitors, we didn’t really keep track of how often people had done that over time.

So one question we had that we were trying to dive into one quarter was, how does a person’s utilization of Optimizely, as measured by the number of visitors that go through their site, change over time as the customer gets older?

There were a couple of problems with that.

  1. One, we weren’t storing impression data.
  2. Two, it was in a non-relational format and we had to get into a relational database.
  3. Three, in the relational database that it was in, we were deleting the data after 45 days, because no one really looked at this data, and Optimizely processes several billions events every week, so it didn’t make sense to store because we didn’t need it after that time, because it had already been displayed and logged and priced.
  4. And then after that, we didn’t aggregate it into a format that could be queried.

The first step was actually getting that data into one centralized place. Theo here spent a quarter doing that. It was interesting that it took a quarter to get this entire pipeline from Kafka to HTFS and then into a Postgres database to actually do that analysis.

The outcome then was that basically we were able to see a maturity curve, and we saw that on average (I know you say you don’t like averages) but on average we found that the general maturity curve of our enterprise customers was they started to reach a steady state around 9-12 months. And then what we came to was we told that to customer success people, and they said, “Yeah, we already knew that,” but engineering and product people didn’t.

But it also made sense. For an A/B test, you’re testing different things, as you are trying to figure out what you want to do, and then finally when you figure out what you want to do, you deploy it, and then you don’t really test that much anymore. But what it showed to the company was that we had a need to want to create new products. A/B testing gets you so far in a maturity metric, so you need to continue to grow it, especially for a company that charges by number of visitors. If you reach a plateau, you need to make more money.

Optimizely was launching a personalization product in the coming months, so we didn’t influence the decision to launch personalization, that was happening in development for years, but it helped to best present the story to engineering as we were going forward: look at this maturity curve, we need to get it to be continued to be growing exponentially, rather than plateauing at some period of time . I think the fact that it took a quarter to build that pipeline, was also a lot of hard work, and was fascinating to see.

So there was a question in the back, actually.

Oh, sure. If I understood correctly, basically, it’s like, I’m in the earlier stages of my own company, how important is it to sales and product communicate with engineering to display those metrics and stuff.

Yeah. One of the benefits is that there are only two of us full-time, and so sales and product is right next to engineering, and so it’s perfect. (laughs) It speaks to a larger thing that a lot of these problems emerge just because your teams get fragmented between different floors, and you’re not able to communicate, and I think actually the biggest improvement as you get to a larger company to help improve your data infrastructure is just to have a point person whose job it is to build bridges between all these different floors.

The biggest improvement as you get to a larger company to help improve your data infrastructure is just to have a point person whose job it is to build bridges.

That was kind of my job was to institute these processes, and basically often what I was doing was just walking between floors and bugging people. I would walk up to them, because some people didn’t respond to emails or something, so I would just walk up to them and ask, “Hey, can we do this?” or “Hey, what do you need?” And so I think that’s for whatever reason more effective in getting that communication across.

Ville: That really relates to my frustration with the aggregates as well. It seems to be the case that, especially when the company grows larger, that it’s easy to justify any kind of thing with data. Looking at some fancy dashboard showing some kind of health metric about the product or something, and then you can use that as an excuse to ignore, maybe, feedback from individual sales people or product managers and so forth.

It’s super important to keep that communication channel open and also, secondly, not only look at the high-level metrics, but, especially if you get some kind of negative feedback that something is not working, and that might not agree with the high-level aggregate you are seeing. And really deep in the data, and really a micro-level analysis at the level of even a single user or a single account and see that if you can find a single case that actually proves the hypothesis that things are not working, since it’s so easy to hide all kinds of issues in these high-level aggregates.

Bilal: Actually, going back to that question, it’s also easier to have that in a B2B sense, because you can just go to one account and probably there are five other accounts that are like that, and you have just 50 customers, that’s 10% of your people probably have that problem. And you can probably figure that out by just talking to your customer success team. You don’t need to do any analysis.

Matin: In your position as analytics leaders, you work with lots of different teams, like marketing, product, sales, customer success. What are their different workflows? How do they interact with data differently? And also, how do you keep them on the same page in terms of what matters with analytics?

Maurice: So, when we moved to Heap, in April of this year, we were coming from GA, Kissmetrics. Both of those data sources are just not at the user level, to put it simply. So we always had a ton of post-processing on top of that. It gets stuck in a database, and then everything turns into a project. You’ve got to work with the stakeholders, so, “What do you need, how do you need to slice this, segments, filters, etc.” As well, that could be the very first time you’re working with that piece of data. You have to clean it up. So that always has to happen.

But once we moved to Heap, we kind of switched the focus entirely. I like to say it’s an organizational change, but it’s really been driven by the product. Because our product team are really the owners of Heap now. They are creating all the new features, something changes on a page, back-end code changes, that works with our AJAX system, and they make the adjustments. But what’s happening there is they’re doing the adjustments to the events at the source, and that does change all the downstream reporting.

So we have to work with them; I sit right next to them now, versus meeting for one-offs. So I’m working with them throughout the day, really through any sort of analysis on a feature, or new changes, and that’s really where we’ve started to get such a good understanding of the product, what people are doing, accuracy, that’s the biggest thing. And then I can kind of funnel what I’m doing with them in the product team to everyone else, really.

So they’ve become some great use cases for Heap: here’s how all these users are converting from these upsell promos on the product side. Purely organic users. They’re doing this because they want to. It shows a great conversion rate. Now, how can we take this and have marketing do it? With some more inbound stuff, or just through promotional material, or content. So that’s been a really good organizational change for us.

Bilal: I guess I’d take a slightly different approach, just based on different experiences. I don’t think I’ve ever seen one tool be able to solve all problems for an organization, because each tool in some ways solves different problems. To the original question, each team has different metrics that they care about. Sales cares about MRR, which is often in Salesforce, based on opportunities or opportunities closed. Marketing cares about Marketing Qualified Leads, or lead conversion rates. Product cares about, depending on your company, some usage metric. Success is based on churn, also in Salesforce.

The differences then become, and the differences in the workflows, is depending on the metrics that they’re measuring their performance by, they have differences and difficulties to actually measure those based on each of those metrics have different latencies. Sales, again, happens quarterly. Churn happens, maybe, yearly. Usage might happen monthly or daily, and so the difficulties that are presented by each team, they are going to live in one system.

I find the only way that we’ve ever been able to get alignment has been less about tooling and more about processes and decisions from the top.

I find the only way that we’ve ever been able to get alignment has been less about tooling and more about processes and decisions from the top, wherein if leadership took an interest to align metrics or goals across different departments, that was the only way that you could get a coordinated view of a user in terms of how you were analyzing it. Otherwise, every different department had a different log or definition for how revenue or usage was being measured, and it was otherwise really hard to get alignment, and the operation’s about how you leverage the data you already have, rather than instituting or instrumenting new tools to fix your problem.

Basically I often find it’s just problems with people, not with the tooling, after a certain necessary instrumentation. If there’s no consensus from the top, then each department below them is not going to know what to agree upon.

Ville: My experience is similar. We have six main types of users of data at AdRoll.

Thinking in terms of a funnel, we have our big marketing department using the big marketing tools like Google Analytics, now increasingly Heap, and so forth. And maybe after that, when people are actually using our product, we are again using Heap, but also we have some internal dashboards built with Tableau that show how people use our products, and then after that, interestingly, we have a number of different roles that are all called data scientists, but in practice they do very different things.

So we have what I would call a product analyst, or a product data scientist. They really like focusing on using tools like Tableau, maybe, or Heap, to analyze really the usage of the product in a certain way, and often that means really working closely with the PMs.

And then we have the next layer, which is more like statisticians, building predictive models, to predict how the business is doing, how successful the customers will be, and so forth. And actually as of today, many of them, well practically all of them are using R. So they’re really into showing you we have many dashboards.

And then in the deepest, darkest corner of AdRoll, then you have machine learning engineers, so data science engineers, and they are interested in building the highest-performance convex optimizers or whatnot.

Marketers, data scientists, product analysts, statisticians, machine learning engineers… the interesting thing is that all of these people at the end of the day look at the same data.

The interesting thing is that all of these people at the end of the day kind of look at the same data. It’s still the same AdRoll, it’s still the same business. I think there is some value in having different tools, since I think, honestly, the tools really shape your thinking. It makes you a different person if you are using Shiny versus Tableau versus Heap and so forth.

And there’s definitely value in having those different viewpoints, but if one wants to reach some kind of a consensus, which is oftentimes useful, it is typically a pretty painful kind of reconciliation process, and I think it’s probably useful to have those checks and balances. You look at your marketing analytics and you get one kind of answer, and then you ask your predictive data scientist and they say another thing, and so forth. And then you try to reconcile what’s going on. Again, looking at the same time, at the very high level, doing the macro-analysis, and then at the same time, looking at the very micro level, forming new hypotheses from that end. So it’s definitely a number of different tools, a number of different people using different tools.