How we integrated a new, acquired platform - in a single sprint!

July 18, 20236 min read

In May of last year, our executive leadership delivered some exciting news: we’d just acquired Auryc Inc, a session replay company.

This was big news. Heap is a product analytics company, and so far our capabilities had been all quantitative. We’d always dreamed of being able to augment these capabilities with qualitative features like session replay, which would give our customers a different way to see how users interacted with their products.

Then the other boot dropped: my team was tasked with integrating Auryc’s session replay capabilities into Heap in a seamless way that looked native. In four months.

Oh, and we also had to deliver the following:

Self-serve trial capabilities for interested users
Auto-enforcement of subscription terms
Internal tooling for solutions and sales partners to set up subscriptions
Settings screens to easily configure privacy, security, and other items

Terrifying!

As it turned out, we not only beat this deadline - we were wildly successful. We hit all of our milestone targets, got a beta into users' hands in August, publicly launched on September 1, and created a sticky, loved-by-users feature that became a new business line for the company.

Sound unbelievable? It did to us too. Here’s how we made it happen.

I. BYOSR (Bring Your Own Session Replays)

To be fair, we did have a bit of a head start.

5 months earlier, we had already started adding session replay capabilities to prominent areas within Heap., The difference was that we were told to not build these functionalities in-house. Instead, we were going to partner with existing session replay companies, so our users could integrate their preferred tools.

Our goal at this point was to make it easy for our existing users to integrate a wide variety of other potential session replay tools into Heap. Because this meant that each integration would come with its unique quirks, we prioritized modularity and plug-and-play integration. This seemed like the best way to anticipate potential future collaborations with different session replay companies.

The main challenge was not over-coupling our solution to any one company’s session replay solution, while not over-convoluting the code with excessive abstraction to make the code hard to maintain. If you’re reading this and you’ve tried to make an integration generic in the past, you know what I’m talking about. It’s a delicate balance between generic/abstract touch points and code that is littered with factories, inheritance, and abstract interfaces that few people can decipher a few months later.

Throughout the process, we stayed laser-focused on not over-genericizing or complicating the solution. Instead of digging ourselves into a hole of excessive design patterns, we kept things simple. We didn’t anticipate fifty session replay partners - we expected maybe a handful. So we just stuck to some basic if-else checks in key spots in the code to determine which behavior to execute. We didn’t prematurely optimize for “what if we end up working with 100 session replay companies, this wouldn’t scale” we leaned into “this would work fine for five companies if we need to support more someday, we’ll figure it out then.”

We embarked on integrating heap.js (our client-side capture code) with various session replay SDKs from partner companies. This integration allowed us to capture session IDs and relevant user data, which we augmented into our Heap events. For instance, a click event captured by Heap integrated with a session replay partner might resemble the following in our database:

{ type: 'click', target_tag: 'div', target_class: 'article', target_text: 'This is an article with some text', session_replay: partner_company_id:9801h23d908123nf91234hf, hierarchy: util.toHierarchy({ elem: { tag: 'div', classes: ['container'] }, child: { elem: { tag: 'div', classes: ['article'], attrs: ['[data-section=News]', '[data-genre=Music]'], }, }, }, }

To propagate the session replay session IDs and their corresponding partner company IDs to the front end, we enhanced our web application's report queries. The user interface (UI) was then equipped to embed iframes of the partner company's session replay players into the relevant surface areas.

In other words, we noticed that all session replay solutions, at their core, are attaching a session ID to captured events, and this session ID is the magic key that lets you “lookup” the replay associated with a given moment in time. We decided the right level of abstraction was attaching this session ID (and in some instances a few bits of extra metadata) to each event stored in our database, so any event matching a user query in Heap, could then lookup the associated replay using knowledge of who the session replay partner is, and what the session ID is.

Within just one month, we successfully built a prototype by collaborating with one session replay company. We tested its functionality, and early-beta versions were distributed to a select group of existing Heap customers who were also users of the partner company's tools.

II. Change of Plans

In May 2022, while we were refining the early-beta version of BYOSR, we received a new directive from the company. Instead of partnering with session replay companies, Heap was going to provide native session replay capabilities. To do this, we were acquiring Auryc Inc. By September 1, we had to meet the goals described in the introduction.

Initially, this seemed like a daunting task. However, we adopted an engineering approach that broke the workload into manageable segments.

The key to our success lay in the flexibility and versatility we had built into the architecture. For example, throughout all of the codebase, we never made any assumptions about who the vendor was, details about how their session replay APIs worked, or the intricacies of their authentication. All of these details were abstracted behind easy-to-understand functions and highly configurable database tables.

Leveraging our existing modular design, we were able to integrate Auryc's session replay capabilities in a similar manner to any other session replay vendor. By integrating their SDK with heap.js on the client side and embedding their iframe into our UI surface areas, we swiftly made progress. In a single sprint, we achieved the core functionality of integrating their session replays into Heap.

Let me say that again: in a single sprint, we achieved the core functionality of integrating their session replays into Heap. We stayed laser-focused on functionality and in a single sprint’s spike got Auryc’s session replays plugged in and functional. This was undoubtedly the key to this whole crazy thing working out: we proved the technology could work, and we did it fast.

This gave us three and a half months to focus on refining the user experience: implementing self-serve trials, automating subscription overage enforcement, creating internal tools and monitoring, settings screens, and generally ensuring Auryc's player blended seamlessly with Heap's native interface.

Within approximately two months, we reached a stage where we felt confident enough to release a beta version to users. By September 1, we were confident enough in our session replay integration that we released it to the general public and allowed any Heap customer to start a session replay trial.

III. Lessons Learned

Had we not already architected session replay with a modular approach, the situation would have played out differently. If we had originally built session replay to only work for one specific partner, this most likely would have taken us a month or more to prototype. Similarly, if we had over-engineered our solution by introducing unnecessary design patterns too early, this could have also taken much longer. Instead, we hit the sweet spot of just generic enough to support another partner by keeping the code simple and easy to read, and so were able to quickly pivot it to support another similar session replay system.

Although not everyone may have the luxury of starting with a modular design, there is an important lesson to be learned here. When embarking on a new complex feature set for your application, you should almost always anticipate curveballs and code defensively for them. Don’t get me wrong: there definitely is a time and place to avoid premature optimization. But if the pressure isn’t tight, make things modular and reusable.

That said, if you’re going to make things generic and optimize a bit upfront, don’t overdo it. Don’t needlessly bring in cool design patterns just because you learned about them in college. Don’t introduce a complex state machine because you read a cool article about it one time. Prioritize writing code that is easily understood by your team (and yourself), and optimize “just enough” to easily flex or pivot if the requirements slightly change, don’t try to boil the ocean in the first iteration.

Similarly, communicate this philosophy to your product team partners. Product teams may suspect which requirements are most likely to change based on user feedback or product strategy and can help encourage a modular approach for those areas of the product. If your product partner knows where you've taken a modular approach, then they'll better understand if new requirements are a big lift or a small change.