Electronic Data Interchange came on the scene in the early 1970s, and ever since we’ve all been trying various ways to make different pieces of software work together. Although we’ve made a lot of progress, it still feels like getting software to cooperate is harder than it should be, and I think that’s because we’re framing the problem wrong. At its core, solving software integration isn’t about building a tool, solving software integration is about building a network.
Let me explain what I mean.
When integrating software using any of today’s methods, there is an inescapable tradeoff between control and cost to build. For example, Unified APIs like Merge (and even open-source variants like Nango) are turn-key solutions, but they might not have the level of detail you need. Connector-based iPaaS solutions like Tray.io and Mulesoft give you more control, but demand more investment. And of course, you can tailor hand-written integrations to your liking, but they’re quite painful to build and maintain.
None of these approaches do anything to make the act of integration fundamentally more efficient. All that’s going on is a customer choosing to offload more or less integration work onto their vendor; every approach sits somewhere on this continuum. Unified APIs are on the far end – they’ll take almost all the work off your hands. iPaas vendors lie in the middle – they’ve built things like connectors and a workflow engine, but you’ll need to handle the transformation logic yourself. For hand-written integrations you offload nothing at all.
The same type and amount of work is being done; what changes is the degree to which your vendor has done the work for you.
I think we can do better. By making a foundational improvement to the way we write and think about integrations, we can create an approach that offers superior control and convenience.
The central insight is this: if, for example, you’ve integrated your app with Hubspot, and Hubspot has an integration with Zendesk, like so:
then building a completely new integration from your app to Zendesk is redundant and a waste of effort. Intuitively, there’s no reason you shouldn’t be able to combine the info in the You → Hubspot link with the info in the Hubspot → Zendesk link to dramatically reduce or eliminate the work involved in connecting your app to Zendesk.
This property is transitive. Consider the case where we have a few more apps and integrations:
Theoretically, you ought to be able to auto-derive integrations between yourself and Salesforce, Zoho, and Vtiger – in fact, you should be able to do it for any app that is reachable from yours in the graph.
I call this concept a “Schema Network” – it’s a network where the nodes are apps and the edges are integrations. The idea is to generate new integrations by composing existing ones instead of starting from scratch, to the greatest extent possible – thus making the process of software integration dramatically more efficient.
Now, there are two major hurdles involved in getting this Schema Network concept production-ready. The first of these is performance: in our earlier example, naively moving the data across two integrations would be inefficient, and Hubspot probably wouldn’t consent to being a middleman between us and Zendesk. To address this, we can decouple data transformation from data movement. This means that, to pull data from Zendesk, we make a call to Zendesk to move the data from them to us, then we transform the data from Zendesk’s format to Hubspot’s format, then we transform the data from Hubspot’s format to our format (note that there are many ways we can squash those two transforms together ahead of time and pre-optimize them to make them faster):
The data goes directly from Zendesk to us, and then we apply the necessary transforms ourselves – Hubspot isn’t involved.
The second problem is coverage. What if Zendesk offers a field that Hubspot doesn’t? That field won’t be included in Hubspot’s integration with Zendesk, so it won’t be accessible to us. The most straightforward way to resolve this is to let services integrate with each other on a field-by-field basis:
In this new version of our network, every line represents a single field in an integration. As you can see, services now have the option to “split up ” their integrations between multiple sources when they want to (for example, when a single partner doesn’t offer everything they need). Fields are all still pulled at the same time, but depending on the structure of the network, they may be transformed in different ways. For example, here’s how we’d pull fields 1, 2, and 3 from Zoho:
This approach solves our earlier coverage problem involving Zendesk and Hubspot. Even though Hubspot doesn’t offer field 2, we can get it directly from Zendesk. Importantly, we haven’t given up any efficiency to get this benefit; we can still reuse our Hubspot integration to pull fields 1 and 3 from Zendesk. In fact, no matter who in the network we integrate with, we only need to make at most one connection per field to pull data from all available sources.
Let’s make our network a little bigger (and let’s scope our focus down to field 1 for now, so the diagram doesn’t get too cluttered):
As mentioned earlier, we can automatically integrate with any service reachable via one or more lines. This means we can integrate with Close, Pipedrive, and Insightly (but not Affinity, Copper, or Capsule, which are on their own little island). For example, here’s what it would look like for us to pull data from Close:
If we wanted to, we could build an integration from us to Copper, at which point we’d get integrations with Affinity and Capsule for free:
Or, if Close happened to integrate with Copper, we’d get all three integrations without needing to build anything:
Notice that when two Schema Networks connect at a single point, as in the example above, they form one larger Schema Network with complete connectivity. This “bottom-up” pattern of growth allows Schema Networks to develop organically, instead of being architected by a centralized entity.
From a developer perspective, Schema Networks are quite easy to use. In fact, they offer roughly the same experience as a Unified API – once you connect your app to the network, you get automatic integrations with every other app available. This is a lot like how you can connect your computer to an ISP and reach any host in the world, or how you can go to any commercial airport and – via a series of flights – reach any destination.
What’s special about Schema Networks is that they can achieve this ease-of-use while still staying flexible and comprehensive. A Unified API is handcrafted by a single group of people, while the view presented by a Schema Network evolves out of a large number of local integrations made by local actors. There is no central bottleneck on a Schema Network, which means many more integrators can be involved, and coordination costs are dramatically reduced. Because of these efficiency gains, Schema Networks are not only able to tie more services together than Unified APIs, they’re able to do so at a much higher level of detail, leading to richer, more sophisticated schemas from the end user’s perspective (it’s also worth mentioning that Schema Networks allows for integrators who are experts in their local domains, in contrast to the master curators of Unified APIs, who can’t be experts in everything).
Networks undergird large sections of our modern lives – we’ve already mentioned air travel and the internet, for example. The largest and most popular networks tend to share some common structural attributes – in particular, they tend to be what are called small world networks. Informally, a small world network has a hierarchical structure, where smaller links feed into large links between highly-connected hubs: consider the backbone links of the internet, or the large international airports in major cities.
Small world networks have the convenient property that any two nodes are connected by a fairly small number of “hops”. The emergent structure of a worldwide Schema Network would probably look like a small world network, with the largest and most popular applications being hubs.
Networks have proven to be a natural way to efficiently move people or packets through physical space at scale. If you think of software integration as the act of moving data through “format space,” Schema Networks should hopefully appear just as natural!