Centralized data management teams focus on managing reusable corporate ESBs, centralized iPaaS solutions, and corporate APIs across the organization. While these teams may not be going away overnight, the paradoxical question of centralization remains: have our our attempts to reduce redundancy and centralize expertise become an intolerable bottleneck for expanding organizations?
The Traditional Approach to Data Management
The traditional approach to data management is virtually synonymous with centralization. Data flows are centralized with an ESB and/or iPaaS tooling platform, and a centralized team ensures that data operations and governance are concentrated into a horizontal function.
his approach lends itself well to maturing startups and mid-sized companies where data decentralization and duplication of data practices occurred out of organic, unplanned growth. A common example would be when a go-to-market team builds their own data pipeline parallel to the BI team’s data pipeline, parallel to the ERP team’s data pipeline. In cases where the company is small and system counts are relatively few, centralization makes sense as you invest in one team and reap the benefits across many vertical functions.
The Centralized Approach Has a Scaling Problem
Over time, the limitations of centralized data teams become evident in a few different ways. The most evident area is that of shared human bandwidth. The “take a number” approach to demand management quickly wears thin on data team customers, and prompts such customers to consider crafting their own teams to foster agility and empowerment.
The second scaling challenge is technical in nature. For starters, not all app-to-app connectivity requires heavy, centralized data integration prowess. Many solutions leverage point-and-click connectors with no development expertise required. Yet over time, these standard, simplistic connections fail to cover all use cases, and thus require additional middleware to cover above-and-beyond use cases. A common example is connecting a sales and marketing system as shown below:
When these two systems are initially connected, the initial off-the-shelf connector may provide sufficient data synchronization capabilities for moving accounts and leads around. But over time, the need to sync opportunities, which isn’t covered in the default connector, becomes a real need. Teams can either rip-and-replace the original connector, or augment it with a centralized iPaaS type solution.
Adding even more complexity to the mix are platform-centric iPaaS solutions. Major business systems, or “planets,” nowadays come with their own data integration platforms. For example:
- Workday Integration Platform
- ServiceNow Integration Hub
- SAP Cloud Integration Gateway
Enterprise software vendors understand the bottlenecks associated with centralized data teams, and as shown in the examples above, wasted no time creating solutions to fill the void for platform owners.
Taking a Page from DevOps and Microservices
When a town is small, it may rely on the county or perhaps even the state to provide certain services such as water, policing, and fire protection. As the town matures into a mid-sized city, it builds police and fire departments of its own. If the city grows into a major metropolis, these police and fire departments will scale out into precincts, and potentially fragment into other agencies entirely. College campuses get their own police departments, airports get their own fire departments, and so on.
In the modern era, pods of responsibility and bounded ownership have been popularized through DevOps and microservice patterns. At the application level, monoliths are generally easy to understand, but are hard to scale and maintain without impacting the entire system. So from an enterprise point of view, why not apply the same methodology? In other words, allow some degree of controlled redundancy in order to foster autonomy and agility.
This sounds wonderfully empowering in theory. In practice, however, you need multiple teams of highly competent data professionals with federated standards across them. This is easier said than done.
The Future is Decentralized and Complex
Here, the “future” is really already starting to happen today. In reality, most organizations don’t map out their entire enterprise physical data flows, so this complexity is occurring without overarching oversight. Like a dense fog that rolls in under the cover of darkness, you don’t realize the complexity has enveloped your organization until first light. And by then, it’s too late to squash the momentum.
If we map out core business systems (planets) and their bolted-on auxiliary systems (satellites) and we additionally trace connectivity by data integration type, it doesn’t take long for the picture to look spaghetti-esque:
Not only are there multiple data connectivity types, but each system may embrace each type simultaneously. With this type of data architecture, “source of truth” is no longer a system, it’s many systems and it’s point-in-time. Years ago I wrote an article on this phenomenon and labeled it as shards of truth. It’s like a science fiction film based on string theory, where there are many interconnected realities; not just one. The “David Torre” living in SAP may be similar, but not the same as the “David Torre” living in ServiceNow.
With enterprise oversight, this approach may be acceptable. However without at least some semblance of federated governance, we enter “The Bizarro Jerry” episode of Seinfeld where known characters look familiar, yet act in strange and unpredictable ways. Data quality, golden records, and lineage tracing go out the window and the pressing question of “which system has the right copy of data” frankly boils down to subjective opinion.
What this Means for Future Data Governance
Like the airport that needs its own fire department, scaling out functions may be inevitable. The key is making the scale-out process deliberate and planned rather than organic and haphazard.
When a planet takes on its own iPaaS, such as when the HRIS team decides to bypass the corporate iPaaS and use its own, there needs to be a degree of not only competence, but coordination between other data teams. Again drawing on the fire department analogy: you don’t reinvent the wheel when you build a fire department. You use standard tools, processes, and training. Moreover, you coordinate with other fire departments in case you need backup help, need to share resources, or serve common customers.
This scaled out mindset is very different from a traditional team building mindset. A scaled out strategy is cross-functional, it’s inherently complex, and it requires holistic (enterprise-scope) understanding of people, processes, and technology. If this sounds familiar, I can help.
