Introducing the Gated Bridge Pattern: Bridging Heterogeneity
Challenges and Context
When I joined Deutsche Telekom , I came across this unique set of challenges and constraints for introducing events integration bridging 10 countries we operate in Europe . DT had a strategy referred to as OneXP (One experience )that aimed to unify our customer-facing product portfolio across all operating countries. Here are the key challenges we faced w.r.t intergation of these central products with systems in operating countries.
- Centralized Development & Management: Products were developed, deployed, and managed centrally, requiring seamless data and process integrations with systems operating across various countries.
- Harmonized API Approach: While our strategy hinged on harmonizing APIs around REST, this approach introduced complexities. The APIs needed to accommodate diverse product requirements and changes across countries, demanding a sophisticated and efficient change management process.
- Multiplicity and Change Management: The shared APIs underpinning multiple products necessitated a more efficient process for coordinating changes. As these APIs were critical for cross-country product and service delivery, any modification impacted the entire network, highlighting the need for a streamlined, agile approach to integration and governance.
- Need for Fluid Integration: The existing integration mechanisms were rigid and often bottlenecked by synchronous workflows and governance models. We needed a solution that could support high throughput and asynchronous workflows, enabling a more fluid, semi-autonomous approach to system integration.
- Diverse Tech Landscapes: Each country operated its own IT systems with unique tech stacks, making it impractical to implement a one-size-fits-all solution. The challenge was to respect the autonomy and diversity of local systems while achieving a cohesive, efficient integration at a continental scale.
These challenges highlighted the need for a novel integration pattern that could accommodate the diverse, dynamic, and distributed nature of our systems while promoting efficiency, scalability, and adaptability.
The above image of a service personnel on top Golden Gate Bridge struck me as a dose of inspiration for a pattern that could be introduced to solve this. This led to the conception of the Gated Bridge pattern, inspired by the notion of creating a unified yet flexible pathway for event integration.
Solution: The Gated Bridge Pattern
This pattern is designed to address our integration challenges effectively and elegantly. Here’s how it works:
- Unified Channel for Event Flow: At its core, the Gated Bridge serves as a unified channel, allowing events to seamlessly flow between different “islands” — representing the operating countries. Each country maintains its internal systems and bridges( event buses) but connects to the central system through this gated bridge.
- Islands and Bridges: Each operating country is visualized as an island with its unique infrastructure. These islands are connected to the central system (the heart of the kingdom) through the Gated Bridge. Internally, countries manage their systems and workflows, but when it comes to integration with the central system, the Gated Bridge provides a standardized, efficient pathway.
- Gated Towers: At the edge of each island (country) and the central system, there are gated towers. These towers are manned by personnel ( analogy in thoery a system referred to as gate) who monitor and manage the flow of events. They ensure that only relevant, agreed-upon events pass through the bridge, maintaining security and efficiency. This represents the governance aspect of the pattern, allowing for flexibility and control in what is shared and integrated.
- Central and Local Specifications: As events travel from the local systems to the central systems, they are transformed or maintained according to flexible, pre-agreed specifications. This ensures that while there is a standard method of communication, there is also room for customization and adaptation to local needs.
- Efficiency and Decentralization: The USP of the Gated Bridge pattern lies in its ability to bring your own event buses. The connecting bridges (central integration points) are constructed centrally, requiring no effort from local provinces. Gates deployed at the edges of each province can be reused, further reducing complexity and cost. Local provinces need only map the events of interest for transmission, significantly simplifying the integration process.
- Technological Implementation: To bring this pattern to life, we utilized off-the-shelf components and leading technologies. NiFi was deployed as the gate for its robust data routing and transformation capabilities. For the central bridge, Kafka was chosen for its scalability and reliability in handling event streams. However, the architecture remains flexible to accommodate other technologies like events over REST, ensuring adaptability to different tech ecosystems.
By implementing the Gated Bridge pattern led by Shashidhar Singhal , we transformed our integration landscape. Integration times for asynchronous workflows reduced drastically from months to days. We established a fully decentralized governance hub, allowing for quicker, more efficient management and adaptation of specifications.
Get Arun Joseph’s stories in your inbox
Join Medium for free to get updates from this writer.
The result was a significant enhancement in our integration capabilities, supporting a wide array of products and handling substantial traffic volumes with ease and agility.
This now powers our events integration with 9 european countries, 5 OneXP Products and is also used for unified vendor Event intergraitons like Medallia , Braintree.
A fully reusable and self governable events pipeline for any products , external vendors has cut the solution rollout time by more than 70 percent.
Technical Brief: Gated Bridge Pattern
In the development of the Gated Bridge pattern, several novel technical approaches were employed to achieve a truly decentralized events integration platform and governance hub. While a detailed exploration of these technicalities will be covered in a follow-up article, here is a brief overview to provide a taste of the innovations and technical robustness of the solution:
- Decentralized Event Buses: The architecture allows for various event buses, supporting a bring-your-own-bus model. This flexibility ensures that regardless of the underlying technology stack in different countries, the system can integrate seamlessly, allowing for a truly decentralized approach.
- Dynamic Data Routing: Apache NiFi was utilized as the gate for its exceptional capabilities in data routing, transformation, and system mediation. Its design allows for dynamic responses to changing data flows, a crucial feature for managing the diverse and dynamic nature of events across various systems.
- Flexible Specifications: A key aspect of the Gated Bridge pattern is its support for flexible and light-touch specifications. This allows for adaptability in how events are formatted and transmitted, catering to local needs while maintaining overall coherence and compatibility with the central system.
- Decentralized Governance Hub: The governance aspect of the Gated Bridge pattern is particularly innovative. By leveraging a decentralized model, it allows for quicker decision-making, efficient change management, and a reduction in the overall time and resources required for governance processes. This approach ensures that while control and standards are maintained, the system remains agile and responsive to changes.
A follow up detailed technical article will delve deeper into each of these components, providing insights into the architecture, technologies, and strategies that make the Gated Bridge pattern .
Closing Thoughts
The Gated Bridge pattern represents a major milestone forward in the field of events integration and governance for the unique challenges we faced . As we continue to build on this foundation, the potential for innovation and improvement is boundless. If you have faced similar challenges look forward to further discussions, insights, and advancements as we collectively explore the full capabilities and future applications of the Gated Bridge pattern.
Gated Bridges :
