Typically, an enterprise service bus (ESB) or other integration solutions like extract-transform-load (ETL) tools have been used to lớn try to lớn decouple systems. However, the sheer number of connectors, as well as the requirement that applications publish and subscribe to the data at the same time, mean that systems are always intertwined. As a result, development projects have sầu lots of dependencies on other systems & nothing can be truly decoupled.

The need for integration—a never ending story

No matter in which enterprise you work, no matter when your company was founded, you will have sầu the requirement lớn integrate your applications with each other to implement your business processes.

This includes many different factors:

Technologies (standards lượt thích SOAPhường, REST, JMS, MQTT, data formats like JSON, XML, Apache Avro or Protocol Buffers, open frameworks like Nginx or Kubernetes and proprietary interfaces lượt thích EDIFACT or SAPhường BAPI)Programming languages and platforms lượt thích Cobol, Java, .NET, Go or PythonApplication architectures like Monolith, Client Server, Service-oriented Architecture (SOA), Microservices or ServerlessCommunication paradigms like batch processing, (near) real time, request-response, fire-and-forget, publish subscribe, continuous queries & rewinding

Many enterprise architectures are a bit messy—something like this:


Every company needs khổng lồ solve sầu these spaghetti architectures. Depending on the decade, you either bought something lượt thích an ETL tool lớn build batch pipelines or an ESB to lớn design a SOA. Some products also changed their names. Today, you are offered things lượt thích middleware messaging, an integration platsize, microservice gateway or API management. The branding & product name vị not matter. You always see the same picture as a solution to move away from your spaghetti architecture to a central integral box in the middle, like this:


This rarely worked well in practice, unfortunately. Most SOA projects in the last two decades failed. Instead of using an ETL tool or ESB for this, enterprises are now moving on to lớn a streaming platsize to lớn solve this issue. Is this the next bubble on the market? Just a new term? Or, did something really change to allow successful integration across an enterprise—whether you integrate legacy mainframes, standard applications lượt thích CRM & ERPs, modern microservices built with any programming platsize, or public cloud services? Why are companies now migrating to Apache Kafka lớn build this streaming platform? Why is everytoàn thân happy & talking about this at conferences, tech talks và blog posts? How does it compare khổng lồ an ESB or ETL tool?

The next sections will answer all these questions, và explain the reason và differences between the ecosystem of Apabít Kafka & other existing integration solutions.Quý Khách vẫn xem: Service bus là gìQuý khách hàng sẽ xem: Enterprise service bus là gì

Event-driven processing và streaming as a key concept in the enterprise architecture

An event streaming platform (you can also enter another buzzword here) leverages events as a core principle. You think in data flows of events and process the data while it is in motion.

Bạn đang xem: Service bus là gì

Many concepts, such as event sourcing, or kiến thiết patterns such as Enterprise Integration Patterns (EIPs), are based on event-driven architecture. The following are some characteristics of a streaming platform:

Event-based data flows as a foundation for (near) real-time and batch processing. In the past, everything was built on data stores (data at rest), making it impossible to lớn build flexible, agile services lớn act on data while it is relevant.Scalable central nervous system for events between any number of sources và sinks. Central does not mean one or two big boxes in the middle but a scalable, distributed infrastructure, built by kiến thiết for zero downtime, handling the failure of nodes và networks and rolling upgrades. Different versions of infrastructure (lượt thích Kafka) và applications (business services) can be deployed và managed in an agile, dynamic way.Integrability of any kind of applications và systems. Technology does not matter. Connect anything: programming language, APIs lượt thích REST, open standards, proprietary tools & legacy applications. Speed does not matter. Read once. Read several times. Read again from the beginning (e.g., add new application, train different machine learning models with the same data).Distributed storage for decoupling applications. Don’t try lớn build your own streaming platsize using your favorite traditional messaging system và in-memory cache/data grid. There is a lot of complexity behind this and a streaming platform simply has it built-in. This allows you lớn store the state of a microservice instead of requiring a separate database, for example. Stateless service & stateful business processes.

Xem thêm: Hệ Thống Giải Đấu Giải Liên Minh Huyền Thoại Thế Giới Hiện Nay

Business processes typically are stateful processes. They often need to lớn be implemented with events và state changes, not with remote procedure calls & request-response style. Patterns lượt thích event sourcing và CQRS help implement this in an event-driven streaming architecture.

Benefits of a streaming platkhung in the enterprise architecture

A streaming platform establishes huge benefits for your enterprise architecture:

These characteristics build the foundation of a streaming platsize, the beginning of your successful digital transformation. With services implementing a limited phối of functions, and services being developed, deployed and scaled independently, you get shorter time lớn results and increased flexibility. This is only possible with a streaming platform having the above sầu characteristics.

Use cases for a streaming platform

Here are some generic scenargame ios for how you can leverage a streaming platform with the characteristics discussed above:

Event-driven processing of big data sets (e.g., logs, IoT sensors, social feeds)Mission-critical, real-time applications (e.g., payments, fraud detection, customer experience)Decoupled integration between different legacy applications và modern applicationsMicroservices architectureAnalytics (e.g., for data science, machine learning)

Producer & consumers of different applications are really decoupled. They scale independently at their speed & requirements. You can add new applications over time, both on the producer và consumer side. Often, one sự kiện is required lớn be consumed by many independent applications to complete the business process. For example, a khách sạn room reservation needs immediate payment fraud detection in real time, the ability lớn process the booking through all backkết thúc systems in near real time, và overnight batch analytics to lớn improve sầu customer 360, aftersales, khách sạn logistics & other business processes.

Xem thêm: Giả Lập Chơi Game 4 Nút Nes Console Trên Pc, Giả Lập Máy Điện Tử 4 Nút Trên Pc

Think about some more use cases that you can build easily with a real decoupled system that is still a well-integrated and scalable streaming platform:

Selling before the customer left the storeAborting a transaction before the fraud happenedReplacing a part of a manufacturing machine before it breaksInforming customers if a flight or train is late (plus sending updates, rebooking or a voucher) You name it—the menu goes on.

Big bang from batch lớn real time?

Now, you underst& the added value of a real decoupled, scalable streaming platsize. So, vày I have to introduce this as a central data platsize for all of our applications?

The following shows the streaming maturity Model that we use to identify the current situation and planning in large enterprises: