5 CNCF Projects for Streaming and Messaging
Event-driven architecture is becoming more commonplace within cloud-native ecosystems. By focusing on events, services can automatically trigger additional functionality, reducing manual legwork and evolving the traditional request-response communication paradigm. In such streaming environments, services either “subscribe” to an event source or “publish” events for other services to listen to. Often, they conduct a combination of both actions. Connecting services in this manner can eliminate over-fetching concerns.
Streaming and events manager platforms, such as Kafka, RabbitMQ or NATS, add a layer to a system of services, explicitly focusing on transmitting messages between services. This helps applications remain decoupled from one another yet still allows the ability to communicate in a common format. Below, I’ll review the CNCF projects related to cloud-native streaming and messaging. These five open source packages are helpful utilities to help you implement a publish-subscribe pattern in your service ecosystem.
1. CloudEvents
A specification for describing event data in a common way
One downside of the proliferation of events is that each event publisher system describes events differently. When you’re connecting a large number of services, you will likely be dealing with various event sources and protocols, requiring conversions. CloudEvents aims to solve these event handling issues. It’s a specification to describe events consistently which could ease integrating event sources and improve interoperability in the field.
CloudEvents specification dictates the formatting for necessary information, such as id, type, source and specversion, as well as for other optional values. Here is an example CloudEvent represented in JSON in accordance with the specification:
{
"specversion" : "1.0",
"type" : "com.github.pull_request.opened",
"source" : "https://github.com/cloudevents/spec/pull",
"subject" : "123",
"id" : "A234-1234-1234",
"time" : "2018-04-05T17:31:00Z",
"comexampleextension1" : "value",
"comexampleothervalue" : 5,
"datacontenttype" : "text/xml",
"data" : "<much wow=\"xml\"/>"
}
CloudEvents is part of the CNCF’s Serverless Working Group. At the time of writing, the CloudEvents project is in incubating status with CNCF.
2. NATS
A high-performance server for NATS.io, the cloud- and edge-native messaging system
NATS is a highly performant and portable communications layer well-suited for distributed systems. The NATS Server is a publish-subscribe server that uses JetStream for real-time data streaming. The community supports over 40 different client libraries in various languages, making it easy to communicate with a NATS server regardless of whether the client is mobile, edge, desktop, web, on-premises or in the cloud. The project was originally created by Derek Collison and is supported by Synadia. At the time of writing, NATS is an incubating project within the CNCF.
3. Pravega
Storage for a streaming world
Another cloud-native streaming architecture to consider is Pravega, a CNCF sandbox project. Building upon the success of streaming platforms such as Pulsar and Kafka, Pravega takes it a step further to support a wider range of applications and features. Some of the unique capabilities of Pravega include long-term retention, ingestion of large amounts of data and consistent state replication. In-stream auto-scaling is a unique feature of Pravega, which could help respond to changes in ingestion rates for certain partitions. Some potential use cases for Pravega include predictive maintenance for IoT, real-time billing and cyberthreat detection.
4. Strimzi
Apache Kafka running on Kubernetes
Strimzi offers a simple way to get Kafka, the popular message bus, up and running on Kubernetes with only a few commands. Strimzi provides container images and operators that simplify the process of deploying and running Kafka clusters and components, configuring access to Kafka and securing access to Kafka, among other capabilities. Strimzi employs four operators to accomplish these tasks: The cluster operator, entity operator, topic operator and user operator. Strimzi is compatible with various Kubernetes deployment configurations. To get started with Strimzi, check out the latest quick start guide.
5. Tremor
An early-stage event processing system for unstructured data
Tremor is an event-processing system to aid platform engineers working in high-density throughput situations where constant uptime is critical. Tremor is ripe with functionality and could replace several other data processing and distribution tools. The tool was designed at Wayfair and has been in production use since 2018, and it continues to evolve with additional use cases. But as it was built with e-commerce in mind, engineers supporting similar situations may find it especially useful. To get started with Tremor, you can pull the Docker image here.
Publish-Subscribe Tools and Projects
In our hyper-connected digital economy, a robust connective tissue is necessary to reliably and securely connect services. Platform middleware like the ones above can help enable a publish-subscribe pattern, a model that has become necessary to enable decoupled architecture and real-time streaming. These packages and projects are all contributing in different ways to incorporating eventing into the cloud-native ecosystem.