5 CNCF Projects for Streaming and Messaging

Event-driven architecture is becoming more commonplace within cloud-native ecosystems. By focusing on events, services can automatically trigger additional functionality, reducing manual legwork and evolving the traditional request-response communication paradigm. In such streaming environments, services either “subscribe” to an event source or “publish” events for other services to listen to. Often, they conduct a combination of both actions. Connecting services in this manner can eliminate over-fetching concerns.

Streaming and events manager platforms, such as Kafka, RabbitMQ or NATS, add a layer to a system of services, explicitly focusing on transmitting messages between services. This helps applications remain decoupled from one another yet still allows the ability to communicate in a common format. Below, I’ll review the CNCF projects related to cloud-native streaming and messaging. These five open source packages are helpful utilities to help you implement a publish-subscribe pattern in your service ecosystem.

1. CloudEvents

A specification for describing event data in a common way

GitHub | Website

One downside of the proliferation of events is that each event publisher system describes events differently. When you’re connecting a large number of services, you will likely be dealing with various event sources and protocols, requiring conversions. CloudEvents aims to solve these event handling issues. It’s a specification to describe events consistently which could ease integrating event sources and improve interoperability in the field.

CloudEvents specification dictates the formatting for necessary information, such as id, type, source and specversion, as well as for other optional values. Here is an example CloudEvent represented in JSON in accordance with the specification:

{
    "specversion" : "1.0",
    "type" : "com.github.pull_request.opened",
    "source" : "https://github.com/cloudevents/spec/pull",
    "subject" : "123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

CloudEvents is part of the CNCF’s Serverless Working Group. At the time of writing, the CloudEvents project is in incubating status with CNCF.

2. NATS

A high-performance server for NATS.io, the cloud- and edge-native messaging system

GitHub | Website

NATS is a highly performant and portable communications layer well-suited for distributed systems. The NATS Server is a publish-subscribe server that uses JetStream for real-time data streaming. The community supports over 40 different client libraries in various languages, making it easy to communicate with a NATS server regardless of whether the client is mobile, edge, desktop, web, on-premises or in the cloud. The project was originally created by Derek Collison and is supported by Synadia. At the time of writing, NATS is an incubating project within the CNCF.

3. Pravega

Storage for a streaming world

GitHub | Website

Another cloud-native streaming architecture to consider is Pravega, a CNCF sandbox project. Building upon the success of streaming platforms such as Pulsar and Kafka, Pravega takes it a step further to support a wider range of applications and features. Some of the unique capabilities of Pravega include long-term retention, ingestion of large amounts of data and consistent state replication. In-stream auto-scaling is a unique feature of Pravega, which could help respond to changes in ingestion rates for certain partitions. Some potential use cases for Pravega include predictive maintenance for IoT, real-time billing and cyberthreat detection.

4. Strimzi

Apache Kafka running on Kubernetes

GitHub | Website

Strimzi offers a simple way to get Kafka, the popular message bus, up and running on Kubernetes with only a few commands. Strimzi provides container images and operators that simplify the process of deploying and running Kafka clusters and components, configuring access to Kafka and securing access to Kafka, among other capabilities. Strimzi employs four operators to accomplish these tasks: The cluster operator, entity operator, topic operator and user operator. Strimzi is compatible with various Kubernetes deployment configurations. To get started with Strimzi, check out the latest quick start guide.

5. Tremor

An early-stage event processing system for unstructured data

GitHub | Website

Tremor is an event-processing system to aid platform engineers working in high-density throughput situations where constant uptime is critical. Tremor is ripe with functionality and could replace several other data processing and distribution tools. The tool was designed at Wayfair and has been in production use since 2018, and it continues to evolve with additional use cases. But as it was built with e-commerce in mind, engineers supporting similar situations may find it especially useful. To get started with Tremor, you can pull the Docker image here.

Publish-Subscribe Tools and Projects

In our hyper-connected digital economy, a robust connective tissue is necessary to reliably and securely connect services. Platform middleware like the ones above can help enable a publish-subscribe pattern, a model that has become necessary to enable decoupled architecture and real-time streaming. These packages and projects are all contributing in different ways to incorporating eventing into the cloud-native ecosystem.

Bill Doerrfeld

Bill Doerrfeld is a tech journalist and analyst. His beat is cloud technologies, specifically the web API economy. He began researching APIs as an Associate Editor at ProgrammableWeb, and since 2015 has been the Editor at Nordic APIs, a high-impact blog on API strategy for providers. He loves discovering new trends, interviewing key contributors, and researching new technology. He also gets out into the world to speak occasionally.

Bill Doerrfeld has 80 posts and counting. See all posts by Bill Doerrfeld