Lightbend Bets on Kubernetes to Converge DevOps and DataOps
Lightbend has announced an update to its Fast Data Platform that can now be deployed on Kubernetes clusters as part of an effort to unify DevOps and DataOps.
Company CEO Mark Brewer says version 2.0 version of the Fast Data Platform makes it easier to build and deploy containerized applications that need to access streaming data via, for example, Apache Kafka or Apache Flink software alongside data from the Apache Spark in-memory computing framework.
The Fast Data Platform is built on top of the Lightbend Reactive Platform, which employs a reactive model based on the Scala programming language to process data in parallel using a message-based runtime across multiple compute cores built on top of a Java Virtual Machine (JVM). Lightbend this week extended that capability via an alliance with Triplequote, which provides tools for optimizing Scala compile time by a factor of three on average.
Brewer says that as DevOps continues to evolve, the management of pipelines employed to feed data into applications needs to evolve. Historically, developers would build an application and wait for database administers to create schemas through which pipelines for access data would be created. Within the context of an integrated DevOps process, the developer now exercises more control over pipelines that are constructed within the limits of policies and guidelines—also known as DataOps—defined by IT operations teams.
Now, however, the rate of speed at which data is being accessed is exponentially faster, thanks to data streaming frameworks that enable applications to access data in near real-time. That shift is creating a need for the Fast Data Platform running on Kubernetes, which makes it easier to build microservices-based applications that access data streaming frameworks, Brewer says.
As those applications evolve, Brewer says a platform-centric approach will only become more important. In time, applications will be accessing multiple data streaming engines simultaneously because the various frameworks that are being made available are optimized for different classes of data.
It may take a while for the worlds of microservices based on containers and data streaming frameworks to converge completely. But it’s clear both classes of technologies are moving in adjacent orbits. As organizations increasingly move away from batch-oriented platforms to embrace a more real-time approach to processing transaction and analytics simultaneously, most of those new applications are being built and deployed as microservices that are easier to update, secure and manage. Organizations that have embraced microservices based on a Reactive programming model include Capital One and Credit Karma.
Of course, when it comes to integrating microservices-based applications with streaming data frameworks, there’s no shortage of runtime options. Lightbend is betting that its approach will be especially appealing to enterprise IT organizations have made major investments in JVMs.
Regardless of the path chosen, however, it’s now only a matter of time before most organizations realize that DevOps processes need to evolve rapidly to accommodate not just microservices and containers, but also data streaming frameworks that are rapidly transforming the state of DataOps.
