Kubeflow Pipelines on Tekton Advance MLOps

The maintainers of Kubeflow, a machine learning operations (MLOps) platform built on top of Kubernetes, have made available a 1.0 release of Kubeflow Pipelines on Tekton (KFP-Tekton) project based on the same open source pipelines now being developed under the auspices of the Continuous Delivery (CD) Foundation.

The 1.0 release of Kubeflow Pipelines on Tekton (KFP-Tekton) makes it possible for data science teams to take advantage of capabilities such as graph recursion, conditional loops, caching, any sequencer and dynamic parameters that have been recently added to the core Tekton project via a Python interface. It also enables extract, transform and load (ETL) bindings that will make it easier to move the large amounts of data typically required to train artificial intelligence (AI) models.

IBM has also announced that Watson Studio Pipelines, which is built on top of KFP-Tekton, is now available in open beta.

Animesh Singh, IBM distinguished engineer and CTO for Watson Data and AI Open Tech at IBM, says data science teams are now applying many of the best practices defined by DevOps teams to build applications to the construction of AI models. Containers have emerged as the primary software artifact that data science teams are employing to build AI models because they enable data sets to be encapsulated in a way that makes building a massive AI model manageable. Kubernetes, as a de facto standard for orchestrating containers, is naturally gaining traction as well.

The challenge is that Kubernetes is complex, so providers of MLOps platforms are providing a layer of abstraction on top of Kubernetes that makes the container orchestration platform more accessible, notes Singh.

As the development of AI continues to advance, there is now a significant amount of debate over the degree to which organizations will require a dedicated MLOps platform to build AI models or whether AI models are themselves just another type of software artifact to be managed within the context of a DevOps workflow. IBM, for its part, maintains that a separate MLOps platform is required because the processes employed to build AI models tend to be much more recursive than traditional software development, says Singh.

It’s too early to say just how the building of AI models and traditional applications might converge, but ultimately an AI model needs to be deployed within the context of an application. Many data science teams require months to build an AI model that is ready to be deployed in a production environment. Aligning those efforts with the application development life cycle that an organization uses to build the applications in which an AI model is embedded can prove challenging. Much like any other software component, AI models need to be regularly updated as the assumptions used to build the model change or drift away from those assumption sets as more data becomes available.

Regardless of the approach to building AI models, the one thing that is certain is there will soon be a lot more of them. The challenge now is determining how best to manage the processes required to build and maintain them.

Mike Vizard

Mike Vizard is a seasoned IT journalist with over 25 years of experience. He also contributed to IT Business Edge, Channel Insider, Baseline and a variety of other IT titles. Previously, Vizard was the editorial director for Ziff-Davis Enterprise as well as Editor-in-Chief for CRN and InfoWorld.

Mike Vizard has 1614 posts and counting. See all posts by Mike Vizard