Istio Extension: Secure Multi-Cluster Communication

Istio is an open source service mesh that seamlessly integrates with Kubernetes. Istio is typically deployed in a single Kubernetes cluster, but as the adoption of Kubernetes increases, the deployment of Istio across multiple clusters is also on the rise. The motivation to deploy multiple Kubernetes clusters is being driven by the need to separate dev/test/stage cycles, as each uses different deployment environments. Additionally, some applications require different scaling (e.g. horizontal scalability) and resiliency requirements or geolocation that requires different clustered deployments.

Service Mesh in Multi-Cluster Deployments

Classic service mesh deployment assumes a single Kubernetes cluster and a single service mesh control-plane deployment. Using service mesh in multiple Kubernetes clusters that need to communicate with each other creates configuration and security challenges. These challenges stem from the need to identify services outside their cluster (“external” cluster) and consequently to authenticate and authorize them.

Istio offers few options to deploy service mesh in multiple Kubernetes clusters. The pros and cons of this will be discussed, as well as the need for a distributed option using multi-mesh in more secure deployments.

Single-Mesh Deployment

You can deploy a single-service mesh (control plane) over a fully connected multi-cluster network, and all workloads can reach each other directly without an Istio gateway, regardless of the cluster on which they are running.

Multi-Mesh Deployment

You can also configure a multi-cluster environment with a multi-mesh, in which each cluster has its own mesh, controlled by one Istio control plane for each domain, and the service exposure between domains is handled separately. In this multi-mesh configuration, each service mesh is treated as an independent “fault domain,” but with inter-mesh communication. With a multi-mesh deployment, you have a greater degree of isolation and availability, but it increases the setup complexity. Meshes that are otherwise independent are loosely coupled together using ServiceEntries and Ingress Gateway, using a common root CA as a base for secure communication. From a networking standpoint, the only requirement is that the ingress gateways be reachable from one another. Every service in a given mesh that needs to be accessed requires a ServiceEntry configuration in the remote mesh.

The multi-mesh configuration has inherent benefits including:

  • Fault isolation and failover: If cluster-1 goes down, failover to cluster-2.
  • Use various control plane models to support different levels of availability.
  • Team or project isolation: Each team runs its own set of clusters.

Multi-Mesh Security Considerations

In multi-mesh deployments, security can become complicated as the environment grows and diversifies. There are security challenges in authenticating and authorizing services between the clusters. The local Mixer (services policies and telemetries) needs to be updated with the attributes of the services in the neighboring clusters. Otherwise, it will not be able to authorize the services when they reach the cluster. To achieve this, each Mixer needs to be aware of the workload identities and their attributes in neighboring clusters.

Federation of granular workload identities (mTLS certificates) and service attributes across multi-mesh control planes can be done in the following ways:

  • Kubernetes Ingress: Exposing HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can terminate SSL/TLS and offer name-based virtual hosting. Yet, it requires an Ingress controller for fulfilling the Ingress rules.
  • Service mesh gateway: The Istio service mesh offers a different configuration model, Istio Gateway. A gateway allows Istio features such as monitoring and route rules to be applied to traffic entering the cluster. An ingress gateway describes a load balancer operating at the edge of the mesh that receives incoming HTTP/TCP connections. It configures exposed ports, protocols, etc. Traffic routing for ingress traffic is configured instead using Istio routing rules, exactly the same way as for internal service requests.

Next-Generation Security for Multi-Cluster Multi-Mesh Architecture

To address the mesh deployment on multi-clusters, DevSecOps are leveraging multi-mesh deployment with replicated control plane elements in each cluster to provide maximum flexibility. Then, a method to share the root certificate authority (root CA) to all clusters was selected and Istio Ingress applied to each cluster to automate the federation of certificates/identities and service attributes between clusters. The Istio ingress gateway then connects the neighboring cluster with other clusters and uses the DNS configuration for external services in all clusters. The Istio ServiceEntry can then be automated for external services in each cluster, leveraging a VirtualService for each external service IP/FQDN.

Summary

Extending the service mesh to address multi-cluster deployments generates a lot of interest these days. There are several options to extend the Istio service mesh to address this challenge, using single or distributed control planes. A new generation of identity-based workload protection technologies is providing a secure multi-mesh multi-cluster solution, with granular identities federation, allowing you to define granular and effective security policies across clusters.

Ran Ilany

Prior to founding Portshift, Ran Ilany held leadership security roles at Checkpoint and MainControl (acquired by IBM). As founder of Bladefusion, Ran formulated the industry standard “Multi Bladed Security Appliance”. He has authored over 3 security-related patents and wrote “Corporate Security & Network Consolidation” published by Larstan Publishing 2004.

Ran Ilany has 1 posts and counting. See all posts by Ran Ilany