Common Istio Errors and How to Solve Them

Istio is an open source service mesh technology that lets you connect, secure, control, monitor and run microservices. Istio is compatible with microservices running on either containers or virtual machines (VMs) running on any platform and written in any programming language.

Istio is platform-agnostic but is often used in Kubernetes clusters. In the context of Kubernetes, Istio can provide efficient communication between services and pods, with security both at the application and network level. Istio adds functionality that is missing from plain Kubernetes—including traffic management, failure handling and circuit breaking.

Like any complex system, Istio is prone to errors, which can have a significant impact on production Kubernetes clusters. Let’s take a look at the most common configuration errors faced by Istio users and how to fix them.

Common Configuration Errors 

Rejection of Seemingly Valid Configuration

To investigate why the configuration was rejected:

Use the following commands to understand why the configuration got rejected:

istioctl validate -f

istioctl analyze

Use the above commands in an istioctl command line environment that has a version similar to the control plane version.

The problem commonly arises because of syntax issues in the YAML configuration, such as indentation and array notation errors. Hence, manually check that the configuration is correct and use Istio’s API reference during the process.    

Receiving ‘no such hosts’ or ‘no endpoints available’ Errors

To investigate errors about no endpoints available: 

1. Run the following commands and ensure that the istiod pod(s) are functioning and the endpoints are ready:

kubectl -n istio-system get pod -lapp=istiod

kubectl -n istio-system get endpoints istiod

1. If the endpoints or the pods aren’t ready, use the following commands to investigate why the webhook pod cannot start and isn’t serving traffic:

for pod in $(kubectl -n istio-system get pod -lapp=istiod -o jsonpath='{.items[*].metadata.name}’); do \

    kubectl -n istio-system logs ${pod} \

done

for pod in $(kubectl -n istio-system get pod -lapp=istiod -o name); do \

    kubectl -n istio-system describe ${pod} \

Done

Kubernetes CrashLoopBackOff

CrashLoopBackOff is a state in which a container inside a Kubernetes pod is stuck in a restart loop: It gets started, crashes and restarts. Kubernetes waits for a back-off time between each restart for you to fix the error, and the duration keeps increasing with every restart. Hence, CrashLoopBackOff indicates an underlying issue with the container and isn’t the error itself.

Istio relies on sidecar containers that run alongside every application instance. Just like any container, these sidecar containers can experience CrashLoopBackOff errors.

Here are some potential issues that can result in CrashLoopBackOff errors:

• Misconfigurations—A typo in the configurations file
• Unavailable resources—Such as an unmounted PersistantVolume
• Wrong CLI arguments—Either missing or just incorrect arguments
• Bugs and exceptions—Anything hindering the container from functioning

Here are some ways to investigate the issue:

• Check Istio configuration applied to the specific application instance
• Check the pod description and logs
• Check the events
• Check the deployment   

Common Traffic Management Problems

Envoy Rejects Requests

To check Envoy’s access logs for reasons behind request rejection:

Use the following to see Envoy’s logs:

kubectl logs DEMO-PODNAME -c istio-proxy -n DEMO-NAMESPACE

According to the default access log format, the response code comes before the Envoy response flags. Include %RESPONSE_FLAGS% if some custom log format is in use.

Here are common response flags:

• NR—No route is configured; investigate your DestinationRule or VirtualService
• UO—Upstream overflow with the circuit breaking; check the circuit breaker configuration present in DestinationRule
• UF—Couldn’t connect to upstream successfully; if the authentication is from Istio,  for a mutual TLS configuration conflict

Route Rules Don’t Affect the Traffic Flow

Envoy’s current sidecar implementation requires up to 100 requests for weighted version distribution to become observable.

If the route rules work as intended for a sample application, but similar routing rules don’t impact your application, the Kubernetes services may need slight modification. After all, these services must stick with specific restrictions to use Istio’s L7 routing features.

It might be possible that the route rules are coming into effect but slowly. The Istio implementation ensures all Envoy sidecars have the right configuration, including route rules, through an eventually consistent algorithm. Therefore, a change in configuration takes some time to take effect in all sidecars, and with larger deployments, it’ll take longer.     

Sending HTTPS Requests to an HTTP Port

Investigate whether the application is attempting to send an HTTPS request and the service is declared to be HTTP. In such a case, the Envoy sidecar will try to parse the request as HTTP while it forwards the request. However, it won’t be able to parse it because the request has a different and unexpected encryption.

For example, consider the following configuration:

apiVersion: networking.istio.io/v1beta1

kind: ServiceEntry

metadata:

  name: httpbin

spec:

  hosts:

  – httpbin.org

  ports:

  – number: 443

    name: http

    protocol: HTTP

  resolution: DNS

While the above configuration may be right If the aim is to send plaintext on the 443 port, the port is normally dedicated to HTTPS traffic. Hence, when it receives an HTTPS request and defaults to the 443 port, it will throw an error like curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number. Additionally, the access logs might also show an error like 400 DPE.

To fix the inconsistency of HTTP and HTTPS ports: 

Change the port protocol to HTTPS in the configuration like below:

spec:

  ports:

  – number: 443

    name: https

    protocol: HTTPS

Common Security Problems 

Failure of End User Authentication

It is possible to enable end user authentication using request authentication policies in Istio.

To troubleshoot the policy specification behind end user authentication failure:

1. Check if the jwksUri field is present in the request authentication policy, as shown below:

apiVersion: security.istio.io/v1beta1

kind: RequestAuthentication

metadata:

  name: “demo”

spec:

  selector:

    matchLabels:

      app: demoapp

  jwtRules:

  – issuer: “[email protected]”

    jwksUri: “https://raw.githubusercontent.com/istio/istio/release-1.15/security/tools/jwt/samples/jwks.json” 

If jwksUri is not set present, check that the JWT issuer has a URL format and that the following URL is valid and can be visited in a browser:

<JWT-issuer-URL> + /.well-known/openid-configuration

2. Ensure that the JWT token is valid if it’s present in the Authorization header of HTTP requests. Various online JWT parsing tools are available to find the token’s fields.

3. Verify the Envoy proxy’s configuration of the intended workload by running the following command:

istioctl proxy-config command 

Mutual TLS Errors

If there is a possibility of issues with mutual TLS, it is important to verify that Citadel is functioning properly. Additionally, run the command below to ensure that the right certificates and keys are reaching the sidecars:

istioctl proxy-config secret <demo-pod-name[.namespace]>

If all of the above is behaving as intended, verify that the configurations involve an appropriate authentication policy and destination rules. 

In case there is a possibility that the client sidecar might be sending mutual TLS or plaintext wrong, look into the associated Grafana Workload dashboard. It annotates all outbound requests with the information if mTLS is or isn’t in use. If there is an indication that the client sidecars behaved unexpectedly, report the issue on GitHub.

Too Rigid or Flexible Authorization

A common mistake is unintentionally adding multiple items in the YAML specification. For example, consider the following:

apiVersion: security.istio.io/v1beta1

kind: AuthorizationPolicy

metadata:

  name: example

  namespace: foo

spec:

  action: ALLOW

  rules:

  – to:

    – operation:

        paths:

        – /demo

  – from:

    – source:

        namespaces:

        – demo

The expectation is that the above policy will allow requests when the path and source namespace are /demo and demo, respectively. However, it will let requests through if either the path is /demo or the source namespace is demo, making it more flexible than intended.

According to YAML syntax, the – before the from: indicates a new addition to the list. Hence, it creates two rules within the policy instead of one. Within the authorization policy, multiple rules mean requests must obey either of them.

To fix the issue, remove the extra – in front of from: so that policy has only one rule and allows requests only if the path and the source namespace is /demo and demo, respectively.

Conclusion

In this article, I reviewed some of the common errors facing Istio users:

• Configuration errors
  • Rejection of Seemingly Valid Configuration
  • Receiving no such hosts or no endpoints available Errors When Creating Configuration
  • Kubernetes CrashLoopBackOff
• Traffic management problems
  • Envoy Rejects Requests
  • Route Rules Don’t Affect the Traffic Flow
  • Sending HTTPS Requests to an HTTP Port
• Security problems
  • Failure of End User Authentication
  • Mutual TLS Errors
  • Too Rigid or Flexible Authorization

I hope this will help you make better use of Istio to manage communications for your microservices applications.