Vistio — Visualize your Istio Mesh Using Netflix’s Vizceral

GitHub Link — https://github.com/nmnellis/vistio

Vizceral is an open source project released by Netflix to monitor network traffic between applications and clusters in near real time. Vistio is an adaptation of Vizceral for Istio and mesh monitoring. It utilizes metrics generated by Istio Mixer which are then fed into Prometheus. Vistio queries Prometheus and stores that data locally to allow for the replaying of traffic.

There are two main visualizations served by Vizceral, global and cluster level. At the global level (shown above) you can visualize network traffic from the Internet to your Istio mesh via an entry point like the Istio Ingress Gateway, or you can display the total network traffic within your Istio mesh.

At the cluster level (shown below) you can visualize the traffic of your internal mesh. You can quickly detect when applications are having issues by setting warning and error level alerts.

Setting Up Vistio for Your Istio Mesh

Requirements

• Prometheus
• Istio => 0.7

Assumptions
The following demo makes these assumptions for an easier deployment. If your environment is setup differently, you may need to checkout the code locally and edit some files.

• Prometheus is running in the istio-system namespace and is accessible via http://prometheus.istio-system:9090
• Istio mixer has istio_request_count metric enabled
• Your kubernetes cluster has a ‘standard’ StorageClass
• Helm installed for easy deployment (Optional)

Preface
If your mesh is not currently deployed you can follow the directions here [Istio Bookinfo Demo] to deploy Istio and its sample application. You will need to be able to generate traffic between the applications. To test that metrics are being sent correctly to Prometheus from mixer you can run the following Prometheus query “istio_request_count” and should see multiple entries.

Deploy Vistio

You can choose to deploy Vistio via kubectl or Helm, the directions will cover both. Some values may need to be modified depending on your environment.

Checkout Vistio (Optional)

If you would like to deploy Vistio via Helm you will need to checkout the project to get its Helm templates. Also if one of the above assumptions does not meet your needs(Ex. prometheus url is different), you should checkout and edit the files manually.

git clone https://github.com/nmnellis/vistio.git

Deploy Via kubectl

kubectl apply -f https://raw.githubusercontent.com/nmnellis/vistio/v0.1.2/vistio-mesh-only.yaml -n default

Deploy Via Helm

cd into the root of Vistio project and run a “helm install”

helm install helm/vistio -f helm/vistio/values-mesh-only.yaml --name vistio --namespace default

Verify and Expose Vistio Web/API

Verify that the applications are up and running. We expose the applications via “kubectl port-forward”.

Verify vistio-api

kubectl describe statefulset vistio-api -n default

Optional check logs

You should be able to see if there are connection/query errors to Prometheus from the vistio-api in the logs.

kubectl logs -n default -c vistio-api $(kubectl -n default get pod -l app=vistio-api -o jsonpath='{.items[0].metadata.name}')

Verify vistio-web

kubectl describe deployment vistio-web -n default

Expose vistio-api

We will use “kubectl port-forward” to expose the vistio-api to http://localhost:9091

kubectl -n default port-forward $(kubectl -n default get pod -l app=vistio-api -o jsonpath='{.items[0].metadata.name}') 9091:9091 &

Verify vistio-api

The vistio-api is called by vistio-web to render your mesh. You should see something similar to the output below

http://localhost:9091/graph

Expose Vistio

In another terminal window, expose the Vizceral UI on http://localhost:8080.

kubectl -n default port-forward $(kubectl -n default get pod -l app=vistio-web -o jsonpath='{.items[0].metadata.name}') 8080:8080 &

Open Vistio

If everything up until now is working you should be able to load the Vistio UI and begin exploring your mesh network. You should see something similar to the image below.

http://localhost:8080

Explore

At the global level you will see the sum of all the requests within you Istio mesh. If you deployed the istio-ingressgateway you could optionally display the traffic you are receiving from outside your mesh with some other configuration[Deploy Vistio with Ingress Gateway]

If you click on the istio-mesh bubble, you will be able to view your mesh network.

Within your Istio mesh you have a number of visualization tools to help you pinpoint troubled applications.

Using the filters in the top right of the screen you can quickly filter to applications that have a high rate of errors. With advanced configuration you can also trigger alerts when error rates exceed certain values. The alerts will show you the current trend of error rates for the given application.

Troubleshooting

If you see the following output from vistio-api you will know that something is not working correctly. The correct output is displayed above in the tutorial.

http://localhost:9091/graph

1. Check vistio-api logs for errors — in most cases vistio-api will log any issues it has with communicating with Prometheus.

kubectl logs -n default -c vistio-api $(kubectl -n default get pod -l app=vistio-api -o jsonpath='{.items[0].metadata.name}')

2. Verify Prometheus Queries — the following queries are used by vistio-api to retrieve its data. You should make sure data exists for both within Prometheus.

# Global Level Query
sum(rate(istio_request_count[1m])) by (response_code)

# Cluster Level Query
sum(rate(istio_request_count[1m])) by (source_service,destination_service,response_code)

3. Submit an Issue — In the event you get stuck, feel free to submit and issue! https://github.com/nmnellis/vistio/issues