8. Cluster Monitoring And Horizontal Autoscaling

In this lab, we will install and use some common Kubernetes cluster monitoring tools. We will then deploy a Kubernetes Horizontal Pod Autoscaler (HPA) and manage a Deployment replica count using the CPU metric.

Chapter Details
Chapter Goal Learn Kubernetes best practices
Chapter Sections
8.1. Deploy Core Metrics Server
8.2. Install kube-prometheus
8.3. Access Grafana Dashboard
8.4. Auto-Scaling

8.1. Deploy Core Metrics Server

Make sure that Core Metrics Server is deployed:

$ kubectl apply -f ~/k8s-examples/extensions/metrics-server-1.8+
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
serviceaccount/metrics-server unchanged
deployment.extensions/metrics-server unchanged
service/metrics-server unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-server unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged

If the Metrics Server is already deployed you will get some warnings, which are harmless.

8.2. Install kube-prometheus

Kube-prometheus is a collection of manifests that combines various monitoring tools such as Prometheus, Grafana, Alertmanager, and etc. It is preconfigured to collect metrics from all Kubernetes components and provide you with a default set of dashboard and alerting rules.

The source code and manifests are available at ~/k8s-examples/addons/kube-prometheus

We will now use this directory to deploy all of our monitoring components.

Step 1 Install the kube-prometheus components by applying the manifests:

$ kubectl apply -f ~/k8s-examples/addons/kube-prometheus/manifests
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus-operator created
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
...

Step 2 Wait a few minutes for the monitoring workload to startup. Get all resources in the monitoring namespace to see its progress:

$ kubectl get all -n monitoring
NAME                                       READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0                    2/2     Running   0          16m
pod/alertmanager-main-1                    2/2     Running   0          15m
pod/alertmanager-main-2                    2/2     Running   0          15m
pod/grafana-545d8c5576-k8mt2               1/1     Running   0          16m
pod/kube-state-metrics-6898cf84bd-mhlkc    4/4     Running   0          16m
pod/node-exporter-278tq                    2/2     Running   0          16m
pod/node-exporter-2tdh5                    2/2     Running   0          16m
pod/node-exporter-562lp                    2/2     Running   0          16m
pod/prometheus-adapter-66fc7797fd-mc6pk    1/1     Running   0          16m
pod/prometheus-k8s-0                       3/3     Running   1          15m
pod/prometheus-k8s-1                       3/3     Running   1          15m
pod/prometheus-operator-6f96f6d567-lnzlh   1/1     Running   0          16m

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/alertmanager-main       ClusterIP   10.100.248.138   <none>        9093/TCP            16m
service/alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   16m
service/grafana                 NodePort    10.107.108.8     <none>        3000:32617/TCP      16m
service/kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   16m
service/node-exporter           ClusterIP   None             <none>        9100/TCP            16m
service/prometheus-adapter      ClusterIP   10.106.31.232    <none>        443/TCP             16m
service/prometheus-k8s          ClusterIP   10.108.119.164   <none>        9090/TCP            16m
service/prometheus-operated     ClusterIP   None             <none>        9090/TCP            15m
service/prometheus-operator     ClusterIP   None             <none>        8080/TCP            16m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
daemonset.apps/node-exporter   3         3         3       3            3           beta.kubernetes.io/os=linux   16m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana               1/1     1            1           16m
deployment.apps/kube-state-metrics    1/1     1            1           16m
deployment.apps/prometheus-adapter    1/1     1            1           16m
deployment.apps/prometheus-operator   1/1     1            1           16m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-545d8c5576               1         1         1       16m
replicaset.apps/kube-state-metrics-6898cf84bd    1         1         1       16m
replicaset.apps/kube-state-metrics-69c95c6466    0         0         0       16m
replicaset.apps/prometheus-adapter-66fc7797fd    1         1         1       16m
replicaset.apps/prometheus-operator-6f96f6d567   1         1         1       16m

NAME                                 READY   AGE
statefulset.apps/alertmanager-main   3/3     16m
statefulset.apps/prometheus-k8s      2/2     15m

As you can see, this is a comprehensive deployment of many services that provide you cluster monitoring.

8.3. Access Grafana Dashboard

Grafana dashboard has already been installed as defined in one of the manifests files from the previous section. Let’s obtain its endpoint and take a look in our browser.

Step 1 Obtain the Grafana service NodePort:

$ kubectl get svc -n monitoring grafana
NAME      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
grafana   NodePort   10.107.108.8   <none>        3000:32617/TCP   33m

The node port in this example is 32617, but yours may be different.

Step 2 In your browser, open the following URL (replace <nodePort> with the nodePort number from the previous step):

http://<lab-ip>:<nodePort>

Step 4 Log-in with admin user and admin password:

_images/k8s-grafana-login.png

You will be prompted to change your password. You can skip this step since this is a lab environment:

_images/k8s-grafana-skip-pw.png

Step 5 Click on the Home Drop-down menu then click Kubernetes / Compute Resources / Cluster:

_images/k8s-grafana-cluster.png

This shows you a great overview of your cluster health. Feel free to explore other dashboards configured in Grafana. Proceed when you are ready. Note that it will take a few minutes for your datapoints to be populated similar to the screenshot shown.

8.4. Auto-Scaling

Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization. Heapster (https://github.com/kubernetes/heapster) needs to be deployed in the cluster as Horizontal Pod Autoscaler uses it to collect metrics.

8.4.1. Create an Application

To demonstrate Horizontal Pod Autoscaler we will use a custom docker image based on the php-apache image. It contains index.php page, which performs some CPU intensive computations.

Step 1 Run the application and expose its port:

$ kubectl run php-apache \
--image=gcr.io/google_containers/hpa-example \
--requests=cpu=200m \
--expose --port=80
service/php-apache created
deployment.apps/php-apache created

Step 2 Check that the application’s pod is running:

$ kubectl get pods -l run=php-apache
NAME                          READY   STATUS    RESTARTS   AGE
php-apache-864967cc4f-gr7hs   1/1     Running   0          4m6s

Step 3 Check that the application responds on the service clusterIP address and port 80:

$ curl http://$(kubectl get svc php-apache -o jsonpath={..clusterIP}):80
OK!

8.4.2. Create a Horizontal Pod Autoscaler

Step 1 Use the following command to create a Horizontal Pod Autoscaler that maintains between 1 and 10 replicas of the Pods controlled by the php-apache deployment we created in the previous steps:

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
deployment "php-apache" autoscaled

Step 2 Check the current status of autoscaler:

$ kubectl get hpa
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS
php-apache   Deployment/php-apache   0% / 50%   1         10        1

Note that the current CPU consumption is 0% as we are not sending any requests to the server. If you see <unknown> instead of 0% wait a few minutes and run the command again: the autoscaler needs some time to gather information for the application.

Note

If the value under TARGETS is not 0% / 50% then apply the fix in ~/k8s-examples/addons/heapster/fix/README.md/.

8.4.3. Increase load

We will start a container, and send an infinite loop of queries to the php-apache service (remebmer, that our page in the php-apache application consumes a lot of CPU resources for each request).

Step 1 Open a new terminal and log in to the lab again (you also can use tmux or screen to open a new window in the same SSH session). In a new terminal, run the following command:

$ kubectl run -i --tty load-generator --image=busybox /bin/sh
If you don't see a command prompt, try pressing enter.
/ #

Step 2 In the load-generator container run the following infinite loop to generate a load for our application:

/ # while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done
OK!OK!..

Keep the command running and wait a few minutes.

Step 3 In the first terminal, check the current status of autoscaler:

$ kubectl get hpa
NAME         REFERENCE               TARGETS      MINPODS   MAXPODS   REPLICAS
php-apache   Deployment/php-apache   122% / 50%   1         10        4

As you see, CPU consumption has increased to 122%. As a result, the deployment was resized to 4 replicas (in you lab, the numbers can be different):

$ kubectl get deployment php-apache
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE
php-apache   4         4         4            4

8.4.4. Stop Load

Step 1 Return to the second terminal and stop the load generator (press Ctrl-C). Delete the load-generator deployment:

$ kubectl delete deployment load-generator
deployment.extensions "load-generator" deleted

You also can close the second terminal, we do not need it any more. Wait a few minutes.

Step 2 In the first terminal, check the current status of autoscaler:

$ kubectl get hpa
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS
php-apache   Deployment/php-apache   0% / 50%   1         10        1

8.4.5. Delete the Application

Delete the php-apache application and service:

$ kubectl delete service php-apache
service "php-apache" deleted

$ kubectl delete deployment php-apache
deployment.extensions "php-apache" deleted

Checkpoint

  • Install InfluxDB
  • Install Heapster
  • Use Horizontal Pod Autoscaler to scales the number of pods in a replication controller

Table Of Contents

Previous topic

7. Kubernetes Architecture

Next topic

9. Kubernetes Security

Navigation

© Copyright 2019, Mirantis.