Introduction to Google Kubernetes Engine
What you'll learn in this page
We have used Kubernetes distribution K3s using Docker containers via k3d. In a production environment the burden of maintaining a Kubernetes cluster is often left to third parties. A managed Kubernetes as a service is often the best choice as the additional work required in maintenance exceeds the benefits of a personal cluster. In some, somewhat rare, cases setting up and maintaining your own cluster is a reasonable option. A case for it would be that your company/organization already has the hardware and/or wants to stay independent from providers, one such example could be a University. Another primary reason for operating your own Kubernetes cluster is that regulations rule out all other choices.
Maintaining your own Kubernetes cluster is one way to increase costs. The other is running servers that are not in use. To avoid costs Serverless (opens in a new tab) (opens in a new tab) solutions could be more cost-efficient. A Kubernetes cluster of improper size can get really expensive pretty fast. An excellent option to start out with would be a high-tier environment for serverless workloads, such as Google Cloud Run or AWS Fargate. In those you can run any container, compared to older solutions like Cloud Functions or AWS Lambda, which have wildly varying support for different languages, environments and tooling.
Let's focus on the Google Kubernetes Engine (GKE) costs for now. Note that the GKE costs a little bit more than its competitors.
The calculator here https://cloud.google.com/products/calculator (opens in a new tab) (opens in a new tab) offers us a picture of the pricing. I decided to try a cheap option: 5 nodes in 1 zonal cluster using 1 vCPU each. The datacenter location is in Finland, and I don't need a persistent disk. If we wanted fewer than 5 nodes, why would we even use Kubernetes? The total cost for this example was around 200 USD per month. Adding additional services, such as a Load balancer, increases the cost. If you find the billing for Google Cloud Platform confusing, you're not alone: Coursera has ~5 hour course for "Understanding Your Google Cloud Platform Costs (opens in a new tab) (opens in a new tab)".
During this Chapter, we will be using GKE either by using the free credits offered by Google. You are responsible for making sure that the credits last for the whole part, and if all of them are consumed, I can not help you.
After redeeming the credits, we can create a project with the billing account. The Google Cloud UI can be confusing. On the resources page (opens in a new tab) (opens in a new tab) we can create a new project and let us name it "dwk-gke" for the purposes of this course.
Install the Google Cloud SDK. Instructions here (opens in a new tab) (opens in a new tab). After that login and set the previously created project to be used. See the project id from the resource page!
$ gcloud -v
Google Cloud SDK 471.0.0
bq 2.1.3
core 2024.03.29
gcloud-crc32c 1.0.0
gsutil 5.27
$ gcloud auth login
...
You are now logged in
$ gcloud config set project dwk-gke-idhere
Updated property [core/project].We can now create a cluster with the command gcloud container clusters create (opens in a new tab) (opens in a new tab). You can choose any zone you want from the list here (opens in a new tab) (opens in a new tab). I chose Finland. Notice that one region (e.g. europe-north1) may have multiple regions (e.g. -a). Let's add another flag: --cluster-version=1.32. This will ask GKE to use a version that will be the default in April 2025. The GKE release schedule can be seen here (opens in a new tab) (opens in a new tab).
Let us try creating the cluster:
$ gcloud container clusters create dwk-cluster --zone=europe-north1-b --cluster-version=1.32 --disk-size=32 --num-nodes=3 --machine-type=e2-micro
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Kubernetes Engine API has not been used in project dwk-gke-xxxxxx before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/container.googleapis.com/overview?project=dwk-gke-xxxxxx then retry.Creation fails. You can visit the link that was provided and enable the Kubernetes Engine API, just note that the URL which is outputted is specific to your project name. Or, you can just execute the following command on the CLI:
$ gcloud services enable container.googleapis.com
Operation "operations/acf.p2-385245615727-2f855eed-e785-49ac-91da-896925a691ab" finished successfully.
$ gcloud container clusters create dwk-cluster --zone=europe-north1-b --cluster-version=1.32 --disk-size=32 --num-nodes=3 --machine-type=e2-micro
...
Creating cluster dwk-cluster in europe-north1-b...
...
kubeconfig entry generated for dwk-cluster.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
dwk-cluster europe-north1-b 1.29.8-gke.2200 35.228.176.118 e2-medium 1.29.8-gke.2200 3 RUNNINGIf the command does not work, you need to install gke-gcloud-auth-plugin by following this (opens in a new tab) (opens in a new tab).
The installation takes some minutes, so be patient! The installation should set the kubeconfig to point to the newly created cluster. You can check the cluster info with kubectl cluster-info to see if the configuration is right:
$ kubectl cluster-info
Kubernetes control plane is running at https://35.238.53.231
GLBCDefaultBackend is running at https://35.238.53.231/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
KubeDNS is running at https://35.238.53.231/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://35.238.53.231/api/v1/namespaces/kube-system/services/https:metrics-server:/proxyIf you still see the URLs looking something like the below, the configuration is still pointing to your local cluster:
$ kubectl cluster-info
Kubernetes control plane is running at https://0.0.0.0:62698
CoreDNS is running at https://0.0.0.0:62698/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:62698/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxyIn this case we can set up the kubeconfig to point to the new cluster like this:
$ gcloud container clusters get-credentials dwk-cluster --zone=europe-north1-b
Fetching cluster endpoint and auth data.
kubeconfig entry generated for dwk-cluster.The cluster we have now is almost like the one we had locally. Let's apply this (opens in a new tab) (opens in a new tab) application that creates a random string and then serves an image based on that random string. This will create 6 replicas of the process "seedimage".
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-hy/material-example/e11a700350aede132b62d3b5fd63c05d6b976394/app6/manifests/deployment.yamlExposing the service is where the differences start. Instead of an Ingress we'll use LoadBalancer (opens in a new tab) (opens in a new tab) service. Now as a warning the next step is going to add into the cost of the cluster (opens in a new tab) (opens in a new tab) as well.
Apply the following file service.yaml:
apiVersion: v1
kind: Service
metadata:
name: seedimage-svc
spec:
type: LoadBalancer # This should be the only unfamiliar part
selector:
app: seedimage
ports:
- port: 80
protocol: TCP
targetPort: 3000A load balancer service asks for Google services to provision us a load balancer. Let us find out what IP the service is given:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.31.240.1 <none> 443/TCP 144m
seedimage-svc LoadBalancer 10.31.241.224 <pending> 80:30215/TCP 94sThe external IP address is still in a pending state. We can wait until the service gets an external IP, with the option --watch :
$ kubectl get svc --watch
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.31.240.1 <none> 443/TCP 144m
seedimage-svc LoadBalancer 10.31.241.224 35.228.41.16 80:30215/TCP 94sAfter a short wait, we will see the external IP. If we now access http://35.228.41.16 with our browser, we'll see the application up and running. By refreshing the page, we can also see that the load balancer sometimes offers us a different image. Note that this is available on port 80, so HTTPS will not work.
To avoid using up the credits delete the cluster whenever you do not need it
$ gcloud container clusters delete dwk-cluster --zone=europe-north1-bAnd when resuming progress create the cluster back with the command shown earlier.
Closing the cluster will also remove everything you've deployed on the cluster. So if you decide to take a days long break during an exercise, you may have to redo it. Thankfully we are using a declarative approach so continuing progress will only require you to apply the yamls.
Google Kubernetes Engine will automatically provision a persistent disk for your PersistentVolumeClaim (opens in a new tab) (opens in a new tab) - just don't set the storage class. If you want you can read more about it here (opens in a new tab) (opens in a new tab).
Exercise:3.1. Pingpong GKE
1
Deploy Ping-pong application into GKE.
In this exercise use a LoadBalancer service to expose the service.
If your Postgres logs say
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
It contains a lost+found directory, perhaps due to it being a mount point.
Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.you can add subPath configuration:
statefulset.yaml
# ...
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
subPath: postgres
# ...This will create a Postgres directory where the data will reside. subPaths also make it possible to use single volume for multiple purposes.
Services are quite simple. They allow us to define a layer 4 (L4) load balancer on Google Cloud Services. Layer 4 here means the layer in OSI model. In other words, TCP/UDP traffic. For advanced traffic rules, we need to use a layer 7 (L7) load balancers which can be created either by an Ingress, or by Gateway resources which we'll talk a little bit more about later. Let's try using the Ingress resource since it's still the more widely used way of defining incoming traffic.
NodePort type service is required with an Ingress in GKE. Even though it is NodePort, GKE does not expose it outside the cluster. Let's test this by continuing with the previous example. Change service.yaml as follows:
apiVersion: v1
kind: Service
metadata:
name: seedimage-svc
spec:
type: NodePort
selector:
app: seedimage
ports:
- port: 80
protocol: TCP
targetPort: 3000Ingress in file ingress.yaml contains no surprises:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: seedimage-ing
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: seedimage-svc
port:
number: 80When you have applied those you can view the port from the list of ingresses:
$ kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
seedimage-ing <none> * 34.120.61.234 80 2m13sNow the address here will be the way to access the application. This will take a moment to deploy, responses may be 404 and 502 as it becomes available. The Ingress performs health checks by GET requesting / and expects an HTTP 200 response.
Exercise:3.2. Back to Ingress
1
Deploy the "Log output" and "Ping-pong" applications into GKE and expose it with Ingress.
"Ping-pong" will have to respond from /pingpong path. This may require you to rewrite parts of the code.
Note that Ingress expects a service to give a successful response in the path / even if the service is mapped to some other path!
New kid in the block: Gateway API
For years, Ingress has been the primary solution for external traffic routing in Kubernetes. The situation is changing; the new Gateway API offers a next-generation solution for routing.
The Gateway API is a set of resources and standards that allow you to define how external traffic should be routed to services within your Kubernetes cluster. It builds on the ingress concept but offers more advanced features, making it easier to handle complex routing and traffic management scenarios.
In order to use the Gateway API, we should enable it to our cluster:
$ gcloud container clusters update clustername --location=europe-north1-b --gateway-api=standardThe command takes some time so be patient!
Gateway API has some more parts to configure compared Ingress:
At the top is GatewayClass, which is a resource within the GatewayAPI that defines a type of Gateway. It specifies the underlying infrastructure or controller that will be used to implement the Gateway. The GatewayClass resources are defined by infrastructure providers. The list of options provided by Google Kubernetes Engine can be seen here.
Our work begins by defining the Gateway, which defines where and how the load balancers listen for traffic to our cluster. It is configured based on a GatewayClass that, amongst other things, selects the load balancer type to use. The Gateway specifies details like which IP addresses or hostnames to listen on, which ports to use:
Our gateway in file gateway.yaml looks the this:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
spec:
gatewayClassName: gke-l7-global-external-managed
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
kinds:
- kind: HTTPRouteOur definition is pretty simple; it picks a GatewayClass following the Google recommendations and says that our cluster accepts HTTP traffic to port 80.
The gateway just defines where and how the load balancers listen for traffic. The routing rules can be defined with HTTPRoute resources. The file route.yaml looks like the following:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: my-route
spec:
parentRefs:
- name: my-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: seedimage-svc
port: 80So the rule looks quite much like the route definition in the Ingress configuration, it just says requests the path / are router to service seedimage-svc, it also has the parentRefs that point to the Gateway resource where it belongs.
We still need to change the Service port type to ClusterIP:
apiVersion: v1
kind: Service
metadata:
name: seedimage-svc
spec:
type: ClusterIP
selector:
app: seedimage
ports:
- port: 80
protocol: TCP
targetPort: 3000The gateway objects tells us the IP address of the cluster:
$ kubectl get gateway my-gateway
NAME CLASS ADDRESS PROGRAMMED AGE
my-gateway gke-l7-gxlb 35.227.224.141 True 106mWe can verify that the app works in http:://35.227.224.141. Note that it could take some time until the gateway is set up and the IP starts responding. The command kubectl describe gateway my-gateway is also helpful.
Ingress and nowadays the Gateway help us define a load balancer as GCP understands them. In other words, we write yaml files, which then get read by GCP and actual infrastructure is provisioned by their automation. When using either resources in a cloud environment, it's wise to read through the cloud service providers documentation. You can also confirm details about the load balancers we just created by going to the GCP UI and searching for load balancer resources or by running `gcloud compute url-maps list`. If you're wondering why we're listing "url-maps", or what GCP load balancers are made of, including forwarding-rules and target proxies, you can learn more here.
Exercise:3.3. To the Gateway
1
Replace the Ingress with Gateway API. See here for more about HTTP routing.
Exercise:3.4. Rewritten routing
1
Your ping-pong app now most likely needs to respond to the URL /pingpong to work in the cluster setup. It would be nice if we were not forced to reflect the cluster-level URL structures in the applications, and instead the app itself could provide the behavior in the root path /. Thanks to the flexibility of the Gateway API, this can be easily done by route rewriting.
Make this change to your ping-pong app and to the HTTP route!
Note that replaceFullPath is not yet supported in GKE so you should use replacePrefixMatch, see here for more.
