(cache)Create an unmanaged gateway controller in AKS with NGINX Gateway Fabric

A gateway controller is the central component that implements the Gateway API, responsible for managing all Gateway resources. With the Gateway API, you can not only route traffic like you would with Ingress, but you also have better control over Gateway API resources. When you use a gateway controller and Gateway API resources, one or more IP addresses can be used to route traffic to multiple services in a Kubernetes cluster.

This article shows you how to deploy the NGINX Gateway Fabric in an Azure Kubernetes Service (AKS) cluster. Two applications will run in different namespaces within the AKS cluster, both accessible via a single IP address. Furthermore, an SSL/TLS certificate will be configured using Kubernetes Secret, cert-manager or Azure Key Vault Provider for Secrets Store CSI Driver.

To learn more about differences between the Ingress API and the Gateway API, see also: Migrating from Ingress.

Before you begin#

Prepare a domain for accessing the demo application. If you don’t have any, you can try free DDNS service like No-IP.

Package/API/add-on preparation#

First, we need to install all the necessary packages and APIs before proceeding with the configuration.

Install the Gateway API There are two ways to install APIs:
a. Refer to the Kubernetes official documentation to install the latest version of the Gateway API.
b. If you prefer to use the Azure managed gateway API, you can also refer to the Azure documentation: Install Managed Gateway API Custom Resource Definitions (CRDs)

NOTE
However, note that some of the Gateway APIs are still not in General Availability (GA) and are subject to change. When using the managed Gateway API, consider the potential impact of version changes when upgrading your AKS cluster version.

TIP
Before using Azure managed Gateway API, you probably want to read the following sections under support policies (Archive: archive.today):

Unsupported alpha and beta Kubernetes features (Key point: AKS only supports stable and beta features within the upstream Kubernetes project)

Preview features or feature flags (Key point: features in public preview fall under best effort support)

Upstream bugs and issues

Install NGINX Gateway Fabric (gateway controller) as below:

1
helm upgrade --install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway

TIP
Note that we will use NGINX Gateway Fabric as the gateway controller in this example. If you want to use a different gateway controller, check out: Gateway Controller Implementation Status

To use NodePort instead of LoadBalancer, using command below:

1
helm upgrade --install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.service.type=NodePort

To generate publicly-trusted SSL/TLS certificate, install cert-manager using the command below:

1
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager --create-namespace --namespace cert-manager \
2
  --set crds.enabled=true \
3
  --set config.enableGatewayAPI=true

If you need to use the Secrets Store CSI Driver to reference the certificate in Azure Key Vault, refer to Use the Azure Key Vault Provider for Secrets Store CSI Driver in an Azure Kubernetes Service (AKS) cluster for installation and permission-granting instructions.

Create a simple Gateway#

Use the following manifest to create a gateway within the “gateway-example” namespace:

1
apiVersion: gateway.networking.k8s.io/v1
2
kind: Gateway
3
metadata:
4
  name: gateway-public
5
  namespace: gateway-example
6
spec:
7
  gatewayClassName: nginx
8
  listeners:
9
  - name: http
10
    port: 80
11
    protocol: HTTP

If you want to use Azure internal load balancer, use the following manifest:

1
apiVersion: gateway.networking.k8s.io/v1
2
kind: Gateway
3
metadata:
4
  name: gateway-internal
5
  namespace: gateway-example
6
spec:
7
  gatewayClassName: nginx
8
  listeners:
9
  - name: http
10
    port: 80
11
    protocol: HTTP
12
  infrastructure:
13
    annotations:
14
      service.beta.kubernetes.io/azure-load-balancer-internal: 'true'

Your Gateways will be assigned different IPs shortly. To check the IPs, use the command below:

1
kubectl get svc -n gateway-example

The output should be as follows:

1
NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)        AGE
2
gateway-internal-nginx   LoadBalancer   10.0.95.126    10.208.0.33       80:32550/TCP   5m26s
3
gateway-public-nginx     LoadBalancer   10.0.175.217   123.45.67.89      80:30124/TCP   5m26s

Showcase: routing traffic to applications in different namespaces#

After understanding how to create Gateway with public and internal IPs, we will attempt to use the Gateway API in a sample case.

NOTE
The previously created Gateways mentioned above will not be utilized in the following demonstration. Instead, a new Gateway will be created.

Run demo applications#

To see the gateway controller in action, run two demo applications in your AKS cluster. In this example, you will use kubectl apply to deploy two instances of a simple Hello world application.

Create namespace called “default-service-and-gateway” for gateway resource and default application deployment

1
cat <<EOF | kubectl apply -f -
2
apiVersion: v1
3
kind: Namespace
4
metadata:
5
  name: default-service-and-gateway
6
  labels:
7
    app.kubernetes.io/part-of: default-service
8
    app.kubernetes.io/component: gateway-api
9
EOF

Deploy default appliation (service-one) in “default-service-and-gateway” namespace

1
cat <<EOF | kubectl apply -f -
2
apiVersion: apps/v1
3
kind: Deployment
4
metadata:
5
  name: service-one
6
  namespace: default-service-and-gateway
7
  labels:
8
    app.kubernetes.io/part-of: default-service
9
spec:
10
  replicas: 1
9 collapsed lines
11
  selector:
12
    matchLabels:
13
      app.kubernetes.io/name: service-one
14
      app.kubernetes.io/part-of: default-service
15
  template:
16
    metadata:
17
      labels:
18
        app.kubernetes.io/name: service-one
19
        app.kubernetes.io/part-of: default-service
20
    spec:
21
      containers:
22
      - name: service-one-app
23
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
24
        ports:
25
        - containerPort: 80
26
        env:
27
        - name: TITLE
28
          value: "Welcome to AKS Default Service Page"
29
---
30
apiVersion: v1
31
kind: Service
32
metadata:
33
  name: service-one
34
  namespace: default-service-and-gateway
35
  labels:
36
    app.kubernetes.io/part-of: default-service
37
spec:
38
  type: ClusterIP
39
  ports:
40
  - port: 80
41
    protocol: TCP
42
    name: http
43
  selector:
44
    app.kubernetes.io/name: service-one
45
    app.kubernetes.io/part-of: default-service
46
EOF

Create namespace called “external-service” for another application (service-two)

1
cat <<EOF | kubectl apply -f -
2
apiVersion: v1
3
kind: Namespace
4
metadata:
5
  name: external-service
6
  labels:
7
    app.kubernetes.io/part-of: external-service
8
EOF

Deploy another appliation (service-two) in “external-service” namespace

1
cat <<EOF | kubectl apply -f -
2
apiVersion: apps/v1
3
kind: Deployment
4
metadata:
5
  name: service-two
6
  namespace: external-service
7
  labels:
8
    app.kubernetes.io/part-of: external-service
9
spec:
10
  replicas: 1
9 collapsed lines
11
  selector:
12
    matchLabels:
13
      app.kubernetes.io/name: service-two
14
      app.kubernetes.io/part-of: external-service
15
  template:
16
    metadata:
17
      labels:
18
        app.kubernetes.io/name: service-two
19
        app.kubernetes.io/part-of: external-service
20
    spec:
21
      containers:
22
      - name: service-two-app
23
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
24
        ports:
25
        - containerPort: 80
26
        env:
27
        - name: TITLE
28
          value: "Welcome to AKS External Service Page"
29
---
30
apiVersion: v1
31
kind: Service
32
metadata:
33
  name: service-two
34
  namespace: external-service
35
  labels:
36
    app.kubernetes.io/part-of: external-service
37
spec:
38
  type: ClusterIP
39
  ports:
40
  - port: 80
41
    protocol: TCP
42
    name: http
43
  selector:
44
    app.kubernetes.io/name: service-two
45
    app.kubernetes.io/part-of: external-service
46
EOF

Create Gateway#

Both applications are now running in your Kubernetes cluster. Next, we will create a Gateway resource using the NGINX Gateway Fabric.

Set your hostname as variable

1
# Set your hostname
2
routeDomain=<your-own-domain>

Deploy Gateway resource

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: Gateway
4
metadata:
5
  name: the-sole-unique-gateway
6
  namespace: default-service-and-gateway
7
  labels:
8
    app.kubernetes.io/component: gateway-api
9
spec:
10
  gatewayClassName: nginx
11
  listeners:
12
  - name: http
13
    port: 80
14
    protocol: HTTP
15
    # Optional: set hostname
16
    hostname: ${routeDomain}
17
  infrastructure:
18
    labels:
19
      app.kubernetes.io/part-of: gateway-api
20
EOF

NOTE
The “hostname” field is optional; however, it is still being used in this demonstration as I want to add a sense of seriousness. I hope you won’t be misled to think that “hostname must be used”.

Set up DNS record using the public IP you received
You will need to set up the DNS record on your own. To obtain the current public IP used for Gateway, use the command below:

1
kubectl get gateway the-sole-unique-gateway -n default-service-and-gateway -o jsonpath='{.status.addresses[0].value}'

Create HTTPRoute#

To route traffic to each application, create a Kubernetes HTTPRoute resource. The HTTPRoute resource configures the rules that route traffic to one of the two applications.

In the following example, HTTPRoute will be created in the samespace with Gateway resource. Traffic to

${routeDomain}/default is routed to the service named service-one (which is in the same namespace with Gateway resource).
${routeDomain}/external is routed to the service named service-two (which is in a different namespace).
${routeDomain}/static is routed to the service named service-one for static assets.

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: HTTPRoute
4
metadata:
5
  name: service-one
6
  namespace: default-service-and-gateway
7
  labels:
8
    app.kubernetes.io/part-of: default-service
9
    app.kubernetes.io/component: gateway-api
10
spec:
11
  parentRefs:
12
  - name: the-sole-unique-gateway
13
    sectionName: http
14
  hostnames:
15
  - ${routeDomain}
16
  rules:
17
  - matches:
18
    - path:
19
        type: PathPrefix
20
        value: /default
21
    filters:
22
    - type: URLRewrite
23
      urlRewrite:
24
        path:
25
          type: ReplacePrefixMatch
26
          replacePrefixMatch: /
27
    backendRefs:
28
    - name: service-one
29
      namespace: default-service-and-gateway
30
      port: 80
31
---
32
apiVersion: gateway.networking.k8s.io/v1
33
kind: HTTPRoute
34
metadata:
35
  name: service-two
36
  namespace: default-service-and-gateway
37
  labels:
38
    app.kubernetes.io/part-of: external-service
39
    app.kubernetes.io/component: gateway-api
40
spec:
41
  parentRefs:
42
  - name: the-sole-unique-gateway
43
    sectionName: http
44
  hostnames:
45
  - ${routeDomain}
46
  rules:
47
  - matches:
48
    - path:
49
        type: PathPrefix
50
        value: /external
51
    filters:
52
    - type: URLRewrite
53
      urlRewrite:
54
        path:
55
          type: ReplacePrefixMatch
56
          replacePrefixMatch: /
57
    backendRefs:
58
    - name: service-two
59
      namespace: external-service
60
      port: 80
61
---
62
apiVersion: gateway.networking.k8s.io/v1
63
kind: HTTPRoute
64
metadata:
65
  name: service-one-static
66
  namespace: default-service-and-gateway
67
  labels:
68
    app.kubernetes.io/part-of: default-service
69
    app.kubernetes.io/component: gateway-api
70
spec:
71
  parentRefs:
72
  - name: the-sole-unique-gateway
73
    sectionName: http
74
  hostnames:
75
  - ${routeDomain}
76
  rules:
77
  - matches:
78
    - path:
79
        type: PathPrefix
80
        value: /static
81
    backendRefs:
82
    - name: service-one
83
      namespace: default-service-and-gateway
84
      port: 80
85
EOF

Create ReferenceGrant#

Since service-two is located in a different namespace than the HTTPRoute resource, we need to use “ReferenceGrant” to enable cross namespace references within Gateway API.

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: ReferenceGrant
4
metadata:
5
  name: external-service-reference
6
  namespace: external-service
7
  labels:
8
    app.kubernetes.io/part-of: external-service
9
    app.kubernetes.io/component: gateway-api
10
spec:
11
  from:
12
  - group: gateway.networking.k8s.io
13
    kind: HTTPRoute
14
    namespace: default-service-and-gateway
15
  to:
16
  - group: ""
17
    kind: Service
18
    name: service-two
19
EOF

NOTE
There is no need to create “ReferenceGrant” resource for service-one, as its HTTPRoute and Service resources reside within the same namespace.

TIP
When the application is deployed in a different namespace than the Gateway resource, you can alternatively choose to allow the Gateway to accept HTTPRoute resources from a different namespace while placing the HTTPRoute in the same namespace as the service, depending on your management architecture.
In this scenario, the use of “ReferenceGrant” is unnecessary. This approach will be demonstrated in the section of SSL/TLS certificate setup using cert-manager.

Test the gateway controller#

To test the routes for the gateway controller, you can use curl to test the link and check for results:

1
curl -s http://${routeDomain}/default | grep "Default Service Page</div>"
2
curl -s http://${routeDomain}/external | grep "External Service Page</div>"

Output:

1
        <div id="logo">Welcome to AKS Default Service Page</div>
2
        <div id="logo">Welcome to AKS External Service Page</div>

Using SSL/TLS certificate in Gateway#

SSL/TLS certificates facilitate SSL/TLS encryption when a client communicates with a website. You can either bring your own certificates and store them in a Kubernetes Secret, integrate them with the Secrets Store CSI driver, or choose to use cert-manager to generate a publicly-trusted certificate.

Simple reference to certificate stored in secret#

The following manifest is a simple example of referencing a certificate from the secret “www-cert,” which is located in the “certificate” namespace:

1
apiVersion: gateway.networking.k8s.io/v1
2
kind: Gateway
3
metadata:
4
  name: gateway
5
  namespace: gateway-example
6
spec:
7
  gatewayClassName: nginx
8
  listeners:
9
  - name: http
10
    port: 80
11
    protocol: HTTP
12
    hostname: ${routeDomain}
13
    allowedRoutes:
14
      namespaces:
15
        from: All
16
  - name: https
17
    port: 443
18
    protocol: HTTPS
19
    hostname: ${routeDomain}
20
    tls:
21
      mode: Terminate
22
      certificateRefs:
23
      - kind: Secret
24
        name: www-cert
25
        namespace: certificate

Remember to create a “ReferenceGrant” to enable cross-namespace references when the certificate secret is not in the same namespace as Gateway resource:

1
apiVersion: gateway.networking.k8s.io/v1
2
kind: ReferenceGrant
3
metadata:
4
  name: www-cert-reference
5
  namespace: certificate
6
spec:
7
  to:
8
  - group: ""
9
    kind: Secret
10
    name: www-cert
11
  from:
12
  - group: gateway.networking.k8s.io
13
    kind: Gateway
14
    namespace: gateway-example

Using cert-manager to generate and configure publicly-trusted certificate#

If you need an application to generate and renew certificates for you, consider using cert-manager.
In this section, we will continue from the showcase above. Note that it is assumed that you have cert-manager installed in the cluster before proceeding.

Create an namespace called “cert-issuer” for issuing certificate

1
cat <<EOF | kubectl apply -f -
2
apiVersion: v1
3
kind: Namespace
4
metadata:
5
  name: cert-issuer
6
  labels:
7
    app.kubernetes.io/component: cert-issuer
8
EOF

Define a Issuer

1
cat <<EOF | kubectl apply -f -
2
apiVersion: cert-manager.io/v1
3
kind: Issuer
4
metadata:
5
  name: domain-cert-issuer
6
  namespace: cert-issuer
7
spec:
8
  acme:
9
    server: https://acme-v02.api.letsencrypt.org/directory
10
    privateKeySecretRef:
11
      name: letsencrypt-acme-secret
12
    solvers:
13
    - http01:
14
        gatewayHTTPRoute:
15
          parentRefs:
16
          - name: the-sole-unique-gateway
17
            namespace: default-service-and-gateway
18
            kind: Gateway
19
EOF

TIP
For the list of well-known ACME servers, refer to: acme.sh wiki
Additionally, check out External Account Bindings when using a CA other than Let’s Encrypt.

NOTE
You do can use ClusterIssuer instead of an Issuer to make it cluster-scoped, depending on your architecture. In this case, a namespace-scoped Issuer will be used.

Create Certificate resource to define request specifications

1
cat <<EOF | kubectl apply -f -
2
apiVersion: cert-manager.io/v1
3
kind: Certificate
4
metadata:
5
  name: domain-cert
6
  namespace: cert-issuer
7
spec:
8
  issuerRef:
9
    name: domain-cert-issuer
10
  dnsNames:
11
  - ${routeDomain}
12
  privateKey:
13
    algorithm: ECDSA
14
    encoding: PKCS1
15
    size: 256
16
  secretName: domain-cert-secret
17
  secretTemplate:
18
    labels:
19
      app.kubernetes.io/component: cert-issuer
20
EOF

The certificate will be stored in the secret “domain-cert-secret” within the “cert-issuer” namespace.

NOTE
After the Gateway references the Issuer (which will be done in a later step), a HTTPRoute resource will be automatically created as shown below:
1
apiVersion: gateway.networking.k8s.io/v1
2
kind: HTTPRoute
3
metadata:
4
  name: cm-acme-http-solver-abcde
5
  namespace: cert-issuer
6
spec:
7
  parentRefs:
8
    - name: the-sole-unique-gateway
9
      namespace: default-service-and-gateway
10
      kind: Gateway
11
  hostnames:
12
  - ${routeDomain}
13
  rules:
14
  - backendRefs:
15
    - port: 8089
16
      name: cm-acme-http-solver-abcde
17
      weight: 1
18
    matches:
19
    - path:
20
        type: Exact
21
        value: /.well-known/acme-challenge/<TOKEN>
By default, Gateway only allows HTTPRoute resources from the namespace where it is located. In the next step, we will modify “allowedRoutes” to accept HTTPRoute resources from other namespaces, allowing the CA issuer to verify the request via HTTP.

Grant permission to allow the Gateway to reference certificate Secret in different namespace

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: ReferenceGrant
4
metadata:
5
  name: domain-cert-reference
6
  namespace: cert-issuer
7
spec:
8
  from:
9
  - group: gateway.networking.k8s.io
10
    kind: Gateway
11
    namespace: default-service-and-gateway
12
  to:
13
  - group: ""
14
    kind: Secret
15
    name: domain-cert-secret
16
EOF

Modify Gateway to use Issuer, referencing uncreated certificate Secret and extra HTTPRoute

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: Gateway
4
metadata:
5
  name: the-sole-unique-gateway
6
  namespace: default-service-and-gateway
7
  labels:
8
    app.kubernetes.io/component: gateway-api
9
spec:
10
  gatewayClassName: nginx
11
  listeners:
12
  - name: http
13
    port: 80
14
    protocol: HTTP
15
    hostname: ${routeDomain}
16
    allowedRoutes:
17
      namespaces:
18
        from: Selector
19
        selector:
20
          matchExpressions:
21
          - key: app.kubernetes.io/component
22
            operator: In
23
            values:
24
            - gateway-api
25
            - cert-issuer
26
  - name: https
27
    port: 443
28
    protocol: HTTPS
29
    hostname: ${routeDomain}
30
    tls:
31
      mode: Terminate
32
      certificateRefs:
33
      - name: domain-cert-secret
34
        namespace: cert-issuer
35
        kind: Secret
36
        group: ""
37
  infrastructure:
38
    labels:
39
      app.kubernetes.io/part-of: gateway-api
40
EOF

Now, Gateway will accept HTTPRoute resources from namespaces that have the label “app.kubernetes.io/component: gateway-api” or “app.kubernetes.io/component: cert-issuer”. This enables Gateway to route traffic to both cert-manager and applications.

TIP

You can also choose to allow HTTPRoute resources from different namespaces based on their names, as shown below:

1
allowedRoutes:
2
  namespaces:
3
    from: Selector
4
    selector:
5
      matchExpressions:
6
      - key: kubernetes.io/metadata.name
7
        operator: In
8
        values:
9
        - default-service-and-gateway
10
        - cert-issuer

Edit existing HTTPRoute to allow traffic to be routed under HTTPS communication

1
kubectl patch httproute service-one \
2
  -n default-service-and-gateway \
3
  --type='json' \
4
  -p='[{"op": "add", "path": "/spec/parentRefs/-", "value": {"name": "the-sole-unique-gateway", "sectionName": "https"}}]'
5

6
kubectl patch httproute service-two \
7
  -n default-service-and-gateway \
8
  --type='json' \
9
  -p='[{"op": "add", "path": "/spec/parentRefs/-", "value": {"name": "the-sole-unique-gateway", "sectionName": "https"}}]'
10

11
kubectl patch httproute service-one-static \
12
  -n default-service-and-gateway \
13
  --type='json' \
14
  -p='[{"op": "add", "path": "/spec/parentRefs/-", "value": {"name": "the-sole-unique-gateway", "sectionName": "https"}}]'

The Gateway manifest should look as follows:


10 collapsed lines
1
apiVersion: gateway.networking.k8s.io/v1
2
kind: HTTPRoute
3
metadata:
4
  name: service-one
5
  namespace: default-service-and-gateway
6
  labels:
7
    app.kubernetes.io/part-of: default-service
8
    app.kubernetes.io/component: gateway-api
9
spec:
10
  parentRefs:
11
  - name: the-sole-unique-gateway
12
    sectionName: http
13
  - name: the-sole-unique-gateway
14
    sectionName: https
15
  hostnames:
16
  - ${routeDomain}
26 collapsed lines
17
  rules:
18
  - matches:
19
    - path:
20
        type: PathPrefix
21
        value: /default
22
    filters:
23
    - type: URLRewrite
24
      urlRewrite:
25
        path:
26
          type: ReplacePrefixMatch
27
          replacePrefixMatch: /
28
    backendRefs:
29
    - name: service-one
30
      namespace: default-service-and-gateway
31
      port: 80
32
---
33
apiVersion: gateway.networking.k8s.io/v1
34
kind: HTTPRoute
35
metadata:
36
  name: service-two
37
  namespace: default-service-and-gateway
38
  labels:
39
    app.kubernetes.io/part-of: external-service
40
    app.kubernetes.io/component: gateway-api
41
spec:
42
  parentRefs:
43
  - name: the-sole-unique-gateway
44
    sectionName: http
45
  - name: the-sole-unique-gateway
46
    sectionName: https
47
  hostnames:
48
  - ${routeDomain}
26 collapsed lines
49
  rules:
50
  - matches:
51
    - path:
52
        type: PathPrefix
53
        value: /external
54
    filters:
55
    - type: URLRewrite
56
      urlRewrite:
57
        path:
58
          type: ReplacePrefixMatch
59
          replacePrefixMatch: /
60
    backendRefs:
61
    - name: service-two
62
      namespace: external-service
63
      port: 80
64
---
65
apiVersion: gateway.networking.k8s.io/v1
66
kind: HTTPRoute
67
metadata:
68
  name: service-one-static
69
  namespace: default-service-and-gateway
70
  labels:
71
    app.kubernetes.io/part-of: default-service
72
    app.kubernetes.io/component: gateway-api
73
spec:
74
  parentRefs:
75
  - name: the-sole-unique-gateway
76
    sectionName: http
77
  - name: the-sole-unique-gateway
78
    sectionName: https
79
  hostnames:
80
  - ${routeDomain}
9 collapsed lines
81
  rules:
82
  - matches:
83
    - path:
84
        type: PathPrefix
85
        value: /static
86
    backendRefs:
87
    - name: service-one
88
      namespace: default-service-and-gateway
89
      port: 80

Wait for the certificate to be generated. Use the command below to check its existence:

1
kubectl get secret domain-cert-secret -n cert-issuer

It usually takes about 1 minute. Proceed to the next step if the certificate Secret is created.

Test if traffic works under HTTPS communication

1
curl -s https://${routeDomain}/default | grep "Default Service Page</div>"
2
curl -s https://${routeDomain}/external | grep "External Service Page</div>"

Output:

1
        <div id="logo">Welcome to AKS Default Service Page</div>
2
        <div id="logo">Welcome to AKS External Service Page</div>

Using Secrets Store CSI Driver with your own certificates#

If you are deeply using managed service in Azure, then using Secrets Store CSI Driver will be a better choice.
In this section, we will continue using the same showcase from the previous section. Whether you skipped to this section or are reading from the start, I will provide the steps to reset the Gateway for consistency.
Note that it is assumed that you have relative add-ons installed on the cluster before proceeding.

Use the following command to reset HTTPRoute and Gateway resources

NOTE
If you haven’t executed any commands in the section “Using cert-manager to generate and configure publicly-trusted certificate”, you can skip this step. However, even if you have executed it, there is no harm done.

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: Gateway
95 collapsed lines
4
metadata:
5
  name: the-sole-unique-gateway
6
  namespace: default-service-and-gateway
7
  labels:
8
    app.kubernetes.io/component: gateway-api
9
spec:
10
  gatewayClassName: nginx
11
  listeners:
12
  - name: http
13
    port: 80
14
    protocol: HTTP
15
    hostname: ${routeDomain}
16
  infrastructure:
17
    labels:
18
      app.kubernetes.io/part-of: gateway-api
19
---
20
apiVersion: gateway.networking.k8s.io/v1
21
kind: HTTPRoute
22
metadata:
23
  name: service-one
24
  namespace: default-service-and-gateway
25
  labels:
26
    app.kubernetes.io/part-of: default-service
27
    app.kubernetes.io/component: gateway-api
28
spec:
29
  parentRefs:
30
  - name: the-sole-unique-gateway
31
    sectionName: http
32
  hostnames:
33
  - ${routeDomain}
34
  rules:
35
  - matches:
36
    - path:
37
        type: PathPrefix
38
        value: /default
39
    filters:
40
    - type: URLRewrite
41
      urlRewrite:
42
        path:
43
          type: ReplacePrefixMatch
44
          replacePrefixMatch: /
45
    backendRefs:
46
    - name: service-one
47
      namespace: default-service-and-gateway
48
      port: 80
49
---
50
apiVersion: gateway.networking.k8s.io/v1
51
kind: HTTPRoute
52
metadata:
53
  name: service-two
54
  namespace: default-service-and-gateway
55
  labels:
56
    app.kubernetes.io/part-of: external-service
57
    app.kubernetes.io/component: gateway-api
58
spec:
59
  parentRefs:
60
  - name: the-sole-unique-gateway
61
    sectionName: http
62
  hostnames:
63
  - ${routeDomain}
64
  rules:
65
  - matches:
66
    - path:
67
        type: PathPrefix
68
        value: /external
69
    filters:
70
    - type: URLRewrite
71
      urlRewrite:
72
        path:
73
          type: ReplacePrefixMatch
74
          replacePrefixMatch: /
75
    backendRefs:
76
    - name: service-two
77
      namespace: external-service
78
      port: 80
79
---
80
apiVersion: gateway.networking.k8s.io/v1
81
kind: HTTPRoute
82
metadata:
83
  name: service-one-static
84
  namespace: default-service-and-gateway
85
  labels:
86
    app.kubernetes.io/part-of: default-service
87
    app.kubernetes.io/component: gateway-api
88
spec:
89
  parentRefs:
90
  - name: the-sole-unique-gateway
91
    sectionName: http
92
  hostnames:
93
  - ${routeDomain}
94
  rules:
95
  - matches:
96
    - path:
97
        type: PathPrefix
98
        value: /static
99
    backendRefs:
100
    - name: service-one
101
      namespace: default-service-and-gateway
102
      port: 80
103
EOF

Generate a SSL/TLS certificate in PFX format

a. Create and change to a temporary directory

1
tempDir=$(mktemp -d) && cd $tempDir

b. Generate a self-signed SSL/TLS certificate

1
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
2
  -keyout private.key -out certificate.pem -days 30 -nodes \
3
  -subj "/CN=${routeDomain}" -addext "subjectAltName=DNS:${routeDomain}"

c. Convert it to PFX format

1
openssl pkcs12 -export -in certificate.pem -passout pass: \
2
  -inkey private.key -out certificate.pfx

The file certificate.pfx will be used in the following steps.

Create an Azure Key Vault

a. Fill in an existing resource group name and a new Key Vault name

1
rG=
2
kv=

b. Create a new Key Vault

1
az keyvault create -n ${kv} -g ${rG} \
2
  --enable-rbac-authorization true -o none

c. Get Key Vault ID and URI for further use

1
kvId=$(az resource list -n ${kv} -g ${rG} \
2
  --resource-type Microsoft.KeyVault/vaults \
3
  --query [0].id -o tsv)
4
kvUri=$(az keyvault show -n ${kv} -g ${rG} \
5
  --query properties.vaultUri -o tsv)

d. Grant permision to yourself

1
userObjectId=$(az ad signed-in-user show --query id -o tsv)
2

3
az role assignment create --role "Key Vault Certificates Officer" \
4
  --assignee-object-id ${userObjectId} -o none \
5
  --scope ${kvId} --assignee-principal-type User

Import the self-signed SSL/TLS certificate

1
az keyvault certificate import --vault-name ${kv} \
2
  --name www-cert --file certificate.pfx -o none

Connect Azure Key Vault with managed identity of Azure Key Vault Secrets Store CSI Driver

NOTE
In this section, managed identity method will be used for granting permissions. For more methods, refer to: Connect your Azure identity provider to the Azure Key Vault Secrets Store CSI Driver in Azure Kubernetes Service (AKS)

1
akvspObjectId=$(az aks show -n ${aks} -g ${rG} -o tsv \
2
  --query addonProfiles.azureKeyvaultSecretsProvider.identity.objectId)
3
akvspClientId=$(az aks show -n ${aks} -g ${rG} -o tsv \
4
  --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId)
5

6
az role assignment create --role "Key Vault Certificate User" \
7
  --assignee-object-id ${akvspObjectId} -o none \
8
  --scope ${kvId} --assignee-principal-type ServicePrincipal

Create a namespace for storing SSL/TLS certificate from key vault

1
cat <<EOF | kubectl apply -f -
2
apiVersion: v1
3
kind: Namespace
4
metadata:
5
  name: certificate
6
  labels:
7
    app.kubernetes.io/component: certificate
8
EOF

Deploy SecretProviderClass to reference SSL/TLS certificate

a. Set your Tenant ID

1
tenantId=

b. Deploy SecretProviderClass

1
cat <<EOF | kubectl apply -f -
2
apiVersion: secrets-store.csi.x-k8s.io/v1
3
kind: SecretProviderClass
4
metadata:
5
  name: www-cert
6
  namespace: certificate
7
spec:
8
  provider: azure
9
  secretObjects:
10
    - secretName: www-cert-secret
11
      type: kubernetes.io/tls
12
      data:
13
        - objectName: www-cert
14
          key: tls.key
15
        - objectName: www-cert
16
          key: tls.crt
17
  parameters:
18
    useVMManagedIdentity: "true"
19
    userAssignedIdentityID: ${akvspClientId}
20
    keyvaultName: ${kv}
21
    objects: |
22
      array:
23
        - |
24
          objectName: www-cert
25
          objectType: secret
26
    tenantId: ${tenantId}
27
EOF

Deploy dummy pods in the “certificate” namespace to trigger certificate synchronization

1
cat <<EOF | kubectl apply -f -
2
apiVersion: apps/v1
3
kind: Deployment
4
metadata:
5
  name: www-cert-dummy
6
  namespace: certificate
7
spec:
8
  replicas: 2
9
  selector:
10
    matchLabels:
11
      app.kubernetes.io/name: azure-secretprovider-dummy
12
  template:
13
    metadata:
14
      labels:
15
        app.kubernetes.io/name: azure-secretprovider-dummy
16
    spec:
17
      priorityClassName: system-node-critical
18
      affinity:
19
        podAffinity:
20
          preferredDuringSchedulingIgnoredDuringExecution:
21
            - weight: 100
22
              podAffinityTerm:
23
                labelSelector:
24
                  matchExpressions:
25
                    - key: app.kubernetes.io/instance
26
                      operator: In
27
                      values:
28
                        - ngf
29
                topologyKey: "kubernetes.io/hostname"
30
      containers:
31
        - name: dummy-pod
32
          image: busybox
33
          command: ["sleep", "infinity"]
34
          volumeMounts:
35
          - name: secrets-store-inline
36
            readOnly: true
37
            mountPath: /mnt/secrets-store
38
      volumes:
39
      - name: secrets-store-inline
40
        csi:
41
          driver: secrets-store.csi.k8s.io
42
          readOnly: true
43
          volumeAttributes:
44
            secretProviderClass: www-cert
45
  strategy:
46
    type: RollingUpdate
47
    rollingUpdate:
48
      maxUnavailable: 1
49
      maxSurge: 1
50
EOF

NOTE
A secret can only be retrieved when the SecretProviderClass is referenced in volumes. SecretProviderClass must be in the same namespace as the Pod, and Secret will also be created in the same namespace as SecretProviderClass only.
This design necessitates the creation of a dummy Pod; otherwise, you can only place certificate Secrets within the “nginx-gateway” namespace.
Attaching volumes within the Nginx Fabric Gateway controller is not recommended, but if you choose to do so, use the following command:
Terminal window
1
helm upgrade --install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway \
2
  --set nginx.pod.nodeSelector."kubernetes\.io/os"=linux \
3
  --set nginxGateway.nodeSelector."kubernetes\.io/os"=linux \
4
  --set certGenerator.nodeSelector."kubernetes\.io/os"=linux \
5
  -f - <<EOF
6
nginxGateway:
7
  extraVolumes:
8
      - name: secrets-store-inline
9
        csi:
10
          driver: secrets-store.csi.k8s.io
11
          readOnly: true
12
          volumeAttributes:
13
            secretProviderClass: www-cert
14
  extraVolumeMounts:
15
      - name: secrets-store-inline
16
        mountPath: "/mnt/secrets-store"
17
        readOnly: true
18
EOF

Enable auto rotation in Azure Key Vault provider for Secrets Store CSI Driver

1
az aks addon update -n [your_aks_name] -g [aks_resource_group] -o none \
2
  --addon azure-keyvault-secrets-provider --enable-secret-rotation

For introduction to auto rotation feature, refer to: Manage auto rotation

Referencing SSL/TLS certificate and set-up ReferenceGrant

a. Reference SSL/TLS certificate in Gateway

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: Gateway
4
metadata:
5
  name: the-sole-unique-gateway
6
  namespace: default-service-and-gateway
7
  labels:
8
    app.kubernetes.io/component: gateway-api
9
spec:
10
  gatewayClassName: nginx
11
  listeners:
12
  - name: http
13
    port: 80
14
    protocol: HTTP
15
    hostname: ${routeDomain}
16
  - name: https
17
    port: 443
18
    protocol: HTTPS
19
    hostname: ${routeDomain}
20
    tls:
21
      mode: Terminate
22
      certificateRefs:
23
      - name: www-cert-secret
24
        namespace: certificate
25
        kind: Secret
26
        group: ""
27
  infrastructure:
28
    labels:
29
      app.kubernetes.io/part-of: gateway-api
30
EOF

b. Grant permission to allow the Gateway to reference certificate Secret in different namespace

1
cat <<EOF | kubectl apply -f -
2
apiVersion: gateway.networking.k8s.io/v1
3
kind: ReferenceGrant
4
metadata:
5
  name: domain-cert-reference
6
  namespace: certificate
7
spec:
8
  from:
9
  - group: gateway.networking.k8s.io
10
    kind: Gateway
11
    namespace: default-service-and-gateway
12
  to:
13
  - group: ""
14
    kind: Secret
15
    name: www-cert-secret
16
EOF

c. Editing existing HTTPRoute to allow traffic to be routed under HTTPS communication

1
kubectl patch httproute service-one \
2
  -n default-service-and-gateway \
3
  --type='json' \
4
  -p='[{"op": "add", "path": "/spec/parentRefs/-", "value": {"name": "the-sole-unique-gateway", "sectionName": "https"}}]'
5

6
kubectl patch httproute service-two \
7
  -n default-service-and-gateway \
8
  --type='json' \
9
  -p='[{"op": "add", "path": "/spec/parentRefs/-", "value": {"name": "the-sole-unique-gateway", "sectionName": "https"}}]'
10

11
kubectl patch httproute service-one-static \
12
  -n default-service-and-gateway \
13
  --type='json' \
14
  -p='[{"op": "add", "path": "/spec/parentRefs/-", "value": {"name": "the-sole-unique-gateway", "sectionName": "https"}}]'

The Gateway manifest should look as follows:


10 collapsed lines
1
apiVersion: gateway.networking.k8s.io/v1
2
kind: HTTPRoute
3
metadata:
4
  name: service-one
5
  namespace: default-service-and-gateway
6
  labels:
7
    app.kubernetes.io/part-of: default-service
8
    app.kubernetes.io/component: gateway-api
9
spec:
10
  parentRefs:
11
  - name: the-sole-unique-gateway
12
    sectionName: http
13
  - name: the-sole-unique-gateway
14
    sectionName: https
15
  hostnames:
16
  - ${routeDomain}
26 collapsed lines
17
  rules:
18
  - matches:
19
    - path:
20
        type: PathPrefix
21
        value: /default
22
    filters:
23
    - type: URLRewrite
24
      urlRewrite:
25
        path:
26
          type: ReplacePrefixMatch
27
          replacePrefixMatch: /
28
    backendRefs:
29
    - name: service-one
30
      namespace: default-service-and-gateway
31
      port: 80
32
---
33
apiVersion: gateway.networking.k8s.io/v1
34
kind: HTTPRoute
35
metadata:
36
  name: service-two
37
  namespace: default-service-and-gateway
38
  labels:
39
    app.kubernetes.io/part-of: external-service
40
    app.kubernetes.io/component: gateway-api
41
spec:
42
  parentRefs:
43
  - name: the-sole-unique-gateway
44
    sectionName: http
45
  - name: the-sole-unique-gateway
46
    sectionName: https
47
  hostnames:
48
  - ${routeDomain}
26 collapsed lines
49
  rules:
50
  - matches:
51
    - path:
52
        type: PathPrefix
53
        value: /external
54
    filters:
55
    - type: URLRewrite
56
      urlRewrite:
57
        path:
58
          type: ReplacePrefixMatch
59
          replacePrefixMatch: /
60
    backendRefs:
61
    - name: service-two
62
      namespace: external-service
63
      port: 80
64
---
65
apiVersion: gateway.networking.k8s.io/v1
66
kind: HTTPRoute
67
metadata:
68
  name: service-one-static
69
  namespace: default-service-and-gateway
70
  labels:
71
    app.kubernetes.io/part-of: default-service
72
    app.kubernetes.io/component: gateway-api
73
spec:
74
  parentRefs:
75
  - name: the-sole-unique-gateway
76
    sectionName: http
77
  - name: the-sole-unique-gateway
78
    sectionName: https
79
  hostnames:
80
  - ${routeDomain}
9 collapsed lines
81
  rules:
82
  - matches:
83
    - path:
84
        type: PathPrefix
85
        value: /static
86
    backendRefs:
87
    - name: service-one
88
      namespace: default-service-and-gateway
89
      port: 80

Test if traffic works under HTTPS communication

1
curl -s -k https://${routeDomain}/default | grep "Default Service Page</div>"
2
curl -s -k https://${routeDomain}/external | grep "External Service Page</div>"

Output:

1
        <div id="logo">Welcome to AKS Default Service Page</div>
2
        <div id="logo">Welcome to AKS External Service Page</div>

Afterword#

This is the first article after my surgery and also the last article of the year.
Before writing this one, I had already written three drafts on different topics. However, one of them is pending a version release, another is stuck due to a bug, and for the last one, I don’t have a subscription to perform the lab. I could have released a new article earlier, but it never happened for various reasons.

This article also honors the archived document: Create an unmanaged ingress controller. This article is no longer being maintained, but it is a classic. To honor this article, I have imitated most of its style while trying to improve it. Most of the basic information can be found in this article.

NOTE
There was something wrong here when this article was initially published, but after clarification by the developer, I removed the incorrect content to maintain accuracy. The discussion history can be checked here.

Let’s set this topic aside for now. As the year comes to a close, let’s take a moment to say:

Happy New Year!

Wishing everyone all the best in the year ahead.

And my cat.