How to expose services in k3s - with automatic Let's Encrypt certs and certificate renewal

In this guide I'll show how to deploy a little API and expose it with the traefik reverse proxy. Traefik is already packaged with k3s, so no need for additional installation.

As described in my instruction manual about how to install k3s, K3s is one of the more lightweight kubernetes distributions out there, created by the well-known kubernetes experts at Rancher. Running kubernetes by using k3s is a real blast - and on top of the little complexity, k3s is CNCF-certified and marked as production-ready.

The application - a resume parsing backend
The goal
Traefik Ingress
Cert-Manager
k8s Resource overview
How-To
Change issuer to Let's Encrypt Production
The Traefik Dashboard
Summary - TLDR

We will also enable automatic SSL certificate handling, using Let's encrypt and kubernetes Cert Manager.

The application - a resume parsing backend

Note: The application is not part of this guide - however I outline some of the details to set the context for the following steps

For a different project, I created a small project which takes pdf resumes (CVs), parses them and returns a machine readable JSON document with the summarized candidate information. The application provides two REST endpoints:

POST: /parseresume: Takes a pdf file as payload and parses it as resume
GET: /parseresume: Simply returns {"answer": "test"}

The application is served using uvicorn - a python low-level webserver. It listens on port 8000.

The goal

In the end, we want to serve the aforementioned application on https://myresume.datascienceengineer.com/parseresume. The communication should be SSL encrypted with Let's-Encrypt certificates and the certificate renewal should be automated.

Traefik Ingress

The Traefik Ingress provider is a Kubernetes Ingress controller based on the popular Traefik reverse proxy. Traefik describes itself as "an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them."

Traefik Overview

And indeed, using traefic is very easy and fun as they say - especially as it comes already prepacked with k3s. One word though: While traefic is able to automatically work with Let's Encrypt certificates, it lacks a lot of features which for example the kubernetes-native cert-manager provides. Most noteworthy, as of time of this writing, the certificates are stored in the file system - meaning they are gone when the pod is restartet. It's also hard to impossible to share the certs across pods and nodes - making high availability setups nearly impossible.

Cert-Manager

Enter cert-manager, the well-known certificate handling solution for kubernetes. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

The certificates are stored as kubernetes secrets - making them easily shareable and a k8s-native resource.

And on top of that, cert-manager is known to be very reliable: I run cert-manager on several installations for years know - I had not a single incident with certificate since.

k8s Resource overview

So, additionally to k3s, what do we need to set all of this up?

The cert-manager
A ClusterIssuer resource which is used by the cert-manager to issue certificates
A traefik-based Ingress resource which handles the incoming traffic, TLS termination and redirects to the right services
A certificate - but which is automatically created by the ClusterIssuer - so nothing todo here

Kubernetes Resources

How-To

Note: We are installing all our own resources into namespace resume

Install cert-manager by running kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Check, that cert-manager was installed: Run kubectl get pods -n cert-manager. You should see something like:

1NAME                                       READY   STATUS    RESTARTS   AGE
2cert-manager-5dd59d9d9b-4q9vp              1/1     Running   0          26s
3cert-manager-cainjector-8696fc9f89-4z5k6   1/1     Running   0          26s
4cert-manager-webhook-7d4b5b8c56-qn2tt      1/1     Running   0          26s

Create a new file staging-issuer.yaml

1apiVersion: cert-manager.io/v1
2kind: ClusterIssuer
3metadata:
4    name: letsencrypt-staging
5    nmamespace: resume
6spec:
7    acme:
8    email: your-email@example.com
9    server: https://acme-staging-v02.api.letsencrypt.org/directory
10    privateKeySecretRef:
11        name: letsencrypt-staging
12    solvers:
13        - http01:
14            ingress:
15            class: traefik

Apply the staging-issuer.yaml: kubectl apply -f staging-issuer.yaml

Optional: Add a https-redirect middleware. This comes handy to automatically redirect http request to https. But, this step is optional. Add a file traefik-https-redirect-middleware.yaml and apply it

1apiVersion: traefik.containo.us/v1alpha1
2kind: Middleware
3metadata:
4name: redirect-https
5namespace: resume
6spec:
7redirectScheme:
8    scheme: https
9    permanent: true

Create the ingress resource. Add file called resume-parser-ingress.yaml and apply it

1apiVersion: networking.k8s.io/v1
2kind: Ingress
3metadata:
4name: resume-parser-tls-ingress
5namespace: resume
6annotations:
7    kubernetes.io/ingress.class: traefik
8    cert-manager.io/cluster-issuer: letsencrypt-staging
9    traefik.ingress.kubernetes.io/router.middlewares: resume-redirect-https@kubernetescrd
10spec:
11rules:
12    - host: myresume.datascienceengineer.com
13    http:
14        paths:
15        - path: /
16            pathType: Prefix
17            backend:
18            service:
19                # this is the service which interfaces the application
20                # it is a simple ClusterIP service, redirecting port 
21                # 8000 to 8000
22                name: resume-parser-cluster-ip-service
23                port:
24                number: 8000
25tls:
26    - secretName: resume-parser-tls
27    hosts:
28        - myresume.datascienceengineer.com

Note: The first part of the traefik.ingress.kubernetes.io/router.middlewares is the namespace. Our namespace is resume.

In your DNS configuration console, make sure to have myresume.datascienceengineer.com pointed to the external IP-Address of the ingress. You can check the external IP by running kubectl get ingress -n resume

1NAME                        CLASS    HOSTS                                  ADDRESS         PORTS     AGE
2resume-parser-tls-ingress   <none>   myresume.datascienceengineer.com   164.68.x.x   80, 443   2m7s

That's actually all the magic. Your ingress is now deployed and your application is served on myresume.datascienceengineer.com. Keep in mind, that a modern browser will still tell you, that your route is insecure, because we used a staging issuer - instead of a production one.

Change issuer to Let's Encrypt Production

Please note, that changing the issuer to the Let's Encrypt production issuer will make browser rate your route as secure. However you only have 5 certificate renewals per week - so make sure to test everything with the staging issuer and only afterwards change to production.

Add a file production-issuer.yaml and apply it

1apiVersion: cert-manager.io/v1
2kind: ClusterIssuer
3metadata:
4name: letsencrypt-prod
5namespace: resume
6spec:
7acme:
8    email: andreas.nigg@devopsandmore.com
9    server: https://acme-v02.api.letsencrypt.org/directory
10    privateKeySecretRef:
11    name: letsencrypt-prod
12    solvers:
13    - http01:
14        ingress:
15            class: traefik

Delete the automatically generated staging certificate: kubectl delete secrets resume-parser-tls -n resume

In your Ingress Definition file, change the cluster-issuer configuration to letsencrypt-prod and apply the file

1apiVersion: networking.k8s.io/v1
2kind: Ingress
3metadata:
4name: resume-parser-tls-ingress
5namespace: resume
6annotations:
7    kubernetes.io/ingress.class: traefik
8    cert-manager.io/cluster-issuer: letsencrypt-prod
9    traefik.ingress.kubernetes.io/router.middlewares: resume-redirect-https@kubernetescrd
10spec:

The Traefik Dashboard

Traefik comes with a quite handy dashboard. To reach the dashboard, we forward the remote port to our local port by running:

1kubectl port-forward -n kube-system "$(kubectl get pods -n kube-system | grep '^traefik-' | awk '{print $1}')" 9000:9000

Now you can access the dashboard with your browser: http://localhost:9000/dashboard/

IMPORTANT: Note the trailing slash in the URL. This is required, otherwise you'll get an error 404!

The dashboard than provides some overview about your Traefik Resources.

Traefik Dashboard

Summary - TLDR

In this guide we saw how one can serve workloads running in their k3s kubernetes cluster by using the pre-installed Traefik controller and cert-manager. The following steps are necessary:

Install cert-manager by running kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Create a new file production-issuer.yaml

1apiVersion: cert-manager.io/v1
2kind: ClusterIssuer
3metadata:
4name: letsencrypt-prod
5namespace: resume
6spec:
7acme:
8    email: andreas.nigg@devopsandmore.com
9    server: https://acme-v02.api.letsencrypt.org/directory
10    privateKeySecretRef:
11    name: letsencrypt-prod
12    solvers:
13    - http01:
14        ingress:
15            class: traefik

Add a http to https redirect middleware:

1apiVersion: traefik.containo.us/v1alpha1
2kind: Middleware
3metadata:
4name: redirect-https
5namespace: resume
6spec:
7redirectScheme:
8    scheme: https
9    permanent: true

Create the ingress resource:

1apiVersion: networking.k8s.io/v1
2kind: Ingress
3metadata:
4name: resume-parser-tls-ingress
5namespace: resume
6annotations:
7    kubernetes.io/ingress.class: traefik
8    cert-manager.io/cluster-issuer: letsencrypt-prod
9    traefik.ingress.kubernetes.io/router.middlewares: resume-redirect-https@kubernetescrd
10spec:
11rules:
12    - host: myresume.datascienceengineer.com
13    http:
14        paths:
15        - path: /
16            pathType: Prefix
17            backend:
18            service:
19                # this is the service which interfaces the application
20                # it is a simple ClusterIP service, redirecting port 
21                # 8000 to 8000
22                name: resume-parser-cluster-ip-service
23                port:
24                number: 8000
25tls:
26    - secretName: resume-parser-tls
27    hosts:
28        - myresume.datascienceengineer.com

------------------

Interested in how to train your very own Large Language Model?

We prepared a well-researched guide for how to use the latest advancements in Open Source technology to fine-tune your own LLM. This has many advantages like:

Cost control
Data privacy
Excellent performance - adjusted specifically for your intended use

Get your free LLM training guide