High availability Kubernetes with k3s

blog preview

As a single-person company, I need to be as efficient as possible when it comes to IT operations. And therefore, I feel very strong about high availability. While a lot of small businesses neglect this aspect of system deployment, I try to invest here very early. The reason for that is very simple: I want to concentrate on growing my business rather than working on (and worrying about) downtime.

The TLDR can be found in the Summary

In a previous article, we set up k3s on a remote server in a cost-efficient single-node environment. For more details, see one of my previous posts. This time, we extend this installation with two more nodes and achieve a highly available k3s cluster. We will see that due to the very straight-forward tooling around k3s, setting up a 3-node HA cluster is equally easy as installing a single-node one. (That being said, what makes HA a little more complex than single-node deployments is actually having at least 3 virtual/physical server nodes to deploy on. However - this is a one time effort and with modern cloud or VPS providers also one you can afford quite easily).

1Setting up a system in high available manners is not much more complex than setting it up in a single-node setup

To proof this point, in this article I'll demonstrate how to set up kubernetes - more specifically k3s - in a high available installation.

Between this post and the one linked above, I found out about the superb tool k3sup - developed by Alex Ellis. This tool will help us today, setting up k3s on 3 remote servers.

About k3sup

k3sup (pronounced ketchup) is a quite intuitive tool which bootstraps k3s on a local or remote virtual or physical machine. All one needs is local or ssh access to the server to install and the k3sup binary. k3sup also manages your KUBECONFIG, merges it with potentially existing KUBECONFIGs and provides immediate kubectl access to your remote, potentially highly available, k3s cluster. Especially the feature of merging KUBECONFIGs is quite nice - as we have seen last time, this aspect is quite complex - compared to other k3s setup steps.

For more details about k3sup, see their official github page.

Prerequisites

For this guide, we assume you have 3 physical or virtual servers which can communicate between each other. As we'll see below, we run k3s with embedded etcd, which requires a quorum for leader election - therefore we need an odd amount of nodes.

Furthermore, you have passwordless SSH access to each one of them. For more information about how to access your remote systems via SSH, see this excellent Redhat guide.

Installing k3sup

As a first step, we need to install k3sup on our local host machine. The binaries are conveniently provided in their github release page. For even more convenience, we can use the k3sup setup script - provided by k3sup themselves. As I like convenience, we use this script.

NOTE: As always, please check scripts you download from the internet, whether they are to be trusted. For the time of this writing, the script was checked and found to do no harm.

  1. If your local machine is a Linux machine, run the following commands:
1curl -sLS https://get.k3sup.dev | sh
2sudo install k3sup /usr/local/bin/
3rm k3sup # (we can remove the downloaded binary after installation)

Windows: For Windows, visit the binary github release page and download the k3sup.exe file. No additional installation necessary.

Setting up a high available, multimaster k3s installation

As established in our last k3s blog entry, k3s comes with two high availability modes:

  1. Having an external database (eg. postgres) to store the metadata
  2. Having etcd embedded in the k3s master nodes

Number 1 has the advantage, that you can have HA with 2 nodes, but you need to manage an external database system. Which - in my opinion - somehow defeats the purpose. So we (and k3sup by default) choose option 2. We set up a k3s cluster, having 3 nodes with the storage (etcd) embedded in the servers.

k3s high available multi-server deployment with embedded storagek3s high available multi-server deployment with embedded storage

The general workflow for setting up is:

  1. Initialize the first server on the first node in "cluster" mode
  2. Join the other two nodes as servers

Initialize the first server on the first node in "cluster" mode

For setting up our first server, simply run, the following command - changing the placeholders as follows:

  • your-servers-ip: The IP-Address where you can reach your server via SSH
  • your-servers-user: The (SSH)-user of the remote server where you want to install k3s to

Important: The server-user needs to either be root or have passwordless root via ssh enabled. Otherwise k3sup will fail.

You can check if both the IP and user are correct by trying to connect via SSH (ssh <your-servers-user>@<your-servers-ip>). If you can't connect via passwordless SSH to the server, please follow this Redhat guide.

1k3sup install --ip <your-servers-ip> --user <your-servers-user> --context k3scontext --local-path ~/.kube/config --cluster --merge

This command now bootstrapped a single k3s server instance with embedded etcd. Joining our two additional nodes is as easy as:

1k3sup join --ip <ip-of-node-2> --user <user-of-node-2> --server-ip <your-servers-ip> --server-user <your-servers-user> --server
2

3k3sup join --ip <ip-of-node-3> --user <user-of-node-3> --server-ip <your-servers-ip> --server-user <your-servers-user> --server

As with the step above, make sure to replace the placeholders as follows:

  • your-servers-ip: The IP-Address where you can reach your first server via SSH (the one we already initialized in the step before)
  • your-servers-user: The (SSH)-user of the remote server where you already bootstrapped k3s (the one we already initialized in the step before)
  • ip-of-node-2: The IP-Address where you can reach your second node via SSH
  • user-of-node-2: The (SSH)-user of the second remote node
  • ip-of-node-3: The IP-Address where you can reach your third node via SSH
  • user-of-node-3: The (SSH)-user of the third remote node

That's already all the magic. You now have three k3s nodes running in high availability mode. Now you might start scheduling your workloads, using kubectl.

As k3sup did all the heavy lifting for your kubeconfig-setup, you can already use kubectl to manage workloads on your cluster. Start by checking your nodes:

1kubectl get node -o wide

This should give you your 3 nodes.

Explanation of k3sup command line parameters

Besides the obvious user and ip flags, we used three additional flags while running our commands. These are:

  • context: Set the name of the kubeconfig context. The default is "default" - which I personally don't like so much, as there are potentially quite a view contexts in your kubeconfig.
  • cluster: starts this server in cluster-mode. It uses an embedded etcd to potentially be HA.
  • server: Flag required to tell k3sup to join the nodes to an existing cluster - as servers rather than agents.
  • merge: Merge the config with existing kubeconfig if it already exists. This is important, as by default, k3sup overwrites existing kubeconfigs. You might omit this flag, if you don't have an existing kubeconfig anyhow.

Besides these basic flags, there are a ton of other customization options. For the sake of completeness, I'll illustrate them here - shamelessly copying them from the command lines help screen:

  • datastore: If you run k3s in "with external datastore" mode, this sets the connection-string for the k3s datastore to enable HA - i.e. "mysql://username:password@tcp(hostname:3306)/database-name"
  • host: Public hostname of node on which to install agent
  • ip ip: Public IP of node (default 127.0.0.1)
  • ipsec: Enforces and/or activates optional extra argument for k3s: flannel-backend option: ipsec
  • k3s-channel: Release channel: stable, latest, or pinned v1.19 (default "stable")
  • k3s-extra-args: Additional arguments to pass to k3s installer, wrapped in quotes (e.g. --k3s-extra-args '--no-deploy servicelb')
  • k3s-version: Set a version to install, overrides k3s-channel
  • local: Perform a local install without using ssh
  • local-path: Local path to save the kubeconfig file (default "kubeconfig")
  • no-extras: Disable "servicelb" and "traefik"
  • print-command: Print a command that you can use with SSH to manually recover from an error
  • print-config: Print the kubeconfig obtained from the server after installation
  • skip-install: Skip the k3s installer
  • ssh-key: The ssh key to use for remote login (default "~/.ssh/id_rsa")
  • ssh-port: The port on which to connect for ssh (default 22)
  • sudo: Use sudo for installation. e.g. set to false when using the root user and no sudo is available. (default true)
  • tls-san: Use an additional IP or hostname for the API server
  • token: the token used to encrypt the datastore, must be the same token for all nodes

Using a different network interface for inter-node communication

The following section is optional and only applies, if you have more than one network interface on your nodes.

By default, k3s uses flannel as network fabric with VXLAN as network backend. In short this means the k3s nodes will communicate via VXLAN between each other. It's important to keep in mind that the traffic through the VXLAN is not encrypted. However, this per se is not an issue in itself - see the following quite funny blog post about what this means for the general security context: https://blog.ipspace.net/2015/04/omg-vxlan-encapsulation-has-no-security.html.

By default, k3s uses the first non-loopback network interface to establish the flannel network. This might be good or might be bad, depending on your network configuration. If you have a private network where your nodes participate, it might be wise to use this network instead of a public one. To check your network configuration, run ip a or ifconfig. The output might be similar to

11: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3    inet 127.0.0.1/8 scope host lo
4       valid_lft forever preferred_lft forever
5    inet6 ::1/128 scope host
6       valid_lft forever preferred_lft forever
72: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
8    link/ether xxxxxxxxxxxxxxxxxxxxxx
9    altname enp0s18
10    inet xxxxxxxxxxxx/22 brd xxxxxxxxx scope global eth0
11       valid_lft forever preferred_lft forever
12    inet6 xxxxxxxxxxxxxxxxxxxx/64 scope global
13       valid_lft forever preferred_lft forever
14    inet6 xxxxxxxxxxxxxxxxxxxxxx/64 scope link
15       valid_lft forever preferred_lft forever
163: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
17    link/ether xxxxxxxxxxxxxxxxxxxx
18    altname enp0s19
19    inet 10.0.0.1/22 brd 10.0.3.255 scope global eth1
20       valid_lft forever preferred_lft forever
21    inet6 fe80::bc06:19ff:fe3d:1597/64 scope link
22       valid_lft forever preferred_lft forever

In the above output, I have 3 interfaces:

  • lo: The loopback interface
  • eth0: A public network, accessing the internet
  • eth1: A private network

If you'd like to not use the first network interface, but a different one, add the following flag to the k3sup commands:

1--k3s-extra-args '--flannel-iface=eth1' # k3s will now use eth1 for flannel to create the network

Creating the k3s cluster therefore might look like

1k3sup install --ip <your-servers-ip> --user <your-servers-user> --context k3scontext --local-path ~/.kube/config --cluster --merge --k3s-extra-args '--flannel-iface=eth1'

Make sure to also add this flag to your k3sup join commands!

Encrypting flannel traffic

As mentioned above, flannel uses VXLAN as network backend. This is not bad per se, however the traffic through VXLAN is not encrypted. If a bad actor gains access to the VXLAN network, the traffic is exposed. To increase resiliency against such attacks, flannel provides different backends which can encrypt traffic. Popular ones are ipsec or wireguard. As wireguard is the more modern and more secury option (with the default configs at least), we'll use wireguard in this guide.

  1. Install wireguard. Detailed instructions are provided here: https://www.wireguard.com/install/ . For Ubuntu/Debian, simply run sudo apt install wireguard.
  2. Add the following flag to your k3sup commands (all of them): --k3s-extra-args '--flannel-backend=wireguard-native'. If you already have a k3s-extra-args flag, make sure to merge them. Eg. --k3s-extra-args '--flannel-iface=eth1 --flannel-backend=wireguard-native'

Summary

We've seen, that k3sup makes it very easy to set up a highly available kubernetes cluster on remote computers. Even with advanced features like different network interface and traffic encryption. 3 simple commands is all it needs to spawn the k3s cluster and also merge your cluster config with already existing kubeconfigs. While I think k3s in itself is already great technology and very easy to use, k3sup makes it really a piece of cake to set up single-node or multi-node k3s clusters.

  1. Prerequisites for HA k3s cluster

    • 3 nodes where to install k3s servers
    • SSH access to all the 3 nodes

  2. Run the following commands to bootstrap your cluster and have your kubectl ready to go

    1k3sup install --ip <your-servers-ip> --user <your-servers-user> --context k3scontext --cluster --merge
    2

    3k3sup join --ip <ip-of-node-2> --user <user-of-node-2> --server-ip <your-servers-ip> --server-user <your-servers-user> --context k3scontext --server --merge
    4

    5k3sup join --ip <ip-of-node-3> --user <user-of-node-3> --server-ip <your-servers-ip> --server-user <your-servers-user> --context k3scontext --server --merge
    6

    7kubectl get node -o wide
    • your-servers-ip: The IP-Address where you can reach your first server via SSH
    • your-servers-user: The (SSH)-user of the first remote server
    • ip-of-node-2: The IP-Address where you can reach your second node via SSH
    • user-of-node-2: The (SSH)-user of the second remote node
    • ip-of-node-3: The IP-Address where you can reach your third node via SSH
    • user-of-node-3: The (SSH)-user of the third remote node
1That's it - happy k3s-ing.

------------------

Interested in how to train your very own Large Language Model?

We prepared a well-researched guide for how to use the latest advancements in Open Source technology to fine-tune your own LLM. This has many advantages like:

  • Cost control
  • Data privacy
  • Excellent performance - adjusted specifically for your intended use
Get your free LLM training guide

Need assistance?

Do you have any questions about the topic presented here? Or do you need someone to assist in implementing these areas? Do not hesitate to contact me.