Install and configure a high available Kubernetes cluster with Ansible

Lorenzo Garuti · August 17, 2022

This ansible role will install and configure a high available Kubernetes cluster. This repo automate the installation process of Kubernetes using kubeadm.

This repo is only a example on how to use Ansible automation to install and configure a Kubernetes cluster. For a production environment use Kubespray

Requirements

Install ansible, ipaddr and netaddr:

pip install -r requirements.txt

Download the role form GitHub:

ansible-galaxy install git+https://github.com/garutilorenzo/ansible-role-linux-kubernetes.git

Role Variables

This role accept this variables:

Var Required Default Desc
kubernetes_subnet yes 192.168.25.0/24 Subnet where Kubernetess will be deployed. If the VM or bare metal server has more than one interface, Ansible will filter the interface used by Kubernetes based on the interface subnet
disable_firewall no no If set to yes Ansible will disable the firewall.
kubernetes_version no 1.24.3 Kubernetes version to install
kubernetes_cri no containerd Kubernetes CRI to install.
kubernetes_cni no flannel Kubernetes CNI to install.
kubernetes_dns_domain no cluster.local Kubernetes default DNS domain
kubernetes_pod_subnet no 10.244.0.0/16 Kubernetes pod subnet
kubernetes_service_subnet no 10.96.0.0/12 Kubernetes service subnet
kubernetes_api_port no 6443 kubeapi listen port
setup_vip no no Setup kubernetes VIP addres using kube-vip
kubernetes_vip_ip no 192.168.25.225 Required if setup_vip is set to yes. Vip ip address for the control plane
kubevip_version no v0.4.3 kube-vip container version
install_longhorn no no Install Longhorn, Cloud native distributed block storage for Kubernetes.
longhorn_version no v1.3.1 Longhorn release.
install_nginx_ingress no no Install nginx ingress controller
nginx_ingress_controller_version no controller-v1.3.0 nginx ingress controller version
nginx_ingress_controller_http_nodeport no 30080 NodePort used by nginx ingress controller for the incoming http traffic
nginx_ingress_controller_https_nodeport no 30443 NodePort used by nginx ingress controller for the incoming https traffic
enable_nginx_ingress_proxy_protocol no no Enable nginx ingress controller proxy protocol mode
enable_nginx_real_ip no no Enable nginx ingress controller real-ip module
nginx_ingress_real_ip_cidr no 0.0.0.0/0 Required if enable_nginx_real_ip is set to yes Trusted subnet to use with the real-ip module
nginx_ingress_proxy_body_size no 20m nginx ingress controller max proxy body size
sans_base no [list of values, see defaults/main.yml] list of ip addresses or FQDN uset to sign the kube-api certificate

Extra Variables

This role accept an extra variable kubernetes_init_host. This variable is used when the cluster is bootstrapped for the first time. The value of this variable must be the hostname of one of the master nodes. When ansible will run on the matched host kubernetes will be initialized.

Cluster resource deployed

Whit this role Nginx ingress controller and Longhorn will be installed.

Nginx ingress controller

Nginx ingress controller is used as ingress controller.

The installation is the bare metal installation, the ingress controller then is exposed via a NodePort Service. You can customize the ports exposed by the NodePort service, use Role Variables to change this values.

Longhorn

Longhorn is a lightweight, reliable, and powerful distributed block storage system for Kubernetes.

Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes.

Vagrant

To test this role you can use Vagrant and Virtualbox to bring up a example infrastructure. Once you have downloaded this repo use Vagrant to start the virtual machines:

vagrant up

In the Vagrantfile you can inject your public ssh key directly in the authorized_keys of the vagrant user. You have to change the CHANGE_ME placeholder in the Vagrantfile. You can also adjust the number of the vm deployed by changing the NNODES variable (Default: 6)

Using this role

To use this role you follow the example in the examples/ dir. Adjust the hosts.ini file with your hosts and run the playbook:

lorenzo@mint-virtual:~$ ansible-playbook -i hosts-ubuntu.ini site.yml -e kubernetes_init_host=k8s-ubuntu-0

PLAY [kubemaster] ***************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : include_tasks] **********************************************************************************************************************
included: /home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/setup_repo_Debian.yml for k8s-ubuntu-0, k8s-ubuntu-1, k8s-ubuntu-2 => (item=/home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/setup_repo_Debian.yml)

TASK [ansible-role-kubernetes : Install required system packages] ***************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : Add Google GPG apt Key] *************************************************************************************************************
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Add K8s Repository] *****************************************************************************************************************
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : Add Docker GPG apt Key] *************************************************************************************************************
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : shell] ******************************************************************************************************************************
changed: [k8s-ubuntu-1]
changed: [k8s-ubuntu-2]
changed: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : Add Docker Repository] **************************************************************************************************************
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : setup] ******************************************************************************************************************************
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : include_tasks] **********************************************************************************************************************
included: /home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/preflight.yml for k8s-ubuntu-0, k8s-ubuntu-1, k8s-ubuntu-2

TASK [ansible-role-kubernetes : disable ufw] ************************************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-0]
ok: [k8s-ubuntu-1]

TASK [ansible-role-kubernetes : Install iptables-legacy] ************************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Remove zram-generator-defaults] *****************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : disable firewalld] ******************************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Put SELinux in permissive mode, logging actions that would be blocked.] *************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Disable SELinux] ********************************************************************************************************************
skipping: [k8s-ubuntu-0]
skipping: [k8s-ubuntu-1]
skipping: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : Install openssl] ********************************************************************************************************************
ok: [k8s-ubuntu-2]
ok: [k8s-ubuntu-1]
ok: [k8s-ubuntu-0]

TASK [ansible-role-kubernetes : load overlay kernel module] *********************************************************************************************************
changed: [k8s-ubuntu-1]
changed: [k8s-ubuntu-0]
changed: [k8s-ubuntu-2]

TASK [ansible-role-kubernetes : load br_netfilter kernel module] ****************************************************************************************************
changed: [k8s-ubuntu-1]
changed: [k8s-ubuntu-0]
changed: [k8s-ubuntu-2]

[...]
[...]
[...]

TASK [ansible-role-kubernetes : Add KUBELET_ROOT_DIR env var] *******************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Add KUBELET_ROOT_DIR env var, set value] ********************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Install longhorn] *******************************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Install longhorn storageclass] ******************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : include_tasks] **********************************************************************************************************************
included: /home/lorenzo/workspaces-local/ansible-role-kubernetes/tasks/install_nginx_ingress.yml for k8s-ubuntu-3, k8s-ubuntu-4, k8s-ubuntu-5

TASK [ansible-role-kubernetes : Check if ingress-nginx is installed] ************************************************************************************************
changed: [k8s-ubuntu-3 -> k8s-ubuntu-0(192.168.25.110)]

TASK [ansible-role-kubernetes : Install ingress-nginx] **************************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : render nginx_ingress_config.yml] ****************************************************************************************************
skipping: [k8s-ubuntu-3]

TASK [ansible-role-kubernetes : Apply nginx ingress config] *********************************************************************************************************
skipping: [k8s-ubuntu-3]

PLAY RECAP **********************************************************************************************************************************************************
k8s-ubuntu-0               : ok=78   changed=24   unreachable=0    failed=0    skipped=25   rescued=0    ignored=3   
k8s-ubuntu-1               : ok=52   changed=12   unreachable=0    failed=0    skipped=30   rescued=0    ignored=1   
k8s-ubuntu-2               : ok=52   changed=12   unreachable=0    failed=0    skipped=30   rescued=0    ignored=1
k8s-ubuntu-3               : ok=58   changed=30   unreachable=0    failed=0    skipped=35   rescued=0    ignored=1   
k8s-ubuntu-4               : ok=52   changed=28   unreachable=0    failed=0    skipped=27   rescued=0    ignored=1   
k8s-ubuntu-5               : ok=52   changed=28   unreachable=0    failed=0    skipped=27   rescued=0    ignored=1   

Now we have a Kubernetes cluster deployed in high available mode, we can check the status of the cluster:

root@k8s-ubuntu-0:~# kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
k8s-ubuntu-0   Ready    control-plane   139m   v1.24.3
k8s-ubuntu-1   Ready    control-plane   136m   v1.24.3
k8s-ubuntu-2   Ready    control-plane   136m   v1.24.3
k8s-ubuntu-3   Ready    <none>          117m   v1.24.3
k8s-ubuntu-4   Ready    <none>          117m   v1.24.3
k8s-ubuntu-5   Ready    <none>          117m   v1.24.3

Check the pods status:

root@k8s-ubuntu-0:~# kubectl get pods --all-namespaces
NAMESPACE         NAME                                           READY   STATUS      RESTARTS       AGE
ingress-nginx     ingress-nginx-admission-create-tsc8p           0/1     Completed   0              135m
ingress-nginx     ingress-nginx-admission-patch-48tpn            0/1     Completed   0              135m
ingress-nginx     ingress-nginx-controller-6dc865cd86-kfq88      1/1     Running     0              135m
kube-flannel      kube-flannel-ds-fm4s6                          1/1     Running     0              117m
kube-flannel      kube-flannel-ds-hhvxx                          1/1     Running     0              117m
kube-flannel      kube-flannel-ds-ngdtc                          1/1     Running     0              117m
kube-flannel      kube-flannel-ds-q5ncb                          1/1     Running     0              136m
kube-flannel      kube-flannel-ds-vq4kk                          1/1     Running     0              139m
kube-flannel      kube-flannel-ds-zshpf                          1/1     Running     0              137m
kube-system       coredns-6d4b75cb6d-8dh9h                       1/1     Running     0              139m
kube-system       coredns-6d4b75cb6d-xq98k                       1/1     Running     0              139m
kube-system       etcd-k8s-ubuntu-0                              1/1     Running     0              139m
kube-system       etcd-k8s-ubuntu-1                              1/1     Running     0              136m
kube-system       etcd-k8s-ubuntu-2                              1/1     Running     0              136m
kube-system       kube-apiserver-k8s-ubuntu-0                    1/1     Running     0              139m
kube-system       kube-apiserver-k8s-ubuntu-1                    1/1     Running     0              135m
kube-system       kube-apiserver-k8s-ubuntu-2                    1/1     Running     0              136m
kube-system       kube-controller-manager-k8s-ubuntu-0           1/1     Running     0              139m
kube-system       kube-controller-manager-k8s-ubuntu-1           1/1     Running     0              136m
kube-system       kube-controller-manager-k8s-ubuntu-2           1/1     Running     0              135m
kube-system       kube-proxy-59jqx                               1/1     Running     0              136m
kube-system       kube-proxy-8mjwr                               1/1     Running     0              139m
kube-system       kube-proxy-8nhbw                               1/1     Running     0              117m
kube-system       kube-proxy-j2rrx                               1/1     Running     0              117m
kube-system       kube-proxy-qwd5r                               1/1     Running     0              117m
kube-system       kube-proxy-vcs7g                               1/1     Running     0              137m
kube-system       kube-scheduler-k8s-ubuntu-0                    1/1     Running     0              139m
kube-system       kube-scheduler-k8s-ubuntu-1                    1/1     Running     0              136m
kube-system       kube-scheduler-k8s-ubuntu-2                    1/1     Running     0              135m
kube-system       kube-vip-k8s-ubuntu-0                          1/1     Running     1 (136m ago)   139m
kube-system       kube-vip-k8s-ubuntu-1                          1/1     Running     0              136m
kube-system       kube-vip-k8s-ubuntu-2                          1/1     Running     0              136m
longhorn-system   csi-attacher-dcb85d774-jrggr                   1/1     Running     0              114m
longhorn-system   csi-attacher-dcb85d774-slhqt                   1/1     Running     0              114m
longhorn-system   csi-attacher-dcb85d774-xcbxn                   1/1     Running     0              114m
longhorn-system   csi-provisioner-5d8dd96b57-74x6h               1/1     Running     0              114m
longhorn-system   csi-provisioner-5d8dd96b57-kdzdf               1/1     Running     0              114m
longhorn-system   csi-provisioner-5d8dd96b57-xmpjf               1/1     Running     0              114m
longhorn-system   csi-resizer-7c5bb5fd65-4262v                   1/1     Running     0              114m
longhorn-system   csi-resizer-7c5bb5fd65-mfjgv                   1/1     Running     0              114m
longhorn-system   csi-resizer-7c5bb5fd65-qw944                   1/1     Running     0              114m
longhorn-system   csi-snapshotter-5586bc7c79-bs2xn               1/1     Running     0              114m
longhorn-system   csi-snapshotter-5586bc7c79-d927b               1/1     Running     0              114m
longhorn-system   csi-snapshotter-5586bc7c79-v99t6               1/1     Running     0              114m
longhorn-system   engine-image-ei-766a591b-hrs6g                 1/1     Running     0              114m
longhorn-system   engine-image-ei-766a591b-n9fsn                 1/1     Running     0              114m
longhorn-system   engine-image-ei-766a591b-vxhbb                 1/1     Running     0              114m
longhorn-system   instance-manager-e-3dba6914                    1/1     Running     0              114m
longhorn-system   instance-manager-e-7bd8b1ff                    1/1     Running     0              114m
longhorn-system   instance-manager-e-aca0fdc4                    1/1     Running     0              114m
longhorn-system   instance-manager-r-244c040c                    1/1     Running     0              114m
longhorn-system   instance-manager-r-39bd81b1                    1/1     Running     0              114m
longhorn-system   instance-manager-r-3b7f12b1                    1/1     Running     0              114m
longhorn-system   longhorn-admission-webhook-858d86b96b-j5rcv    1/1     Running     0              135m
longhorn-system   longhorn-admission-webhook-858d86b96b-lphkq    1/1     Running     0              135m
longhorn-system   longhorn-conversion-webhook-576b5c45c7-4p55x   1/1     Running     0              135m
longhorn-system   longhorn-conversion-webhook-576b5c45c7-lq686   1/1     Running     0              135m
longhorn-system   longhorn-csi-plugin-f7zmn                      2/2     Running     0              114m
longhorn-system   longhorn-csi-plugin-hs58p                      2/2     Running     0              114m
longhorn-system   longhorn-csi-plugin-wfpfs                      2/2     Running     0              114m
longhorn-system   longhorn-driver-deployer-96cf98c98-7hzft       1/1     Running     0              135m
longhorn-system   longhorn-manager-92xws                         1/1     Running     0              116m
longhorn-system   longhorn-manager-b6knm                         1/1     Running     0              116m
longhorn-system   longhorn-manager-tg2zc                         1/1     Running     0              116m
longhorn-system   longhorn-ui-86b56b95c8-ctbvf                   1/1     Running     0              135m

we can see, longhorn, nginx ingress and all the kube-system pods.

We can also inspect the service of the nginx ingress controller:

root@k8s-ubuntu-0:~# kubectl get svc -n ingress-nginx
NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller                NodePort    10.111.203.177   <none>        80:30080/TCP,443:30443/TCP   136m
ingress-nginx-controller-admission      ClusterIP   10.105.11.11     <none>        443/TCP                      136m

we can see the nginx ingress controller listening port, in this case the http port is 30080 and the https port is 30443. From an external machine we can test the ingress controller:

lorenzo@mint-virtual:~$ curl -v http://192.168.25.110:30080
*   Trying 192.168.25.110:30080...
* TCP_NODELAY set
* Connected to 192.168.25.110 (192.168.25.110) port 30080 (#0)
> GET / HTTP/1.1
> Host: 192.168.25.110:30080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Wed, 17 Aug 2022 12:26:17 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host 192.168.25.110 left intact

Twitter, Facebook