Kubernetes Cluster

Akash leases are deployed via Kubernetes pods on provider clusters. This guide details the build of the provider’s Kubernetes control plane and worker nodes.
The setup of a Kubernetes cluster is the responsibility of the provider. This guide provides best practices and recommendations for setting up a Kubernetes cluster. This document is not a comprehensive guide and assumes pre-existing Kubernetes knowledge.

Create a Kubernetes cluster and start your first provider
Already have a Kubernetes cluster? Start here!

We recommend using the Kubespray project to deploy a cluster. Kubespray uses Ansible to make the deployment of a Kubernetes cluster easy.
The recommended minimum number of hosts is four for a production Provider Kubernetes cluster. This is meant to allow:
  • Three hosts serving as a redundant control plane/master instances
  • One host to serve as Kubernetes worker node to host provider leases
  • NOTE - the number of control plane nodes in the cluster should always be an odd number to allow the cluster to reach consensus.
While you could use a single Kubernetes host in testing and dev this would not be recommended for production.

Kubernetes Master Node Requirements
  • Minimum Specs
    • 2 CPU
    • 4 GB RAM
    • 30 GB disk
  • Recommended Specs
    • 4 CPU
    • 8 GB RAM
    • 40 GB disk
Kubernetes Work Node Requirements
  • Minimum Specs
    • 4 CPU
    • 8 GB
    • 100 GB disk
  • Recommendations
    • The more resources the better depending on your goal of maximum number of concurrent deployments.
    • Especially important to note that worker node needs to have as much CPU as possible, because if it's got, say 8 CPU and, 100 GB RAM, and 2 TB disk -> the cpu would likely be a bottleneck. Since people tend to deploy at least 1 CPU per deployment, the server could only host 8 deployments maximum and likely about 6 deployments as other ~2 CPU will be reserved by the Kubernetes system components.

Install Kubespray on a machine that has connectivity to the three hosts that will serve as the Kubernetes cluster.
Obtain Kubespray and navigate into the created local directory:
git clone https://github.com/kubernetes-sigs/kubespray.git ; cd kubespray

When you launch Kubespray it will use an Ansible playbook to deploy a Kubernetes cluster. In this step we will install Ansible.
Depending on your operating system it may be necessary to install OS patches, pip3, and virtualenv. Example steps for a Ubuntu OS are detailed below.
apt-get update ; apt-get install -y python3-pip virtualenv
Within the kubespray directory use the following commands for the purpose of:
  • Opening a Python virtual environment for the Ansible install
  • Installing Ansible and other necessary packages specified in the requirements.txt file
virtualenv --python=python3 venv
source venv/bin/activate
rm requirements.txt
cat > requirements.txt << 'EOF'
ansible==4.8.0
ansible-core==2.11.6
cryptography==2.8
jinja2==2.11.3
netaddr==0.7.19
pbr==5.4.4
jmespath==0.9.5
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.6
MarkupSafe==1.1.1
EOF
pip3 install -r requirements.txt

Ansible will configure the Kubernetes hosts via SSH. The user Ansible connects with must be root or have the capability of escalating privileges to root.
Commands in this step provide an example set up of SSH access to Kubernetes hosts and testing those connections.

ssh-keygen -t rsa -C $(hostname) -f "$HOME/.ssh/id_rsa" -P "" ; cat ~/.ssh/id_rsa.pub

  • The keys will be stored in the user’s home directory.
  • Use these commands to verify keys.
cd ~/.ssh ; ls
  • Files created:
authorized_keys id_rsa id_rsa.pub

  • Template:
ssh-copy-id -i ~/.ssh/id_rsa.pub <username>@<ip-address>
  • Example:
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

  • Ansible should be able to access Kubernetes hosts with no password
  • Template:
ssh -i ~/.ssh/id_rsa <username>@<ip-address>
  • Example:
ssh -i ~/.ssh/id_rsa [email protected]
ssh -i ~/.ssh/id_rsa [email protected]
ssh -i ~/.ssh/id_rsa [email protected]
ssh -i ~/.ssh/id_rsa [email protected]

Ansible will use an inventory file to determine the hosts Kubernetes should be installed on.

  • Use the following commands on the Ansible host and in the “kubespray” directory
  • Replace the IP addresses in the declare command with the addresses of your Kubernetes hosts
  • Running these commands will create a hosts.yaml file within the kubespray/inventory/akash directory
cp -rfp inventory/sample inventory/akash
declare -a IPS=(10.0.10.136 10.0.10.239 10.0.10.253 10.0.10.9)
CONFIG_FILE=inventory/akash/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
  • Expected result:
(venv) [email protected]:/home/ubuntu/kubespray# CONFIG_FILE=inventory/akash/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
DEBUG: Adding group all
DEBUG: Adding group kube_control_plane
DEBUG: Adding group kube_node
DEBUG: Adding group etcd
DEBUG: Adding group k8s_cluster
DEBUG: Adding group calico_rr
DEBUG: adding host node1 to group all
DEBUG: adding host node2 to group all
DEBUG: adding host node3 to group all
DEBUG: adding host node4 to group all
DEBUG: adding host node1 to group etcd
DEBUG: adding host node2 to group etcd
DEBUG: adding host node3 to group etcd
DEBUG: adding host node1 to group kube_control_plane
DEBUG: adding host node2 to group kube_control_plane
DEBUG: adding host node3 to group kube_control_plane
DEBUG: adding host node1 to group kube_node
DEBUG: adding host node2 to group kube_node
DEBUG: adding host node3 to group kube_node
DEBUG: adding host node4 to group kube_node
  • Example of the generated hosts.yaml file
  • Update the kube_control_plane category if needed with full list of hosts that should be master nodes
all:
hosts:
node1:
ansible_host: 10.0.10.136
ip: 10.0.10.136
access_ip: 10.0.10.136
node2:
ansible_host: 10.0.10.239
ip: 10.0.10.239
access_ip: 10.0.10.239
node3:
ansible_host: 10.0.10.253
ip: 10.0.10.253
access_ip: 10.0.10.253
node4:
ansible_host: 10.0.10.9
ip: 10.0.10.9
access_ip: 10.0.10.9
children:
kube_control_plane:
hosts:
node1:
node2:
node3:
kube_node:
hosts:
node1:
node2:
node3:
node4:
etcd:
hosts:
node1:
node2:
node3:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}

  • Open the hosts.yaml file in VI (Visual Editor) or nano to make the following file updates to the hosts.yaml file
  • Within the YAML file’s “all” stanza and prior to the “hosts” sub-stanza level - insert the following vars stanza
vars:
cluster_id: "1.0.0.1"
ansible_user: root
gvisor_enabled: true
  • The hosts.yaml file should look like this once finished
all:
vars:
cluster_id: "1.0.0.1"
ansible_user: root
gvisor_enabled: true
hosts:
node1:
ansible_host: 10.0.10.136
ip: 10.0.10.136
access_ip: 10.0.10.136
node2:
ansible_host: 10.0.10.239
ip: 10.0.10.239
access_ip: 10.0.10.239
node3:
ansible_host: 10.0.10.253
ip: 10.0.10.253
access_ip: 10.0.10.253
node4:
ansible_host: 10.0.10.9
ip: 10.0.10.9
access_ip: 10.0.10.9
children:
kube_control_plane:
hosts:
node1:
node2:
node3:
kube_node:
hosts:
node1:
node2:
node3:
node4:
etcd:
hosts:
node1:
node2:
node3:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}

In this section we will enable gVisor which provides basic container security.
  • From the “kubespray” directory:
cd inventory/akash/group_vars/k8s_cluster
  • Using VI or nano edit the k8s-cluster.yml file:
vi k8s-cluster.yml
  • Update the container_manager key if necessary to containerd
container_manager: containerd
  • From the “kubespray” directory:
cd inventory/akash/group_vars
  • Using VI or nano edit the etcd.yml file:
vi etcd.yml
  • Update the etcd_deployment_type key if necessary to host
etcd_deployment_type: host

If you are using a newer systemd version, your container will get stuck in ContainerCreating state on your provider with gVisor enabled. Please reference this document for details regarding this issue and the recommended workaround.

With inventory in place we are ready to build the Kubernetes cluster via Ansible.
  • Note - the cluster creation may take several minutes to complete
  • From the “kubespray” directory:
ansible-playbook -i inventory/akash/hosts.yaml -b -v --private-key=~/.ssh/id_rsa cluster.yml

A couple of quick Kubernetes cluster checks are in order before moving into next steps.
  • SSH into Kubernetes node01 (AKA Kubernetes master node)

kubectl get nodes
  • Example output from a healthy Kubernetes cluster:
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane,master 8m47s v1.22.4
node2 Ready control-plane,master 8m17s v1.22.4
node3 Ready control-plane,master 8m17s v1.22.4
node4 Ready <none> 7m11s v1.22.4

kubectl get pods -n kube-system
  • Example output of the pods that are the brains of the cluster
[email protected]:/home/ubuntu# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5788f6558-mzm64 1/1 Running 1 (4m53s ago) 4m54s
calico-node-2g4pr 1/1 Running 0 5m29s
calico-node-6hrj4 1/1 Running 0 5m29s
calico-node-9dqc4 1/1 Running 0 5m29s
calico-node-zt8ls 1/1 Running 0 5m29s
coredns-8474476ff8-9sgm5 1/1 Running 0 4m32s
coredns-8474476ff8-x67xd 1/1 Running 0 4m27s
dns-autoscaler-5ffdc7f89d-lnpmm 1/1 Running 0 4m28s
kube-apiserver-node1 1/1 Running 1 7m30s
kube-apiserver-node2 1/1 Running 1 7m13s
kube-apiserver-node3 1/1 Running 1 7m3s
kube-controller-manager-node1 1/1 Running 1 7m30s
kube-controller-manager-node2 1/1 Running 1 7m13s
kube-controller-manager-node3 1/1 Running 1 7m3s
kube-proxy-75s7d 1/1 Running 0 5m56s
kube-proxy-kpxtm 1/1 Running 0 5m56s
kube-proxy-stgwd 1/1 Running 0 5m56s
kube-proxy-vndvs 1/1 Running 0 5m56s
kube-scheduler-node1 1/1 Running 1 7m37s
kube-scheduler-node2 1/1 Running 1 7m13s
kube-scheduler-node3 1/1 Running 1 7m3s
nginx-proxy-node4 1/1 Running 0 5m58s
nodelocaldns-7znkj 1/1 Running 0 4m28s
nodelocaldns-g8dqm 1/1 Running 0 4m27s
nodelocaldns-gf58m 1/1 Running 0 4m28s
nodelocaldns-n88fj 1/1 Running 0 4m28s

Akash uses two Kubernetes Custom Resource Definitions (CRD) to store each deployment.
  • On the Kubernetes master node, download and install the Akash CRD files.

wget https://raw.githubusercontent.com/ovrclk/akash/master/pkg/apis/akash.network/crd.yaml
kubectl apply -f ./crd.yaml

kubectl get crd | grep akash

# kubectl get crd | grep akash
manifests.akash.network 2022-04-27T14:39:32Z
providerhosts.akash.network 2022-04-27T14:39:47Z

On the Kubernetes master node, download the network YAML file
wget https://raw.githubusercontent.com/ovrclk/akash/mainnet/main/_docs/kustomize/networking/network-policy-default-ns-deny.yaml
  • Install the YAML File
kubectl apply -f ./network-policy-default-ns-deny.yaml

  • Apply the Akash namespace YAML file and other networking customizations
git clone --depth 1 -b mainnet/main https://github.com/ovrclk/akash.git
cd akash
kubectl apply -f _docs/kustomize/networking/namespace.yaml
kubectl kustomize _docs/kustomize/akash-services/ | kubectl apply -f -
cat >> _docs/kustomize/akash-hostname-operator/kustomization.yaml <<'EOF'
images:
- name: ghcr.io/ovrclk/akash:stable
newName: ghcr.io/ovrclk/akash
newTag: 0.14.1
EOF
kubectl kustomize _docs/kustomize/akash-hostname-operator | kubectl apply -f -

The Akash provider requires an ingress controller in the Kubernetes cluster.

  • On the Kubernetes master node, download and install the ingress controller YAML files
wget https://raw.githubusercontent.com/ovrclk/akash/v0.14.1/_run/ingress-nginx-class.yaml
kubectl apply -f ./ingress-nginx-class.yaml
wget https://raw.githubusercontent.com/ovrclk/akash/v0.14.1/_run/ingress-nginx.yaml
kubectl apply -f ./ingress-nginx.yaml
  • Expected result:
[email protected]:~# kubectl apply -f ./ingress-nginx.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
configmap/ingress-nginx-tcp created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

A Kubernetes node needs to be labeled for ingress use. This will cause the NGINX ingress controller to live only on the labeled node.
NOTE - if a wildcard domain is created, the pointers should point to the labeled node's IP address. Additional nodes can be labeled to load balance any ingress communications.
kubectl label nodes node4 akash.network/role=ingress

The Kubelet process on Kubernetes worker nodes does not support swap. Issue the following command on each worker node to disable swap.
swapoff -a
We should in addition ensure that swap is permanently disabled on the Linux hosts via:
  • Open the /etc/fstab file
  • Search for a swap line and add a # (hashtag) sign in front of the line to comment out the entire line
vi /etc/fstab

The following firewall rules are applicable to internet facing Kubernetes components.
Akash Provider
8443/tcp - for manifest uploads
Akash Ingress Controller
80/tcp - for web app deployments
443/tcp - for web app deployments
30000-32767/tcp - for Kubernetes node port range for deployments
30000-32767/udp - for Kubernetes node port range for deployments

If local firewall instances are running on Kubernetes control-plane and worker nodes, add the following policies.
Etcd Key Value Store Policies
Ensure the following ports are open inbound on all Kubernetes etcd instances:
- 2379/tcp for client requests; (Kubernetes control plane to etcd)
- 2380/tcp for peer communication; (etcd to etcd communication)
API Server Policies
Ensure the following ports are open inbound on all Kubernetes API server instances:
- 6443/tcp - Kubernetes API server
Worker Node Policies
Ensure the following ports are open inbound on all Kubernetes worker nodes:
- 10250/tcp - Kubelet API server; (Kubernetes control plane to kubelet)
Copy link
On this page
Kubernetes Cluster Setup for Akash Providers
Overview
Quickstart Guides
STEP 1 - Clone the Kubespray Project
STEP 2 - Install Ansible
STEP 3 - Ansible Access to Kubernetes Cluster
STEP 4 - Ansible Inventory
STEP 5 - Enable gVisor
STEP 6 - Create Kubernetes Cluster
STEP 7 - Confirm Kubernetes Cluster
STEP 8 - Custom Resource Definition
STEP 9 - Network Policy And Akash Namespace
STEP 10 - Ingress Controller
STEP 11 - Disable Swap on Kubernetes Hosts
STEP 12 - Review Firewall Policies