Airgap k8s

11. Juni 2023 - Lesezeit: 13 Minuten

Eine Airgap Installation muss nicht kompliziert sein. Früher ging es ja auch, es gab kein Netzwerk und man hat die Software von einer (oder mehreren) Diskette(n) installiert. Wenn das damals funktioniert hat, dann geht das auch heute...

Zuerst einmal skizziere ich die Idee wie man Kubernetes airgaped installieren kann. Das Ganze soll natürlich einfach sein und schnell gehen. Wenn Du jetzt denkst, dass eine "normale" Kubernetes Installation schon kompliziert ist wie soll dann eine airgaped Installation einfach sein?

1. Voraussetzung

So eine Airgap Installation ist natürlich auf eine bestimmte Umgebung zugeschnitten. Und ich habe hier Alpine Linux in der Version 3.18 genommen. Das sollte jeder in 10 Minuten installiert bekommen. 

2. Vorbereitung

Jetzt braucht es nur noch ein Script mit dem man das Installations Paket erstellt.

#!/bin/bash

STOPWATCH_START=$(date +%s)

################################################################################
# apk packages for airgap installation
readarray -t PACKAGES <<EOL_PACKAGES
# bash
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/readline-8.2.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/bash-5.2.15-r5.apk
# iptables
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libmnl-1.0.5-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnftnl-1.2.5-r1.apk
https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/libxtables-1.8.9-r4.apk
https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/iptables-1.8.9-r4.apk
https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/iptables-openrc-1.8.9-r4.apk
# kubelet
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubelet-1.27.2-r0.apk
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubelet-openrc-1.27.2-r0.apk
# iproute2
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/musl-fts-1.2.7-r5.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libbz2-1.0.8-r5.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libelf-0.189-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-minimal-6.3.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-tc-6.3.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-ss-6.3.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-6.3.0-r0.apk
# socat
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/socat-1.7.4.4-r1.apk
# ethtool
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/ethtool-6.2-r1.apk
# conntrack-tools
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnfnetlink-1.0.2-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_conntrack-1.0.9-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_cthelper-1.0.1-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_cttimeout-1.0.1-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_queue-1.0.5-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/conntrack-tools-1.4.7-r1.apk
# cri-tools
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/cri-tools-1.27.0-r1.apk
# kubeadm
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubeadm-1.27.2-r0.apk
# containerd
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libseccomp-2.5.4-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/runc-1.1.7-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containerd-1.7.2-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containerd-ctr-1.7.2-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containerd-openrc-1.7.2-r0.apk
# kubectl
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubectl-1.27.2-r0.apk
# k9s
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/k9s-0.27.4-r0.apk
# cni-plugins
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/cni-plugins-1.3.0-r0.apk
# cni-plugin-flannel
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/cni-plugin-flannel-1.1.2-r2.apk
# flannel
https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/flannel-0.21.5-r0.apk
https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/flannel-contrib-cni-0.21.5-r0.apk
https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/flannel-openrc-0.21.5-r0.apk
# podman
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libintl-0.21.1-r7.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libffi-3.4.4-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libmount-2.38.1-r8.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/pcre2-10.42-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/glib-2.76.3-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/conmon-2.1.7-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/catatonit-0.1.7-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/libslirp-4.7.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/slirp4netns-1.2.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/linux-pam-1.5.2-r10.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/shadow-libs-4.13-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/shadow-subids-4.13-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containers-common-0.52.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libgcc-12.2.1_git20220924-r10.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/netavark-1.6.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/aardvark-dns-1.6.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libgpg-error-1.47-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libassuan-2.5.5-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libgcrypt-1.10.2-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/npth-1.6-r4.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/pinentry-1.2.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gnupg-gpgconf-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libksba-1.6.3-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gdbm-1.23-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libsasl-2.1.28-r4.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libldap-2.6.4-r3.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gnupg-dirmngr-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/sqlite-libs-3.41.2-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gnupg-keyboxd-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gpg-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gpg-agent-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gpgsm-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/gpgme-1.20.0-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/podman-4.5.1-r0.apk
# helm
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/helm-3.11.3-r0.apk
EOL_PACKAGES

SKIP_PACKAGES=0
DOWNLOAD=true
INSTALL=true
RETRY=3

for PACKAGE in "${PACKAGES[@]}" ; do

  # don't process commented out packages
  [[ ${PACKAGE:0:1} = \# ]] && continue

  # skip packages
  [[ ${SKIP_PACKAGES} -gt 0 ]] && ((SKIP_PACKAGES--)) && continue

  # parse package url
  PACKAGE_PATH=$(echo ${PACKAGE} | cut -d/ -f4-)
  PACKAGE_NAME=$(echo ${PACKAGE_PATH} | cut -d/ -f5-)
  MIRROR=${PACKAGE%$PACKAGE_PATH}
  PACKAGE_DIR=${PACKAGE_PATH%$PACKAGE_NAME}

  # create package dir
  mkdir -p ${PACKAGE_DIR}

  # download package
  if [[ ${DOWNLOAD} = true ]] ; then

    # skip download if already available
    [[ -f ${PACKAGE_PATH} ]] && continue

    for N in $(seq 2 $RETRY) ; do
      wget ${PACKAGE} -O ${PACKAGE_PATH} 
      RC=$?
      if [[ $RC = 0 ]] ; then 
        break
      else
        echo "Try #$N in 10 seconds..."
        sleep 10
      fi 
    done

    [[ $RC != 0 ]] && exit 1

  fi

  # install package
  if [[ ${INSTALL} = true ]] ; then

    apk add --repositories-file=/dev/null --allow-untrusted --no-network --no-cache ${PACKAGE_PATH}

    # exit on install error
    [[ $? != 0 ]] && exit 1

  fi

done

################################################################################
# container images for airgap installation
readarray -t IMAGES <<EOL_IMAGES
# k8s
registry.k8s.io/coredns/coredns:v1.10.1
registry.k8s.io/defaultbackend-amd64:1.5
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/kube-apiserver:v1.27.2
registry.k8s.io/kube-controller-manager:v1.27.2
registry.k8s.io/kube-proxy:v1.27.2
registry.k8s.io/kube-scheduler:v1.27.2
registry.k8s.io/pause:3.9
# cni flannel
docker.io/flannel/flannel-cni-plugin:v1.1.2
docker.io/flannel/flannel:v0.22.0
# csi openebs
docker.io/openebs/linux-utils:3.4.0
docker.io/openebs/node-disk-exporter:2.1.0
docker.io/openebs/node-disk-manager:2.1.0
docker.io/openebs/node-disk-operator:2.1.0
docker.io/openebs/provisioner-localpv:3.4.0
# ingress controller nginx
registry.k8s.io/ingress-nginx/controller:v1.7.1
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
# cert-manager
quay.io/jetstack/cert-manager-acmesolver:v1.11.2
quay.io/jetstack/cert-manager-cainjector:v1.11.2
quay.io/jetstack/cert-manager-controller:v1.11.2
quay.io/jetstack/cert-manager-ctl:v1.11.2
quay.io/jetstack/cert-manager-webhook:v1.11.2
# monitoring prometheus and grafana
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0
quay.io/prometheus-operator/prometheus-config-reloader:v0.65.1
quay.io/prometheus/alertmanager:v0.25.0
quay.io/prometheus/node-exporter:v1.5.0
quay.io/prometheus/prometheus:v2.43.1
docker.io/prom/pushgateway:v1.5.1
docker.io/grafana/grafana:9.5.1
# wordpress
docker.io/bitnami/mariadb:10.11.3-debian-11-r5
docker.io/bitnami/wordpress:6.2.2-debian-11-r3
EOL_IMAGES

SKIP_IMAGES=0
PULL=true
CONTAINER_IMAGES=""

for IMAGE in "${IMAGES[@]}" ; do

  # don't process commented out images
  [[ ${IMAGE:0:1} = \# ]] && continue

  # skip images
  [[ ${SKIP_IMAGES} -gt 0 ]] && ((SKIP_IMAGES--)) && continue

  # pull image
  if [[ ${PULL} = true ]] ; then

    podman pull ${IMAGE}

    # exit on install error
    [[ $? != 0 ]] && exit 1

  fi

  CONTAINER_IMAGES+=" "
  CONTAINER_IMAGES+=${IMAGE}

done

SAVE=true

if [[ ${SAVE} = true ]] ; then

  # save images as tar file
  mkdir -p container

  # cleanup container images tar file
  [[ -f container/images.tar ]] && rm -f container/images.tar

  # save all images in container images tar file
  podman save --multi-image-archive --output container/images.tar ${CONTAINER_IMAGES}

fi

################################################################################
# helm charts
readarray -t HELM_CHARTS <<EOL_HELM_CHARTS
https://prometheus-community.github.io/helm-charts prometheus 22.5.0
https://grafana.github.io/helm-charts grafana 6.56.2
https://charts.jetstack.io cert-manager v1.11.2
https://kubernetes.github.io/ingress-nginx ingress-nginx 4.6.1
https://charts.bitnami.com/bitnami wordpress 16.1.2
EOL_HELM_CHARTS

SKIP_CHARTS=0

mkdir -p helm/

for CHART in "${HELM_CHARTS[@]}" ; do

  # don't process commented out helm charts
  [[ ${CHART:0:1} = \# ]] && continue

  # skip helm charts
  [[ ${SKIP_CHARTS} -gt 0 ]] && ((SKIP_CHARTS--)) && continue

  # parse chart data
  CHART_DATA=($CHART)
  CHART_REPO="${CHART_DATA[0]}"
  CHART_NAME="${CHART_DATA[1]}"
  CHART_VERSION="${CHART_DATA[2]}"

  # add helm repo
  helm repo add ${CHART_NAME} ${CHART_REPO}

  # create folder for helm chart
  mkdir -p helm/${CHART_NAME}/

  # pull helm chart
  helm pull ${CHART_NAME}/${CHART_NAME} --version ${CHART_VERSION}

  # exit on pull error
  [[ $? != 0 ]] && exit 1

  # move helmchart to folder
  mv ${CHART_NAME}-${CHART_VERSION}.tgz helm/${CHART_NAME}/

done

################################################################################
# download manifests for cni (flannel) and csi (openebs)
mkdir -p manifest

wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml -O manifest/kube-flannel.yaml

wget https://openebs.github.io/charts/openebs-operator.yaml -O manifest/openebs-operator.yaml
# patch openebs-operator because udev is not available for alpine linux
sed -i '497s/true/false/' manifest/openebs-operator.yaml
sed -i '595,596s/^ /#/' manifest/openebs-operator.yaml
sed -i '639,642s/^ /#/' manifest/openebs-operator.yaml

################################################################################
# create setup.sh
cat - > setup.sh <<EOF_SETUP
#!/bin/sh

################################################################################
# airgap no repos
echo "# airgap no repos" > /etc/apk/repositories

################################################################################
# add kernel module for networking stuff
echo "br_netfilter" > /etc/modules-load.d/k8s.conf
modprobe br_netfilter
echo "net.bridge.bridge-nf-call-iptables=1" >> /etc/sysctl.conf
sysctl net.bridge.bridge-nf-call-iptables=1
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl net.ipv4.ip_forward=1

################################################################################
# disable swap
sed -ie '/swap/ s/^/#/' /etc/fstab
swapoff -a

################################################################################
# fix prometheus errors
mount --make-rshared /
echo "#!/bin/sh" > /etc/local.d/sharedmetrics.start
echo "mount --make-rshared /" >> /etc/local.d/sharedmetrics.start
chmod +x /etc/local.d/sharedmetrics.start
rc-update add local default

################################################################################
# install packages
packages=\$(find alpine -name "*.apk")
apk add --repositories-file=/dev/null --allow-untrusted --no-network --no-cache \$packages

################################################################################
# add services and start container runtime
rc-update add kubelet
rc-update add containerd
# hotfix pause container use same version as kubernetes
sed -i 's/pause:3.8/pause:3.9/' /etc/containerd/config.toml
/etc/init.d/containerd start && sleep 10

################################################################################
# import container images
echo "Be patient import container images..."
ctr -n=k8s.io image import container/images.tar

################################################################################
# create cluster
kubeadm init --pod-network-cidr=10.244.0.0/16 --node-name=\$HOSTNAME

################################################################################
# copy kube config
mkdir ~/.kube
ln -s /etc/kubernetes/admin.conf ~/.kube/config

################################################################################
# remove no schedule taint for control plane
kubectl taint nodes \$HOSTNAME node-role.kubernetes.io/control-plane=:NoSchedule-

################################################################################
# add overlay network flannel
kubectl apply -f manifest/kube-flannel.yaml

################################################################################
# add storage openebs
kubectl apply -f manifest/openebs-operator.yaml
# set default storage class
kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

################################################################################
# cleanup
rm -rf setup.sh alpine/ container/ helm/ manifest/
EOF_SETUP

chmod +x setup.sh

################################################################################
# create self extracting archive
NAME="k8s-install"
VERSION="1.27.2-edge"
TAR_FILE="${NAME}-${VERSION}.tgz"
SELF_EXTRACTABLE="$TAR_FILE.self"
PACK=true

if [[ ${PACK} = true ]] ; then

  echo "Be patient creating self extracting archive..."
  # pack and create self extracting archive
  tar -czf ${TAR_FILE} alpine/ container/ helm/ manifest/ setup.sh

  echo '#!/bin/sh' > $SELF_EXTRACTABLE
  echo 'echo Be patient extracting archive...' >> $SELF_EXTRACTABLE
  echo 'dd bs=`head -5 $0 | wc -c` skip=1 if=$0 | gunzip -c | tar -x' >> $SELF_EXTRACTABLE
  echo 'exec ./setup.sh' >> $SELF_EXTRACTABLE
  echo '######################################################################' >> $SELF_EXTRACTABLE

  cat $TAR_FILE >> $SELF_EXTRACTABLE
  chmod a+x $SELF_EXTRACTABLE

fi

################################################################################
# cleanup

CLEANUP=true

if [[ ${CLEANUP} = true ]] ; then
  rm -rf $TAR_FILE setup.sh alpine/ container/ helm/ manifest/
fi

################################################################################
# finish

STOPWATCH_END=$(date +%s.%N)
DURATION=$(echo "$STOPWATCH_END - $STOPWATCH_START" | bc)

echo "build $SELF_EXTRACTABLE took $DURATION seconds"

Hier die wichtigsten Funktionen des Scripts:

• APK Pakete herunterladen
• Container Images herunterladen und in einem Archiv speichern
• Helm Charts herunterladen
• Setup Script erstellen
• Alle Artefakte in ein großes Archiv packen
• Aus dem Archiv ein selbst extrahierendes Installations Paket erstellen 

Um das Script auszuführen wird bash benötigt.

node1:~# apk add bash
fetch http://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
(1/2) Installing readline (8.2.1-r1)
(2/2) Installing bash (5.2.15-r5)
Executing bash-5.2.15-r5.post-install
Executing busybox-1.36.1-r0.trigger
OK: 166 MiB in 71 packages
node1:~# ./k8s-build.sh
Connecting to dl-cdn.alpinelinux.org (151.101.2.133:443)
saving to 'alpine/v3.18/main/x86_64/readline-8.2.1-r1.apk'
readline-8.2.1-r1.ap 100% |*************************************************************************************|  121k  0:00:00 ETA
'alpine/v3.18/main/x86_64/readline-8.2.1-r1.apk' saved
OK: 166 MiB in 71 packages
....
Be patient creating self extracting archive...
build k8s-install-1.27.2-edge.tgz.self took 1089 seconds

Abhängig von der Rechenleistung und der Netzanbindung dauert das erstellen des Installations Pakets ein paar Minuten (auf meinem alten Laptop mit 50 Mbit Internet ca. 18 Minuten)

3. Installation

Die Installation ist dann ganz einfach. Man kopiert das selbst extrahierende Installations Paket auf den Zielrechner und führt es aus. Wenn man eine 100% Airgap Installation durchführen möchte macht man das mit Hilfe eines USB Sticks.

kopieren

node1:~# scp k8s-install-1.27.2-edge.tgz.self 192.168.178.52:~
root@192.168.178.52's password:
k8s-install-1.27.2-edge.tgz.self                                                                    69%  895MB  25.1MB/s   00:15 ETA

installieren

node2:~# ./k8s-install-1.27.2-edge.tgz.self
Be patient extracting archive...

Finish ;-)