Airgap k8s
11. Juni 2023 - Lesezeit: 13 Minuten
Eine Airgap Installation muss nicht kompliziert sein. Früher ging es ja auch, es gab kein Netzwerk und man hat die Software von einer (oder mehreren) Diskette(n) installiert. Wenn das damals funktioniert hat, dann geht das auch heute...
Zuerst einmal skizziere ich die Idee wie man Kubernetes airgaped installieren kann. Das Ganze soll natürlich einfach sein und schnell gehen. Wenn Du jetzt denkst, dass eine "normale" Kubernetes Installation schon kompliziert ist wie soll dann eine airgaped Installation einfach sein?
1. Voraussetzung
So eine Airgap Installation ist natürlich auf eine bestimmte Umgebung zugeschnitten. Und ich habe hier Alpine Linux in der Version 3.18 genommen. Das sollte jeder in 10 Minuten installiert bekommen.
2. Vorbereitung
Jetzt braucht es nur noch ein Script mit dem man das Installations Paket erstellt.
#!/bin/bash
STOPWATCH_START=$(date +%s)
################################################################################
# apk packages for airgap installation
readarray -t PACKAGES <<EOL_PACKAGES
# bash
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/readline-8.2.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/bash-5.2.15-r5.apk
# iptables
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libmnl-1.0.5-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnftnl-1.2.5-r1.apk
https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/libxtables-1.8.9-r4.apk
https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/iptables-1.8.9-r4.apk
https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/iptables-openrc-1.8.9-r4.apk
# kubelet
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubelet-1.27.2-r0.apk
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubelet-openrc-1.27.2-r0.apk
# iproute2
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/musl-fts-1.2.7-r5.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libbz2-1.0.8-r5.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libelf-0.189-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-minimal-6.3.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-tc-6.3.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-ss-6.3.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/iproute2-6.3.0-r0.apk
# socat
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/socat-1.7.4.4-r1.apk
# ethtool
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/ethtool-6.2-r1.apk
# conntrack-tools
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnfnetlink-1.0.2-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_conntrack-1.0.9-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_cthelper-1.0.1-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_cttimeout-1.0.1-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libnetfilter_queue-1.0.5-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/conntrack-tools-1.4.7-r1.apk
# cri-tools
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/cri-tools-1.27.0-r1.apk
# kubeadm
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubeadm-1.27.2-r0.apk
# containerd
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libseccomp-2.5.4-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/runc-1.1.7-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containerd-1.7.2-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containerd-ctr-1.7.2-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containerd-openrc-1.7.2-r0.apk
# kubectl
https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/kubectl-1.27.2-r0.apk
# k9s
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/k9s-0.27.4-r0.apk
# cni-plugins
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/cni-plugins-1.3.0-r0.apk
# cni-plugin-flannel
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/cni-plugin-flannel-1.1.2-r2.apk
# flannel
https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/flannel-0.21.5-r0.apk
https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/flannel-contrib-cni-0.21.5-r0.apk
https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/flannel-openrc-0.21.5-r0.apk
# podman
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libintl-0.21.1-r7.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libffi-3.4.4-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libmount-2.38.1-r8.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/pcre2-10.42-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/glib-2.76.3-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/conmon-2.1.7-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/catatonit-0.1.7-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/libslirp-4.7.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/slirp4netns-1.2.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/linux-pam-1.5.2-r10.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/shadow-libs-4.13-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/shadow-subids-4.13-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/containers-common-0.52.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libgcc-12.2.1_git20220924-r10.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/netavark-1.6.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/aardvark-dns-1.6.0-r0.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libgpg-error-1.47-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libassuan-2.5.5-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libgcrypt-1.10.2-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/npth-1.6-r4.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/pinentry-1.2.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gnupg-gpgconf-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libksba-1.6.3-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gdbm-1.23-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libsasl-2.1.28-r4.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/libldap-2.6.4-r3.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gnupg-dirmngr-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/sqlite-libs-3.41.2-r2.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gnupg-keyboxd-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gpg-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gpg-agent-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/gpgsm-2.4.1-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/gpgme-1.20.0-r1.apk
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/podman-4.5.1-r0.apk
# helm
https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/helm-3.11.3-r0.apk
EOL_PACKAGES
SKIP_PACKAGES=0
DOWNLOAD=true
INSTALL=true
RETRY=3
for PACKAGE in "${PACKAGES[@]}" ; do
# don't process commented out packages
[[ ${PACKAGE:0:1} = \# ]] && continue
# skip packages
[[ ${SKIP_PACKAGES} -gt 0 ]] && ((SKIP_PACKAGES--)) && continue
# parse package url
PACKAGE_PATH=$(echo ${PACKAGE} | cut -d/ -f4-)
PACKAGE_NAME=$(echo ${PACKAGE_PATH} | cut -d/ -f5-)
MIRROR=${PACKAGE%$PACKAGE_PATH}
PACKAGE_DIR=${PACKAGE_PATH%$PACKAGE_NAME}
# create package dir
mkdir -p ${PACKAGE_DIR}
# download package
if [[ ${DOWNLOAD} = true ]] ; then
# skip download if already available
[[ -f ${PACKAGE_PATH} ]] && continue
for N in $(seq 2 $RETRY) ; do
wget ${PACKAGE} -O ${PACKAGE_PATH}
RC=$?
if [[ $RC = 0 ]] ; then
break
else
echo "Try #$N in 10 seconds..."
sleep 10
fi
done
[[ $RC != 0 ]] && exit 1
fi
# install package
if [[ ${INSTALL} = true ]] ; then
apk add --repositories-file=/dev/null --allow-untrusted --no-network --no-cache ${PACKAGE_PATH}
# exit on install error
[[ $? != 0 ]] && exit 1
fi
done
################################################################################
# container images for airgap installation
readarray -t IMAGES <<EOL_IMAGES
# k8s
registry.k8s.io/coredns/coredns:v1.10.1
registry.k8s.io/defaultbackend-amd64:1.5
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/kube-apiserver:v1.27.2
registry.k8s.io/kube-controller-manager:v1.27.2
registry.k8s.io/kube-proxy:v1.27.2
registry.k8s.io/kube-scheduler:v1.27.2
registry.k8s.io/pause:3.9
# cni flannel
docker.io/flannel/flannel-cni-plugin:v1.1.2
docker.io/flannel/flannel:v0.22.0
# csi openebs
docker.io/openebs/linux-utils:3.4.0
docker.io/openebs/node-disk-exporter:2.1.0
docker.io/openebs/node-disk-manager:2.1.0
docker.io/openebs/node-disk-operator:2.1.0
docker.io/openebs/provisioner-localpv:3.4.0
# ingress controller nginx
registry.k8s.io/ingress-nginx/controller:v1.7.1
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
# cert-manager
quay.io/jetstack/cert-manager-acmesolver:v1.11.2
quay.io/jetstack/cert-manager-cainjector:v1.11.2
quay.io/jetstack/cert-manager-controller:v1.11.2
quay.io/jetstack/cert-manager-ctl:v1.11.2
quay.io/jetstack/cert-manager-webhook:v1.11.2
# monitoring prometheus and grafana
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0
quay.io/prometheus-operator/prometheus-config-reloader:v0.65.1
quay.io/prometheus/alertmanager:v0.25.0
quay.io/prometheus/node-exporter:v1.5.0
quay.io/prometheus/prometheus:v2.43.1
docker.io/prom/pushgateway:v1.5.1
docker.io/grafana/grafana:9.5.1
# wordpress
docker.io/bitnami/mariadb:10.11.3-debian-11-r5
docker.io/bitnami/wordpress:6.2.2-debian-11-r3
EOL_IMAGES
SKIP_IMAGES=0
PULL=true
CONTAINER_IMAGES=""
for IMAGE in "${IMAGES[@]}" ; do
# don't process commented out images
[[ ${IMAGE:0:1} = \# ]] && continue
# skip images
[[ ${SKIP_IMAGES} -gt 0 ]] && ((SKIP_IMAGES--)) && continue
# pull image
if [[ ${PULL} = true ]] ; then
podman pull ${IMAGE}
# exit on install error
[[ $? != 0 ]] && exit 1
fi
CONTAINER_IMAGES+=" "
CONTAINER_IMAGES+=${IMAGE}
done
SAVE=true
if [[ ${SAVE} = true ]] ; then
# save images as tar file
mkdir -p container
# cleanup container images tar file
[[ -f container/images.tar ]] && rm -f container/images.tar
# save all images in container images tar file
podman save --multi-image-archive --output container/images.tar ${CONTAINER_IMAGES}
fi
################################################################################
# helm charts
readarray -t HELM_CHARTS <<EOL_HELM_CHARTS
https://prometheus-community.github.io/helm-charts prometheus 22.5.0
https://grafana.github.io/helm-charts grafana 6.56.2
https://charts.jetstack.io cert-manager v1.11.2
https://kubernetes.github.io/ingress-nginx ingress-nginx 4.6.1
https://charts.bitnami.com/bitnami wordpress 16.1.2
EOL_HELM_CHARTS
SKIP_CHARTS=0
mkdir -p helm/
for CHART in "${HELM_CHARTS[@]}" ; do
# don't process commented out helm charts
[[ ${CHART:0:1} = \# ]] && continue
# skip helm charts
[[ ${SKIP_CHARTS} -gt 0 ]] && ((SKIP_CHARTS--)) && continue
# parse chart data
CHART_DATA=($CHART)
CHART_REPO="${CHART_DATA[0]}"
CHART_NAME="${CHART_DATA[1]}"
CHART_VERSION="${CHART_DATA[2]}"
# add helm repo
helm repo add ${CHART_NAME} ${CHART_REPO}
# create folder for helm chart
mkdir -p helm/${CHART_NAME}/
# pull helm chart
helm pull ${CHART_NAME}/${CHART_NAME} --version ${CHART_VERSION}
# exit on pull error
[[ $? != 0 ]] && exit 1
# move helmchart to folder
mv ${CHART_NAME}-${CHART_VERSION}.tgz helm/${CHART_NAME}/
done
################################################################################
# download manifests for cni (flannel) and csi (openebs)
mkdir -p manifest
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml -O manifest/kube-flannel.yaml
wget https://openebs.github.io/charts/openebs-operator.yaml -O manifest/openebs-operator.yaml
# patch openebs-operator because udev is not available for alpine linux
sed -i '497s/true/false/' manifest/openebs-operator.yaml
sed -i '595,596s/^ /#/' manifest/openebs-operator.yaml
sed -i '639,642s/^ /#/' manifest/openebs-operator.yaml
################################################################################
# create setup.sh
cat - > setup.sh <<EOF_SETUP
#!/bin/sh
################################################################################
# airgap no repos
echo "# airgap no repos" > /etc/apk/repositories
################################################################################
# add kernel module for networking stuff
echo "br_netfilter" > /etc/modules-load.d/k8s.conf
modprobe br_netfilter
echo "net.bridge.bridge-nf-call-iptables=1" >> /etc/sysctl.conf
sysctl net.bridge.bridge-nf-call-iptables=1
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl net.ipv4.ip_forward=1
################################################################################
# disable swap
sed -ie '/swap/ s/^/#/' /etc/fstab
swapoff -a
################################################################################
# fix prometheus errors
mount --make-rshared /
echo "#!/bin/sh" > /etc/local.d/sharedmetrics.start
echo "mount --make-rshared /" >> /etc/local.d/sharedmetrics.start
chmod +x /etc/local.d/sharedmetrics.start
rc-update add local default
################################################################################
# install packages
packages=\$(find alpine -name "*.apk")
apk add --repositories-file=/dev/null --allow-untrusted --no-network --no-cache \$packages
################################################################################
# add services and start container runtime
rc-update add kubelet
rc-update add containerd
# hotfix pause container use same version as kubernetes
sed -i 's/pause:3.8/pause:3.9/' /etc/containerd/config.toml
/etc/init.d/containerd start && sleep 10
################################################################################
# import container images
echo "Be patient import container images..."
ctr -n=k8s.io image import container/images.tar
################################################################################
# create cluster
kubeadm init --pod-network-cidr=10.244.0.0/16 --node-name=\$HOSTNAME
################################################################################
# copy kube config
mkdir ~/.kube
ln -s /etc/kubernetes/admin.conf ~/.kube/config
################################################################################
# remove no schedule taint for control plane
kubectl taint nodes \$HOSTNAME node-role.kubernetes.io/control-plane=:NoSchedule-
################################################################################
# add overlay network flannel
kubectl apply -f manifest/kube-flannel.yaml
################################################################################
# add storage openebs
kubectl apply -f manifest/openebs-operator.yaml
# set default storage class
kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
################################################################################
# cleanup
rm -rf setup.sh alpine/ container/ helm/ manifest/
EOF_SETUP
chmod +x setup.sh
################################################################################
# create self extracting archive
NAME="k8s-install"
VERSION="1.27.2-edge"
TAR_FILE="${NAME}-${VERSION}.tgz"
SELF_EXTRACTABLE="$TAR_FILE.self"
PACK=true
if [[ ${PACK} = true ]] ; then
echo "Be patient creating self extracting archive..."
# pack and create self extracting archive
tar -czf ${TAR_FILE} alpine/ container/ helm/ manifest/ setup.sh
echo '#!/bin/sh' > $SELF_EXTRACTABLE
echo 'echo Be patient extracting archive...' >> $SELF_EXTRACTABLE
echo 'dd bs=`head -5 $0 | wc -c` skip=1 if=$0 | gunzip -c | tar -x' >> $SELF_EXTRACTABLE
echo 'exec ./setup.sh' >> $SELF_EXTRACTABLE
echo '######################################################################' >> $SELF_EXTRACTABLE
cat $TAR_FILE >> $SELF_EXTRACTABLE
chmod a+x $SELF_EXTRACTABLE
fi
################################################################################
# cleanup
CLEANUP=true
if [[ ${CLEANUP} = true ]] ; then
rm -rf $TAR_FILE setup.sh alpine/ container/ helm/ manifest/
fi
################################################################################
# finish
STOPWATCH_END=$(date +%s.%N)
DURATION=$(echo "$STOPWATCH_END - $STOPWATCH_START" | bc)
echo "build $SELF_EXTRACTABLE took $DURATION seconds"
Hier die wichtigsten Funktionen des Scripts:
- APK Pakete herunterladen
- Container Images herunterladen und in einem Archiv speichern
- Helm Charts herunterladen
- Setup Script erstellen
- Alle Artefakte in ein großes Archiv packen
- Aus dem Archiv ein selbst extrahierendes Installations Paket erstellen
Um das Script auszuführen wird bash benötigt.
node1:~# apk add bash
fetch http://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
(1/2) Installing readline (8.2.1-r1)
(2/2) Installing bash (5.2.15-r5)
Executing bash-5.2.15-r5.post-install
Executing busybox-1.36.1-r0.trigger
OK: 166 MiB in 71 packages
node1:~# ./k8s