온프레미스나 클라우드 인프라에서 직접 Kubernetes 클러스터를 구축하기

안녕하세요! VM에서 직접 클러스터를 구축해보던 중에 이것을 쉘 스크립트로 자동화 해놓으면 새로운 프로젝트 및 클러스터 구축에 앞으로 편하지 않을까 싶어서 한 번 쉘 스크립트를 통해 자동화 시켜보는 것은 어떨까 싶어서 만들어보게 되었습니다.

 

아래의 쉘 스크립트를 진행하기 전 수행하는 단계

VM 최소 요구사항 : CPUs 2, Memory 1736MB,

CPU 아키텍처 arm64 기준으로 작성된 내용입니다.

환경 버전 : Ubuntu 24.04 LTS, kubeadm 1.30

• kubectl 관련 명령어는 root 유저 수준의 관리자만 명령어 권한이 있기 때문에 UTM VM으로 리눅스 인스턴스 생성한 후에 아래의 명령어를 입력하여 비밀번호 초기화로 변경한 후에 다음 명령어를 통해서 관리자로 사용자 전환해준다.

sudo passwd root # root 비밀번호 초기화
su root # root 관리자로 사용자 전환

주의사항 : 아래의 쉘 스크립트 중 맞는 인스턴스의 쉘 스크립트를 복사한 다음의 특이사항 반영해서 실행시켜야합니다.

쉘 스크립트 수행

• master 인 경우 master 전용 쉘 스크립트 중 변수 API_SERVER_ADDRESS= “master ip addr” 값 찾아서 변경하여야함
• worker인 경우 쉘 스크립트 수정 없이 복사 붙여넣기 하여서 설치한 후에 worker 인스턴스에 terminal에 만들어두었던 토큰(토큰 생성 방법 아래 참고) 복붙으로 입력하면 join이 됨

cd / # root로 이동
vi setup_k8s_for_local_env.sh # 쉘 스크립트 생성 및 수정, 파일 이름 무관
# 중요한 사항 : master전용 쉘 스크립트, worker 전용 쉘 스크립트 구분할 것 
#            master 쉜 스크립트 안에 작성할 때 중간 정도에 master ip address 넣어야 하는 곳 있음
chmod +x setup_k8s_for_local_env.sh # 실행 권한 부여
./setup_k8s_for_local_env.sh # 쉘 스크립트 실행
# setup_k8s_for_local_env_master.sh 실행한 후에
kubeadm token create --print-join-command # only put in master terminal
# 생성한 토큰은 worker join할때 쓸 예정이기때문에 다른 곳에 복붙으로 기록해두어야 함
# UTM으로 하는 경우 네트워크 고정 IP 할당 설정 해야함

#UTM으로 생성한 인스턴스 재부팅 후에 자동으로 swapoff -a 하는 방법
#이 과정 없이 진행하게 되면 재부팅후에 swapon 이 되어서 kubectl 명령어 에러발생
#su root 로 root 계정 전환 추천함
sudo vi /etc/systemd/system/swapoff.service
#아래 내용 추가한 후에 :wq
[Unit]
Description=Turn off swap
After=network.target

[Service]
Type=oneshot
ExecStart=/sbin/swapoff -a

[Install]
WantedBy=multi-user.target

sudo systemctl enable swapoff.service
sudo systemctl start swapoff.service

#UTM 인스턴스 고정 IP 할당 하는 방법
#네트워크 설정 따로 하셨으면 여기는 skip 하셔도 됩니다.
#UTM UI에서 인스턴스 우클릭 edit -> Devices/network tab에서 Network mode : Bridged로 변경
#Bridged Interface는 en0으로 설정(자동으로 해도 될듯?)
#인스턴스 접속 후 su root로 root 계정 전환
# vi /etc/netplan/50-cloud-init.yaml (각자 이름은 다를수도 있으나 대체로 저 이름과 비슷함)
# 아래 내용 덮어쓰고 :wq
network:
    ethernets:
        enp0s1:
          addresses:
            - 192.168.0.10/24 #{{ 원하는 임의의 ip_addr_ }} 
          routes:
            - to: default
              via: 192.168.0.1 # 현재 호스트(노트북)의 네트워크 설정에서 라우터 엔드포인트 확인한 값
          nameservers:
            addresses: [ 8.8.8.8, 8.8.4.4 ]
    version: 2

netplan apply
# 인스턴스 재시작 해서 명령어(ip a)를 통해서 변경된 ip address 확인하고 맞으면 끝 

 

 

 

- setup_k8s_for_local_env_master.sh

# setup_k8s_for_local_env.sh
# for kubeadm, this for master instance
#!/bin/bash

echo "Updating APT package index..."
apt update
echo "APT package index updated."

echo "Installing Vim and Git..."
apt install -y vim git
echo "Vim and Git installation completed."

echo "Updating and upgrading apt packages..."
sudo apt update && sudo apt upgrade -y
echo "Completed updating and upgrading apt packages."

echo "Disabling swap..."
swapoff -a
echo "Swap disabled."

echo "Commenting out the swap partition in /etc/fstab..."
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
echo "Swap partition commented out in /etc/fstab."

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf > /dev/null
overlay
br_netfilter
EOF

echo "Loading br_netfilter module... and Load overlay module"
modprobe br_netfilter
modprobe overlay
echo "br_netfilter module loaded."


tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
echo "iptables configuration completed."

echo "Applying sysctl settings..."
sysctl --system
echo "Sysctl settings applied."

#install kubernetes packge
echo "Installing APT transport HTTPS and other dependencies..."
sudo apt install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
echo "APT transport HTTPS and other dependencies installed."

# add aarch arm64
sudo dpkg --add-architecture arm64

# Ensure /etc/apt/keyrings directory exists

sudo mkdir -p /etc/apt/keyrings
sudo chmod 755 /etc/apt/keyrings

# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg


# Add Docker repository
if [ ! -f /etc/apt/sources.list.d/docker.list ]; then
	echo "deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
fi

# Update apt cache
sudo apt-get update

# Install containerd.io.package
sudo apt-get install -y containerd.io

# Generate default containerd config
sudo containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1

# Modify containerd config for SystemdCgroup
echo "Modifying containerd config for SystemdCgroup..."
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
echo "Containerd config modification completed."

# Restart containerd service
echo "Restarting containerd service..."
sudo systemctl restart containerd
echo "Containerd service restarted."

# Enable containerd service on boot
echo "Enabling containerd service on boot..."
sudo systemctl enable containerd
echo "Containerd service enabled."

# Ensure /etc/apt/keyrings directory exists and add Kubernetes apt-key
echo "Adding Kubernetes apt-key..."
if [ ! -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg ]; then
  sudo mkdir -p /etc/apt/keyrings
  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | gpg --dearmor | sudo tee /etc/apt/keyrings/kubernetes-apt-keyring.gpg > /dev/null
  echo "Kubernetes apt-key added."
else
  echo "Kubernetes apt-key already exists. Skipping."
fi

# Add Kubernetes APT repository
echo "Adding Kubernetes APT repository..."
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null
echo "Kubernetes APT repository added."

# Update apt cache
echo "Updating apt cache..."
sudo apt-get update
echo "APT cache updated."

# Install kubeadm, kubelet, kubectl
echo "Installing kubeadm, kubelet, kubectl..."
sudo apt-get install -y kubeadm kubelet kubectl
echo "Installation of kubeadm, kubelet, kubectl completed."

# Mark Kubernetes packages to hold version
echo "Holding Kubernetes package versions..."
sudo apt-mark hold kubelet kubeadm kubectl
echo "Kubernetes package versions held."

# Enable and start kubelet service
echo "Enabling and starting kubelet service..."
sudo systemctl enable kubelet
sudo systemctl start kubelet
echo "Kubelet service enabled and started."

# Variables (Replace with your actual values if needed)
POD_NETWORK_CIDR="172.16.0.0/16" # CNI Calico에서 default 값 192.168.0.0/16
API_SERVER_ADDRESS="{{ master_ip_addr }}" # master instance ip addr

echo "Initializing Kubernetes Cluster..."
kubeadm init --pod-network-cidr=$POD_NETWORK_CIDR --apiserver-advertise-address=$API_SERVER_ADDRESS
if [ $? -eq 0 ]; then
  echo "Kubernetes Cluster initialized successfully."
else
  echo "Failed to initialize Kubernetes Cluster." >&2
  exit 1
fi

echo "Setting up kubeconfig for the user..."
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
echo "kubeconfig setup completed."

# Download Calico manifest
echo "Downloading Calico manifest..."
curl -o /tmp/calico.yaml https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/calico.yaml
if [ $? -eq 0 ]; then
  echo "Calico manifest downloaded successfully."
else
  echo "Failed to download Calico manifest." >&2
  exit 1
fi

# Apply Calico network plugin
echo "Applying Calico network plugin..."
kubectl apply -f /tmp/calico.yaml
if [ $? -eq 0 ]; then
  echo "Calico network plugin applied successfully."
else
  echo "Failed to apply Calico network plugin." >&2
  exit 1
fi

echo "Script execution completed."

 

 

- setup_k8s_for_local_env_worker.sh

 

# setup_k8s_for_local_env.sh
# for kubeadm, this for worker instance
#!/bin/bash

echo "Updating APT package index..."
apt update
echo "APT package index updated."

echo "Installing Vim and Git..."
apt install -y vim git
echo "Vim and Git installation completed."

echo "Script execution finished."

echo "Updating and upgrading apt packages..."
sudo apt update && sudo apt upgrade -y
echo "Completed updating and upgrading apt packages."

echo "Disabling swap..."
swapoff -a
echo "Swap disabled."

echo "Commenting out the swap partition in /etc/fstab..."
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
echo "Swap partition commented out in /etc/fstab."

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf > /dev/null
overlay
br_netfilter
EOF

echo "Loading br_netfilter module... and Load overlay module"
modprobe br_netfilter
modprobe overlay
echo "br_netfilter module loaded."


tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
EOF
echo "iptables configuration completed."

echo "Applying sysctl settings..."
sysctl --system
echo "Sysctl settings applied."

#install kubernetes packge
echo "Installing APT transport HTTPS and other dependencies..."
sudo apt install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
echo "APT transport HTTPS and other dependencies installed."

sudo dpkg --add-architecture arm64

sudo apt update

# Ensure /etc/apt/keyrings directory exists
if [ ! -d /etc/apt/keyrings ]; then
	sudo mkdir -p /etc/apt/keyrings
	sudo chmod 755 /etc/apt/keyrings
fi

# Add Docker GPG key
if [ ! -f /etc/apt/keyrings/docker.gpg ]; then
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor | sudo tee /etc/apt/keyrings/docker.gpg > /dev/null
fi

# Add Docker repository
if [ ! -f /etc/apt/sources.list.d/docker.list ]; then
	echo "deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
fi

# Update apt cache
sudo apt-get update

# Install containerd.io.package
sudo apt-get install -y containerd.io

# Generate default containerd config
sudo containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1

# Modify containerd config for SystemdCgroup
echo "Modifying containerd config for SystemdCgroup..."
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
echo "Containerd config modification completed."

# Restart containerd service
echo "Restarting containerd service..."
sudo systemctl restart containerd
echo "Containerd service restarted."

# Enable containerd service on boot
echo "Enabling containerd service on boot..."
sudo systemctl enable containerd
echo "Containerd service enabled."

# Ensure /etc/apt/keyrings directory exists and add Kubernetes apt-key
echo "Adding Kubernetes apt-key..."
if [ ! -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg ]; then
  sudo mkdir -p /etc/apt/keyrings
  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | gpg --dearmor | sudo tee /etc/apt/keyrings/kubernetes-apt-keyring.gpg > /dev/null
  echo "Kubernetes apt-key added."
else
  echo "Kubernetes apt-key already exists. Skipping."
fi

# Add Kubernetes APT repository
echo "Adding Kubernetes APT repository..."
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null
echo "Kubernetes APT repository added."

# Update apt cache
echo "Updating apt cache..."
sudo apt-get update
echo "APT cache updated."

# Install kubeadm, kubelet, kubectl
echo "Installing kubeadm, kubelet, kubectl..."
sudo apt-get install -y kubeadm kubelet kubectl
echo "Installation of kubeadm, kubelet, kubectl completed."

# Mark Kubernetes packages to hold version
echo "Holding Kubernetes package versions..."
sudo apt-mark hold kubelet kubeadm kubectl
echo "Kubernetes package versions held."

# Enable and start kubelet service
echo "Enabling and starting kubelet service..."
sudo systemctl enable kubelet
sudo systemctl start kubelet
echo "Kubelet service enabled and started."