Skip to main content

Kubeadm


warning

๊ธฐ๋ณธ์ ์œผ๋กœ ์„ค์น˜๋˜์–ด์•ผํ•˜๋Š” ํ”„๋กœ๊ทธ๋žจ์˜ ๊ฒฝ์šฐ ๋ฏธ๋ฆฌ ์„ค์น˜๋œ ์ด๋ฏธ์ง€๋ฅผ ์ค€๋น„ํ•˜๋ฉด ์ข‹์Šต๋‹ˆ๋‹ค.

Installation

  • ์ตœ์†Œ ํ•„์š” ์กฐ๊ฑด
    • 2 CPU
    • 2 GB memory
kubeadm-installation.yaml
---
- hosts: all
become: yes
vars:
version: 1.19.15-00
tasks:
- name: Install requirements
apt:
name: '{{ item }}'
state: latest
update_cache: yes
loop: ['apt-transport-https', 'ca-certificates', 'curl']

- name: Add kubernetes GPG apt key
apt_key:
url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
keyring: /usr/share/keyrings/kubernetes-archive-keyring.gpg

- name: Add kubernetes repository
apt_repository:
repo: deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
https://apt.kubernetes.io/ kubernetes-xenial main
filename: kubernetes

- name: Install kubeadm kubectl kubelet version={{ version }}
apt:
name: '{{ item }}={{ version }}'
update_cache: yes
force: yes
loop: ['kubelet', 'kubeadm', 'kubectl']

- name: Disable swap
command: swapoff -a
when: ansible_swaptotal_mb > 0

- name: Remove swapfile from /etc/fstab
replace:
path: /etc/fstab
regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
replace: '# \1'

- name: Add br_netfilter to modules-load.d
lineinfile:
path: /etc/modules-load.d/k8s.conf
line: br_netfilter
mode: 0644
create: yes

- name: modprobe br_netfilter
modprobe:
name: br_netfilter

- name: Add netbridge config ip6
lineinfile:
path: /etc/sysctl.d/k8s.conf
line: 'net.bridge.bridge-nf-call-ip6tables = 1'
mode: 0644
create: yes

- name: Add netbridge config ip4
lineinfile:
path: /etc/sysctl.d/k8s.conf
line: 'net.bridge.bridge-nf-call-iptables = 1'
mode: 0644
create: yes

- name: Update sysctl
shell: sysctl --system

- name: Add kubectl completion to /home/{{ ansible_user }}/.bashrc
lineinfile:
path: /home/{{ ansible_user }}/.bashrc
line: |
source <(kubectl completion bash)

alias k=kubectl
complete -F __start_kubectl k
mode: 0644

# AWS
- name: Set hostname to aws private dns name
shell: hostnamectl set-hostname $(curl http://169.254.169.254/latest/meta-data/local-hostname)

Complition

.bashrc
source <(kubectl completion bash)

alias k=kubectl
complete -F __start_kubectl k
.zshrc
source <(kubectl completion zsh)

alias k=kubectl
complete -F __start_kubectl k

Control plane node

ํ”„๋กœํ† ์ฝœ๋ฐฉํ–ฅํฌํŠธ ๋ฒ”์œ„๋ชฉ์ ์‚ฌ์šฉ์ž
TCP์ธ๋ฐ”์šด๋“œ6443*์ฟ ๋ฒ„๋„คํ‹ฐ์Šค API ์„œ๋ฒ„๋ชจ๋‘
TCP์ธ๋ฐ”์šด๋“œ2379-2380etcd ์„œ๋ฒ„ ํด๋ผ์ด์–ธํŠธ APIkube-apiserver, etcd
TCP์ธ๋ฐ”์šด๋“œ10250kubelet API์ž์ฒด, ์ปจํŠธ๋กค ํ”Œ๋ ˆ์ธ
TCP์ธ๋ฐ”์šด๋“œ10251kube-scheduler์ž์ฒด
TCP์ธ๋ฐ”์šด๋“œ10252kube-controller-manager์ž์ฒด
kubeadm-master.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
clusterName: kubernetes
kubernetesVersion: <version> # v1.19.15
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16 # pod network์— ํ• ๋‹น๋˜๋Š” IP ์ฃผ์†Œ ๋ฒ”์œ„
serviceSubnet: 10.96.0.0/12 # 10.96.0.0 ~ 10.111.255.255
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

Remote์—์„œ kubectl ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์‹ถ์€ ๊ฒฝ์šฐ ์•„๋ž˜ ๋‚ด์šฉ์„ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค.

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
certSANs:
- <private-ip>
- <public-ip>

ํด๋ผ์šฐ๋“œ ์„œ๋น„์Šค๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ cloud-provider ์‚ฌ์šฉ์„ ์œ„ํ•ด ์•„๋ž˜ ๋‚ด์šฉ์„ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค.

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
extraArgs:
cloud-provider: external
controllerManager:
extraArgs:
cloud-provider: external
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
warning

Highly Available topology ๊ตฌ์„ฑ์„ ์œ„ํ•ด์„œ๋Š” controlPlaneEndpoint๋ฅผ ๋กœ๋“œ๋ฐธ๋Ÿฐ์„œ๋กœ ์„ค์ •ํ•ด์•ผํ•ฉ๋‹ˆ๋‹ค. ์ถ”๊ฐ€ ์˜ต์…˜์ด ์žˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ…Œ์ŠคํŠธ ํ›„ ์—…๋ฐ์ดํŠธ ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.(TODO)

sudo kubeadm init --config kubeadm-master.yaml
mkdir -p $HOME/.kube \
&& sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config \
&& sudo chown $(id -u):$(id -g) $HOME/.kube/config
info

cloud์—์„œ self managed kubernetes๋ฅผ ์šด์˜ํ•˜๋Š” ๊ฒฝ์šฐ cloud-provider๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

calico

kubectl get pods -n kube-system

coredns๊ฐ€ Pending ์ƒํƒœ์ธ ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Calico ๊ธ€์„ ์ฝ์–ด๋ณด์‹œ๊ธฐ ๋ฐ”๋ž๋‹ˆ๋‹ค.

token

kubeadm token list
kubeadm token create --print-join-command --ttl 10m

hash

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex | sed 's/^.* //'

Control plane node ์ถ”๊ฐ€

warning

Highly Available topology ๊ตฌ์„ฑ์„ ์œ„ํ•ด์„œ๋Š” apiServerEndpoint๋ฅผ ๋กœ๋“œ๋ฐธ๋Ÿฐ์„œ๋กœ ์„ค์ •ํ•ด์„œ ์ง„ํ–‰ํ•ด์•ผํ•ฉ๋‹ˆ๋‹ค.

๊ธฐ์กด ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์—์„œ ์•„๋ž˜ ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด 2์‹œ๊ฐ„ ๋™์•ˆ ์œ ํšจํ•œ certificateKey๋ฅผ ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

kubeadm init phase upload-certs --upload-certs
kubeadm-master.yaml
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
bootstrapToken:
apiServerEndpoint: <load balencer>:6443
token: <token>
caCertHashes:
- <hash>
controlPlane:
localAPIEndpoint:
advertiseAddress: <master ip>
certificateKey: <certificate key>

ํด๋ผ์šฐ๋“œ ์„œ๋น„์Šค๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ cloud-provider ์‚ฌ์šฉ์„ ์œ„ํ•ด ์•„๋ž˜ ๋‚ด์šฉ์„ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค.

nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
sudo kubeadm join --config kubeadm-master.yaml

Worker node

ํ”„๋กœํ† ์ฝœ๋ฐฉํ–ฅํฌํŠธ ๋ฒ”์œ„๋ชฉ์ ์‚ฌ์šฉ์ž
TCP์ธ๋ฐ”์šด๋“œ10250kubelet API์ž์ฒด, ์ปจํŠธ๋กค ํ”Œ๋ ˆ์ธ
TCP์ธ๋ฐ”์šด๋“œ30000-32767NodePort์„œ๋น„์Šคโ€  ๋ชจ๋‘
kubeadm-worker.yaml
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
bootstrapToken:
apiServerEndpoint: <master endpoint>:6443
token: <token>
caCertHashes:
- <hash>

ํด๋ผ์šฐ๋“œ ์„œ๋น„์Šค๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ cloud-provider ์‚ฌ์šฉ์„ ์œ„ํ•ด ์•„๋ž˜ ๋‚ด์šฉ์„ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค.

nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
sudo kubeadm join --config kubeadm-worker.yaml

apiServer์— SAN ์ถ”๊ฐ€

SAN(Subject Alternative Name)

kubectl get configmap -n kube-system kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm-conf.yaml
apiServer:
certSANs:
- <private-ip>
- <public-ip>

์œ„ ๋‚ด์šฉ์„ ์ถ”๊ฐ€ํ•œ ํ›„ /etc/kubernetes/pki/apiserver.*์„ ์‚ญ์ œํ•˜๊ณ  ์•„๋ž˜ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

sudo kubeadm init phase certs apiserver --config kubeadm-conf.yaml
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

reset

sudo kubeadm reset -f
sudo rm -r /etc/kubernetes/manifests $HOME/.kube/config

Reference