kubernetes 는
오픈소스 기반의 컨테이너화 된 애플리케이션의 자동 디플로이, 스케일링 등을 제공하는 관리 시스템이다. 궁극적 목적은 여러 클러스터의 호스트 간 애플리케이션 컨테이너의 배치, 스케일링, 운영자동화 등을 위한 플랫폼 제공이다.
쿠버네티스 관리를 위한 도구이므로 쿠버네티스가 필요하다.
kubernetes 설치
https://dl.k8s.io/release/stable.txt에 최근 안정 버전 정보가 적혀있다.
글 작성일 기준으로 v1.22.1 이다. 아래 명령줄의 괄호 안 curl 명령을 v1.22.1로 치환해도 무방하다. (-k 옵션은 인증서 검증 생략 옵션이다.)
haedong@haedong:~:]$ curl -LO -k "https://dl.k8s.io/release/$(curl -L -s -k https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 154 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 44.7M 100 44.7M 0 0 4892k 0 0:00:09 0:00:09 --:--:-- 10.3M
또는
haedong@haedong:~:]$ curl -LO -k "https://dl.k8s.io/release/v1.22.1/bin/linux/amd64/kubectl"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 154 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 44.7M 100 44.7M 0 0 4004k 0 0:00:11 0:00:11 --:--:-- 6095k
다운로드한 파일 검증
haedong@haedong:~:]$ curl -LO "https://dl.k8s.io/$(curl -L -s % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 154 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 64 100 64 0 0 52 0 0:00:01 0:00:01 --:--:-- 52
haedong@haedong:~:]$ echo "$(<kubectl.sha256) kubectl" | sha256sum --check
kubectl: 성공
kubectl 설치
haedong@haedong:~:]$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
[sudo] haedong의 암호:
haedong@haedong:~:]$ ll /usr/local/bin/ | grep kubectl
-rwxr-xr-x 1 root root 46907392 8월 24 15:06 kubectl
haedong@haedong:~:]$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
컨테이너 런타임 설치
containerd
haedong@haedong:~:]$ sudo vi /etc/modules-load.d/containerd.conf
# 아래 내용을 붙여 넣는다.
overlay
br_netfilter
haedong@haedong:~:]$ sudo modprobe overlay
haedong@haedong:~:]$ sudo modprobe br_netfilter
haedong@haedong:~:]$ sudo vi /etc/sysctl.d/99-kubernetes-cri.conf
# 아래 내용을 붙여 넣는다.
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
haedong@haedong:~:]$ sudo sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-kubernetes-cri.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1
* Applying /etc/sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1
설치
Centos-Base repository 활성화
기본적으로 활성화 되어있지만, local repository를 구축했거나, 비활성화 했다면 활성화 해야 한다.
haedong@haedong:~:]$ sudo vi /etc/yum.repos.d/bak/CentOS-Base.repo
# [extras] 꼭지의 enabled 값이 0이라면 1로 수정한다.
[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Docker 리포지터리 추가
rpm 파일들을 다운로드한 뒤 여기와 여기를 참고하여 로컬 리포지터리를 구축해도 된다.
haedong@haedong:~:]$ sudo sudo yum install -y yum-utils
haedong@haedong:~:]$ sudo sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
오래된 버전의 docker 관련 패키지 삭제
haedong@haedong:~:]$ sudo sudo sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
Docker engine 설치
haedong@haedong:/home:]$ sudo yum install docker-ce docker-ce-cli containerd.io
Loaded plugins: fastestmirror
...중략...
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed
---> Package python-IPy.noarch 0:0.75-6.el7 will be installed
---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================================================================================================
Package Arch Version Repository Size
==========================================================================================================================================================================================
Installing:
containerd.io x86_64 1.4.9-3.1.el7 docker 30 M
docker-ce x86_64 3:20.10.8-3.el7 docker 23 M
docker-ce-cli x86_64 1:20.10.8-3.el7 docker 29 M
...중략...
slirp4netns x86_64 0.4.3-4.el7_8 local_centos-extra 81 k
Transaction Summary
==========================================================================================================================================================================================
Install 3 Packages (+13 Dependent packages)
Total download size: 96 M
...중략...
(16/16): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
...중략...
Verifying : libcgroup-0.41-21.el7.x86_64 16/16
Installed:
containerd.io.x86_64 0:1.4.9-3.1.el7 docker-ce.x86_64 3:20.10.8-3.el7 docker-ce-cli.x86_64 1:20.10.8-3.el7
...중략...
Complete!
Docker 서비스 시작
haedong@haedong:/home:]$ sudo systemctl start docker.service
컨테이너 런타임 설치
containerd 설치
haedong@haedong:/home:]$ sudo mkdir -p /etc/containerd
haedong@haedong:/home:]$ containerd config default | sudo tee /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0
[grpc]
address = "/run/containerd/containerd.sock"
tcp_address = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[ttrpc]
address = ""
uid = 0
gid = 0
[debug]
address = ""
uid = 0
gid = 0
level = ""
[metrics]
address = ""
grpc_histogram = false
[cgroup]
path = ""
[timeouts]
"io.containerd.timeout.shim.cleanup" = "5s"
"io.containerd.timeout.shim.load" = "5s"
"io.containerd.timeout.shim.shutdown" = "3s"
"io.containerd.timeout.task.state" = "2s"
[plugins]
[plugins."io.containerd.gc.v1.scheduler"]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"
[plugins."io.containerd.grpc.v1.cri"]
disable_tcp_service = true
stream_server_address = "127.0.0.1"
stream_server_port = "0"
stream_idle_timeout = "4h0m0s"
enable_selinux = false
selinux_category_range = 1024
sandbox_image = "k8s.gcr.io/pause:3.2"
stats_collect_period = 10
systemd_cgroup = false
enable_tls_streaming = false
max_container_log_line_size = 16384
disable_cgroup = false
disable_apparmor = false
restrict_oom_score_adj = false
max_concurrent_downloads = 3
disable_proc_mount = false
unset_seccomp_profile = ""
tolerate_missing_hugetlb_controller = true
disable_hugetlb_controller = true
ignore_image_defined_volumes = false
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "overlayfs"
default_runtime_name = "runc"
no_pivot = false
disable_snapshot_annotations = true
discard_unpacked_layers = false
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
runtime_type = ""
runtime_engine = ""
runtime_root = ""
privileged_without_host_devices = false
base_runtime_spec = ""
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
runtime_type = ""
runtime_engine = ""
runtime_root = ""
privileged_without_host_devices = false
base_runtime_spec = ""
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
runtime_engine = ""
runtime_root = ""
privileged_without_host_devices = false
base_runtime_spec = ""
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
max_conf_num = 1
conf_template = ""
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = ""
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
[plugins."io.containerd.internal.v1.opt"]
path = "/opt/containerd"
[plugins."io.containerd.internal.v1.restart"]
interval = "10s"
[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "shared"
[plugins."io.containerd.monitor.v1.cgroups"]
no_prometheus = false
[plugins."io.containerd.runtime.v1.linux"]
shim = "containerd-shim"
runtime = "runc"
runtime_root = ""
no_shim = false
shim_debug = false
[plugins."io.containerd.runtime.v2.task"]
platforms = ["linux/amd64"]
[plugins."io.containerd.service.v1.diff-service"]
default = ["walking"]
[plugins."io.containerd.snapshotter.v1.devmapper"]
root_path = ""
pool_name = ""
base_image_size = ""
async_remove = false
CRI-O 설치
# .conf 파일을 만들어 부팅 시 모듈을 로드한다
haedong@haedong:/home:]$ cat <<EOF | sudo tee /etc/modules-load.d/crio.conf
> overlay
> br_netfilter
> EOF
haedong@haedong:/home:]$ sudo modprobe overlay
haedong@haedong:/home:]$ sudo modprobe br_netfilter
# 요구되는 sysctl 파라미터 설정, 이 설정은 재부팅 간에도 유지된다.
haedong@haedong:/home:]$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
haedong@haedong:/home:]$ sudo sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-kubernetes-cri.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1
* Applying /etc/sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1
환경변수 설정
/etc/profile 등에 기재해도 된다.
haedong@haedong:/home:]$ export OS=CentOS_7
haedong@haedong:/home:]$ echo $OS
CentOS_7
$VERSION 을 사용자의 쿠버네티스 버전과 일치하는 CRI-O 버전으로 설정한다. 이번 예시의 경우 v1.21.2이다.
haedong@haedong:/home:]$ sudo curl -L -k -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
haedong@haedong:/home:]$ sudo curl -k -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
환경 탓인지 포스트를 작성하는 환경에서는 정상 동작을 하지 않는다.
안되면 다운받자. 여기에서 kubic_libcontainers repo 파일을 받거나 여기에 있는 rpm 파일들을 다운받아 yum을 이용해 설치, 여기에서 kubic_libcontainers repo 파일을 받거나 여기에 있는 rpm 파일들을 다운받고 yum으로 설치. 당연히 yum repository를 만들어놓고 사용해도 된다.
haedong@haedong:/home:]$ sudo yum -y install cri-o
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package cri-o.x86_64 0:1.21.2-4.2.el7 will be installed
...중략...
---> Package python-ipaddress.noarch 0:1.0.16-2.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================================================================================================
Package Arch Version Repository Size
==========================================================================================================================================================================================
Installing:
cri-o x86_64 1.21.2-4.2.el7 libcontainers_cri-o 25 M
...중략...
Transaction Summary
==========================================================================================================================================================================================
Install 1 Package (+19 Dependent packages)
Total download size: 67 M
Installed size: 210 M
Downloading packages:
(1/20): conmon-2.0.29-1.el7.3.2.x86_64.rpm ...중략...
(20/20): containernetworking-plugins-1.0.0-0.2.rc1.el7.6.2.x86_64.rpm | 39 MB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 83 MB/s | 67 MB 00:00:00
Running transaction check
...중략...
Installed:
cri-o.x86_64 0:1.21.2-4.2.el7
...중략...
Complete!
CRI-O 시작
※ 다음 절에서 docker 데몬을 실행할 것이므로 서비스를 시작하지 않는다.
# 다음절에서 docker 데몬을 시작할 것이므로 수행하지 않는다.
haedong@haedong:/home:]$ sudo systemctl daemon-reload
haedong@haedong:/home:]$ sudo systemctl enable crio --now
Created symlink from /etc/systemd/system/multi-user.target.wants/crio.service to /usr/lib/systemd/system/crio.service.
cgroup 드라이버 설정
haedong@haedong:/home:]$ sudo vi /etc/crio/crio.conf
# [crio.runtime] 섹션에 아래 내용을 추가한다.
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"
Docker
컨테이너 cgroup 관리에 systemd를 사용하기 위한 docker 데몬 구성
haedong@haedong:/home:]$ sudo mkdir /etc/docker
haedong@haedong:/home:]$ cat <<EOF | sudo tee /etc/docker/daemon.json
> {
> "exec-opts": ["native.cgroupdriver=systemd"],
> "log-driver": "json-file",
> "log-opts": {
> "max-size": "100m"
> },
> "storage-driver": "overlay2"
> }
> EOF
docker 재시작 및 서비스 설정
haedong@haedong:/home:]$ sudo systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
haedong@haedong:/home:]$ sudo systemctl daemon-reload
haedong@haedong:/home:]$ sudo systemctl restart docker
