[태그:] kubesphere

kubernetes¹그리스어로 조타수(키잡이)를 의미한다. 는
오픈소스 기반의 컨테이너화 된 애플리케이션의 자동 디플로이, 스케일링 등을 제공하는 관리 시스템이다. 궁극적 목적은 여러 클러스터의 호스트 간 애플리케이션 컨테이너의 배치, 스케일링, 운영자동화 등을 위한 플랫폼 제공이다. ²위키백과 발췌

쿠버네티스 관리를 위한 도구이므로 쿠버네티스가 필요하다.

kubernetes 설치
https://dl.k8s.io/release/stable.txt에 최근 안정 버전 정보가 적혀있다.
글 작성일 기준으로 v1.22.1 이다. 아래 명령줄의 괄호 안 curl 명령을 v1.22.1로 치환해도 무방하다. (-k 옵션은 인증서 검증 생략 옵션이다.)

haedong@haedong:~:]$ curl -LO -k "https://dl.k8s.io/release/$(curl -L -s -k https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   154    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100 44.7M  100 44.7M    0     0  4892k      0  0:00:09  0:00:09 --:--:-- 10.3M
또는
haedong@haedong:~:]$ curl -LO -k "https://dl.k8s.io/release/v1.22.1/bin/linux/amd64/kubectl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   154    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100 44.7M  100 44.7M    0     0  4004k      0  0:00:11  0:00:11 --:--:-- 6095k

다운로드한 파일 검증

haedong@haedong:~:]$ curl -LO "https://dl.k8s.io/$(curl -L -s   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   154    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    64  100    64    0     0     52      0  0:00:01  0:00:01 --:--:--    52
haedong@haedong:~:]$ echo "$(<kubectl.sha256) kubectl" | sha256sum --check
kubectl: 성공

kubectl 설치

haedong@haedong:~:]$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
[sudo] haedong의 암호:
haedong@haedong:~:]$ ll /usr/local/bin/ | grep kubectl
-rwxr-xr-x 1 root root 46907392  8월 24 15:06 kubectl
haedong@haedong:~:]$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

컨테이너 런타임 설치

containerd

haedong@haedong:~:]$ sudo vi /etc/modules-load.d/containerd.conf
# 아래 내용을 붙여 넣는다.
overlay
br_netfilter
haedong@haedong:~:]$ sudo modprobe overlay
haedong@haedong:~:]$ sudo modprobe br_netfilter
haedong@haedong:~:]$ sudo vi /etc/sysctl.d/99-kubernetes-cri.conf
# 아래 내용을 붙여 넣는다.
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
haedong@haedong:~:]$ sudo sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-kubernetes-cri.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1
* Applying /etc/sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1

설치

Centos-Base repository 활성화
기본적으로 활성화 되어있지만, local repository를 구축했거나, 비활성화 했다면 활성화 해야 한다.

haedong@haedong:~:]$ sudo vi /etc/yum.repos.d/bak/CentOS-Base.repo
# [extras] 꼭지의 enabled 값이 0이라면 1로 수정한다.
[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Docker 리포지터리 추가
rpm 파일들을 다운로드한 뒤 여기와 여기를 참고하여 로컬 리포지터리를 구축해도 된다.

haedong@haedong:~:]$ sudo sudo yum install -y yum-utils
haedong@haedong:~:]$ sudo sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

오래된 버전의 docker 관련 패키지 삭제

haedong@haedong:~:]$ sudo sudo sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

Docker engine 설치 ³자세한 설치 관련 내용은 docker 홈페이지를 참고.

haedong@haedong:/home:]$ sudo yum install docker-ce docker-ce-cli containerd.io
Loaded plugins: fastestmirror
...중략...
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed
---> Package python-IPy.noarch 0:0.75-6.el7 will be installed
---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================================================================================================
 Package                                            Arch                            Version                                             Repository                                   Size
==========================================================================================================================================================================================
Installing:
 containerd.io                                      x86_64                          1.4.9-3.1.el7                                       docker                                       30 M
 docker-ce                                          x86_64                          3:20.10.8-3.el7                                     docker                                       23 M
 docker-ce-cli                                      x86_64                          1:20.10.8-3.el7                                     docker                                       29 M
...중략...
 slirp4netns                                        x86_64                          0.4.3-4.el7_8                                       local_centos-extra                           81 k
Transaction Summary
==========================================================================================================================================================================================
Install  3 Packages (+13 Dependent packages)
Total download size: 96 M
...중략...
(16/16): setools-libs-3.3.8-4.el7.x86_64.rpm                                                                                                                       | 620 kB  00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
...중략...
  Verifying  : libcgroup-0.41-21.el7.x86_64                                                                                                                                         16/16
Installed:
  containerd.io.x86_64 0:1.4.9-3.1.el7                           docker-ce.x86_64 3:20.10.8-3.el7                           docker-ce-cli.x86_64 1:20.10.8-3.el7
...중략...
Complete!

Docker 서비스 시작

haedong@haedong:/home:]$ sudo systemctl start docker.service

컨테이너 런타임 설치

containerd 설치

haedong@haedong:/home:]$ sudo mkdir -p /etc/containerd
haedong@haedong:/home:]$ containerd config default | sudo tee /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = ""
  uid = 0
  gid = 0

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[plugins]
  [plugins."io.containerd.gc.v1.scheduler"]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"
  [plugins."io.containerd.grpc.v1.cri"]
    disable_tcp_service = true
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    stream_idle_timeout = "4h0m0s"
    enable_selinux = false
    selinux_category_range = 1024
    sandbox_image = "k8s.gcr.io/pause:3.2"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    disable_cgroup = false
    disable_apparmor = false
    restrict_oom_score_adj = false
    max_concurrent_downloads = 3
    disable_proc_mount = false
    unset_seccomp_profile = ""
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    ignore_image_defined_volumes = false
    [plugins."io.containerd.grpc.v1.cri".containerd]
      snapshotter = "overlayfs"
      default_runtime_name = "runc"
      no_pivot = false
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
        privileged_without_host_devices = false
        base_runtime_spec = ""
      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
        privileged_without_host_devices = false
        base_runtime_spec = ""
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          privileged_without_host_devices = false
          base_runtime_spec = ""
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      max_conf_num = 1
      conf_template = ""
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = ""
    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"
  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"
  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"
  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false
  [plugins."io.containerd.runtime.v1.linux"]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]
  [plugins."io.containerd.snapshotter.v1.devmapper"]
    root_path = ""
    pool_name = ""
    base_image_size = ""
    async_remove = false

CRI-O 설치

# .conf 파일을 만들어 부팅 시 모듈을 로드한다
haedong@haedong:/home:]$ cat <<EOF | sudo tee /etc/modules-load.d/crio.conf
> overlay
> br_netfilter
> EOF
haedong@haedong:/home:]$ sudo modprobe overlay
haedong@haedong:/home:]$ sudo modprobe br_netfilter
# 요구되는 sysctl 파라미터 설정, 이 설정은 재부팅 간에도 유지된다.
haedong@haedong:/home:]$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables  = 1
> net.ipv4.ip_forward                 = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
haedong@haedong:/home:]$ sudo sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-kubernetes-cri.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1
* Applying /etc/sysctl.conf ...
net.ipv4.conf.all.arp_notify = 1

환경변수 설정
/etc/profile 등에 기재해도 된다.

haedong@haedong:/home:]$ export OS=CentOS_7
haedong@haedong:/home:]$ echo $OS
CentOS_7

$VERSION 을 사용자의 쿠버네티스 버전과 일치하는 CRI-O 버전으로 설정한다. 이번 예시의 경우 v1.21.2이다. ⁴작성일 기준으로 kubuernetes는 1.22.1이었는데 CRI-O는 1.21.2까지 존재한다. 그래서 1.22.1 이 아니고 1.21.2

haedong@haedong:/home:]$ sudo curl -L -k -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
haedong@haedong:/home:]$ sudo curl -k -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0

환경 탓인지 포스트를 작성하는 환경에서는 정상 동작을 하지 않는다.
안되면 다운받자. 여기에서 kubic_libcontainers repo 파일을 받거나 여기에 있는 rpm 파일들을 다운받아 yum을 이용해 설치, 여기에서 kubic_libcontainers repo 파일을 받거나 여기에 있는 rpm 파일들을 다운받고 yum으로 설치. 당연히 yum repository를 만들어놓고 사용해도 된다.

haedong@haedong:/home:]$ sudo yum -y install cri-o
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package cri-o.x86_64 0:1.21.2-4.2.el7 will be installed
...중략...
---> Package python-ipaddress.noarch 0:1.0.16-2.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================================================================================================
 Package                                                      Arch                         Version                                        Repository                                 Size
==========================================================================================================================================================================================
Installing:
 cri-o                                                        x86_64                       1.21.2-4.2.el7                                 libcontainers_cri-o                        25 M
...중략...
Transaction Summary
==========================================================================================================================================================================================
Install  1 Package (+19 Dependent packages)
Total download size: 67 M
Installed size: 210 M
Downloading packages:
(1/20): conmon-2.0.29-1.el7.3.2.x86_64.rpm                                                                                                                         ...중략...
(20/20): containernetworking-plugins-1.0.0-0.2.rc1.el7.6.2.x86_64.rpm                                                                                              |  39 MB  00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                      83 MB/s |  67 MB  00:00:00
Running transaction check
...중략...
Installed:
  cri-o.x86_64 0:1.21.2-4.2.el7
...중략...
Complete!

CRI-O 시작
※ 다음 절에서 docker 데몬을 실행할 것이므로 서비스를 시작하지 않는다.

# 다음절에서 docker 데몬을 시작할 것이므로 수행하지 않는다.
haedong@haedong:/home:]$ sudo systemctl daemon-reload
haedong@haedong:/home:]$ sudo systemctl enable crio --now
Created symlink from /etc/systemd/system/multi-user.target.wants/crio.service to /usr/lib/systemd/system/crio.service.

cgroup 드라이버 설정

haedong@haedong:/home:]$ sudo vi /etc/crio/crio.conf
# [crio.runtime] 섹션에 아래 내용을 추가한다.
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"

Docker

컨테이너 cgroup 관리에 systemd를 사용하기 위한 docker 데몬 구성

haedong@haedong:/home:]$ sudo mkdir /etc/docker
haedong@haedong:/home:]$ cat <<EOF | sudo tee /etc/docker/daemon.json
> {
>   "exec-opts": ["native.cgroupdriver=systemd"],
>   "log-driver": "json-file",
>   "log-opts": {
>     "max-size": "100m"
>   },
>   "storage-driver": "overlay2"
> }
> EOF

docker 재시작 및 서비스 설정

haedong@haedong:/home:]$ sudo systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
haedong@haedong:/home:]$ sudo systemctl daemon-reload
haedong@haedong:/home:]$ sudo systemctl restart docker