Kubephere는 플러그앤 플레이 아키텍쳐를 통해 타사 애플리케이션을 해당 에코시스템에 원활하게 통합할 수 있는 kubernetes를 커널로 하는 클라우드 네이티브 애플리케이션 관리를 위한 분산 운영체제 이다.kubesphere 홈페이지 발췌
설치는 Kubesphere 홈페이지를 참고했다.
Kubesphere는 다음 Kubernetes 런타임을 지원한다.
| Supported Container Runtime | Version |
|---|---|
| Docker | 19.3.8 + |
| containerd (experimental, not fully tested) | Latest |
| CRI-O (experimental, not fully tested) | Latest |
| iSula (experimental, not fully tested) | Latest |
본 예제는 총 4대의 가상머신 (CentOS7, 4core CPU, 8GB RAM, 128GB storage)으로 테스트 하였다.
작업을 시작하기 전에 이 포스트를 참고하여 kubesphere 구성을 할 서버들 간의 SSH key를 맞춰주도록 한다.
의존성 패키지 설치
kubernetes v1.18 이상은 socat과 conntrack 패키지가 필요하고 ebtables와 ipset 패키지는 권장, v1.18 미만은 모두 설치를 권장한다.
haedong@haedong:~/kubesphere:]$ sudo yum install socat conntrack ebtables ipset Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile Package socat-1.7.3.2-2.el7.x86_64 already installed and latest version ...중략... ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================================================== Installing: conntrack-tools x86_64 1.4.4-7.el7 ...중략... local_centos 23 k Transaction Summary ========================================================================================================================================================================================== Install 1 Package (+3 Dependent packages) Total download size: 245 k (4/4): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm ...중략... ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 1.6 MB/s | 245 kB 00:00:00 Running transaction check ...중략... Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 4/4 Installed: conntrack-tools.x86_64 0:1.4.4-7.el7 ...중략... Complete!
Kubesphere는 다음 port들을 사용한다. 본인 소유의 시스템이 아니라면 관리자에 다음 표의 포트 개방을 요청 해야 한다.
| Service | Protocol | Action | Start Port | End Port | Notes |
|---|---|---|---|---|---|
| ssh | TCP | allow | 22 | ||
| etcd | TCP | allow | 2379 | 2380 | |
| apiserver | TCP | allow | 6443 | ||
| calico | TCP | allow | 9099 | 9100 | |
| bgp | TCP | allow | 179 | ||
| nodeport | TCP | allow | 30000 | 32767 | |
| master | TCP | allow | 10250 | 10258 | |
| dns | TCP | allow | 53 | ||
| dns | UDP | allow | 53 | ||
| local-registry | TCP | allow | 5000 | For the offline environment | |
| local-apt | TCP | allow | 5080 | For the offline environment | |
| rpcbind | TCP | allow | 111 | Required if NFS is used | |
| ipip | IPENCAP / IPIP | allow | Calico needs to allow the ipip protocol | ||
| metrics-server | TCP | allow | 8443 |
Kubekey 다운로드
haedong@kubesphere-01:~:]$ curl -ksfL https://get-kk.kubesphere.io | VERSION=v1.1.1 sh - Downloading kubekey v1.1.1 from https://github.com/kubesphere/kubekey/releases/download/v1.1.1/kubekey-v1.1.1-linux-amd64.tar.gz ... Failed to download Kubekey v1.1.1 ! Please verify the version you are trying to download.
잘 안된다…… github 릴리즈 페이지에서 다운로드 하자.
haedong@kubesphere-01:~:]$ wget --no-check-certificate https://github.com/kubesphere/kubekey/releases/download/v1.1.1/kubekey-v1.1.1-linux-amd64.tar.gz --2021-08-25 14:53:24-- https://github.com/kubesphere/kubekey/releases/download/v1.1.1/kubekey-v1.1.1-linux-amd64.tar.gz Connecting to 168.219.61.252:8080... connected. ...중략... Length: 13341858 (13M) [application/octet-stream] Saving to: ‘kubekey-v1.1.1-linux-amd64.tar.gz’ 100%[================================================================================================================================================>] 13,341,858 3.14MB/s in 4.4s 2021-08-25 14:53:30 (2.88 MB/s) - ‘kubekey-v1.1.1-linux-amd64.tar.gz’ saved [13341858/13341858]
압축을 해제하고 권한을 변경한다.
haedong@kubesphere-01:~:]$ tar -xvzf kubekey-v1.1.1-linux-amd64.tar.gz README.md README_zh-CN.md kk haedong@kubesphere-01:~:]$ chmod +x kk haedong@kubesphere-01:~:]$ ll 합계 26428 -rw-r--r-- 1 haedong haedong 22906 7월 12 16:00 README.md -rw-r--r-- 1 haedong haedong 22845 7월 12 16:00 README_zh-CN.md -rwxr-xr-x 1 haedong haedong 13668116 7월 12 16:02 kk -rw-rw-r-- 1 haedong haedong 13341858 7월 12 16:03 kubekey-v1.1.1-linux-amd64.tar.gz
설치
다운로드한 kk가 지원하는 kubernetes 버전을 확인한다. 1예제는 Kubekey를 이용해 Kubesphere와 Kubernetes를 함께 설치하는 방법을 나열한다. 기존에 설치되어있는 Kubernetes를 이용하는 경우는 여기를 눌러 설치 방법을 확인하면 된다.
haedong@kubesphere-01:~:]$ ./kk version --show-supported-k8s v1.15.12 v1.16.8 v1.16.10 v1.16.12 v1.16.13 v1.17.0 v1.17.4 v1.17.5 v1.17.6 v1.17.7 v1.17.8 v1.17.9 v1.18.3 v1.18.5 v1.18.6 v1.18.8 v1.19.0 v1.19.8 v1.19.9 v1.20.4 v1.20.6
다음 템플릿대로 실행하면 된다
./kk create cluster [–with-kubernetes version] [–with-kubesphere version]
만약 부족한 패키지가 있다면 다음과 같이 표시될 것이다.
haedong@kubesphere-01:~:]$ sudo ./kk create cluster --with-kubernetes v1.20.4 --with-kubesphere v3.1.1 [sudo] haedong의 암호: +---------------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+ | name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time | +---------------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+ | kubesphere-01 | y | y | y | y | | y | | | y | | | KST 15:17:51 | +---------------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+ kubesphere-01: conntrack is required.
필요한 패키지가 모두 설치 되어있다면 다음과 같이 진행 된다.
haedong@kubesphere-01:~:]# sudo./kk create cluster --with-kubernetes v1.20.4 --with-kubesphere v3.1.1
+--------+------+------+---------+----------+-------+-------+-----------+---------+------------+-------------+------------------+--------- -----+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+--------+------+------+---------+----------+-------+-------+-----------+---------+------------+-------------+------------------+--------- -----+
| kube01 | y | y | y | y | y | y | y | 20.10.8 | y | | | KST 09:5 2:43 |
+--------+------+------+---------+----------+-------+-------+-----------+---------+------------+-------------+------------------+--------- -----+
This is a simple check of your environment.
Before installation, you should ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations
Continue this installation? [yes/no]: yes
INFO[09:52:45 KST] Downloading Installation Files
INFO[09:52:45 KST] Downloading kubeadm ...
INFO[09:52:49 KST] Downloading kubelet ...
INFO[09:53:00 KST] Downloading kubectl ...
INFO[09:53:07 KST] Downloading helm ...
INFO[09:53:11 KST] Downloading kubecni ...
INFO[09:53:18 KST] Configuring operating system ...
...중략...
clusterconfiguration.installer.kubesphere.io/ks-installer created
#####################################################
### Welcome to KubeSphere! ###
#####################################################
Console: http://0.0.0.0:30880
Account: admin
Password: P@88w0rd
NOTES：
1. After you log into the console, please check the
monitoring status of service components in
"Cluster Management". If any service is not
ready, please wait patiently until all components
are up and running.
2. Please change the default password after login.
#####################################################
https://kubesphere.io 2021-09-07 10:00:00
#####################################################
INFO[10:00:11 KST] Installation is complete.
Please check the result using the command:
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
You have new mail in /var/spool/mail/root
kubekey를 이용한 설치 시 31개의 Docker image를 가져오게 되는데 운영 환경 제약 등으로 실패할 경우 아래 명령을 이용해 설치 전 이미지를 다운 받으면 된다.
docker pull docker.io/kubesphere/ks-installer:v3.1.1 &&\ docker pull docker.io/kubesphere/ks-controller-manager:v3.1.1 &&\ docker pull docker.io/kubesphere/ks-apiserver:v3.1.1 &&\ docker pull docker.io/kubesphere/ks-console:v3.1.1 &&\ docker pull docker.io/openebs/provisioner-localpv:2.10.1 &&\ docker pull docker.io/kubesphere/notification-manager:v1.0.0 &&\ docker pull docker.io/kubesphere/notification-manager-operator:v1.0.0 &&\ docker pull docker.io/openebs/linux-utils:2.10.0 &&\ docker pull docker.io/kubesphere/kubectl:v1.20.0 &&\ docker pull docker.io/prom/prometheus:v2.26.0 &&\ docker pull docker.io/kubesphere/kube-proxy:v1.20.4 &&\ docker pull docker.io/kubesphere/kube-apiserver:v1.20.4 &&\ docker pull docker.io/kubesphere/kube-controller-manager:v1.20.4 &&\ docker pull docker.io/kubesphere/kube-scheduler:v1.20.4 &&\ docker pull docker.io/csiplugin/snapshot-controller:v3.0.3 &&\ docker pull docker.io/kubesphere/kube-rbac-proxy:v0.8.0 &&\ docker pull docker.io/calico/node:v3.16.3 &&\ docker pull docker.io/calico/pod2daemon-flexvol:v3.16.3 &&\ docker pull docker.io/calico/cni:v3.16.3 &&\ docker pull docker.io/calico/kube-controllers:v3.16.3 &&\ docker pull docker.io/kubesphere/prometheus-config-reloader:v0.42.1 &&\ docker pull docker.io/kubesphere/prometheus-operator:v0.42.1 &&\ docker pull docker.io/kubesphere/etcd:v3.4.13 &&\ docker pull docker.io/prom/alertmanager:v0.21.0 &&\ docker pull docker.io/kubesphere/kube-state-metrics:v1.9.7 &&\ docker pull docker.io/kubesphere/k8s-dns-node-cache:1.15.12 &&\ docker pull docker.io/coredns/coredns:1.6.9 &&\ docker pull docker.io/kubesphere/pause:3.2 &&\ docker pull docker.io/jimmidyson/configmap-reload:v0.3.0 &&\ docker pull docker.io/prom/node-exporter:v0.18.1 &&\ docker pull docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.4
일반적인 환경에서는
haedong@kubesphere-01:~:]$ curl -ksfL https://get-kk.kubesphere.io | VERSION=v1.1.1 sh - haedong@kubesphere-01:~:]# sudo./kk create cluster --with-kubernetes v1.20.4 --with-kubesphere v3.1.1
위 두 줄 명령으로 설치가 진행되지만 특수한 환경에서는2 Proxy를 통한 외부 연결 등
Docker runtime 설치 -> Docker service 스크립트에 proxy 정보 추가 -> image pull -> 설치 의 순서로 진행해야 할 수 있다.