[태그:] bigdata

작성일자

Packstack #1 사전 작업 및 패키지 설치

※ 이 포스트는 실제 설치및 설정을 수행하면서 작성하고 있습니다. 계속 업데이트 됩니다.

Openstack #1 개요

CentOS 7 에 install 하는것으로 가정한다.
CentOS 8 Linux 설치를 참고하여 최소설치 모드로 리눅스를 설치한다.
네트워크 설정을 변경한다. (NIC는 2개 이상이 필요하다. 외부 연결용, 노드간 통신 용)
root 권한으로 해야 하는 작업이 많다. root 패스워드를 설정하거나, SuDo : SuperUser Do를 참고하여 sudo 사용 설정을 한다.
클러스터간 설치 및 통신 용의성 확보를 위해 SSH설정을 참고하여 키를 등록한다.
openstack train 기준으로 진행한다.



네트워크설정, ssh 키 설정 등이 완료 됐으면 서비스 설정을 변경한다.
openstack 클러스터의 구성 노드 모두에서 작업 해줘야 한다.

 # 노드간 통신의 용의성을 위해 방화벽 서비스를 종료한다.
[HOSTNAME:/haedong]$ sudo systemctl disable firewalld
[HOSTNAME:/haedong]$ sudo service firewalld stop

 # Network 관리 서비스인데 개인적으로 아주 고약한 녀석이다. 
 # 아무리 설정을 바꿔도 제 멋대로 설정을 덮어 써버리는 경우가 허다하므로 종료한다.
 # 어차피 네트워크는 별도로 관리해야 한다. 
[HOSTNAME:/haedong]$ sudo systemctl disable NetworkManager
[HOSTNAME:/haedong]$ sudo service NetworkManager stop

 # 만약을 위해 서비스를 재시작하고, 항상 서비스가 구동도록 설정한다.
[HOSTNAME:/haedong]$ sudo systemctl enable network
[HOSTNAME:/haedong]$ sudo service network restart

packstack1RDO project의 puppet module을 이용한 CentOS 및 Redhat linux 용 openstack 자동 배포 유틸리티이다. 관련 리포지터리 및 패키지 설치

 # 만약 epel-release 리포지터리가 추가되어있다면 삭제한다.(혹은 disable로 변경해도 된다.)
[HOSTNAME:/home/haedong]$ sudo rm /etc/yum.repos.d/epel
rm: remove 일반 파일 `epel-testing.repo'? y
rm: remove 일반 파일 `epel.repo'? y
 # packstack repo 설치
[HOSTNAME:/home/haedong:]$ sudo yum install -y https://www.rdoproject.org/repos/rdo-release.rpm
[sudo] haedong의 암호:
Loaded plugins: fastestmirror, langpacks
rdo-release.rpm                                                                                                                                               | 6.7 kB  00:00:00
Examining /var/tmp/yum-root-URQvAZ/rdo-release.rpm: rdo-release-train-1.noarch
Marking /var/tmp/yum-root-URQvAZ/rdo-release.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package rdo-release.noarch 0:train-1 will be installed
--> Finished Dependency Resolution
base/7/x86_64                                                                                                                                                 | 3.6 kB  00:00:00
Dependencies Resolved
=====================================================================================================================================================================================
 Package                                      Arch                                    Version                                    Repository                                     Size
=====================================================================================================================================================================================
Installing:
 rdo-release                                  noarch                                  train-1                                    /rdo-release                                  3.1 k
Transaction Summary
=====================================================================================================================================================================================
Install  1 Package

Total size: 3.1 k
Installed size: 3.1 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : rdo-release-train-1.noarch                                                                                                                                        1/1
  Verifying  : rdo-release-train-1.noarch                                                                                                                                        1/1
Installed:
  rdo-release.noarch 0:train-1
Complete!
 # openstack train 패키지 설치
[HOSTNAME:/home/haedong:]$ sudo yum install -y centos-release-openstack-train
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
 * base: mirror.kakao.com
 * extras: mirror.kakao.com
 * openstack-train: mirror.kakao.com
 * rdo-qemu-ev: mirror.kakao.com
 * updates: mirror.kakao.com
openstack-train                                                                                                                                               | 3.0 kB  00:00:00
rdo-qemu-ev                                                                                                                                                   | 3.0 kB  00:00:00
(1/2): rdo-qemu-ev/x86_64/primary_db                                                                                                                          |  57 kB  00:00:00
(2/2): openstack-train/x86_64/primary_db                                                                                                                      | 1.1 MB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package centos-release-openstack-train.noarch 0:1-1.el7.centos will be installed
--> Processing Dependency: centos-release-qemu-ev for package: centos-release-openstack-train-1-1.el7.centos.noarch
--> Processing Dependency: centos-release-ceph-nautilus for package: centos-release-openstack-train-1-1.el7.centos.noarch
--> Running transaction check
---> Package centos-release-ceph-nautilus.noarch 0:1.2-2.el7.centos will be installed
--> Processing Dependency: centos-release >= 7-5.1804.el7.centos.2 for package: centos-release-ceph-nautilus-1.2-2.el7.centos.noarch
--> Processing Dependency: centos-release-storage-common for package: centos-release-ceph-nautilus-1.2-2.el7.centos.noarch
--> Processing Dependency: centos-release-nfs-ganesha28 for package: centos-release-ceph-nautilus-1.2-2.el7.centos.noarch
---> Package centos-release-qemu-ev.noarch 0:1.0-4.el7.centos will be installed
--> Processing Dependency: centos-release-virt-common for package: centos-release-qemu-ev-1.0-4.el7.centos.noarch
--> Running transaction check
---> Package centos-release.x86_64 0:7-5.1804.el7.centos will be updated
---> Package centos-release.x86_64 0:7-9.2009.1.el7.centos will be an update
---> Package centos-release-nfs-ganesha28.noarch 0:1.0-3.el7.centos will be installed
---> Package centos-release-storage-common.noarch 0:2-2.el7.centos will be installed
---> Package centos-release-virt-common.noarch 0:1-1.el7.centos will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================================================================================================
 Package                                                  Arch                             Version                                           Repository                         Size
=====================================================================================================================================================================================
Installing:
 centos-release-openstack-train                           noarch                           1-1.el7.centos                                    extras                            5.3 k
Installing for dependencies:
 centos-release-ceph-nautilus                             noarch                           1.2-2.el7.centos                                  extras                            5.1 k
 centos-release-nfs-ganesha28                             noarch                           1.0-3.el7.centos                                  extras                            4.3 k
 centos-release-qemu-ev                                   noarch                           1.0-4.el7.centos                                  extras                             11 k
 centos-release-storage-common                            noarch                           2-2.el7.centos                                    extras                            5.1 k
 centos-release-virt-common                               noarch                           1-1.el7.centos                                    extras                            4.5 k
Updating for dependencies:
 centos-release                                           x86_64                           7-9.2009.1.el7.centos                             updates                            27 k
Transaction Summary
=====================================================================================================================================================================================
Install  1 Package  (+5 Dependent packages)
Upgrade             ( 1 Dependent package)
Total download size: 62 k
Downloading packages:
No Presto metadata available for updates
(1/7): centos-release-7-9.2009.1.el7.centos.x86_64.rpm                                                                                                        |  27 kB  00:00:00
(2/7): centos-release-ceph-nautilus-1.2-2.el7.centos.noarch.rpm                                                                                               | 5.1 kB  00:00:00
(3/7): centos-release-nfs-ganesha28-1.0-3.el7.centos.noarch.rpm                                                                                               | 4.3 kB  00:00:00
(4/7): centos-release-qemu-ev-1.0-4.el7.centos.noarch.rpm                                                                                                     |  11 kB  00:00:00
(5/7): centos-release-storage-common-2-2.el7.centos.noarch.rpm                                                                                                | 5.1 kB  00:00:00
(6/7): centos-release-openstack-train-1-1.el7.centos.noarch.rpm                                                                                               | 5.3 kB  00:00:00
(7/7): centos-release-virt-common-1-1.el7.centos.noarch.rpm                                                                                                   | 4.5 kB  00:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                650 kB/s |  62 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : centos-release-7-9.2009.1.el7.centos.x86_64                                                                                                                       1/8
warning: /etc/yum/vars/contentdir created as /etc/yum/vars/contentdir.rpmnew
  Installing : centos-release-storage-common-2-2.el7.centos.noarch                                                                                                               2/8
  Installing : centos-release-nfs-ganesha28-1.0-3.el7.centos.noarch                                                                                                              3/8
  Installing : centos-release-ceph-nautilus-1.2-2.el7.centos.noarch                                                                                                              4/8
  Installing : centos-release-virt-common-1-1.el7.centos.noarch                                                                                                                  5/8
  Installing : centos-release-qemu-ev-1.0-4.el7.centos.noarch                                                                                                                    6/8
  Installing : centos-release-openstack-train-1-1.el7.centos.noarch                                                                                                              7/8
  Cleanup    : centos-release-7-5.1804.el7.centos.x86_64                                                                                                                         8/8
  Verifying  : centos-release-openstack-train-1-1.el7.centos.noarch                                                                                                              1/8
  Verifying  : centos-release-nfs-ganesha28-1.0-3.el7.centos.noarch                                                                                                              2/8
  Verifying  : centos-release-7-9.2009.1.el7.centos.x86_64                                                                                                                       3/8
  Verifying  : centos-release-ceph-nautilus-1.2-2.el7.centos.noarch                                                                                                              4/8
  Verifying  : centos-release-virt-common-1-1.el7.centos.noarch                                                                                                                  5/8
  Verifying  : centos-release-storage-common-2-2.el7.centos.noarch                                                                                                               6/8
  Verifying  : centos-release-qemu-ev-1.0-4.el7.centos.noarch                                                                                                                    7/8
  Verifying  : centos-release-7-5.1804.el7.centos.x86_64                                                                                                                         8/8
Installed:
  centos-release-openstack-train.noarch 0:1-1.el7.centos
Dependency Installed:
  centos-release-ceph-nautilus.noarch 0:1.2-2.el7.centos        centos-release-nfs-ganesha28.noarch 0:1.0-3.el7.centos        centos-release-qemu-ev.noarch 0:1.0-4.el7.centos
  centos-release-storage-common.noarch 0:2-2.el7.centos         centos-release-virt-common.noarch 0:1-1.el7.centos
Dependency Updated:
  centos-release.x86_64 0:7-9.2009.1.el7.centos

Complete!

 # 설치 된 패키지 업데이트.
[HOSTNAME:/home/haedong:]$ sudo yum -y update
Loaded plugins: fastestmirror, langpacks
Repository rdo-trunk-train-tested is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirror.kakao.com
 * centos-ceph-nautilus: mirror.kakao.com
 * centos-nfs-ganesha28: mirror.kakao.com
 * centos-openstack-train: mirror.kakao.com
 * centos-qemu-ev: mirror.kakao.com
 * extras: mirror.kakao.com
 * openstack-train: mirror.kakao.com
 * rdo-qemu-ev: mirror.kakao.com
 * updates: mirror.kakao.com
Resolving Dependencies
--> Running transaction check
---> Package GeoIP.x86_64 0:1.5.0-11.el7 will be updated
---> Package GeoIP.x86_64 0:1.5.0-14.el7 will be an update
--> Processing Dependency: geoipupdate for package: GeoIP-1.5.0-14.el7.x86_64
---> Package LibRaw.x86_64 0:0.14.8-5.el7.20120830git98d925 will be updated
---> Package LibRaw.x86_64 0:0.19.4-1.el7 will be an update
---> Package ModemManager.x86_64 0:1.6.10-1.el7 will be updated
---> Package ModemManager.x86_64 0:1.6.10-4.el7 will be an update
---> Package ModemManager-glib.x86_64 0:1.6.10-1.el7 will be updated
---> Package ModemManager-glib.x86_64 0:1.6.10-4.el7 will be an update
---> Package NetworkManager.x86_64 1:1.10.2-13.el7 will be updated
---> Package NetworkManager.x86_64 1:1.18.8-2.el7_9 will be an update
---> Package NetworkManager-adsl.x86_64 1:1.10.2-13.el7 will be updated
---> Package NetworkManager-adsl.x86_64 1:1.18.8-2.el7_9 will be an update
---> Package NetworkManager-glib.x86_64 1:1.10.2-13.el7 will be updated
---> Package NetworkManager-glib.x86_64 1:1.18.8-2.el7_9 will be an update
---> Package NetworkManager-libnm.x86_64 1:1.10.2-13.el7 will be updated
---> Package NetworkManager-libnm.x86_64 1:1.18.8-2.el7_9 will be an update
---> Package NetworkManager-ppp.x86_64 1:1.10.2-13.el7 will be updated
---> Package NetworkManager-ppp.x86_64 1:1.18.8-2.el7_9 will be an update
--> Running transaction check
...중략...
---> Package mokutil.x86_64 0:15-8.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================================================================================================
 Package                                                 Arch                     Version                                             Repository                                Size
=====================================================================================================================================================================================
Installing:
 freerdp-libs                                            x86_64                   2.1.1-2.el7                                         base                                     851 k
     replacing  freerdp-plugins.x86_64 1.0.2-15.el7
 gnome-dictionary                                        x86_64                   3.26.1-2.el7                                        base                                     642 k
     replacing  gnome-dictionary-libs.x86_64 3.20.0-1.el7
 gnome-shell                                             x86_64                   3.28.3-32.el7                                       updates                                  2.1 M
     replacing  caribou.x86_64 0.4.21-1.el7
     replacing  caribou-gtk2-module.x86_64 0.4.21-1.el7
     replacing  caribou-gtk3-module.x86_64 0.4.21-1.el7
     replacing  python2-caribou.noarch 0.4.21-1.el7
 xorg-x11-xauth                                          x86_64                   1:1.0.9-1.el7                                       base                                      30 k
 xorg-x11-xinit                                          x86_64                   1.3.4-2.el7                                         base                                      58 k
 xorg-x11-xkb-utils                                      x86_64                   7.7-14.el7                                          base                                     103 k
...중략...
Transaction Summary
=====================================================================================================================================================================================
Install   36 Packages (+95 Dependent packages)
Upgrade  940 Packages

Total download size: 1.1 G
Downloading packages:
No Presto metadata available for centos-openstack-train
No Presto metadata available for openstack-train
No Presto metadata available for base
No Presto metadata available for updates
No Presto metadata available for centos-ceph-nautilus
(1/1071): GeoIP-1.5.0-14.el7.x86_64.rpm                                                                                                                       | 1.5 MB  00:00:00
(2/1071): LibRaw-0.19.4-1.el7.x86_64.rpm                                                                                                                      | 308 kB  00:00:00
(3/1071): ModemManager-1.6.10-4.el7.x86_64.rpm                                                                                                                | 738 kB  00:00:00
(4/1071): ModemManager-glib-1.6.10-4.el7.x86_64.rpm                                                                                                           | 232 kB  00:00:00
(5/1071): NetworkManager-adsl-1.18.8-2.el7_9.x86_64.rpm                                                                                                       | 163 kB  00:00:00
(6/1071): NetworkManager-glib-1.18.8-2.el7_9.x86_64.rpm                                                                                                       | 1.5 MB  00:00:00
(7/1071): NetworkManager-libnm-1.18.8-2.el7_9.x86_64.rpm                                                                                                      | 1.7 MB  00:00:00
...중략...
(1068/1071): yum-utils-1.1.31-54.el7_8.noarch.rpm                                                                                                             | 122 kB  00:00:00
(1069/1071): zlib-1.2.7-18.el7.x86_64.rpm                                                                                                                     |  90 kB  00:00:00
(1070/1071): zlib-devel-1.2.7-18.el7.x86_64.rpm                                                                                                               |  50 kB  00:00:00
(1071/1071): zenity-3.28.1-1.el7.x86_64.rpm                                                                                                                   | 4.0 MB  00:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                 46 MB/s | 1.1 GB  00:00:25
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
Importing GPG key 0xE451E5B5:
 Userid     : "CentOS Storage SIG (http://wiki.centos.org/SpecialInterestGroup/Storage) <security@centos.org>"
 Fingerprint: 7412 9c0b 173b 071a 3775 951a d4a2 e50b e451 e5b5
 Package    : centos-release-storage-common-2-2.el7.centos.noarch (@extras)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
...중략...
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : libgcc-4.8.5-44.el7.x86_64                                                                                                                                     1/2055
  Installing : urw-base35-fonts-common-20170801-10.el7.noarch                                                                                                                 2/2055
  Installing : xorg-x11-proto-devel-2018.4-1.el7.noarch                                                                                                                       3/2055
...중략...
  Verifying  : 10:qemu-kvm-1.5.3-156.el7.x86_64                                                                                                                            2053/2055
  Verifying  : python-dmidecode-3.12.2-2.el7.x86_64                                                                                                                        2054/2055
  Verifying  : freerdp-1.0.2-15.el7.x86_64                                                                                                                                 2055/2055
Installed:
  freerdp-libs.x86_64 0:2.1.1-2.el7                             gnome-dictionary.x86_64 0:3.26.1-2.el7                  gnome-shell.x86_64 0:3.28.3-32.el7
  grub2.x86_64 1:2.02-0.86.el7.centos                           grub2-tools.x86_64 1:2.02-0.86.el7.centos               grub2-tools-extra.x86_64 1:2.02-0.86.el7.centos
...중략...
  qemu-img.x86_64 10:1.5.3-156.el7                         qemu-kvm.x86_64 10:1.5.3-156.el7                               qemu-kvm-common.x86_64 10:1.5.3-156.el7
  sip-macros.x86_64 0:4.14.6-4.el7                         urw-fonts.noarch 0:2.4-16.el7                                  webkitgtk4-plugin-process-gtk2.x86_64 0:2.16.6-6.el7

Complete!
[HOSTNAME:/home/haedong:]$ sudo yum install -y openstack-packstack
Loaded plugins: fastestmirror, langpacks
Repository rdo-trunk-train-tested is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirror.kakao.com
Resolving Dependencies
--> Running transaction check
---> Package openstack-packstack.noarch 1:15.0.1-1.el7 will be installed
--> Processing Dependency: openstack-packstack-puppet = 1:15.0.1-1.el7 for package: 1:openstack-packstack-15.0.1-1.el7.noarch
--> Processing Dependency: python-docutils for package: 1:openstack-packstack-15.0.1-1.el7.noarch
--> Processing Dependency: python2-pbr for package: 1:openstack-packstack-15.0.1-1.el7.noarch
--> Running transaction check
---> Package openstack-packstack-puppet.noarch 1:15.0.1-1.el7 will be installed
--> Processing Dependency: puppet-aodh for package: 1:openstack-packstack-puppet-15.0.1-1.el7.noarch
--> Processing Dependency: puppet-apache for package: 1:openstack-packstack-puppet-15.0.1-1.el7.noarch
...중략...
  Verifying  : 1:openstack-packstack-15.0.1-1.el7.noarch                                                                                                                       76/78
  Verifying  : python2-pbr-5.1.2-2.el7.noarch                                                                                                                                  77/78
  Verifying  : puppet-cinder-15.4.0-1.el7.noarch                                                                                                                               78/78
Installed:
  openstack-packstack.noarch 1:15.0.1-1.el7
Dependency Installed:
  boost159-atomic.x86_64 0:1.59.0-2.el7.1                 boost159-chrono.x86_64 0:1.59.0-2.el7.1                      boost159-date-time.x86_64 0:1.59.0-2.el7.1
...중략...
  ruby-facter.x86_64 1:3.9.3-7.el7                        ruby-shadow.x86_64 0:1.4.1-23.el7                            rubygem-pathspec.noarch 0:0.2.1-3.el7
  rubygem-rgen.noarch 0:0.6.6-2.el7                       yaml-cpp.x86_64 0:0.5.1-6.el7
Complete!










작성일자

Hadoop # 2. 싱글 Namenode 설치,설정

Hadoop #1. 개요

사전 작업

하둡을 구동할 계정은 hadoop, 그룹도 hadoop 으로 한다. hadoop 계정의 홈 디렉토리는 /home/hadoop 이다. 키-쌍 생성 관련 내용은 별도의 포스트를 통해 설명하겠다

 # 사용자 생성
[root@hadoop01 ~]# useradd -g hadoop hadoop
[root@hadoop01 ~]# su - hadoop
[root@hadoop01 ~]# passwd hadoop
hadoop 사용자의 비밀 번호 변경 중
새  암호:
새  암호 재입력:
passwd: 모든 인증 토큰이 성공적으로 업데이트 되었습니다.
 #여기까지 모든 서버에서 작업한다.
 # 키 쌍 생성
[hadoop@hadoop01 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:kQyuv0LCUMDOBwicLr4VyFSskm0wYIGHLIv1pTbALF0 hadoop@centos7
The key''s randomart image is:
+---[RSA 2048]----+
|%O=oE .          |
|XBO. ..o .       |
|O@+o o. +        |
|BBoo=.   .       |
|++..o.  S        |
| .o...           |
|  oo  .          |
| .  .  .         |
|     ..          |
+----[SHA256]-----+
[hadoop@hadoop01 ~]$ chmod 700 ~/.ssh
[hadoop@hadoop01 ~]$ cat ~/.ssh/id_rsa.pub >> authorized_keys
[hadoop@hadoop01 ~]$ chmod 600 ~/.ssh/*
[hadoop@hadoop01 ~]$ ls -l ~/.ssh/
합계 12
-rw------- 1 hadoop hadoop  396  9월 14 15:32 authorized_keys
-rw------- 1 hadoop hadoop 1675  9월 14 15:24 id_rsa
-rw------- 1 hadoop hadoop  396  9월 14 15:24 id_rsa.pub
 # 하둡 클러스터로 사용할 모든 서버에 키를 복사하고 권한을 변경한다.
[hadoop@hadoop01 ~]$ scp ~/.ssh/authorized_keys hadoop@hadoop02:/home/hadoop/.ssh/authorized_keys
The authenticity of host 'hadoop02 (192.168.0.2)' can't be established.
ECDSA key fingerprint is SHA256:nhIT6XvSamWF1mgXDkAuM64eZj5XCJww5T2NEojH2iU.
ECDSA key fingerprint is MD5:7b:a5:40:02:c3:cd:0f:e7:36:77:dd:3c:cc:3b:ba:d2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
hadoop@hadoop02's password:
authorized_keys     
[hadoop@hadoop01 ~]$  ssh hadoop02 chmod 700 /home/hadoop/.ssh
[hadoop@hadoop01 ~]$  ssh hadoop02 chmod 600 /home/hadoop/.ssh/authorized_keys
hadoop@hadoop02's password:

설치

Apache Hadoop 홈페이지에서 바이너리를 다운로드 하고 압축을 해제한다.
※ hadoop 계정으로 진행

[hadoop@hadoop01 ~]$ wget http://mirror.apache-kr.org/hadoop/common/hadoop-3.1.4/hadoop-3.1.4.tar.gz
--2020-09-14 15:28:30--  http://mirror.apache-kr.org/hadoop/common/hadoop-3.1.4/hadoop-3.1.4.tar.gz
Resolving mirror.apache-kr.org (mirror.apache-kr.org)... 125.209.216.167
Connecting to mirror.apache-kr.org (mirror.apache-kr.org)|125.209.216.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 348326890 (332M) [application/octet-stream]
Saving to: ‘hadoop-3.1.4.tar.gz’
100%[=====================================================================================================================================>] 348,326,890 37.9MB/s   in 11s
2020-09-14 15:28:41 (30.1 MB/s) - ‘hadoop-3.1.4.tar.gz’ saved [348326890/348326890]
[hadoop@hadoop01 ~]$ tar -xvzf  hadoop-3.1.4.tar.gz
hadoop-3.1.4/
hadoop-3.1.4/lib/
중략
hadoop-3.1.4/libexec/mapred-config.sh
hadoop-3.1.4/libexec/hadoop-config.cmd
hadoop-3.1.4/libexec/hdfs-config.cmd
[hadoop@hadoop01 ~]$ ln -s  ~/hadoop-3.1.4 ~/hadoop
[hadoop@hadoop01 ~]$ ls -l
합계 340164
lrwxrwxrwx 1 hadoop hadoop        12  9월 14 15:31 hadoop -> hadoop-3.1.4
drwxr-xr-x 9 hadoop hadoop       169  9월 14 15:31 hadoop-3.1.4
-rw-rw-r-- 1 hadoop hadoop 348326890  8월 24 21:40 hadoop-3.1.4.tar.gz
drwxrwxr-x 2 hadoop hadoop         6  9월 14 15:22 perl5
 #나머지 노드에서 모두 동일한 작업을 수행한다. (다운로드, 압축 풀기, 심볼릭 링크 생성)
[hadoop@hadoop01 ~]$  vi  ~/.bas_profile
 # 여기부터 파일의 맨 뒤에 붙여넣기 한다.
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[hadoop@hadoop01 ~]$  source ~/.bas_profile
 나머지 노드들에서 동일하게 수행 해 준다.

설정

 # Hadoop 환경변수 설정 (하둡이 구동될 때 읽어들이는 환경변수)
[hadoop@hadoop01 ~]$ vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
 # 기존 내용을 모두 지우고 여기서부터 붙여넣기 한다.
 export HADOOP_HOME=/home/hadoop/hadoop
 export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

 # 시스템 사양에 맞추어 적절히 증감한다.
 export HADOOP_HEAPSIZE_MAX= 8g
 export HADOOP_HEAPSIZE_MIN=8g

export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
 # core-site.xml
[hadoop@hadoop01 ~]$ vi $HADOOP_HOME/etc/hadoop/core-site.xml
<!-- 모두 지우고 여기서부터 붙여넣기 한다. -->
<configuration>
  <property>
      <name>fs.defaultFS</name>
      <value>hdfs://hadoop01:8020</value>
<!-- 기본 HDFS 주소. 예를 들어 "hdfs dfs -ls / " 를 수행하면  / 앞에 value의 주소가 들어간다고 이해하면 된다 -->
  </property>
  <property>
      <name>hadoop.proxyuser.hive.groups</name>
      <value>*</value>
  </property>
  <property>
      <name>hadoop.proxyuser.hive.hosts</name>
      <value>*</value>
  </property>
</configuration>
 # hdfs-site.xml
[hadoop@hadoop01 ~]$ vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<!-- 모두 지우고 여기서부터 붙여넣기 한다. -->
<configuration>
   <property>
        <name>dfs.replication</name>
        <value>3</value>
<!-- 
데이터 복제 횟수 
-->
   </property>
   <property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/hadoop/namenode</value>
<!--
로컬(리눅스 시스템의) 디렉토리 경로이며, 반드시 hadoop을 구동하는 계정이 디렉토리에 대한 모든 권한을 가지고 있어야 한다.
-->
   </property>
   <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>/home/hadoop/namesecondary</value>
<!-- 
namenode 디렉토리와 동일하게 로컬 디렉토리이고 모든 권한을 부여해야 한다. 
-->
   </property>
   <property>
        <name>dfs.datanode.data.dir</name>
        <value>/home/hadoop/datanode</value>
<!-- 
실제 hdfs 에 쌓인 데이터가 저장되는 경로. 여유공간이 넉넉한 파티션의 디렉토리를 지정해야 한다.
 datanode로 동작하는 클라이언트, 즉 workers 파일에 등록되어있는 호스트에 모두 동일하게 적용해야 한다.
 당연히 hadoop을 구동하는 계정이 모든 권한을 가진 디렉토리여야 한다.
-->
   </property>
   <property>
         <name>dfs.http.address</name>
         <value>hadoop01:50070</value>
<!--namenode http 주소 및 포트. hadoop web-UI -->
   </property>
   <property>
         <name>dfs.secondary.http.address</name>
         <value>hadoop01:50090</value>
<!-- 
Secondary namenode http 주소 및 포트
주의 할 점은 secondary-namenode는 백업, 복구를 담당하는 것이 아니라 보조의 역할이라는 점이다. 
-->
   </property>

<property>
         <name>dfs.block.size</name>
         <value>16777216</value>
<!-- 
HDFS의 block size, Byte 단위.  다음 값들을 참고하자.
8MB=8,388,608 / 16MB=16,777,216 / 32MB=33,554,432 / 64MB=67,108,864 / 128MB=134,217,728
-->
    </property>
    <property>
      <name>dfs.permissions.enabled</name>
      <value>false</value>
<!-- 
값이 false일 경우 파일 시스템의 퍼미션 관련 기능이 비활성화된다. 
-->
    </property>
    <property>
      <name>fs.trash.interval</name>
      <value>3</value>
<!-- 
휴지통과 같은 기능이다. 파일을 삭제하면 임시 공간으로 이동되고 지정한 시간 이후 삭제된다.
value의 값(=분)에 따라 휴지통이 비워진다.  
-->
    </property>
</configuration>
 # yarn-site.xml
[hadoop@hadoop01 ~]$ vi $HADOOP_HOME/etc/hadoop/yarn-site.xml
<!-- 모두 지우고 여기서부터 붙여넣기 한다. -->
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop02</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop02:8025</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop02:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop02:8050</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop02:8055</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<!--
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler
- FIFO : 먼저 들어온 작업을 먼저 처리한다. 후순위 작업은 이전 작업이 끝날 때까지 대기한다.
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
- FAIR : 작업에 균등하게 자원을 할당한다.
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
- 트리 형태로 큐를 선언하고 각 큐 별로 이용할 수 있는 자원의 용량을 정하여 할당한다. 
-->
        </property>
</configuration>
 # mapreduce-site.xml
[hadoop@hadoop01 ~]$ vi $HADOOP_HOME/etc/hadoop/mapreduce-site.xml
<!-- 모두 지우고 여기서부터 붙여넣기 한다. -->
<configuration>
     <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
     </property>
     <property>
        <name>mapred.local.dir</name>
        <value>/home/hadoop/mapred</value>
     </property>
     <property>
        <name>mapred.system.dir</name>
        <value>/home/hadoop/mapred</value>
     </property>
     <property>
        <name>mapred.map.memory.mb</name>
        <value>2048</value>
<!-- map 작업을 위한 최대 리소스 제한 -->
     </property>
     <property>
        <name>mapred.map.java.opts</name>
        <value>-Xmx1024M</value>
<!-- map 작업을 위한 최대 child jvms  -->
     </property>
     <property>
        <name>mapred.reduce.memory.mb</name>
        <value>3072</value>
<!-- reduce 작업을 위한 최대 리소스 제한  -->
     </property>
     <property>
        <name>mapred.reduce.java.opts</name>
        <value>-Xmx2560M</value>
     <!-- reduce 작업을 위한 최대 child jvms  -->
     </property>
     <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop02:10020</value>
<!-- Job history server 주소 (기본 포트는 10020)  -->
     </property>
     <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop02:19888</value>
<!-- Job history server Web UI주소 (기본 포트는 19888)  -->
     </property>
     <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/home/hadoop/mr-history/tmp</value>
     </property>
     <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/home/hadoop/mmr-histroy/done</value>
     </property>
</configuration>
[hadoop@hadoop01 ~]$ vi $HADOOP_CONF_DIR/workers
 # datanode 구동을 위한 노드를 기록한다.
hadoop01
hadoop02
192.168.0.3
hadoop04.net
 # 수정한 설정 파일을 모든 노드에 복사한다.
[hadoop@hadoop01 ~]$ scp $HADOOP_HOME/etc/hadoop/* hadoop02:/home/hadoop/etc/hadoop/


구동 및 확인. 그리고 종료

 # 구동 스크립트를 이용한 방법

 # 하둡 구동
[hadoop@hadoop01 ~]$  $HADOOP_HOME/sbin/start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as hadoop in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [centos7]
Starting datanodes
Starting secondary namenodes [centos7]
Starting resourcemanager
Starting nodemanagers

 # 프로세스 확인
[hadoop@hadoop01 ~]$  jps -l
92880 org.apache.hadoop.yarn.server.nodemanager.NodeManager
92339 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
93780 sun.tools.jps.Jps
92685 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
92111 org.apache.hadoop.hdfs.server.datanode.DataNode

 # 종료
[hadoop@hadoop01 ~]$  sbin/stop-all.sh
WARNING: Stopping all Apache Hadoop daemons as hadoop in 10 seconds.
WARNING: Use CTRL-C to abort.
Stopping namenodes on [centos7]
Stopping datanodes
Stopping secondary namenodes [centos7]
Stopping nodemanagers
Stopping resourcemanager