Hadoop 叢集 - HackMD

--- title: Hadoop 叢集 tags: Guide, HD Cluster, TW GA: UA-155999456-1 --- {%hackmd @docsharedstyle/default %} # Hadoop 叢集 TWCC 提供建立 Hadoop 叢集的服務，可讓使用者進行巨量資料的運算及分析使用。 ## 建立 Hadoop 叢集 * 進入「Hadoop 叢集」服務，在「Hadoop 叢集管理」頁面，點擊「＋建立」。 ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_43bf769f1747a9afc4a885bd87a5f991.png) * 在基本資訊頁面填寫名稱、附件數、映像檔及硬體設定，接著點擊「下一步：鑰匙對>」。 ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_f5c61d1d29060a741b667bd312b189ce.png) * 選擇欲使用的鑰匙對，或是點選「＋建立鑰匙對」，接著點擊「下一步：網路>」 ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_1e1a07f1770e162e7bfdf4f3cd0b23ab.png) * 選擇欲使用的虛擬網路，接著點擊「下一步：檢閱 + 建立>」。 :::info :book: 參見：[虛擬網路](https://www.twcc.ai/doc?page=virtual_network) ::: ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_03d40d952d1991356efb0019b4c521d2.png) * 檢視整個 Hadoop 叢集的設定和預估使用額度，按下「建立」即完成，幾分鐘後等狀態變成 Ready 即可開始連線使用。 ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_66d9f6ca5e2ebf111d7c6c4f233f7627.png) ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_9971e4a7180d36971f1a46250bcea8f9.png) ## 檢視大數據叢集內容 * 在「Hadoop 叢集管理」頁面，點擊欲檢視的項目列表，即可檢視 Hadoop 叢集的詳細內容。點擊上方按鈕，可對此大數據叢集進行啟動、停止、刪除及重新整理。 ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_4248812986ec74fc5bfbaefba7219b34.png) ## 登入 Hadoop 主機 * 登入 Hadoop 主機與[連結虛擬運算個體](https://man.twcc.ai/uaKL6SkBT9GePrMILRpomg#%E9%80%A3%E7%B5%90%E8%99%9B%E6%93%AC%E6%A9%9F%E5%99%A8)的方式相同，可在命令提示字元視窗下使用 SSH 指令登入。 * 開啟命令提示字元視窗，依照以下範例輸入您的金鑰存放路徑、名稱及欲登入的 Hadoop 主機連結資訊。 :::info 注意：Hadoop 叢集服務提供虛擬機器預設的主機帳號為 fedora。 ::: <div style="background-color:black;color:white;padding:20px;"> C:\Users\Janice_Chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">icacls d:\key\janicekey01.pem /reset</span> 已處理的檔案: d:\key\janice0624.pem 已順利處理 1 個檔案; 0 個檔案處理失敗 C:\Users\Janice_Chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">icacls d:\key\janice0624.pem /GRANT:R "%USERNAME%:`(`R`)`"</span> 已處理的檔案: d:\key\janice0624.pem 已順利處理 1 個檔案; 0 個檔案處理失敗 C:\Users\Janice_Chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">icacls d:\key\janice0624.pem /inheritance:r</span> 已處理的檔案: d:\key\janice0624.pem 已順利處理 1 個檔案; 0 個檔案處理失敗 C:\Users\janice_chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">ssh -i d:\key\janice0624.pem fedora@203.145.220.143</span> The authenticity of host ‘203.145.220.143 (203.145.220.143)’ can’t be established. ECDSA key fingerprint is SHA256:lvnoNcwvZi04gjgizztLbqmSHBCYgT9cpFeRwfv11So. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ‘203.145.220.143’ (ECDSA) to the list of known hosts. Last login: Wed Jun 26 02:19:23 2019 from 59.124.220.27 [fedora@hadoop-master ~]$ </div> </br> ### 檢查 Hadoop 主機狀態 * 輸入 hdfs dfsadmin -report　指令檢查 DataNode 是否正常啟動 <div style="background-color:black;color:white;padding:20px;"> [fedora@hadoop-master ~]$ <span style="background-color:#fcf8e3; color: #000; padding:3.2px">hdfs dfsadmin -report</span> Configured Capacity: 321869430784 (299.76 GB) Present Capacity: 295097856000 (274.83 GB) DFS Remaining: 295097741312 (274.83 GB) DFS Used: 114688 (112 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 Pending deletion blocks: 0 Live datanodes (4): Name: 192.168.211.27:50010 (hadoop-master) Hostname: hadoop-master Decommission Status : Normal Configured Capacity: 80467357696 (74.94 GB) DFS Used: 28672 (28 KB) Non DFS Used: 3223433216 (3.00 GB) DFS Remaining: 73764782080 (68.70 GB) DFS Used%: 0.00% DFS Remaining%: 91.67% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Jun 28 08:00:03 UTC 2019 Last Block Report: Fri Jun 28 06:54:06 UTC 2019 Name: 192.168.211.30:50010 (hadoop-slave1) Hostname: hadoop-slave1 Decommission Status : Normal Configured Capacity: 80467357696 (74.94 GB) DFS Used: 28672 (28 KB) Non DFS Used: 3200741376 (2.98 GB) DFS Remaining: 73787473920 (68.72 GB) DFS Used%: 0.00% DFS Remaining%: 91.70% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Jun 28 08:00:02 UTC 2019 Last Block Report: Fri Jun 28 02:36:34 UTC 2019 Name: 192.168.211.42:50010 (hadoop-slave2) Hostname: hadoop-slave2 Decommission Status : Normal Configured Capacity: 80467357696 (74.94 GB) DFS Used: 28672 (28 KB) Non DFS Used: 3220684800 (3.00 GB) DFS Remaining: 73767530496 (68.70 GB) DFS Used%: 0.00% DFS Remaining%: 91.67% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Jun 28 08:00:04 UTC 2019 Last Block Report: Fri Jun 28 05:40:40 UTC 2019 Name: 192.168.211.69:50010 (hadoop-slave3) Hostname: hadoop-slave3 Decommission Status : Normal Configured Capacity: 80467357696 (74.94 GB) DFS Used: 28672 (28 KB) Non DFS Used: 3210260480 (2.99 GB) DFS Remaining: 73777954816 (68.71 GB) DFS Used%: 0.00% DFS Remaining%: 91.69% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Jun 28 08:00:02 UTC 2019 Last Block Report: Fri Jun 28 04:57:40 UTC 2019 [fedora@hadoop-master ~]$ </div> ### 檢查 Hadoop 主機運行程序 * 輸入 jps 指令檢查各個 Hadoop 主機上的運行程序，如 NameNode、DataNode、ResourceManager、NodeManager等。 <div style="background-color:black;color:white;padding:20px;"> [fedora@hadoop-master ~]$<span style="background-color:#fcf8e3; color: #000; padding:3.2px">jps</span> 3395 SecondaryNameNode 24900 Jps 3158 NameNode 3575 ResourceManager 3768 JobHistoryServer 3262 DataNode [fedora@hadoop-master ~]$ </div> ### 連線 Hadoop Web UI * Hodoop 服務啟動成功後，可以透過瀏覽器連線到 Hadoop Web UI，查看服務運行狀況。例如："http://Master IP:50070" ![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_d01d3e51d1579fb152bbeff36361753a.png)