---
title: Hadoop 叢集
tags: Guide, HD Cluster, TW
GA: UA-155999456-1
---
{%hackmd @docsharedstyle/default %}
# Hadoop 叢集
TWCC 提供建立 Hadoop 叢集的服務,可讓使用者進行巨量資料的運算及分析使用。
## 建立 Hadoop 叢集
* 進入「Hadoop 叢集」服務,在「Hadoop 叢集管理」頁面,點擊「+建立」。
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_43bf769f1747a9afc4a885bd87a5f991.png)
* 在基本資訊頁面填寫名稱、附件數、映像檔及硬體設定,接著點擊「下一步:鑰匙對>」。
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_f5c61d1d29060a741b667bd312b189ce.png)
* 選擇欲使用的鑰匙對,或是點選「+建立鑰匙對」,接著點擊「下一步:網路>」
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_1e1a07f1770e162e7bfdf4f3cd0b23ab.png)
* 選擇欲使用的虛擬網路,接著點擊「下一步:檢閱 + 建立>」。
:::info
:book: 參見:[虛擬網路](https://www.twcc.ai/doc?page=virtual_network)
:::
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_03d40d952d1991356efb0019b4c521d2.png)
* 檢視整個 Hadoop 叢集的設定和預估使用額度,按下「建立」即完成,幾分鐘後等狀態變成 Ready 即可開始連線使用。
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_66d9f6ca5e2ebf111d7c6c4f233f7627.png)
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_9971e4a7180d36971f1a46250bcea8f9.png)
## 檢視大數據叢集內容
* 在「Hadoop 叢集管理」頁面,點擊欲檢視的項目列表,即可檢視 Hadoop 叢集的詳細內容。點擊上方按鈕,可對此大數據叢集進行啟動、停止、刪除及重新整理。
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_4248812986ec74fc5bfbaefba7219b34.png)
## 登入 Hadoop 主機
* 登入 Hadoop 主機與[連結虛擬運算個體](https://man.twcc.ai/uaKL6SkBT9GePrMILRpomg#%E9%80%A3%E7%B5%90%E8%99%9B%E6%93%AC%E6%A9%9F%E5%99%A8)的方式相同,可在命令提示字元視窗下使用 SSH 指令登入。
* 開啟命令提示字元視窗,依照以下範例輸入您的金鑰存放路徑、名稱及欲登入的 Hadoop 主機連結資訊。
:::info
注意:Hadoop 叢集服務提供虛擬機器預設的主機帳號為 fedora。
:::
<div style="background-color:black;color:white;padding:20px;">
C:\Users\Janice_Chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">icacls d:\key\janicekey01.pem /reset</span>
已處理的檔案: d:\key\janice0624.pem
已順利處理 1 個檔案; 0 個檔案處理失敗
C:\Users\Janice_Chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">icacls d:\key\janice0624.pem /GRANT:R "%USERNAME%:`(`R`)`"</span>
已處理的檔案: d:\key\janice0624.pem
已順利處理 1 個檔案; 0 個檔案處理失敗
C:\Users\Janice_Chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">icacls d:\key\janice0624.pem /inheritance:r</span>
已處理的檔案: d:\key\janice0624.pem
已順利處理 1 個檔案; 0 個檔案處理失敗
C:\Users\janice_chiang><span style="background-color:#fcf8e3; color: #000; padding:3.2px">ssh -i d:\key\janice0624.pem fedora@203.145.220.143</span>
The authenticity of host ‘203.145.220.143 (203.145.220.143)’ can’t be established.
ECDSA key fingerprint is SHA256:lvnoNcwvZi04gjgizztLbqmSHBCYgT9cpFeRwfv11So.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘203.145.220.143’ (ECDSA) to the list of known hosts.
Last login: Wed Jun 26 02:19:23 2019 from 59.124.220.27
[fedora@hadoop-master ~]$
</div>
</br>
### 檢查 Hadoop 主機狀態
* 輸入 hdfs dfsadmin -report 指令檢查 DataNode 是否正常啟動
<div style="background-color:black;color:white;padding:20px;">
[fedora@hadoop-master ~]$ <span style="background-color:#fcf8e3; color: #000; padding:3.2px">hdfs dfsadmin -report</span>
Configured Capacity: 321869430784 (299.76 GB)
Present Capacity: 295097856000 (274.83 GB)
DFS Remaining: 295097741312 (274.83 GB)
DFS Used: 114688 (112 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0
Live datanodes (4):
Name: 192.168.211.27:50010 (hadoop-master)
Hostname: hadoop-master
Decommission Status : Normal
Configured Capacity: 80467357696 (74.94 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 3223433216 (3.00 GB)
DFS Remaining: 73764782080 (68.70 GB)
DFS Used%: 0.00%
DFS Remaining%: 91.67%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Jun 28 08:00:03 UTC 2019
Last Block Report: Fri Jun 28 06:54:06 UTC 2019
Name: 192.168.211.30:50010 (hadoop-slave1)
Hostname: hadoop-slave1
Decommission Status : Normal
Configured Capacity: 80467357696 (74.94 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 3200741376 (2.98 GB)
DFS Remaining: 73787473920 (68.72 GB)
DFS Used%: 0.00%
DFS Remaining%: 91.70%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Jun 28 08:00:02 UTC 2019
Last Block Report: Fri Jun 28 02:36:34 UTC 2019
Name: 192.168.211.42:50010 (hadoop-slave2)
Hostname: hadoop-slave2
Decommission Status : Normal
Configured Capacity: 80467357696 (74.94 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 3220684800 (3.00 GB)
DFS Remaining: 73767530496 (68.70 GB)
DFS Used%: 0.00%
DFS Remaining%: 91.67%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Jun 28 08:00:04 UTC 2019
Last Block Report: Fri Jun 28 05:40:40 UTC 2019
Name: 192.168.211.69:50010 (hadoop-slave3)
Hostname: hadoop-slave3
Decommission Status : Normal
Configured Capacity: 80467357696 (74.94 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 3210260480 (2.99 GB)
DFS Remaining: 73777954816 (68.71 GB)
DFS Used%: 0.00%
DFS Remaining%: 91.69%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Jun 28 08:00:02 UTC 2019
Last Block Report: Fri Jun 28 04:57:40 UTC 2019
[fedora@hadoop-master ~]$
</div>
### 檢查 Hadoop 主機運行程序
* 輸入 jps 指令檢查各個 Hadoop 主機上的運行程序,如 NameNode、DataNode、ResourceManager、NodeManager等。
<div style="background-color:black;color:white;padding:20px;">
[fedora@hadoop-master ~]$<span style="background-color:#fcf8e3; color: #000; padding:3.2px">jps</span>
3395 SecondaryNameNode
24900 Jps
3158 NameNode
3575 ResourceManager
3768 JobHistoryServer
3262 DataNode
[fedora@hadoop-master ~]$
</div>
### 連線 Hadoop Web UI
* Hodoop 服務啟動成功後,可以透過瀏覽器連線到 Hadoop Web UI,查看服務運行狀況。例如:"http://Master IP:50070"
![](https://cos.twcc.ai/SYS-MANUAL/uploads/upload_d01d3e51d1579fb152bbeff36361753a.png)