資料轉移 | zh

--- title: 資料轉移 | zh tags: Guide, TWNIA3, TW GA: --- {%hackmd @docsharedstyle/default %} <div style="text-align: right">更新日期：2023/12/28</div> 台灣杉一號資料轉移至台灣杉三號說明 ===  [TOC] ## 登入資訊 <table> <tr> <th>類別</th> <th>台灣杉一號</th> <th>台灣杉三號</th> </tr> <tr> <th>登入節點</th> <th>clogin1.twnia.nchc.org.tw clogin2.twnia.nchc.org.tw</th> <th>twnia3.nchc.org.tw twnia3a.nchc.org.tw</th> </tr> <tr> <th>傳輸節點</th> <th>xdata1.twnia.nchc.org.tw xdata2.twnia.nchc.org.tw</th> <th>t3-x1.nchc.org.tw t3-x2.nchc.org.tw</th> </tr> <tr> <th>家目錄</th> <th>/home/$user1</th> <th>/home/$user</th> </tr> <tr> <th>工作目錄</th> <th>/work1/$user</th> <th>/work/$user</th> </tr> <tr> <th>Project目錄</th> <th>/project/$計畫代號2</th> <th>/project/$計畫代號</th> </tr> </table> :::info **備註**： 1.$user為使用者帳號，勿直接使用$user 2.$計畫代號為使用者於國網中心使用之計畫編號，如MST113001 ::: --- :::success **由台灣杉三號至台灣杉一號傳輸檔案資料** ::: 製作與傳送SSH金鑰 --- 1. 以帳號、密碼、OTP登入台灣杉三號登入節點 ``` login as: demo_user Keyboard-interactive authentication prompts from server: | Password: | Changing MOTP: 537268 End of keyboard-interactive prompts from server Auth MOTP: PASS Auth Password: PASS Last login: Tue Dec 26 15:26:33 2023 ... [demo_user@lgn301 ~]$ ``` 2. 以ssh-keygen製作SSH金鑰 [demo_user@lgn301 ~]$ ssh-keygen -t rsa > rsa可替換為dsa、ecdsa、ed25519、rsa1或使用預設值（不加任何參數）亦可 ``` Generating public/private rsa key pair. Enter file in which to save the key (/home/demo_user/.ssh/id_rsa): Enter passphrase (empty for no passphrase): <--建議輸入密碼保護，省略亦可使用，但有安全性疑慮 Enter same passphrase again: Your identification has been saved in /home/demo_user/.ssh/id_rsa. Your public key has been saved in /home/demo_user/.ssh/id_rsa.pub. The key fingerprint is: SHA256:RPWBSlXmUJ9ysUMxcxnqo5cj7zdkAK+qHJie1XxUovw demo_user@lgn301.nchc-2020-hpc The key's randomart image is: +---[RSA 2048]----+ | .o++= *o+| | .. .* +.O | | ... .*o* | | .o . += . | | So ..o. | | o o o.. oo | | o o o.E +o | | . + ... + .o | | o o. .o. .| +----[SHA256]-----+ ``` :::info **備註**：產生/home/demo_user/.ssh/id_rsa（私鑰）與 /home/demo_user/.ssh/id_rsa.pub（公鑰） ::: 3. 以ssh-copy-id傳送SSH金鑰至遠端（台灣杉一號） [demo_user@lgn301 ~]$ ssh-copy-id -i /home/demo_user/.ssh/id_rsa.pub demo_user@clogin1.twnia.nchc.org.tw :::info **備註**： -i 指定公鑰，若非rsa型式，請依實際名稱輸入 ::: ``` /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/demo_user/.ssh/id_rsa.pub" /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys Password: Changing MOTP: 344464 Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'demo_user@clogin1.twnia.nchc.org.tw'" and check to make sure that only the key(s) you wanted were added. [demo_user@lgn301 ~]$ ``` 傳輸方式（任一皆可） --- ### SCP * 傳輸檔案 1. 語法： scp 來源檔案目的檔名/目錄 scp -r 來源目錄/檔案目的目錄/檔案 2. 範例： [demo_user@lgn301 demo_user]$ scp demo_user@xdata1.twnia.nchc.org.tw:/work1/demo_user/CentOS-7-x86_64-Everything-2009.iso /work/demo_user/ ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': CentOS-7-x86_64-Everything-2009.iso 100% 9728MB 190.0MB/s 00:51 [demo_user@lgn301 demo_user]$ ls -l CentOS-7-x86_64-Everything-2009.iso -rw-r--r-- 1 demo_user TRI107056 10200547328 Dec 21 15:15 CentOS-7-x86_64-Everything-2009.iso ``` :::info **備註**：因二系統間帳號是一樣的，故亦可省略帳號，如：scp xdata1.twnia.nchc.org.tw:/work1/demo_user/CentOS-7-x86_64-Everything-2009.iso /work/demo_user/ ::: [demo_user@lgn301 ~]$ scp -r xdata1.twnia.nchc.org.tw:/work1/demo_user/*.c /work/demo_user/ ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': cpi.c 100% 1685 88.9KB/s 00:00 hello.c 100% 123 16.5KB/s 00:00 [demo_user@lgn301 ~]$ ``` :::info **備註**：檔案名稱可使用regular expression方式表示，如*、?、.之類的字元 ::: * 傳輸目錄 1. 語法： scp -r 來源目錄目的目錄 2. 範例： [demo_user@lgn301 demo_user]$ scp -r demo_user@xdata1.twnia.nchc.org.tw:/work1/demo_user/batch /work/demo_user/ ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': sub_ansys182fluent 100% 2192 277.6KB/s 00:00 sub_ansys190fluent 100% 2223 127.1KB/s 00:00 [demo_user@lgn301 demo_user]$ ``` ### SFTP * 範例 [demo_user@lgn303 ~]$ sftp demo_user@xdata1.twnia.nchc.org.tw ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': <--若有密碼保護，請輸入密碼 Connected to xdata1.twnia.nchc.org.tw. sftp> pwd <--顯示遠端目錄位置 Remote working directory: /home/u2/demo_user sftp> cd /work1/demo_user <-- 切換遠端目錄至 /work1/demo_user sftp> ls -l <-- 列出遠端目錄/檔案 drwxr-xr-x 2 demo_user TRI107053 4096 May 3 2018 APP -rw-r--r-- 1 demo_user TRI107056 10200547328 Sep 2 2021 CentOS-7-x86_64-Everything-2009.iso -rw-r--r-- 1 demo_user TRI107056 30601641984 Sep 2 2021 CentOS-7-x86_64-Everything-2009.iso.1 drwxr-xr-x 2 demo_user TRI107053 4096 Apr 12 2018 Desktop drwxr-xr-x 2 demo_user root 4096 Aug 21 2018 batch -rw-r--r-- 1 demo_user TRI107056 215416484 Sep 29 2021 chr_21.zip -rw-r--r-- 1 demo_user TRI107056 71615948757 Sep 28 2021 chr_21.zip.1 -rw-r--r-- 1 demo_user TRI107053 1685 Mar 30 2018 cpi.c drwxr-xr-x 2 demo_user TRI107056 4096 Sep 20 2018 hello -rw-r--r-- 1 demo_user TRI107053 123 Mar 23 2018 hello.c drwxr-xr-x 4 demo_user TRI107056 4096 Jan 6 2020 hfss -rwxr--r-- 1 demo_user TRI107053 271 Mar 30 2018 mpi_job.sh -rw------- 1 demo_user TRI107056 2001 Aug 17 2018 mpijob.o259008 drwxr-xr-x 10 demo_user TRI107053 4096 Feb 1 2019 src drwxr-xr-x 6 demo_user TRI107053 4096 Dec 20 2018 test -rwxr--r-- 1 demo_user TRI107056 275 Aug 17 2018 test.job -rwxr--r-- 1 demo_user TRI107056 275 Sep 5 2018 test.sh drwxr-xr-x 2 demo_user root 4096 Aug 21 2018 test2 drwxr-xr-x 2 demo_user root 4096 Aug 21 2018 test3 -rw-r--r-- 1 demo_user TRI107053 3017 Aug 25 2016 vim.tgz sftp> lcd /work/demo_user <-- 切換本地端（台灣杉三號）目錄至 /work/demo_user sftp> get CentOS-7-x86_64-Everything-2009.iso <-- 下載檔案CentOS-7-x86_64-Everything-2009.iso Fetching /work1/demo_user/CentOS-7-x86_64-Everything-2009.iso to CentOS-7-x86_64-Everything-2009.iso /work1/demo_user/CentOS-7-x86_64-Everything-2009.iso 100% 9728MB 211.5MB/s 00:46 sftp> get -r batch <-- 下載batch 目錄 Fetching /work1/demo_user/batch/ to batch Retrieving /work1/demo_user/batch /work1/demo_user/batch/sub_ansys182fluent 100% 2192 173.1KB/s 00:00 /work1/demo_user/batch/sub_ansys190fluent 100% 2223 136.3KB/s 00:00 sftp> help <-- 指令功能說明 Available commands: bye Quit sftp cd path Change remote directory to 'path' chgrp grp path Change group of file 'path' to 'grp' chmod mode path Change permissions of file 'path' to 'mode' chown own path Change owner of file 'path' to 'own' df [-hi] [path] Display statistics for current directory or filesystem containing 'path' exit Quit sftp get [-afPpRr] remote [local] Download file reget [-fPpRr] remote [local] Resume download file reput [-fPpRr] [local] remote Resume upload file help Display this help text lcd path Change local directory to 'path' lls [ls-options [path]] Display local directory listing lmkdir path Create local directory ln [-s] oldpath newpath Link remote file (-s for symlink) lpwd Print local working directory ls [-1afhlnrSt] [path] Display remote directory listing lumask umask Set local umask to 'umask' mkdir path Create remote directory progress Toggle display of progress meter put [-afPpRr] local [remote] Upload file pwd Display remote working directory quit Quit sftp rename oldpath newpath Rename remote file rm path Delete remote file rmdir path Remove remote directory symlink oldpath newpath Symlink remote file version Show SFTP version !command Execute 'command' in local shell ! Escape to local shell ? Synonym for help ``` ### RSYNC 1. 語法：rsync 參數來源檔案/目錄目的檔案/目錄 2. 基本參數說明： -v：verbose模式，輸出較詳細的資訊 -a：封裝模式備份，等同使用參數-rlptgoD -r：遞迴（--recursive），備份目錄下的所有檔案與子目錄 -z：在傳輸過程中壓縮資料，以節省頻寬 -h：以易讀的數字（單位）顯示 --progress：傳輸過程中顯示進度詳細參數說明，請登入系統man rsync。範例： * 單一檔案 [demo_user@lgn303 ~]$ rsync -azvh demo_user@xdata1.twnia.nchc.org.tw:/work1/demo_user/CentOS-7-x86_64-Everything-2009.iso ./ ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': receiving incremental file list CentOS-7-x86_64-Everything-2009.iso sent 30 bytes received 10.20G bytes 131.66M bytes/sec total size is 10.20G speedup is 1.00 ``` * 顯示進度 --progress [demo_user@lgn303 ~]$ rsync -azvh --progress demo_user@xdata1.twnia.nchc.org.tw:/work1/demo_user/CentOS-7-x86_64-Everything-2009.iso ./ ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': receiving incremental file list CentOS-7-x86_64-Everything-2009.iso 1.44G 14% 138.40MB/s 0:01:01 ``` * 傳輸目錄 [demo_user@lgn303 ~]$ rsync -azvh --progress demo_user@xdata1.twnia.nchc.org.tw:/work1/demo_user/hfss ./ ``` Enter passphrase for key '/home/demo_user/.ssh/id_rsa': receiving incremental file list hfss/ hfss/HFSS_test.o5328178 34.22K 100% 32.63MB/s 0:00:00 (xfr#1, to-chk=41/43) hfss/HFSS_test.o5328760 34.62K 100% 1.65MB/s 0:00:00 (xfr#2, to-chk=40/43) hfss/HFSS_test.o5328900 960 100% 44.64kB/s 0:00:00 (xfr#3, to-chk=39/43) hfss/HFSS_test.o5328919 960 100% 18.38kB/s 0:00:00 (xfr#4, to-chk=38/43) ...（略） 258 100% 0.56kB/s 0:00:00 (xfr#33, to-chk=1/43) hfss/phased_array_unitcell.aedtresults/mf_2/svcache/ hfss/phased_array_unitcell.aedtresults/mf_2/svcache/HDR4MlOn19068615771756414.tmp 320 100% 0.68kB/s 0:00:00 (xfr#34, to-chk=0/43) sent 693 bytes received 112.46K bytes 17.41K bytes/sec total size is 558.48K speedup is 4.94 ``` 反向設定範例 --- :::success **由台灣杉一號往台灣杉三號傳輸檔案資料** ::: 1. 以帳號、密碼、OTP登入台灣杉一號 2. 製作SSH 金鑰 [demo_user@clogin1 ~]$ ssh-keygen -t ecdsa ``` Generating public/private ecdsa key pair. Enter file in which to save the key (/home/demo_user/.ssh/id_ecdsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/demo_user/.ssh/id_ecdsa. Your public key has been saved in /home/demo_user/.ssh/id_ecdsa.pub. The key fingerprint is: a7:43:1a:66:da:32:38:11:b8:08:a2:40:cd:8b:ea:0c demo_user@clogin1 The key's randomart image is: +--[ECDSA 256]---+ | .o | |.. o | |= o . | |*o o | |= . + S . | |E o = + o | |+ o + o o | | o . o . | | | +-----------------+ ``` 3. 設定SSH Key Authorization，傳送公鑰至Server端（台灣杉三號） [demo_user@clogin1 ~]$ ssh-copy-id -i ~/.ssh/id_ecdsa.pub demo_user@twnia3.nchc.org.tw ``` /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys Password: Changing MOTP: 149501 Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'demo_user@twnia3.nchc.org.tw'" and check to make sure that only the key(s) you wanted were added. ``` :::info 由於製作的ssh金鑰是以ecdsa演算法產生，設定時須注意公鑰檔名。 ::: * scp [demo_user@clogin1 ~]$ scp ./vim.tgz demo_user@twnia3.nchc.org.tw:~/ ``` Password: Changing MOTP: 725223 vim.tgz 100% 3017 3.0KB/s 00:00 [demo_user@clogin1 ~]$ ``` * sftp ``` [demo_user@clogin1 ~]$ sftp demo_user@t3-x1.nchc.org.tw Connected to t3-x1.nchc.org.tw. sftp> put vim.tgz Uploading vim.tgz to /home/demo_user/vim.tgz vim.tgz 100% 3017 3.0KB/s 00:00 sftp> ``` :::warning **備註**： **台灣杉三號登入節點不接受sftp，檔案傳輸請透過傳輸節點進行。** ::: * rsync [demo_user@clogin1 ~]$ rsync -azvh ./abc_9.1.3.tbz demo_user@t3-x1.nchc.org.tw:~/ ``` sending incremental file list abc_9.1.3.tbz sent 428.23M bytes received 31 bytes 77.86M bytes/sec total size is 428.08M speedup is 1.00 ```