Miniconda 建立自己的使用環境

# Miniconda 建立自己的使用環境 [TOC] :::success :warning: 執行環境須與編譯安裝的環境設定一致，勿混用！！！使用conda安裝者，請以conda環境執行。 ::: ## 1. 建立自己的環境 ``` console [user@cbi-lgn01 ~]$ ml purge [user@cbi-lgn01 ~]$ ml list No modules loaded [user@cbi-lgn01 ~]$ ml load miniconda3 (base) [user@cbi-lgn01 ~]$ ml list # 自動切換成 base Currently Loaded Modules: 1) miniconda3/24.11.1 # 建立新的環境，自訂你的環境要叫什麼名字，依據你想要的Python版本安裝 (base) [user@cbi-lgn01 ~]$ conda create -n pytorch-cuda python=3.12 Channels: - defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done (... stuff deleted ...) Proceed ([y]/n)? y Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate pytorch-cuda # # To deactivate an active environment, use # # $ conda deactivate ``` 如果您想列出所有已經建立的 conda 環境，可以執行以下命令 ``` console (base) [user@cbi-lgn01 ~]$ conda env list # conda environments: # pytorch-cuda /home/user/.conda/envs/pytorch-cuda base * /work/HPC_software/LMOD/miniconda3/miniconda3_app/24.11.1 ``` 如果您想啟動在此之前所建立的 `pytorch-cuda` 的環境，可以執行以下命令 ``` console (base) [user@cbi-lgn01 ~]$ conda activate pytorch-cuda (pytorch-cuda) [user@cbi-lgn01 ~]$ ``` 或是關閉目前所啟動的環境，可以執行以下命令 ``` console (pytorch-cuda) [user@cbi-lgn01 ~]$ conda deactivate (base) [user@cbi-lgn01 ~]$ ``` 抑或是完全的離開 base 虛擬環境，請再執行以下命令 ``` console (base) [user@cbi-lgn01 ~]$ conda deactivate [user@cbi-lgn01 ~]$ ml list Currently Loaded Modules: 1) miniconda3/24.11.1 [user@cbi-lgn01 ~]$ ml purge # 卸載所有 module，完完全全的離開 miniconda， ``` ## 2. 安裝 tools 在你自己的 PyTorch-CUDA 環境中啟動 `pytorch-cuda` 環境 ``` console [user@cbi-lgn01 ~]$ ml load miniconda3 (base) [user@cbi-lgn01 ~]$ conda activate pytorch-cuda ``` 因為 conda solver 太「聰明」，會幫您安裝了一組「實際上不能用」的組合，所以要先強制要求 conda 降級所安裝的 Intel OpenMP / MKL，接著才能這在這環境中安裝您的 PyTorch。 ``` console (pytorch-cuda) [user@cbi-lgn01 ~]$ conda install -c defaults "intel-openmp<2024" "mkl<2024" Channels: - defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done (... stuff deleted ...) Proceed ([y]/n)? y Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done (pytorch-cuda) [user@cbi-lgn01 ~]$ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia Channels: - pytorch - nvidia - defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done (... stuff deleted ...) Proceed ([y]/n)? y Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done (pytorch-cuda) [user@cbi-lgn01 ~]$ ```  ## 3. 產生測試程式附註：請直接將 `cat <<-EOF >pytorch_cuda_example.py` 一直到包含 `EOF` 的所有內容複製，並直接在命令列貼上，就會產生測試程式 `pytorch_cuda_example.py` ``` console (pytorch-cuda) [user@cbi-lgn01 pytorch_cuda]$ cat <<-EOF >pytorch_cuda_example.py import torch # 檢查 CUDA 是否可用 if torch.cuda.is_available(): device = torch.device("cuda") # 使用 GPU print(f"使用裝置：{torch.cuda.get_device_name(0)}") # 顯示使用的 GPU 名稱 else: device = torch.device("cpu") # 使用 CPU print("CUDA 不可用，使用 CPU。") # 定義一個簡單的模型 class SimpleModel(torch.nn.Module): def __init__(self): super(SimpleModel, self).__init__() self.linear = torch.nn.Linear(10, 1) def forward(self, x): return self.linear(x) # 創建模型實例並移動到指定裝置 (GPU 或 CPU) model = SimpleModel().to(device) # 創建一些隨機資料並移動到指定裝置 input_data = torch.randn(32, 10).to(device) # 32 筆資料，每筆 10 個特徵 target_data = torch.randn(32, 1).to(device) # 定義損失函數和優化器 criterion = torch.nn.MSELoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.001) # 訓練迴圈 num_epochs = 100 for epoch in range(num_epochs): # 前向傳播 outputs = model(input_data) loss = criterion(outputs, target_data) # 反向傳播和優化 optimizer.zero_grad() loss.backward() optimizer.step() if (epoch+1) % 10 == 0: print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}') print("Finished！") EOF ``` ## 4. 產生測試 script 附註：請直接將 `cat <<-EOF >pytorch_cuda_example.sh` 一直到包含 `EOF` 的所有內容複製，並直接在命令列貼上，就會產生測試程式 `pytorch_cuda_example.sh` ``` console (pytorch-test) [user@cbi-lgn01 pytorch_cuda]$ cat <<-EOF >pytorch_cuda_example.sh #!/bin/bash #SBATCH --job-name=pytorch_cuda_job # 工作名稱 #SBATCH --partition=dev # 使用的 partition (請根據你的系統修改) #SBATCH --time=00:30:00 # 執行時間上限 (小時:分鐘:秒) #SBATCH --account=GOV113080 ####### 請記得換成您的計畫代碼 ####### #SBATCH --nodes=1 # (-N) Maximum number of nodes to be allocated #SBATCH --gpus-per-node=2 # Gpus per node #SBATCH --cpus-per-task=4 # (-c) Number of cores per MPI task #SBATCH --ntasks-per-node=4 # Maximum number of tasks on each node #SBATCH -o pytorch_cuda_example_%j.log # output file (%j expands to jobId) #SBATCH -e pytorch_cuda_example_%j.err # output file (%j expands to jobId) ml load miniconda3/24.11.1 CONDA_DEFAULT_ENV="pytorch-cuda" # 你的 conda 環境名稱 conda run -n "$CONDA_DEFAULT_ENV" python pytorch_cuda_example.py EOF ```  ## 5. 提交 Slurm 執行 ``` console (pytorch-test) [user@cbi-lgn01 pytorch_cuda]$ sbatch pytorch_cuda_example.sh Submitted batch job 10640 ``` ## 6. 檢查執行結果 ``` console (pytorch-test) [user@cbi-lgn01 pytorch_cuda]$ ls -al -rw-r--r-- 1 user TRI1123529 0 Jan 22 10:12 pytorch_cuda_example_job_10640.err -rw-r--r-- 1 user TRI1123529 345 Jan 22 10:12 pytorch_cuda_example_job_10640.log (pytorch-test) [user@cbi-lgn01 pytorch_cuda]$ cat pytorch_cuda_example_job_10640.log 使用裝置：NVIDIA H100 80GB HBM3 Epoch [10/100], Loss: 1.2595 Epoch [20/100], Loss: 1.2277 Epoch [30/100], Loss: 1.1992 Epoch [40/100], Loss: 1.1734 Epoch [50/100], Loss: 1.1499 Epoch [60/100], Loss: 1.1283 Epoch [70/100], Loss: 1.1087 Epoch [80/100], Loss: 1.0909 Epoch [90/100], Loss: 1.0747 Epoch [100/100], Loss: 1.0600 Finished！ ```

Read more

晶創25（Nano5）佇列清單與說明

@font-face ByNgJpz5kg (HackMD)

Slurm常用指令與參數

登入ThinLinc 互動式繪圖節點和操作範例說明