1.下載軟件
wget https://codeload.github.com/wilicc/gpu-burn/zip/master
2.解壓縮
unzip gpu-burn-master.zip
3.進入目錄編譯(確保cuda環境變量已經配置成功 nvcc -v能顯示結果)
cd gpu-burn-master
make
4.編譯成功后,會在當前目錄生成 gpu_burn 這個文件
gpu_burn
5.默認執行,跑全部GPU卡,空格后面參數為時間,一般快速測試設置100,穩定性測試為500
[root@localhost gpu-burn-master]#
./gpu_burn 100
GPU 0: Tesla V100 (UUID: GPU-6250466c-35ed-c279-fc0b-3b9b613a586f)
GPU 1: Tesla V100 (UUID: GPU-0a4a2b9c-d32c-1ba2-42a0-151ed9907d57)
GPU 2: Tesla V100 (UUID: GPU-f6cf184f-9173-1edd-648f-71e841afe152)
GPU 3: Tesla V100 (UUID: GPU-044f96e6-cc66-cc93-6283-07b829216f91) Initialized device 2 with 11178 MB of memory (10993 MB available, using 9894 MB of it), using FLOATS
Initialized device 1 with 11178 MB of memory (10993 MB available, using 9894 MB of it), using FLOATS
Initialized device 3 with 11178 MB of memory (10993 MB available, using 9894 MB of it), using FLOATS
Initialized device 0 with 11178 MB of memory (10993 MB available, using 9894 MB of it), using FLOATS
6.可以指定某幾張卡跑,比如指定0和1號卡
export CUDA_VISIBLE_DEVICES=0,1
./gpu_burn 100
如何找出故障卡
1. dmesg -l err 篩選出錯誤卡的Bus-Id
.png){kind=small}
2. 根據Bus-Id找出對應的GPU卡編號,在跑測試的時候排除它,比如機器8張卡,device 2 故障,那個參數這 樣寫:
export CUDA_VISIBLE_DEVICES=0,1,3,4,5,6,7 #2不寫在里面
./gpu_burn 100
3. 跑完之后關機,找出那張沒有溫度的卡,即故障卡