Cheatnvidia-smi GPU monitoring
Light · Postcard
102 × 152 mm · PDF
Dark · Postcard
102 × 152 mm · PDF
Ink saver · Postcard
102 × 152 mm · PDF
Light · A4 landscape
297 × 210 mm · PDF
Dark · A4 landscape
297 × 210 mm · PDF
Ink saver · A4 landscape
297 × 210 mm · PDF
Light · A3 landscape
420 × 297 mm · PDF
Dark · A3 landscape
420 × 297 mm · PDF
Ink saver · A3 landscape
420 × 297 mm · PDF
Start here
nvidia-smiGPU summary
nvidia-smi -LList GPUs and UUIDs
nvidia-smi -l 2Refresh every 2 s
nvidia-smi -qFull detail dump
nvidia-smi pmonPer-process usage
nvidia-smi dmonRolling metrics
nvidia-smi topo -m
Link topology
nvidia-smi -i <id> -q
One GPU only
Concepts
- nvidia-smi summary table
- Detail or CSV pick fields
- Live monitor loop or stream
- Tuning needs root
Inspect
nvidia-smiShow GPU summary table
nvidia-smi -LList GPUs with UUIDs
nvidia-smi -qShow all details for every GPU
nvidia-smi -i <id>
Limit output to one GPU
<id> is an index, UUID, serial or PCI bus ID. Works with
most options.
nvidia-smi -q -d MEMORY,UTILIZATION
Show only chosen sections
Other types: ECC, TEMPERATURE, POWER, CLOCK, PERFORMANCE.
nvidia-smi -q -x
Print details as XML
nvidia-smi -f <file> -q
Write output to a file
nvidia-smi --version
Show tool and driver version
nvidia-smi -hPrint help
Monitor
nvidia-smi -l <sec>
Repeat every N seconds
nvidia-smi -lms <ms>
Repeat every N milliseconds
nvidia-smi dmonStream power, temp, clocks, load
nvidia-smi dmon -s um
Choose metric groups to stream
Groups: p power/temp, u utilization, c clocks, v violations, m
memory, e ECC, t PCIe.
nvidia-smi pmonShow load per process
nvidia-smi pmon -s um
Per-process utilization and memory
nvidia-smi dmon -c <n>
Stop after N samples
watch -n 1 nvidia-smi
Refresh the table in place
Scripted queries
nvidia-smi --query-gpu=<fields> --format=csv
Query chosen fields as CSV
nvidia-smi --query-gpu=name,memory.used,memory.total
--format=csv
Memory use per GPU
nvidia-smi --query-gpu=temperature.gpu
--format=csv,noheader,nounits
Bare numbers for scripts
nvidia-smi
--query-compute-apps=pid,process_name,used_memory
--format=csv
Compute processes and memory
nvidia-smi --help-query-gpu
List every queryable field
nvidia-smi --query-gpu=<fields> --format=csv -l
1
Log CSV every second
nvidia-smi --query-supported-clocks=gr,mem
--format=csv
List supported clock pairs
nvidia-smi --help-query-compute-apps
List compute-app fields
Persistence and compute mode
sudo nvidia-smi -pm 1
Enable persistence mode
Keeps the driver loaded so launches are faster. Resets on
reboot; NVIDIA recommends the persistence daemon.
sudo nvidia-smi -c <mode>
Set compute mode
DEFAULT, EXCLUSIVE_PROCESS or PROHIBITED. Resets on reboot.
sudo nvidia-smi -r
Reset the GPU
Fails if processes still use the GPU.
sudo nvidia-smi -e <0|1>
Disable or enable ECC
Takes effect after a reboot or reset.
sudo nvidia-smi -am 1
Turn on process accounting
Power and clocks
sudo nvidia-smi -pl <watts>
Set power limit
Must stay within the range shown by -q -d POWER.
sudo nvidia-smi -lgc <min>,<max>
Lock GPU clock range in MHz
sudo nvidia-smi -rgc
Unlock GPU clocks
sudo nvidia-smi -lmc <min>,<max>
Lock memory clock range in MHz
sudo nvidia-smi -rmc
Unlock memory clocks
nvidia-smi -q -d CLOCK
Show current and max clocks
nvidia-smi -q -d PERFORMANCE
Show throttle reasons
Topology and others
nvidia-smi topo -m
Show GPU interconnect matrix
Shows NVLink, PCIe and CPU affinity between devices.
nvidia-smi nvlink -s
Show NVLink status
nvidia-smi mig -lgi
List MIG GPU instances
MIG is Ampere and newer; enabling it needs -mig 1 plus a
reset.
nvidia-smi -mig 1
Enable MIG mode
nvidia-smi drain -h
Show GPU drain options
Recipes
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,power.draw \ --format=csv,noheader -l 5 >> gpu.csvnvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheaderps -o pid,user,cmd -p <pid>sudo nvidia-smi -pm 1sudo nvidia-smi -pl 250nvidia-smi -q -d POWER | head -20nvidia-smi -i 0 -l 1 \ --query-gpu=temperature.gpu,utilization.gpu,memory.used --format=csv
