{"id":17254,"date":"2023-08-09T13:49:11","date_gmt":"2023-08-09T17:49:11","guid":{"rendered":"https:\/\/carleton.ca\/scs\/?page_id=17254"},"modified":"2025-07-25T09:15:44","modified_gmt":"2025-07-25T13:15:44","slug":"scs-gpu-vm-fall-2023","status":"publish","type":"page","link":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/","title":{"rendered":"DEPRECATED &#8211; SCS GPU VM  2023-4"},"content":{"rendered":"<p><a href=\"#details\">GPU VM Account Details<\/a><br \/>\n<a href=\"#testing\">Testing the GPU VM software<\/a><br \/>\n<a href=\"#troubleshooting\">Troubleshooting<\/a><\/p>\n<p>This page will show you how to test your virtual machine&#8217;s GPU and access and test the software that is pre-installed on it.<\/p>\n<h2><a id=\"details\"><\/a>GPU VM Account Details<\/h2>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-image-information\" aria-expanded=\"false\" aria-controls=\"slideme-image-information\" class=\"slideme__heading slideme__trigger\">Image Information<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-image-information\" aria-hidden=\"true\"><p><\/p>\n<p><strong>Image Name<\/strong>: SCS-GPU-fall-2023-08-16-v3<br \/>\n<strong>Creation Date<\/strong>: August 16, 2023<br \/>\n<strong>Operating System<\/strong>: Ubuntu 22.04<br \/>\n<strong>Window Manager<\/strong>: XFCE<br \/>\n<strong>Intended usage<\/strong>: Openstack GPU virtual machine with AI programming support<\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-account\" aria-expanded=\"false\" aria-controls=\"slideme-account\" class=\"slideme__heading slideme__trigger\">Account<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-account\" aria-hidden=\"true\"><p><\/p>\n<p>Accessing your GPU Virtual Machine. You will be given a username and password once your VM is ready.<\/p>\n<blockquote><p>Please change your password as soon as the VM is provisioned for you. This can be done by logging into your VM and then opening a terminal window and typing \u2018passwd\u2019.<\/p><\/blockquote>\n<p>Please note:<\/p>\n<ul>\n<li>Change the default password for your account!<\/li>\n<li>Your account does not have Openstack dashboard access. Access is by IP address only.<\/li>\n<li>There are no system backups for your VM, that is, you are responsible for your own backups!<\/li>\n<\/ul>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-accessing-your-gpu-virtual-machine\" aria-expanded=\"false\" aria-controls=\"slideme-accessing-your-gpu-virtual-machine\" class=\"slideme__heading slideme__trigger\">Accessing your GPU Virtual Machine<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-accessing-your-gpu-virtual-machine\" aria-hidden=\"true\"><p><\/p>\n<p>1. From outside of Carleton you will need to VPN to Carleton in order to access the VM<\/p>\n<ul>\n<li><strong>VPN<\/strong>:\u00a0<a href=\"https:\/\/carleton.ca\/its\/help-centre\/remote-access\/\" target=\"_blank\" rel=\"noopener noreferrer\">Carleton VPN<\/a>\u00a0when connecting from outside of the campus<\/li>\n<\/ul>\n<p>2. Listed are the ways you can connect to your VM:<\/p>\n<ul>\n<li><strong>x2go <\/strong>(graphical desktop): download\u00a0<a href=\"https:\/\/wiki.x2go.org\/doku.php\/download:start\" target=\"_blank\" rel=\"noopener noreferrer\">x2go client<\/a>\u00a0to get a\u00a0 full graphical desktop (use\u00a0<strong>Session Type:<\/strong> XFCE). More info: <a href=\"https:\/\/carleton.ca\/scs\/2021\/ssh-connection-with-x2go-remote-desktop-client\/\" target=\"_blank\" rel=\"noopener noreferrer\">SSH Connection with x2go Remote Desktop Client<\/a><\/li>\n<li><strong>ssh<\/strong> (command line): <a href=\"https:\/\/carleton.ca\/scs\/2021\/connecting-to-an-openstack-instance-using-ssh-putty\/\" target=\"_blank\" rel=\"noopener noreferrer\">use a terminal window to gain ssh access<\/a><\/li>\n<\/ul>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-system-administration\" aria-expanded=\"false\" aria-controls=\"slideme-system-administration\" class=\"slideme__heading slideme__trigger\">System Administration<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-system-administration\" aria-hidden=\"true\"><p><\/p>\n<ul>\n<li>You have full root\/sudo privileges on your GPU VM.<\/li>\n<li>You can restart your VM using the &#8216;reboot&#8217; command<\/li>\n<li>If you shutdown your VM then you need to contact an SCS Sysadmin to Start the VM for you<\/li>\n<li>If your VM is in an unusable state or it is difficult to fix errors then you have the option to re-launch your VM. Please contact the SCS System Administrator to re-launch the VM for you. Re-launching it means terminating the VM (you lose all local data) and launching a new instance.<\/li>\n<li>Sometimes the VM is not accessible via <a href=\"https:\/\/carleton.ca\/scs\/2021\/ssh-connection-with-x2go-remote-desktop-client\/\">x2go<\/a> but you can access it using an <a href=\"https:\/\/carleton.ca\/scs\/2021\/connecting-to-an-openstack-instance-using-ssh-putty\/\">ssh terminal like putty<\/a><\/li>\n<li>run the command &#8216;passwd&#8217; to change your local Ubuntu password<\/li>\n<\/ul>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-software\" aria-expanded=\"false\" aria-controls=\"slideme-software\" class=\"slideme__heading slideme__trigger\">Software<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-software\" aria-hidden=\"true\"><p><\/p>\n<p>This is the tested software installed on this virtual machine.<\/p>\n<table style=\"width: 45.4126%; height: 203px;\" border=\"1\">\n<tbody>\n<tr>\n<td style=\"width: 39.9824%;\"><strong>Software<\/strong><\/td>\n<td style=\"width: 101.276%;\"><strong>Version<\/strong><\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">NVIDIA Driver Ver.<\/td>\n<td style=\"width: 101.276%;\">12.0<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">CUDA Runtime<\/td>\n<td style=\"width: 101.276%;\">11.8<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">cuDNN<\/td>\n<td style=\"width: 101.276%;\">8.6.0<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">GCC<\/td>\n<td style=\"width: 101.276%;\">11.4.0 + (9.4.0)<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Conda<\/td>\n<td style=\"width: 101.276%;\">23.5.2<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Bazel<\/td>\n<td style=\"width: 101.276%;\">5.3.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Python software:<\/p>\n<table style=\"width: 45.4126%; height: 203px;\" border=\"1\">\n<tbody>\n<tr>\n<td style=\"width: 39.9824%;\"><strong>Software<\/strong><\/td>\n<td style=\"width: 101.276%;\"><strong>Version<\/strong><\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Python<\/td>\n<td style=\"width: 101.276%;\">3.10.12<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Pip3<\/td>\n<td style=\"width: 101.276%;\">22.1.1<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Tensorflow<\/td>\n<td style=\"width: 101.276%;\">2.9.1<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Python-torch<\/td>\n<td style=\"width: 101.276%;\">1.8.0a0<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Keras<\/td>\n<td style=\"width: 101.276%;\">2.9.0<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Keras-processing<\/td>\n<td style=\"width: 101.276%;\">1.1.2<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Pandas<\/td>\n<td style=\"width: 101.276%;\">2.0.3<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 39.9824%;\">Numpy<\/td>\n<td style=\"width: 101.276%;\">1.23.1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><\/p>\n<p><\/p><\/dd><dl><\/div>\n<h2><a id=\"testing\"><\/a>Testing the GPU VM software<\/h2>\n<p>Here are some helpful tests to see if your software is running correctly.<\/p>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-probing-your-gpu\" aria-expanded=\"false\" aria-controls=\"slideme-probing-your-gpu\" class=\"slideme__heading slideme__trigger\">Probing your GPU<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-probing-your-gpu\" aria-hidden=\"true\"><p><\/p>\n<p>This command allows you test if the GPU is being detected, identifies the GPU and shows any running jobs, utilisation, and memory usage in real time:<\/p>\n<p><code>nvidia-smi -l<\/code><\/p>\n<p>You should be able to view a realtime command prompt window that shows:<\/p>\n<ul>\n<li>CUDA version<\/li>\n<li>GPU name<\/li>\n<li>What job is running on your GPU<\/li>\n<li>Stats about your GPU: realtime heat, load and memory usage<\/li>\n<\/ul>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-testing-cuda\" aria-expanded=\"false\" aria-controls=\"slideme-testing-cuda\" class=\"slideme__heading slideme__trigger\">Testing CUDA<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-testing-cuda\" aria-hidden=\"true\"><p><\/p>\n<p>There are CUDA samples that you can download, compile and run for your version of CUDA. These samples have been compiled for you in your account. One of the samples probes your GPU and gives you detailed spec&#8217;s about it. You can try to run this sample code in your account (long output):<\/p>\n<p><code>\/home\/student\/cuda-samples\/Samples\/1_Utilities\/deviceQuery\/deviceQuery<\/code><\/p>\n<p><\/p>\n<p><strong>CUDA Sample code output:<\/strong><\/p>\n<p><\/p>\n<p>CUDA Device Query (Runtime API) version (CUDART static linking)<\/p>\n<p>Detected 1 CUDA Capable device(s)<\/p>\n<p>Device 0: &#8220;NVIDIA TITAN V&#8221;<br \/>\nCUDA Driver Version \/ Runtime Version 12.0 \/ 11.8<br \/>\nCUDA Capability Major\/Minor version number: 7.0<br \/>\nTotal amount of global memory: 12057 MBytes (12642746368 bytes)<br \/>\n(080) Multiprocessors, (064) CUDA Cores\/MP: 5120 CUDA Cores<br \/>\nGPU Max Clock rate: 1455 MHz (1.46 GHz)<br \/>\nMemory Clock rate: 850 Mhz<br \/>\nMemory Bus Width: 3072-bit<br \/>\nL2 Cache Size: 4718592 bytes<br \/>\nMaximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)<br \/>\nMaximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers<br \/>\nMaximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers<br \/>\nTotal amount of constant memory: 65536 bytes<br \/>\nTotal amount of shared memory per block: 49152 bytes<br \/>\nTotal shared memory per multiprocessor: 98304 bytes<br \/>\nTotal number of registers available per block: 65536<br \/>\nWarp size: 32<br \/>\nMaximum number of threads per multiprocessor: 2048<br \/>\nMaximum number of threads per block: 1024<br \/>\nMax dimension size of a thread block (x,y,z): (1024, 1024, 64)<br \/>\nMax dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)<br \/>\nMaximum memory pitch: 2147483647 bytes<br \/>\nTexture alignment: 512 bytes<br \/>\nConcurrent copy and kernel execution: Yes with 7 copy engine(s)<br \/>\nRun time limit on kernels: No<br \/>\nIntegrated GPU sharing Host Memory: No<br \/>\nSupport host page-locked memory mapping: Yes<br \/>\nAlignment requirement for Surfaces: Yes<br \/>\nDevice has ECC support: Disabled<br \/>\nDevice supports Unified Addressing (UVA): Yes<br \/>\nDevice supports Managed Memory: Yes<br \/>\nDevice supports Compute Preemption: Yes<br \/>\nSupports Cooperative Kernel Launch: Yes<br \/>\nSupports MultiDevice Co-op Kernel Launch: Yes<br \/>\nDevice PCI Domain ID \/ Bus ID \/ location ID: 0 \/ 0 \/ 5<br \/>\nCompute Mode:<br \/>\n&lt; Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) &gt;<\/p>\n<p>deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.0, CUDA Runtime Version = 11.8, NumDevs = 1<br \/>\nResult = PASS<\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-test-cudnn\" aria-expanded=\"false\" aria-controls=\"slideme-test-cudnn\" class=\"slideme__heading slideme__trigger\">Test CuDNN <\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-test-cudnn\" aria-hidden=\"true\"><p><\/p>\n<p>The sample code is already compiled in the &#8216;student&#8217; user. First change to the correct directory:<\/p>\n<p><code>cd \/home\/student\/cudnn-samples\/mnistCUDNN<\/code><\/p>\n<p>After that run the program:<\/p>\n<p><code>.\/mnistCUDNN<\/code><\/p>\n<p><\/p>\n<p><strong>CuDNN Sample code output:<\/strong><\/p>\n<p><\/p>\n<p>Executing: mnistCUDNN<br \/>\ncudnnGetVersion() : 8600 , CUDNN_VERSION from cudnn.h : 8600 (8.6.0)<br \/>\nHost compiler version : GCC 11.4.0<\/p>\n<p>There are 1 CUDA capable devices on your machine :<br \/>\ndevice 0 : sms 80 Capabilities 7.0, SmClock 1455.0 Mhz, MemSize (Mb) 12057, MemClock 850.0 Mhz, Ecc=0, boardGroupID=0<br \/>\nUsing device 0<\/p>\n<p>Testing single precision<br \/>\nLoading binary file data\/conv1.bin<br \/>\nLoading binary file data\/conv1.bias.bin<br \/>\nLoading binary file data\/conv2.bin<br \/>\nLoading binary file data\/conv2.bias.bin<br \/>\nLoading binary file data\/ip1.bin<br \/>\nLoading binary file data\/ip1.bias.bin<br \/>\nLoading binary file data\/ip2.bin<br \/>\nLoading binary file data\/ip2.bias.bin<br \/>\nLoading image data\/one_28x28.pgm<br \/>\nPerforming forward propagation &#8230;<br \/>\nTesting cudnnGetConvolutionForwardAlgorithm_v7 &#8230;<br \/>\n^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory<br \/>\n^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory<br \/>\n^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 0 memory<br \/>\n^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 178432 memory<br \/>\n^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 184784 memory<br \/>\n^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 2057744 memory<br \/>\n^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory<\/p>\n<p>&#8230;&#8230;..<\/p>\n<p>^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.179200 time requiring 64000 memory<br \/>\n^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory<br \/>\n^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory<br \/>\nResulting weights from Softmax:<br \/>\n0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000714 0.0000000 0.0000000 0.0000000 0.0000000<br \/>\nLoading image data\/five_28x28.pgm<br \/>\nPerforming forward propagation &#8230;<br \/>\nResulting weights from Softmax:<br \/>\n0.0000000 0.0000008 0.0000000 0.0000002 0.0000000 1.0000000 0.0000154 0.0000000 0.0000012 0.0000006<\/p>\n<p>Result of classification: 1 3 5<\/p>\n<p>Test passed!<\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-check-tensorflow\" aria-expanded=\"false\" aria-controls=\"slideme-check-tensorflow\" class=\"slideme__heading slideme__trigger\">Check Tensorflow<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-check-tensorflow\" aria-hidden=\"true\"><p><\/p>\n<p><\/p>\n<p>Tensorflow version:<\/p>\n<p><code>python3 -c 'import tensorflow as tf; print(tf.__version__)'<\/code><\/p>\n<p>Test tensorflow CPU support:<\/p>\n<pre class=\"clear-for-copy\"><code dir=\"ltr\"><span class=\"pln\">python3 <\/span><span class=\"pun\">-<\/span><span class=\"pln\">c <\/span><span class=\"str\">\"import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))\"<\/span><\/code><\/pre>\n<p>Test tensorflow GPU support:<\/p>\n<p><code dir=\"ltr\"><span class=\"pln\">python3 -c \"import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))\"<\/span><\/code><\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-ai-benchmark\" aria-expanded=\"false\" aria-controls=\"slideme-ai-benchmark\" class=\"slideme__heading slideme__trigger\">AI Benchmark<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-ai-benchmark\" aria-hidden=\"true\"><p><\/p>\n<p>This is useful to check if your tensorflow install is <em><strong>actually\u00a0<\/strong><\/em>using your GPU. It is not uncommon for tensorflow to run on CPU. This is one way to check if tensorflow is using GPU and to see if it is performing as expected.<\/p>\n<p>Test using AI benchmark (may take 20 minutes):<\/p>\n<p><code>python3 -c 'from ai_benchmark import AIBenchmark;benchmark = AIBenchmark();results = benchmark.run()'<\/code><\/p>\n<p>Then lookup the results on the Ai Benchmark link below. The AI website chart will tell you how well your GPU is performing. If its not performing well then you are not using the GPU!<\/p>\n<p>Here are the AI Benchmark scores: \u00a0<a href=\"http:\/\/ai-benchmark.com\/ranking_deeplearning.html\" target=\"_blank\" rel=\"noopener noreferrer\">http:\/\/ai-benchmark.com\/ranking_deeplearning.html<\/a><\/p>\n<p><\/p>\n<p><strong>Successful AI Benchmark output:<\/strong><\/p>\n<p><\/p>\n<p>&gt;&gt; AI-Benchmark-v.0.1.2<br \/>\n&gt;&gt; Let the AI Games begin..<\/p>\n<p>* TF Version: 2.9.1<br \/>\n* Platform: Linux-5.15.0-37-generic-x86_64-with-glibc2.35<br \/>\n* CPU: N\/A<br \/>\n* CPU RAM: 47 GB<br \/>\n* GPU\/0: NVIDIA GeForce RTX 2080 SUPER<br \/>\n* GPU RAM: 6.5 GB<br \/>\n* CUDA Version: 11.7<br \/>\n* CUDA Build: V11.7.64<\/p>\n<p>The benchmark is running&#8230;<br \/>\nThe tests might take up to 20 minutes<br \/>\nPlease don&#8217;t interrupt the script<\/p>\n<p>1\/19. MobileNet-V2<\/p>\n<p>1.1 &#8211; inference | batch=50, size=224&#215;224: 60.8 \u00b1 12.9 ms<br \/>\n1.2 &#8211; training | batch=50, size=224&#215;224: 203 \u00b1 2 ms<\/p>\n<p>2\/19. Inception-V3<\/p>\n<p>2.1 &#8211; inference | batch=20, size=346&#215;346: 63.6 \u00b1 2.6 ms<br \/>\n2.2 &#8211; training | batch=20, size=346&#215;346: 199 \u00b1 4 ms<\/p>\n<p>3\/19. Inception-V4<\/p>\n<p>3.1 &#8211; inference | batch=10, size=346&#215;346: 62.2 \u00b1 3.8 ms<br \/>\n3.2 &#8211; training | batch=10, size=346&#215;346: 216 \u00b1 1 ms<\/p>\n<p>4\/19. Inception-ResNet-V2<\/p>\n<p>4.1 &#8211; inference | batch=10, size=346&#215;346: 85.2 \u00b1 6.2 ms<br \/>\n4.2 &#8211; training | batch=8, size=346&#215;346: 232 \u00b1 8 ms<\/p>\n<p>5\/19. ResNet-V2-50<\/p>\n<p>5.1 &#8211; inference | batch=10, size=346&#215;346: 44.0 \u00b1 2.3 ms<br \/>\n5.2 &#8211; training | batch=10, size=346&#215;346: 127 \u00b1 1 ms<\/p>\n<p>6\/19. ResNet-V2-152<\/p>\n<p>6.1 &#8211; inference | batch=10, size=256&#215;256: 55.9 \u00b1 2.3 ms<br \/>\n6.2 &#8211; training | batch=10, size=256&#215;256: 178 \u00b1 3 ms<\/p>\n<p>7\/19. VGG-16<\/p>\n<p>7.1 &#8211; inference | batch=20, size=224&#215;224: 102 \u00b1 1 ms<br \/>\n7.2 &#8211; training | batch=2, size=224&#215;224: 166 \u00b1 1 ms<\/p>\n<p>8\/19. SRCNN 9-5-5<\/p>\n<p>8.1 &#8211; inference | batch=10, size=512&#215;512: 77.3 \u00b1 1.9 ms<br \/>\n8.2 &#8211; inference | batch=1, size=1536&#215;1536: 90.8 \u00b1 3.5 ms<br \/>\n8.3 &#8211; training | batch=10, size=512&#215;512: 253 \u00b1 4 ms<\/p>\n<p>9\/19. VGG-19 Super-Res<\/p>\n<p>9.1 &#8211; inference | batch=10, size=256&#215;256: 112 \u00b1 3 ms<br \/>\n9.2 &#8211; inference | batch=1, size=1024&#215;1024: 178 \u00b1 1 ms<br \/>\n9.3 &#8211; training | batch=10, size=224&#215;224: 266.1 \u00b1 0.8 ms<\/p>\n<p>10\/19. ResNet-SRGAN<\/p>\n<p>10.1 &#8211; inference | batch=10, size=512&#215;512: 112 \u00b1 1 ms<br \/>\n10.2 &#8211; inference | batch=1, size=1536&#215;1536: 99.7 \u00b1 1.9 ms<br \/>\n10.3 &#8211; training | batch=5, size=512&#215;512: 166 \u00b1 3 ms<\/p>\n<p>11\/19. ResNet-DPED<\/p>\n<p>11.1 &#8211; inference | batch=10, size=256&#215;256: 115.0 \u00b1 0.8 ms<br \/>\n11.2 &#8211; inference | batch=1, size=1024&#215;1024: 185 \u00b1 4 ms<br \/>\n11.3 &#8211; training | batch=15, size=128&#215;128: 185.6 \u00b1 0.7 ms<\/p>\n<p>12\/19. U-Net<\/p>\n<p>12.1 &#8211; inference | batch=4, size=512&#215;512: 216 \u00b1 1 ms<br \/>\n12.2 &#8211; inference | batch=1, size=1024&#215;1024: 209.7 \u00b1 1.0 ms<br \/>\n12.3 &#8211; training | batch=4, size=256&#215;256: 220 \u00b1 1 ms<\/p>\n<p>13\/19. Nvidia-SPADE<\/p>\n<p>13.1 &#8211; inference | batch=5, size=128&#215;128: 95.4 \u00b1 1.0 ms<br \/>\n13.2 &#8211; training | batch=1, size=128&#215;128: 175 \u00b1 1 ms<\/p>\n<p>14\/19. ICNet<\/p>\n<p>14.1 &#8211; inference | batch=5, size=1024&#215;1536: 233 \u00b1 16 ms<br \/>\n14.2 &#8211; training | batch=10, size=1024&#215;1536: 776 \u00b1 35 ms<\/p>\n<p>15\/19. PSPNet<\/p>\n<p>15.1 &#8211; inference | batch=5, size=720&#215;720: 415 \u00b1 7 ms<br \/>\n15.2 &#8211; training | batch=1, size=512&#215;512: 156 \u00b1 2 ms<\/p>\n<p>16\/19. DeepLab<\/p>\n<p>16.1 &#8211; inference | batch=2, size=512&#215;512: 114 \u00b1 2 ms<br \/>\n16.2 &#8211; training | batch=1, size=384&#215;384: 127 \u00b1 4 ms<\/p>\n<p>17\/19. Pixel-RNN<\/p>\n<p>17.1 &#8211; inference | batch=50, size=64&#215;64: 1019 \u00b1 33 ms<br \/>\n17.2 &#8211; training | batch=10, size=64&#215;64: 5049 \u00b1 216 ms<\/p>\n<p>18\/19. LSTM-Sentiment<\/p>\n<p>18.1 &#8211; inference | batch=100, size=1024&#215;300: 763 \u00b1 46 ms<br \/>\n18.2 &#8211; training | batch=10, size=1024&#215;300: 2314 \u00b1 142 ms<\/p>\n<p>19\/19. GNMT-Translation<\/p>\n<p>19.1 &#8211; inference | batch=1, size=1&#215;20: 271 \u00b1 14 ms<\/p>\n<p>Device Inference Score: 10318<br \/>\nDevice Training Score: 10631<br \/>\n<strong>Device AI Score: 20949<\/strong><\/p>\n<p>For more information and results, please visit http:\/\/ai-benchmark.com\/alpha<\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-check-pytorch\" aria-expanded=\"false\" aria-controls=\"slideme-check-pytorch\" class=\"slideme__heading slideme__trigger\">Check Pytorch<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-check-pytorch\" aria-hidden=\"true\"><p><\/p>\n<p>Version check:<\/p>\n<p><code>python3 -c \"import torch; print(torch.__version__)\"<\/code><\/p>\n<p>Test code:<\/p>\n<p><code>python3 -c 'import torch;x=torch.rand(5,3);print(x)'<\/code><\/p>\n<p><\/p><\/dd><dl><\/div>\n<h2><a id=\"troubleshooting\"><\/a>Troubleshooting<\/h2>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-x2go-connection-failed-errors\" aria-expanded=\"false\" aria-controls=\"slideme-x2go-connection-failed-errors\" class=\"slideme__heading slideme__trigger\">x2go connection failed errors<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-x2go-connection-failed-errors\" aria-hidden=\"true\"><p><\/p>\n<p>If you cannot login to your VM using x2go the most common issue is that your VM ran out of space. In this case you can:<\/p>\n<ol>\n<li>login to your VM using the ssh-terminal (<a href=\"https:\/\/carleton.ca\/scs\/2021\/connecting-to-an-openstack-instance-using-ssh-putty\/\">putty for Windows<\/a>)<\/li>\n<li>free up space by deleting files &amp; folders (Identify large folders using: du -s -h *) (filesystem space: df -h)<\/li>\n<li>try to login again using x2go<\/li>\n<\/ol>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-extend-the-lvm-filesystem\" aria-expanded=\"false\" aria-controls=\"slideme-extend-the-lvm-filesystem\" class=\"slideme__heading slideme__trigger\">Extend the LVM filesystem<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-extend-the-lvm-filesystem\" aria-hidden=\"true\"><p><\/p>\n<p>Usually the file system is using all the allocated space assigned to it. There is a simple script that can expand the filesystem to use all the allocated space. In case you need to expand the space you can run the provided script:<\/p>\n<p><code>sudo \/home\/student\/extend-lvm\/extend-lvm.sh \/dev\/vda<\/code><\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-numa-node-read-from-sysfs\" aria-expanded=\"false\" aria-controls=\"slideme-numa-node-read-from-sysfs\" class=\"slideme__heading slideme__trigger\">NUMA node read from SysFS...<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-numa-node-read-from-sysfs\" aria-hidden=\"true\"><p><\/p>\n<p>Because this is &#8216;bleeding&#8217; edge software you can expect some software issues. This one in particular is concerning NUMA node error reading from memory. If you want to fix this particular &#8216;warning&#8217; there is a bash script you can run:<\/p>\n<p><code>sudo \/home\/student\/os_scripts\/numa-node-fix.bash<\/code><\/p>\n<p><strong>Reference<\/strong>: <a href=\"https:\/\/gist.github.com\/zrruziev\/b93e1292bf2ee39284f834ec7397ee9f\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/gist.github.com\/zrruziev\/b93e1292bf2ee39284f834ec7397ee9f<\/a><\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-need-to-use-another-version-of-gcc\" aria-expanded=\"false\" aria-controls=\"slideme-need-to-use-another-version-of-gcc\" class=\"slideme__heading slideme__trigger\">Need to use another version of gcc?<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-need-to-use-another-version-of-gcc\" aria-hidden=\"true\"><p><\/p>\n<p>There are two versions of gcc installed on this VM. You can use the update-alternatives to switch between the versions:<\/p>\n<pre>  sudo update-alternatives --config gcc<\/pre>\n<p><\/p>\n<p><\/p><\/dd><dl><\/div>\n<div class=\"slideme\"><dl class=\"slideme__list\"><dt class=\"slideme__term\"><a href=\"#slideme-tensorflow-error-sysfs-had-negative-value-1\" aria-expanded=\"false\" aria-controls=\"slideme-tensorflow-error-sysfs-had-negative-value-1\" class=\"slideme__heading slideme__trigger\">Tensorflow error: SysFS had negative value (-1)<\/a><\/dt><dd class=\"slideme__description\" id=\"slideme-tensorflow-error-sysfs-had-negative-value-1\" aria-hidden=\"true\"><p><\/p>\n<pre>  I tensorflow\/stream_executor\/cuda\/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<\/pre>\n<p>Add this environment variable to \/etc\/environment<\/p>\n<pre>  TF_ENABLE_ONEDNN_OPTS=0<\/pre>\n<p>Find the NVIDIA GPU ID:<\/p>\n<p>lspci -D | grep NVIDIA<\/p>\n<p>In my case it was<\/p>\n<p>0000:00:05.0<\/p>\n<p>Then add a line to the crontab (note YOUR device id):<\/p>\n<p>sudo crontab -e<\/p>\n<p>Then add this line:<\/p>\n<p>@reboot (echo 0 | tee -a &#8220;\/sys\/bus\/pci\/devices\/0000:00:05.0\/numa_node&#8221;)<\/p>\n<p>After that reboot and try the tensorflow GPU command again. That should fix this issue.<\/p>\n<p><\/p><\/dd><dl><\/div>\n","protected":false},"excerpt":{"rendered":"<p>GPU VM Account Details Testing the GPU VM software Troubleshooting This page will show you how to test your virtual machine&#8217;s GPU and access and test the software that is pre-installed on it. GPU VM Account Details Testing the GPU VM software Here are some helpful tests to see if your software is running correctly. [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":0,"parent":6535,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","_mi_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":"","_links_to":"","_links_to_target":""},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>DEPRECATED - SCS GPU VM 2023-4 - School of Computer Science<\/title>\n<meta name=\"description\" content=\"GPU VM Account Details Testing the GPU VM software Troubleshooting This page will show you how to test your virtual machine&#039;s GPU and access and test the\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/\",\"url\":\"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/\",\"name\":\"DEPRECATED - SCS GPU VM 2023-4 - School of Computer Science\",\"isPartOf\":{\"@id\":\"https:\/\/carleton.ca\/scs\/#website\"},\"datePublished\":\"2023-08-09T17:49:11+00:00\",\"dateModified\":\"2025-07-25T13:15:44+00:00\",\"description\":\"GPU VM Account Details Testing the GPU VM software Troubleshooting This page will show you how to test your virtual machine's GPU and access and test the\",\"breadcrumb\":{\"@id\":\"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/carleton.ca\/scs\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Technical Support\",\"item\":\"https:\/\/carleton.ca\/scs\/tech-support\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"DEPRECATED &#8211; SCS GPU VM 2023-4\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/carleton.ca\/scs\/#website\",\"url\":\"https:\/\/carleton.ca\/scs\/\",\"name\":\"School of Computer Science\",\"description\":\"Carleton University\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/carleton.ca\/scs\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"DEPRECATED - SCS GPU VM 2023-4 - School of Computer Science","description":"GPU VM Account Details Testing the GPU VM software Troubleshooting This page will show you how to test your virtual machine's GPU and access and test the","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/","twitter_misc":{"Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/","url":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/","name":"DEPRECATED - SCS GPU VM 2023-4 - School of Computer Science","isPartOf":{"@id":"https:\/\/carleton.ca\/scs\/#website"},"datePublished":"2023-08-09T17:49:11+00:00","dateModified":"2025-07-25T13:15:44+00:00","description":"GPU VM Account Details Testing the GPU VM software Troubleshooting This page will show you how to test your virtual machine's GPU and access and test the","breadcrumb":{"@id":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/carleton.ca\/scs\/"},{"@type":"ListItem","position":2,"name":"Technical Support","item":"https:\/\/carleton.ca\/scs\/tech-support\/"},{"@type":"ListItem","position":3,"name":"DEPRECATED &#8211; SCS GPU VM 2023-4"}]},{"@type":"WebSite","@id":"https:\/\/carleton.ca\/scs\/#website","url":"https:\/\/carleton.ca\/scs\/","name":"School of Computer Science","description":"Carleton University","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/carleton.ca\/scs\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"}]}},"acf":{"banner_image_type":"upload","banner_button":"no","banner_uploaded_image":{"ID":18486,"id":18486,"title":"GPU Image","filename":"GPU-1600x700-1.jpg","filesize":246203,"url":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1.jpg","link":"https:\/\/carleton.ca\/scs\/tech-support\/scs-gpu-vm-fall-2023\/gpu-1600x700\/","alt":"GPU Image","author":"7","description":"GPU Image","caption":"","name":"gpu-1600x700","status":"inherit","uploaded_to":17254,"date":"2024-04-22 19:45:52","modified":"2024-04-22 19:46:12","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/carleton.ca\/scs\/wp\/wp-includes\/images\/media\/default.png","width":1600,"height":700,"sizes":{"thumbnail":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-160x70.jpg","thumbnail-width":160,"thumbnail-height":70,"medium":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-240x105.jpg","medium-width":240,"medium-height":105,"medium_large":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-768x336.jpg","medium_large-width":768,"medium_large-height":336,"large":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-400x175.jpg","large-width":400,"large-height":175,"gallery-thumb":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-300x230.jpg","gallery-thumb-width":300,"gallery-thumb-height":230,"1536x1536":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-1536x672.jpg","1536x1536-width":1536,"1536x1536-height":672,"2048x2048":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1.jpg","2048x2048-width":1600,"2048x2048-height":700,"banner":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1.jpg","banner-width":1600,"banner-height":700,"people":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-200x200.jpg","people-width":200,"people-height":200,"post-thumb":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-300x230.jpg","post-thumb-width":300,"post-thumb-height":230,"rotator-image":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-800x600.jpg","rotator-image-width":800,"rotator-image-height":600,"video-thumb":"https:\/\/carleton.ca\/scs\/wp-content\/uploads\/GPU-1600x700-1-360x158.jpg","video-thumb-width":360,"video-thumb-height":158}},"banner_opacity":"default"},"_links":{"self":[{"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/pages\/17254"}],"collection":[{"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/comments?post=17254"}],"version-history":[{"count":3,"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/pages\/17254\/revisions"}],"predecessor-version":[{"id":21906,"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/pages\/17254\/revisions\/21906"}],"up":[{"embeddable":true,"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/pages\/6535"}],"wp:attachment":[{"href":"https:\/\/carleton.ca\/scs\/wp-json\/wp\/v2\/media?parent=17254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}