NVIDIA drivers on Linux

This page installs the NVIDIA driver and the CUDA toolkit on a GPU server from NVIDIA's own package repository, on Ubuntu 24.04, Debian 13 or AlmaLinux 10.

The driver is built as a kernel module through DKMS, which is why the compiler and the kernel headers are needed. DKMS rebuilds the module by itself after a kernel update.

nvidia-open is the kernel module NVIDIA recommends for these cards. cuda-toolkit is only needed to build CUDA code, such as llama.cpp. Ollama and PyTorch ship their own CUDA runtime and need only the driver, so leave cuda-toolkit out if that is all you run. The tested versions were driver 615.71.09 and CUDA 13.4; the repository hands you whatever is current.

Secure Boot is off on the GPU servers, so the self-built module loads without signing. A fresh install loads the nouveau driver for the cards; the NVIDIA packages blacklist it, and it is gone after the reboot at the end.

Ubuntu 24.04

Update first. If the upgrade pulls in a new kernel, the reboot makes it the running one, so the headers you install next match it.

apt update && apt full-upgrade -y
reboot

Install the build tools and headers, add NVIDIA's repository and install the driver and toolkit:

apt install -y build-essential linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt install -y nvidia-open cuda-toolkit
reboot

For another Ubuntu release, replace ubuntu2404 in the URL (ubuntu2204 for 22.04, ubuntu2604 for 26.04). Continue with Check that it works.

Debian 13

Debian's minimal install does not include curl; install it with the rest if you use it later (the Ollama installer does).

apt update && apt full-upgrade -y
reboot

apt install -y build-essential linux-headers-$(uname -r) curl
wget https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt install -y nvidia-open cuda-toolkit
reboot

For Debian 12, replace debian13 with debian12 in the URL.

AlmaLinux 10

The DKMS package comes from EPEL, and EPEL needs the CRB repository.

dnf upgrade -y
reboot

dnf install -y epel-release
dnf config-manager --set-enabled crb
dnf install -y kernel-devel-$(uname -r) kernel-headers gcc make dkms
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel10/x86_64/cuda-rhel10.repo
dnf install -y nvidia-open cuda-toolkit
reboot

For AlmaLinux 9, use rhel9 in the repository URL.

Check that it works

After the last reboot, nvidia-smi lists the cards. This is a GPU-2 server:

$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.71.09              KMD Version: 615.71.09     CUDA UMD Version: 13.4     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                On  |   00000000:01:00.0 Off |                  Off |
| 33%   28C    P0             51W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                On  |   00000000:02:00.0 Off |                  Off |
| 33%   30C    P0             53W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Off in the ECC column is the default, see ECC memory. The CUDA compiler is installed under /usr/local/cuda and is not on the PATH by default:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
nvcc --version

When nvidia-smi fails

• NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver: the module is not loaded. Run dkms status. It should list nvidia/<version>, <kernel>, x86_64: installed for the kernel in uname -r. If the kernel is missing from the list, the headers for it were not installed: install them (linux-headers-$(uname -r) or kernel-devel-$(uname -r)), run dkms autoinstall and reboot.
• The server does not come back after a reboot: open the Console to see where the boot stopped.