Nvidia/CUDA Docker Image: Tags, Sizes and the Setup Guide

Nvidia/CUDA docker image looks simple until you see the tag list and realise there are hundreds, differing by CUDA version, flavour, cuDNN inclusion and base OS, with no obvious guidance on which one you want. Pick wrong and you either ship a 6GB image that needed to be 300MB, or you get a runtime error at 2am because nvcc is not in the container. This page decodes the tag scheme, gives you the commands, and shows the multi-stage pattern that fixes the size problem. Copy what you need and get back to your Dockerfile.

Understanding the Tag Scheme

Every tag follows one pattern, and once you can read it the hundreds of options collapse into a small decision tree. The format is {cuda_version}-{flavour}-{os}, optionally with cudnn inserted before the flavour.

The Three Flavours: base, runtime, devel

This is the choice that matters most and the one people get wrong.

The rule most people need: you almost certainly want runtime, not devel. If you are pip-installing PyTorch and running a script, nvcc never gets called. Shipping devel to production means shipping a compiler nobody will use, for 3GB.

Use devel in your build stage only. The last section shows how.

Reading a Tag Correctly

Take 12.8.0-cudnn-devel-ubuntu22.04 and decompose it: CUDA 12.8.0, includes cuDNN, has the full toolkit, based on Ubuntu 22.04. Every element is there once you know to look.

Two practical notes. The CUDA version must match what your code needs — and remember that Blackwell cards require CUDA 12.8 or newer, so 12.4.0 images will fail on an RTX 50-series GPU with a “no kernel image available” error that never mentions CUDA versions.

The OS choice matters less than people think. Ubuntu 22.04 is the safest default with the widest package availability. Ubuntu 24.04 works and occasionally trips older Python wheels. Pin the OS explicitly rather than taking a floating tag — a base OS changing underneath you is a bad afternoon.

Where to Pull From: Docker Hub or NGC

Two registries, and the difference has bitten people.

# Docker Hub - familiar, but tag availability has changed over time
docker pull nvidia/cuda:12.8.0-runtime-ubuntu22.04

# NGC - Nvidia's own registry, the canonical source
docker pull nvcr.io/nvidia/cuda:12.8.0-runtime-ubuntu22.04

NGC is the authoritative registry and keeps a longer tail of versions. Docker Hub has had tags removed and lifecycle policies applied over the years, which means a Dockerfile that built fine in 2023 can fail today with a manifest-not-found error on a tag that simply no longer exists there.

If you are pinning a specific older CUDA for a reproducible build, use nvcr.io. If you are on something current, either works.

Getting It Running

Less setup than people expect, provided one host prerequisite is in place. That prerequisite is where every first attempt fails.

Host Prerequisites: Driver and Container Toolkit

Two things on the host, and only two. An Nvidia driver, and the NVIDIA Container Toolkit.

You do not need CUDA on the host. This surprises people and it is the best property of the whole approach. The container carries the toolkit; the host provides only the driver, which is exposed into the container by the runtime.

# Ubuntu: install the container toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Driver version still governs which CUDA you can run — a 525 driver will not run a CUDA 12.8 container. Drivers are backward compatible, so a newer host driver is nearly always safe.

Your First Pull and Run

Verify the whole chain in one command:

docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu22.04 nvidia-smi

If that prints your GPU, everything works — driver, toolkit, runtime, container. If it does not, the troubleshooting section covers the two errors you will see.

The --gpus all flag is what exposes the hardware. Omit it and the container runs with no GPU at all, silently, which produces confusing CPU-only behaviour rather than an error.

# Specific GPUs only
docker run --rm --gpus '"device=0,1"' nvidia/cuda:12.8.0-base-ubuntu22.04 nvidia-smi

Dockerfile and Compose Patterns

The pattern most projects need:

FROM nvidia/cuda:12.8.0-runtime-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
      python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

COPY . /app
WORKDIR /app
CMD ["python3", "main.py"]

For Compose, GPU access uses the device reservation syntax rather than a flag:

services:
  app:
    build: .
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Note --no-cache-dir on the pip line. Without it, pip’s cache adds a gigabyte or more of files nobody will ever read.

Pros and Cons, and the Image Size Problem

Containerising CUDA is the right call for almost everyone, and it has one genuinely annoying property that deserves its own section because the fix is not obvious.

Why devel Images Are Enormous

A cudnn-devel image can exceed 8GB before you add a single line of your own code. That is nvcc, every header, every static library, the profiler, the debugger and the full cuDNN distribution.

None of it runs in production. It exists to compile things. But because the obvious Dockerfile starts FROM ...devel and ends with a CMD, all of it ships — to every node, on every deploy, forever.

The cost is real: slower pulls, larger registries, longer cold starts, and a bigger attack surface in the form of a compiler sitting in your production container.

Multi-Stage Builds That Fix It

Compile in devel, ship in runtime or base. This single pattern routinely turns a 6GB image into 300MB.

# Stage 1: build
FROM nvidia/cuda:12.8.0-devel-ubuntu22.04 AS builder
WORKDIR /build
COPY . .
RUN nvcc -O3 -o myapp main.cu

# Stage 2: ship
FROM nvidia/cuda:12.8.0-base-ubuntu22.04
COPY --from=builder /build/myapp /usr/local/bin/myapp
CMD ["myapp"]

The final image contains your binary and the CUDA runtime libraries it links against. No compiler, no headers, no profiler.

For Python projects that do not compile anything, the equivalent win is simply starting from runtime instead of devel — which most people do not, because devel is what the tutorial they copied used.

Pros and Cons Summary

The verdict for anyone maintaining more than one CUDA project: this is the answer, and the version conflicts you have been fighting simply stop existing.

Troubleshooting the Two Errors You Will See

Nearly every container GPU problem is one of these, and both have a specific cause rather than a vague one.

“could not select device driver with capabilities: [[gpu]]”

The container toolkit is not installed or Docker was not reconfigured after installing it. This is the first-attempt error and it means the host, not the image.

# Verify the toolkit is present
nvidia-ctk --version

# Reconfigure and restart - the step people skip
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The restart is not optional. Installing the toolkit without restarting the Docker daemon leaves it unaware the runtime exists, and the error message does not hint at this at all.

Driver and CUDA Mismatch Inside the Container

You pulled a CUDA 12.8 image onto a host with a 525 driver, and you get “CUDA driver version is insufficient for CUDA runtime version”. The container’s CUDA is newer than the host driver supports.

Two fixes: update the host driver, or use an older image tag matching what the driver supports. On a shared machine where you cannot touch the driver, the second is your only option — check the ceiling with nvidia-smi and pull a tag beneath it.

The related trap: an RTX 50-series card with a CUDA 12.4 image throws “no kernel image is available for execution on the device”. That is not a driver problem — Blackwell needs CUDA 12.8+, and updating the driver again will not help. Change the image tag.

The Hardware Underneath

Once the container runs, VRAM becomes your constraint permanently and no image tag changes it. Roughly: 8GB handles small vision models and modest batch sizes, 12GB is the practical floor for comfortable work, 16GB opens up 7B language models, 24GB is where you stop fighting the machine.

Containers add negligible overhead — GPU access is passed through, not virtualised — so whatever your card does bare metal, it does in Docker. If you are hitting out-of-memory errors during model load rather than during compute, that is not a configuration problem. If a VRAM ceiling is what actually blocks your work, compare what 16GB and 24GB cards cost before spending more evenings on image tags.

See More: 

• GTX 1650 vs RTX 3050
• Nvidia DIGITS
• Nvidia cuDNN
• Radeon RX 9070 XT vs RTX 5090
• PNY GeForce RTX 5080 review

Conclusion: Using the Nvidia/CUDA Docker Image Well

The nvidia/cuda docker image ecosystem is straightforward once three things are clear. The tag format is {version}-{flavour}-{os} and you almost certainly want runtime rather than devel. The host needs a driver and the container toolkit and nothing else — no CUDA install at all. And if you compile anything, use a multi-stage build to ship the binary without the compiler, which turns 6GB into 300MB for the cost of four extra lines.

Pin your tags explicitly, prefer nvcr.io over Docker Hub for anything you need to build reproducibly in two years, and remember that Blackwell requires CUDA 12.8 or newer regardless of what your driver reports. Bookmark the flavour table above — it is the decision you will make again on the next project.