OBS NVENC Error: Causes and Fixes for Encoder Failures

An NVENC error in OBS means the program asked the NVIDIA hardware encoder to start and the request was refused or failed. The usual causes are an outdated or broken graphics driver, encoder settings the GPU does not accept, OBS selecting the wrong graphics adapter on a laptop, too many simultaneous encode sessions, or not enough video memory. The fastest fixes are to update the driver, lower the output resolution or disable advanced encoder options, and select the correct GPU.

The error text often includes a number, such as “Failed to open NVENC codec” followed by a code. That number is the useful clue, because it points to a category of failure. This guide explains how to read the message, then works through the fixes in a safe order, from the quickest checks to the more involved ones.

What an NVENC Error Actually Tells You

NVENC is the dedicated video encoding block on NVIDIA GPUs. OBS opens an encoding session on it, sends frames, and receives compressed video back. When the session cannot open, OBS shows a message in a pop-up or in the log, and streaming or recording will not start.

The messages usually carry a short name and a number from NVIDIA’s encoder interface. Commonly cited meanings include the following.

• Out of memory (10): the GPU could not allocate enough video memory for the session.
• Unsupported device (2): the selected GPU cannot use the chosen encoder or codec.
• Unsupported parameter (12): one of the settings, such as resolution, bitrate, B-frames or look-ahead, is outside what the encoder accepts.
• Invalid version (15): the driver and the software disagree on the interface version, which usually means an old driver.
• Incompatible client key (21): a version mismatch between OBS and the driver library.

Open the OBS log from the Help menu and look for the line near the error. It often names the adapter and the exact code, which saves guesswork.

Error Causes at a Glance

Step-by-Step Fixes

Step 1: Update the graphics driver and OBS

Many NVENC errors come from a driver that is older than the OBS build expects. Download the current driver from NVIDIA’s official site or through the NVIDIA app, install it, and restart the computer. Also update OBS to its current release, since mismatches between old OBS builds and new drivers can trigger interface errors. If the problem began right after a driver update, the opposite may apply, and you can try the previous driver release.

Step 2: Reset encoder settings to safe values

In Settings, open Output and switch to Advanced mode. Under the encoder options, remove anything unusual: set the preset to a standard quality level, turn off look-ahead and psycho visual tuning, set B-frames to a low value, and use a regular rate control like CBR for streaming. Then match your base canvas and output resolution to values the encoder supports. Extremely large outputs or odd resolutions can trigger an unsupported parameter error.

Step 3: Select the right GPU

If you have more than one GPU, OBS may be using the wrong one. In the Advanced output settings, there is a GPU index field. Try the other number if the first one fails. On a laptop, open Windows Settings, go to System, then Display, then Graphics, add the OBS executable, open its options and choose High performance so Windows runs it on the NVIDIA GPU. Restart OBS afterward.

Step 4: Check encoder session limits and VRAM

Consumer GPUs have a limit on how many video encode sessions can run at the same time, and NVIDIA has changed that limit across driver releases. If you run OBS along with another recorder, a second OBS instance, a cloud gaming client or a streaming plugin that uses NVENC, you may reach the cap. Close those programs, then start OBS again. Also watch video memory: a game running at high texture settings plus a browser, a video editor and OBS can leave too little memory for the encoder to open.

Laptop-Specific Problems

Laptops with both integrated and NVIDIA graphics cause a large share of NVENC reports. The display may be driven by the integrated GPU while the game runs on the NVIDIA GPU, so OBS ends up on the wrong adapter, and game capture can fail or show a black screen along with encoder errors.

The fix is to make OBS and the game use the same GPU, as in Step 3. Some laptops also have a hardware switch or BIOS option that controls whether the NVIDIA GPU drives the display directly. Changing that is a firmware setting, so check your laptop maker’s documentation first and only change it if you understand the effect.

If you use the NVIDIA Control Panel, the global setting for the preferred graphics processor can be set to the high-performance NVIDIA processor, but a per-app setting in Windows is usually enough.

When Switching Encoders Is the Right Move

If the NVENC error persists, you can switch encoders in Settings, Output, to keep recording while you investigate. Options vary by hardware.

• x264 (software): works on any computer but uses the CPU heavily, so lower the preset to a faster one if you see dropped frames.
• Another hardware encoder: if the PC has an Intel or AMD GPU with its own encoder, OBS may list it as an alternative.
• A different NVENC codec: H.264 is widely supported, while HEVC and AV1 require compatible GPU generations. AV1 hardware encoding needs a newer card, so an older one will show an unsupported device message.

Switching is a workaround and not a repair, but it keeps a stream or recording session alive.

Deeper Fixes If Nothing Works

If the earlier steps fail, try these in order, and stop once the error is gone.

1. Do a clean driver install by choosing the clean installation option in the NVIDIA installer.
2. Run OBS as administrator once to check whether a permissions issue is blocking the capture.
3. Disable overlays and overclocking tools temporarily, since some overlays hook the same graphics pipeline.
4. Rename the OBS profile folder to create a fresh configuration, which removes the risk of a corrupted setting.
5. Reinstall OBS if the program files appear damaged.

Third-party driver removal tools exist, but they change system files, so use them only if you are comfortable with the process and have created a restore point first.

Preventing the Error From Returning

Once OBS works again, a few habits keep it that way. Keep a note of the driver version that worked, so you can return to it if a later update causes trouble. Avoid running several capture or recording programs at the same time, since each may open its own encoder session. Finally, save a copy of your OBS profile and scene collection after a working setup, so you can restore known-good settings quickly.

Related guides

• What Does Manage Gpu Performance Counters Mean On Nvidia Hardware
• Low Profile Nvidia Graphics Cards For Small Form Factor Pcs
• Nvidia App Error Your System Configuration Is Not Supported

Frequently Asked Questions

What does NVENC error 12 mean in OBS?

Error 12 is commonly listed as an unsupported parameter, meaning one of your encoder settings is outside what the hardware accepts. Reset the encoder to default values, turn off look-ahead and B-frames, and make sure your output resolution is reasonable. Updating the driver is a good second step.

Why does OBS say out of memory when my GPU has plenty of VRAM?

The message can appear when other programs hold video memory or when the encoder session limit is reached, not only when the memory is truly full. Close other GPU-heavy programs and extra recorders, then try again. If it keeps happening, lower the game’s texture settings to leave room for the encoder.

Does NVENC work on any NVIDIA GPU?

Most GeForce cards from the last several generations include a hardware encoder, but the supported codecs depend on the generation. Very old cards may lack H.264 or HEVC support, and AV1 needs a recent generation. Check the manufacturer’s current documentation for your exact model.

Is software encoding a good fallback?

Yes, x264 works on almost any computer and is a reliable fallback. It uses the processor instead of the GPU, so pick a faster preset if your CPU struggles, and expect higher CPU load while gaming.