View Full Version : NVENCForge v1.33 / GUI v1.12 - VMAF Auto-CQ, AV1/H.265, Split/Join DaVinci ready


Burnersen
29th August 2026, 23:48
Hi everyone,

I built a batch encoder for my own video library over the last
few months. In its current state, it handles about 99% of my
encoding work to my full satisfaction, so I thought it might
be useful to others here as well.

NVENCForge is a free, source-available Windows batch encoder
built around FFmpeg and NVENC. The CLI executable handles the actual
encoding, but I recommend using the GUI (NVENCForgeGUI) for daily
use. It makes batch queues, watched folders, audio track selection,
and profiles a lot easier than typing CLI flags.

Why another encoder?

Every batch tool I tried forced me to set a single fixed CQ or CRF
value for an entire folder. That number is always wrong somewhere:
too generous for simple material, too tight for complex scenes.

NVENCForge measures quality per file instead of guessing. Each video
gets a quick analysis first to find the cheapest CQ that still hits
the quality target (default: VMAF 97, tolerance 0.5). On a normal
movie, this check takes under a minute.

How Auto-CQ works:

Reads the bitrate profile from the container without decoding
to place three 8-second sample windows on demanding scenes.
Encodes those samples at two anchor CQ values and scores them
with libvmaf against the source using identical filter chains and fps.
Derives the target CQ and runs one more quick encode to confirm it.
Detects quality saturation on pre-compressed sources (steps down
to cheaper CQs) and sets a spending cap on grainy video (default:
max 45% of source bitrate).
Testing showed window length matters more than window count:
3x8s windows gave identical CQ picks as 4x8s while cutting analysis
time by 22-32%.


Main features:

Probes files first: skips or remuxes videos that are already
efficient. Per-file bitrate caps keep re-encodes smaller than the source.
GPU pipeline: NVDEC decoding and CUDA Lanczos scaling (files over
50 Mbit/s automatically process on CPU).
Passes through HDR10 and HLG color tags plus static mastering metadata.
Auto-crop for letterboxed material, with a `-cropcheck` preview mode.
Supports H.265 (default), AV1 (`av1_nvenc` on RTX 40+), MP4 and 8-bit
compatibility modes, and a `-cpu` mode (libx265 / SVT-AV1).
DaVinci Resolve helper: stream-copies video while converting
unsupported audio (DTS, TrueHD, etc.) to AAC and cleaning subtitles.
Includes 1:1 lossless split and join queues.
Validates outputs before moving originals to an `originals/` folder.
Interrupted encodes leave a playable `.preview.mkv` file.


The GUI & Getting Started

To get started, simply download the latest release of NVENCForgeGUI
and launch it. In the top right corner, click "NVENCForge - check github...".
This will automatically download the latest NVENCForge engine, initialize
all required config files, and set up the tools folder next to the executable.

The GUI supports drag and drop, up to 3 parallel jobs, live progress
(speed, ETA, projected output size), and folder watching. It also has
a visual track chooser, auto-generated config docs, and profile support.
It does not encode anything itself - it drives the CLI executable
via a JSON event stream.

Requirements:
Windows 10/11 x64. NVIDIA GPU (Pascal or newer for HEVC 10-bit, RTX 40+
for AV1). CPU mode and lossless tools work on any machine.

Known limitations:

Sampled VMAF on short windows is an estimate for the full file
and inherits standard VMAF blind spots (such as color banding).
NVENC `p5` matches `libx265 fast` in quality per byte at 4x speed,
but a slow x265 preset will pull ahead in compression efficiency
(~1 VMAF gain for triple the render time).
Windows-only, unsigned binary (SmartScreen warnings will appear).
PolyForm Noncommercial 1.0.0 license: free to use for personal
and non-commercial purposes (source available, no commercial resale).


Links:
GUI: https://github.com/burnersen/NVENCForgeGUI
CLI Engine: https://github.com/burnersen/NVENCForge

I would be very interested in your thoughts on the measurement logic:
Is 3 windows of 8 seconds a defensible sample size for a two-hour film,
or do you have material where scene placement like this misses hard scenes?

I don't check in here every day, but I do try to reply within a
reasonable time.

Cheers,
burnersen

Z2697
30th August 2026, 01:09
files over
50 Mbit/s automatically process on CPU

Why? There doesn't seem to ba a good reason

In the source file

Extreme-bitrate video has been known to crash display drivers


Are you sure this is not just some LLM hallucination? I've never encountered it.

Z2697
30th August 2026, 01:12
The window selection relies on the source material being a quality-based VBR, that's not always the case.

Burnersen
30th August 2026, 06:32
Thanks for taking a look at the code and testing the project, Z2697!

Regarding the 50 Mbit/s GPU threshold:
That threshold came from real-world testing on my machine. When feeding extremely high-bitrate 4K HDR remuxes into NVDEC while simultaneously performing CUDA Lanczos downscaling, I experienced occasional NVDEC hangs / TDRs (driver resets).

To prevent an overnight batch encode from stalling, I added a conservative safety limit to fall back to CPU decoding for heavy files. That value is fully configurable in the config settings if your GPU handles heavy NVDEC pipelines without hiccups. The comment in the source is indeed a bit overly dramatic, ...

Regarding the VBR assumption for sample windows:
Spot on observation. The heuristic looks at bitrate peaks, which correlates well with scene complexity on standard VBR/CRF sources (like most BD/WEB-DL rips).

On flat CBR or near-constant bitrate material, frame sizes don't reflect visual complexity, so the sample windows end up placed semi-randomly. The VMAF target evaluation still works on those samples, but it loses the "guaranteed hardest scene" advantage.

Do you have any suggestions for a lightweight heuristic to spot complex scenes without needing a full decode pass first?

tebasuna51
30th August 2026, 07:32
W10 Pro v1607
Ryzen 5 1600X
NVIDIA GeForce GT 1030 (without HVEC encoder)

Can't use your software:

---
OK FFmpeg auto-download complete: ffmpeg.exe + ffprobe.exe installed.
INFO CPU mode (from configuration: encoder=cpu): encoding H.265 with libx265 — no Nvidia card needed, but much slower than a GPU.
INFO Keep-source enabled via configuration: originals stay exactly where they are.
INFO Checking CPU encoder (libx265 H.265 10-bit)...
ERROR The CPU encoder is not available in this FFmpeg build.
ERROR Delete ffmpeg.exe next to NVENCForge.exe and restart to download a complete build.
Detail: main.go: checkCPUEncoderCapability: libx265 dummy encode failed: exit status 0xc0000139 |

After delete ffmpeg.exe and restart I obtain the same.

Burnersen
30th August 2026, 08:25
Hi tebasuna51, thanks for testing and sharing the detailed log!

The exit code 0xc0000139 (STATUS_ENTRYPOINT_NOT_FOUND) means the auto-downloaded FFmpeg build is calling Windows API functions that simply don’t exist in Windows 10 v1607.

I currently target modern Windows 10 (1809+) and Windows 11, so this older build isn’t covered in my regular testing. When the libx265 capability check failed with that OS-level error, the program incorrectly reported it as “libx265 missing from the FFmpeg build”.

If you want to run it on that system, the easiest workaround is to manually place an older FFmpeg build that still works on v1607 next to NVENCForge.exe (a local copy always takes priority).

I’ll improve the startup error handling in the next release so an OS compatibility crash is no longer misdiagnosed as a missing codec.

Just as a side note: NVENCForge is mainly built around modern NVIDIA GPUs and current Windows versions. The CPU mode exists as a fallback for systems without a suitable NVENC card, but full compatibility testing on very old Windows builds (like 1607) isn’t something I can realistically maintain as a single developer.

Still, thanks for taking the time to test it and report the issue!

tebasuna51
30th August 2026, 10:19
...
If you want to run it on that system, the easiest workaround is to manually place an older FFmpeg build that still works on v1607 next to NVENCForge.exe (a local copy always takes priority).
Thanks for the reply.

I download the latest full ffmpeg build from gyan.dev and work fine.

...
INFO > Auto-CQ: analyzing 3 sample windows (24 s) for VMAF target 96.5 (97 - 0.5 tolerance)...
OK Auto-CQ: CQ 19 -> predicted VMAF 94.0 (target 96.5) (cost cap 45%: CQ 18 would spend 52% of the source - CQ 19 measured 94.0 at 46%)
· anchors: CQ 18 = 94.55, CQ 22 = 91.56 · windows: bitrate-guided · analysis took 2:52
...
Then... what is the CQ used?
Can I run that test without launch the encoder?

We can select the tracks to encode? Maybe 1/2 audio/subtitles only from sources with many of them.

We can select the audio encoder? I create my ffmpeg builds with fdkaac encoder and use -vbr 4.

Another questions

iAvoe
30th August 2026, 11:32
Why? There doesn't seem to ba a good reason

In the source file


Are you sure this is not just some LLM hallucination? I've never encountered it.
This is somewhat true, my OnePlus 12r has some kind of hardware defect, playing any video fullscreen for long enough may trigger a system crash (reboot to slot B), the HEVC high bitrate videos can generate extra heat, and crashes the system faster :o

Burnersen
30th August 2026, 12:29
A big thank you to tebasuna51, Z2697, and iAvoe for taking the time to test NVENCForge, inspect the code, and share your feedback! Your input directly helped shape the v1.29.0 and v1.30.0 updates.

Answers to tebasuna51's questions:

1. Which CQ was used?
The final selected value was CQ 19. The algorithm initially aimed for CQ 18 (VMAF 94.55), but that was predicted to use 52% of the original size, exceeding your 45% cost cap. It automatically fell back to CQ 19 (46% cost, VMAF 94.0) to honor your safety limit. You were completely right about the log output: citing the rejected CQ in the same sentence was confusing. v1.30.0 completely rewrites this output to make the final chosen CQ crystal clear.

2. Can I run that test without launching the encoder?
Yes! v1.30.0 adds a new -cqcheck flag, along with a dedicated button in GUI v1.8.1. It runs the Auto-CQ analysis, logs the picked CQ for every file in the queue, and exits without encoding, exactly like -cropcheck.

3. Can we select tracks to encode?
Not inside the main converter pipeline. The converter is built for unattended overnight batch jobs, so interactive prompts per file would break automation. For custom track filtering, you can use the built-in lossless split/join tools before or after encoding without re-encoding video.

4. Can we select the audio encoder (e.g. libfdk_aac -vbr 4)?
Currently, audioMode only supports FFmpeg's native aac or copy. Since I cannot easily test custom FDK-AAC builds here, adding an unverified switch would be risky. I've noted it for future consideration, though!

Bonus Fix: While rewriting the log logic based on your report, I spotted a hidden bug in an Auto-CQ edge case where a fallback dropped the measured result and defaulted to an interpolated CQ. That is now fixed in v1.30.0. Your question paid for itself!

Notes to Z2697 & iAvoe:
@Z2697: Thanks for questioning the 50 Mbit/s comment. The source code comment and help text regarding the NVDEC threshold have been cleaned up in v1.29.0 to be factual rather than dramatic.
@iAvoe: Thanks for sharing that hardware anecdote! High-bitrate HEVC decoding can indeed put substantial thermal stress on tight hardware setups.

Release Summary (NVENCForge v1.30.0 / GUI v1.8.1):
- Fixed Windows NT system-level crashes (like 0xc0000139) being misdiagnosed as missing codecs.
- Added -cqcheck CLI flag and GUI button for analysis-only runs.
- Cleaned log output and fixed the interpolated CQ fallback bug.

Burnersen
30th August 2026, 13:03
Oops, I'm only human and totally forgot to address one of your points in my previous post!

@Z2697 - regarding CBR / flat-bitrate sources:

NVENCForge actually already has a fallback in place for this! If the bitrate variation in the container is too flat (the heaviest section stays under 1.25x the median), the guided window placement automatically gives up and falls back to an even 20% / 45% / 70% grid (logged as "fixed positions").

So CBR or flat-rate material won't throw off the sampling
it simply switches to the fixed grid instead.

Sorry for missing that