View Full Version : What is the Nvidia HEVC encoder doing?


rwill
11th August 2026, 04:48
Hi,

in the last couple of weeks I tried to implement a software realtime HEVC encoder (realtime as in... 1080p@60fps on some small Ryzen 5600) for maybe offering an additional option for streaming boxes or the like. I am doing this for learning stuff regarding realtime software encoders etc.


So while it was easy to push over x265 in its veryfast or superfast mode quality wise I took an arrow to the knee when I compared Nvidias hardware HEVC encoder to mine on this test sequence at 4Mbit CBR with 2 seconds buffer:

https://media.xiph.org/video/derf/twitch/Uncompressed/CSGO.mov

For comparisons you can imagine x265's superfast mode to perform similar to my encoder mechanics wise. The problem zones are the HUD elements, especially the crosshair. The crosshair fades in and out, wiggles around a bit on the moving background etc. This is caused because it is perfectly in the picture center, thus being 3 pixels wide its spread over 4 CU boundaries, and not doing overlapped ME makes the encoder 'forget' about it being an important detail.

The Nvidia encoder is reproducing the HUD and crosshair almost perfectly while blurring the game field a little. I am pulling my hair out trying to come up with a theory how they might accomplish this. Maybe its something like non motion compensated MBTree thats working on individual pixel, but this would be specific to games with a HUD only and I cannot imagine them implementing something like this in their general purpose encoder.

While Nvidia wins on PSNR slightly its loosing in SSIM, so they seem to be mostly tuning for PSNR. They have more compute available but even when I do an exhaustive RD Mode Decision I cannot fix the damn crosshair and the other HUD elements....

So what are they doing?

Z2697
11th August 2026, 11:55
I have finally downloaded that file after hours...
WTF raw RGBA? 1/4 of the file is wasted :)

Z2697
11th August 2026, 16:45
If you do "full RD" on top of preset 2...
To me this is stable enough (more stable than my RTX40 nvenc)
https://workupload.com/file/ux3Qy3TD7Qw

ffmpeg -i asdhna.nut -vf format=gbrp,zscale=m=709:f=spline64,format=yuv420p -c:v libx265 -preset 2 -b:v 4M -x265-params sao=0:aq-mode=1:ctu=32:vbv-maxrate=4000:vbv-bufsize=8000:keyint=120:rd=6:rskip=0 asd.mp4
(I don't think it really matters, but it's my modded x265 inside this FFmpeg)

Cmptr
11th August 2026, 17:36
If you do "full RD" on top of preset 2...
To me this is stable enough (more stable than my RTX40 nvenc)
https://workupload.com/file/ux3Qy3TD7Qw

ffmpeg -i asdhna.nut -vf format=gbrp,zscale=m=709:f=spline64,format=yuv420p -c:v libx265 -preset 2 -b:v 4M -x265-params sao=0:aq-mode=1:ctu=32:vbv-maxrate=4000:vbv-bufsize=8000:keyint=120:rd=6:rskip=0 asd.mp4
(I don't think it really matters, but it's my modded x265 inside this FFmpeg)


Aren't rd levels above 4 broken?

Z2697
11th August 2026, 18:41
I think the answer is it really has enough compute to implement a RD that's at least close to or even better than x265 rd3/4?

Cmptr
11th August 2026, 18:44
I think the answer is it really has enough compute to implement a RD that's at least close to or even better than x265 rd3/4?

From what I've been told from other members on the forum, rd above 4 is broken and gives worse results.

Z2697
11th August 2026, 18:52
From what I've been told from other members on the forum, rd above 4 is broken and gives worse results.

That was not a reply to you...
The reply was "Just a myth" in your thread.

rwill
11th August 2026, 19:00
I think the answer is it really has enough compute to implement a RD that's at least close to or even better than x265 rd3/4?

I am starting to think this is the case. I think the times were software encoders in fast presets are able to beat hardware encoders in quality/rate are over.