News cards Nvidia (NVEnc) Can you compete with x265?

[DMD]

Good evening.
I ask if the new Nvidia RTX 3xxxx video cards (NVENC) can compete with X265 at a qualitative level.
I ask this if I should apply a less expensive video card and mount a multicore processor like Ryzen 9 5900x
Thank you

[RanmaCanada]

This is answered in the hardware encoding thread. The answer is NO.

[tonemapped]

The quality is slowly improving, and for non-grainy sources most people probably wouldn't notice. That being said, even the best NVENC encode with an RTX 3080 looks worse than a software --fast encode with x265.

One use I've found for NVENC is SD content, such as old DVDs. It does a good enough job there as there's not much detail to start with.

[Emulgator]

I ask this if I should apply a less expensive video card and mount a multicore processor like Ryzen 9 5900x
Seeing the h.265 output of a RTX3080 here:
Yes, I would weight expenses towards CPU in your case, qualitywise it may be not worth to pay up for GPU too much.
Besides, IIRC the 20xx series are reported to have the same encoding engine.
But if you intend to go for AI resizing sooner or later: You will need a beefy GPU as well.

[Yups]

I made two tests some time ago. GTX 1660S and RTX 3000 same encoder, no difference.


https://forum.doom9.org/showpost.php?p=1940526&postcount=451
https://forum.doom9.org/showpost.php?p=1940485&postcount=450


There are 2-3 encoding options which can boost the quality quite a bit. With these the quality in best preset is respectable around x265 faster level.

[RanmaCanada]

Seeing the h.265 output of a RTX3080 here:
Yes, I would weight expenses towards CPU in your case, qualitywise it may be not worth to pay up for GPU too much.
Besides, IIRC the 20xx series are reported to have the same encoding engine.
But if you intend to go for AI resizing sooner or later: You will need a beefy GPU as well.

That is correct. Turing and Ampere use the same ASICS so the encoding quality will be exactly the same. It's a step up from Pascal and Maxwell, but still not as good as software. Next gen maybe.

[DMD]

Thank you for the confirmation.
I will orient my bugget on the CPU, at this point you do not need a powerful latest generation video card, perhaps a smaller RTX 3050.
The problem is finding a good compromise setting for x265 (StaxRip), but I assume that a Slow or Very Slow preset is essential for good quality.
Unfortunately, there is no solution to long CPU processing times.:(

[RanmaCanada]

Thank you for the confirmation.
I will orient my bugget on the CPU, at this point you do not need a powerful latest generation video card, perhaps a smaller RTX 3050.
The problem is finding a good compromise setting for x265 (StaxRip), but I assume that a Slow or Very Slow preset is essential for good quality.
Unfortunately, there is no solution to long CPU processing times.:(

I find it odd that you didn't ask about Quicksync. Quicksync is better than NVENC and is almost on par with x265.

Again, go through the hardware encoding thread and near the end there is discussion of the new XE ASICS and how the quality is far superior to NVENC and almost borderline to x265.

[DMD]

I find it odd that you didn't ask about Quicksync. Quicksync is better than NVENC and is almost on par with x265.

Again, go through the hardware encoding thread and near the end there is discussion of the new XE ASICS and how the quality is far superior to NVENC and almost borderline to x265.
Should I enable Intel or AMD> H265 in StaxRip?
Can it be compared to x265 in terms of quality?
Thank you

[excellentswordfight]

I find it odd that you didn't ask about Quicksync. Quicksync is better than NVENC and is almost on par with x265.

Again, go through the hardware encoding thread and near the end there is discussion of the new XE ASICS and how the quality is far superior to NVENC and almost borderline to x265.
What is the preferred encode mode with quicksync? I just got my hands on an Turing GPU so I'm currently doing some tests. I just tried icq and found nvenc to be superior.

Results of an very quick test: x265 medium, x264 veryslow and QSVEncC64 all reached around 50fps on an 12700k for an 1080p Bluray RIP @ 4Mbps, NVEncC64 200fps. In terms of quality x264>x265>NVEncC64>QSVEncC64. Nvenc did indeed seems to be similar to x265 preset fast.

[Yups]

What is the preferred encode mode with quicksync? I just got my hands on an Turing GPU so I'm currently doing some tests. I just tried icq and found nvenc to be superior.

Results of an very quick test: x265 medium, x264 veryslow and QSVEncC64 all reached around 50fps on an 12700k for an 1080p Bluray RIP @ 4Mbps, NVEncC64 200fps. In terms of quality x264>x265>NVEncC64>QSVEncC64. Nvenc did indeed seems to be similar to x265 preset fast.


CBR/VBR is better on Nvidia, CQP is better on Intel. You have to use CQP and 14 bframes on Intel for best quality, Nvidia is limited to 5 bframes. ICQ mode is worse than CQP on Intel.

Ok I actually posted the preferred settings for Xe here: https://forum.doom9.org/showpost.php?p=1949008&postcount=494


Also you are using the hybrid mode obviously, try low power mode and you get a huge performance boost. Quality is a bit lower in this mode but performance is a different world.

[DMD]

I am in favor of using the x265 encoder for high quality sources such as upscaled 4K movies or footage from professional camcorders, where the time spent is worth the final quality.
Use the NVEnc encoder only for videos coming from smartphones or low quality sources, it might be a good one compromise?

[benwaggoner]

That is correct. Turing and Ampere use the same ASICS so the encoding quality will be exactly the same. It's a step up from Pascal and Maxwell, but still not as good as software. Next gen maybe.
I doubt any HW encoder will ever be as good as the best SW encoder for the best codec of any given era. As codec complexity increases, optimal algorithms are way much more about little branchy early-exit software-defined decisions that are really well suited to having lots of cores + SIMD sharing L3 cache.

[RanmaCanada]

I doubt any HW encoder will ever be as good as the best SW encoder for the best codec of any given era. As codec complexity increases, optimal algorithms are way much more about little branchy early-exit software-defined decisions that are really well suited to having lots of cores + SIMD sharing L3 cache.

Isn't the Turing/Ampere hardware implementation of x264 on NVENC better than x264 slow (https://obsproject.com/forum/resources/ultimate-encoder-quality-analysis-2020-nvenc-vs-amf-vs-quicksync-vs-x264.998/)? We've had hardware HEVC encoding for almost a decade now, and with AV1 adoption moving at a snail's pace, I figure the next implementation of HEVC hardware will be even better than the current. At least at consumer level. We are already starting to see close results with QuickSync from Intel. Pretty sure professional unobtainium hardware already meets or beats current software implementations. (Tencent)

[benwaggoner]

Isn't the Turing/Ampere hardware implementation of x264 on NVENC better than x264 slow (https://obsproject.com/forum/resources/ultimate-encoder-quality-analysis-2020-nvenc-vs-amf-vs-quicksync-vs-x264.998/)? We've had hardware HEVC encoding for almost a decade now, and with AV1 adoption moving at a snail's pace, I figure the next implementation of HEVC hardware will be even better than the current. At least at consumer level. We are already starting to see close results with QuickSync from Intel. Pretty sure professional unobtainium hardware already meets or beats current software implementations. (Tencent)
I can see hardware encoders with good content adaption beating default settings with a SW encoder. But it's not like x265 quality improvements end with --preset placebo. There are plenty of content-depending tuning beyond that. For example, using --tskip with animation, or --cu-lossless with screen captures.

Nor is x265 the best encoder for every HEVC scenario. Beamr and Elemental can beat even an optimally tuned x265 with very grainy content, for example. All the professional-grade HEVC encoders use SW encoders last I deep dived.

I've not seen an example of a HEVC HW encode that couldn't be matched or beat by a properly tuned software encoder. HW encoders shine most with quality @ perf and quality @ CPU. HW on GPU is great for game streaming, for example, as the CPU and memory bandwidth saved can really improve the game's performance.

[Blue_MiSfit]

I wasn't aware Elemental had improved significantly. I was pretty underwhelmed at their HEVC encoding quality last I looked :) It's been several years though.

[benwaggoner]

I wasn't aware Elemental had improved significantly. I was pretty underwhelmed at their HEVC encoding quality last I looked :) It's been several years though.
Elemental's HEVC has a lot of valuable improvements in the last few years. It's definitely superior to x265 for realtime 4K encoding on equivalent hardware. Lots of psychovisual tuning, improved adaptive quantization, etc.

[bitspyer]

I made two tests some time ago. GTX 1660S and RTX 3000 same encoder, no difference.


https://forum.doom9.org/showpost.php?p=1940526&postcount=451
https://forum.doom9.org/showpost.php?p=1940485&postcount=450


There are 2-3 encoding options which can boost the quality quite a bit. With these the quality in best preset is respectable around x265 faster level.

Can you post or PM your detailed Settings for nvencc or the encoder you used for gtx1660 encoding?

[DMD]

For the moment the only problem of hardware encoding NVEnc HEVC
It has no support for b-frames, nor 4:2:2 chroma subsampling and the decoder only support 4:2:0 chroma subsampling.
From the beginning the Nvidia codec has been finalized for streaming play, I don't know if this problem will be solved.
I found some interesting articles:
https://codecalamity.com/hardware-encoding-4k-hdr10-videos/
https://www.linkedin.com/pulse/gpu-vs-cpu-encoding-part-1-bj%C3%B6rn-isakson
https://www.linkedin.com/pulse/video-encoding-using-gpu-part-2-bj%C3%B6rn-isakson

[Selur]

For the moment the only problem of hardware encoding NVEnc HEVC
It has no support for b-frames,
most newer cards (6th generation and newer) support b-frames, see: https://en.wikipedia.org/wiki/Nvidia_NVENC

[DMD]

most newer cards (6th generation and newer) support b-frames, see: https://en.wikipedia.org/wiki/Nvidia_NVENC
Thank you for the link.
This confirms that b-frame and 4:2:2 and 4:4:4 color space are supported, I am more reassured now at least on the RTX 3xxx series with seventh generation GPUs.
https://www.nvidia.com/it-it/geforce/graphics-cards/compare/?section=compare-specs

What I can't understand now if the qualitative differences with x265 software encoder are so obvious.

[Selur]

afaik 4:2:2 is not supported. (4:4:4 and 4:2:0 are)

What I can't understand now if the qualitative differences with x265 software encoder are so obvious.
That depends on tons of things, like x265 settings, bit rate used, resolution used, how well you are trained to spot differences due to compresseion,...
-> if your data rate is high enough you probably can not tell the difference during normal playback

[benwaggoner]

For the last ~15 years, the best software encoders have offered reliably better quality @ bitrate than fixed-function HW encoders or GPU-accelerated encoders. And the delta keeps getting bigger as codecs become more complex. So many little branchy early exits needed between all the potential ways a given 4x4 block could be encoded. So many psychovisual considerations to factor in.

Even for live encoding, the high end systems are all software based these days.

There is certainly interesting R&D about using FPGA, or GPU-based encoders for a first pass for SW to refine. I've yet to see something that offers a real-world advantage yet.

A modern high-end CPU has a whole lot of cores and a whole lot of SIMD power, and can do a tons of GPU-like parallel vector processing and complex branchy logic at the same time, all within the same L3 cache.