Updated (Grainy Source): x264 vs x265 vs NVENC H.265 vs QSV H.265 [2-pass @ 5000kbps]

[tonemapped]

Update: Please see this post for updated encoding and five screenshots per encode

OLD POST

Introduction
I've spent about 30 hours testing the same four minute clip of a very grainy source using 2-pass @ 4,500 kbps to control for size and bitrate (for comparisons), as well as a couple of CRF encodes thrown in to see what bitrate would be required (for x265).

The aim of my experiment is to keep the grain, or as much of it as possible - or perceivably possible -, for the bitrate allocated. I've spent about 80 hours reading threads on doom9 the x265 documentation over a two week period. However, I have played around with heavy noise removal using x265 and x264 built-in noise reduction. I have done this to see what sort of difference it makes to the grain quality (I apologise if that doesn't make sense to anyone else).

For creating a 'perceived rating', I tested scenes with motion, darkness, detail, edge detail (plants in the corner), and other small details. All encodes were 10 bit to create a level playing field (as much as possible). The resolution is 1920x1080. Sensible flags, such as veryslow for x264 and tuning for grain, and not tuning for grain on x265 but using slow and changing options manually that should preserve more grain, but with speed optimisations (just to get a rough idea of the output).

I did create a different thread with screenshots, but it went to moderation and was never seen again (I don't believe it broke the rules at all, as I read each one - I've not received a message or notification about the thread so assume it has been lost).


Software/Hardware Versions
NVENC (Pascal 1080 and Turing 2080)
x265 3.5 (3.5.0.9)
x264 161 r3048 (10 bit)

Results
From testing, I've found (perceived rating out of 10):

x265
- NVENC (Pascal): 6/10
- NVENC (Turing): 6/10
- x265: 3/10

x264
- NVENC (Pascal): 7/10
- NVENC (Turing): 7/10
- x264: 8/10

NOTE: These are 2-pass where stated, or CRF. One of the CRF files has a bitrate of over 8mbps (x265) and is OK, but also very large. I have lost many of the logs - and please ignore the () after -x265 as that's for my own sanity to try and make sure the information is correct.

Screenshots - SOURCE



Screenshots - NVENC (Turing)

Screenshots - QSV (Gemini Lake Refresh - 'Gen 9.5')

Screenshots - x264

Screenshots - x265 (test3)

Screenshots - x265 (test4)

Screenshots - x265 (test6)

Screenshots - x265 (test8)

Screenshots - x265 (x265n)

What I'm trying to understand, and I have read many, many threads, is why x265 appears to have such a problem with grain - even with a 10mbps 3-pass encode. The grain on x265 appears to be static, and tuning for grain seems to be counterproductive. I've read threads where it's stated that --tune grain isn't effective, but without an explanation.

Conclusion
I hope this can lead to a discussion about the state of grain retention with x265 in 2021. It's come a long way in a relatively short period of time and much of the guidance and opinion appears to apply to older builds.


Edit: I've not included screenshots as I believe this might have been the reason the previous thread (with about eight screenshots) was not allowed.
Edit2: Included screenshots and basic information.
Edit3: Updated top of post to reflect new encodes and screenshots

[Boulder]

Screenshots should not be an issue, they are used all the time.

It would be nice to see the command line parameters for x265. The presets definitely will smooth the image.

[tonemapped]

Originally Posted by Boulder

Screenshots should not be an issue, they are used all the time.

It would be nice to see the command line parameters for x265. The presets definitely will smooth the image.

Thank you for the reply. I'll try and get about 10 different screenshots and the cli used for each one shortly. At this point, I've added QSV into the mix and it seems to produce better results at the same bitrate, which is clearly absurd!

[benwaggoner]

Have you tried --tune grain yet? That would be good for a baseline.

[tonemapped]

Originally Posted by benwaggoner

Have you tried --tune grain yet? That would be good for a baseline.

That was the first option I tried, but it looks atrocious. The grain became static and it looked worse than medium preset with no changes.

Originally Posted by Stereodude

How did you evaluate it? Pouring over stills or playing it back in motion at normal speed?

First stills, then watching a four minute clip on a 10 bit 4K calibrated monitor (workstation), and then on a TV from about 3m away.

[fauxreaper]

Originally Posted by tonemapped

That was the first option I tried, but it looks atrocious. The grain became static and it looked worse than medium preset with no changes.

I think that you shold try --ctu 16 --qg-size 8 --rskip 0 --no-early-skip --rdoq-level 1 with high --psy-rdoq.

[Stereodude]

How did you evaluate it? Pouring over stills or playing it back in motion at normal speed?

[Boulder]

If you use x265 for any real work, you should use at least --preset slow as the base, I personally use --preset slower. Nevertheless, the source is very difficult. I wouldn't call it grain, it's more like that your source comes from a grainy original encoded at a lowish bitrate - just look at the dark flat areas near the top of the first image. What's the average bitrate of the source clip?

Also try aq-mode 1 although I'm not sure it will help with this case since you use quite a low average bitrate. Out of interest, what kind of average bitrate would you get for CRF 19 and your test 8 command line?

[tonemapped]

Originally Posted by Boulder

I wouldn't call it grain, it's more like that your source comes from a grainy original encoded at a lowish bitrate - just look at the dark flat areas near the top of the first image. What's the average bitrate of the source clip?

About 17mbps. I've recently done some more QSV testing and found a some good quality encodes (relatively). I'll upload the screenshots and details when I have the info on me.

[Emulgator]

What Boulder said: Give the encoder time for precision ! I am using at least x264 veryslow, x265 slower and can suggest that with confidence.

Seeing stills only...anyway for such content type I would call 4,5Mbps @ FHD starved for x264, wet patches coming up where HF coefficients had been dropped and LF coefficients sustain.

If someone must have 4,5Mbps on that clip, 265 will try hard to impersonate grain and seems to do a bit better.
test 3 and 4 are quite close for 4,5Mbps.

(Parameter "tune grain" means encoder is allowed to throw in some calculation-cheap HF coefficients, not really encode whats given.
Lots of Blu-ray encodes have introduced such morphing grain which often moves with Luma borders. Strange to the eye.)

NVEnc just slips off my table into the mud, happy to see QSV cope better.

I would just give more bitrate and move on.
Only if you need to sail along that cliff of damage for educational reasons, well...

I would use zopti for that.

[tonemapped]

Originally Posted by Emulgator

anyway for such content type I would call 4,5Mbps @ FHD starved for x264, wet patches coming up where HF coefficients had been dropped and LF coefficients sustain.

I only show x264 for comparison. With my eyes, the x264 bitrate-starved encode shows better results than the x265 encode. I generally encode using CRF but 2-pass for testing.

Originally Posted by benwaggoner

Also, I strongly recommend you don't compare with --crf, as you wind up with output that varies in both quality and size. Better to use 2-pass encoding at the same average bitrate, so you're comparing quality at a given bitrate between encoders/modes.

I lost some of the log files so only posted examples of those with logs. The x264 vs x265 at the same bitrate is a good example of 2-pass at the same bitrate, with reasonably comparable parameters (to a point).

[Stereodude]

I like these options for x265: -F 1 -p veryslow --no-sao

I found -F 1 specifically to help with temporal grain issues (but it slows things down by reducing parallelism)

[benwaggoner]

Also, I strongly recommend you don't compare with --crf, as you wind up with output that varies in both quality and size. Better to use 2-pass encoding at the same average bitrate, so you're comparing quality at a given bitrate between encoders/modes.

[benwaggoner]

Some more notes looking over your bitrate comparisons

x264's --nr and x265's --nr-inter are the same core algorithm underneath and use the same scale (x264 doesn't have a --nr-intra equivalent). Comparing 50 to 500 and 2000 is pretty different! If you are combining --tune grain with --nr-inter 2000, I'd absolutely expect to see the "frozen grain" you describe. The -nr-inter is essentially an adaptive deadzone for high frequency coefficients in predicted blocks. Which means it filters out small sharp details that vary from the pixels it predicts from. Using a --nr-intra at 25-50% the --nr-inter value can help stabilize things somewhat. I think the highest --nr-inter I've ever found useful was 750, and that was with some edge-case crazy content. I think 250 is about the highest I'd use for most 1080p encoding, grainy or otherwise.

For a setting that can handle pretty grainy content well without losing too much speed or efficiency for normal content, I'd probably start with:

--preset slower
-F 2
--aq-mode 1
--aq-strength 0.5
--cutree 0
--ipratio 1.2
--pbratio 1.1
--qpstep 1
--sao 0
--psy-rd 2.0
--psy-rdoq 5.0
--recursion-skip 0
--nr-intra 100
--nr-inter 250

[tonemapped]

I'm currently working on some new comparisons, but keeping the logs and doing the following tests (all at 2-pass/QSV at CRF but same bitrate):

▶ Nvidia NVENC H265 @ 5000kbps, P7, 10-bit
▶ x264 @ 5000kbps, veryslow, 8-bit
▶ x265 @ 5000kbps, slow, waggoner (settings suggested above), 10-bit [slow & veryslow is too slow to justify encoding, but doing so for testing]
▶ x265 @ 5000kbps, slow, grain, 10-bit [slow & veryslow is too slow to justify encoding, but doing so for testing]
▶ Intel QSV @ 5080kbps, ICQ, 'best quality', 10-bit [2-pass not available]

They are completing now, but it's 31°C in my office so might need to wait until later to do them all (it's hot).

These should provide a fairer comparison, and the logs should be useful as well. I'm thinking of posting three screenshots per encode and would post the four minute clip but clearly copyright is an issue. I could technically cut 30 seconds from each one but then I wouldn't know where to upload it.

[Emulgator]

I could technically cut 30 seconds from each one but then I wouldn't know where to upload it.

wetransfer.com, anything below 2GB per file is free and stays up for 1 week.
zippyshare.com, below 500MB per file, stays up as long as there is a download now and then.

[tonemapped]

Originally Posted by Emulgator

I could technically cut 30 seconds from each one but then I wouldn't know where to upload it.

wetransfer.com, anything below 2GB per file is free and stays up for 1 week.
zippyshare.com, below 500MB per file, stays up as long as there is a download now and then.

Thank you. Does 30s comply with the rules? It seems many people upload ~2m of footage (from threads I've read).

[tonemapped]

- Updated encodes with more screenshots.
- Due to forum limit of 20 images per post, and there being 35 images to show, this is post one out of two.
- I hope the formatting is clear. I tried to make the layout as simple as possible.

1. ▶ source

bitrate: ~17mbps
resolution: 1920*1080
depth: 8-bit
length: 00:04:05 (245s)

2. ▶ nvenc-hevc-2pass-5000kbps-p7

parameters: --vbr 5000 --codec h265 --preset P7 --output-depth 10 --profile main10 --level 4 --multipass 2pass-full --aq --aq-temporal --bframes 4 --lookahead 32 --colormatrix bt709 --colorprim bt709 --transfer bt709 --cabac

frame type IDR 93
frame type I 93, total size 15.36 MB
frame type P 5784, total size 131.40 MB
(not sure where the bframes went)

depth: 10-bit
time: 00:00:50 (50s)

3. ▶ x264-2pass-5000kbps-film-veryslow

parameters: parameters: --pass 2 --bitrate 5000 --preset veryslow --tune film --profile high --level 4.1 --no-fast-pskip --nr 150 --aq-mode 2 --aq-strength 0.8 --subme 9 --bframes 6 --ref 4 --no-dct-decimate --colorprim bt709 --colormatrix bt709 --transfer bt709 --deblock -3:-3

x264 [info]: frame I:51 Avg QP:28.56 size:141590
x264 [info]: frame P:1586 Avg QP:31.48 size: 54644
x264 [info]: frame B:4240 Avg QP:33.94 size: 13992

depth: 8-bit
time: 00:10:42 (642s)

4. ▶ x264-2pass-5000kbps-grain-veryslow

parameters: --pass 2 --bitrate 5000 --preset veryslow --tune grain --profile high --level 4.1 --colorprim bt709 --colormatrix bt709 --transfer bt709 --deblock -3:-3

x264 [info]: frame I:47 Avg QP:31.66 size: 79442
x264 [info]: frame P:1584 Avg QP:32.91 size: 51823
x264 [info]: frame B:4246 Avg QP:34.21 size: 16192

depth: 8-bit
time: 00:08:12 (492s)

[tonemapped]

5. ▶ x265-2pass-5000kbps-grain-slow

parameters: --pass 2 --bitrate 5000 --preset slow --tune grain --output-depth 10 --profile main10 --level-idc 4 --no-high-tier --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --deblock -3:-3

x265 [INFO]: frame I: 103, Avg QP:33.96 kb/s: 13553.77
x265 [INFO]: frame P: 1548, Avg QP:34.69 kb/s: 6417.86
x265 [INFO]: frame B: 4226, Avg QP:34.82 kb/s: 4287.61

depth: 10-bit
time: 01:21:32 (4,892s)

6. ▶ x265-2pass-5000kbps-waggoner-slow (suggestions by Ben Waggoner)

parameters: --pass 2 --bitrate 5000 --preset slow --output-depth 10 --profile main10 --level-idc 4 --no-high-tier --psy-rdoq 5 --rskip 0 --aq-mode 1 --aq-strength 0.5 --qpstep 1 --nr-intra 100 --nr-inter 250 --ipratio 1.2 --pbratio 1.1 --no-cutree --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --deblock -3:-3 --no-sao

x265 [INFO]: frame I: 103, Avg QP:31.05 kb/s: 18137.07
x265 [INFO]: frame P: 1548, Avg QP:32.60 kb/s: 7904.58
x265 [INFO]: frame B: 4226, Avg QP:33.58 kb/s: 3617.51

depth: 10-bit
time: 01:23:09 (4,989s)

7. ▶ intel-hevc-qsv-qp30-icq

parameters: icq, qp30, highest quality options (don't have specifics to hand) - overall bitrate: ~5080kbps (this shouldn't make a massive difference when comparing to the 2-pass encodes)

frame type IDR 93
frame type I 93, total size 15.36 MB
frame type P 5784, total size 131.40 MB
(not sure where the bframes went)

depth: 10-bit
time: 00:06:40 (400s) (Atom-based Pentium Silver J5005, so don't expect it to be fast!)

8. ▶ x265-2pass-5000kbps-boulder-slower (suggestions by Boulder)

parameters: --pass 2 --bitrate 5000 --preset slower --output-depth 10 --profile main10 --level-idc 4 --no-high-tier --rd-refine --rskip 2 --rskip-edge-threshold 2 --limit-refs 0 --merange 58 --max-merge 2 --bframes 10 --ref 4 --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --deblock -1:-1 --no-sao

x265 [INFO]: frame I: 89, Avg QP:28.97 kb/s: 28745.94
x265 [INFO]: frame P: 1497, Avg QP:31.49 kb/s: 11857.99
x265 [INFO]: frame B: 4291, Avg QP:36.23 kb/s: 2116.03

depth: 10-bit
time: 04:01:40 (14,500s - that's four hours for a four minute clip!)

[Boulder]

Since your x265 don't have "regular" settings (--tune grain or --no-cutree are not very common), I cannot say anything about those QPs. They are rather high but that's probably because both encodes disable the cu-tree.

How about with something like this: --preset slower --deblock -1:-1 --no-sao --merange 58 --rskip 2 --rskip-edge-threshold 2 --rd-refine --max-merge 2 --ref 4 --bframes 10 --limit-refs 0 ?