Still trying to work out x265's 'static grain' issue

[tonemapped]

Originally Posted by rwill

You know, Blu-Rays are copy protected and are not meant to be used as a source for encoding the content further. So I guess keep your disc ?

Why are you registered if that's your primary view? It's legal to make a copy of owned content.

Originally Posted by rwill

You also seem to have a basic misunerstanding of data processing. Please read this Wikipedia article:

Garbage in, garbage out

I didn't create this thread to debate VC-1 vs H.264 quality. I'm not sure where you got that impression from.

Originally Posted by rwill

As I wrote previously: 'My guess is that most bits are spent on compression artifacts. The Grain is stuck to the hair in this "Source" already, an encoder cannot magically unstuck it.'

I'm more than aware that encoding it isn't going to remove defects on the source. Again, that's not the problem - the problem is x265 having poor handling compared to x264 when more complex grain is involved. Based on your test file of the same scene, you've simply stripped all the grain from it. That's not a solution.

Originally Posted by rwill

Now x265 is not among the best HEVC encoders that are available but it is not really bad either - I mean at least its free. I think using crappy sources like yours prevent proper detail retention with x265 further.

This is a problem I've encountered on a number of Blu-ray sources, including the German remastered Charmed collection (which is know for its excellent quality).

Why won't you name the encoder you claim you've created? The file you shared shows GPAC. Are you claiming you created that?

Originally Posted by rwill

The quality issue you try to solve is really negligible if one takes the state of the source material into account and what an encoder can still do given such input.

The issue I'm trying to solve is x265's poor handling of grain, compared to x264 at the same bitrate.

Originally Posted by rwill

I only posted the output of a different encoder to prove that it can be done in higher 'quality' compared to x265.

Which encoder?

Originally Posted by rwill

Proving that something can be done better is very important in compression for some people. It differentiates between realistic and unrealistic demands in the ever present request for higher compression efficiency.

And the relevance of that statement is...?

[tonemapped]

Originally Posted by Boulder

At lower bitrates, you are better off with aq-mode 2 and SAO (maybe selective SAO could be used). Yes, SAO does eat grain for breakfast but you can't have both detail and a low bitrate if the source is not easy to compress. HEVC blurs while AVC creates blocking when the bitrate is not enough to keep the details. Blocking may look better in motion, depending on your viewing distance etc.

A lot of people think that HEVC is better because it is newer, but in fact the use cases are more or less related to being able to create a watchable video at a low bitrate at higher resolutions, at least for x265. You can easily see how most of the actual development is done on that part, and that is why I've always taken the developers' fancy new things with a grain of salt. I myself use x265 because of HW support for 10bit encodes and of course HDR encodes pretty much require using it.

I'm happy to use CRF, but used 6mbps for the sake of comparison, or do you mean low bitrate sources? The annoying this is, the majority of the encode looks great with grain at ~5mbps - apart from those types of scenarios where the grain 'clings' to moving objects. This is not a source problem, but something x265 fails at (even encoding 2-pass, very slow, 100mbps doesn't get rid of the issue), or something I'm doing wrong.

I don't use x265 for everything, but for less noisy sources it's excellent. Even with some noisy sources it's not bad. Sex and the City is one of the worst 'remasters' I've ever seen, with the usual trick of adding a thick layer of noise to help with perceived quality. Using slow (with a few changes), CRF 23, the file produced is 1.24GB (5,910 kb/s) and looks great.

[Boulder]

Originally Posted by tonemapped

I'm happy to use CRF, but used 6mbps for the sake of comparison, or do you mean low bitrate sources? The annoying this is, the majority of the encode looks great with grain at ~5mbps - apart from those types of scenarios where the grain 'clings' to moving objects. This is not a source problem, but something x265 fails at (even encoding 2-pass, very slow, 100mbps doesn't get rid of the issue), or something I'm doing wrong.

I don't use x265 for everything, but for less noisy sources it's excellent. Even with some noisy sources it's not bad. Sex and the City is one of the worst 'remasters' I've ever seen, with the usual trick of adding a thick layer of noise to help with perceived quality. Using slow (with a few changes), CRF 23, the file produced is 1.24GB (5,910 kb/s) and looks great.

I mean low-bitrate encodes. Of course, that's pretty much a subjective thing as it depends on how well the source compresses. 6 Mbps might not be that much for 1080p - for example I'm currently encoding the LOTR movies and it looks like The Two Towers will end up at around 8 Mbps at CRF 19 using my slightly denoising script. If it had an AR of 1,78:1, it would require much more.

Having said that, I still suggest testing some very slight denoising to combat your issues.

[benwaggoner]

Some things to try when strugging with fine grain content in x265:

1. use --nr-inter to reduce grain swirling and bitrate consumption. It can go up to 2000; don't be afraid to test high values.
2. Raise --nr-intra to where I frames don't have visibly more detail
3. Reduce --ipratio and --pbratio for less frame strobing
4. Raise psy-rdoq to retain a more even appearance of grain detail
5. Reduce --aq-strength to keep grain detail more consistent across the frame
6. Use --aq-mode 4 instead of 2
7. Use --rskip 2 instead of 1, with a lower threshold. The original rskip was terrible with grain
8. Try --rd 4 if --rd 6 isn't working well

[tonemapped]

Originally Posted by Boulder

I mean low-bitrate encodes. Of course, that's pretty much a subjective thing as it depends on how well the source compresses. 6 Mbps might not be that much for 1080p - for example I'm currently encoding the LOTR movies and it looks like The Two Towers will end up at around 8 Mbps at CRF 19 using my slightly denoising script. If it had an AR of 1,78:1, it would require much more.

Having said that, I still suggest testing some very slight denoising to combat your issues.

That's an impressive bitrate for a film with some complex scenes. What's the basis of your denoising script?

Originally Posted by benwaggoner

Some things to try when strugging with fine grain content in x265:

1. use --nr-inter to reduce grain swirling and bitrate consumption. It can go up to 2000; don't be afraid to test high values.
2. Raise --nr-intra to where I frames don't have visibly more detail
3. Reduce --ipratio and --pbratio for less frame strobing
4. Raise psy-rdoq to retain a more even appearance of grain detail
5. Reduce --aq-strength to keep grain detail more consistent across the frame
6. Use --aq-mode 4 instead of 2
7. Use --rskip 2 instead of 1, with a lower threshold. The original rskip was terrible with grain
8. Try --rd 4 if --rd 6 isn't working well

• I find intra frame noise reduction produces horrible results no matter what I do (and change elsewhere). I conducted a test with Inter-frame denoising at an absurd value (1000) and it produced a really great result. I also conducted a test with inter-frame noise reduction at 75 and SAO enabled with selective SAO at 1. That also produced a good result. Both were at CRF 21.
  
• Using intra-frame NR seems to produce blurred results with a lot of detail lost (not just the perceived detail grain/noise offers). Is this something you've experienced?
  
• I generally always reduce the ipratio and pbratio to 1.30 and 1.20, or if a complex source then 1.2 and 1.1 (or thereabouts).
  
• I've tried values from the default 1.00 up to 30.00 and, perhaps it's my eyes, not really seen much difference between them - and that's with rskip on 2.
  
• Oddly, I've found an AQ of 1 producing the best results with x265, with 3 and 4 prone to a strobing effect. Not sure how else to describe it. I usually set AQ strength to 0.80 for most content, and then ~0.60 for very noisy content. However, it appears AQ acts very differently on x265 compared to x264 and I've not managed to find a paper or even a simple answer on how it works differently etc.
  
• I'll try changing RD to 4. Have you experienced content where 4 produces a better result compared to 6, and if so, why do you think that occurred?

I've replied with bullet points for clarity - I don't want it to be seen as rudeness.

[tonemapped]

Since I managed to get the Sex and the City boxset for a bargain price, here's what some testing produces. The idea was not to be transparent since the Blu-ray isn't great, but for it to be 'close enough' at a much reduced file size.

I'm curious if anyone can tell which is which - it should be obvious. This is was of the worst examples I could find (in terms of quality). Since the grain/noise is 'bigger' on this, I'm using --aq-mode 3 which seems to work better. For finer grain, AQ 1 (or disabled) seems the best.

One is taken from the Blu-ray disc and is just over 6.31GB at 1920*1080. The other is the x265 encode and is 1.40GB at 1440*1080 (screenshots using spline36 to resize to 1920*1080). The screenshot from the encoded file is a b-frame.

--crf 23 --preset slow --output-depth 10 --profile main10 --rd 5 --psy-rd 2.45 --psy-rdoq 3 --rskip 2 --no-rect --aq-mode 3 --aq-strength 0.7 --nr-inter 75 --ipratio 1.35 --pbratio 1.25 --subme 6 --bframes 8 --rc-lookahead 120 --deblock -4:-4 --no-sao --no-strong-intra-smoothing

[Boulder]

Originally Posted by tonemapped

That's an impressive bitrate for a film with some complex scenes. What's the basis of your denoising script?

I can share the denoiser function after some cleanup. It's just similar stuff that many others do: 1) predenoise the clip used for motion search, 2) do analysis on that clip and 3) use MDegrain on the sharpened original clip for limited motion compensated denoising to avoid excessive detail loss. Does slight spatial denoising where block matching is bad (usually blocks with lots of motion).



The final average bitrate for the TTT was around 8.1 Mbps. However, I must stress that there is something wrong with CTU 64 and 1080p resolution at least when it comes to CRF encodes (EDIT: and rskip mode 2). I did a color corrected encode of FOTR EE (cropped resolution 1920x800) and started wondering why the flat backgrounds don't look good at all in motion. I then compared CTU 64 and 32 on a shorter sample from the movie and the filesize difference was very big. I've already noticed this earlier but thought it was only in case of --limit-tu 0. I am quite sure that the encoder does some stupid decisions with CTU 64 and rskip 2 because the avg QP differs a lot but frame types are of course the same. I really need to test if it applies to >1080p resolutions as well. Hope not, because it would mean some more rework for me

https://forum.doom9.org/showthread.p...47#post1919347

[Boulder]

And forget the exotic aq-modes, they just don't work well with regular encodes. I'd say modes 1 or 2 are the ones to use, and 1 preferred in most cases. The functionality was ported from x264 and I don't think the devs have done any real work on them. That's why they may not work like in x264 because the environment is not the exact same.

[Boulder]

I've pasted my function here: https://pastebin.com/0KegSZiR - I suppose you know how to handle Avisynth functions. If you don't have it already, I strongly recommend the latest official Avisynth+ build by pinterf and then possibly his latest test build from the Avisynth+ thread in the Avisynth Development section. Chances are that some of the dependencies require that.

Dogway's filter packs are used by the function, you'll probably need at least ExTools, TransformsPack, SPresso and SMDegrain. https://forum.doom9.org/showthread.php?t=182881
Get pinterf's MVTools and DFTTest as well.

What I usually do is:
1) Use debug=1 to select the prefiltering (adjusted by prefilter=1, 2 or 3) on the motion search clip. This depends on the amount of noise and grain as you want to have a rather clean clip to do the analysis on.

2) Use debug=6 to set thscd1 and thscd2 based on the amount of blocks it detects as "changed too much". In the upper left corner, if the bottom value exceeds thscd2, it triggers a scene change in the denoising. This must be set for each source as they differ a lot. Thscd1 affects which blocks are detected as "changed too much".

3) Finally, debug=2 to set limit (limits MDegrain) and blurlimit (limits spatial blur), it interleaves between the original and denoised frame. Debug=3 is the same but it shows the differences in luma amplified. Debug=4 does the same for chroma.

The defaults are set so that the denoising is very slight as I like to keep the grain. Still, it can easily shave off a nice amount of bits from the final encode without actually being noticable.

For example, my ROTK script looks like this (on a 3900X):

part1=DGSource("M:\lotr_rotk_part1.dgi",ct=144,cb=144,cl=4,cr=4).Trim(0,183541)
part2=DGSource("M:\lotr_rotk_part2.dgi",ct=144,cb=144,cl=4,cr=4).Trim(24,0)
part1+part2
MDG(limit=0.1, prefilter=1, blurlimit=0.3, thscd1=420, thscd2=90)
Neo_f3kdb(preset="medium", grainy=0, grainc=0, output_depth=16, sample_mode=4, mt=false)
AddBorders(8,0,8,0)
Prefetch(threads=24, frames=12)

[tonemapped]

Originally Posted by Boulder

... However, I must stress that there is something wrong with CTU 64 and 1080p resolution at least when it comes to CRF encodes (EDIT: and rskip mode 2). I ... started wondering why the flat backgrounds don't look good at all in motion. I then compared CTU 64 and 32 on a shorter sample from the movie and the filesize difference was very big.

https://forum.doom9.org/showthread.p...47#post1919347

Genuinely fascinating. I've also noticed that CTU 64 vs CTU 32 for 1080p encodes produces results I perceived to be odd, but thought it was normal and perhaps an oddity of x265. I've found in tests that 3-pass (slow first pass) encodes at 7,000 kbps in 720p, backgrounds look more natural compared to the 1080p encodes at 16,000 kbps. This is on a source with very little noise and backgrounds should be clean with no banding at the bitrates used.

What's your results from n-pass tests?

Originally Posted by Boulder

I've pasted my function here: https://pastebin.com/0KegSZiR - I suppose you know how to handle Avisynth functions.

I do.

Originally Posted by Boulder

Dogway's filter packs are used by the function, you'll probably need at least ExTools, TransformsPack, SPresso and SMDegrain. https://forum.doom9.org/showthread.php?t=182881
Get pinterf's MVTools and DFTTest as well.

I believe I have most of those, although SMDegrain is having issues on Windows 11 Pro (for me). No idea why.

Originally Posted by Boulder

What I usually do is:
1) Use debug=1 to select the prefiltering (adjusted by prefilter=1, 2 or 3) on the motion search clip. This depends on the amount of noise and grain as you want to have a rather clean clip to do the analysis on.

2) Use debug=6 to set thscd1 and thscd2 based on the amount of blocks it detects as "changed too much". In the upper left corner, if the bottom value exceeds thscd2, it triggers a scene change in the denoising. This must be set for each source as they differ a lot. Thscd1 affects which blocks are detected as "changed too much".

3) Finally, debug=2 to set limit (limits MDegrain) and blurlimit (limits spatial blur), it interleaves between the original and denoised frame. Debug=3 is the same but it shows the differences in luma amplified. Debug=4 does the same for chroma.

The defaults are set so that the denoising is very slight as I like to keep the grain. Still, it can easily shave off a nice amount of bits from the final encode without actually being noticable.

For example, my ROTK script looks like this (on a 3900X):

part1=DGSource("M:\lotr_rotk_part1.dgi",ct=144,cb=144,cl=4,cr=4).Trim(0,183541)
part2=DGSource("M:\lotr_rotk_part2.dgi",ct=144,cb=144,cl=4,cr=4).Trim(24,0)
part1+part2
MDG(limit=0.1, prefilter=1, blurlimit=0.3, thscd1=420, thscd2=90)
Neo_f3kdb(preset="medium", grainy=0, grainc=0, output_depth=16, sample_mode=4, mt=false)
AddBorders(8,0,8,0)
Prefetch(threads=24, frames=12)

Thank you for sharing your snippet. What's your view of something like TemporalDegrain2 instead of MDegrain, and perhaps GradFun3 instead of neo_f3kdb? Certainly in the latter I see better results with noisy sources.

[Boulder]

Originally Posted by tonemapped

Genuinely fascinating. I've also noticed that CTU 64 vs CTU 32 for 1080p encodes produces results I perceived to be odd, but thought it was normal and perhaps an oddity of x265. I've found in tests that 3-pass (slow first pass) encodes at 7,000 kbps in 720p, backgrounds look more natural compared to the 1080p encodes at 16,000 kbps. This is on a source with very little noise and backgrounds should be clean with no banding at the bitrates used.

What's your results from n-pass tests?

I've never properly tested a multipass IIRC, since I don't do any of that for production use. I would expect that it still shows the same symptoms. My findings earlier were that the encoder favours merges a lot when CTU is 64 and you use rskip mode 2.

Thank you for sharing your snippet. What's your view of something like TemporalDegrain2 instead of MDegrain, and perhaps GradFun3 instead of neo_f3kdb? Certainly in the latter I see better results with noisy sources.

As far as I know, TemporalDegrainV2 also does similar things, i.e. MDegrain is used there too. I've noticed that neo_f3kdb does very slight debanding with my settings so I've kind of settled on that. Strong settings can easily start removing detail from the image.

[tonemapped]

Off topic, but ...

Currently running three x265 jobs - including one testing out TemporalDegrain2 as a substitute (as mentioned above). Still amazed after going from a 4C/8T 6700K @ ~4.5 GHz (overclocked) to a 12C/24T 5900X at ~4.30 GHz (stock). The new CPU also runs 20°C cooler (although it does have a 360mm radiator)!



Boulder, the reason I've gone off topic is because you mentioned you also have a 12C CPU. What sort of thread/pool allocation do you tend to use with x265? In x264 it's easy to work out, but with x265 it seems that the extra threads really hurt quality if allocated to a single job, hence running three jobs at the same time. I've done tests on a 32C/64T server at work, but that's different as the default setup is four simultaneous jobs.

[Boulder]

Originally Posted by tonemapped

Boulder, the reason I've gone off topic is because you mentioned you also have a 12C CPU. What sort of thread/pool allocation do you tend to use with x265? In x264 it's easy to work out, but with x265 it seems that the extra threads really hurt quality if allocated to a single job, hence running three jobs at the same time. I've done tests on a 32C/64T server at work, but that's different as the default setup is four simultaneous jobs.

That is an interesting question indeed. I've not encountered any quality issues with the default settings - I believe the biggest thing with faster presets is that they use lookahead slices which can affect frame type decisions negatively. The slower ones run just one slice.

I've also tested -F 1 to -F 4 and have not noticed any negative impacts by running four frame threads as per default. I think it may affect reference frame decisions negatively, but nothing that I could see with my own eyes (frame by frame or in motion).

So in short; I just run with the default settings and also assign 24 threads to Avisynth so the total CPU usage is usually 80-99%.