Dealing With Constant, Sharp, Fine Grain?

[RainyDog]

I'm currently re-encoding 'Mirrors' (blu-ray source) to 720p for archiving and am having a hell of a time keeping the grain structure as transparent as possible. It's some of the sharpest, finest grain I've seen, and most likely a post-processing effect instead of natural. It's also pretty much constant throught the film, making it a bit of a nightmare to encode.

I've been experimenting with psy-rd and psy-trellis, and whilst a little bit of the latter helps, nothing really seems to do keep it constant except raising the bitrate to levels I'd rather not go to at 720p. Being a sub 2-hour film (and not a particularly great one, though that's another matter:)), I want to keep it under 6-7 gig which means an avg bitrate of no more than 7000kpbs. However, anything under 9000kpbs leads to the grain turning to mush in a few too many scenarios. I'm not demanding absolute transparency from non-existant magical settings, just to try and squeeze as much as I can out of the filesize/bitrate limits I've set for myself.

So, from others' experiences with similar material, is there any other settings other than the Psy settings that might help? Maybe a higher qcomp, and/or higher than 1 aq-strength as I believe this will re-direct bits to the backgrounds (the films image is somewhat '300'-ish in that the grain eats actual detail).

Cheers in advance for any pointers.

[Chengbin]

Do you have deblock at -3,-3?

[RainyDog]

Do you have deblock at -3,-3?

Yes, that's a default for me.

[elguaxo]

Have you tried --tune grain already?

[RainyDog]

Have you tried --tune grain already?

Thanks elguaxo, yeah I added the individual tweaks to my script but with slightly higher psy-trellis (0.4) and aq-strength (0.6), and lower deblock (-3,-3). I think I'll have a mess about with aq-strength and qcomp and just settle for as consistant as I can get it at DVD-5 x 1.5 size. It's not the greatest film in the world anyway, as I mentioned :)

[Blue_MiSfit]

If you want perfect grain retention, you may have difficulty doing so at 720p and relatively low bitrates associated with 1.5x DVD5.

--crf 19 --tune grain would be my suggestions :)

~MiSfit

[kevinsert]

grain good or bad for bluray encodes? i mean can we say less grain better PQ ?

[Blue_MiSfit]

"can way say less grain better PQ?"

No, we cannot. IMO, perfect picture quality (when doing BluRay backups, or other transcodes) is preserving the look of the source accurately, in motion.

~MiSfit

[Daemon404]

"can way say less grain better PQ?"

No, we cannot. IMO, perfect picture quality (when doing BluRay backups, or other transcodes) is preserving the look of the source accurately, in motion.

~MiSfit

http://japland.org/img/Untitled.jpg

This man thinks you should rephrase that a bit.

[Lyris]

...assuming the source was good to start with :P

[Blue_MiSfit]

A poorly deinterlaced, blurry anime character with ghosting and aliasing artifacts is asking me to rephrase? Erm... no? :)

To get back on topic:

If your goal is to accurately preserve grain, then be prepared to spend a lot of bits to do so.

If you can't spend a lot of bits, then do some degraining :) MDegrain2 is a great place to start.

~MiSfit

[nm]

A poorly deinterlaced, blurry anime character with ghosting and aliasing artifacts is asking me to rephrase? Erm... no? :)
Even if he's your source? Personally, I'd rather fix that up rather than leave him as-is. ;)

Same goes to high-frequency artificial noise if I got to decide.

[RainyDog]

Well, I did a few 2 min segment test encodes with different settings and found that a higher aq-strength looked best to my eyes for this source. At least for emulating the grain structure.

Which begs the question... Why is it recommended to lower aq-strength with grainy sources? Am I correct in thinking that higher aq redistributes bits from sharp edges to backgrounds/flat areas, and lower aq visa versa? So, a lower aq-strength for grainy sources seems counter productive to me... Or is it to keep too many bits from being distributed, wasted even, on background grain for a more balanced bit distribution?

[Dark Shikari]

Well, I did a few 2 min segment test encodes with different settings and found that a higher aq-strength looked best to my eyes for this source. At least for emulating the grain structure.

Which begs the question... Why is it recommended to lower aq-strength with grainy sources? Am I correct in thinking that higher aq redistributes bits from sharp edges to backgrounds/flat areas, and lower aq visa versa? So, a lower aq-strength for grainy sources seems counter productive to me... Or is it to keep too many bits from being distributed, wasted even, on background grain for a more balanced bit distribution?It's because at very high bitrates on a very grainy source, the bit cost of every macroblock is going to be almost the same because 99% of your bits will be spent on the grain. As a result, at this point, it doesn't make sense to redistribute bits based on spatial complexity when the only complexity that really matters is grain.

Particularly, there's the problem of sharp edges and such that, nevertheless, have grain as well.

[RainyDog]

It's because at very high bitrates on a very grainy source, the bit cost of every macroblock is going to be almost the same because 99% of your bits will be spent on the grain. As a result, at this point, it doesn't make sense to redistribute bits based on spatial complexity when the only complexity that really matters is grain.

Particularly, there's the problem of sharp edges and such that, nevertheless, have grain as well.

Thanks Dark Shikari, that does make perfect sense. I went ahead with the full encode at aq-strength 1.2 anyway, so I'll see what it looks like when its done. I'll probably then do a full second encode using aq-strength 0.5 or 0.6, plus qcomp 0.7 or 0.8, and compare the two. On the subject of qcomp; is constantly active grain something qcomp would generally identify as 'motion'? Just also wondering about the intended purpose of higher qcomp (0.8) in --tune grain...

Cheers.

[Dark Shikari]

Thanks Dark Shikari, that does make perfect sense. I went ahead with the full encode at aq-strength 1.2 anyway, so I'll see what it looks like when its done. I'll probably then do a full second encode using aq-strength 0.5 or 0.6, plus qcomp 0.7 or 0.8, and compare the two. On the subject of qcomp; is constantly active grain something qcomp would generally identify as 'motion'? Just also wondering about the intended purpose of higher qcomp (0.8) in --tune grain...

Cheers.The higher qcomp does the exact same thing as lower AQ except for the purposes of keeping temporal bit distribution relatively the same.

Basically the thought is that if you're trying to get perfect grain retention over the whole video, you don't want lower quality in one section of the video to lose you that grain retention there. While in a normal situation you could get away with lower quality in, say, a high motion scene.

[Shevach]

Despite I am suspicious that I don't understand the debated issue, i would like to add my two coins (sorry if my comments are irrelevant, respect then as grain noise).

H.264 standard provides a particular tool (film grain characteristics SEI) in order to enable for a decoder to syntesize film grain before displaying.
As a stream creator if you wish create a stream with grain, you can encode raw data (which is noise free) and insert into the stream film-grain parameters.
Once a decoder gets a film-grain SEI message it should generate grain noise according to parameters signaled in the SEI message and add the noise to the picture (before display).

Let's suppose that raw data (i.e. pixels) already contains grain noise. In such case one can reduce the noise by applaying a noise-removal filter (e.g. pseudo-inverse filter). Then one should encode this noise-free frame with signalling the film-grain SEI (in order to signal to a decoder to syntesize the removed noise).
Of course the above method has a disadvantage - noise removal might cause over-smoothing of the original picture.

[Dark Shikari]

Despite I am suspicious that I don't understand the debated issue, i would like to add my two coins (sorry if my comments are irrelevant, respect then as grain noise).

H.264 standard provides a particular tool (film grain characteristics SEI) in order to enable for a decoder to syntesize film grain before displaying.
As a stream creator if you wish create a stream with grain, you can encode raw data (which is noise free) and insert into the stream film-grain parameters.
Once a decoder gets a film-grain SEI message it should generate grain noise according to parameters signaled in the SEI message and add the noise to the picture (before display).

Let's suppose that raw data (i.e. pixels) already contains grain noise. In such case one can reduce the noise by applaying a noise-removal filter (e.g. pseudo-inverse filter). Then one should encode this noise-free frame with signalling the film-grain SEI (in order to signal to a decoder to syntesize the removed noise).
Of course the above method has a disadvantage - noise removal might cause over-smoothing of the original picture.Yes, this has been discussed a number of times, but it's a chicken-and-egg situation. Almost no decoders support it; while HD-DVD mandated it, HD-DVD's death has led to most decoders dropping support. No open source decoder supports it either (libavcodec). So until someone writes support, it's useless.

... but if nobody writes support, we can't test it in x264...

[Shevach]

Almost no decoders support it; while HD-DVD mandated it, HD-DVD's death has led to most decoders dropping support. No open source decoder supports it either (libavcodec). So until someone writes support, it's useless.

... but if nobody writes support, we can't test it in x264...

Good answer although disappointing.
This is challenging to encode video material with film-grain while preserving the film-grain noise.

There is an article (published Panasonic, by the way the idea of film-grain SEI message was of Panasonic):
"Quantization Offsets for Video Coding", by Thomas Wedi and Steffen Wittmann, Circuits and Systems, 2005. ISCAS 2005.

In the above paper are discussed methods clever quantization (adaptive dead-zones and offsets) in order to preserve film-grain.
I think the paper can be interesting for this discussion.