Improving grain in dark-gray areas.

[tombstone]

This is the first time I'm posting and I have searched exhaustively in archives on this and other sites for the answer to this, so please don't berate me.

So far, I've been very happy with the picture quality I've been able to obtain with the x264 encoder, but I'm stuck on one issue. Poor film grain on dark gray areas on 'some' movies (not all films are affected). One particular example is an old movie I'm trying to archive called The Ten Commandments, with Charleton Heston (I own the movie, I buy all my movies on Blu-Ray, and merely want to cut it's size down while maintaining as much quality as possible for my main Home Theater PC).

The scene with the old Pharoah where Moses is in chains (around 1:33:05 in the movie). The room has many high contrast shaded/bright areas. The film grain just isn't right behind Moses in that scene on the medium & dark gray pillar (there are other moments in the film where this happens as well, not just here, but that pillar really stands out as the best example). I have to bump the bitrate up to 20Mbps and turn off MBTREE just to get it 'close' to the original film grain noise level, otherwise there just isn't enough random noise describing that part of the video. There should be WAY MORE than enough bits to describe that dark area but it seems like the x264 encoder is flat out refusing to apply bits to it. The brighter areas of the film have perfect film grain with MUCH lower bitrates, a fraction of the bitrate, but the dark-gray (even some regular gray) areas are pretty much getting ignored by the encoder. It's very distracting to me.

In the vast majority of my other movies, I have gotten perfect film grain, seemingly even in the gray areas, but on a few select, stubborn movies, there is a problem with film grain on shaded medium/dark gray areas. Does anyone have a potential fix for this other than shutting off MBTREE and brute-force bitrate? There should be enough bits for those areas but the encoder just won't distribute them on some movies.

And yes, I have my home theater calibrated for correct black level, so that should not be an issue.

[xooyoozoo]

Unfortunately, I think your best bet is manually using `--zones` to give a bitrate multiplier to problematic scenes, as x264 doesn't have luminance aware AQ.

IIRC, the problem is that there hasn't been a good, generalized proposal for light-based AQ, though I've always wondered how easily color difference formulas could be tweaked and applied in an encoding setting. They adapt to picture lightness and focus on quantifying noticeable differences, so for example, one could try to effectively flatten these curves:

http://i.imgur.com/ymnbYNV.jpg

[Emulgator]

What I am doing in such scenarios is adding grain using GrainFactory3,
(just a tiny bit, and do not exclude high frequencies, just 1 tick helps)
until it tickles x264 just enough to finally spend the desired bits onto shadows and almost blacks.

This helps to get rid of wet spots (where locally noise has been exempt from encoding), residual banding as well...
and delivered very acceptable blu-ray masters.

Yes, I was suggesting such a thing (luminance-aware AQ) as well back then
(hank315 has implemented this in HCEnc and it works there),
but it was said by the main developer that it wouldn't make much sense to code the same approach into x264.

[benwaggoner]

Isn't quality in textured low luma areas what --aq-mode 3 is supposed to address? Still flagged experimental, but maybe worth a try.

[Lyris]

When I've encoded movies for BD, I've also wanted to "trick" x264 into not compressing dark scenes too much. Heavily compressing dark scenes makes a lot of sense when bandwidth is at a premium, but when you have 50gb to play with I'd rather forego it... also, it only takes an LCD-based display viewed off-axis to show up the heavily compressed dark areas. (Not that that is x264's fault!)

A lot of the time I've ended up using these settings to get a CBR-ish result:

--no-mbtree --ratetol 0.0 --qcomp 0.0

I wouldn't advise that in your case when you're only going to 20mbps, but adjustment of ratetol and qcomp might bring you some sort of result?

[benwaggoner]

When I've encoded movies for BD, I've also wanted to "trick" x264 into not compressing dark scenes too much. Heavily compressing dark scenes makes a lot of sense when bandwidth is at a premium, but when you have 50gb to play with I'd rather forego it... also, it only takes an LCD-based display viewed off-axis to show up the heavily compressed dark areas. (Not that that is x264's fault!)

A lot of the time I've ended up using these settings to get a CBR-ish result:

--no-mbtree --ratetol 0.0 --qcomp 0.0

I wouldn't advise that in your case when you're only going to 20mbps, but adjustment of ratetol and qcomp might bring you some sort of result?
The VC-1 encoder for HD DVD and Blu-ray could spend up to 25% of its bits on the 16-30 luma range with typical film stock sources, in order to avoid any blocking or visible quantization on those blasted early LCDs.

Getting great 8-bit from noisy sources is quite labor intensive. Good preprocessing can help more than anything the encoder can do.

10-bit may not be perfect, but it does help a lot.

[Lyris]

Interesting to know :)

Although, it's not just the early LCDs either from my experience, the problem is still there. Makes me pine for plasma :(

Especially on the larger sized LCDs, the display gamma is often only correct (or close to it) in the center of the panel, getting brighter towards the edges.

[tombstone]

Unfortunately, I think your best bet is manually using `--zones` to give a bitrate multiplier to problematic scenes, as x264 doesn't have luminance aware AQ.

IIRC, the problem is that there hasn't been a good, generalized proposal for light-based AQ, though I've always wondered how easily color difference formulas could be tweaked and applied in an encoding setting. They adapt to picture lightness and focus on quantifying noticeable differences, so for example, one could try to effectively flatten these curves:

http://i.imgur.com/ymnbYNV.jpg

I just did a test with my staxrip frontend and used the deadzone command line with 6,6 at around 9.5 Mbps and compared it with my old encode test at 9.5 Mbps and there is an improvement in the gray areas for film grain (this is with trellis still 'on'). It's not perfect, but "busier" looking, and just as before, the "brighter" areas have "perfect" film grain.

I will try deadzone 3,3 next. This is with MBTree turned 'off', which I found is an absolute 'must' to improve the grain on this movie. Many other movies have grain that comes out fine, even in gray areas with MBTree turned on, so I'm scratching my head as to 'why' this movie is different than others.

I would really like the idea of "light-based AQ". Right now, there is no "one size fits all" setting, because no one has written a detection method that determines when there is grain, anime, smooth/flat areas, etc and when to tell the encoder to change its parameters to compensate. Users are having to encode, evaluate, then use their experience and judgement from past successes to improve the results of the next encode.

[tombstone]

Isn't quality in textured low luma areas what --aq-mode 3 is supposed to address? Still flagged experimental, but maybe worth a try.

I've got 3 choices in Staxrip:

Disabled
Variance
Auto-Variance

Is that experimental option one of these? If not, then what custom command do I add in Staxrip to enable this? I would like to test this.

[jpsdr]

To use aq mode 3, you first need to get a non official version build of x264 (like t_mod version).
After, the command line to add is --aq-mode 3.
I don't know on gui interface how you can replace it.
I personnaly don't use gui and always encode with a .bat file.

[Sparktank]

non official version build of x264 (like t_mod version)...
I don't know on gui interface how you can replace it.
I personnaly don't use gui and always encode with a .bat file.

In my expereience, replacing the vanilla x264 version provided with the bundle-ware is that it's not always compatible.
I tried (to replace the binraies) with a few (bundles) and the process always aborted prematurely with errors in the x264 process stage.

Since, x264 is open-source, I suspect the developers of the bundled software do some modifications to (not only) the x264 binary but (most probably) the GUI that works with everything to only recognize the vanilla (provided) x264, and aborts operations if the check against the provided x264 doesn't match.

Like you, I don't use GUI's that much (at all) and rely on batch scripts with different builds of x264 (tMod).

[tombstone]

In my expereience, replacing the vanilla x264 version provided with the bundle-ware is that it's not always compatible.
I tried (to replace the binraies) with a few (bundles) and the process always aborted prematurely with errors in the x264 process stage.

With Staxrip, it seems that if I setup the encode and save the project, then rename the old x264 file to .old, then change the new x264 file to x264.exe, then the encoder seems to start. Odd. I haven't verified this with experimental encoders however. I just tried this with success last night using a newer Staxrip 1.1.9.0 and using the newest x264 version that I dropped into the folder and renamed files.

Question, why doesn't Staxrip use the 64-bit version of x264? That's got to be making things potentially slower being stuck with 3GB of memory.

[tombstone]

To use aq mode 3, you first need to get a non official version build of x264 (like t_mod version).
After, the command line to add is --aq-mode 3.
I don't know on gui interface how you can replace it.
I personnaly don't use gui and always encode with a .bat file.

Shucks, I just started a long encode last night with --aqmode 3 on the newest r2431 version in Staxrip 1.1.9.0. lol. And it's running. I got no errors, so I figured I'd just let it run. Funny. Do you have a link to download an experimental encoder? I'd be willing to try it out.

I've done all my encoding so far with Staxrip and I'm open to doing this another way, but need to find a little tutorial for the command line to use and what programs to put into the same directory. Currently, the Staxrip frontend does a lot in terms of muxing automatically that I haven't done by command line. So, while I'm sure I could figure it out, it's a bit of a pain working with the old command line method until I iron out the steps needed and get used to making command lines (which shouldn't be difficult by now for me).

I mean, I can figure it out. I've figured our how to use eac3to to encode DTS @ 786kb/sec, which is my favorite quality/size ratio choice for the sound on my movies. I don't hear any difference between 768kb/sec and 1.5Mb/sec. But I do hear a difference between Dolby Digital and DTS. I prefer DTS on most material and only rarely use Dolby Digital, mostly for "drama's" and old movies with limited surround usage where lower bitrate doesn't affect quality. But that's another discussion.

Anyway, do you have step by step instructions I could use?

[jpsdr]

Anyway, do you have step by step instructions I could use?

It's realy not easy and long, for apparently someone not used to more low-level work/tools. A lot of peoples, a lot of different ways of doing things.
It's not step by step, but globaly, personnaly :
- I rip my Blu-Rays with AnyDVD.
- I created dgi files with DGIndex from m2ts files.
- I open the dgi files in avisynth scripts, and do all my pre-processing with avisynth/VirtualDub filters,
and save the result final files in lossless YV12 avi file (UT Video, expect around 120GB/150GB for a 2h anime movie without noise, never done live and if there is noise i denoise, so it may be bigger for live of with noise).
- I create the avisynth script i'll give to x264 with the finals result files (on this final step, the avisynth script use only AVISource, and eventualy "+").
- Encode with x64 version of x264 using only .bat files. Need Avisynth+ for x64 avisynth version.

An exemple of an encode file :

CALL x264_x64_FHD-1.bat Noir_BR-1 31988 K:\ animation Noir_01-05_Chapters_x264.txt
CALL x264_x64_FHD-1.bat Noir_BR-2 32032 K:\ animation Noir_06-10_Chapters_x264.txt
CALL x264_x64_FHD-1.bat Noir_BR-3 32032 K:\ animation Noir_11-14_Chapters_x264.txt
CALL x264_x64_FHD-1.bat Noir_BR-4 31988 K:\ animation Noir_15-18_Chapters_x264.txt
CALL x264_x64_FHD-1.bat Noir_BR-5 36008 K:\ animation Noir_19-22_Chapters_x264.txt
CALL x264_x64_FHD-1.bat Noir_BR-6 36008 K:\ animation Noir_23-26_Chapters_x264.txt

shutdown -f -s -t 0


Note : This is an exemple of encode targetting Blu-Ray => I want to create a Blu-Ray, and using tmod version.

x264_x64_FHD-1.bat :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET CHAPTERS=%6%5
SET STAT_FILE=%6%1.stats
SET LOG_FILE_1=%6%1_log_1.txt
SET LOG_FILE_2=%6%1_log_2.txt
SET BITRATE=%2

if %4==animation (
set AQS=--aq-mode 3 --aq-strength 0.7 --aq2-strength 0.8 --aq2-sensitivity 17 --aq3-mode 1 --aq3-strength 0.35 --deblock 1:0 --psy-rd 0.6:0.0
SET TUNING=animation
) else (
if %4==cartoon (
set AQS=--aq-mode 3 --aq2-strength 0.8 --aq2-sensitivity 17 --aq3-mode 1
SET TUNING=animation
) else (
set AQS=--aq-mode 3
SET TUNING=%4
)
)

REM 1rst pass
x264_x64.exe --device bluray --preset placebo --tune %TUNING% --pass 1 --bitrate %BITRATE% --stats %STAT_FILE% --keyint 24 --min-keyint 1 --mvrange 511 --ref 4 --bframe 3 --slices 4 --b-pyramid strict --open-gop --fade-compensate 0.8 --subme 7 --me umh %AQS% --aud --nal-hrd vbr --colorprim bt709 --transfer bt709 --colormatrix bt709 --videoformat ntsc --sar 1:1 --qpfile %CHAPTERS% --threads 14 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2nd pass
x264_x64.exe --device bluray --preset placebo --tune %TUNING% --pass 2 --bitrate %BITRATE% --stats %STAT_FILE% --keyint 24 --min-keyint 1 --mvrange 511 --ref 4 --bframe 3 --slices 4 --b-pyramid strict --open-gop --fade-compensate 0.8 --subme 10 %AQS% --aud --nal-hrd vbr --colorprim bt709 --transfer bt709 --colormatrix bt709 --videoformat ntsc --sar 1:1 --qpfile %CHAPTERS% --threads 14 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%

[tombstone]

Oh, and another issue I found with dark gray scenes. The movie Cowboys & Aliens at 43:04, we have a scene where the cowboys are on their horses approaching a boat sitting upside down the land in front of them. The scene is 'dim' with a mix of both Film Grain AND rain together in a dark-gray and multi-shaded back round that 'shifts' as the rain is blown by the wind.

This scene appears to be inadequately encoded by the SOURCE Blu-ray. I don't believe the makers of this Blu-Ray gave this scene enough bits to describe it and it's pretty obvious the original encode settings/encoder version they used STRUGGLED to do this visually complex scene any justice. And if you think the original Blu-Ray must of have struggled with the scene, well x264 gets busted as completely inadequate to duplicate the Blu-Ray here, even at excessive bitrates like 14Mbps - 18Mbps. I tried even super high encodes and no dice on my 'normal' settings.

[foxyshadis]

Did you give zones a try? You should be able to use --zones 61954,64000(?),crf=12 or something like that, maybe even lower if necessary. Then the rest of the movie isn't hit with the ludicrous bitrate penalty that one scene needs. Maybe also add ,nr=100 to clean it up a little bit too, or filter it lightly in Avisynth.

That won't fix the wrong-match problem, but at least it'll cover it up better.

[shae]

Getting great 8-bit from noisy sources is quite labor intensive. Good preprocessing can help more than anything the encoder can do.

10-bit may not be perfect, but it does help a lot.
How/why does 10-bit help with dark content?

in order to avoid any blocking or visible quantization on those blasted early LCDs.The current LCDs can be just as blasted. Viewing angles? What's that? :-/

[benwaggoner]

How/why does 10-bit help with dark content?
You get 4x the gradations in that difficult low black. For example, instead of having a visible banding/contouring between Y'=17 and Y'=18, you've got Y'=67, 68, 69, 71, 72, 73, and 74 to cover the same band of the luma range. So gradients instead of contours.

[Asmodian]

Also, as part of the same effect, if the error from compression changes a 67 into a 69 it is less noticeable than changing a 17 into an 18.

[Mr VacBob]

Also, as part of the same effect, if the error from compression changes a 67 into a 69 it is less noticeable than changing a 17 into an 18.

The sRGB gamma curve was supposed to solve this, i.e. gamma-corrected 8-bit video is supposedly already visually linear. That's why you can have dark images in 8-bit at all and keep any detail.

But it may not be quite good enough, and the loss of the 0-16 and 240-255 codes in video doesn't help either. However, nobody ever produced a sound theory for which codes were the "least linear" and therefore most sensitive ones, which is why x264 doesn't have any brightness-based AQ.

[Asmodian]

Visually linear and visible steps aren't mutually exclusive with 8-bit. I can see faint banding throughout a full screen 8-bit gradient.

On a perfect display I don't think increasing the bit-rate near black is necessary as errors have the same visual significance at any brightness. Sadly no one has a perfect display.

Most LCDs have very visible steps near black which makes the improvement due to using 10-bit more significant near black. This is also the reason for BT.1886, BT.1886 compresses the steps more near black and expands them near white but by changing the gamma with brightness instead of simply increasing the gamma (allowing the use of the standard gamma in the mid range).

I wonder if comparing a pure power or sRGB gamma curve to a "stock" BT.1886 type gamma curve would be a way to implement a brightness-based AQ. Use the difference in the gamma curves as a weight, near black would be weighted higher and near white would be weighted less. This is optimizing AQ for the display technology instead of the content; it would look better on the BT.1886 display that gamma curve was from but worse on a perfect pure power display.

[shae]

You get 4x the gradations in that difficult low black.Is this just to have less severe rounding during the intermediate decoding steps? How come higher bitrate is another way to improve the situation if the problem is actually not enough precision? And if 10-bit helps or is needed, why not 12- or 16-bit?

[foxyshadis]

Is this just to have less severe rounding during the intermediate decoding steps? How come higher bitrate is another way to improve the situation if the problem is actually not enough precision? And if 10-bit helps or is needed, why not 12- or 16-bit?

12- and 16-bit are slower and need more memory, it's a tradeoff. Even 10 has seen minimal adoption because of that. The reason higher bitrate usually works is that noise overwhelms the precision problems, but lower bitrates remove more noise. (If noise has been filtered out, or it's a CG project made without grain, obviously bitrate won't matter.)

[benwaggoner]

Is this just to have less severe rounding during the intermediate decoding steps? How come higher bitrate is another way to improve the situation if the problem is actually not enough precision? And if 10-bit helps or is needed, why not 12- or 16-bit?
Less than good enough isn't good enough, but more than good enough is still good enough :)? That's the answer to many mysteries of life, actually.

Anyway, the dynamic range of Rec. 709 is such that human vision is generally happy with 10-bits of precision. If you want to do super high dynamic range without some kind of nonexponential gamma function you need at least 12 bits.

[shae]

If custom QMs in h264 are considered standard enough, can this be used to improve banding by keeping more noise, and without using 10-bit?

And BTW, how's that 10-bit output as 8-bit in the end? Just rounding, or dithering?

If ... it's a CG project made without grain, obviously bitrate won't matter.Is the common solution here is to add noise/dither prior to encoding (assuming one wants to avoid 10-bit)?

Less than good enough isn't good enough, but more than good enough is still good enough :)? That's the answer to many mysteries of life, actually.Quite a mysterious answer.

the dynamic range of Rec. 709 is such that human vision is generally happy with 10-bits of precision. If you want to do super high dynamic range without some kind of nonexponential gamma function you need at least 12 bits.So 12 is enough for Rec.2020?

[foxyshadis]

In the end it all gets converted to RGB, but if you can keep it in 10+ bits from start to the final RGB 8-bit conversions, you can minimize banding. 10-bit RGB is very rare, heck even full 8-bit monitors aren't universal, but MadVR and MPDN support it on compatible hardware. And yeah, dithering or noise are used whenever you want to hide banding.

Custom matrices really don't help much. Keeping extra noise means sacrificing detail elsewhere.

Rec. 2020 is nearly twice as wide as old gamuts, so you need 1 more bit to keep a similar amount of banding. 10-bit would look like 9-bit 709; still better than 8-bit, but wasting the extra precision. Not as noticeable if you're actually working in deep color from shoot to final display.