Suggestions: Optimize flat animation for encoding with x264

[xackphair]

1080p BD source with no real grain and light dithering. Ill post some example screen shots later today.

The problem I am having is a lot of the details are chewed up in x264 at crf 17 especially in darker scenes. Without completely modifying my x264 commands, I want find a way to optimize the video with AviSynth for encoding with x264. The only items I would preferably increase is pys-rd and what ever else. Since the animation is so flat (no real details and no grain), it is hard to keep what small details there are. I would either end up with banding or very chewed up dithers. I already run a sharpening mod on it (I don't remember the name of the filter off the top of my head atm; all I know is that it starts with "L".)

I just need some suggestions from people with experience on this problem. I will post avs scripts and x264 command lines if needed. Thanks in advance to those that help.

[ajp_anton]

I've heard adding a little grain might help. Maybe even using some sort of mask to only add it to dark areas to force x264 to use more bits there?

[xackphair]

I've heard adding a little grain might help. Maybe even using some sort of mask to only add it to dark areas to force x264 to use more bits there?

I have never used MaskTools or any other masking filter by itself; I have only used it though mods and what not. Is there any possible way for you to give me an idea how to use it or point me in the direction of some kind guide/tutorial? A sample line or script would work too. I can probably figure it out from that.

Dfttests dither option works great too. Mixed with masking might do the trick for me.

If there are any other suggestion, please shoot. :thanks:

SCREEN SHOTS
x264:
http://sadpanda.us/images/360394-4A4Z4FW.png

Original:
http://sadpanda.us/images/360398-V7SQCEF.png

[Mr VacBob]

Adding high frequency grain won't compress very well and leads to blocking in the output (much worse if it's moving between frames). Try Udeband/Gradfun2dbMod, but modifying x264 settings (giving it more bitrate, stronger AQ, -1 to deblock) is more likely to help.

That said yours looks fine to me already, but it also looks upscaled.

[Emulgator]

GrainFactory3. Mmmh. Scalable Grain sizes and amounts for dark/mid/bright areas.