x264: Film Grain Optimization

[Dark Shikari]

example:
with (http://mirror05.x264.nl/Dark/FGO.png), without (http://mirror05.x264.nl/Dark/noFGO.png)

fprofiled win32 build (http://files.x264.nl/x264.826.fgo.1.exe)
patch (http://pastebin.com/m2e601b27) (updated to work on latest revision and on 64-bit)
explanation (how it works, etc) (http://x264dev.blogspot.com/2008/05/film-grain-optimization.html)

usage:

--fgo, see --help for more info. Requires RDO, i.e. --subme 6 or higher and --b-rdo. Obviously doesn't work well on non-grainy sources.

[Sagittaire]

You use nssd?
Come from libavcodec?

[Dark Shikari]

You use nssd?
Come from libavcodec?Yes, I ported NSSD from libavcodec and drastically simplified the assembly. Other changes include some modifications to RD thresholds.

[IgorC]

Is it good idea if I try to encode with fgo enabled and trellis 2 + deblock -1:-1?

[Dark Shikari]

Is it good idea if I try to encode with fgo enabled and trellis 2 + deblock -1:-1?It should work fine with trellis 2. I haven't experimented much, so feel free to do whatever you want and see how well it works.

[Adub]

So, would this be used instead of your Grain Optimizer? Or in conjunction with?

[Dark Shikari]

So, would this be used instead of your Grain Optimizer? Or in conjunction with?Unrelated, though I really don't think grainoptimizer is that effective nor that reliable.

[ToS_Maverick]

now that you created another mysteriously patch, would you please explain how it is supposed to work? i'm very interested, and can only trust a patch if i know what it does :D

[Blue_MiSfit]

<3!!!!

I've been waiting for this one for a long time!!

I was under the impression that FGM had to be supported decoder side as well?

~misfit

[Dark Shikari]

now that you created another mysteriously patch, would you please explain how it is supposed to work? i'm very interested, and can only trust a patch if i know what it does :DRDO normally uses sum of squared differences as its metric. That is, it rewards modes which approximate the source the most accurately according to that metric, and which don't cost too many bits.

FGO uses NSSD instead. NSSD consists of two components:

1) SSD as before.

2) abs( Noise(Reference Block) - Noise(Coded Block) ) * Weight, where "weight" is the --fgo parameter (this parameter also controls a few other things in FGO).

The resulting score is SSD + ND (Noise Difference).

Noise is measured as follows:

If X(i,j) is a pixel in a block, the noise score for that pixel is abs(X(i,j) - X(i+1,j) - X(i,j+1) + X(i+1,j+1)). This is summed up over all pixels in the block.

In other words, NSSD rewards modes which have a similar amount of noise and approximate the source reasonably well. This noise need not be exactly like the source's noise; it merely needs to be good enough.

I was under the impression that FGM had to be supported decoder side as well?This isn't FGM, its FGO.

[Blue_MiSfit]

ooooo...

right :D

I guess I mis-read it.

I like the results, at any rate!

[mahsah]

The pictures and Binary are down... anyone got a mirror?

Also should this be used with AQ or not...?

and for that matter, how exactly does it work (in layman's terms perhaps? :D)

[Dark Shikari]

The pictures and Binary are down... anyone got a mirror?

Also should this be used with AQ or not...?

and for that matter, how exactly does it work (in layman's terms perhaps? :D)It should be used with AQ, the server isn't down (works fine here), and a somewhat simple explanation was posted above.

In incredibly simple terms: it measures only that the amount of noise in a candidate block is the same as it should be, not that its the same noise as in the source.

[K0zi]

What CQM's would you recommend to use with FGO?
How about prestige? :D

[Dark Shikari]

What CQM's would you recommend to use with FGO?
How about prestige? :DI have no idea; I haven't tried any.

I've never really had good experiences with Prestige.

[ToS_Maverick]

OMG Dark Shikari, you just wrote the XviD-KILLER!!!

everything that was better with xvid is GONE! with FGO i simply got the best encode of the BlackPearl-Sample. looks like 60 mb, at only 32 mb!!!

and forget about matrices, who needs matrices now?

if you don't believe it, here are some screens :D
Left AQ (29,4 MB), Right AQ + FGO (32,1 MB)

http://img291.imageshack.us/img291/8816/74aqsmallcu1.th.jpg (http://img291.imageshack.us/my.php?image=74aqsmallcu1.jpg)http://img232.imageshack.us/img232/7581/74fgosmallqw6.th.jpg (http://img232.imageshack.us/my.php?image=74fgosmallqw6.jpg)

http://img232.imageshack.us/img232/9311/182aqsmallws6.th.jpg (http://img232.imageshack.us/my.php?image=182aqsmallws6.jpg)http://img120.imageshack.us/img120/2057/182fgosmallzz9.th.jpg (http://img120.imageshack.us/my.php?image=182fgosmallzz9.jpg)

and the most amazing image-pair:
http://img241.imageshack.us/img241/2042/590aqsmallwk8.th.jpg (http://img241.imageshack.us/my.php?image=590aqsmallwk8.jpg)http://img241.imageshack.us/img241/3584/590fgosmallns1.th.jpg (http://img241.imageshack.us/my.php?image=590fgosmallns1.jpg)

commandline:

--crf 20.0 --level 4.1 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --b-rdo --bime --weightb --filter -2,-2 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 9781 --vbv-maxrate 29400 --threads auto --thread-input --progress --no-dct-decimate --output "output" "input" --fgo 10

[Sagittaire]

--fgo, see --help for more info. Requires RDO, i.e. --subme 6 or higher and --b-rdo. Obviously doesn't work well on non-grainy sources.

The resulting score is SSD + ND (Noise Difference).



Perhaps SSD problem here (with non-grainy source certainely that ND ~ 0). Why not choose other metric like SAD/SATD if ND ~ 0 or if ND < Threshold. Why not SAD + ND or SATD + ND too.

[Dark Shikari]

Perhaps SSD problem here (with non-grainy source certainely that ND ~ 0). Why not choose other metric like SAD/SATD if ND ~ 0 or if ND < Threshold. Why not SAD + ND or SATD + ND too.SAD/SATD are really measures of bit cost, not distortion.

[Inventive Software]

Wow, Dark Shikari with another wonder-patch that'll surely make it's way to SVN within a matter of weeks! :D

How optimised is it? And do I now have to re-encode most of my movies to fit? :p

[Dark Shikari]

Wow, Dark Shikari with another wonder-patch that'll surely make it's way to SVN within a matter of weeks! :D

How optimised is it? And do I now have to re-encode most of my movies to fit? :pHow optimized?

Its probably erring on the side of caution in the disabling of certain RD thresholds; I'm guessing I might want to reinstate them, just at a higher level than normal.

The "noise" operation is SIMD-optimized but its not nearly as optimized as it could be; it can definitely get faster.

[Ranguvar]

Massive ownage!

This is gonna be fun to play with. Thanks DS!

[Snowknight26]

Would there be any reason not to use FGO for something like transcoding 300?

[Dark Shikari]

Would there be any reason not to use FGO for something like transcoding 300?Sure, what if you wanted all the grain to be removed? :p

[[P]ako]

Awesome! It has even the aq patch... well I shouldn't be surprised about the aq patch, now that it is part of x264 cvs.

[Snowknight26]

Sure, what if you wanted all the grain to be removed? :p

Is it ready enough to be used for non-development environments?

[Dark Shikari]

Is it ready enough to be used for non-development environments?Probably not, but you can try :p

[Dust Signs]

Hi Dark Shikari,

looks like a very nice patch, but there is one thing which keeps bothering me when talking about H.264 film grain... the standard specifies a SEI film grain message which I have not seen yet in any implementation. Wouldn't there be a much bigger gain in quality if the grain was just removed with a filter, the filtered movie encoded, and the grain added with this SEI message? This would save a lot of noise from being encoded with an enourmous amount of bits... just an idea, as I don't know if there are any decoders which consider this SEI message for it is optional.

Dust Signs

[Sagittaire]

Hi Dark Shikari,

looks like a very nice patch, but there is one thing which keeps bothering me when talking about H.264 film grain... the standard specifies a SEI film grain message which I have not seen yet in any implementation. Wouldn't there be a much bigger gain in quality if the grain was just removed with a filter, the filtered movie encoded, and the grain added with this SEI message? This would save a lot of noise from being encoded with an enourmous amount of bits... just an idea, as I don't know if there are any decoders which consider this SEI message for it is optional.

Dust Signs

Decoder (hardware and software) must be compatible with FGM. As far I know it's not the case for libavcodec, coreavc ...

[McCauley]

Hi Dust Signs,

the problem is that there's no way to remove the grain without loosing any details, you can degrain and sharpen afterwards to (re-)amplify the softened details, but you are always altering the picture. If it would be that easy, you could just degrain and apply AddGrain(x.x) during playback to it.

@DS
Thanks for all your work on x264, this patch could be the next big step after AQ.

Regards
McCauley

[CruNcher]

Subjective wise this works really well especially on B-frames where prestige fails even with b-rdo (b-frames) FGO has no problems to keep the grainy Look & Feel constantly (and very visible) throughout every frame very impressive, indeed it fakes the eye really efficient and can even bring back the grain feeling after resizing :)


Prestige = 15.30 fps Global:45.183 Y:0.9806016
RD Prestige = 10.74 fps Global:45.404 Y:0.9799006
RD FGO 15 = 9.40 fps Global:44.612 Y:0.9758523
RD FGO 100 = 9.40 fps Global:44.153 Y:0.9739887


Speed wise it's also not much slower and for sure you can optimize on other options and still keep the grain but get almost as fast as without RD (not with the full detail preservation then but still with the grain feeling) :)

[Dark Shikari]

Hi Dark Shikari,

looks like a very nice patch, but there is one thing which keeps bothering me when talking about H.264 film grain... the standard specifies a SEI film grain message which I have not seen yet in any implementation. Wouldn't there be a much bigger gain in quality if the grain was just removed with a filter, the filtered movie encoded, and the grain added with this SEI message? This would save a lot of noise from being encoded with an enourmous amount of bits... just an idea, as I don't know if there are any decoders which consider this SEI message for it is optional.

Dust SignsFilm Grain Modelling is a great idea but nearly nobody has implemented it (not even JM), so there's very little to go off of. We'd basically be starting from nothing, and we'd need to implement it in libavcodec too.

Subjective wise this works really well especially on B-frames where prestige fails even with b-rdo (b-frames) That's partially because I cheated; FGO lowers B-frame quantizers somewhat to ensure they stay grainy :p

[ToS_Maverick]

That's partially because I cheated; FGO lowers B-frame quantizers somewhat to ensure they stay grainy :p

oh your such a bad boy :p

i tested your new patch in combination with fft3dgpu > gradfun2db > x264. the results were excellent.
the banding is almost reduced to nothing.

your work is really amazing. this is a real great visual upgrade!

[mahsah]

Ah, the links are working for me now, thanks!

Anyway, this looks like a real breakthrough in encoding. The results I'm getting are fantastic!

[woah!]

same settings with --fgo 10 turned on and off



OFF
http://www.cif-forums.com/png/small/FGO_SMALL.png (http://www.cif-forums.com/png/FGO_NO.png)

ON
http://www.cif-forums.com/png/small/FGO_SMALL.png (http://www.cif-forums.com/png/FGO_10.png)

the bitrate used would give me a dvd9 result at 1080p...

[squid808]

Hello Dark,

I'm curious about the reasoning behind the formula for "noise score". Would you mind to elaborate?


Noise is measured as follows:

If X(i,j) is a pixel in a block, the noise score for that pixel is abs(X(i,j) - X(i+1,j) - X(i,j+1) + X(i+1,j+1)). This is summed up over all pixels in the block.

In other words, NSSD rewards modes which have a similar amount of noise and approximate the source reasonably well. This noise need not be exactly like the source's noise; it merely needs to be good enough.

[Dark Shikari]

Hello Dark,

I'm curious about the reasoning behind the formula for "noise score". Would you mind to elaborate?
1. The noise score itself represents the absolute gradient at each point, summed up; this is a relatively good indicator of noise level.

2. Because it doesn't matter exactly what noise we retain as long as its similar, two similar noise scores are a good thing.

[survivant001]

oh your such a bad boy :p

i tested your new patch in combination with fft3dgpu > gradfun2db > x264. the results were excellent.
the banding is almost reduced to nothing.

your work is really amazing. this is a real great visual upgrade!

can you put a screenshoot, I'm curious :)

[Razorholt]

In incredibly simple terms: it measures only that the amount of noise in a candidate block is the same as it should be, not that its the same noise as in the source.

So, with a non-grainy source the picture shouldn't be degraded when using FGO, correct? Also, how FGO will "play" with non-grainy source + addGrain? I guess I should try... :)


Thanks,
- Dan

[elguaxo]

amazing! thanks!!

[Dark Shikari]

So, with a non-grainy source the picture shouldn't be degraded when using FGO, correct?Oh, it should, but not that badly.

[burfadel]

With a non-grainy source, any chance of a film grain threshold? Adaptive strength based on threshold? That would be particularly useful for mixed videos!

I have noticed with the couple of clips I did the file size increased significantly, but I guess thats expected! (the results are great).

I read somewhere that the b-frame decision of x264 could be improved, is this true? Would that affect the FGO? Would storing the main film grain in b-frames and using the principles of your grain optimiser plugin actually work, in an effort to reduce the output size impact (maybe as a different mode to fgo)?

[Dark Shikari]

With a non-grainy source, any chance of a film grain threshold? Adaptive strength based on threshold? That would be particularly useful for mixed videos!If we can get a good measurement of film grain that doesn't have false positives (this method, for example, would get a high "noise" value on a very sharp edge, which would be a problem for very clean sources), that might be possible.
I read somewhere that the b-frame decision of x264 could be improved, is this true?Yes.
Would storing the main film grain in b-frames and using the principles of your grain optimiser plugin actually work, in an effort to reduce the output size impact (maybe as a different mode to fgo)?You mean storing the grain in P-frames? That's actually already done in a sense since the B-frames often simply take the grain from the neighboring P-frames instead of re-encoding their own new grain. FGO makes this much more likely, since the bidirectional mode, which generally gives the best RD score, tends to blur out grain (making FGO less likely to select it).

[MfA]

Hell, someone is being paid this summer to work on improving the b-frame decision ... someone we know?

[Dark Shikari]

Hell, someone is being paid this summer to work on improving the b-frame decision ... someone we know?Its a GSOC project; Robert Deaton will be the student responsible for B-frame decision improvement.

[donis]

Awesome news :thanks:

[cestfait]

on fgo:

I've never liked the flat matrix so much. Always wanted to use it, but it was too prone to banding and blocking.

This patch is a keeper. Spectacular work, Dark Shikari.

:)

[bokonon]

Decoder (hardware and software) must be compatible with FGM. As far I know it's not the case for libavcodec, coreavc ...

Does anyone know if it is supported by the sigma decoder in Network Media Tanks such as Popcorn Hour

[MfA]

the problem is that there's no way to remove the grain without loosing any details
There is no way to do lossy compression without loosing any detail either, there is going to be a crossover point where the saved bits provide a net gain.

[Razorholt]

@DS: I remember you say to enable p4x4 with GrainOptimizer. Do you recommend to do the same with FGO?

Thanks,
- Dan

[Dark Shikari]

@DS: I remember you say to enable p4x4 with GrainOptimizer. Do you recommend to do the same with FGO?

Thanks,
- DanNo, they're totally different; no benefit here to p4x4 really.