New Filter: GrainOptimizer (2.02 -- Bug fixed with certain temporal denoisers)

[Wishbringer]

@Dark Shikari: Does FFT3Dfilter count as Denoiser for your plugin?

[Didée]

Source: strong Grain sample (http://home.arcor.de/dhanselmann/_samples/Alien2_scene1_source.m2v)

Using GrainOptimizer v1.11.

Achieved relative filesizes:

no filters GrainOpt() GrainOpt(str=24) GrainOpt(str=50)

Xvid q2 100% 100.5% 101.2% 100.3%

h264 q18 100% 101.7% 101.3% 97.5%


On this source I get at best minimal bitrate savings with very high strenght.

With default settings of GrainOptimizer, the achieved bitrate *increases*.

Who's to blame -- me, the filter, the source?

[Dark Shikari]

Source: strong Grain sample (http://home.arcor.de/dhanselmann/_samples/Alien2_scene1_source.m2v)

Using GrainOptimizer v1.11.

Achieved relative filesizes:

no filters GrainOpt() GrainOpt(str=24) GrainOpt(str=50)

Xvid q2 100% 100.5% 101.2% 100.3%

h264 q18 100% 101.7% 101.3% 97.5%


On this source I get at best minimal bitrate savings with very high strenght.

With default settings of GrainOptimizer, the achieved bitrate *increases*.

Who's to blame -- me, the filter, the source?
That's extremely strong grain--much stronger than the sources I tested on. Try a very, very high strength value--perhaps 250 or higher--the default settings aren't meant for grain with that insane of a standard deviation. I just tested 250 and there didn't seem to be any particularly noticeable visual artifacting in a quick scan of the encoded stream.

[foxyshadis]

Having a little trouble here myself, I just can't seem to make it strong enough to help much. You can get a copy of the source yourself from:
BlankClip(pixel_type="YV12",color=$808080).AddGrain(1000,0,0)

(Well, it was the next logical step.... j/k :p)

[rfmmars]

Having a little trouble here myself, I just can't seem to make it strong enough to help much. You can get a copy of the source yourself from:
BlankClip(pixel_type="YV12",color=$808080).AddGrain(1000,0,0)

(Well, it was the next logical step.... j/k :p)

Same here not large reduction of grain. When you are using the word "film", to me that means 8mm, Super 8mm home movie film which has large grain, did you really mean "Video" or video made to look like film?

Richard
photorecall.net

[Terranigma]

I had Shikari do a test last night with addgrain through irc, with mild settings ((20,.2,.2): Not as strong as yours, but I did suggest 10,000 in place of 20 =P) with and without grainoptimizer (encoding using x264 with a fixed crf). Without grainoptimizer, he ended up with a file having a bitrate of 25139kbps. Using GrainOptimizer, he ended up with a file having a bitrate of 14553kbps (I'm not sure, but I think he used the default settings of grainoptimizer). I think that's pretty impressive if you ask me.

[Dark Shikari]

I had Shikari do a test last night with addgrain through irc, with mild settings ((20,.2,.2): Not as strong as yours, but I did suggest 10,000 in place of 20 =P) with and without grainoptimizer (encoding using x264 with a fixed crf). Without grainoptimizer, he ended up with a file having a bitrate of 25139kbps. Using GrainOptimizer, he ended up with a file having a bitrate of 14553kbps (I'm not sure, but I think he used the default settings of grainoptimizer). I think that's pretty impressive if you ask me.And improvements are coming--I discussed a few of the finer aspects of inter/intra MB coding with Akupenguin last night and I'm working on some modifications to improve efficiency.

[Fizick]

for real film grain you need in real degrainer (degrainmedian, etc). BTW they will decrease filesize too ... ;)

[Dark Shikari]

for real film grain you need in real degrainer (degrainmedian, etc). BTW they will decrease filesize too ... ;)The entire point of this plugin is to not actually remove the grain--to keep the grainy feel while decreasing the filesize ;)

Ideally it should work on any "level" of grain as long as the grain is weaker than the image itself--I may decide to "linearize" the strength so that you don't have to massively raise to deal with grain that is, say, 2x stronger.

[rfmmars]

for real film grain you need in real degrainer (degrainmedian, etc). BTW they will decrease filesize too ... ;)

Yes I am very happy with your plugins, so far its the best.

Richard

[rfmmars]

The entire point of this plugin is to not actually remove the grain--to keep the grainy feel while decreasing the filesize ;)

Ideally it should work on any "level" of grain as long as the grain is weaker than the image itself--I may decide to "linearize" the strength so that you don't have to massively raise to deal with grain that is, say, 2x stronger.

Ok I should have read more into the name of the plugin. For me file size isn't important at all, so you have answer my question.

Many thanks,

Richard

[burfadel]

Hey Dark Shikari? habe you checked the before and after the resizer thing I mentioned? or is there an issue in the filter that will be fixed with the new improvements beyond 1.11 that you mentioned?

Its great filter btw, simple in concept but exremely effective!

[Raere]

Pardon my stupidity, but by grain, do you mean noise? If so, should I use the optimizer, then a denoiser?

[burfadel]

by grain its means the (usually) slight fuzzy dots which appear in the picture. This optimiser is much better than a denoiser in my opionion, as some noise can actually make the picture look better, at least a very slight noise (heavy noise is different). For heavy noise a denoiser would probably be more appropriate depending on the situation.

The other thing with denoisers is that they often remove some picture quality as well, whereas the optimiser doesn't (not noticeably). The best thing to do is use a short 30 second clip and judge the best thing for your use. If you keep the clip the same, use the denoiser in one, and the optimiser in the other. Running a denoiser and the optimiser together doesn't work properly as the denoiser will annialate the work of the optimiser, and if the optimiser is user after it won't work properly on the now modified blocks.

[MfA]

Instead of copying the blocks from previous frames couldn't you try to copy the noise/grain? Ie. subtract degrained images from the originals and add the result to the following degrained frames for static blocks, and just copy the original for motion blocks. The damage which can be done due to misdetected motion is far less than when you copy previous frame blocks.

[Dark Shikari]

Instead of copying the blocks from previous frames couldn't you try to copy the noise/grain? Ie. subtract degrained images from the originals and add the result to the following degrained frames for static blocks, and just copy the original for motion blocks. The damage which can be done due to misdetected motion is far less than when you copy previous frame blocks.That's a lot more difficult though--defining exactly what the grain is is much more difficult than defining whether or not a block is grainy.

Odds are that would probably create more artifacting due to misdetection of what is and is not grain in a grainy block.

[Didée]

That's extremely strong grain--much stronger than the sources I tested on. Try a very, very high strength value--perhaps 250 or higher--the default settings aren't meant for grain with that insane of a standard deviation.
Of course that is strong grain -- that's where the fun is. Dealing with elfin grain (can only be seen at full moon) is easy ...

I just tested 250 and there didn't seem to be any particularly noticeable visual artifacting in a quick scan of the encoded stream.
... where in contrary, at times there are way too much artfefacts even with the conservative default settings:

source:
http://img232.imageshack.us/img232/6950/sourcegrainoe5.th.png (http://img232.imageshack.us/my.php?image=sourcegrainoe5.png)

GrainOptimizer(strength=8) # default
http://img119.imageshack.us/img119/5998/defaultgrainoptimizerar3.th.png (http://img119.imageshack.us/my.php?image=defaultgrainoptimizerar3.png)

Zoom in - to my eyes, that's one huge 4x4 blocking festival. Isn't it?

[Kurth]

The plugin is not loading here =(

http://img213.imageshack.us/img213/8574/errorhr3.png

Any ideas on how to make this plugin load correctly?

[Boulder]

Try installing the the Visual Studio 2005 SP1 Redistributable Package, it worked for me (forgot to report).

[Kurth]

Thanks Boulder :)

[Sagekilla]

@Didee, I noticed something similar to that myself. I was doing a quick visual inspection through mpc and I ended up noticing in certain areas there was some really funky artifacts going on.

Shikari, any word on why this is happening? I'll see if it happens again and try to post some screenshots or the videos themselves.

[Dark Shikari]

@Didee, I noticed something similar to that myself. I was doing a quick visual inspection through mpc and I ended up noticing in certain areas there was some really funky artifacts going on.

Shikari, any word on why this is happening? I'll see if it happens again and try to post some screenshots or the videos themselves.I'll try to work on it a bit over the weekend if I can.

[burfadel]

This filter is already fast, how much faster do you think it would be once is it optimised? also, did you find out why the filter works better before the resizer? or is that only occurring with me? lol oh, I also don't get any artifacts at 25, or seem to anyway, but when I ramped it up to 250 I noticed that there were bad artifacts on darkish late afternoon sky (with a bit of thin cloud).

[Dark Shikari]

New version released--lots of improvements made. The bug previously mentioned should be fixed. Full improvement list in the original post.

Note that it now seems to work effectively on all sources, even those with no grain, at least bitrate-wise. That means its even useful after, say, FFT3Dfilter :)

[Terranigma]

Is this filter now safe to use, or will Didée show us some more bad examples? =P

[Dark Shikari]

Is this filter now safe to use, or will Didée show us some more bad examples? =PPerhaps, so far it seems fine (Didee's sample clip works fine for me with the new version at 4,25,8,true,8, for example).

[ChrisW77]

Great filter, love it.

Just a couple of questions,

Does this filter work well if you add grain to a source ?

For example, I have a lot of old VHS material, and it looks good if you add grain near the end of the script, as it does a good job of fooling the eyes into thinking your source is actually better than it actually is.

Lastly, what codec do you recommend to use on a grainy source ?
Is x264, able to keep most of the grain, using (for example) a high quality profile ?

[Dark Shikari]

Great filter, love it.

Just a couple of questions,

Does this filter work well if you add grain to a source ?
Yes; better than on the original grain, in fact. My test on artificial grain showed over a 40% bitrate reduction versus without the filter when on the normal grain it gave roughly a 30% bitrate reduction. With the improvements now I'm guessing the reduction could approach 50% depending on the source and settings.
Lastly, what codec do you recommend to use on a grainy source ?
Is x264, able to keep most of the grain, using (for example) a high quality profile ?Depends. With the GrainOptimizer plugin, x264 likely does a much better job than it normally would. Try it out, but make sure to compare to Xvid, which does a very good job on grain retention, though requires pretty high bitrates to do it.

Some other H.264 encoders (payware) have some various forms of grain optimization, also.

[Dark Shikari]

OK, did the test on 500 frames of Cruncher's footage at CRF 18. Denoised with Mocomp'd FFT3DGPU and then AddGrain(20,0.2,0.2)'d.

Result:

No GrainOptimizer: 18379kbps, 0.946 SSIM, 42.035 OPSNR.
GrainOptimizer(4,25,8,true,8): 12499kbps, 0.959 SSIM, 42.857 OPSNR.

[ChrisW77]

Yes; better than on the original grain, in fact. My test on artificial grain showed over a 40% bitrate reduction versus without the filter when on the normal grain it gave roughly a 30% bitrate reduction. With the improvements now I'm guessing the reduction could approach 50% depending on the source and settings.

Thanks for the quick reply. Sounds great.
I was thinking of removing the usual VHS noise with Fitz's excellent mvtools + mvdegrain, than adding some grain to the end of the script. It really works well on this particular piece, but I've never been able to capture the grain with a codec. Xvid, need far too much bitrate just to preserve some of it.

Depends. With the GrainOptimizer plugin, x264 likely does a much better job than it normally would. Try it out, but make sure to compare to Xvid, which does a very good job on grain retention, though requires pretty high bitrates to do it.

Yeah, cheers. I'll probably give x264 a go, probably at a bitrate of around 1500+.

[Dark Shikari]

Turns out there was a major bug with the chroma search that decreased efficiency. This has existed since the introduction of the chroma search but only manifested itself significantly with blocksize 8 (which by the way, proves to be "curiously" effective :) ). Fix will come tonight or tomorrow.

Edit: Perhaps I was premature on the 8x8 block bit. It seems that 8x8 blocks have an inherent problem--they're less effective at detecting very slight motion within the block. Example--if the block is at the border between a slightly lighter and slightly darker area, and the block shifts a pixel to the right due to motion, you'll notice the artifact if the block is retained, so it shouldn't be retained. With an 8x8 block, the amount of area the border covers is 2x larger... but the block is 4x larger, so its "half as good" at detecting the movement. The only way to resolve this IMO is... yet another heuristic, of some sort, that would deal with the issue. 8x8 blocks do give a valid benefit--somewhat more efficient B-frame coding--but until this problem is fixed they're basically useless IMO.

[cestfait]

WOW!! :eek: This filter has SERIOUS potential! I was just noticing how LESS denoised clips are often better compressed than smooth ones with x264. Guess it's just a matter of optimization!

However, I have a problem: In certain dark or flat areas, entire blocks seem to "ghost" in a strange way (similar to problems I have had with high AQ strength and low sensitivity). Here are some screenshots of the animated source I discovered this bug with (although it shows up in other ones I have tried, as well).

Original:

http://img155.imageshack.us/my.php?image=originaliq3.png

With Artefacts:

http://img148.imageshack.us/my.php?image=blockghostinglv1.png

[Yes, I accidentally let it crop and resize the artefacted clip, but I'm pretty sure that's not the problem. Anyway, see below script:]

The problem persists even on lower settings ("grainoptimizer(4,1,1,false)"), although I used defaults for this example. The script for the example was:

LoadPlugin("C:\Documents and Settings\Adam\Desktop\yatta\plugins\dgdecode.dll")
LoadPlugin("C:\Documents and Settings\Adam\Desktop\grainopt.dll")


function Preset0(clip c) {
#Name: Default
c
grainoptimizer(4,8,6,false)
return last
}

DGDecode_Mpeg2Source("C:\Documents and Settings\Adam\Desktop\haibane 01\haibane.d2v")


PresetClip0=Preset0()

PresetClip0.Trim(0,41229)


Crop(10,6,-6,0)
Spline36Resize(704,480)

Another thing I have noticed is that when I preview a frame in YATTA, the frame does not have any artefacting unless I move a few more preview frames in. The problem still shows up in the encode, though.

This filter looks like a wonderful idea, and I would love to start using it on a regular basis! Any idea what could be causing this problem?

EDIT: I was using 1.2, by the way.

[Dark Shikari]

Can you upload a short clip of your original source for me to test on? Use Mediafire if you need a hosting site. I love to find clips where the filter fails--its the best way to improve it :)

The reason that it doesn't show up in YATTA is because the filter is sequential--that is, if you call it on an arbitrary frame, it does nothing; it only does anything if its called for many frames in sequence.

The only way to avoid this would be to call the function recursively in the code, which would be a TERRIBLE idea.

[cestfait]

Wow! That was one hell of a fast reply! :) Forgot to add that I had also tried to MoComp with DePanEstimate, which failed. Then again, I had no idea why I was doing it; just thought I would poke my nose into areas of encoding that I don't understand...

Here's another little something I can't quite make out: How can I cut up my VOB to upload to you?

[Dark Shikari]

Wow! That was one hell of a fast reply! :) Forgot to add that I had also tried to MoComp with DePanEstimate, which failed. Then again, I had no idea why I was doing it; just thought I would poke my nose into areas of encoding that I don't understand...

Here's another little something I can't quite make out: How can I cut up my VOB to upload to you?I think VobBlanker can do it?

[Boulder]

Or simpler: use DGIndex to select a small range, then choose "save project and demux video". You'll get a demuxed m2v file out of that.

[cestfait]

Ok, the clips are up. The "haibane_renmei_clip.m2v" has the ugliest artefacts, but the artefacts manifest in a bunch of... creative... ways in the "mononoke-hime_clip.m2v". I especially think that you should note the panning shot of the ironworks and, right at the end, the shot of the group walking toward the camera. The whole frame shakes there, for the most part... freaky!

Mononoke:

http://www.mediafire.com/?5veythzwdi3

Haibane:

http://www.mediafire.com/?d9fhn1eumlg

[Didée]

Is this filter now safe to use, or will Didée show us some more bad examples? =P
Well, I didn't want to appear as an old nagger who makes everything seem bad, so I waited a while. ;)

v1.2 of GrainOptimizer is a good improvement over the previous versions. Still, issues are not difficult to find.

Perhaps, so far it seems fine (Didee's sample clip works fine for me with the new version at 4,25,8,true,8, for example).
Can't agree, sorry. "Fine" is something else, definetly. Did you try all the samples, or just one or two?

snip1 - no blocking artefacts. (*)
snip2 - artefacts on Mollari's forehead, and in the background when there is slow movement. Scene is degraded.
snip3 - no blocking artefacts. (*)
snip4 - mostly good. At the end of the sequence, there is the "static-curtain" effect on Garibaldi's face.
snip5 - some small artefacts, but visual appearance is good.
snip6 - there appear some bad blocking artefacts on the light beams.
snip7 - blocking / wrong block freezing of the slowly moving textures. A major degradation of the scene, inacceptable.

(*) on these scenes, the "slowing" of the grain in fact is very annoying to my eyes. Instead of the constant deviation of the original grain, there is a "blinking" effect created due to the grain freezing.

My temporary conclusion: for sources with stronger grain like this here, the filter is not well suited:
- when there is grain, I don't like the effect. Somehow reminds me of an encode that suffers bitrate.
- when there is no grain, the filter is still not safe against producing artefacts.
- when the filter mistakes (texture+motion) for grain, the resultant effect is particularly poor.


Regarding bitrate reduction - average over all seven test samples @ q18:

source: 100%
default, strength=08: 88.0% (-12.0%) - slightly degraded
default, strength=16: 84.5% (-15.5%) - slightly more degraded
default, strength=25: 83.0% (-17.0%) - noticeably degraded. Partly OK, partly unacceptable
default, strength=50: 80.4% (-19.6%) - seriously degraded, unacceptable
default, strngth=100: 79.3% (-20.7%) - seriously degraded, unacceptable

A link to the used test samples is in this post (http://forum.doom9.org/showthread.php?p=1051972#post1051972).

For the interested, here is a RAR archive to download (http://maxupload.com/8AC2CAB2) (30MB) that includes:

- source directly encoded at q18
- encode of GrainOptimizer(4,25,8,true,8) at q18

and for comparison:

- encode of a thorough grain remover (WIP snapshot) at q18. Overall bitrate reduction: 45%. (60%-80%% on strong-grain sequences).

[cestfait]

Could any insight be drawn from anyone's knowledge of the AQ patch for x264? For sources without much grain (including the ones I posted), high strengths had a tendency to produce similar lagging or jerking effects, although they did not take the shape of blocks.

[Dark Shikari]

As a result of your clip I made some huge changes to the code, changing a lot of the algorithms and hopefully making it a lot better--and best of all, you don't need to specify the strength anymore--it'll automagically find the correct strength per frame.

The downside is that now, though the result is more accurate, you need a denoised clip as input for the filter.

Improvement with artificial grain using AddGrain():

http://i24.tinypic.com/11smhdg.png

[cestfait]

WOWZER! :eek:

Nice changes! *goes off to test*

By the way, what do you mean by "denoised," to begin with? How [much] denoised? (fft3d, I assume, maybe mocomped, but settings?)

Would you mind posting an example script with the full chain beginning with "denoising" and adding grain and finally your filter?

Thanks again, and good luck on your other new projects (although I think this one is the most groundbreaking :) )

Edit: wow, I'm slow... just got that... still wouldn't mind an example, though.

[buzzqw]

yes, an example and a more accurate readme.txt in zip is more then welcome :D

BHH

[Manao]

How to you compute quality improvments ? If you use the SSIM figure at the end of the encoding process, it's completely bogus since you changed the source...

[Dark Shikari]

How to you compute quality improvments ? If you use the SSIM figure at the end of the encoding process, it's completely bogus since you changed the source...The point of the "quality" measure is that since the filter doesn't change the data in a spatial manner, each frame is individually just as hard to encode as the original, and therefore a higher SSIM means that you could lower the bitrate and achieve the same quality, ignoring the quality change caused by the filter. Basically its making the assumption that the quality change caused by the filter is already considered "acceptable" and is not quantifiable.

This wouldn't be valid for an ordinary denoiser, for example; using fft3dfilter one can easily get 0.99 SSIM at half the bitrate of 0.95 SSIM without denoising, but most of the "lost quality" in the original encode is because of the grain not being flawlessly retained, and therefore that detail is lost, lowering the SSIM. Since the GrainOptimizer, ideally, doesn't remove any grain spatially, this shouldn't be a problem.

Basically the assumption is that the filter does not change the source in a negative manner, or that if it does, it is not quantifiable and is assumed that the user of the filter has no problem with it.

The "quality" measure is not meant to measure deviation from the source--rather its meant to hopefully approximate visual quality relative to what is assumed to be a flawless input. This assumption is invalid if the user dislikes the change in the grain (in which case they shouldn't use the filter!) or if the filter causes spatial artifacts (which it shouldn't, but might).

Anyways, if you don't like the quality aspect of the graph, ignore it and just look at the bitrate aspect--but in my experience it is accurate, as making the grain "easier" to encode results in x264 retaining more of it, increasing quality.

[burfadel]

How do you actually use the new filter? it doesn't seem to work!

If you just put in:
Grainoptimizer()

It comes up with:
Evaluate: System Exception - Access Violation

A few examples on the proper usage would be greatly appreciated!

If you really do need a pre-denoised clip as a reference, and the grainoptimiser after the denoiser, then I'd like to suggest adding another fuction (predenoise will be used in this example), to simplify the process such that: (the dots are commands or a seriouis of commands, <resizer> is your chosen resizer, <denoiser> is your chosen denoiser)
.....
<resizer>
Grainstore()
<denoiser>
Grainoptimizer<>
.....

The purpose of the grainstore command would be just to store the pre-denoised clip in memory, then the grainoptimiser command can work its magic from the avisynth input and the frame stores in memory. This is a more simplistic way I think, and would be easy for anyone to understand how to use.

[Didée]

The syntax is not that difficult:

-----
o = YourSource
d = o.YourDenoiser()

GrainOptimizer( o, d, [parameters] )
-----

[cestfait]

:script:

yeah, I hate to harp on this point, but a sample script with specific filters and motion compensation techniques would be really nice. what kind of a script allows the filter to reference a suitably denoised clip for each frame?

I can imagine referencing a depan/fft3d/ttemporalsoften chain getting pretty arabesque...

EDIT: oops, didée slipped in afore me. still wouldn't mind the creator's feelings on the specific denoisers though... personally, I'm inclined just to do a really strong motion compensated temporalsoften or fluxsmooth for a reference clip...

[Dark Shikari]

Here's a very simple script:

SetMemoryMax(128)
clip1=DirectShowSource("300.divx").ConvertToYV12()
LoadPlugin("GrainOpt.dll")
clip2=clip1.fft3dgpu(sigma=3,bt=3).TTempSmooth(maxr=3,strength=4)
clip1.GrainOptimizer(clip2)

I should add an error catching feature to return an error if a second clip isn't given in the argument (i.e. using GrainOptimizer() instead of GrainOptimizer(clip2) ).

[cestfait]

:thanks:

Cheers! (until the next bug.... just kidding?)

Good work!!!

[squid_80]

I should add an error catching feature to return an error if a second clip isn't given in the argument (i.e. using GrainOptimizer() instead of GrainOptimizer(clip2) ).

Leave required parameters unnamed in the AddFunction call and avisynth will do the error catching for you. For example instead of:
env->AddFunction("GrainOptimizer", "[clip]c[denoisedclip]c[BlockSize]i[strength]f[tdist]i[minrep]i", Some_function, 0);
/* I guess this is what it looks like? */
You want something like this:
env->AddFunction("GrainOptimizer", "cc[BlockSize]i[strength]f[tdist]i[minrep]i", Some_function, 0);
/* any parameters that don't have default values *
* should be unnamed too */