Need Suggestions for VERY GRAINY source

[Terranigma]

I was right to a certain extent, but there's some differences.

MVDegrain2:
http://img218.imageshack.us/img218/9749/mvd2io8.png

2x MVDegrain1:
http://img218.imageshack.us/img218/7426/2mvd1cl2.png

So it seems my way actually removes a bit more noise, so you'll end up with mvdegrain3+

[Sagekilla]

Originally Posted by Terranigma

I was right to a certain extent, but there's some differences.

MVDegrain2:
http://img218.imageshack.us/img218/9749/mvd2io8.png

2x MVDegrain1:
http://img218.imageshack.us/img218/7426/2mvd1cl2.png

So it seems my way actually removes a bit more noise, so you'll end up with mvdegrain3+

Think we're both right and wrong to some extent, since MVDegrain3 does use 6 frames to denoise, but how it does it I have no idea. So I guess that would make your method some sort of.. MVDegrain3.5 huh?

I'm going to do some testing for myself and see which method I like better: Didee's or yours. Then I'm gonna run back and implement the one I like better.

[Terranigma]

Originally Posted by Sagekilla

Think we're both right and wrong to some extent, since MVDegrain3 does use 6 frames to denoise, but how it does it I have no idea. So I guess that would make your method some sort of.. MVDegrain3.5 huh?

I'm going to do some testing for myself and see which method I like better: Didee's or yours. Then I'm gonna run back and implement the one I like better.

Well, I don't use Didee's script, nor the script you posted. I was using another script with 2 mvdegrain calls.

Anyways, here's a screenshot comparison of the normal script with mvd2 vs. this mvdegrain.

mvd2:
http://img297.imageshack.us/img297/8881/mvd2og5.png

mvd3ish:
http://img147.imageshack.us/img147/4870/mvd3tn5.png

[Sagekilla]

Originally Posted by Terranigma

Well, I don't use Didee's script, nor the script you posted. I was using another script with 2 mvdegrain calls.

Anyways, here's a screenshot comparison of the normal script with mvd2 vs. this mvdegrain.

mvd2:
http://img297.imageshack.us/img297/8881/mvd2og5.png

mvd3ish:
http://img147.imageshack.us/img147/4870/mvd3tn5.png

Hmm, neat. I might end up just using yours since it looks like this one does a much better job then the faux MVD3 by Didee. I did a quick visual comparison and there was little improvement through his method in my testing.

[Didée]

@ Terranigma: Basically correct, but not quite like you posted it. In ^^that way, the weightings of the "wing" frames are screwed: Too much weights go to the frames [current-3] and [current+3].

You need to do

degrain_2 = source.mvdegrain2(delta1,delta2,delta1,delta2,idx=2)
degrain_2_1 = source2.mvdegrain1(delta3,delta3,idx=2)

degrain_3 = degrain_2 .merge( degrain_2_1 , 0.4286)

to get the correct weightings.

Originally Posted by Terranigma

So it seems my way actually removes a bit more noise, so you'll end up with mvdegrain3

Without corrected weightings, it will denoise less, not more. (Unless you split the idx'es, then it will denoise more indeed.)


@ Sagekilla: That way is not some funky "denoise some, then denoise some more". It's quite possible to do so.
MVDegrain3 melts 7 frames together at one.
Since Terranigma used the *same* idx values (which in this case is correct), what happens is that the 2nd MVDegrain uses the 1st MVDegrain'ed clip as base, but will average with the not-yet processed wingframes. I.e. first there are 5 frames melted together. Then, in the 2nd step, the missing two frames are additionally melted into the first melting.
When done with the correct weighting, then the result is the same, except for rounding differences.

---

ColorYUV-analyzing a diff, I see min/max 126/130, loose min/max 127/129. That's rounding differences, yep.

[Terranigma]

Originally Posted by Didée

@ Terranigma: Basically correct, but not quite like you posted it. In ^^that way, the weightings of the "wing" frames are screwed: Too much weights go to the frames [current-3] and [current+3].

You need to do

degrain_2 = source.mvdegrain2(delta1,delta2,delta1,delta2,idx=2)
degrain_2_1 = source2.mvdegrain1(delta3,delta3,idx=2)

degrain_3 = degrain_2 .merge( degrain_2_1 , 0.4286)

to get the correct weightings.


Without corrected weightings, it will denoise less, not more. (Unless you split the idx'es, then it will denoise more indeed.)
---

ColorYUV-analyzing a diff, I see min/max 126/130, loose min/max 127/129. That's rounding differences, yep.

Oh ok, so I did another test , and now I see what you mean.

MVDeGrain2:
http://img206.imageshack.us/img206/2051/mvd2origlr5.png

MVDegrain1 x 2:
http://img258.imageshack.us/img258/6622/mvd2nc4.png

The small nuances between the 2 must be what you mean by rounding differences.

[Didée]

@Terranigma: Yeah, whatever you're doing there. BTW did you use the correct weightings to simulate MVDegrain2 by 2xMVDegrain1...?

However ... the reason to make a step-by-step rebuild of MVDegrain3 was more towards adding some features in that MVDegrainX does not have at all. (When going to make suggestions to a dev, ears are wider open when you can show a working solution instead of just boring with theoretical blabla.)

Par ex: Seamless integration of using an alternative filter (resp. an alternatively filtered clip), weighted by thSAD:



Note the motion regions where MVDegrain3 folds its arms & says "no, here I do nothing".
(That's more interesting than discussing how to simulate MVDgr7 with MVDgr1.)

[Terranigma]

Originally Posted by Didée

@Terranigma: Yeah, whatever you're doing there. BTW did you use the correct weightings to simulate MVDegrain2 by 2xMVDegrain1...?

Yes

However ... the reason to make a step-by-step rebuild of MVDegrain3 was more towards adding some features in that MVDegrainX does not have at all. (When going to make suggestions to a dev, ears are wider open when you can show a working solution instead of just boring with theoretical blabla.)

Par ex: Seamless integration of using an alternative filter (resp. an alternatively filtered clip), weighted by thSAD:



Note the motion regions where MVDegrain3 folds its arms & says "no, here I do nothing".
(That's more interesting than discussing how to simulate MVDgr7 with MVDgr1.)

Ahh, so that's what that is: I noticed this the other day With 1 call of mvdegrain, whereas adding a second call totally eliminated it; and I thought that was just because I was using mvdegrain4 (though improperly).. but otoh, raising the thsad did absolutely nothing for the 1 call.

[Didée]

Originally Posted by Terranigma

I noticed this the other day With 1 call of mvdegrain, whereas adding a second call totally eliminated it;

Nope. Those not-processed areas are because block SAD is too high there (bad compensation because ME did not find a good match.)
It doesn't matter how often you call MVDegrain over and over. Not-compensateable areas keep being not-compensateable, block SAD keeps being high, the not-filtered areas keep being not filtered.

If you have scripts that in practice make happen what in theory should not happen, that's fine. But without posting them, nobody can follow.

[Terranigma]

Originally Posted by Didée

If you have scripts that in practice make happen what in theory should not happen, that's fine. But without posting them, nobody can follow.

Well, wouldn't combining mvmask with the analysis solve the error? By the way, since you mentioned non-compensateable, one could surely fix this by using truemotion=false right? I'm doing a frame-by-frame step through as of now to see where the problem lied. I changed some parameters in mvmask, so surely it's going to have some effect on the analysis. Maybe I should revert to what the mask was originally, then try the new mask.

I'll report back with updated info.

[Didée]

Sure, there's lots of tweaking possible with MVAnalyse's settings.
Regarding "not-compensateable": elimination of that problem is of course not possible. Possible is a very noticeable improvement on the problem, by using different settings than the default ones ... truemotion=false makes a difference. searchparam=8 makes a difference. Both together make a big difference.
However: Though a good part of those differences are good ones (better compensation of problematic motion areas), some of those differences are bad ones (losing motion coherence, getting either "swimming" [or just "less pleasing"] denoising in flat areas like sky, fog, etc.)

As always, there is no 'best'. Only different pro's and con's.
Did I ever mention the "you win here, you lose there" proverb ...

[Terranigma]

Originally Posted by Didée

Sure, there's lots of tweaking possible with MVAnalyse's settings.
Regarding "not-compensateable": elimination of that problem is of course not possible. Possible is a very noticeable improvement on the problem, by using different settings than the default ones ... truemotion=false makes a difference. searchparam=8 makes a difference. Both together make a big difference.
However: Though a good part of those differences are good ones (better compensation of problematic motion areas), some of those differences are bad ones (losing motion coherence, getting either "swimming" [or just "less pleasing"] denoising in flat areas like sky, fog, etc.)

As always, there is no 'best'. Only different pro's and con's.
Did I ever mention the "you win here, you lose there" proverb ...

That proverb seems to hold true. I thought a certain clip of mines was a lost cause. I usually use search 3, which in turns, also seems to lock the searchparam at 255 (max) , but there have been downsides as you said: Swimming in once grainy, flat areas (doesn't matter if it's prefiltered and denoised completely), and weak denoising for flat skies (Though I pinned down what was causing the weak denoising for the sky, but never for the swimming.) so search 2 should be just fine for most clips it seems.

Anyways, would it be possible for you to upload a sample of that clip so that I can test this script with mvdegrain with it? After stepping through frame-by-frame, all images seems to be fine all of a sudden (maybe because I can't seem to recall the previous settings i've used completely. )

[Didée]

The samples were uploaded by Zep - see this post and this post.

Originally Posted by Terranigma

I usually use search 3 [...] but there have been downsides as you said: Swimming in once grainy, flat areas (doesn't matter if it's prefiltered and denoised completely)

Inspired by one of Dark Shikari's early posts in this thread (though only after recovering from the sickness when seeing RemoveGrain(19)^2 being used), I tried the almost-forgotten HQDN3D and found it to be a pretty effective way to eliminate remaining flicker from flat areas for the search clip. It's a really tricky thing - in those areas that are [supposed to be] *very* flat, even the most minor flicker will invite the motion engine to follow it. E.g., the "bands" that are often produced by fft3dfilter usually are hopping-around a bit. Although they're mostly only +/-1 pixel value , it's enough to attract motion vectors ...

Originally Posted by Terranigma

[...] so search 2 should be just fine for most clips it seems.

Probably, yes.


P.S.: Added some comments to the script posted in #107. ;-)

[Terranigma]

instead of posting new updated posts, how about just editing your posts with the new code, oh and use size=1?

[Zep]

Originally Posted by Didée

@Terranigma: Yeah, whatever you're doing there. BTW did you use the correct weightings to simulate MVDegrain2 by 2xMVDegrain1...?

However ... the reason to make a step-by-step rebuild of MVDegrain3 was more towards adding some features in that MVDegrainX does not have at all. (When going to make suggestions to a dev, ears are wider open when you can show a working solution instead of just boring with theoretical blabla.)

Par ex: Seamless integration of using an alternative filter (resp. an alternatively filtered clip), weighted by thSAD:



Note the motion regions where MVDegrain3 folds its arms & says "no, here I do nothing".
(That's more interesting than discussing how to simulate MVDgr7 with MVDgr1.)

ok you have got to be kidding me lol I just spent 4 days encoding pass1 and it JUST finished and I went in and looked at that very frame you posted above and sure enough it is showing even worse that what you posted. My guess MVDegrain2 not as good as MVDegrain3 again lol Thank goodness I stopped the 2nd pass because I see other flaws in your posted pic as well as my scans now. In this regard it is worse than my first encode but I didn't notice it because I was too focused on grain! grrr.....

Ok I now have to decide what to do. I do have the pass 1 stats file which should be plenty good no matter how I do pass 2. I think I will try your scripted_MVdegrain3 and run pass2 with it.

I also smacked my head about 2 days into the encode when I realized it was taking so long that what I should have done was use tritical's Twrite() and saved a lossless file so that I could run pass2 from it and not have to redo all the grain removable calcs for pass2. Normally Twrite() (well the encoder and massive HD I/O anyway) is too slow to use for me but when I am doing something like this and just getting 1.1 FPS a second it is more than fast enough and 60 gig lossless VBLE file is no problem for this project :P

now I will go and see just how slow the scripted version is lol
if it is like under 1 FPS oh my this may take a week to encode (UPDATE .3 FPS lol will take 9 days to encode lol)


BTW - what is pelclip for? (or what will it be used for in the future)

thanks


Zep

[Zep]

Originally Posted by Didée

Definetly CRT here. I'll try to avoid LCD until SED comes around (though not sure if I manage to bear up that pledge...)

MVDegrain2 uses 4 frames to denoise the current frame. MVDegrain3 uses 6 frames. 6/4 = 1.5 means that MVDegrain3 is potentially 50% more effective in denoising than MVDegrain2. Upper bound, that is.
So yes, on the strong-grain sequences you'll definetly see more residual flicker when only MVDegrain2 is used.

I tried to re-build MVDegrainX with a script function, mainly because I wanted to plug one or two features into it which MVDegrain does not provide at all. Theoretically, it should be no problem to make it with MVMask+TTempsmooth ... but somehow I don't get similar enough. It's quite similar, but there's something about the weight reduction that just won't line up. Either the script is somewhat less sensitive to errors, or it is more ... but never exactly the same.

Here's the outline of the function: (attention, not all features of MVDegrain are implemented yet)

edit: corrected for proof-of-concept usability.

Didée have you compared frames from the MVdedrain3 to this scripted version? I ask because at least here on my LCD the scripted version is not as sharp and seems to have slightly less detail. Perhaps TTempSmooth is the cause? I'll mess with TTempSmooth settings to tomorrow it is late and I need sleep.

The good news is none of the MVdegrain3 missed areas you showed later on in this thread so I like the scripted version better over all I just want to tweak it a little so that TTempSmooth is not so strong (I think TTempSmooth . like I said willl mess with it tomorrow)


Thanks

Zep

[ankurs]

this is what i got when i tried to use Temporal Degrain V1.07 with all the required plugins(latest versions) in the script ( lots of them .. wish they could be autoloaded )

well thing is this is what i have at line 7 of my script

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\HQdn3D.dll")

never used this filter ever before .. just saw that it was required by the script and downloaded it of the link which was given in the script itself by Sagekilla .. think there is some incompatibility with it of the filters which i am calling in the script ( P.S : Used to have the same problem until placed the ffw3.dll file in system 32 folder and then it started wrking ) no idea if even this requires a file of such sorts to be placed in that folder ? if so then please help me out

[ankurs]

also getting this even though all the required plugins are there ,,



this is the filter loading part of my script

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ColorMatrix.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MVtools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrain.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Repair.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\HQdn3D.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\asharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpenFaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\ylevelsS.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\soothe.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\Temporal Degrain.avs")

??

[ankurs]

the first instance is when HQdn3D.dll was not there in the plugins directory :shy: my mistake but after putting it there i am getting the second error which is posted above this post ..

[Didée]

MinBlur() is a well-hidden little function. Find it e.g. in the script posted in #61 of this thread.