calm function

[mathmax]

Hi

Could anyone help me to understand this function?


function Calm (clip clp, int "rep0")
{
ox = clp.width()
oy = clp.height()

# create linearily weighted temporal averaging
ts1 = clp.temporalsoften(1,255,255,28,2)
ts2 = clp.temporalsoften(2,255,255,28,2)

# construct temporal gaussian average from linear averages
t = ts1.merge(ts2,0.357).merge(clp,0.125)

# Create clip for motion search.
searchclip = t.removegrain(11).gaussresize(ox,oy,0,0,ox+.0001,oy+.0001,p=2)
\ .merge(t,0.1).mt_lutxy(t3,"x 7 + y < x 2 + x 7 - y > x 2 - x 51 * y 49 * + 100 / ? ?",U=3,V=3)

return (searchclip)
}


Especially I don't understand these parts:
- construct temporal gaussian average from linear averages
- create clip for motion search

What are gaussian average & gaussresize()? What are they used for?

Thank you in advance for any help.

[Didée]

That's a code snipplet from my TGMC function. Terka extracted that snipplet and made a separate function out of it, in order to use it for his own scripting investigations.


"temporal gaussian average"

When doing a simple TemporalSoften(), then several frames are merged together (for pixels that don't difer more than a threshold). E.g. when calling TemporalSoften(2,10,15,20,2), then the pixels of frames (current-2), (current-1), (current), (current+1) and (current+2) will be merged together. Ech frame has equal weighting, i.e. there are 5 frames merged with a weighting of 1:1:1:1:1. That's what I called "linear weighting".

For the gaussian case, a different weighting is used. Instead of 1:1:1:...., the weighting rather is 1:2:1 (for 3 pixels), or 1:4:6:4:1 (for 5 pixels).

In the context of TGMC, I needed to use gaussian weighting because that is what fits to the actual problem, namely: calming of bob flicker/shimmer. Normal linear weighting would not have worked correctly for this purpose. (Imagine a pixel sequence of 50-100-50-100-50-100-.... with linear averaging, it will still flicker. With gaussian averaging, the sequence is calmed-down to a static value.)


"create clip for motion search"

That's a construct that is specifically "designed" to prepare the calmed clip for the motion search engine of MVTools. I ultimately needed to reduce the local contrast of the search clip to get a working result in the end. By the nature of things, the SAD values gathered by/after motion search are relatively big in areas of detail - when there is e.g. a black line on a white background, even the slighest mismatch of the motion engine will quickly produce a big SAD. Now, the behaviour of MVTools' MDegrain() filter is that it reduces its strength in areas with big SAD, up to completely disabling itself into a do-nothing operation.
In the context of TGMC this is highly counterproductive: areas with "prominent" high-contrast detail are exactly the areas where bob flicker is visible the most. And since removing bob flicker is the main intend of TGMC, it is not wished that MDegrain reduces its strength there. (I cannot have MDegrain disable itself in exactly those places where it is most urgently needed to do its work!)

BTW, in the original function, TGMC, this section is noted as follows:

"# Create clip for motion search.
# Trick: weight-in a gaussian blur, to reduce local contrast of searchclip. (Needed to later make MVDegrain work as expected ...the "SAD concept" doesn't fit to the realities -- so, let's tailor the realities to fit the darn concept ...

[Terka]

yes, its cutted (stolen) part of TGMC.
i use it for clip prefiltering, before motion search.
(sometimes also for progressive content)

[mathmax]

Thank you for this explanation. I'm novice with avisynth script and I don't understand everything..


In the context of TGMC, I needed to use gaussian weighting because that is what fits to the actual problem, namely: calming of bob flicker/shimmer.

First what do you mean by calming bob flicker? What does TGMC mean? Could you show me an example of image with bob flicker?


Imagine a pixel sequence of 50-100-50-100-50-100-....

you mean the luma values?


with linear averaging, it will still flicker. With gaussian averaging, the sequence is calmed-down to a static value.)

ok so you have


ts1 = clp.temporalsoften(1,255,255,28,2)
ts2 = clp.temporalsoften(2,255,255,28,2)

ts2 should be the more soften, then ts1, then clp
Why do you merge in this order


t = ts1.merge(ts2,0.357).merge(clp,0.125)


shouldn't it be:

t = clp.merge(ts1,0.357).merge(ts2,0.125)



"create clip for motion search"

That's a construct that is specifically "designed" to prepare the calmed clip for the motion search engine of MVTools. I ultimately needed to reduce the local contrast of the search clip to get a working result in the end. By the nature of things, the SAD values gathered by/after motion search are relatively big in areas of detail - when there is e.g. a black line on a white background, even the slighest mismatch of the motion engine will quickly produce a big SAD.
Now, the behaviour of MVTools' MDegrain() filter is that it reduces its strength in areas with big SAD, up to completely disabling itself into a do-nothing operation.

I don't really understand this passage... What is a SAD value?

Thank you very much for your help... and sorry for my lacks of understanding.

[Didée]

I'm novice with avisynth script and I don't understand everything..
So it seems indeed. Sorry, but I won't explain all that stuff now. Your questions are covering topics of "advanced" level ... and when the background knowledge is not sufficient, then it's difficult to give sufficient answers.

I don't really understand this passage... What is a SAD value?
SAD is "Sum of Absolute Differences", a measure that is used by MVTools during motion search.

Inform yourself about motion compensation in general, and about MVTools specifically.


what do you mean by calming bob flicker? What does TGMC mean? Could you show me an example of image with bob flicker?
Bob Flicker is what usually comes up when an interlaced source is deinterlaced with a bob filter.
TGMC is my "TemporalGaussMC" filter.

For an example of bob flicker, see e.g. this sample (http://www.mediafire.com/download.php?kzhgmutjgtz). (MediaFire, ~11MB)

You might want to visit the discussion thread of TGMC (http://forum.doom9.org/showthread.php?t=133826). There's quite some interesting information in it, as well as answers to many of your questions.



Why do you merge in this order


t = ts1.merge(ts2,0.357).merge(clp,0.125)


shouldn't it be:

t = clp.merge(ts1,0.357).merge(ts2,0.125)


No. It should be exactly the way as it is now.

The present relative weighting are

clp: [0 0 1 0 0]
ts1: [0 1 1 1 0]
ts2: [1 1 1 1 1]

The resultant combined weighting shall be [1 4 6 4 1].

The merge weightings used by now do result in the wished final weighting. There's no room for tastings or preferences. Its simple math.

***

Without any doubt, I would prefer to use a dedicated temporal-soften-alike filter for this operation, instead of doing this cumbersome kind of manual averaging. But alas, there is no filter that does. The closest would be ttempsmooth, which at least offers some different temporal weightings. But alas, none of the available weightings correspond to gaussian weighting.

Another alternative would be to use the Average() plugin ... much easier to get the correct weightings, and moreover it would be much faster. But alas, with this method there is no handling of scenechanges anymore, which would be a regression.

Summed up, the current state maybe is somewhat complicated, but it's more or less the fastest way to do the intended operations correctly.

[Gavino]

No. It should be exactly the way as it is now.
Actually, it could be done in the order suggested by mathmax, but different merge weights would then need to be used:
t = clp.merge(ts1, 9.0/11).merge(ts2, 5.0/16)
would give the same final result.

[Terka]

how will be the merge, if ts3 or even ts4 will be used?
i know the Pascal triangle, know what weightings are but dont understand how you count the ts1, ts2, clp weightings.
is temporalsoften weihtings are 1/N where N=nr of frames?

[Didée]

A question to ask the math professor. It's easier when using the Average() function to merge them all-in-one-go: sum up total weightings, distribute individual relative weightings, done. When doing it with peacemeal sequences of Merge(), then figuring the correct weightings is a tiny little bit awkward.:devil:

However ... believe me, it is not important. You don't need that. Really, you don't need that. The "gaussian thingy" is only for the bob-flip-flop, and for that, radius=1 is basically enough. Radius=2 already provides more than enough safety margin.

For what it's supposed to do, there is nothing to gain from using even bigger radii.

Don't think that a temporal gauss would be the "better filter" just because it's used in TGMC. Better think about what either a gaussian or a linear filter are doing, about what are the differences/consequences, and about which filter fits to which problem.

If you're after more stabilization for the darn grass:p, then well, stabilize it. But you don't need a temporal gauss for that. For that purpose it mostly creates headaches, but doesn't improve anything.


Example for the easier radius=2 case when using Average():
c1: [0 0 1 0 0] *2 [0 0 2 0 0] {2}
c2: [0 1 1 1 0] *3 [0 3 3 3 0] {9}
c3: [1 1 1 1 1] *1 [1 1 1 1 1] {5}

Sum:[1 2 3 2 1] [6] [1 4 6 4 1] {16}

=> Average(c1,2./16, c2,9./16, c3,5./16)

[Didée]

Another possibility is, of course, to just chain several radius=1 filters:

last.temporalsoften(1,255,255,SC,2).Merge(last,0.25) # now we have radius=1
last.temporalsoften(1,255,255,SC,2).Merge(last,0.25) # now we have radius=2
last.temporalsoften(1,255,255,SC,2).Merge(last,0.25) # now we have radius=3
last.temporalsoften(1,255,255,SC,2).Merge(last,0.25) # now we have radius=4
...etc...

However, while this is "mathematically correct", due to rounding issues it will saturate at some point, similar to how blur(1).blur(1).blur(1)..... saturates for bigger radii.

[Gavino]

i know the Pascal triangle, know what weightings are but dont understand how you count the ts1, ts2, clp weightings.
If you really want to know how to do the maths, it's like this.

Extending the temporal radius to 3 (using a total of 7 pixels), for the Gaussian average we would want to merge them in the proportions [1 6 15 20 15 6 1]*1/64 (from Pascal's triangle)

The outer pixels come only from t3, which has weights [1 1 1 1 1 1 1]*1/7.
So to get from 1/7 to 1/64, we need t3 weight to be 7/64.

Remaining contributions from t2, t1 and c for the other 5 pixels must then be [5 14 19 14 5]*1/64
So similarly we deduce t2 overall weight must be 25/64 (to go from 1/5 to 5/64).

Remaining inner pixels from t1 and c then must be [9 14 9], from which we deduce t1 weight is 27/64 and for c is 5/64.

So we merge c and t1 in ratio [5, 27], merge this with t2 in ratio [32, 25] and finally merge result with t3 in ratio [57, 7].

c.Merge(t1, 27.0/32).Merge(t2, 25.0/57).Merge(t3, 7.0/64)

[Didée]

While we're at math ... it seems my suggested usage of Average is not correct. Darn math!

Stacking the weights in the obvious way:
c: [0 0 0 1 0 0 0] *5 [0 0 0 5 0 0 0]
t1: [0 0 1 1 1 0 0] *9 [0 0 9 9 9 0 0]
t2: [0 1 1 1 1 1 0] *5 [0 5 5 5 5 5 0]
t3: [1 1 1 1 1 1 1] *1 [1 1 1 1 1 1 1]

Sum:[1 2 3 4 3 2 1] [20] [1 6 15 20 15 6 1]

=> Average(c,5./20, t1,9./20, t2,5./20, t3,1./20)

Seems good to me, but I must be thinking wrong at some point. Evaluating with a little script shows that this kind of weighting is not correct:
c = blankclip(pixel_type="YV12",width=320,height=320).showframenumber(true)
# c = interleave(c,c.mt_lut("255 x -",U=3,V=3))

c
t1 = temporalsoften(1,255,255,255,2)
t2 = temporalsoften(2,255,255,255,2)
t3 = temporalsoften(3,255,255,255,2)


t1.merge(c,0.25)
temporalsoften(1,255,255,255,2).merge(last,0.25)
temporalsoften(1,255,255,255,2).merge(last,0.25)
chained=last
chained_diff = mt_makediff(c,chained).greyscale

piecemeal = c.Merge(t1, 27.0/32).Merge(t2, 25.0/57).Merge(t3, 7.0/64)
piecemeal_diff = mt_makediff(c,piecemeal).greyscale

averaged = Average(c,5./20,t1,9./20,t2,5./20,t3,1./20)

stackhorizontal(averaged,piecemeal,chained)
stackvertical(last,stackhorizontal(c.mt_lut("x",Y=-128,V=-128,U=-128),piecemeal_diff,chained_diff))

Where is my error in reasoning?


Edit:
Oh my, yeah ... the weightings really should be summed over the individual elements, not over the "multiplicators" of the "matrices".
Correcting of posted scripts in progress ...... :rolleyes:

[Terka]

Gavino, thank you for explanation, understand it until the last
line:
c.Merge(t1, 27.0/32).Merge(t2, 25.0/57).Merge(t3, 7.0/64)
which i dont understand.
using next line is clear for me.
Average(c,5/64, t1,27/64, t2,25/64, t3,7/64)

[Terka]

Didee, thank you,

For what it's supposed to do, there is nothing to gain from using even bigger radii
ok.
Don't think that a temporal gauss would be the "better filter" just because it's used in TGMC. Better think about what either a gaussian or a linear filter are doing, about what are the differences/consequences, and about which filter fits to which problem.
some time ago i was a bit into math, but i didnt know that gaussian weightnings make the flicker -> constant. (I tried it in excel and it works :) )

If you're after more stabilization for the darn grass, then well, stabilize it. But you don't need a temporal gauss for that. For that purpose it mostly creates headaches, but doesn't improve anything.
how?

[Gavino]

Where is my error in reasoning?
Should be
c: [0 0 0 1 0 0 0] *5 [0 0 0 5 0 0 0]
t1: [0 0 1 1 1 0 0] *27 [0 0 9 9 9 0 0]
t2: [0 1 1 1 1 1 0] *25 [0 5 5 5 5 5 0]
t3: [1 1 1 1 1 1 1] *7 [1 1 1 1 1 1 1]

Sum:[1 2 3 4 3 2 1] [64] [1 6 15 20 15 6 1]

=> Average(c,5./64, t1,27./64, t2,25./64, t3,7./64)

[Terka]

should`t the sum be 64?
edit: late post, Gavino is showing it.

[Didée]

how?

By any method that suits you. There is no "correct" way or method. The fluctuations you are concerned about are just pixel changes, and there's no easy way to separate the good from the bad. Human brain easily separates because of complex evaluation. Dumb programatical filters can't do so.

[Gavino]

Gavino, thank you for explanation, understand it until the last line:
c.Merge(t1, 27.0/32).Merge(t2, 25.0/57).Merge(t3, 7.0/64)
which i dont understand.
Because Merge only takes two clips, we have to merge t1, t2 and t3 one after the other. In the final result, c has overall weight 5/64 and t1 27/64, so they need to be merged in ratio 5:27, ie c.Merge(t1, 27.0/32).

This intermediate result needs weight 32/64 in the final result, while t2 has weight 25/64, so merge them with ratio 32:25 - hence Merge(t2, 25.0/57).
Finally merge in t3 with its weight of 7/64 - Merge(t3, 7.0/64).

[Terka]

yes, yes.
to be sure i will form the question once more.
its not about flicker. degrain (degrain1) and probably v. coherence cause that when there is a small movement of bigger area, sometimes this area doesnot move as whole, but by parts.
tried fft3dfilter, which was able to remove this 'artefact' but.
but blurs alot. tried to have same frame with 1pixel up/dn/left/right shift and do temporalfiltering on them. or corrector on them.
you probably saw my posts.
compared degrain2 and degrain3, degrain3 isnot superior on degrain2 in this case.

[Terka]

Gavino
aha, 5:27, ie c.Merge(t1, 27.0/32). 32=5+27
Thank you, its clear now. Before i didnt see the 5

[Didée]

@Terka - BTW, did you see that latest TGMC version(s) have a "Terka mode" in it? (it's hardcoded-but-deactivated, buried somewhere in the script). :p
## brute-force calming when the motion tool finds too much motion. Better don't use.
# stage99 = stage99.ttempsmooth(2,6,6,2,2,strength=5,pfclip=searchclip)


Another proposal:

Use MMask to mask areas with very small motion (range 0 to 1~2 pixels), then use some strong temporal filter on only those areas. Filter might be TemporalSoften, ttempsmooth, FluxsmoothT, Clense, dfttest(purely_temporal_mode), ...

[Terka]

tried the 'Terka only, better dont use' code, its alredy some time ago. also tried the 4x speedup mode
dfttest-played with it also, but couldnt see it better that fft3d -could you suggest some settings for dfftest?

[Boulder]

Didée,

do you recommend creating a prefiltered clip for analysis when running the 1st pass in Deshaker (which is when the motion estimation is done)? I've got a lot of digitized 8mm material to process and I've been wondering whether it is worth it. As all material was shot using handheld cameras, there is a lot of motion there. There's also quite a lot of visible film grain, spots and scratches in some clips.

Or is there some other way to improve the motion estimation? Speed is not an issue as this is a one-time job only. Although this one time lasts for over 4 hrs :p

[Didée]

@Boulder - I've hardly ever used deshaker, so I'm not sure what it needs or needs not, or what it likes or likes not. Don't know what to recommend.


dfttest-played with it also, but couldnt see it better that fft3d -could you suggest some settings for dfftest?
What comes in your way with FFT3DFilter (always) and dfttest (at default) is the "spatial" processing. Spatial processing is the "main mode" of these filters, and temporal processing just comes as an additional feature. When used, they work 3D (spatial+temporal), but not 2D-temporal.

With FFT3D there's nothing you can do, short of some exhaustive script friggling.

With dfttest, purely temporal operation is achieved by specifying "sbsize=1":
>> dfttest(smode=0,sbsize=1,sosize=1,tbsize=5)

However, dfttest in pure temporal mode is losing lots of it's "magic", becoming ~somewhat similar~ to a plain temporalsoften. (Not the same, but similar.)

[Boulder]

@Boulder - I've hardly ever used deshaker, so I'm not sure what it needs or needs not, or what it likes or likes not. Don't know what to recommend.What about MDepan (MSuper/MAnalyse/MRecalculate)? It seems to be able to produce a log file to feed in Deshaker's second pass. I've not yet seen MRecalculate being used anywhere, but I wondered whether it could be utilized properly when the clip to process is quite noisy and shaky.

[Didée]

Still don't know what to say or advice. Generally, de-shaking is a process that (should) evaluate full-frame. Full-frame evaluation is expected to be pretty robust against noise (unless the noise is *overly* severe).

Hence it depends on the actual source characteristics, and on whether deshaker has or has not any obvious problems due to present noise.

MRecalculate is no universal magic, either. The obvious appliances are a) breaking down from bigger block sizes into smaller blocksizes, and b) compairing block SADs between differently processed clips. Neither of which is closely related to depanning, or to any frame-global evaluation.

I can't tell. There's no universal cure.

[mathmax]

Thank you for your answers.

This calm() function is used in a script posted by Terka in an other thread. It was intended to remove the halos.
I just wonder this function really fit to this script as it was supposed to fix bob flicker...

Especially the second part "Create clip for motion search" should not fit to remove halos, should it?

[Vitaliy Gorbatenko]

Is it possible to make a temporary stabilization, but without temporalsoften? The fact that it generates ghosts and if you use this as reference clip in compensated clip ghosts are also present. I'm talking about DegrainX script which (may ?) is used to suppress flicker.