Interpolating duplicated frames

[lisztfr9]

Hi all,

I was trying out gavino's nice script for interpolating duplicated frames :

# ScriptClip(last, "Subtitle(String(YDifferenceFromPrevious))")


#Interpolation_for_second_frame_of_duplicate_(by_Gavino).avs
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", "2.0")
# ConvertToRGB24() # for easy display in VirtualDub

Sometimes, 2.0 is fine but when Y vary not much, it should be ~0.75... So i was wondering if this could be made adaptive, to the average luma difference ?

YDifferenceFromPrevious > 10 threshold ~ 2

YDifferenceFromPrevious < 7 threshold ~ 1

But the threshold should not jerk too much, i mean i cannot simply made it like f(YDifferenceFromPrevious)...

Well how to write this function ?

Thanks ,

L

[raffriff42]

Try this, I think it's a better dupe detector:

#Interpolation_for_second_frame_of_duplicate_(by_Gavino).avs
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
#ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", "2.0")
S=Subtract(Last, BlankClip(Last, length=1)+Last)
ConditionalFilter(inter.Loop(delta, 0, 0), Last, "YPlaneMinMaxDifference(S, 2) ", "<", "1")
# ConvertToRGB24() # for easy display in VirtualDub

[johnmeyer]

Several quick thoughts ...

1. The point of the function is to interpolate and then replace duplicated frames. By definition, a duplicated frame with have a Ydiff value of zero. So, for true duplicates, any value, including zero, will work.

2. If the two frames are not perfectly identical, but a "visually identical," then you are correct that the Ydiff value can be quite different at various points in the clip, and yet the adjacaent frames may still look the same. My advice is to simply use a larger value for the threshold because if two frames are not perfectly identical, but are really, really close, and you end up replacing one of these "almost identical" frames with an interpolated frame, you probably won't even be able to tell the difference (i.e., you won't create a really bad frame).

3. If you truly want to create an adaptive difference value, I recommend playing around with taking ratios of Ydiff values, using the current frame to last frame, and the last frame to the frame before that (and even more frame pairs, if needed). There are more complex variations on the idea, but the basic concept is that by taking ratios, you end up with a function that only does something when the difference is much less than its neighbors. Depending on your function, and depending on exactly what you are trying to accomplish, you may need to incorporate scene detection logic.

Another variation on the idea is to create a moving average of Ydiff values, and then compare the current Ydiff to that moving average, and only invoke the conditional logic when the current Ydiff exceeds that moving average by a set amount.

[lisztfr9]

Thanks for replying, you'r right, per definition duplicated frames should be identical, but there is some blocking, which rise a little diff.

i added such a function :

function fmin(float f1, float f2) {
return (f1>f2) ? 2 : 1
}

ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceToNext", "<", "fmin(YDifferenceFromPrevious, 8.0)")

i had to change for YDifferenceToNext in order to get it work, - still testing it.

If you are interested, i'm trying to rescue an old movie, Fährmann Maria Which is only available on youtube or in that IQ.

Now at scene changes, the last frame is repeated after the first new one !

[bxyhxyh]

If it is grainy source, difference is always higher and not really dependent to detect duplicates.
So I think testclip should be denoised.
One boolean argument called grainy or something should be there.

[lisztfr9]

Yes, i have also a series of black frames, all seen as dupes... ! And other very static sequence.. transitions with fade out, rise very little Y. I will try raffriff42' suggestion.

[lisztfr9]

[lisztfr9]

@raffriff42

I don't seem to understand how your Subtraction statement works, since i can't see any differences (see 1th pic), but obviously there are (see 2nd pic). And it seems to detect dupes very well !

Blancklip + last is AlignedSplice...

Ok you skip 1 frame

But how to skip black frames ?

L

[lisztfr9]

I was trying some formula like this, but still not working :

video = ScriptClip(video, "Subtitle(String(YPlaneMinMaxDifference(S, 2) + LumaDifference(video, BlankClip($FFFFFF))) )")

To skip black frame, a White Clip to show max difference to black frames...

-----

Blanc = BlankClip(video, $FFFFFF).ConvertToYV12()

video = ScriptClip(video, "Subtitle(String(LumaDifference(video, Blanc)))")

Keeps crashing... ? why

---------------

This is not crashing :

Blanc = BlankClip(Width=Width(video), Height=Height(video), color=$FFFFFF)
Blanc = Blanc.ConvertToYV12()

S=Blanc

video = ScriptClip(video, "Subtitle(String(LumaDifference(Blanc)))")

I guess ScriptClip cannot evaluate LumaDifference(video, Blanc) but leave clip1 as implicit, it works : i get average lumadiff = 218, to black frames (max would be 255). Well, why Scriptclip can't take LumaDiff with 2 arguments ? And there are many better solutions i guess, i'm just puting junk together

[Gavino]

Originally Posted by lisztfr9

video = ScriptClip(video, "Subtitle(String(LumaDifference(video, Blanc)))")

Keeps crashing... ? why

Infinite recursion - see here.

Write it like this:
video = ScriptClip(video, "Subtitle(String(LumaDifference(last, Blanc)))")

EDIT: last can also be left implicit, as you have discovered:
video = ScriptClip(video, "Subtitle(String(LumaDifference(Blanc)))")

[lisztfr9]

Found :

function fmin(float f1, float f2) {
return (f1<f2) ? 100 : f1
}

T = 20

S = ScriptClip(S, "Subtitle(String(fmin(AverageLuma(last), T)))")

To insert in raffriff42' suggestion :

#ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", "2.0")
S=Subtract(Last, BlankClip(Last, length=1)+Last)
ConditionalFilter(inter.Loop(delta, 0, 0), Last, "YPlaneMinMaxDifference(S, 2) ", "<", "1")

I will made the threshold variable, rising it up for black frames.

But as johnmeyer said, a better script should use ratio instead.

------

I was using a denoiser in depan today. I found theses settings working very well, detecting scene changes :

mdata = DePanEstimate(i, dxmax=8, dymax=8, zoommax=1, trust=1.0, stab=1.0)
DePanInterleave(i, data=mdata, info=true)
# FluxSmoothST()
DeGrainMedian(limitY=5,limitUV=7,mode=2)
SelectEvery(3, 1)

[raffriff42]

Yes, BlankClip(length=1)+Clip makes a clip "delayed" by 1 frame. To "advance" by one frame, use Clip.Trim(1, 0). Check the first & last frames of the final video; they will probably be invalid and will need to be trimmed off.

The "Subtract" method breaks down somewhat if noise or moving artifacts are present. If the source is clean, subtracting the previous frame - if it's a duplicate - will give you a plain gray image, and YPlaneMinMaxDifference will be exactly zero. If there's noise present, it won't be zero, but it *should* still be lower on a dupe frame than any non-dupe frame.

For fine tuning the threshold values, you'll want to examine the video, noting the difference values for known good & bad frames with something like this:

## S=Subtract(...
ScriptClip(Last, 
\ """Subtitle("diff="+String(YPlaneMinMaxDifference(S, 2)))""")

...BTW the YPlaneMinMaxDifference threshold argument of "2" was chosen at random; another value might be better. It should be greater than 0 though; not sure why.

If there is grain etc, it helps to denoise the 2 clips being subtracted:

## LP=BlankClip(...
S=Subtract(Last.Blur(1.0).Blur(1.0), LP.Blur(1.0).Blur(1.0))

...or call RemoveGrain or whatever works best for the type of noise you have. The denoised clips won't be seen in the final output, so the processing can be very aggressive.

[lisztfr9]

Very helpful post from raffriff42, in fact i didn't dare to use such a amount of blur. Now using Blur(1.0).Blur(1.0), and only 1 as threshold argument for YPlaneMinMaxDifference, almost all my duplicated frames are detected. I need to deflicker the source, maybe i get even better results.

Edit

o = last
sm = o.bicubicresize(174,88) #25% of source
smm = sm.temporalsoften(1,12,255,24,2).merge(sm,0.25)
smm = smm.temporalsoften(2,7,255,20,2)
o2 = o.mt_makediff(mt_makediff(sm,smm,U=3,V=3).bicubicresize(width(o),height(o),0,0),U=3,V=3)

but i let it as it is... it doesn't improve very much.

[lisztfr9]

I will try it with MVtools.

[lisztfr9]

I'm not sure it performs better as dupes detection than previous methods.

# LOAD PLUGIN :

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ac3source.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\NicAudio.dll")

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\depanestimate.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\depan.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\FluxSmooth.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DeGrainMedian.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mvtools2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\FFT3DFilter.dll")

a1 = ColorBars().Trim(0,399)
a2 = ImageSource("dot.png").ConvertToRGB32

# a2.GreyScale returns a grey dot on a black background;
# Levels makes the dot white
mask_clip = Mask(a2, a2.GreyScale.Levels(0, 0.001, 75, 0, 255))

# Layer(a1, mask_clip)

Overlay(a1, a2, mask=ShowAlpha(mask_clip), y=0, x=0, mode="blend", opacity=1)

ConditionalReader("C:\Program Files\AviSynth 2.5\Examples\xoffset.txt", "ol_x_offset", false)
ConditionalReader("C:\Program Files\AviSynth 2.5\Examples\yoffset.txt", "ol_y_offset", false)

ConvertToYV12(last)

S = last

vectors = MSuper().MAnalyse(isb = false)
MShow(Msuper(), vectors, scale = 1)

Crop(last, 8, 8, -8, -8)

Subtract(S, last)


/*
# cinetic blur :
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
MFlowBlur(super, backward_vectors, forward_vectors, blur=200)
*/

mt_edge(last, "prewitt").Grayscale()

# info()