'mvfps-like function' based on MVFlowFPS?

[WorBry]

Hi Scharfis_Brain,

I originally raised this subject in the following thread,

http://forum.doom9.org/showthread.php?p=758733#post758733

but was diplomatically advised that it would be better addressed under AVISynth usage. So here it is.

My interest is in conversion of bob-deinterlaced 50fps video back to 25fps in a manner that preserves good fluidity of motion.

Theoretically, SelectEven() alone should be sufficient, but, with my archived DV sources at least (all shot in automatic shutter mode), the motion is noticeably jerky. A simple blend or merge of adjacent frames gives fairly fluid motion but is prone to ghosting. Prior interpolation to a higher frame-rate (e.g. 400fps) followed by blending (e.g. over 16 frames) reduces ghosting, but (in my experience) slower moving objects tend to suffer from a slight loss in detail. Given these limitations, I have turned to methods intended to simulate ‘cine-like’ motion blur. Of the methods I’ve tried so far, your (MVConvertFPS based) mvfps function (over-sampling with temporal blur) with improved scene detection (mvfpsscd), has produced the most convincing motion blur with good preservation of detail. However, the encoding is very slow and is not very practical for clips longer than a few minutes. The new MVFlow functions that Fizick has brought to MVTools offer the promise of faster processing with ‘true motion’ compensation, and include an MVFlowBlur tool for simulating ‘cine-like’ motion blur. Certainly, prior treatment with MVFlowBlur() improves on SelectEven() alone but I’m still seeing some motion flicker. Having achieved good results with mvfps(scd) it seemed reasonable to see whether a similar function based on MVFlowFPS could achieve comparable results with faster processing. Allowing for the fact that I am not a ‘developer’ and have a limited knowledge of motion graphics, here is my stab at it:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\MVTools-v1.0.1\MVTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MVBob\MaskTools.dll")

function mvfpsflowscd(clip i, float fps, int "oversample", int "blurradius")
{
oversample=default(oversample,1)
blurradius=default(blurradius,1)
j=i.temporalsoften(2,4,5,2)
forward_vec=mvanalyse(j,isb=false,truemotion=true, pel=2, idx=1)
backward_vec=mvanalyse(j,isb=true,truemotion=true, pel=2, idx=1)
mv=i.MVFlowFps(backward_vec, forward_vec, num=fps*oversample, den=1, ml=100, idx=1).deblock()
global mv= (oversample>1) ? mv.temporalsoften(blurradius,255,255,mode=2).selectevery(oversample,0) : mv
global ch=i.changefps(fps) #. subtitle("ch")

global msk0=ch.duplicateframe(0).reduceby2().reduceby2().motionmask(thsd=255,y=3,u=1,v=1)
global msk1=msk0.trim(1,0)
global msk2=msk0.trim(2,0)
global msk3=msk0.trim(3,0)

f0=scriptclip(ch,"""sc""") #.subtitle(string(a0)+" "+string(a1)+" "+string(a2))""")
f1=f0.frameevaluate("""sc= (sc2 || sc1) ? ch : mv""")
f2=f1.frameevaluate("""sc2=( (a2 > (a1*5+25)) && (a2 > (a3*5+25)) ) ? true : false""")
f3=f2.frameevaluate("""sc1=( (a1 > (a0*5+25)) && (a1 > (a2*5+25)) ) ? true : false""")
f4=f3.frameevaluate("a3=averageluma(msk3)")
f5=f4.frameevaluate("a2=averageluma(msk2)")
f6=f5.frameevaluate("a1=averageluma(msk1)")
f6.frameevaluate("a0=averageluma(msk0)")
}

avisource("C:\Video\MVBob Filmlook 50fps.avi")
ConvertToYV12()
mvfpsflowscd(25,oversample=16,blurradius=4)
#

If this approach seems reasonable, I would be very grateful for any advice you could provide on fine-tuning of the MVFlowFPS and TemporalSoften settings. I will send you the links for the video clips, scripts and images you requested.

Many thanks,

[Mug Funky]

my 2 cents: assuming a perfect motion compensation is used, using it to halve a framerate will always give you the same picture as selecteven/odd. the only difference being slight filling of interpolated lines (which is a job better done with a decent deinterlacer/interpolator).

as far as adding motion-blur goes, you seem to have answered your own question: it looks less jerky, but still juddery when compared to the original.

the only way you're going to get motion as smooth as 50p, is to keep it at 50p.

even on a cinema screen, auto-shutter stuff is going to look jerky. one advantage cinema has though is the blinking of the shutter and a longer persistence than a CRT has - you wont get flicker, and you'll perceive more motion-blur from the movement of your own eyes as they track objects.

i'd say only use mvflowblur when you need to, and only when the source allows - artefacts are still quite common, even though fizick has done incredible things hiding them. a simple situation like an arm passing in front of a brick wall or a face (or a chain-link fence... eek) will completely throw off the mvflow technique, and we currently don't have the means to hide these artefacts. one problem with knowing where motion compensation will go wrong is that you need a near perfect algorithm to find these places, and if you had that, such artefact protection becomes less necessary.

[WorBry]

MugFunky,

i'd say only use mvflowblur when you need to, and only when the source allows - artefacts are still quite common, even though fizick has done incredible things hiding them. a simple situation like an arm passing in front of a brick wall or a face (or a chain-link fence... eek)

Yes, I've noticed that. Below are some frameshots of a mvbob (the new MVFlow based version) deinterlaced clip (arm moving across foreground) converted to 25fps by three different blurring methods.

The mvfpsflowscd script is the one I posted above.

The mvfpsscd scipt I posted here:

http://forum.doom9.org/showthread.php?p=757580#post757580

The MVFlowBlur.SelectEven script:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\MVTools-v1.0.1\MVTools.dll")

a=avisource("C:\mvbobbed 50fps clip.avi").ConvertToYV12()
backward_vectors = a.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vectors = a.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
b=a.MVFlowBlur(backward_vectors, forward_vectors, idx=1)
Return b.SelectEven().Subtitle("MVFlowBlur Pel2 SelectEven")

For both mvfpsflowscd and MVFlowBlur.SelectEven there are two frameshots, one with MVAnalyze set to Pel=2 and the other at default (Pel=1).

As you can see Pel=1 generally results in less distortion and MVFlowBlur.SelectEven produces a slightly cleaner blur than mvfpsflowscd. However, in terms of observed motion fluidity, mvfpsflowscd is pretty much on par with mvfpsscd and encodes much faster. I'm hoping Scharfis or someone can help me out with some fine tuning of the mvfpsflowscd parameters.

[Mug Funky]

one more you can try is (after finding the appropriate Motion plugins by mg262/clouded):

motionprotectedfps(clip.framerate*7,warp).blendfps(clip.framerate*7,aperture=7).selectevery(7,3)

this uses a different motion-search algo, and may be more reliable on some kinds of motion. it's strange that pel=1 and pel=2 can be so different! i wasn't expecting that...

[WorBry]

Thanks for the advice Mug Funky.

As I understand it, BlendFPS simply blends frames to achieve the desired frame rate i.e. doesnt attempt to create a motion blur by interpolation. I had previously taken a similar approach before experimenting with motion blur i.e. increase the framerate from 50fps to, say, 400fps using mvfps (or now MVFlowFPS) and then perform an average merge (luma/chroma) over 16 frames to convert to 25fps, like so:


gg=avisource("C:\Video\MVBob 50fps.avi").ConvertToYV12()
backward_vec = gg.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = gg.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
a=gg.MVFlowFps(backward_vec, forward_vec, num=400, den=1, ml=100, idx=1)

b=a.trim(1,0)
q=a.mergeluma(b,0.5).mergechroma(b,0.5).SelectEven()
#
c=a.trim(2,0)
d=a.trim(3,0)
r=c.mergeluma(d,0.5).mergechroma(d,0.5).SelectEven()
#
e=a.trim(4,0)
f=a.trim(5,0)
s=e.mergeluma(f,0.5).mergechroma(f,0.5).SelectEven()
#
g=a.trim(6,0)
h=a.trim(7,0)
t=g.mergeluma(h,0.5).mergechroma(h,0.5).SelectEven()
#
i=a.trim(8,0)
j=a.trim(9,0)
u=i.mergeluma(j,0.5).mergechroma(j,0.5).SelectEven()
#
k=a.trim(10,0)
l=a.trim(11,0)
v=k.mergeluma(l,0.5).mergechroma(l,0.5).SelectEven()
#
m=a.trim(12,0)
n=a.trim(13,0)
w=m.mergeluma(n,0.5).mergechroma(n,0.5).SelectEven()
#
o=a.trim(14,0)
p=a.trim(15,0)
x=o.mergeluma(p,0.5).mergechroma(p,0.5).SelectEven()
#
aa=q.mergeluma(r,0.5).mergechroma(r,0.5).SelectEven()
#
bb=s.mergeluma(t,0.5).mergechroma(t,0.5).SelectEven()
#
cc=u.mergeluma(v,0.5).mergechroma(v,0.5).SelectEven()
#
dd=w.mergeluma(x,0.5).mergechroma(x,0.5).SelectEven()
#
ee=aa.mergeluma(bb,0.5).mergechroma(bb,0.5).SelectEven()
#
ff=cc.mergeluma(dd,0.5).mergechroma(dd,0.5).SelectEven()
#
LAST=ee.mergeluma(ff,0.5).mergechroma(ff,0.5).SelectEven()
#
FINAL=LAST
Return(FINAL)
#

Please dont cringe at the very elementary method I used for the frame merging. I'm sure there is a more elegant way of presenting it (i.e. BlendFPS), but it worked. As I commented above, the resulting motion is very fluid and ghosting is minimized but it tends to reduce the sharpness of slower moving objects.

I'll experiment some more with MotionProtectedFPS but I've done some quick comparisons with MVFlowFPS and my first impressions are that they achieve quite similar results. Attached, for example, are frameshots obtained with the MotionProtectedFPS/BlendFPS method you suggested, the above script, and the same with MVFlowFPS replaced by MotionProtectedFPS. I also did the same substitution with the 'mvfpsflowscd' script that I posted at the beginning of this thread, taking the liberty to call this function 'mpfpscd' (hope that's OK Clouded?) i.e.

avisource("C:\Video\MVbobbed 50fps.avi")
ConvertToYV12()
mpfpsscd(25,oversample=16,blurradius=4)
#
function mpfpsscd(clip i, float fps, int "oversample", int "blurradius")
{
oversample=default(oversample,1)
blurradius=default(blurradius,1)
mv=i.MotionProtectedFPS(fps*oversample,warp)
global mv= (oversample>1) ? mv.temporalsoften(blurradius,255,255,mode=2).selectevery(oversample,0) : mv
global ch=i.changefps(fps) #. subtitle("ch")

global msk0=ch.duplicateframe(0).reduceby2().reduceby2().motionmask(thsd=255,y=3,u=1,v=1)
global msk1=msk0.trim(1,0)
global msk2=msk0.trim(2,0)
global msk3=msk0.trim(3,0)

f0=scriptclip(ch,"""sc""") #.subtitle(string(a0)+" "+string(a1)+" "+string(a2))""")
f1=f0.frameevaluate("""sc= (sc2 || sc1) ? ch : mv""")
f2=f1.frameevaluate("""sc2=( (a2 > (a1*5+25)) && (a2 > (a3*5+25)) ) ? true : false""")
f3=f2.frameevaluate("""sc1=( (a1 > (a0*5+25)) && (a1 > (a2*5+25)) ) ? true : false""")
f4=f3.frameevaluate("a3=averageluma(msk3)")
f5=f4.frameevaluate("a2=averageluma(msk2)")
f6=f5.frameevaluate("a1=averageluma(msk1)")
f6.frameevaluate("a0=averageluma(msk0)")
}


Again I've attached a frameshot. The results (and encoding times) are pretty similar to those obtained with 'mvfpsflowscd' (frameshots attached with my last post) although with a touch more distortion. It might be noted that I omitted ''pre-vector softening'' with TemporalSoften from the 'mpfpsscd' script a) because I wasnt quite sure where to put it (?? maybe before FindMotion in the ProtectedMotionFPS function) and b) in case it upset the function.

I'll experiment with these alternatives some more. Again I would appreciate any expert advice on the settings.

Thanks again.

[WorBry]

Well, after much testing, I've finally settled on the above 'mpfpsscd' function as my method of choice for conversion of my 50fps (mv-bobbed DV) sources (shot in automated shutter mode) to 25fps with motion blur. Whilst Scharfis_Brain's original mvfpsscd function, (based on MVConvertFPS) appears a little less prone to distortions than the 'true motion compensation' alternatives (MotionProtectedFPS and MVFlowFPS), 'mpfpsscd' processes much faster, and after conversion to DivX 6 (Extreme, 2-pass, 3200kbps, no B-frames, GMC, Optimized H263) I am achieving superior quality metric results, at least in terms of DRF values (measured with DRF Analyzer). Pre-smoothing of the motion analysis by inclusion of TemporalSoften before FindMotion in the MotionProtectedFPS improves the DRF results still further, at a slight expense of encoding speed. Here's the script:

avisource("C:\mvbobbed 50fps source.avi")
ConvertToYV12()
mpfpsscd(25,oversample=16,blurradius=4)
#
function mpfpsscd(clip i, float fps, int "oversample", int "blurradius")
{
oversample=default(oversample,1)
blurradius=default(blurradius,1)
mv=i.MotionProtectedFPS(fps*oversample,warpfast)
global mv= (oversample>1) ? mv.temporalsoften(blurradius,255,255,mode=2).selectevery(oversample,0) : mv
global ch=i.changefps(fps) #. subtitle("ch")

global msk0=ch.duplicateframe(0).reduceby2().reduceby2().motionmask(thsd=255,y=3,u=1,v=1)
global msk1=msk0.trim(1,0)
global msk2=msk0.trim(2,0)
global msk3=msk0.trim(3,0)

f0=scriptclip(ch,"""sc""") #.subtitle(string(a0)+" "+string(a1)+" "+string(a2))""")
f1=f0.frameevaluate("""sc= (sc2 || sc1) ? ch : mv""")
f2=f1.frameevaluate("""sc2=( (a2 > (a1*5+25)) && (a2 > (a3*5+25)) ) ? true : false""")
f3=f2.frameevaluate("""sc1=( (a1 > (a0*5+25)) && (a1 > (a2*5+25)) ) ? true : false""")
f4=f3.frameevaluate("a3=averageluma(msk3)")
f5=f4.frameevaluate("a2=averageluma(msk2)")
f6=f5.frameevaluate("a1=averageluma(msk1)")
f6.frameevaluate("a0=averageluma(msk0)")
}


function MotionProtectedFPS(
\ clip input, float FPS,
\ int "mode", int "protection",
\ int "iterate", int "reset", int "initialise")
{
mode = default(mode, warpfast)
protection = default(protection, 80)
iterate = default(iterate, 2)
reset = default(reset, 50)
initialise = default(initialise, 6)

originalFPS = input.framerate()
input

mp = temporalsoften(2,4,5,2).FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise)
p = MotionFPS(FPS,mode,Motion = mp)
maskp = mp.SumStretchFPSMask(FPS, protection)

mn = temporalsoften(2,4,5,2).FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise, from = next)
n = MotionFPS(FPS,mode,Motion = mn, source = next)
maskn = mn.SumStretchFPSMask(FPS, protection, source = next)

maskp
assumeFPS(originalFPS)
CombineFPS(FPS, maskp, maskn.Invert())
bilinearresize(width*8, height*8)
crop(0, 0, input.width(), input.height())
fity2uv()

maskedmerge(p, n, last, u=3, v=3)
}

As for the crude blending (Merge Chroma/Luma) method that I posted above, I realized afterwards that I could have achieved the same result with BlendBob as follows:

a=avisource("C:\MVBobbed 50fps source.avi").ConvertToYV12()
b=a.MotionProtectedFPS(25*16,warp)
c=b.AssumeBFF().BlendBob(forcematch=1)
d=c.AssumeBFF().BlendBob(forcematch=1)
e=d.AssumeBFF().BlendBob(forcematch=1)
Return e.AssumeBFF().BlendBob(forcematch=1)
#
function MotionProtectedFPS(
\ clip input, float FPS,
\ int "mode", int "protection",
\ int "iterate", int "reset", int "initialise")
{
mode = default(mode, warpfast)
protection = default(protection, 80)
iterate = default(iterate, 2)
reset = default(reset, 50)
initialise = default(initialise, 6)

originalFPS = input.framerate()
input

mp = temporalsoften(2,4,5,2).FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise)
p = MotionFPS(FPS,mode,Motion = mp)
maskp = mp.SumStretchFPSMask(FPS, protection)

mn = temporalsoften(2,4,5,2).FindMotion(iterate = iterate, reset = reset,
\ initialise = initialise, from = next)
n = MotionFPS(FPS,mode,Motion = mn, source = next)
maskn = mn.SumStretchFPSMask(FPS, protection, source = next)

maskp
assumeFPS(originalFPS)
CombineFPS(FPS, maskp, maskn.Invert())
bilinearresize(width*8, height*8)
crop(0, 0, input.width(), input.height())
fity2uv()

maskedmerge(p, n, last, u=3, v=3)
}

Indeed this is my method of choice after mpfpsscd when I need a faster encode or in the rare cases that mpfpsscd doesnt quite achieve the 'judder-free' fluidity of motion that I am looking for.

With that, I have now set myself the task of converting 250GB of archived DV footage. Should keep me busy for a few weeks.

[Buddy Casino]

I tried using your mpfpsscd script with MVtools 1.3.0, AVISynth 2.5 and VirtualDubMod 1.5.10.1, but I got the following error:
Avisynth open failure:
I don't know what warpfast means
[...] Line 9
[...] Line 4

Any idea what the problem could be?

[WorBry]

The script uses mg262(Clouded)'s 'Motion' tools plugin instead of MVTools. Here's the link for the last posted (I think) version:

http://forum.doom9.org/showthread.php?p=750037#post750037

'mpfpsscd', as I called it, merely replaces MVConvertFPS (from MVTools) in Scharfis_Brains mvfpsscd (mvfps with improved scene detection) function with mg262(Clouded)'s 'MotionProtectedFPS' script.

I'm still very happy with the results, although for PC playback I've come to prefer leaving my deinterlaced (mvbob) and denoised (hqdn3d/RemoveGrain) DV sources at 50fps (as an FFDShow HuffYuv-YV12 intermediate) before converting to XviD using AutoGK at 75% quality. This invariably results in every second frame being enoded as a B-frame, which lowers the average quant, but not with any noticable loss in visual quality and the fluidity of 50fps motion is preserved without blurring.