Another 60i-to-24p routine with 'true' motion blur

[actionman133]

Hey all,

Having recently discovered MVTools (hats off to the guys who contribute to it, it's awesome!) and its potential to change framerates, I've been looking out to try and create a better 60ito24p converter. I'm aware of MVConv () which is very effective with smoothing motion inconsistencies, but still suffers from the high shutter speed effect; not enough motion blur.

Here is a potential solution. Using the core of MVConv (), rather than downconverting 60 to 24, this filter UPconverts to 120 fps. It then grabs every 5 frames and blends them together to simulate motion blur. The weighting for the 5 frames are as follows... 0.1, 0.2, 0.4, 0.2, 0.1. This allows the motion blur to fade out at its extremities.

To use the function on a 60p source...
p60to24p (clip, motionblur = true)

Since the render time for motion blur is insanely long (thanks to MVTools and the huge upsample), I've Divx'd two test clips, one with motion blur and one without. They are available at:

http://actionman133.isa-geek.net:8080/preview/preview.html

Function p60to24p (clip full, int "blksize", int "pel", int "search", \
int "searchparam", int "idx", bool "truemotion", bool "motionblur") {
LoadPlugin ("x:\...\mvtools.dll")

# error check variables
blksize = Default (blksize, 8)
blksize = (blksize == 4) || (blksize == 8) || (blksize == 16) ? blksize : 8
pel = Default (pel, 1)
pel = (pel == 1) || (pel == 2) ? pel : 1
search = Default (search, 2)
search = (search == 1) || (search == 2) || (search == 3) ? search : 2
searchparam = Default (searchparam, 1)
idx = Default (idx, 1)
idx = (idx < 1) ? 1 : idx
truemotion = Default (truemotion, false)
motionblur = Default (motionblur, false)

# idx's allow multiple instances of function in script with any integer
idx1 = (idx * 2) - 2
idx2 = (idx * 2) - 1

# cropping for 2nd pass - depends on blksize
cropped = (blksize == 4) ? full.Crop (2, 2, -6, -6) : \
(blksize == 8) ? full.Crop (4, 4, -4, -4) : \
(blksize == 16) ? full.Crop (8, 8, -8, -8) : Last

f1 = full.MVAnalyse (blksize, pel, truemotion = truemotion, \
isb = false, idx = idx1)
b1 = full.MVAnalyse (blksize, pel, truemotion = truemotion, \
isb = true, idx = idx1)
f2 = cropped.MVAnalyse(blksize, pel, truemotion = truemotion, \
isb = false, idx = idx2)
b2 = cropped.MVAnalyse(blksize, pel, truemotion = truemotion, \
isb = true, idx = idx2)

p24 = full.MVFlowFPS2 (b1, f1, b2, f2, 24000, 1001,\
idx = idx1, idx2 = idx2)
p120 = full.MVFlowFPS2 (b1, f1, b2, f2, 120000, 1001,\
idx = idx1, idx2 = idx2)

# everything above this point is basically MVConv (), slightly optimized.
# now apply motion blur via Merge ()

a = p120.SelectEvery (5, 0)
b = p120.SelectEvery (5, 1)
c = p120.SelectEvery (5, 2)
d = p120.SelectEvery (5, 3)
e = p120.SelectEvery (5, 4)

ae = Merge (a, e) # final weight on frames - 0.1
bd = Merge (b, d) # final weight on frames - 0.2
abde = Merge (ae, bd, 2/3.0)
abcde = Merge (abde, c, 0.4) # final weight on - 0.4

Return (motionblur == True) ? abcde : p24

}

[foxyshadis]

Scharfis has made several routines in this area...

http://forum.doom9.org/showthread.php?p=615027#post615027
http://forum.doom9.org/showthread.php?p=684808#post684808

I tried to modify mvfps to fully take advantage of new mvtools routines a while back, but I broke it instead; but without doing that you should be able to use:

function mvfps(clip i, float fps, int "oversample", int "blurradius")
{
oversample=default(oversample,1)
blurradius=default(blurradius,1)
j=i.temporalsoften(2,4,5,2)
fwd=mvanalyse(j,isb=false,truemotion=true,idx=1,lambda=4000)
bwd=mvanalyse(j,isb=true, truemotion=true,idx=1,lambda=4000)
i.mvflowfps(bwd,fwd,idx=1,num=fps*oversample*1000,den=1000)
(oversample>1) ? last.temporalsoften(blurradius,255,255,mode=2).selectevery(oversample,0) : last
}

and call it with mvfps(24,2,2) to approximate your filter (I think). Weighting the frames differently could be done by tweaking the last line. Yours looks like the extension of mkvfps into mvflowfps2 and frame-weighting territory, along with giving you more tweaking options, so it's not bad at all, it's just that forcing it to one pattern of framerates restricts it pretty severely.

There's also an mvflowblur built-in now, but I don't think it works as well as mvfps.

[actionman133]

Oh yeah, I vaguely remember mvfps but I never really liked it's blur function too much. and the MVFlowBlur seems to be quite simple at the moment and in my opinion, not very effective, either.

I know I restrict my filter's general usefulness by forcing the 24/120 fps output, but I wanted this to be specialised in 60-24 conversions.

One thing I'm thinking about, is the possibility of avoiding the two step process of acquiring motion vectors. Currently, you have to MVBob () to 60p, then MVConv () to 24p. That would give the conversion a decent speed boost, since motion vectors are quite cycle-hungry.

Would it be possible to trim that down to a single step? By the looks of it, it *should* be possible, although it's a little confusing. I've been looking at MVBob (), but a lot of it goes over my head. I don't understand what Corrector does or how it works, for instance... anyone able to explain that and it's two modes would be a great help, as the help file is very full of jargon...

Thanks

[foxyshadis]

mvbob now comes with securedeint in the script as its pre-mocomp bobber, and it's quite effective at avoiding artifacts. You could speed it up still more by using type=0-2 (bob,tomsmocomp,kernelbob) instead of the default 3 (eedi2). Since you're blurring away a lot of the fine extra detail in motion where mvbob really shines, securedeint should be plenty.

Alternately for even more speed you could just use leakkerneldeint, tdeint, tomsmocomp, or even bob, directly, but you risk quite visible artifacts.

Corrector really just limits the effects of the motion-compensation, to pick the best mix of several different compensations, and limit the creation of artifacts and blocking, iirc. There isn't much in the way of documentation on it, it's sort of a one-off made for mvbob.

I think it would be fairly trivial to generalize your method for applying the blur into something that would work just as well for any given framerate, but, I'm much too tired at the moment to think that hard. ^^

[Fizick]

mvflowblur is probably simple, but is the only true motion blur. :)

[Korben]

The best 60 -> 24 converter I ve seen.
Good job actionman

[Korben]

I used your script with truemotion = TRUE.
Thank you.

[actionman133]

Your welcome, Korben. Glad it was of some help

[scharfis_brain]

actionman133, you are creating stepped-motionblur with your script.

Creating correct motion blur depends on the present motion blur of the source.
Film like motion blur is always 1/2 the time of one frame. this mean you got a shutter speed of 1/48 sec.

examlpes:
1) your source has 1/60 sec shutter speed. here you only have 2 options to create relistic (uniform) motion blur: merging two adjacent sourch frames into each other simulating 1/30 sec shutter.
This means: up-interpolate to 120 fps, blur each frame with its pre-predecessor and then select every fifth frame:
input.mvconv(120)
merge(last.duplicateframe(0), last.deleteframe(0))
selectevery(5, 2)

2) with 1/120 sec shutter footage you can do more film like motion blur, cause you have these option for simulating motion blur:
1/60 sec (merge 2 frames)
1/40 sec (merge 3 frames) # this is the most filmlike here!
1/30 sec (merge 4 frames)

so here is the pseudocode for 1/40 sec shutter simulation with 1/120 sec footage on input:
Input.mvconv(120)
merged=merge(last.duplicateframe(0), last.deleteframe(0))
merge(merged, last, 0.33) # or 0.667 forgot the steering direction of merge's wheighting...
selectevery(5,2)

with higher shutter speeds at the input you need to use a higher intermediate interpolated framerate.
eg.: 1/240 sec shutter requires mvconverting to 240 fps and then blurring 5 frames (temporalsoften(2,255,255)) and then selectevery(10,5)

even shorter shutter speeds are better handled via mvflowblur() cause it will be faster and more precise in these cases.

so actionman133, if you like to create an ultimate motionblurring script you might consider to implement several modes of motionblur depending on a shutter speed parameter.
(I#ve been too lazy to do so ;) )

[actionman133]

Yes, Scharfis I was thinking the same thing just now.

I'm actually thinking of a far more insane motion blur algorithm that might hopefully be customisable. I'm thinking I could get 1/60 sec shutter to look like 1/48 sec shutter...

But one question about the first code segment you posted, scharfis... shouldn't it be:
input.mvconv(120)
merge(last, last.deleteframe(0))
selectevery(5, 2)

Using DeleteFrame () AND DuplicateFrame () would create a gap between the merged frames and expose the effect, wouldn't it? Just wondering if you made a small mistake or if I'm missing a crucial bit of information that I oughta know. :confused:

I'll have a look at this tomorrow, see if I can work this out a little more...

[scharfis_brain]

you cannt create uniform 1/48 sec motionblur out of 1/60 sec footage.
You may achieve Pyramidal motion blur using mvflowblur.

you have to use delete & duplicateframe(0) the same time with 1/60 sec footage, cause after interpolation to 120 fps you'll get 50% overlapping motionblurs.
and selecting every other frame will result in touching motion blurs. Or differently spoken:
at 60 fps with 1/60 sec shutter, the motion bludr is as long as the frame duration.
at 120 fps with 1/60 sec shutter the motion blur is TWICE as long as the frame duration. that's why one needs to skip one frame during merging frames at 1/60 sec shutter footage.

[actionman133]

If your source is 60p, and you upconvert to 120p via MVFlowFPS, then it simply adds a midpoint frame between every existing frame. So to skip a frame would simply skip the interpolated frame and merge the two frames that already existed, wouldn't it? And if so, why bother upconverting in the first place?

I was actually relying on the overlapping motionblur for my filter. This helped the individual frames be less distinguishable because there are more of them.

This is a draft of what I came up with for a test.

'a' is the regular use of MVConv for the conversion - equivalent to MVConv (24, 1) or (24000, 1001)

'p24' is a slight variation on my original motionblur algorithm, except that it now lacks the overlapping motion blur from the earlier version. This might help 60i video look like 24p video with a 1/24th shutter (like Collateral, perhaps).

'p48' should properly imitate film. Film shutter speed (1/48) is 25% longer than video (1/60). To imitate this, it blends two interpolate frames together. The first is frame 'x', and the second is interpolated 25% towards frame 'x+1'. This should expand the motion blur by a factor of 1.25, thus making 1/60 look like 1//48. As such, this is currently designed to only work with video with a shutter speed of 1/60.

I know this goes against your stepping idea Scharfis, but a couple tests that I did on my limited amount of NTSC footage showed no noticeable artifacts. Would you (I assume you have plenty of interlaced clips based on your avatar ;) ) be able to run this through a few clips, and see how it looks?

AssumeFPS (60, True)
MVConv (240, 1)

a = SelectEvery (10, 0)
b = SelectEvery (10, 1)
c = SelectEvery (10, 2)
e = SelectEvery (10, 4)
g = SelectEvery (10, 6)
i = SelectEvery (10, 8)
p48 = Merge (a, b)

ac = Merge (a, c)
eg = Merge (e, g)
p24 = Merge (ac, eg)

# this final line is for comparison purposes only...
a + p24 + p48

[scharfis_brain]

If your source is 60p, and you upconvert to 120p via MVFlowFPS, then it simply adds a midpoint frame between every existing frame. So to skip a frame would simply skip the interpolated frame and merge the two frames that already existed, wouldn't it? And if so, why bother upconverting in the first place?

You didn't got it ;) I'll try it by visualizing:

60p Source with 1/60 shutter:

A B C D E F G H I J (Source frames are A-J)

after interpolating to 120 fps (interpolated images are a-j):

A a B b C c D d E e F f G g H h I i J j

Now we need to select every fifth frame to get back to 24fps:

A c F h

In order to create non-overlapping/seamless motionblur I blur every other frame and the result is:

AB cd FG hi

As you can see: I still need the interpolated images.

[Alain2]

Sorry to introduce myself in your conversation, but I find this interesting (never thought about it before) and I just want to make sure I understand...

Your method Sharfis will give you a 24p with motion blur equivalent to a 1/30th shutter speed from the 1/60th source, isn't it? Why do we want 1/30th equiv motion blur, as you seem to say that 1/40th is film-like ?

[scharfis_brain]

because you have a compromise to make:
1) make 1/40 sec overlapping motionblur, which is close to the shutter time of 1/48 sec of Film
or
2) make 1/30 sec non-overlapping motionblur which is uniform motionblur as film produces it.

[actionman133]

Okay Scharfis, I see how your method works now.... thanks for that.

But is there actually a problem with the overlapping motion blur, that I'm using? You've said that it shouldn't be done, but you haven't mentioned a reason why, and I'm curious. The samples I've tested have looked fine (although as I said, my testing environment is limited), so I don't find any issues with it. How is it 'wrong' or 'bad'?

I always thought overlapping the motion blur would help smooth out the contours at the edges of moving objects to make the blended frames less noticeable (which might happen in high motion footage with blending 2 or 3 consecutive frames together, like that simple 24p converter script you put together a while ago).

[scharfis_brain]

Here is an example with artifical addad motionblur an a 1/60 sec shutter video:
http://home.arcor.de/scharfis_brain/samples/motionblur.png

of course it depends on personal preference, which kind of motion blur one prefers.

[actionman133]

Aah, I see now... it appears as though non-uniform motionblur only becomes apparent when an small objects in the frame move quickly (like the triangular formation in your sample frame)... the rest of the non-uniform overlapping frame seems pretty good.

Thanks for helping me out, Scharfis. I'll take what I've learnt and see what I can come up with...

[XviDEncoder]

I've been messing with this stuff, and I'm going for a non-overlapping motion blur. My camera has shutter speeds of 60, 100, 250, and beyond.
I'm going for a shutter of 100, mvflowfps > 95.904fps (4x23.976). blur two frames together (temporal soften), selectevery 4.
Avisource("C:\Documents and Settings\Compaq_Administrator\Desktop\Cap.avi") #1/100th shutter speed
converttoYUY2()
Import("C:\Program Files\AviSynth 2.5\plugins\mvbob\mvbob.avs")

mvbob()

backward_vec = MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
cropped = crop(4,4,-4,-4) # by half of block size 8
backward_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, pel=2, idx=2)
forward_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, pel=2, idx=2)
MVFlowFps2(last,backward_vec,forward_vec,backward_vec2,forward_vec2,num=95904,den=1000,idx=1,idx2=2)

temporalsoften(2,255,255,mode=2)
#merge(last.duplicateframe(0), last.deleteframe(0))
selectevery(4, 2)

I can't seem to get a non-overlapping motionblur using merge. Is there any reason using merge is better than temporal soften?
anyhow, i have a feeling i'm not doing things the best way in the first place.

[2Bdecided]

I wondered if anyone had a new take on this?

I'm trying to drop 50i (filmed at a high shutter rate) down to 24p with appropriate simulated 1/48th shutter motion blur.

Tried...

mcbob()

source=last

backward_vec = source.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1, divide=2)
forward_vec = source.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1, divide=2)
source.MVFlowFps(backward_vec,forward_vec,num=120,den=1,idx=1)
#source.MVFlowBlur(backward_vec,forward_vec,blur=50.0,prec=1,idx=1)

temporalsoften(2,255,255,mode=2)
selectevery(4,0)


The motion blur just isn't convincing. Look at an object that's small compared with the amount of blur, and you can usually see the start and end of the blur more clearly than the centre - or there's a bit within the blur that's clearer than the rest. You can tell it's multiple frames added together (badly), rather than real blur.

I tried going up to 240fps, softening over 4, and selecting 1 in 8, but it doesn't help much at all.

Does anyone have any suggestions/ideas?

Cheers,
David.

[WorBry]

You may achieve Pyramidal motion blur using mvflowblur.


Sorry to dig up an old chestnut, but in describing the motion blur acheived by mvflowblur as 'pyramidal', does this mean that temporal blur is effectively gaussian in nature? Certainly there is less tail-streak and ghosting than the 'temporal supersample/averaged linear blur methods' (mvfps and the like).

mvflowblur is probably simple, but is the only true motion blur. :)

Again, what are the distinguishing characterstics of MVFlowBlur that qualify this statement i.e. 'true' in what respect?