mvtools: 2 videos with different fps together

[Terka]

Idea is to increase the HD video framerate.
Is possible following?:
2 vids:
1st 1920x1080@50i
2nd 640x 480@160fps
use the Motion Estimation on the 160fps vid for Motion Compensation on the 50i?

[Didée]

Not in a really "good" way. If the 640 one would be 150fps, it would be much better, because of (relatively) easy matching between 50i fields and 150p frames. But 50i and 160p are not synchronized (only every 80th frame of 160p is a direct match to a field of 50i), so the conditions are not good to start with.

A possibility could be to simply not care, just go ahead and try it: dumbchange the HD's framerate to 160, interleave it with the upscaled 160p, and see what MVTools will find ...

Rough outline:


50iHD = [this] .crop(as,needed)
160pSD = [that]

160pUS = 160pSD.spline36resize(50iHD.width,50iHD.height)
50pHD = 50iHD.bob()
50pHD_work1 = 50pHD.changeFPS(160.0)
50pHD_work2 = 50pHD_work1.blur(1.0,0.7).blur(1.0,0.7)

alter1 = interleave(160pUS,50pHD_work1)
alter2 = interleave(160pUS,50pHD_work2)

sup1 = alter1.MSuper()
sup2 = alter2.MSuper()

bv = sup2.MAnalyse(isb=true)

bc = sup1.MCompensate(bv)

bc.selecteven()


Of course, both sources should be "reasonably" aligned, both spatially and temporally.

[Terka]

Thank you Didee,
im now trying it on IDEAL CONDITIONS test source:
150p higher res. vs 50p with lower resolution 'manually done' from 50i clip

put a on variables begining because of problems with syntax.



clp=avisource("c:\Program Files\AviSynth\plugins\ski.avi").bob(0,0.5) #.Crop(left,top,-right,-bottom)
f=3


a150pUS = clp.BilinearResize(720/f,576/f).spline36resize(720,576)
a50pHD = clp.selectevery(3,0)


a50pHD_work1 = a50pHD.ChangeFPS(150)
a50pHD_work2 = a50pHD_work1.blur(1.0,0.7).blur(1.0,0.7)

alter1 = interleave(a150pUS,a50pHD_work1)
alter2 = interleave(a150pUS,a50pHD_work2)

sup1 = alter1.MSuper()
sup2 = alter2.MSuper()

bv = sup2.MAnalyse(isb=true)

#bc = sup1.MCompensate(bv)
bc = MCompensate(alter1, sup1, bv)

bc.selecteven()

but something is wrong is script i think.

[Didée]

Wrong was that I missed the source clip for MCompensate - you have it right now. Apart, it looks correct to me ... what is wrong with the result?

Note - in areas where mo'comp is not successful, the output will show the content of {the small Clip with high fps}.


Ah, wait ... your source is not really 150fps, hence the hardcoded value in Changefps is wrong, of course. Try like this:

clp = avisource("c:\Program Files\AviSynth\plugins\ski.avi").bob(0,0.5) #.Crop(left,top,-right,-bottom)
clp = clp.Assumefps(150.0)
f=3


a150pUS = clp.BilinearResize(720/f,576/f).spline36resize(720,576)
...

[Terka]

Yes, correct.
The wrong for me was the blocky-ness. Smaller blocks and overlap brought better results.
Of course the goal is to compare the result with HD only mflowfps.
(=compare triple fps using :1 HD clip fps versus 1 HD clip + small already trippled


-------------
so did a comparsion and used your Note above-it looks like wrong mo'comp everywhere??
see the 4MB testclip & code
-------------

http://rapidshare.com/files/283009611/ski-TempGaussMC_beta1x264q20.avi



clp=avisource("c:\Program Files\AviSynth\didee_compare\ski-TempGaussMC_beta1x264q20.avi") .Assumefps(150.0)
f=3
bs=4

a150pUS = clp.BilinearResize(720/f,576/f).spline36resize(720,576).tweak(sat=3.75).tweak(sat=3.75)
a50pHD = clp.selectevery(3,0)


a50pHD_work1 = a50pHD.ChangeFPS(150)
a50pHD_work2 = a50pHD_work1.blur(1.0,0.7).blur(1.0,0.7)

alter1 = interleave(a150pUS,a50pHD_work1)
alter2 = interleave(a150pUS,a50pHD_work2)

sup1 = alter1.MSuper(pel=2)
sup2 = alter2.MSuper(pel=2)

bv = sup2.MAnalyse(isb=true,blksize=bs, overlap=bs/2)
bc = MCompensate(alter1, sup1, bv)

d=bc.selecteven()

#########
clp
super = MSuper(pel=2)
backward_vec = MAnalyse(super, overlap=4, isb = true, search=3)
forward_vec = MAnalyse(super, overlap=4, isb = false, search=3)
f=MFlowFps(super, backward_vec, forward_vec, num=3*FramerateNumerator(last), den=FramerateDenominator(last))

stackhorizontal(d,f,Overlay(d,f, mode="difference", pc_range=true).tweak(sat=3.75))

[Terka]

Played with your script abit. Cant say if the 150fps clip brings something. Why not to do all the motion search on the 150fps clip?
Wont be it better if the frames are closer together (not compare frames 1-1, 1-2, 1-3, but 1-2, 2-3 etc.)?


left=2*100 right=2*100 top=2*88 bottom=2*60 #ski

clp=avisource("c:\Program Files\AviSynth\didee_compare\ski-TempGaussMC_beta1x264q20.avi") .Assumefps(150.0).ocisluj()
f=3
bs=4

#
a150pUS = clp.spline36resize(720/f,576/f).spline36resize(720,576)
a50pHD = clp.selectevery(3,0)


a50pHD_work1 = a50pHD.ChangeFPS(150)
a50pHD_work2 = a50pHD_work1.blur(1.0,0.7).blur(1.0,0.7)

alter1 = interleave(a150pUS,a50pHD_work1)
alter2 = interleave(a150pUS,a50pHD_work2)

sup1 = alter1.MSuper(pel=4)
sup2 = alter2.MSuper(pel=4)

bv = sup2.MAnalyse(isb=true,blksize=bs, overlap=bs/2)
bc = MCompensate(alter1, sup1, bv)

d=bc.selecteven().Crop(left,top,-right,-bottom).subtitle("Didee")

#########
clp.Assumefps(150.0).selectevery(3,0)
bs=8
super = MSuper(pel=2)
backward_vec = MAnalyse(super,blksize=bs, overlap=bs/2, isb = true, search=4,lambda=1000*bs*bs/64)
forward_vec = MAnalyse(super,blksize=bs, overlap=bs/2, isb = false, search=4,lambda=1000*bs*bs/64)
f=MFlowFps(super, backward_vec, forward_vec, num=3*FramerateNumerator(last), den=FramerateDenominator(last)).Crop(left,top,-right,-bottom).subtitle("Fizick")


clp=clp.Crop(left,top,-right,-bottom).subtitle("orig")
US=a150pUS.Crop(left,top,-right,-bottom).subtitle("upscale")
stackhorizontal(clp,us,d,f)

function ocisluj (clip c)
{
ShowFrameNumber(c,x=10, y=25, font="arial", size=24, text_color=$c00000)
}

[Terka]

Could someone check my last script for errors?
If its err. free, my conclusion is:

combining 2 cameras to get 1080@150fps
source cameras:
1st: 1080@50fps
2nd 480@150fps
wont be better than simply use mvtools to calculate tripple framerate. In both cases artefacts are present.



SetMemoryMax(1024)
left=2*100 right=2*100 top=2*88 bottom=2*60 #ski

clp=avisource("c:\Program Files\AviSynth\didee_compare\ski-TempGaussMC_beta1x264q20.avi") .Assumefps(150.0).ocisluj()
f=3
bs=16
thSAD=20000
lambda=1000*bs*bs/64


sd_150 = clp.spline36resize(720/f,576/f).spline36resize(720,576)
hd_50 = clp.selectevery(3,0)
hd_50dupl =hd_50.ChangeFPS(150)
SD_to_HD_150 = sd_150.ChangeFPS(150).blur(1.0,0.7).blur(1.0,0.7)


super_sd2hd = SD_to_HD_150.MSuper(pel=4)
super_hd = hd_50dupl.MSuper(pel=4)

vb1 = super_sd2hd.MAnalyse(isb=true ,blksize=bs, overlap=bs/2, search=4,lambda=lambda)
vf1 = super_sd2hd.MAnalyse(isb=false,blksize=bs, overlap=bs/2, search=4,lambda=lambda)


f1 = MCompensate(hd_50dupl, super_hd, vf1, thSAD=thSAD)
b1 = MCompensate(hd_50dupl, super_hd, vb1, thSAD=thSAD)


b1=b1.selectevery(3,0)
f1=f1.SelectEvery(3,2)
final=interleave(b1,hd_50,f1)

#########from mvtools documentation ########
clp.Assumefps(150.0).selectevery(3,0)
#bs=8
super = MSuper(pel=2)
backward_vec = MAnalyse(super,blksize=bs, overlap=bs/2, isb = true, search=4,lambda=lambda)
forward_vec = MAnalyse(super,blksize=bs, overlap=bs/2, isb = false, search=4,lambda=lambda)
f=MFlowFps(super, backward_vec, forward_vec, num=3*FramerateNumerator(last), den=FramerateDenominator(last)).Crop(left,top,-right,-bottom).subtitle("Fizick")


clp=clp.Crop(left,top,-right,-bottom).Subtitle("original")
final=final.Crop(left,top,-right,-bottom).Subtitle("final")
stackhorizontal(clp,final,f)


function ocisluj (clip c)
{
ShowFrameNumber(c,x=10, y=25, font="arial", size=24, text_color=$c00000)
}

[Didée]

Well ... let's say you didn't give enough information about the sources, in order to get an optimized result. ;)

After some quick tests, I found that your proposal seems to work. (Using my method with minor adjustments - I didn't test any of your scripts yet.)

There *are* artifacts - but to me they mostly appear less distracting than the warping-artifacts of MFlow. And there are more ways to reduce them, read on.

Then, "it looks like wrong mo'comp everywhere??" - erh, what? That's not what I see. The result I get seems basically correct. Perhaps there's a 'lil twist somewhere in your script, can't tell.

More ways of artifact supression:
- of course, block overlapping (trivial)
- using "thSAD=300~800" in MCompensate (trivial)
- tweaking the blurring of the HD, to better match the upscaled SD (not too hard)
- using an additional forward compensation, then selecting the better one of forward/backward
(Should improve very much on border regions during pannings etc.)
(not trivial, requires MMask comparisons)



Also: I really wonder what exactly we are talking about! (And what the result is for.)

Initially, you mentioned a 160fps source. Now all of a sudden, it's solely about a 150 fps source. That's a difference!

With the method I had shown (motion search between HighFPS<-->LowFPS frames), it is basically possible to combine clips of any arbitrary framerate. 160p<>50i is possible. 150p<>50i is possible. 987.65p<>43.21i is possible.

With the method you asked about (motion search on the HighFPS clip exclusively), only non-fractional framerates can be combined: 150p<>50i is possible. 160p<>50i is not possible. 987.65p<>43.21i is not possible.

More questions:

How do you acquire two sources, one being 50i HD, one being 150/160p SD, that show exactly the same content?
(I could think of a game engine running a fixed demo, or something like that. But I can't think that easily of real-world cases using one camcorder, and even with two camcorders it's tricky - you'd need to mount two cam's together, pretty close, on some carrier.)

Finally, what's the goal of having a 1920x1080 clip at 150 or 160 fps? If it's for doing some fine slowmo's, that's okay, of course. But if it's just for "motion fluidity", then I fear you'll get problems with realtime playback of 1080p @ 150 fps ...

[Terka]

The main idea was following: Buy another camera with 160fps (dont know if 150 fps is also available) and combine the framerates together to get higer framerate.
This will be used on sport-technique analysis.

So did a test on 150 fps.
With the method I had shown (motion search between HighFPS<-->LowFPS frames), it is basically possible to combine clips of any arbitrary framerate. 160p<>50i is possible. 150p<>50i is possible. 987.65p<>43.21i is possible.

With the method you asked about (motion search on the HighFPS clip exclusively), only non-fractional framerates can be combined: 150p<>50i is possible. 160p<>50i is not possible. 987.65p<>43.21i is not possible
yes.

exactly the same content
is of course a problem.
tried 2 1080i sources, on tripods, controlled by remote controller. Of course the angle is different. Maybee depan could help?
But most of all i wanted to try it.