GenMotion and DePan plugins for global motion compensation

[krieger2005]

You are right. It seems to be something with the clip. Thanks

[Beave]

I posted my problem in two other threads as well, but I still wanna ask here, because I tried your plugin.

I have a few 720p capture that shakes about one line up and down. So the whole image moves by one pixel. I don't wanna correct any other shaking. I used:
mdata = DePanEstimate(i)
DePanStabilize(i, data=mdata, dymax=1, dxmax=0, info =false)

Now not all the shaking is caught and it seems only parts of the image move. But it is smoother then before.
Could I only specify for example the bottom 20 lines for finding the shaking? There is a small black edge where I can see the picture moving quite easily.

Sorry for bothering you and thank you for your great plugins!

[Beave]

I'm getting closer to solving my problem. I used this script to first analyze only the bottom border:

mpeg2source("test_720p.d2v")
Trim(0,400)
i = ConvertToYV12()
j = ConvertToYV12().crop(0,712,-0,-0)
mdata = DePanEstimate(j,info=false)
DePanStabilize(i, data=mdata, dymax=1, dxmax=0, info=true)

Now I can see the data being diplayed on the video. the "dy" is either between -0.15 and +0.10 or between +0.25 and +0.55 when the video jumps. Now I now the values, but how do I translate that into my DePanStabilize function? So I have the sweet spot at around 0.18 and I want the picture to shift one pixel up when going over that threshold. How do I do that?

Thanks for the help in advance!

[johnmeyer]

I want to only compensate in the X direction. I thought I could do this by setting dymax=0, but this has no effect. I describe the reasons for this script in this thread: http://forum.doom9.org/showthread.php?t=109687&highlight=depan

Here is the script:# Script to recover film frames from film projected on shutterless 16mm projector.
# Second Pass
# Copyright 2006 John H. Meyer
# Revision May 9, 2006
#-----------------------------

loadPlugin("c:\Program Files\AviSynth 2.5\plugins\TIVTC.dll")
loadPlugin("c:\Program Files\AviSynth 2.5\plugins\Depan.dll")
loadplugin("c:\Program Files\AviSynth 2.5\plugins\MultiDecimate.dll")

AVISource("D:\OPRFHS\Intersquad 1st reel)0002.avi")

converttoYUY2(interlaced=true)
AssumeBFF()
tfm(display=false,micout=2,mode=0,cthresh=45,mi=1600,pp=6,metric=0,field=1,micmatching=2,slow=2,blockx=256,blocky=256,sco=-1,debug=false,input="d:\tfm.txt")

a=MultiDecimate(pass=2)
b=AssumeFrameBased(a)
c=b.separatefields()
d=c.colorYUV(autogain=true)

e=interleave(blankclip(c),selectodd(c),selecteven(c),blankclip(c))
f=interleave(blankclip(d),selectodd(d),selecteven(d),blankclip(d))

mdata2 = DePanEstimate(f,range=1,dxmax=32,dymax=0,pixaspect=0.9091,info=false)
g=e.Depan(data=mdata2,matchfields=false,offset=1,info=false)
h=g.selectevery(4,2,3)
i=h.ComplementParity().weave()
AssumeFPS(18)

i

As you can see, I use an exposure-compensated version (d) to generate the depan parameters, and then use these parameters to compensate the original. I then force a scene change after each pair of fields by inserting two blank fields (thanks to Avil for that hint). Thus, only the second of each pair of fields is shifted, and it only uses the information between from the previous field, which is what I want.

The problem is that this second field is shifting vertically as well as horizontally. I have tried using small non-zero settings for dymax, but it doesn't seem to do anything.

The depan.dll reports version 1.7.0.0, and is dated 9/5/2005.

I also am trying to figure out how to get the framecount back to where it should be. The Interleave function with the blank frames added doubles the frame count so I end up with the second half of the video being duplicates of the last frame. I can probably figure that out eventually, but the dymax has got me stumped.

[Fizick]

Probably I can add option "not estimate dy" in next version (in preparation, with some improvement in DePanstabilization), if you are still interested in it.

[Alain2]

some improvement in DePanstabilization
I am definitely, I have some bugs with current version (or I don't understand what I am doing with parameters, which is also possible but i doubt it)
Thank you Fizick :)

[johnmeyer]

It would help to be able to turn off dy, but I have a workaround I've created, so I am OK for now.

[JuanC]

Probably I can add option "not estimate dy" in next version (in preparation, with some improvement in DePanstabilization), if you are still interested in it.It is really interesting!It would help to be able to turn off dy, but I have a workaround I've created, so I am OK for now.I will appreciate if you share how you managed to work around it.

[johnmeyer]

I will appreciate if you share how you managed to work around it.

My workaround is very specific to my application. You have to go back and look at my earlier posts, but I am transferring film to video using a patent-pending process that I developed where I remove the shutter from a film projector, videotape by pointing the camera directly at the lens, and then, using software available for AVISynth, remove redundant frames, and frames where the film is being pulled down. This leaves me with two fields of video for every frame of film. Unfortunately, when a film projector is run at full speed, the film never quite comes to rest in the film gate (it is only there for a few milliseconds before the next frame of film is pulled down). I was therefore trying to remove the residual horizontal motion. I preferred to leave the residual vertical motion in order to get the proper offset between fields (they need to be offset vertically by one scan line). However, since I couldn't get dy=0 to work, I just let Depan line them up vertically, and then used AVISynth commands to move the odd fields by one scan line. The results are not quite as good as I wanted, but better than if I don't do the correction.

[Chainmax]

I am trying to make a conversion from 15fps to 23.976fps using DePan, but even after reading the corresponding portion of DePan's readme several times, I haven't been able to figure out how to do it. Can someone help me out?

[Fizick]

No, but here is simple fps doubler (from 15 to 30):

data = DePanEstimate()
f1_2 = DePan(data=data, offset=1./2)
Interleave(f1_2,last)

[Chainmax]

How about from 15fps to 23.976? Would something along the lines of:

i=last
data=DePanEstimate(i,range=1,trust=5)
f=DePan(i,data,offset=x)
b=DePan(i,data,offset=-x)
Interleave(f,i,b)
SelectEvery(???)

be ok? If so, what should I substitute x and ??? for?

[Fizick]

Look to fps conversion examples in DePan doc and try understand them.

[Chainmax]

I'm not being lazy, the framerate changes appear out of the blue in the readme, and the only two examples (2-to-3 and 3-to-5) are too different for a common template to be extracted from them. About the only thing I see in common is that both examples create some kind of backward and forward information, but that's it.

[johnmeyer]

Here's some code I used quite awhile ago. Haven't tested it with the latest version. However, it worked when I last used it. Hopefully it will help you fill in the gaps in your knowledge and get you going.#Slow motion
loadplugin("c:\Program Files\AviSynth 2.5\plugins\depan.dll")
AVISource("E:\Volleyball\Volleyball (Soldedad)\Volleyball (soledad)0022.avi")

converttoYV12()
AssumeBFF()
fps3to5()


function fps3to5(clipYV12)
{
# change FPS from 3 to 5 (or 15 to 25, or 18 to 30 and so on), i.e. with factor=5/3
# uses global motion compensation
# input must be YV12 progressive (or separated fields probably ?)
SeparateFields(clipYV12)
data = DePanEstimate(clipYV12)
t3_5 = DePan(clipYV12, data, offset=-2./5)
t6_5 = DePan(clipYV12, data, offset=1./5).trim(2,0)
t9_5 = DePan(clipYV12, data, offset=-1./5).trim(1,0)
t12_5 = DePan(clipYV12, data, offset=2./5).trim(3,0)
Interleave(clipYV12, t3_5, t6_5, t9_5, t12_5)
SelectEvery(15,0,1,2,3,4)
weave()
}

[Chainmax]

Thanks, but that 3-to-5 example is in DePan's readme, and the only other example (2-to-3) is much too different to extract a general framework from them.

[johnmeyer]

I guess none of know what you want. Yes, this was from the readme, but I actually used it, and this script is what I used. In other words, this is a working script. Isn't that what you wanted?

[Chainmax]

Not exactly, I need a 15-to-23.976 script that doesn't result in the need of resampling/repitching the source audio.

[scharfis_brain]

that won't work with depan.

[Chainmax]

I assume 15-to-29.97 won't workt with DePan either, am I stuck with duping/interpolating frames with ChangeFPS then?

[scharfis_brain]

you may upinterpolate it to 150 fps (15fpsx10) and then use cangefps(29.97).
so the remaining jitter is about 1/150 sec

[Chainmax]

You mean extrapolation to 150fps and then ChangeFPS(23.976), right? And what should I use for the extrapolation?

[scharfis_brain]

it is still interpolation.

modify fps3to5 so that it delivers 1/10th steps to achieve 10x of the input framerate

[krieger2005]

... and then use cangefps(29.97).

... and then ChangeFPS(23.976), right?
No comment.

[Chainmax]

it is still interpolation.

modify fps3to5 so that it delivers 1/10th steps to achieve 10x of the input framerate

I really have no clue how to do it, 3to5 has 3_5, 6_5, 9_5 and 12_5 but it really isn't clear why 4 steps are made and why the first numbers of each step are multiples of 3. I also don't understand the logic behind the offsets on each step or the SelectEvery line.


[edit]If the final conversion is going to be to 23.976, would it benefitial to do the up-interpolation to 120fps instead?

[scharfis_brain]

the more up-interpolation, the less the remaining jerkyness due to the 1000/1001 factor.

[Chainmax]

But 119.88 (120) is a multiple of 23.976 (24), whereas 150 is not. Does that not make a difference when the filter tries to calculate which of the up-interpolated frames to discard?

[Alain2]

That's why sharfis is not talking of an exact convertion, he says you'll have remaining jerkiness. With the depan way as I understand from sharfis and the 3to5 script, you can only divide by decimal numbers the gap between 2 frames to introduce the interpolated ones. This is not compatible with 15 to 23.976 (or more precisely 24000/1001, and this is not an decimal number)

Have a look in the avisynth doc to what changefps does... You'll see why with the depan method "the more up-interpolation, the less the remaining jerkyness due to the 1000/1001 factor."

[Chainmax]

I see. So, what would the modified fps3to5 with 1/10th steps to achieve 150fps be?

[Chainmax]

Alain2? scharfis_brain?

[scharfis_brain]

I have no time right now.

Why aren't you trying it out yourself?

[Chainmax]

Because like I said several times before on this thread, the docs are not clear enough in that regard, and the two framerate examples are too different for me to extract a common principle that would allow me to build a function with your suggestion.

[scharfis_brain]

what about trail & error then? That is what I do, when something isn't clear to me.

I remember made a similar conversion function when depan came out
(50 -> 300 -> 59.94 fps)

And the docs are clear IMO.

hint: leave out the selectevery - line.
Its job will be done by changefps afterwards.

[Chainmax]

It's not like I haven't been trying, but whatever.

[Fizick]

Chainmax,
Do you clear understand my third (simplest) example, which is in my post:
http://forum.doom9.org/showthread.php?p=840636#post840636

If not, other discussions is useless.
And did you try this doubler? Do you like it results at 30 fps (smooth)? If not, you must not spend you time with 24 fps.

[Chainmax]

I understand that example, but it only interleaves one component instead of two and it results in 30fps. I need either 23.976 or 29.97 fps output without having to resample/repitch audio.

[krieger2005]

You must find two numbers (not floats). The first one is the factor how much more picture you want to get an (not-float frame-rate) (let's say in your case 7992, because 15*7992=119880fps (or 999 -> 15*999=14985)). The second factor defines which frames of this 119880 you want have in your movie (in your case: 5000, because 119880/23.976=5000 (or 625-> 14985/23.976=625)). And then you have your 23.976fps.

So, in first case you must find a factor, to push up your framerate to a number (not float) framerate, which can be can be devided in the second step by a number (not float) to get 23.976. These two numbers must be as low as possible. Here 999 and 625 would work. This mean an interleave of 999 frames, where you take 625 of it.

[Alain2]

I understand that example, but it only interleaves one component instead of two and it results in 30fps. I need either 23.976 or 29.97 fps output without having to resample/repitch audio.
krieger2005 is right, and this is working only for decimal numbers, its not possible to change from a decimal framerate to a float framerate 24000/1001 using discrete numbers of intermediate interpolated frames ; but why can't you just interpolate to 24fps and then slow down to 24000/1001? I don't think you will notice the pitch change, and it will but easier to implement (15*8=120=24*5, thus interleave 7 frames between original ones, and select 5)

[Chainmax]

It's just that I was already trying to make a 5.1 mix of the mono track and having to resample/repitch means I'll have to do it all over again...*sigh*

Fine, I cave :(.

[edit]Not good. Using DePan creates huge black borders. What other alternatives do I have?


One more thing then: I'm now trying to have a small 15fps clip converted to 23.976fps but have it play at half the speed. I tried this:

fps3to5()

AssumeFPS(23.976)

SelectEven()

ChangeFPS(23.976)

But it doesn't seem to play at half speed. What am I doing wrong?

[johnmeyer]

You could try MotionPerfect from Dynapel. They have a free trial, I believe.

[scharfis_brain]

Not good. Using DePan creates huge black borders.It has parameters to avoid that!

I repeat mysel. it is possible to do your task
pseudocode may be like this:

input=xxxsource("blah")

vectors=depanestimate(input, bla bla bla)

input_1_of_10 = input
input_2_of_10 = input.depan(+1/9)
input_3_of_10 = input.depan(+2/9)
.
.
.
interleave(input_1_of_10, input_2_of_10, ...)
changefps(23.976)


WHAT is so HARD about this task?!?
That is basically typed from my mind.

[videoFred]

WHAT is so HARD about this task?!?
That is basically typed from my mind.

Hello Scharfis,

It's simple for you....

But you and some other members here have this special mathematic talent.
Be happy with it, it's a gift! I admire this talent very much!

But we, mortal people :p , do not have this talent.
You must understand this.
So it would be good if someone would explain more about this.

Fred.

[scharfis_brain]

what is so hard about simple multiplication and division? and also some calculations with the rule of three?

pupils are learning this in 5th grade...

[videoFred]

what is so hard about simple multiplication and division? and also some calculations with the rule of three?

pupils are learning this in 5th grade...

It's not about the multiplication and division, but where and how to use it.
Working with frames, fields, frame rates etc can look very complicated if you are not used to do it.

Fred.

[Fizick]

Take a white paper and draw a time graphs of shifted frames. It must help.

[Chainmax]

Thanks for your example scharfis_brain, I made a 16-step function that extrapolates to 240.002fps, then I use AssumeFPS(239.76) and set the final framerate with ChangeFPS. Motion appears to be acceptably smooth and as far as I can see there are zero artifacts. I am extremely happy and I apologize if I came out as too insistent. I have but two more questions:

1) Will using ChangeFPS(23.976) and then using pulldown or ChangeFPS(29.97) directly make any difference in motion fluidity in the end result mpeg2? Empirically, both seem pretty much equal to me, but I'd like to know a theoretical reason.

2) Would using SelectEvery yield better results than using ChangeFPS?

[scharfis_brain]

you don't need assumefps follower by changefps.

1)just use changefps(23.976) and become happy.
you will not notice the small error of 23.976 <-> 24.000 fps.
but do NOT use changefps(29.97 ) after changefps(23.976) it will make your video jerky.
either use 23.976 fps MPEG2 with pulldown, or go straightforward to 29.97 fps and encode it as 29.97 fps MPEG2

2)selectevery only can reflect integer decimations, while changefps can do arbitrary decibmations.

[Chainmax]

...
but do NOT use changefps(29.97 ) after changefps(23.976) it will make your video jerky.
either use 23.976 fps MPEG2 with pulldown, or go straightforward to 29.97 fps and encode it as 29.97 fps MPEG2
...

Oh, you misunderstood me. What I meant is the second thing: either ChangeFPS(23.976)+Pulldown or directly ChangeFPS(29.97). The former is probably better anyway because the intermediate huffyuv file will occupy much less. Again, thanks for your advice :).

[scharfis_brain]

go directly 29.97.

this won't introduce telecine judder.

[Chainmax]

Ok. The final MPEG2 was done extrapolating to 240.002fps, then using AssumeFPS(239.76) and ChangeFPS(29.97). If I mux that with the original audio, will there be any noticeable A/V asynch or will it be ok?