GenMotion and DePan plugins for global motion compensation

[smok3]

iam trying to stabilize rotation using DePanStabilize, but it is a nogo no matter what i use for 'rotmax' value..., any ideas?

[Fizick]

Recently I developed first beta (alpha) version of MVtools to Depan interface, as MVDepan function of MVTools.

The global motion extraction algorithm is still not very clever and robust in this version, but it works as idea.

But it can estimate rotation too.

Try latest version on MVTools 0.9.8.2 and above
http://forum.doom9.org/showthread.php?s=&threadid=82015&perpage=20&pagenumber=3

[Fizick]

Version 1.1.4, December 15, 2004 - damping parameter in DePanStabilize is now variable (was accidentally fixed =0.9 in all previous versions :scared:

[Fizick]

Version 1.1.5, December 31, 2004 - bug fixed in DepanEstimate (erroneous motion data) for fftw=true with show=false and info=false

I really wander why it bug was not fond before
(it present since v.1.1)

Happy Mew Year!

[scharfis_brain]

I would really like to know, how a mewing year sounds alike ;)

Anyway. thanks for the fixed version :)

[nmap]

Hello Fizick,

Excellent work on a wonderful plugin! I do have but one problem, it may be due to my own stupidity though.

Glabally speaking, if I use

mdata = DePanEstimate(range = 4, trust=5, dxmax=8, dymax=8, improve=true, fftw=true)
DePanStabilize(data=mdata, dxmax=8, dymax=8, prev=1, next=1, info=false, cutoff=0.5)

I still obtain black bars where the panning occurs. Would this be because I've limted the dx and dy? I'd really like to remove the black bars if possible.

Thanks
nmap

[scharfis_brain]

increase the next and prev parameters

[Fizick]

nmap,
it is most hard question, we do no know what must be in place of black borders.

try use "addzoom", and "mirror" to fill it.
I often use use greater "cutoff" = 2.0

Note. The used values of dxmax and dymax are too low,
try defaults.

Lucky depaning! :)

[Fizick]

Version 1.2, April 1, 2005 - added fitlast parameter to fit some last frames range to original position


Version 1.3, April 29, 2005 - added blur parameter to somewhat hide the sharp mirrored details; blur is horizontal only, at left and right border.

[Fizick]

Version 1.4, May 7, 2005 (published May 29)- DePanStabilize: Zoom adaptive mode addzoom is improved. Adaptive zoom decreasing rate is slower than zoom increasing rate now. Thus, the black empty borders are decreased, and zoom value is more stable now.

[vinetu]

and now the WWW button is... unvisible! (http://www.avisynth.org/fizick/)
:)

[Seed]

and now the WWW button is... unvisible! (http://www.avisynth.org/fizick/)
:)

I know that your "invisible" link links to his homepage mirror. But just for information, if one click on the username (e.g. "Fizick") insie the left pane, a box will slides out and it contains the usual user profile links, including the link to homepage.

[Fizick]

Yes, this is a great forum enhancement. :angry:

BTW, DePab version 1.4.1 released (bug fixed with log)

[SCIF]

There is new version DePan v.1.5 (http://bag.hotmail.ru/depan/depan150.zip) - Stabilizer adaptive zoom mode improved more.
Thanks to Fizick for regular improve!

[J&J]

Fizick,

rotation.......
in the near future?

I still use deshaker 1.6 but waiting for Depan's next step....

:)

[Fizick]

J%J,
Did you try MVDePan?
http://forum.doom9.org/showthread.php?p=580027#post580027

Hm, I now found that MVDePan info is missed from Depan 1.5 English doc after edition :confused:
So, i restore it and relese a new 1.5.0 zip archive (the same binary DLL).

[J&J]

Wow, MVDepan can work!
But I can't get the stable outcome as Deshake.

My Avs :


LoadPlugin("C:\Program Files\GordianKnot\AviSynthPlugins\deflicker.dll")
LoadPlugin("C:\Program Files\GordianKnot\AviSynthPlugins\depan.dll")
LoadPlugin("C:\Program Files\GordianKnot\AviSynthPlugins\MV\MVTools.dll")

sourcefile="CIMG0297"

BaseLayer=DirectShowSource(sourcefile+".avi",fps=29.970).AddBorders(96,96,96,96)

vectors = BaseLayer.MVAnalyse(isb = false,lambda=40000)

globalmotion = BaseLayer.MVDepan(vectors, pixaspect=1, thSCD1=40000,thSCD2=40000,zoom = false,error =90)
Midway=DepanStabilize(BaseLayer, data=globalmotion, cutoff=2.0, mirror=0, pixaspect=1).deflicker(percent=99,scene=25)

Return(Midway)


Thanks for your answering.....

[videoFred]

Wow, MVDepan can work!
But I can't get the stable outcome as Deshake.

Yes, MVDepan works better than Depan only, because now we have not just global motion compensation but also compensation of moving objects etc..

:D Thank you, Fizick :D
:D Thank you, Manao :D

I must do many more tests, but it seems to be a huge improvement.
But I have noticed the rotation estimation is not 100%. (sometimes in the wrong direction.)

Also, on scenes with very fast moving objects and fast panning at the same time sometimes there is a jump in time beween two frames. But this is easy to solve with very low thSCD1 parameter. But then stabilising is not so good anymore.

Yes, Deshaker is still a little better, but needs two passes. On the other hand, too much stabilising looks artificial sometimes, and you get lots of blurred frames.

Resumed: I am verry happy with MVDepan, I think I'm gonna use it on all my files in the future. But if someone has a better solution for the time-jump between frames, I would be even more happy.

Fred.

[Fizick]

now we have not just global motion compensation but also compensation of moving objects etc..
No, we have global motion only with MVDePan.

[videoFred]

No, we have global motion only with MVDePan.

Realy???

You made it.. You know it.
But movement is smoother with MVDepan, then why?

Fred.

[Fizick]

I release new version DePan 1.6, August 5, 2005. Here is changelog:
- added YUY2 support (It is the main addition);
- disabled improved=true mode in DePanEstimate. (It seems, it was always broken );
- changed default subpixel=1 (bilinear) for DepanInterleave as quite sufficient for denoising and more fast

[Fizick]

New Version 1.7, September 5, 2005 - DePanStabilize: added parameter initzoom - minimal zoom;
changed limits dxmax, dymax, zoommax, rotmax from hard to soft with larger slope non-linearity.
Changed cache.

[scharfis_brain]

can you add an expanding feature to depanstabilize?

so I can do some kind of panorama out of quick pans.

[Fizick]

I do not understand you.

[scharfis_brain]

When I process VHS, I need to crop away 8px around the image, because there is garbage at this image borders.
So 704x576 becomes 688x560.
This video I feed into depanstabilize.
It is set up to use prev. & next frames to fill the borders.
After this I need to add an 8px border round the image to get 704x576 again.

Is it possible to skip the addborders? So that I get more visible image due to GMC from depanstabilize in shaking scenes?

EDIT: I do not mean a zoom here!

[Fizick]

We can not fill all (and whole) borders so.

[scharfis_brain]

with deshaker I can output a bigger than the input frame without zooming.

Assume an 320x240 input video (Digital photo camera)
http://home.arcor.de/scharfis_brain/samples/deshaker-expand.png

the resolution at the right side of the panel is set to 640x480 (doubled height & width). to compensate zooming, the extra zoom factor is set to 0.5 .

This leads to an expanded frame without resizing and without constant black borders (every movement fills them!)
, but for sure not all borders are filled. I am aware of that.

I would be happy to see this featur in depan, too :)

[Fizick]

So, in Deshaker you get big output frame, with main video in central part only, with big black (partially filled) borders?
So, you want, that output video from depanstabilize will have more width and height size, than input?
it is good for deshaker with many frames used for filling.
Some time ago i think about it, but for more simple boundary proccesing.
Some code re-design will be needed.

May be it is better to add "cropping" option to depan?
and use the "cropping" boundaries as limited mirror positions or solid fill.
Of course, you may use external Avisynth command for these now.
May be i can add some partial fill by previous frame too. But i am not sure.

[videoFred]

I use Depan for frame rate conversion, now.
With very nice results.

AssumeFPS(12.50) > ChangeFPS(25.00) >> Depan Stabilise
Gives very s-m-o-o-t-h panning.

But I do not understand the 'cutof' parameter :(
Suppose I set it to 2HZ.
2HZ of what?

I also do not understand the 'damping' parameter :(

But I begin to understand all the other parameters :D

Fred.

[Fizick]

I use innertial method. consider camera mounted on some inertial platform, with some damped spring (not fixed).
there is some intrinsic frequency for this mechanic system as oscillator.
and threre is its damping (relative factor).
Cutoff frequency is really not strict cutoff, but simply some characteristic frequency value.
Higher frequencies are suppressed. The higher, the more.
(it is simplified algo).

2 Hz is real frequency of shaking.

[loba]

2 Hz is real frequency of shaking.
Does it mean 2 periods per sec?

[Chainmax]

A long time ago, I read a recommendation from scharfis_brain to Boulder of using DePanInterleave in order to improve a script, saying that it helped a lot on video stuff as it usually contains panning/zooming. I have a few very crappy short clips from a digital camera and want to make something discernable out of.

In the previous thread I made about this, the following recommendations were made to me:

i=ConvertToYV12()
mdata=DePanEstimate(i)
r=2 # interleaveradius
DePanInterleave(i,data=mdata,prev=r,next=r)
yourfiltershere()
selectevery(r+r+1,r)
DePanStabilize(last,data=mdata)

Code:
i=ConvertToYV12()
mdata=DePanEstimate(i)
r=2 # interleaveradius
DePanInterleave(i,data=mdata,prev=r,next=r)
yourfiltershere()
i=selectevery(r+r+1,r)
mdata=DePanEstimate(i)
DePanStabilize(i,data=mdata)

None of those options seem to work, in both cases black borders appear and change size in a quite dizzying manner. I can provide one of the clips for you guys to examine if needed.

[johnmeyer]

Are you trying to take 10 fps progressive "video" from a still camera and change its frame rate? On what device do you plan to view it? (NTSC TV, HD TV, computer monitor, etc.).

I have had good luck with Motionperfect as a way to introduce intermediate frames in order to smooth MJPEG video from Canon still cameras. I haven't gotten around to doing it, but I have a 90% complete AVISynth script that accomplishes more or less the same thing using MVTools.

[Chainmax]

The sources are indeed extremely crappy 160x112 videos from a still camera, but the source is 25fps and I don't intend to change the framerate, just deshake the badly done pans. After that, the stream would be cleaned up as much as possible while resizing to 320x256, borders will be added, and it will all be encoded to MPEG2 in order to author a miniDVD.

[Fizick]

Chainmax,
Both your script are generally correct (use first, i.e. second simply slower).
But you must try tune optional parameters.
1. Firstly check, that depanestimate make correct motion estimation. Temporary add "return mdata" line to script after depanestimate,
and add parameter to it:

i=ConvertToYV12()
mdata=depanestimate(info=true, trust=4)
return mdata

Check data, base frame must be changed on scenechanges only.
If not, decrease the "trust" value.
2.Now tune depanstimate (remove "return mdata" of course)
It can not create perfect picture. DEPan does not know, what must be in place of moved borsers (= empty black spaces).
But you may try add some data from previous frames, use mirror, or zoom.
Read DePanStabilize documentation. Then i will aswer your questions.
3. You also may conside apply motion stabilization after resising (320x256) for more smooth results.

[Chainmax]

I see, thanks for the suggestions.

[Chainmax]

I didn't understand how to tweak the trust parameter, but after some experimenting I found that lower values increased the thickness of the black borders, so I raised it until there were no black borders. I also put the DePanStabilize line after the resize as per your recommendation. This is the script I'm currently using:

i=ConvertToYV12()
mdata=DePanEstimate(i,trust=23)
r=2
DePanInterleave(i,data=mdata,prev=r,next=r,mirror=15)
ffdshow("default")
SelectEvery(r+r+1,r)
Lanczos4Resize(320,256)
DePanStabilize(last,data=mdata)
LimitedSharpen(strength=500)
AddGrain(10,0,0)
AddBorders(16,16,16,16)

Thanks for the help :).


P.S: how do different values for r affect the result?

[Fizick]

No,no! It is wrong! :)
1. You really disabled the motion compensation when you rise trust so much.
So, why use DePan?
Decrease trust to default!
2. If you resize clip, its motion is changed, so you MUST use second DePanEstimate.
Or resise at begining of the script (before first estimation and denoise), but it may be not good.
3. Add prev=r,next=r,mirror=15 to DePanStabilize too.
4. r is frame range of motion estimation. It must be used for estimation too.
ffdshow("default") is denoiser? I do not know how good is it.
But try use r=1.

Try this for example:

i=ConvertToYV12()
r=1
mdata=DePanEstimate(i,trust=4,range=r)
DePanInterleave(i,data=mdata,prev=r,next=r,mirror=15)
ffdshow("default")
SelectEvery(r+r+1,r)
Lanczos4Resize(320,256)
i=last
mdata=DePanEstimate(i,trust=4,range=r)
DePanStabilize(i,data=mdata,cutoff=1,prev=1,next=1,mirror=15)
LimitedSharpen(strength=500)
AddGrain(10,0,0)
AddBorders(16,16,16,16)


may be you will need in initzoom

[Fizick]

What video does you try to process (link to short clip)?

[Chainmax]

So, the more trust the more compensation? No wonder the borders disappeared at trust=23 :o.
ffdshow("default") is a line that allows you to use whatever postprocessing options you select in ffdshow's "video decode configuration" section. In this case, I was using SPP deblocking at strength 100%. It's not good as it loses too much detail, but it's the only deblocker (and I tried lots) that could get rid of the horrible blocking in the clip. I'll try your suggestion and report back. In the meantime, you can download one of the clips here (http://www.31012.com/~azulftp/MOV01825.MPG).

[Fizick]

"trust" parameter value is threshold. If real trust of current frame is below "trust", motion compensation will be disabled.

P.S. Why you sometimes do not disable your long signature? I already read it.

[Chainmax]

Well, I tried your suggestion and the black bars are indeed gone, but since they are so thick the mirroring becomes very evident. I guess I should stick to using

i=ConvertToYV12()
mdata=DePanEstimate(i)
r=2 # interleaveradius
DePanInterleave(i,data=mdata,prev=r,next=r)
yourfiltershere()
selectevery(r+r+1,r)
DePanStabilize(last,data=mdata)

and if that doesn't work (like in this case) then just accept that motion compensation can't be done.

[Chainmax]

By the way, I'm going to try the combination on another video I have, would this script

i=ConvertToYV12()
r=1
MData=DePanEstimate(i)
DePanInterleave(i,Data=MData,Prev=r,Next=r,Mirror=15)
DeDot()
FixChromaBleeding()
Crop(whatever,align=true)
Cnr2("xxx",4,5,255)
DeBlock_QED()
DeGrainMedian(mode=0)
SelectEvery(r+r+1,r)
GaussResize(656,448,p=100)
DePanStabilize(last,data=mdata,Prev=r,Next=r,Mirror=15)
DeHalo_Alpha()
LimitedSharpen(SMode=4,LMode=3)
AddBorders(24,16,24,16)

be ok?

[Fizick]

1. Motion compensation (for best denoising) and motion stabilization (deshaking) are different things. So, if you want decrease mirrored borders (and the strength of motion stabilization), try increase cutoff parameter (of DePanStabilize in my suggestion script) to about 4. Only high frequency vibration (above cutoff) will be stabilized.

2. Too many questions. But your second script is not correct.
2.1 You must use second DeOanEstimate after resize. Do you understand the my script?
Please, re-read my post http://forum.doom9.org/showthread.php?p=749324#post749324
2.2 If you use many temporal denoisers, else increase the r parameter
or use DePaninterleave(data=mdata).SomeTemporalFilter(...).Selectevery(3,1) block for every temporal filter.
2.3 Why crop?

[krieger2005]

I have noticed a strange behaivor with depan. I used it on a relativly static scene and set "trust" to 2. Then i let show me the info of "DepanEstimate". The "strange" was, that when info show a trust below 2 it show also "SCENE CHANGE!". When trust was above 2 it show me not scene-change. I thought, that on high motion/many changes it would be classified as "scene-change" and not on low motion/less changes in the frames.

Here my script:
h=height
i=StackVertical(SelectEvery(3,0), SelectEvery(3,1), SelectEvery(3,2))
i=RemoveDirt(i,mthreshold=80, pthreshold=3, tolerance=12).RemoveGrain(17)
i=interleave(crop(i,0,0,width,h), crop(i,0,h,width,h), crop(i,0,h*2,width,h))

mdata=DePanEstimate(i,range=2, trust=2, info=true)

[Fizick]

It is by design.
trust - limit of relative maximum correlation difference from mean value at scene change.

[krieger2005]

This is not logical for me. Can you explain what you exactly thought when you implement this this way? Why do you make it so?
Sorry, but this sentense i does not unterstand: limit of relative maximum correlation difference from mean value at scene change

[Fizick]

To detect scenechange and not compensate (stabilize) motion across this scenechange.
Frames correlation is the best way to detect scanenange.

[krieger2005]

But why the filter make something to a "scene-change" when there is no movement? Again: When i set a value to "trust" the frame correlation must be lower than this "trust"-value to be a scene-change? Is it not the different way, that the frame correlation must be higher than the "trust"-value to became a scene-change, because lower frame correlation values mean lower movement, or not? Sorry, i'm a little bit confused.

[Fizick]

Current frames correlation must be lower than the "trust" parameter value to became a scene-change.
You get different processing?
I am confused.