GenMotion and DePan plugins for global motion compensation

[SoonUDie]

Thanks! I'll try it out as soon as possible.

[scharfis_brain]

the download link doesn't work :(

[SoonUDie]

Originally posted by scharfis_brain
the download link doesn't work :( There's a typo in the link. It currently says http://bag.hotmail.ru/genmotion/depan01.zip when it should be http://bag.hotmail.ru/depan/depan01.zip

[scharfis_brain]

already tried that!

I am always getting to a blueish site

Save File

but whatever I do, I am in loop, when clicking in the download-link of this blueish site :(

[scharfis_brain]

I HATE Sites being not compatible with non-IE browsers!

[SoonUDie]

Actually, I use Firefox, and I didn't have any problems... you just have to wait until the download starts on its own (10 seconds?).

[Fizick]

Sorry, now i correct the link. It was one of bugs. :(
(By the way, I use Opera browser.)
Probably, I must make a mirror (what is the recomended for free?)

[scharfis_brain]

depan 0.1 has a bug.

depan(offset=x) doesn't work for x being in the range of 1>x>-1
even the info=true screen disappears, if offset is between 1 and -1.

this means depan(offset=0.5) won't work.

[Fizick]

Released version 0.2, April 27, 2004 - bug for non-integer OFFSET values is fixed.

[scharfis_brain]

thanks!

I'll try that out.

I hope, I can realease some kind of 'smoothpan' standards conversion function (NSTC <-> PAL) within the next days
(will be nearly the same code like the one I coded for morsa's 30p to 24p conversion)

[scharfis_brain]

okay it does work.

but:
It seems to work very unprecise
I always blend fwd & bckwd compensation within my scripts for better results, but fwd & bckwd do not always match each other, thus introducing false blending.
depanestimate seems to deliver only quantisized motion information.
Example: if a pan slowly accelerates (pan is getting faster and faster), the motioncompensation doesn't adapt smoothly. It compensates in 'steps'

Depan doesn't compensate zooms.

deshakers analyse is much better compared to v0.2 of depan.

okay, enough critisism

the concept of master/slave is pretty cool!

I've made some tests with my smoothpan standards conversion.
It works! But it isn't very unusable for now :(

can't wait the next version of depan :)

would it be a copyright violation just using deshaker's analyse code within depan?

[Fizick]

The motioncompensation doesn't adapt smoothly. It compensates in 'steps'
Try subpixel=2 (bicubic). Default is =0 (nearest pixel), in next version it will be =2. It is document bug. (also i know about Depanstabilize bug with input paarameters - will be fixed later).

Depan doesn't compensate zooms.
Yes, but it may be implemented later.

deshakers analyse is much better compared to v0.2 of depan.
Yes, but it often has error with rotation.
And depan method (phase-shift) is more fast.

would it be a copyright violation just using deshaker's analyse code within depan?
What code? No any source.
May be, I shall add reading of Deshaker log, as in genmotion, but it only useful with zoom added. I do'nt want to add rotation.

[Fizick]

New DePan version 0.3, May 15, 2004 - fixed bug with DePanStabilize input parameters, set subpixel=2 (bicubic) as default, add zoom estimation and ZoomMax parameter, zoom and rotation compensation, log file input, MatchFields parameter (for best denoising).

[Fizick]

New DePan version 0.5:
improve DepanStabilize and some bug fixed.

Anybody test the plugin? Or it is fully unusable?
Any proposal for next version?

[scharfis_brain]

I do not have access to my PCs to test all the new stuff.
For at least the whole next week :(

please stay tuned. I'll post my results asap, I can get back to the faster machine...

(or: recompile for avisynth 2.08 and Pentium 1, than I can test it on my 120MHz Notebook)

[Fizick]

Version 0.6, May 28, 2004 - minor changed and documented client-server format.
DepanEstimate: add stab parameter, scenechange at sharp trust changes, range may be 0, show correlation, crop output.
DepanStabilize: Freqmax is renamed to Cutoff, add adaptive zoom, Fill borders by neibours frames.

Version 0.7, May 30, 2004 - DepanEstimate: add improve zoom estimation.

[Fizick]

I release a new version of Depan, developed about month ago.
Version 0.9, June 13, 2004 - all clients: added "Mirror" parameter to fill empty borders.

mirror parameter - fill empty borders with mirrored from frame edge pixels (instead of black):
0 - no mirror (default);
1 - top;
2 - bottom;
4 - left;
8 - right;
sum any of above - combination (15 - all ).

Mirror mode is unique but slightly strange ;)
It can give remarkable results for example for sky.

[Fizick]

Serg Belyansky,
You wrote:
Unfortunately one problem persists: any solid surface with a large number of frequent simular elements (e.g. brick wall or polka-dot dress) will exhibit an erratic motion estimation with elements disappearing and reappearing randomly. It can be partially remedied by increasing the "range" parameter of Depanestimate, but it slows the process and introduces artifacts on other surfaces. So I'm still scratching my head.

It is a hard problem with no simple and general answer. :(
But I can give one idea to you (seems i omitted it in doc).
Input clip for DePanEstimate may be not strictly same as source,
but prefiltered, or brightness-contrast adjusted, or masked by some method, or cropped, and so on.

If you get some new ideas from your head after scratching, please post here they, or at least the your scratching method. :)
And do'nt use my DeScratch filter until ! :D

P.S. It seems, i found some nice (and big!) Russian forum thread about Avisynth.
http://forum.ixbt.com/topic.cgi?id=29:9331

[scharfis_brain]

thanks for that great tool, it helps a lot denoising shaky/panning analogue camcorder video.

but there seems to be one issue:

everytime I try to enable zoom compensation, using zoommax > 1, I receive the following error MSG from AVISynth:

AVISynth read error: Avisynth: caught an access violation at 0x025d80f5, attempting to readfrom 0x025ffb10

it seems that there is a memory leak, cause, the value of setmemorymax() is in direct corellation to the framecount that is rendered, until avisynth crashes.

[Fizick]

I found some memory leakage in Depan, but only at constuctor stage (not per frame).
I can not get such errors.
May you give me more info? (Avisynth and plugin versions, script, CPU, etc)

[scharfis_brain]

using this (or similar script)

i=converttoYV12()
mdata=i.DePanEstimate(zoommax=1,improve=false,pixaspect=1.094)
i.DePanInterleave(data=mdata,pixaspect=1.094)
temporalsoften(1,10,13)
selectevery(3,1)

when I set zoommax=2 and improve=true

avisynth crashes after some frames (depends on the value of setmemorymax) with above mentioned error msg.

I am using:
newest AVS build.
newest DePan release
AthlonXP 1600+ with ASUS A7VE-266 and 768 MB of DDR-RAM

[Fizick]

I tried similar script.
At what frame number it crashes (100, 1000, 10000, 100000 )?
And for what Setmemorymax ?
Newest AVS build (20.08.2004) is buggy. See development forum.

[Fizick]

I am sorry, but i really try version 0.8 :scared:
0.9 is buggy, yes. :( (i almost did not test it )

So, good news:
New version 0.9.1 is out. Seems, zoom bug is fixed.

[scharfis_brain]

Many thanks for this bugfix :)

But: I finished my current project before you did the bugfix :(

nontheless, DePan becomes my standard aid for denoising handycam footage. it is just great, cause it is the only thing allowing low contrasted moving detail (like grey trexturized walls) being denoised while their texture remains untouched.

a blockbased motioncompensation without global motion compensation first, is not able to catch this correctly...

[Fizick]

I think about using more fast library (FFTW).

[Serg Belyansky]

Originally posted by Fizick
Input clip for DePanEstimate may be not strictly same as source,
but prefiltered, or brightness-contrast adjusted, or masked by some method, or cropped, and so on.
Yes, I tried it this way. And I must say I'm now think that I have no clear understanding of the process at all.

The motion compensation helps to match neighbor frames, right? But DePan works with global motion only, compensating only for "main" object (background or close-up), not both. This the local motion of "polka-dotted" dress should not affect it at all. And judging by undecimated debug output, it actually does not.

I tried MVTools, which is by design "local" and should compensate for _everything_, making the perfect match between frames. But that's NOT IT -- it actually makes DeSpot to work much worse, with a lot of nasty edge effects on moving subjects. The "disappearing polka-dot" problem is somewhat remedied, but I can't say that the difference is easily noticeable.

So I think I just don't understand the "motion compensation" part well. For example, what "motpn" parameter exactly do with local motion compensation performed?

Thresholding with "sign" parameter helps, but not much, since I can't set an actual threshold: the "polka dots" are not as bright as brightest parts of dirt blobs, but they are still falsely keyed and removed by DeSpot (my film dirt is bright white, scanned from negative).

(Bugreport: the "sign" parameter description in Russian manual is erratic; the values are mixed up.)


P.S. It seems, i found some nice (and big!) Russian forum thread about Avisynth.
http://forum.ixbt.com/topic.cgi?id=29:9331 [/B]

Yes, but my firewall prohibits me from posting there :rolleyes:

[Fizick]

Serg Belyansky,
Your (our) problem is serious.
Yes, DePan works with global motion only.

And DesPot is not ideal solution still.
What "motpn" parameter exactly do with local motion compensation performed? - I wrote about it in doc, you also may see a C source. :)

BTW, I do not understand, what is "polka-dots" ? :confused:
May your post some example image? (Or in russian)

If the most question now is about Despot, please return to Despot thread. May be we can improve it with some ideas (not detect very small dots or very round spots :confused: ).

Thanks for discussion!

[hartford]

Polka Dots (http://images.google.com/images?q=polka%20dot&hl=en&lr=&ie=UTF-8&sa=N&tab=gi)

[Dreassica]

My guess is that he means dot crawl and the like.. could be wrong thre.

[Serg Belyansky]

Originally posted by Dreassica
My guess is that he means dot crawl and the like.. could be wrong thre. No-no-no-no. We are talking about regular structures in an image (such as dress fabric pattern or brick wall pattern) which are prone to be partially removed by DeSpot side-effect.

Fizick, I'll try to look at the sources. Maybe the "sign" parameter can be improved by introducing the thresholds for it.

[Fizick]

I release version 1.0 of DePan.
Since version 1.0, DePan can also use more fast FFTW library version 3 (http://www.fftw.org)
as Windows binary DLL (compiled with gcc under MinGW by Alessio Massaro),
which support for threads and have AMD K7 (3dNow!) support in addition to SSE/SSE2.
It may be downloaded from ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip
For fftw using, you must put FFTW3.DLL file from this package to some directory in path (for example, C:\WINNT).

Please post test (speed) report.

[morsa]

WOW!!
A lot faster now on this Athlon XP machine I'm using to test it!!!
As always Fizick, THANK YOU!!:thanks:

(Taste the Russian advanced knowledge ):D

[Fizick]

I get some e-mail from Andreas J. and decide post it here for discussion:

I trying to use DePanEstimate for scene detection & Extraction of JPEG-FIles.
As far as I have tested, It will do what I want.
I would like to do detect the scenes an extract JPEG thumbnails in one step.
My problem is, that I havent found a way to formulate the if-clause because I Havent Found a way to get the "Trust" Value or the dx-value for each frame
As I Understand your Plugin, it is currently not possible to get these values during runtime.
Is it possible to extend DePanEstimate to do this?

My Ideas for this are:
1) getting the "trust" - value as a variable during the processing of the each frames
in AviSynth a condition like
DePanEstimate.Trust < 3.5 ? ExtractKeyframe(currentFrame):nop()
(Okay, I need todo it on every frame with EvaluateFrame() or ScriptClip() but my Scripting Experiance
ist only only 10 Hours with Avisynth ;-):
2) by adding a function "DePanIsKeyFrame" with nearly the same parameters as DePanEstimate
(as far as I understand DePanEstimate SceneChange will come if myTrust-value > Trust(CurrentFrame)
3) another possibility could ba a function (DePanExtractKeyFrames (trust = 3.5, log = "c:\myfilename, KeyFramesEvery=1500, SkipMultiKeyframes=25…)
The log will consist of lines where myTrust <= Trust (Frame) where
"KeyframeEvery" will force-add keyframes(in this case every minute 25fps*60= 1500 and
"SkipMultiKeyframes" will not allow new keyframes for x frames if another (not forced) keyframe was detected before
(it should prevent to get 25 keyframes a second…. )
The log should also contain the trust-value and a info if KeyframeEvery-conditions was positive
4) my last idea is: a extlog="c:\myFilename.log" the result is a the same as the "log=" -LogFile
plus 2 additdional values: the trust value and "Scenechange"-Info.
So it would be goot to parse the logfile with PHP.
But a second step is needed to extract the thumbnails.

A Perfect solution would be a combination out of 1) & 3) & 4)

Sorry for so many ideas, but your tool has so much power….
Do you think you will find / a way to make scene-detection & frame extraction easier?



Here is my current script:
MayBe it ist much easier as I thought…..


LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MPEGDecoder.dll")
global ThumbNailPath = "D:\_Szenenerkennung\videos\Bilder\"
global LogFilename = "D:\_Szenenerkennung\MyLog.txt"
myMPEGSource = MPEGSource("D:\_Szenenerkennung\videos\25.mpg")
myVideo = ReduceBy2(myMPEGSource)
myVideo = ConvertToYV12(myVideo)


function SaveFrame (myClip, myFrameNumber)
{
###Uncomment to display frame number
myClip = Subtitle( myClip, string(myFrameNumber))

### Export JPEG Image
ImageWriter(myClip, file = ThumbNailPath, start = myFrameNumber, end = myFrameNumber, type = "jpeg",info=true)
### WriteLogFile
WriteFile(LogFilename, """ "CurrentFrame: " """ ,"current_frame")
return (last)
}

#myVideo = ScriptClip(myVideo,"depanestimate(myVideo,trust=4,log=""" "scenes.log" """,info=true,show=true)")
myVideo = depanestimate(myVideo,trust=4,log= "scenes.log",info=true,show=true)



###Export KeyFrame
### a Key Frame yould only be exported when myTrust <= (DePanEstimate.Trust(frame))
myRGBVideo = ConvertToRGB24(myVideo)
ScriptClip(myRGBVideo, "SaveFrame (myRGBVideo,current_frame)")



It would make me very happy to get an Idea how this problem could be solved..

[Fizick]

My first answer to Andreas is:

I thinked about implementing scenedetection function some time ago.
For example, I can easy to realize a function DePanScenes(clip, inputlog),
which uses for input either DePanestimate motion data clip, or log file.
DePanScenes output will be clip with all pixels luma (or croma) = 0 for all ordinary frames, and = 255 for scenechanges.
This clip can be analyzed by AverageLuma() for example, and results used in ScriptClip.

But DePanEstimate make some calculation at constructor stage (sinus table for fft etc), and frame processing will be slow.

[Manao]

Thinking of this, I could also quickly make a filter from the ME engine in MVTools, which decides whether a frame is a scenechange or not. I don't know however if the scenechange I get from vectors would be more reliable than those you're getting with your filter.

Anyway, if you want to test the basic scenechange detection in mvtools, you can use the following trick :mv = source.mvanalyse(isb = false)

mask= source.mvmask(mv, ml = -1, gamma = 100)

white = source.blankclip(color = $ffffff)

black = source.blankclip(color = $000000)

scmask = conditionalfilter(mask, white, black, "AverageLuma()", ">", "0.2")

return scmask( It's a hack, if there is no motion at all, it will be considered as a scenechange, but anyway, it gives a result )

[Fizick]

New version 1.1, September 16, 2004 - Added experimental DepanScenes function.

[Ark]

I found this script on Depan help file regarding FPS conversion from 16,6 fps to 25 fps:

Avisource("kino.avi")
loadplugin("depan.dll")
i=ConvertToYV12()

data=DePanEstimate(i, range=1, trust=5)
f1_3=DePan(i,data,offset=1./3)
b1_3=DePan(i,data,offset=-1./3)
interleave(f1_3,i,b1_3)
selectevery(6,0,1,2)

But I've some movies at 15 FPS that I want to convert to 25 FPS, how do I have to change values to achieve that?

(the above script result in 22.5 FPS...)

[johnmeyer]

When trying to load Depan 1.1 or Depan 1.0, AVISynth 2.5 returns the error message:

Unable to load "depan.dll"

If I copy the version 0.91 depan.dll to the same directory, the dll loads and the script runs just fine. I have no problems loading other dlls, so I assume there is some bug in these two latest versions.

I am running XP Pro (SP1). I am running AVISynth 2.54.

Edit: I upgraded to the latest (AVISynth 2.55) and still couldn't load the two latest versions. I was hoping to use this plugin to generate slow motion from standard NTSC DV video. Can this be done using the DePanEstimate function?

[Fizick]

Ark,
Here is a possible function for framerate converting (progressive) with factor=5/3, for example from 15 fps to 25 fps :



function fps3to5(clipYV12)
{
# change FPS from 3 to 5 (or 15 to 25, or 18 to 30 and so on), i.e. with factor=5/3
# uses global motion compensation
# input must be YV12 progressive (or separated fields probably ?)
data = DePanEstimate(clipYV12)
t3_5 = DePan(clipYV12, data, offset=-2./5)
t6_5 = DePan(clipYV12, data, offset=1./5).trim(2,0)
t9_5 = DePan(clipYV12, data, offset=-1./5).trim(1,0)
t12_5 = DePan(clipYV12, data, offset=2./5).trim(3,0)
Interleave(clipYV12, t3_5, t6_5, t9_5, t12_5)
SelectEvery(15,0,1,2,3,4)
}

AviSource("e:\video.avi")
LoadPlugin("depan.dll")
ConvertToYV12()
fps3to5()

[Fizick]

johnmeyer,
I have similar reports from some users.
Probably you must have FFTW3.DLL
I will look later.
What is your CPU ?

[johnmeyer]

Probably you must have FFTW3.DLL

I'll look into it tomorrow. I have a Pentium P4 2.8 GHz.

[Fizick]

Version 1.1.1, October 07, 2004 - bug fixed with compensation near right bottom and top corners2,
change from FFTW_MEASURE to FFTW_ESTIMATE for more short init, without speed change (for power-2 windows),
compiled without /G7 flag (as before v.1.0), FPS script functions added to doc.

(not public)

Version 1.1.2, October 09, 2004 - delayed loading of fftw3.dll (now optional). Compiled with VC6.

[Ark]

Originally posted by Fizick
Ark,
Here is a possible function for framerate converting (progressive) with factor=5/3, for example from 15 fps to 25 fps :



function fps3to5(clipYV12)
{
# change FPS from 3 to 5 (or 15 to 25, or 18 to 30 and so on), i.e. with factor=5/3
# uses global motion compensation
# input must be YV12 progressive (or separated fields probably ?)
data = DePanEstimate(clipYV12)
t3_5 = DePan(clipYV12, data, offset=-2./5)
t6_5 = DePan(clipYV12, data, offset=1./5).trim(2,0)
t9_5 = DePan(clipYV12, data, offset=-1./5).trim(1,0)
t12_5 = DePan(clipYV12, data, offset=2./5).trim(3,0)
Interleave(clipYV12, t3_5, t6_5, t9_5, t12_5)
SelectEvery(15,0,1,2,3,4)
}

AviSource("e:\video.avi")
LoadPlugin("depan.dll")
ConvertToYV12()
fps3to5()



Thanks a lot Fizick, I'll try thah script tomorrow! :)

[johnmeyer]

fftw3.dll fixes the problem. Thanks!

[scharfis_brain]

Bug in depanstabilize found:
it produces dark-green frames (I think Y=U=V=0 )sometimes.
also it makes one or two lines at the bottom looking dark green.

the error is reproduceable, but I didn't tried it with another script for now.

[Fizick]

:script: :)

[scharfis_brain]

you won't like it:


setmemorymax(512)
loadplugin("d:\x\masktools.dll")
loadplugin("d:\x\depan.dll")
loadplugin("d:\x\removedirt.dll")

function repairfieldswap(clip c)
{d=c.swapfields().crop(0,0,0,-1)
d=stackvertical(d.crop(0,0,0,1),d)
e=c.converttoyv12()

global edges=e.yv12convolution(horizontal="1",vertical="1 -2 1",Y=3,U=0,V=0,usemmx=true)
global swapedges=e.swapfields().yv12convolution(horizontal="1",vertical="1 -2 1",Y=3,U=0,V=0,usemmx=true)

ConditionalFilter(e, c, d, "float(averageluma(edges)-averageluma(swapedges))", "lessthan", "0.0")
}
a=avisource("capture (2004-11-14 at 14-34-24).avi")
b=avisource("capture-55630.avi").converttoyuy2().levels(16,1,235,0,255)
c=avisource("capture-81443.avi")
d=avisource("unblended.avi").changefps(25).killaudio().addborders(2,0,6,8)

a1=a.trim(0,55630)
a2=a.trim(55638,81442)
a3=a.trim(81447,0)
a1+b+a2+c+a3
trim(580,100296)
assumetff()
doubleweave().selectodd()
audiox=last
last.killaudio()
last.trim(0,2205)+d+last.trim(3074,0)
audiodub(last,audiox)
r=2
c=2
mergechroma(crop(0,c,0,0).addborders(0,0,0,c))

repairfieldswap()


crop(2,0,-6,-8)

converttoyv12()
x=depanestimate(last,pixaspect=1.094,trust=0.0,range=1)
depaninterleave(last,data=x,next=r,prev=r,mirror=15)
removedirt()
selectevery(r+r+1,r)

de0=last
de1=de0.depanstabilize(data=x,dxmax=12,dymax=12,zoommax=1.0,rotmax=0.0,pixaspect=1.094,mirror=15) #next=12,prev=12)
de1.trim(0,190)+de0.trim(191,540)+de1.trim(541,0)

[Fizick]

Confirmed :(
The bug was becouse of infinite calculated shifts for improperly Depanestimate() parameters setting.
Bug fixed in new version 1.1.3. Now base frame reseted.
Thanks.
Try!

[scharfis_brain]

Wow, you're damn fast in fixing bugs!

but can you explain this:
"The bug was becouse of infinite calculated shifts for improperly Depanestimate() parameters setting."

a little bit more detailed?

what are infinte shifts?
why improper settings?

[Fizick]

1. improper settings - some scenechange was not found,
and resulted in very big false pan by Depanestimate.

2. Depanstabilize try calculate shift for these false data,
and internal get divergence of caclulation (infinite shift).