New toy: MVTools version with true motion compensation

[mg262]

Fizick,

There are many reasons; this is the quickest to explain: I need a very compact format (e.g. to allow cacheing of many motion vectors) so I am thinking of four bytes per vector; what you suggest seems like twelve bytes per vector. 3x overhead...

[Manao]

We could deinterleave sad from mvs in the MVTools. Furthermore, having a header saying how the content is organized would help. But I don't see the point in caching many motion vectors. That won't speed up a filter as soon as it needs to do some processing.

In the header, we could have the following data :
* colorspace
* width of the analysed clip
* height of the analysed clip
* block width
* block height
* mv precision
* direction of the analysis ( forward / backward )
* number of frames between the analysed frame and the reference
* payload id
* payload size
* padding to align to 16

Then we would have the motion vectors ( each coordinate on a 16 bits signed integer ), and the payload ( which could contain all the data needed by MVTools but not by Motion )

[mg262]

But I don't see the point in caching many motion vectors. That won't speed up a filter as soon as it needs to do some processing.The method I am using in Motion is recursive; computing the motion vectors for frame 50 requires the motion vectors for frame 49, which requires the motion vectors for frame 48, which... motion vectors for frame 1, which requires the motion vectors for frame 0. Some methods of dealing with issues raised by this require caching many motion vectors.

In order to use the motion vectors in MVTools, you need to compute whatever additional information is requested by filters; so you need to run them through a conversion filter to generate this. What is the difficulty with having that filter read motion vectors in the Motion format and write them in the MVTools format?

[Fizick]

Didée,
Thanks for test. It was pel=1 or pel=2?


mg262,
but you do not write any header info. and do not output the SAD.
O.K. Forget about it now. May be, when you impement more functions, you will need in more complex format. we will return to this topic.

[Didée]

Ah, I tend to forget that pel=1 does exist ;) Tests were with pel=2, blksize=16, searchparam=3.

Strangely, decreasing number of compensated frames to -3 ... +3, speed seemed to drop a little more, to about ~90% of MVT 0.9.11 (142 frames/min vs. 160 frames/min). However the script *is* complex, so the exact reasons might be in the darks. But atleast it seems the changes didn't make things faster.

***

Another thought:
Could sort of mimicking the "inter4v" mode of mpeg-4 codecs make sense? Because, while small blocksizes (8 or even 4) sometimes deal better with moving edges, and mostly better with occlusion areas, the large blocksize 16 seems much more reliable when large, uniform areas are moving. The wish to get the best of them both sometimes is coming up ...
Or is this, in some way, already lying inbetween the new parameters you've added to mvanalyze? I'm not fully clear about these, and it's hard to see & understand what's happening _during_ the process of motion search, when you're looking "from outside" and only see the end result.

[Fizick]

is it possible for you to add a motion blurrer based on mvFlow() ?
possible parameters like
blurlength: 0 - no blur ; 0.5 blur for the half the duration of a frame; 1 - blur for the duration of one frame ; 2 - blur over the duration of 2 frames
precision: 1 - do nothing; 2 - two pixels; 10 - pixels etc...

maybe you could leave out the precision parameter, if you are drawing lines of 'blurlength' (vectorlength*blurlength) instead of pixels, so there would never occur any ghosting.

v0.9.12.2 (Released 13.11.2005)
Added experimental precise but slow MVFlowBlur function as scharfis_brain asked.
Temporary changed size of internal frames buffer to 5.

Parametersof MVFlowBlur are almost as you asked:

blur: blur time interval between frames (in percent, default=25.0)
prec: blur precision in pixel units. Maximal step between compensated blured pixels. Default =1 (most precise).

But it is simple (without occlusion mask yet) and slow.

[Fizick]

Didée,
you say about variable block size at motion estimation stage?
it is hardly.

[Fizick]

v0.9.12.3 (Released 14.11.2005)
Fixed bug for croma for width not divisible by 16 in MVMask (introduced in v0.9.11).
Some speed optimizing of MVFlowFps.
Reset size of internal frames buffer to original value 10. Try various versions.

[sh0dan]

Fizick, your work is such a big improvement over the earlier versions! It's still quite buggy, but it shows so much potential!

Bugs observed in new functions:

MvFlowFPS ml = 0 -> division by zero.
MvFlowBlur blur > 100 -> occational access violation.

The docs are very hard to read. I call out to everyone who understand them, and have the time, to help out in creating good tutorials and docs.

[Fizick]

sh0dan,
I will add parameters bound checking.

About docs. It is more hard item. Soon I will create Russian doc. May be it will be more readeable (for those who understand Russian) :)

Or you say about Manao's part of doc too?

[sh0dan]

@Fizick: The doc section wasn't as much aimed at you, as it was to some helpful soul wishing to help out. This is a great package, but it still requires solid code and very good documentation. I'm sure you can help with the solid code, as you've already turned Manaos basecode into something very useful!

I think the concepts and parameters are difficult to understand, and even harder to control. This is not a design problem as such, but it requires a lot more documentation to be easy to use. Compare it to Telecide. A "simple" function, but it still it requires massive documentation to ensure that it's useable. MvTools are rather fast becoming just as useful.

Maybe it would be a good idea to open a section on avisynth.org?

[Fizick]

May be some parameters can be renamed or rescaled, but I do not want to do it (and do not want ask Manao to do it) without really big needs (as I do for "showsad"), to not broke existing scripts.

[Fizick]

v0.9.12.4 (Released 15.11.2005)
Changed type of ml parameter in MVMask, MVFlowInter, MVFlowFps from int to float.
Added bound check of ml, time, blur parameters.
Small possible bug fixed (emms).
Partially updated documentation. But I am not sure that sh0dan requested exactly such updating :). It is still not user guide but functions reference.

[videoFred]

Hello Fizick,

MVFlowFps works very very fine :)
I did some first tests on this slow (600mgz) machine.

No blocks anymore!
But sometimes 'rubber' artifacts like with Motion Perfect.

I used this script:

Loadplugin("MVTools.dll")
Loadplugin("Depan.dll")
Loadplugin("fft3dfilter.dll")
Loadplugin("degrainmedian.dll")


source=avisource(film).assumefps(16).converttoYV12()
clipC=DeGrainMedian(source,limitY=5,limitUV=10,mode=2)
clipD=FFT3DFilter(clipC,sigma=2,sharpen=1.0)

backward_vectors = clipD.MVAnalyse(isb = true, lambda = 1000, lsad=1000, pnew=50, pel=2)
forward_vectors = clipD.MVAnalyse(isb = false, lambda = 1000, lsad=1000, pnew=50, pel=2)
frameclip=MVFlowFps(clipD,backward_vectors, forward_vectors, num=25, den=1, ml=100)

mdata=DePanEstimate(frameclip,trust=0)
stab=DePanStabilize(frameclip,data=mdata,cutoff=1.0)

stab

Now I must test it at home on my fast (3Ghz) machine.
I think it's gonna run faster than the original MVTools :cool:

And I'm gonna compare MVConvertFps with MVFlowFps.

But I do not understand the use of the 'den' parameter??? :eek:
I mean, 'num=50, den=2' is the same as: 'num=25, den=1'
Both are giving me a result file with 25fps.
What's the difference?

I have another question, too:
Dynapels Videoscope tells me ALL the frames are originals!
This means MVFlowFps and MVConvertFps creates complete new frames, right?




Fred.

[foxyshadis]

There is no difference, FpS always = num/den. For PAL it's not so important, but with NTSC it has to be num=30000, den=1001 because both are integer values. (Any other franctional fps the same.)

[Fizick]

videoFred,
thanks!
I am very interested in old 8mm film conversions too.

1)If you want, some frames will be old, when you set fps relation to some simple relation.
For example, you can change source fps from 16 to 16.66666 (=50/3) so way:
AssumeFps(50,3)

2) Yes, MotionPerfect probably uses similar algo.

MVFlowFps is experinental. DePan gives more stable result.

As a workaround, we can try use block overlap.
Try this my function:



loadplugin("blockoverlap.dll")
loadplugin("mvtools.dll")

function OverlapFps(clip source, int "num", int "den", int "ml", int "blksize", float "kernel", int "lambda", bool "chroma", int "pel", int "thSCD1", int

"lsad",int "pnew")
{
# Overlapped block motion fps conversion by shifted clips processing and windowed blending
# Fizick, 2005
# Uses MVTools v0.9.12.4 plugin by Manao & Fizick
# Uses BlockOverlap v0.2 plugin by Fizick
# source clip must have YV12 format,
# all other parameters are optional
# new fps = num/den
num=default(num,25) # numerator
den=default(den,1) # denominator
ml=default(ml,100) # mask scale
blksize=default(blksize,8)
kernel=default(kernel,0.5)
lambda=default(lambda,2000) # vector smooth
chroma=default(chroma,true) # use chroma
pel=default(pel,2) # pixel precision
thSCD1=default(thSCD1,300) # scenechange threshold
lsad=default(lsad,2000) # lambda SAD limit
pnew=default(pnew,50) # penalty new
add = (blksize>=8) ? blksize : 8
vecF = source.MVAnalyse(blksize=blksize, isb = false, lambda = lambda, chroma=chroma, pel=pel, lsad=lsad, pnew=pnew,idx=101)
vecB = source.MVAnalyse(blksize=blksize, isb = true, lambda = lambda, chroma=chroma, pel=pel, lsad=lsad, pnew=pnew,idx=101)
comp = source.mvflowfps(mvbw=vecB, mvfw=vecF, num=num, den=den, ml=ml, thscd1 = thscd1, idx=1)
shifted = source.addborders(blksize/2,blksize/2,add-blksize/2,add-blksize/2) # diagonal shift by half of block
vecFshifted = shifted.MVAnalyse(blksize=blksize, isb = false, lambda = lambda, chroma=chroma, pel=pel, lsad=lsad, pnew=pnew,idx=102)
vecBshifted = shifted.MVAnalyse(blksize=blksize, isb = true, lambda = lambda, chroma=chroma, pel=pel, lsad=lsad, pnew=pnew,idx=102)
compshifted = shifted.mvflowfps(mvbw=vecBshifted, mvfw=vecFshifted, num=num, den=den, ml=ml, thscd1 = thscd1, idx=102)
compshiftedback = compshifted.crop(blksize/2,blksize/2,blksize/2-add,blksize/2-add) # remove shifting borders
ov = BlockOverlap(comp,compshiftedback,xblksize=blksize, yblksize=blksize, kernel=kernel)
return ov
}

source=avisource("film.avi")
source.overlapFps(num=50, den=1)



It is not perfect too.
May be i will try to average not pixels, but its vectors in next MVTools versions.

P.S. Please, do not ask mg262, "did he ever try blockoverlap?" :)

[videoFred]

videoFred,
thanks!
I am very interested in old 8mm film conversions too.

Me too... I do nothing else...
I do not even have a digital video cam, except my filmtransfer camera :)


For example, you can change source fps from 16 to 16.66666 (=50/3) so way:

Yes, I understand... 18,75 must be possible, too.

I'm gonna try blockoverlap, too.
No, I'm not gonna ask mg262 :scared:

But Depan after MVFlowFps works realy great!
Have you tried this? It's almost perfect!
I use the stand alone Depan for this..
Maybe it could be even faster with MVDepan?

I also noticed FFT denoising before MVFlowFps is not good.
Then MVFlowFps gives duplicate frames, why?

Soon I'm gonna post some MVFlowFps example clips on my website.

Fred.

[Fizick]

videoFred,
If MVtools duplicates frames, threafore it detect a scenechange. Try to increase thSCD1 parameter to about 500.

[videoFred]

Fizick,

Thank you, I try this, too.

First tests on my HP pavillion (3ghz):
Original files progressive 1024x768 Hyffyuv.
script: Resizing-frame rate conversion-denoising-stabilising-cropping.

Script speed with old MVTools MVconvertFps: 1-2fps
Script speed with your MVFlowFps: 2-3fps :cool:

At play speeds from 15-16-18fps, converted to 25fps, almost no visible artefacts while playing on most files. Very smooth zooming and panning and general motion.

With some files even extreme slow motion is possible (1fps!!)
So Alpha version is already very useful for me. :)

Fred.

[Fizick]

v.9.13 (Released 20.11.2005 )
Added truemotion preset to MVAnalyse.
Added experimental MVFlowFps2.
Change interpolated vector rounding method in all MVFLow... functions.
Edited documentation a little.

Probably i can put it to Avisynth usage MVtools topic now.
http://forum.doom9.org/showthread.php?t=84770
Al least i take a some its develipment pause in any case.

[johnmeyer]

Unable to get 9.13 to run. I get the following error:

AVISynth open failure:
Evaluate: System exception - Access violation

The line on which this is reported is the line with MVFlowfps.

Here's the script:

loadplugin("c:\Program Files\AviSynth 2.5\plugins\kerneldeint140.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools.dll")

AVISource("J:\Senior Video Tribute (do not erase)\Video\Video Highlights (JHM)\Baseball\Baseball-0013 - 000.avi")

assumebff()
changespeed(4, 1)

function changespeed(clip a, float factor, int th)
{

ord = getparity(a) ? 1 : 0
f=a.kerneldeint(order=ord, sharp=false, twoway=false, threshold=th)
e=a.separatefields.trim(1,0).weave.kerneldeint(order=1-ord, sharp=false, twoway=false, threshold=th)
g=interleave(f,e).assumeframebased
h=converttoyv12(g)


vec_back = h.MVAnalyse(isb = true, lambda = 1000, lsad=1000, pnew=50, pel=2)
vec_forw = h.MVAnalyse(isb = false, lambda = 1000, lsad=1000, pnew=50, pel=2)
MVFlowFps(vec_back, vec_forw, num=30, den=1, ml=100, idx=1)
converttoyuy2()
assumefps(30,true)
separatefields()
(ord==0) ? selectevery(4,0,3) : selectevery(4,1,2)
weave()
}

[videoFred]

You forgat your source clip:
It must be: h.MVFlowFPS(etc...)
or: MVFlowFPS(h, etc...)

Fred.

[Didée]

Not really related, but still:

Instead of that "interleave-two-times-kerneldeint"-thingy, why not simply use LeakKernelBob() instead? Less script code, faster operation, same result ...

[Fizick]

johnmeyer,
I hope your problem now solved (thanks videoFred).

To all:
This thread is for programming questions only now.

For MVTools usage please post to other thread:
http://forum.doom9.org/showthread.php?t=84770

[johnmeyer]

u forgat your source clip
-----------------
johnmeyer,
I hope your problem now solved (thanks videoFred).
-----------------------

Thanks videoFred. I was so excited, I started working on this as soon as I read about the new version -- way past my bedtime.

Yes, it works fine now.

[Neil Lee]

Below is the example in MVTOOL1.0 Documentation:

======================================
To denoise with pel = 2, efficiently :

backward_vec2 = source.MVAnalyse(isb = true, lambda = 1000, delta = 2, pel = 2, idx = 1)
backward_vec1 = source.MVAnalyse(isb = true, lambda = 1000, delta = 1, pel = 2, idx = 1)
forward_vec1 = source.MVAnalyse(isb = false, lambda = 1000, delta = 1, pel = 2, idx = 1)
forward_vec2 = source.MVAnalyse(isb = false, lambda = 1000, delta = 2, pel = 2, idx = 1)
return source.MVDenoise(backward_vec2,backward_vec1,forward_vec1,forward_vec2,tht=10,thSAD=300)
===========================================


I was thinking instead of using (delta>1), why not just
add motion vectors between adjacent frames?
(Don't know how MVanalyse work internally, maybe this is it?)

Say I want MotionVector(Frame1,Frame4), I would rather
have a function to ADD motion vectors of adjacent frames.

MV14 = MV12 + MV23 + MV34

This way you won't ignore additional info you can get in Frame2,Frame3

Let's say you have a fast moving object, It's hard to find any
matching patterns between Frame1 and Frame4, it's out of search range.
But you'll have a better chance to find it in adjacent frames,
and then you just add them up ..)

Don't know if there's any precision issue for adding Motion Vectors,
How about use it as a secondary method for finding unmatched blocks?

.

[Fizick]

Neil Lee,
Thanks for interesting question.
Generally, there are many known ways to do motion estimation and compensation in special literature.
I am not Manao, who implemented it so way. But IMHO, current way:
1. Easy internally to implement.
2. Do not accumulate estimation error (mismatching). Not motion vector precision, but similarity (correspondense) of matched blocks by intensity (SAD).
3. Quite good vector MV13 can be found if some occlusion in frame 2 may prevent matching motion MV12 and MV23.

But I see advantage of your suggestion: It can results in more fast motion estimation when you use both delta=1 and delta>1 in complex scrips.
(we will estimate only delta=1 and have all deltas from it).

[Manao]

Don't be afraid of high motion. The motion estimation is done first on subsampled pictures, where even high motion in easy to catch.

Anyway, you can already try your idea if you want, with mvcompensate(mode=1) : mvs1 = source.mvanalyse()
mvs2 = source.mvanalyse(delta=2)
comp1 = source.mvcompensate(mvs1)
comp2 = comp1.mvcompensate(mvs1, mode = 1)
comp3 = source.mvcompensate(mvs2)
return interleave(source, comp1, comp2, comp3)

[Isochroma]

Thanks for all the work on your motion compensated denoiser! It performs very well indeed...

I am wondering about an inconsistency in the syntax of your mvdenoise example...

backward_vectors2 = source.MVAnalyse(isb = true, lambda = 1000, delta = 2)
backward_vectors1 = source.MVAnalyse(isb = true, lambda = 1000, delta = 1)
forward_vectors1 = source.MVAnalyse(isb = false, lambda = 1000, delta = 1)
forward_vectors2 = source.MVAnalyse(isb = false, lambda = 1000, delta = 2)
return source.MVDenoise(backward_vectors1, backward_vectors1, forward_vectors1, forward_vectors2, tht = 10, sadt = 300)

how are the parameters of mvdenoise arranged? backward_vectors1 is duplicated, while backward_vectors2 is not used? Your documentation does not provide enough syntax information for me to determine whether your example contains a mistake.

I'm interested in using the >2 frame denoise feature introduced in 0.9.9.1, but because I can't figure out the syntax, how would I write the script?

Also, clarification on the nature of the parameters, ie. clip, clip mvs [,...] ... is clip the current frame, and clip mvs a comma-delimited sequence of motion-vector variable names? Does their order matter?

Finally, the sadt argument causes an error stating that the function does not support it. I have installed the 0.9.9.1 version and am using the documentation from that version as written... any ideas?

Thanks!

[Mug Funky]

the docs are out of date for that one...

btw, i think the backward vectors 1 being used twice looks like a typo to me.

[Fizick]

Isochroma,
this thread is about new MVTools versions AFTER 0.9.9.1.
All these typos are corrected in v1.0.
Please, be not so lazy to read the thread first.

[Neil Lee]

True Motion too Agressive?

Hi Fizick/Manao,

It seems that TrueMotion is sensitive to noises a bit too much,
I have some videos with absolute fixed background, but even
a few noise in between will cause it to think the background
is moving, a very large area is affected.

What's even stranger, is that when I strip the video down to it's
edges, a cartoon-like video with a near noise-free background, it
still thinks the background has movements. (default settings)

Looks like a slight of movement/noise will affect a large static area
if there's no distinctive lines in between to seperate them.

Maybe a gradient detection(compare) will help preventing this?
Or imploy a temporal vector trembling threshold/smoother to
eliminate incoherent small vectors trembling along the time axis..?

----- using the sample code with truemotion on ---------
source=converttoyv12(source)
vectors = source.MVAnalyse(isb = false,truemotion=true)
return source.MVShow(vectors)
---------------------------------------------------------

original vid (1M) (http://dabadee.hypermart.net/mvtool/original.avi)

edge only vid (1M) (http://dabadee.hypermart.net/mvtool/edges.avi)

vectors vid (1M) (http://dabadee.hypermart.net/mvtool/vectors.avi)

http://dabadee.hypermart.net/mvtool/original.jpg

http://dabadee.hypermart.net/mvtool/edges.jpg

http://dabadee.hypermart.net/mvtool/vectors.jpg

original vid (1M) (http://dabadee.hypermart.net/mvtool/original.avi)

edge only vid (1M) (http://dabadee.hypermart.net/mvtool/edges.avi)

vectors vid (1M) (http://dabadee.hypermart.net/mvtool/vectors.avi)

[Isochroma]

I must apologize - after checking the new 1.0 documentation, that typo was fixed... but my syntax questions remain. Does the order of the motion vector parameters matter? Could you provide an example of the syntax for 3- or 4-frame denoising using MVDenoise?

[Fizick]

Isochroma,
IMHO, any order. This function was implemened by Manao. I never use it.
Try use forum search (old MVTools thread).
And please post your "usage" question in other thread (in "usage" forum).
http://forum.doom9.org/showthread.php?p=749503

Current thread is about "development".

[Isochroma]

Oops, posted in wrong thread again! Indeed, the parametric order doesn't seem to matter so much.

[Fizick]

Neil Lee,
probably i must restore zero vector predictor preference (Now i give some penalty to it). It is dilemma.
But do you try to tweak advanced truemotion parameters, for example decrease pnew to 0?
I will try your video later.

[scharfis_brain]

Fizick, here http://rapidshare.de/files/3042060/buga-2.avi.html you can download a clip, which totally seems to cunfuse mvanalyse. I never puts out any motion vectors!

avisource("buga-2m.avi").assumebff()
bob()
converttoyv12()
backward_vec = MVAnalyse(isb = true)
last.mvshow(backward_vec)

[videoFred]

Nice chickens :)

I found out lowering the contrast and brightness of the MVAnalyse clip brings the motion vectors back.

tweak(bright=-20,cont=0.4)

I tried it with your clip, it works!

Fred.

[scharfis_brain]

yeah, lowering contrast or blurring the source bringing back the vectors.

but both are NO option.

[Fizick]

scharfis_brain,
1. You can use different clips for analysing and compensation.
2. 30 MB is too much for my dial-up.

[scharfis_brain]

1) I know. But I will not modify existing scripts, that are working extremely well with all other sources. plus: lowering contrast and/or sharpness will degrade motion estimation, won't it?

2) how much will be acceptable?

[Didée]

a clip, which totally seems to cunfuse mvanalyse. ...
backward_vec = MVAnalyse(isb = true)
Last time I checked, there were some more parameters available for MVAnalyze ;) ... probably you'd have to adjust some of them. Seems like this clip forces high SADs because of its sharpness, or something like that (can't check out right now).
The possible characteristics of possible sources are without any limits - you can't expect the default values to work out on just everything.

[scharfis_brain]

you can't expect the default values to work out on just everything.
Why not? I could have lived with inaccurate motion vectors, but NO vectors at all is not acceptable at all.

Last time I checked, there were some more parameters available for MVAnalyze ... probably you'd have to adjust some of them. Seems like this clip forces high SADs because of its sharpness, or something like that
So which parameters need to be altered then?

[Didée]

Poking in the blue: thSCD1 (prhps thSCD2 too) are good candidates. Could be that the clip constantly causes SADs that overflow thSCD1=300. Try 9999. Perhaps lsad/pnew play a role too, not sure about that.

Does the behaviour change when simply giving "blksize=16" or "4" ?

[scharfis_brain]

setting thscd1 on the compensation functions to 9999 helps! thanks.

does it hurt to leave it 9999 all the time?
eg. in mvbob?

[Didée]

Well, you loose MVTools' ability to detect scenechanges, so it'll compensate "over" the SCs, no matter how crappy the result. For MVbob it shouldn't matter much, because of the corrector it's using.

[Fizick]

For my (analog, noisy) sources I always use thSCD1=400 or 500.
BTW, today I will release new bugfixed version (memory leakage).
I can also change default thSCD1=400 if you (and Manao) will not protest against it.

[Fizick]

New MVTools 1.0.1 (Released 24.12.2005 by Fizick).
Fixed memory leakage bug in MVAnalyse with global motion (thanks to AI for report - probably nobody else used MVTools for real long encode :) )
Removed penalty for zero vector predictor in MVAnalyse (was introduced in v1.0).
Changed chroma=true as default in MVAnalyse.

I am still did not change thSCD1 default (compatibility ...).
But IMHO it must be block-normalized value (about 5 to 6).

[AI]

I used - I add 21 jobs - every 10 000 frames *.xx.avi :cool:
(for reload MVTools)

[Fizick]

My "to do" list:
1. Overlapped blocks motion estimation for using with MVFlowXXX
2. YUY2 colorspase