New toy: MVTools version with true motion compensation

[WorBry]

Howdy Folks,

For conversion of (mvbob) 50fps sources to 25fps, I’ve been using the following script, a slight modification of Scharfis_Brain's mvfpsscd function:

function mvfpsscd(clip i, float fps, int "oversample", int "blurradius")
{
blurradius=default(blurradius,1)
oversample=default(oversample,1)
j=i.temporalsoften(2,4,5,2)
fwd=mvtools0962_mvanalyse(j,isb=false,sx=4,sy=4,lambda=4000)
bwd=mvtools0962_mvanalyse(j,isb=true,sx=4,sy=4,lambda=4000)
mv=i.mvtools0962_mvconvertfps(bwd,fwd,fps=fps*oversample).deblock()

global mv= (oversample>1) ? mv.temporalsoften(blurradius,255,255,mode=2).selectevery(oversample,0) : mv
global ch=i.changefps(fps) #. subtitle("ch")

global msk0=ch.duplicateframe(0).reduceby2().reduceby2().motionmask(thsd=255,y=3,u=1,v=1)
global msk1=msk0.trim(1,0)
global msk2=msk0.trim(2,0)
global msk3=msk0.trim(3,0)

f0=scriptclip(ch,"""sc""") #.subtitle(string(a0)+" "+string(a1)+" "+string(a2))""")
f1=f0.frameevaluate("""sc= (sc2 || sc1) ? ch : mv""")
f2=f1.frameevaluate("""sc2=( (a2 > (a1*5+25)) && (a2 > (a3*5+25)) ) ? true : false""")
f3=f2.frameevaluate("""sc1=( (a1 > (a0*5+25)) && (a1 > (a2*5+25)) ) ? true : false""")
f4=f3.frameevaluate("a3=averageluma(msk3)")
f5=f4.frameevaluate("a2=averageluma(msk2)")
f6=f5.frameevaluate("a1=averageluma(msk1)")
f6.frameevaluate("a0=averageluma(msk0)")
}

avisource("C:\mvbobbed 50fps.avi")
ConvertToYV12()
mvfpsscd(25,oversample=16,blurradius=4)
#

The oversample and blur radius settings, as suggested by Scharfis, produce a superbly smooth film effect, but the processing is extremely slow (7 hours for a 5 min clip on my AMD XP2800 rig). The new MVFlowFPS and MVFlowBlur functions would seem to offer a faster method, but I am not entirely clear about the ‘num’ and ‘den’ parameters. Specifically;

1. For a 50fps to 25fps conversion, with a basic setting of num=25, den=1, does MVFlowFPS simply select the original (even) frames or synthesize new frames at 25fps?

2. If new frames are created, does increasing the ‘num’/’den’ values proportionately (50/2, 100/4, 200/8 etc) achieve over-sampling (like mvfps) or have no overall effect on the motion quality?

3. Also, how should MVFlowFPS and MVFlowBlur be used together? I’m assuming:

a=avisource("C:\mvbobbed 50fps.avi").ConvertToYV12()
backward_vec = a.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = a.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
b= a.MVFlowFps(backward_vec, forward_vec, num=25, den=1, ml=100, idx=1)

backward_vectors = b.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vectors = b.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
return b.MVFlowBlur(backward_vectors, forward_vectors, blur=15, idx=1)

Is this correct? (I’m using the latest MVTools v1.0.1 with default chroma=true)

Thanks a lot.

[scharfis_brain]

Why do poeple think about 'converting' 50 fps to 25 fps this weird way?

mvbob().selecteven() will do this job.
(selecting evedy 2nd frame)

[WorBry]

Scharfis_Brain,

Because the motion produced by selecteven() is not very smooth, at least with my DV sources which were all shot with automatic shutter speed (Sony DCR-PC115E). Maybe using a fixed 50fps shutter speed would improve this?

However, I am getting excellent results with your mvfps(scd) function and the oversample/blurradius settings you recommended:

http://forum.doom9.org/showthread.php?p=735419#post735419

I was simply wondering whether the same could be achieved more quickly with MVFlowFPS and MVFlowBlur. From my initial tests however MVFlowFPS (alone) tends to produce some flickering, much like SelectEven, which led me to ask whether, in converting 50fps to 25fps, MVFlowFPS selects even frames anyway.

Thanks.

[scharfis_brain]

Ahh, you want to get motionblur. Cinematic film usually gets exposed 50% the time of the duration of one frame (50% shutter = 1/48 sec with 24 fps).

There mainly are two ways to achieve filmlike motionblur depending on the shutter speed of your source video. Filmlike motionblur means 50% shutter again which is 1/50 sec with PAL-Video (Speedup Filmtransfers, too)

a) if the shutter speed is
1/100 ; 1/150 ; 1/200 or 1/250 sec then it is useful to interpolate the framerate to
2x50 ; 3x50 ; 4x50 or 5x50 fps and afterwards blend together
2 ; 3 ; 4 or 5 Frames and then select every
4th ; 6th ; 8th or 10th frame to return to 25 fps.

This method will produce high quality motionblurring. I obviously becomes slower the shorter the shutter speed is. You could do this even with 1/1000 sec shutter but this won't be useful, cause it would be slow.

b) if you have such a short shutter like 1/1000 sec, it is better to use mvflowblur() and do selecteven() afterwards.

[WorBry]

Thanks Scharfis,

All my archived DV sources were shot in automatic mode, so the shutter speed was probably quite variable. Tried mvflowblur().selecteven() as you suggested. The motion is certainly smoother than selecteven() alone, but none of the alternative methods I've tried match mvfpsscd for motion and image quality, particularly after conversion to MPEG4. The next best method I've found so far is using mvflowfps to interpolate the framerate to 400fps and then performing an average ''temporal merge'' (of luma and chroma) over 16 frames. It gives good fluidity of motion and the encoding is considerably faster than mvfpsscd, but it just doesnt achieve the same look as a good motion blur. Is there any prospect of a faster ''mvfps-like'' function that provides the same options for oversampling and blur?

Thanks.

[Fizick]

WorBry,
Current thread is about MVTools "development".
Please post here your suggestion or bugs reports.
Please post your "usage" question in other thread (in "usage" forum).
http://forum.doom9.org/showthread.php?t=84770&page=9&pp=20

[scharfis_brain]

provide the following:
a) the unprocessed source video (a snipplet of it. NO recompression! at last a few seconds)
b)the script using
- mvfpsscd
- mvflowblur
- 400fps mvflowfps
c) one (the same!) special image of your choice each for the methods with
- mvfpsscd
- mvflowblur
- 400fps mvflowfps
The screenshot should be taken from the middle of the sample video you'll provide.


I need these things to clarify what exacly is your wish.
Thanks.

[WorBry]

Fizick. I receive your point.
Scharfis, I'll correspond with you privately on this. However, I've had to pack my PC away for a few days whilst moving apartment.
Cheers.

[Neil Lee]

Neil Lee,
probably i must restore zero vector predictor preference (Now i give some penalty to it). It is dilemma.

Wow, TrueMotion is absolutely amazing after removing penalty
for zero vector predictor, I find that if you feed it with interlaced
video, TureMotion can become a very good tool for deinterlace.

Of all the de-interlacer I've used, none can do a better job in identifying
moving areas, Most of all, they can't handle a moving gradient area, they all
think it's static, thus leaving a bunch of scattered comb-like residual.

Maybe we can utilize MVMask and Masktool and do a "truemotion de-interlacing",
to help removing unwanted extra residual, Think create a 100% de-int clip and
then mask 2 clips using vector data..) But since I've just started to learn
avisynth, maybe someone wanna give it a shot first?

.

[foxyshadis]

mvbob does just that. ;) Newest version uses mvflow.

[Revgen]

where is the newest one?

[sh0dan]

I have a rather small request: A filter to extract statistical data for conditional filters. I had a short look at MvShow, and it seems like it wouldn't be that much to implement:

Example of code (in GetFrame):

double mean = 0;
for ( int i = 0; i < mvClip.GetBlkCount(); i++ )
mean += mvClip.GetBlock(0, i).GetSAD();

env->SetGlobalVar("mv_avg_sad", AVSValue(mean / nBlkCount));

another could be:

double mean = 0;
[for each block in Y plane] {
x = plane.getXlength();
y = plane.getYlength();
mean += sqrt(x*x+y*y);
}
mean /= scale * nBlkCount;
env->SetGlobalVar("mv_avgY_length", AVSValue(mean));


I imagine a command like MvStats(video, vectors,.....) that would pass through video untouched. It could then set a number of variables that could be used by other filters.

[foxyshadis]

Sorry revgen, scharfis linked to it here (http://forum.doom9.org/showthread.php?p=757259#post757259).

[akoten]

Ahh, you want to get motionblur. Cinematic film usually gets exposed 50% the time of the duration of one frame (50% shutter = 1/48 sec with 24 fps).

There mainly are two ways to achieve filmlike motionblur depending on the shutter speed of your source video. Filmlike motionblur means 50% shutter again which is 1/50 sec with PAL-Video (Speedup Filmtransfers, too)

a) if the shutter speed is
1/100 ; 1/150 ; 1/200 or 1/250 sec then it is useful to interpolate the framerate to
2x50 ; 3x50 ; 4x50 or 5x50 fps and afterwards blend together
2 ; 3 ; 4 or 5 Frames and then select every
4th ; 6th ; 8th or 10th frame to return to 25 fps.

This method will produce high quality motionblurring. I obviously becomes slower the shorter the shutter speed is. You could do this even with 1/1000 sec shutter but this won't be useful, cause it would be slow.

b) if you have such a short shutter like 1/1000 sec, it is better to use mvflowblur() and do selecteven() afterwards.
My problem is that (maybe only in my case) the clip coming out of mvbob() is not usable as input to a subsequent mvanalyse() which is necessary for mvflowblur() or any other motion blur algorithm. Almost none of the motion is being detected and where it's not detected the "behaviour" of the resulting clip is MUCH worse than with a simple "merge(even,odd)".

However, mvbob() works perfectly, so my idea was to use the motion vectors from there. I just tried that but I must admit that I gave up very soon because I was not sure where and how it would be best to integrate the motion blur routines in "mvbob.avs".

Can you help? Maybe a new version of mvbob() with a switch named "smooth" (default="true" for compatibility)?

regards
AKo

P.S. I use "mvflowblur() with "blur=100", because I think that together with selecteven() this is the only way to achieve 50% shutter time on the resulting 25fps movie. Am I right? It's just because I found other posts here that claim values like 15, 50 or the like...

[Didée]

My problem is that (maybe only in my case) the clip coming out of mvbob() is not usable as input to a subsequent mvanalyse() which is necessary for mvflowblur() or any other motion blur algorithm. Almost none of the motion is being detected and where it's not detected the "behaviour" of the resulting clip is MUCH worse than with a simple "merge(even,odd)".
This sounds pretty much like "idx desease". Read carefully the MVTools documentation about the "idx" parameter. In a nutshell, I explained it here (http://forum.doom9.org/showthread.php?p=775411#post775411) (well ... at least I tried to ;) ).

I'm not fully sure how MVBob currently handles the idx parameter ... if it's still the same way that I hacked it in some time ago :o, then it's a hardcoded value - which could be an explanation for the poor results you get.

BTW, an at least halfway safe way of using idx in such a way that no conflicts occur, while still having the benefit of using explicit idx values to spare ressources, is done by initiating and up-counting a global variable for this purpose. Look at the MVNR_simple2 (http://home.arcor.de/dhanselmann/_stuff/MCNR_simple2.rar) script I had once posted - there idx is handled in that way. However it's needed that all involved filters or functions refer to that global counter, else it won't work.

(Should note that this wasn not my idea ... I once saw krieger200? doing it this way, and had the BINGO! effect.)

The other option is to not use idx at all ... which is no problem, as long as your machine has at least 4 GB of RAM installed ... :eek:

[akoten]

One thing to clarify:
I convert a DV source clip to MJPEG using mvbob(). Then I load this MJPEG in another VDub and try to put motion blur to it. This is because I get "low memory" crashes with mvbob all the time (even having 2,5GB of RAM!). So "idx" of mvbob currently can't interfere with the motion blur script.

I think I even tried to have different idx for mvanalyses and mvflowblur in this script (by accident while playing around with all kinds of parameters in mvanalyse, but will need to confirm this in the evening).

AKo

[krieger2005]

I think there is somewhere a bug in the MVTools. When i use complex scripts with MVTools and load the same script more the one time after some times it crashes with low-memory... However, since i use in such much different filters it could be also something other. I can live with that "crashes". And that the MVTools produce that crashes is only a speculation of myself (Maybe i should make a test with different versions... hm...)

[AI]

I think there is somewhere a bug in the MVTools. When i use complex scripts with MVTools and load the same script more the one time after some times it crashes with low-memory... However, since i use in such much different filters it could be also something other. I can live with that "crashes". And that the MVTools produce that crashes is only a speculation of myself (Maybe i should make a test with different versions... hm...)
use version 1.0.1+ (http://bag.hotmail.ru/mvtools/MVTools-v1.0.3.zip) ?

1.0.1 (Released 24.12.2005 by Fizick)

* Fixed memory leakage bug in MVAnalyse with global motion (thanks to AI for report).

[akoten]

use version 1.0.1+ (http://bag.hotmail.ru/mvtools/MVTools-v1.0.3.zip) ?

I also read this thread, but I tried with both 1.0.0 and 1.0.3, both behave the same. Maybe it's another leakage bug that was fixed there.

I'm VERY sure, that one of the tools used in mvbob has a leakage bug. Too many indicators point to this one and only conclusion.

[krieger2005]

I use the newest official version, so 1.0.3.

[Fizick]

I am still did not get memory leakage.

But I returned :)

1)I create a new site (mirror) http://avisynth.org.ru

2)MVTools v1.1.1 is released for download. Thanks to all who supported!
v1.1 (non-public build 9.01.2006 by Fizick)
Quite large revision (beta). New option for overlapped block motion estimation in MVAnalyse for usage in MVFlow, MVFlowInter, MVFlowFps for improved motion compensation.
Lookup tables for motion interpolation.
Small correction of displacement value in MVFlowFps2.
v1.1.1 (16.02.2006 by Fizick)
Removed DeBlock and Corrector filters (will be separate plugins)
Documented old MVSCDetection function.
Cleaned project from unused source files.

3)And New beta version MVTools 1.2 with YUY2 support is developed, it is available for registered users.

[sh0dan]

MVTools 1.2 with YUY2 support is developed
Do you unpack to an internal planar format?

That would be great, since it would make 4:4:4 and 4:2:2 planar formats a breeze for AviSynth 2.6.

[Fizick]

sh0dan,
Yes, I unpuck to internal planar 4:2:2 format (currently in draft, slow, without assembler).
So YV16 support will be trivial.

BTW, I missed your old MvStats suggestion. I will look to it.

[sh0dan]

You can grab the MMX assembler from the 2.6 sources here:

http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/convert/Attic/convert_planar.cpp?view=markup&rev=1.1.2.19&sortby=date&only_with_tag=avisynth_2_6

Look for "conv422toYUV422" and "convYUV422to422". Width must be mod 8, but you already have the C-code for fallback.

[Fizick]

It seems, there is stiil some bug with overlap mode in v1.1.1. At least it produces strange results.

sh0dan,
I add mmx convert in next beta, but it gives 3 percent speed only.

About MVStats: fixed names of global variables are not very good, consider two MVanalyse for example.
May be integer type function like MVAveregeSAD(vector) will be better?

[Fizick]

Today: Soviet Red Army Day!

Released MVTools 1.2.1:
Fixed bug of v1.2,
Speed restored,
mmx YUY2 conversion (from avisynth 2.6 function by sh0dan)
But it seems, overlap mode still does not work properly

[Boulder]

Thanks for the YUY2 support, it's very useful for my captures :)

I'd donate but unfortunately student's monthly income can barely pay the rent and bills..not to mention food :(

[Richard Berg]

Can you release the separate plugin with Corrector & DeBlock? Corrector especially is needed for MvBob.

[Fizick]

Richard,
I released them two days ago :)
(at my new site)

[Fizick]

Boulder,
In any case, your test results and suggestions are welcomed!

[Fizick]

Richard,
Sorry, that day I forgot to add a link to it!
Fixed now.

[Fizick]

sh0dan,
Do not forget fix a bug with frame shift in convertYUY2 function at avisynth 2.6 cvs!
I fixed it in released MVTools v.1.2.2

[scharfis_brain]

mvtools.dll (even the old one without YUY2 support) isn't downloadable anymore.
is this intended?

[Fizick]

scharfis_brain,
probably you use wrong (old) site.
see my signature :)

[scharfis_brain]

I know. But the download link redirects to your old site, which doesn't contain the download anymore. So I am trapped into an endless loop....

If I missed something, please tell me the exact clicks step by step starting from you new stie here: http://avisynth.org.ru/

thanks!

[Fizick]

At avisynth.org.ru click "Fizick's plugins for avisynth" in right (English) part of table,
at new opened page (fizick.html) search and click "mvtools" in table, at opened mvtools.html see last link at page bottom.

May be, your browser cache problem? Try refresh at every step.

Somebody else has some trouble?
I download mvtools right now :)

[scharfis_brain]

see last link at page bottom.

That was the problem. I didn't expect the Download at the bottom of the page!

Mabye the word "download" at the top of the is a little bit misleading...

[Fizick]

O.K., I shall update top link too.

[Boulder]

sh0dan,
I add mmx convert in next beta, but it gives 3 percent speed only.

I'm not 100% sure if you're talking about the same thing, but kassandro's SSETools contains functions called Interleaved2Planar and Planar2Interleaved which are needed if you use any function from the latest RemoveGrain or Repair dlls with YUY2 data.

He apparently hasn't released the sourcecode for SSETools as he calls the latest version preliminary, but as he hasn't been active anywhere lately, you might want to ask him if he could share the code which is supposed to be highly optimized.

[Fizick]

Boulder,
3 percent is for total MVTools speed (MMX conversion itself is faster and quite good optimized).

SSETools uses special packed format for storing YUY2 planar in ordinary YUY2 clip. But new avisynth 2.6 will be use separate format for it. It seems as more pespective. Wait.

[KillNoise]

Ever thought of implementing fast subpixel ME by fitting the subpixel resolution vector to the SAD-curvature samples calculated at integer vector positions ?
Instead of interpolating an intermediate 2x resoution "blow-up" image to perform the search at half-pixel positions, you have to search only in the original size image to find the SAD minimum integer position, then approximate the subpixel position from the values of SAD-minimum and its 4 (or 8) neighbours.
Searching blocks in lower resolution saves a lot SAD-calculations, making ME much faster. Curve-fit approximation of the block vectors (yielding resolution of 1/2-pixel or beyond) can be implemented fast and easy. (If you doubt, ask me to go into detail.)

[Fizick]

I released new MVTools version 1.3.0 (beta?)
Implemented overlapped block motion compensation to MVCompensation(mode=1).
Changed default to mode=1 in MVCompensation as the most universal.
Added sharp subpixel interpolation method for pel=2.
Fixed bug for blksize=16 with YUY2.
(To-do list: assembler SSE speed optimization for new compensation and interpolation methods.)

[scharfis_brain]

would it be possible to implement edge directed internal sampling to increase subpixel accuracy over the linear subpixel accuracy?

[Fizick]

Did you try set sharp=1?

[scharfis_brain]

No I didn't I was just asking.

[Fizick]

Answer: Yes, it is possible to imlement any pixel interpolation. :)
Really I firstly want to add Wiener interpolation. But "true" Wiener (local adaptive) is complex. So, I implement sort of invariant Wiener 6 taps interpolation (do not know why :)).

Coeffs= (1; -5; 20 ; 20; -5; 1)/32

About edge-directed. Really Manao used horizontal, vertical and diagonal bilinear interpolation.

[Didée]

Hey, a big Thank-You, Fizick. Block overlapped compensation is doing a very good job, and the sharp interpolation comes nice, too. :)


- One bug discovered:

An error is thrown when compensating after "padding(x,y)":
vectors = source .padding(4,4) .mvanalyse( [parameters] )
comp = source .padding(4,4) .mvcompensate(vectors, [parameters]) .crop(4,4,-4,-4)
will give "MVTools: MVCompensate does not have a named parameter 'truemotion' / 'blksize' / 'pnew' " etc.pp. :confused:
I noticed this when just plugging MVT 1.3.0 into a script where a manual block overlapped ME+MC is done. Probably no big deal, since padding() isn't documented, hence not used very much. Still, it's not exactly the expected behaviour ...


- A question:

"sharp" interpolation can only be specified in MVAnalyse(). Now, in case if compensation is done with new interpolation:
vectors = source_1 .mvanalyse( sharp=1, idx=a )
comp = source_2 .mvcompensate( vectors, idx=b, mode=1 )
will MVCompensate also use sharp interpolation (reckognizing some flag, or whatever, in the vector clip), or will it use normal bilinear interpolation?


- Question / Suggestion:

Compairing the block overlapped estimation + compensation with the scripted approach I tried, your method seems clearly superior in, say, 80% to 90% of the problematic areas. (Darn you! :D ) However, there are some percent left. ;)

What I tried was a different method, more kind of a "best match":

- create vectors on orignal, and on padded original
- create compensations with both vectors
- weight both compensations acc. to their amount of difference to the original (including some 8x8-block-restricted spatial blurring of all three clips)

It seems feasable that in areas where compensation #1 shows an obvious bigger difference to the original than compensation #2, the weighting of compensation #1 should be reduced (and vice versa).

Does your block summation method already include something like this? If not, it could be worth a try.

[dirio49]

Help

i have no idea how to use it

can somebody post a sample working script with everythink that is need?

thanks

[johnmeyer]

can somebody post a sample working script with everythink that is need?
What exactly are you trying to do? And, what part don't you understand?

[dirio49]

I went to the site and don't really understand what to do?

i just want to try it out in some low bitrate encode

1500 avg bitrate..

I want to try motion blur.

thanks


EDIT:

@Fizick

sorry thanks for pointing it out