MVTools

[SubJunk]

Thanks for the reply
The reason I want it isn't because of speed (it is irrelevant because I encode), but because of the quality.
blksize=16 on 720p content is a great setting for most content IMO, while blksize=16 on 1080p content is worse, as is blksize=32 on 1080p content.
It didn't matter much to me before because I mostly watched 720p content, but now I am mostly watching 1080p content so I would like the same interpolation benefits I had with 720p content but with the resolution increase.

[pbristow]

So ideally you want a blocksize somewhere between 16 and 32, such as 24, but that's not available with MVTools2 as it stands.

What overlap setting are you using? You may be able to approximate the effect of a blocksize of 24, by using 32 with a larger overlap. (e.g. blocksize=24 with overlap=4 has the same inter-block spacing as blocksize=32 with overlap=12 (spacing is 20)) A fairly large overlap is advisable when using large blocks anyway, to avoid any large gaps appearing between blocks that diverge from each other.

[SubJunk]

Hi pbristow, thanks for the ideas
I tried them and it seems it doesn't work like that; the overlap didn't seem to bring it close to it.
I compared blksize=16, overlap=4 with blksize=32, overlap=20 and blksize=32, overlap=16 but it seems that overlap doesn't have much effect when using blksize=32.

Attached are examples.
The first one is the 720p clip (resized to 1080p afterwards for this comparison) using blksize=16 and overlap=4.
The second one is 1080p using blksize=32 and overlap=20.

So the second one is just like an average of 2 frames, while the first is fairly accurate at merging the panning parts.

[pbristow]

OK, that's weird... The features it's failing to track seem big enough that a 32x32 block should be able to lock onto them OK. That belt-pouch next to the tail, for example, should be no trouble at all.

Does this problem occur throughout the movie, or only on certain scenes? (High speed action, for example?) The problem may be with the limitation of vector lengths, rather than anything to do with block sizes. With a 50% wider & taller frame showing the same action, all the movement vectors will be 50% longer in pixel terms, so in regions of fast movement they may be just too long for mvtools2 to handle.

Some other things to check:

0. First, double-check whether the 1080p source contains all clean progressive frames, or if it's actually a 1080i30 encode of a 24fps original that's being mis-read as 1080p, as that would lead to blended frames like the one I'm seeing above.

1. Are you definitely using the latest version of mvtools2? The 32 blocksize option is pretty new, and I don't know if the older version would abort when you set blocksize=32, or just default to a smaller size, or something. If you have an older version sitting in your plugins directory, that could be overriding the newer one...?

2. Check that your script is handling the two cases exactly the same way (is it exactly the same script running with a different input file?)

3. Consider whether any other mvtools parameters need scaling to match the larger block size you're setting, or the larger feature size of the source.

Hope this helps.

[-Vit-]

Originally Posted by SubJunk

Is there a way to make MVTools2 get vectors from a smaller source and apply them to a bigger source?

I have a little plugin for this. It's going to be used in a later version of QTGMC for optimization. It hasn't been tested much:

Download MScaleVectors 0.1

It lets you use motion vectors on a different sized clip than they were analyzed on. Clip sizes must be related by powers of 2.

Example

scale = 2  # 2, 4 or 8. How much to downscale the clip for analysis (and then how much upscale the resultant vectors)
pad   = 16 # Padding used in full-scale super clip, must be divisible by scale (downscaled padding needed in downscaled super clip)

clip = YourSource( "Your\Video" )
clipScaled = clip.BicubicResize( clip.Width()/scale, clip.Height()/scale )

superScaled = clipScaled.MSuper( pel=2, hpad=pad/scale,vpad=pad/scale )
bVecScaled  = superScaled.MAnalyse( blksize=8, isb=true ) # Blocksize can't go > 32 after scaling
fVecScaled  = superScaled.MAnalyse( blksize=8, isb=false )

bVec = bVecScaled.MScaleVectors( scale ) # Increase blocksize for vectors
fVec = fVecScaled.MScaleVectors( scale )

super = clip.MSuper( pel=2, hpad=pad,vpad=pad )
clip.MFlowFPS( super, bVec, fVec, den=0 )

Other Documentation

INFO

Plugin that scales MVTools motion vectors / blocksize. Main purpose is to allow vectors to be used on a differently
sized clip than they were analyzed from.

For example
- Use MAnalyze on a half-sized clip at block size 16
- Use this plugin to scale the vectors by 2 to block size 32
- Use resulting vectors for MFlowFPS, MDegrain,... on the full sized frame. 
Saves doing the MAnalyze on the full size frame, which may be faster and saves memory (good for multi-threading).

Note that you need a super clip for each frame size. The padding (hpad, vpad) on each super clip must be
manually scaled to match the vector scaling. See the example below.

Similar functionality was available in MVTools through the function MVIncrease, but it was removed.


SYNTAX

MScaleVectors( int Vectors, float Scale, int Mode )

Returns a new motion vectors clip

  Vectors :
     Motion vectors returned from MAnalyse
     [No default]

  Scale :
    Mode = 0,1: Scale is the amount to up/downscale blocksize, only valid values are 2,4 or 8. Blocksize cannot drop below 4 or exceed 32
    Mode = 2: Scale is (floating point) amount to scale the vectors, blocksize is unaffected. Only the range 0.0 to 1.0 is currently safe
    [Default Scale = 2 (for default Mode=0)]

  Mode :
    Method of scaling:
    Mode = 0: Increase the blocksize by 2, 4 or 8 (see above)
    Mode = 1: Decrease the blocksize by 2, 4 or 8
    Mode = 2: Blocksize is not scaled, only vectors themselves
    [Default Mode = 0]

    Mode 0 allows you to MAnalyse at a smaller size, then process (MDeGrain, MFlowFPS, whatever) at full size
    Mode 1 allows you to MAnalyse at full size, but have some subsequent processes run at a smaller size
    Mode 2 is for esoteric uses

[SubJunk]

Great idea Vit, thanks a lot! I'll check it out soon

[SubJunk]

Should these 2 scripts produce the same output (except the first with less definition than the second because of the resize)?
They produce different results and I'm trying to see why, I'm sure I'm missing something very obvious

Input = video
Input.BicubicResize(Input.Width()/2, Input.Height()/2)
Super = MSuper(Input, hpad=8, vpad=8, rfilter=4)
backward = MAnalyse(Super, blksize=16, searchparam=4, plevel=0, isb=true , badSAD=2000, overlap=4, search=3, dct=5, pelsearch=3)
forward  = MAnalyse(Super, blksize=16, searchparam=4, plevel=0, isb=false, badSAD=2000, overlap=4, search=3, dct=5, pelsearch=3)
backward = MRecalculate(Super, backward, blksize=8, searchparam=1, overlap=2, search=3, dct=5)
forward  = MRecalculate(Super, forward,  blksize=8, searchparam=1, overlap=2, search=3, dct=5)
MFlowFps(Input, Super, Backward, Forward, num=48000, den=1001, ml=10000)
Spline36Resize(Input.Width()*2, Input.Height()*2)

Input = video
InputScaled = Input.BicubicResize(Input.Width()/2, Input.Height()/2)
Super = MSuper(InputScaled, hpad=8, vpad=8, rfilter=4)
backward = MAnalyse(Super, blksize=16, searchparam=4, plevel=0, isb=true , badSAD=2000, overlap=4, search=3, dct=5, pelsearch=3)
forward  = MAnalyse(Super, blksize=16, searchparam=4, plevel=0, isb=false, badSAD=2000, overlap=4, search=3, dct=5, pelsearch=3)
backward = MRecalculate(Super, backward, blksize=8, searchparam=1, overlap=2, search=3, dct=5)
forward  = MRecalculate(Super, forward,  blksize=8, searchparam=1, overlap=2, search=3, dct=5)
BackwardFull = backward.MScaleVectors(2)
ForwardFull  = forward.MScaleVectors(2)
SuperFull = MSuper(Input, hpad=16, vpad=16, rfilter=4)
MFlowFps(Input, SuperFull, BackwardFull, ForwardFull, num=48000, den=1001, ml=10000)

[-Vit-]

Originally Posted by SubJunk

I'm missing something very obvious

The first script ignores the downsize. Should be something like:

Input = video
Input = Input.BicubicResize(Input.Width()/2, Input.Height()/2)
...

[SubJunk]

Duh

Edit: After fixing that error the results are still different, is there another mistake?

[-Vit-]

I tested those scripts, corrected, no problems. How are the results different?

[SubJunk]

Originally Posted by -Vit-

I tested those scripts, corrected, no problems. How are the results different?

They are mostly the same but there are slight differences. I wonder if hpad and vpad are being used properly, since it seems to occur more frequently at edges.

Example attached.

Edit: Removed attachments since there are newer ones in my next post

[-Vit-]

Haven't looked at your images because they're still pending. But after looking at the edges on some examples I'm beginning to wonder if hpad/vpad aren't causing problems themselves. Because explicitly setting them both to 0 removes several major edge artefacts for both versions of the script above. As the MScaleVector version is clearer the problems are more obvious on that version. Try setting hpad/vpad to 0 and see if you're still getting a difference (in fact do hpad/vpad do anything useful for this example?).

[SubJunk]

Originally Posted by -Vit-

Try setting hpad/vpad to 0 and see if you're still getting a difference

It doesn't seem to be the hpad/vpad settings after all. :s

Here are screenshots of the 2 scripts with hpad/vpad=0:
First - Second

The scripts used in those screenshots are much simpler than before in order to rule out as many variables as possible:

Input = video
InputScaled = Input.BicubicResize(Input.Width/2, Input.Height/2)
Super = MSuper(InputScaled, hpad=0, vpad=0)
backward = MAnalyse(Super, isb=true)
forward  = MAnalyse(Super, isb=false)
BackwardFull = backward.MScaleVectors(2)
ForwardFull  = forward.MScaleVectors(2)
SuperFull = MSuper(Input, hpad=0, vpad=0)
MFlowFps(Input, SuperFull, BackwardFull, ForwardFull, num=48000, den=1001)

Input = video
Input = Input.BicubicResize(Input.Width/2, Input.Height/2)
Super = MSuper(Input, hpad=0, vpad=0)
backward = MAnalyse(Super, isb=true)
forward  = MAnalyse(Super, isb=false)
MFlowFps(Input, Super, Backward, Forward, num=48000, den=1001)
Spline36Resize(Input.Width*2, Input.Height*2)

[-Vit-]

This is either a question for Fizick, or you will have to wait until I have time to go through the MVTools2 source code in detail. I'm busy with other things atm. The plugin scales the block size, overlap, padding and the vectors themselves. Would expect that to work, and it broadly does [I have a version of QTGMC that uses it successfully]. Your differences are on fast/complex motion - without some detailed code knowledge it's hard to say if this is what to expect or not [the motion vector interpolation might correctly be different between these scripts].

However, I did just notice that I didn't scale the SAD per-block (need to check the MVTools2 source to see if it is internally scaled, but I would guess it isn't) but I found no difference when I changed it. My test case was simpler, so you might want to try. So here's a version of MScaleVectors that scales the block SAD too:

Download MScaleVectors 0.11

[SubJunk]

Originally Posted by -Vit-

This is either a question for Fizick, or you will have to wait until I have time to go through the MVTools2 source code in detail. I'm busy with other things atm. The plugin scales the block size, overlap, padding and the vectors themselves. Would expect that to work, and it broadly does [I have a version of QTGMC that uses it successfully]. Your differences are on fast/complex motion - without some detailed code knowledge it's hard to say if this is what to expect or not [the motion vector interpolation might correctly be different between these scripts].

However, I did just notice that I didn't scale the SAD per-block (need to check the MVTools2 source to see if it is internally scaled, but I would guess it isn't) but I found no difference when I changed it. My test case was simpler, so you might want to try. So here's a version of MScaleVectors that scales the block SAD too:

Download MScaleVectors 0.11

You're right that both scripts are mostly the same, there is probably little practical difference between them. It seems in my short tests that neither one produces superior interpolation to the other; for every area that is better in the first script there is another that is worse.

For the record, I tried again with MScaleVectors 0.11 and the 2 scripts still produce different results.

Thanks a lot for the plugin, it's really cool

[pbristow]

Hey -Vit-, does MScaleVectors allow one to negate/reverse the original vectors, i.e. multiply them by -1? If not, could that be added?

That would get us halfway toward using MVTools for extrapolation, as opposed to interpolation. (See this post by Didee: http://forum.doom9.org/showthread.ph...40#post1498840)

E.g. to extrapolate Frame N+1 from the previous two frames, you would take the backward vectors that run from frame N to N-1, and negate them before feeding them MFlowInterp as forward vectors (the appropriate flags would need to be set, I think, to allow them to be seen as forward vectors. I tried swapping the vector clips around when calling MFlowInterp once, and it complained). For the other (backward) set of vectors, give MFlowInterp a set of zero vectors (i.e. indicating no movement), all with a confidence rating of zero (i.e. "don't trust these at all"). We'd need a new function to create such a null vector set, which should be dead easy to produce. MFlowInterp should then shift the blocks of Frame N forward in the same direction, by the same amount, as they moved when coming from N-1 to their positions in N.

[-Vit-]

Originally Posted by pbristow

Hey -Vit-, does MScaleVectors allow one to negate/reverse the original vectors, i.e. multiply them by -1? If not, could that be added?

It almost does. That's one of the "esoteric uses" of mode 2. I nearly replied to the other thread, but....
I haven't actually finished mode=2 yet, hence the comment in the docs about "Only the range 0.0 to 1.0 is currently safe". Using scale <0 or >1 in mode 2 works sometimes and crashes sometimes, because when you reverse or lengthen the vectors you need to ensure they don't point outside the frame. I will add that code later. I only posted the plugin because it already had the block size scaling that SubJunk wanted.

Basic extrapolation:
- Calculate backward vectors (bVec) from Frame-1 to Frame-0
- fVec = bVec.MScaleVectors(Scale=-1.0, Mode=2)
- MCompensate Frame-1 using fVec to get approximation of unknown Frame-2 (only need one set of vectors for MCompensate)

There are a few more vector tools I might add, when I find the time:
- Converting a set of vectors (b3,b2,b1,f1,f2,f3) to an ordinary clip, which can be saved and reused. This would be useful for a *very* slow script I'm working on that needs to operate in several passes.
- Statistics on vectors for a frame (faster than analysing MMask clips)
- Combining vectors (V' = A.V1 + B.V2), could be useful combined with above
- Relating sequential vectors (by pixel), i.e. what is the motion of the target of this pixel, next frame? Feels useful...
- etc.

[pbristow]

Originally Posted by -Vit-

when you reverse or lengthen the vectors you need to ensure they don't point outside the frame.

Actually, that's exactly the case that interests me. If the *full length* vector points outside the frame, that doesn't mean the vector isn't useful, as you may only want to interpolate to a point in time, say, 50% of the way between frame N and frame N+1, based on previous motion in frames N-1 and N. The case that interests me especially at the moment is rate-doubling a scene where a person falls out of shot between two frames. With current MVtools methods, the frame created in between the last appearance of the falling body and the first frame where it's absent has a kind of shrivelled, distorted & faded version of the falling body left in the place where it was last seen. Now, some distortion of the body probably can't be avoided; but what matters is that it should *not* appear frozen space, but continue to fall along its original trajectory.

I'd suggest that you allow switching between two or more strategies for "beyond the frame" vectors, e.g.:
1. Zero them, and give them zero confidence ratings (the "no tolerance" option! )
2. Truncate them to the edge of the frame, and give them low confidence ratings (the "right direction, wrong speed" option);
3. Let them point to where they naturally would (i.e. don't modify them because of the frame boundary), and calculate confidence ratings as you would for any other vector (the "this is the truth, whether you can handle it or not" option. ).

The idea with option 2 is to avoid any problems if whatever filter the vectors are passed on to (e.g., MFlowInterp or MCompensate) can't cope with any vector that points beyond the frame: You can still at least indicate that the object/block keeps moving in that general direction, even if the rate of movement is reduced.

Option 3 will only be useful if the following filter will accept vectors regardless of where they point, and then handle them appropriately... Which I *hope* MFlowInterp does, or at least can be made to with a slight modification.

[Jeremy Duncan]

What would be cool is if there was a MSCDetection built into mflowfps or mblockfps so the thscd could be tweaked on a single colored screen. So the MSCDetection would be off by default and could be activated by a parameter. I find having the thscd1 and thscd2 properly calibrated to be very important to frame doubling code.

[SubJunk]

I wonder if it would be possible for the hpad and vpad parameters to be used without causing warping on the edges.
Hpad and vpad are cool and they work, but the tradeoff is that the borders are sometimes "pulled" in and the only way to stop that is to set hpad and vpad to 0.
example