Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 31st October 2005, 01:11   #1  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
New toy: MVTools version with true motion compensation

EDITED: Please post here MVTools DEVELOPMENT questions only.
Use other threads for USAGE questions, for example:
http://forum.doom9.org/showthread.php?t=84770
---------------------------------------------------------

I at last build new version of MVTools plugin for local motion estimation and compensation.
It is experimental alpha version. The number is 0.9.10.
Manao says, "Go ahead!".
I am afraid, i go to some side (offside ? )

News:
Added some true motion (smoothed) estimation options to MVAnalyse
Added function MVFlow for per-pixel motion compensation
Added function MVInter for motion interpolation (very experimental)

More info:

Some new parameters of MVAnalyse are mostly for true motion estimation. Some block may be not so similar to sample blocks, but motion field will be more coherent, instead of some random vectors distribution. It is important for motion interpolation.

lsad: SAD limit for lambda using. Local lambda is reset to 0 for vector predictor with greater values. Default=400 (was introduced and fixed internally since v0.9.7). Larger values (about 1000) are recomended for true motion.

pnew: penalty to SAD cost for new candidate vector. New vector SAD (cost) must be better than predictor on this value to be accepted. Default=0. Values about 100 are recomended for true motion.

plen: penalty factor (similar to lambda) for vector length. to prevent chaotic long vectors. Default=0. Values about 100 are recomended for true motion.

plevel: penalty factors (lambda, plen) level scaling mode. Default=0. Value=0 - no scaling, 1 - linear, 2 - quadratic dependance from hierarchical level size. Note, that vector length is small at lower level.

MVFlow

MVFlow (clip, clip "vectors", float "time", int "mode", int "idx")

Do a motion compensation of the frame not by blocks (like MVCompensation), but by pixels. Motion vector for every pixel is calculated by bilinear interpolation of motion vectors of currect and neibour blocks (according to pixel position). It means that the pixels pointed by the mvs in the reference frame will be moved (flow) along the vectors to reach their places in the current frame. This flow motion compensation does not produce any blocking artifactes, but sometimes can create very strange deformed pictures . True motion estimation is strongly recomended for this function. Motion compensation may be full or partial (at intermediate time).

time: percent of motion compensation (default=100.0, full compensation), define time moment between reference and current frame.

mode can be either 0 ( default ), or 1.
mode=0 - fetch pixels to every place of destination. It is main producing mode.
mode=1 - shift pixels from every place of source (reference). It is debug (learning) mode with some empty spaces (with null intensity). It may be used for occlusion mask creation.

idx works the same way as idx in MVAnalysis.



MVInter

MVInter (clip, clip "mvbw", clip "mvfw", float "time", int "mode", int "thres", int "idx")

Motion interpolation function. Very experimental (alpha version). It is not the same as MVinterpolate function of older MVTools verion. It uses backward "mvbw" and forward "mvfw" motion vectors to create picture at some intermediate time moment between current and next frame. It uses moved blocks from both frames, some combinations of motion vectors, and masked averaging to prevent various artefactes (blockiness, occlusions). The used occlusion mask is cleaned, expanded and smoothed. True motion estimation is strongly recomended for this function.

time: interpolation time position between frames (in percent, default=50.0, half-way)

mode can be either 0 ( default ), or some number, currently from 1 to 15, they are mostly for learning and experimenting. .
0 - some main (best) production mode, complex interpolation method;
1, 2 - forward or backward shift ;
3, 4 - occlusion mask of forward or backward shift ;
5, 6 - forward or backward fetch;
7 - MC fetched average;
8 - static median;
9 - dynamic median;
10 - use shifted pixels if not occlusion, otherwise use fetched forward;
11 - fetch forward masked by shift backward;
12 - fetch backward masked by shift forward;
13 - time weigthed combination of 11 and 12;
14 - common occlusion mask;
15 - mode 13 mixed with simple time average by occlusion mask, it is mode=0 currently.

thres: threshold for count of occlusions per block for mask binarization (default = blocksize*blocksize/4)


Download link:
http://www.avisynth.org/fizick/mvtools/

ATTENTION: Experimental version, do not use it for real work!
But try
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.

Last edited by Fizick; 29th December 2005 at 12:26.
Fizick is offline   Reply With Quote
Old 31st October 2005, 12:45   #2  |  Link
scharfis_brain
brainless
 
scharfis_brain's Avatar
 
Join Date: Mar 2003
Location: Germany
Posts: 3,607
I'll try it. especially for the Standards conversion stuff

I'd like to see a mvtools version that supports the YUY2 colorspace.
May I hope for this, or is it hopeless?

Cause building mocomped deinterlacing with proper chroma handling is impossible with YV12. So the current mvbob() still uses simple motion adaption for the chroma handling.
__________________
Don't forget the 'c'!

Don't PM me for technical support, please.
scharfis_brain is offline   Reply With Quote
Old 31st October 2005, 14:17   #3  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,547
oooh! must try this at once! thanks for your work

i'll edit this post with feedback rather than take up forum real estate...

[edit]

no surprise here, but sharp luma changes tend to throw this off a tad (a hand with a cigarette lighter flashing is what i'm looking at atm). mvflow looks really good though. very promising already.

[edit 2]

scharfi beat me to it, but there's also a rounding error on the luma plane (i ignored the chroma error by just putting "greyscale" after everything).

@ scharfi: are we running the exact same script here?

[edit 3]

take this for now:

yv12lut(yexpr="x 1 +",uexpr="x 1 +",vexpr="x 1 +",y=3,u=3,v=3)
__________________
sucking the life out of your videos since 2004

Last edited by Mug Funky; 31st October 2005 at 14:56.
Mug Funky is offline   Reply With Quote
Old 31st October 2005, 14:21   #4  |  Link
scharfis_brain
brainless
 
scharfis_brain's Avatar
 
Join Date: Mar 2003
Location: Germany
Posts: 3,607
possible bugs found:
- mvinter-ed frames are greenish compared to the non interpolated frames. probably a rounding error.

- no chroma interpolation with mvinter?
__________________
Don't forget the 'c'!

Don't PM me for technical support, please.
scharfis_brain is offline   Reply With Quote
Old 31st October 2005, 15:19   #5  |  Link
mg262
Clouded
 
mg262's Avatar
 
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
Looking very interesting! I already had a script at hand to compare Motion against MVTools, so I plugged this in and played with it. The compensated frame is closer to the reference than Motion, although Motion produces a slightly smoother vector field (both as one might expect). MVTools produces fewer blocks, and with MVFlow I couldn't see any artefacts at all on the short clip I was playing back (I didn't try looking at stills).

I'm looking forward to playing with the (impressively many) interpolation modes. Thank you for the new version, Fizick!
mg262 is offline   Reply With Quote
Old 31st October 2005, 15:34   #6  |  Link
scharfis_brain
brainless
 
scharfis_brain's Avatar
 
Join Date: Mar 2003
Location: Germany
Posts: 3,607
this source seems to be pain for mvinter() http://tinyurl.com/7kxrb

I prepared it by bobbing an scaling to 704x576 and then applied motion interpolation.
__________________
Don't forget the 'c'!

Don't PM me for technical support, please.
scharfis_brain is offline   Reply With Quote
Old 31st October 2005, 17:41   #7  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
scharfis_brain,
YUY2 wiil be very useful, when somebody implement it.
But there are many parts in code where siple relation /2 for croma width and heigth is assumed
I think about intermediate solution: Analyse YV12 clip, get vectors, but compensate YUY2 (or planar YV16).
But thus we can not use half-pel.
By the way, I forgot to say, that half-pel "npel=2" is not implemented for MVFlow and MVInter.
Also I forget to say, that vector component values vx, vy must be not above 127 for MVFlow (I use 1 byte arrays for interpolation)
Rounding error is quite possible. (division to 255 or 256 ?)

And what do you want to say about your 704x576 clip?

- no chroma interpolation with mvinter?
Really? May be I forget to implement (uncomment) it? Must check it, wait.

So, probably i will prepare bugfixed version soon.
(without new features).

Mug Funky,
yes, sharp luma changes is very bad for MVTools.

mg262,
most modes are not disabled specially for you


Manao (or somebody else),
I remember you (or somebody else )say about interpolation, that all avisynth resizers is not correct for MVTools
Not centered?
For MVFlow, I need in interpolation of small array (vx or vy) (one value per block) to full frame.
Currently I use very code of fast SimpleResize avisynth plugin by Tom Barry. How I can correct interpolated results to be consistant with MVTools? Shift it to left and top by half-block?
Fizick is offline   Reply With Quote
Old 31st October 2005, 19:15   #8  |  Link
Manao
Registered User
 
Join Date: Jan 2002
Location: France
Posts: 2,856
Quote:
that all avisynth resizers is not correct for MVTools
When I said not correct, it was related to the fact that SAD computation for block matching are made in the interpolated plane only. So, basically, if you use bilinearresize to compute halfpel, and if you search halfpel position, the SAD will be computed between the halfpel plane and the current block ( without considering the 3 other halfpel planes ). So you can use avisynth's resizers ( you just have to ensure that the pel plane is preserved, which can be done ), but you've got afterwards to reorganize the data into planes in order SAD computations to be efficient.

In the MVTools, in hpel, it works like this :
Code:
X  1  X  1  X  1  X  1  X  1
2  3  2  3  2  3  2  3  2  3
X  1  X  1  X  1  X  1  X  1
2  3  2  3  2  3  2  3  2  3
1 is interpolated by averaging both its horizontal neighbours, 2 both its vertical neighbours and 3 its 4 four neighbours. The 1 to the right of an X correspond to the vector (0.5,0), and so on. I hope it'll answer your second question.
__________________
Manao is offline   Reply With Quote
Old 1st November 2005, 19:54   #9  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
New version 0.9.10.1
Fixed bug with chroma and luma small changes in MVInter (rounding error).
Vector interpolation in MVFlow and mask in MVInter are correct now (due to fixing bug in SimpleResize).

http://www.avisynth.org/fizick/mvtools/

ToDO: pel=2 in MVFlow ?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.

Last edited by Fizick; 12th November 2005 at 15:51.
Fizick is offline   Reply With Quote
Old 2nd November 2005, 23:31   #10  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
I prepared occlusion mask function, but probably the best its place is MVMask.
So, i want replace boolean "showsad" parameter to integer "mode"
and robably will change SAD normalization factor and gamma.
By the way, MVMask has some bug in top of picture,
so i will change its interpolation code to modified SimpleResize (bilinear too).
Fizick is offline   Reply With Quote
Old 3rd November 2005, 03:35   #11  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,547
about SAD and mvmask:

with true motion, SAD can be very high even if the motion is perfectly predicted (luma change on an object for example). is there a good (fast) metric that can tell you "motion accuracy" or "motion confidence" so we can use that for a mask? this would make interpolation much safer as we'd have a good way to tell if there's likely to be artefacts and use weighted blending in those cases.

btw, i've been using mvinter to get the opposite of "film look"... it's amazing how well it works even at this early stage.

[edit]

also, would motion-searching an edge-mask rather than the actual clip make it more immune to luma changes? or would that just mung up other things?

[edit 2]


oh, btw, last night when i was doubling the framerate of some film and re-interlacing, i noticed memory usage became quite excessive after about 10000 frames worth - like 1.2 gigs of system RAM and 1.4 gigs of virtual memory while encoding with QuEnc. i can post my script, but i should "minimise" it first to eliminate unnecessary stuff.
__________________
sucking the life out of your videos since 2004

Last edited by Mug Funky; 3rd November 2005 at 05:30.
Mug Funky is offline   Reply With Quote
Old 4th November 2005, 01:25   #12  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Mug Funky,
Suggested occlusion measure is a good (fast) metric that can tell you "motion accuracy" or "motion confidence" so we can use that for a mask.
I detect bad blocks (occlusion) due to rupture, tensile. If left neibour has vector x coordinate lesser than current block, then we have some tension, gap here (potentially empty space which will be filled by some strange pixels).
I count sum of absolute gaps values for 4 neibours.
After normalization and resizing we have occusion mask.

New kind parameter defines kind of mask in MVMask function.
Mode kind=0 creates motion mask from the motion vectors' length.
kind=1 allows to build a mask of the SAD values instead of the vectors' length (like old showsad).
kind=2 allows to build a occlusion mask (bad blocks due to rupture, tensile).

Scpipt to blur problem (blocky) areas of compensated frame with occlusion mask:
Code:
vectors = source.MVAnalyse(isb = false, lambda = 1000)
compensation = source.MVCompensate(vectors) # or use MVFlow function here
# prepare blured frame with some strong blur or deblock function (we use resiser for blur for example):
blured = compensation.BilinearResize(width/4, height/4).BilinearResize(width,height) 
badmask = source.MVMask(vectors, ml=100, kind = 3)
overlay(compensation,blured,mask=badmask) # or use more fast MaskedMerge function of MaskTools plugin
Download:
http://www.avisynth.org/fizick/mvtools/

2. using edge mask was discussed previously (by Manao in MVTools thread). We lost too many info for algorithm used (SAD comparing). But we can try to use external image intensity stabilizer, such as my DeFlicker (global only). May be some internal correction can be implemented later (for example by using additional cycle with some block intensity variation). But it may result in some errors. I do not know now.
Threre are no ideal method, and never will be.

try get clear memory leakage condition, and i (and manao ?) will try cure it.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.

Last edited by Fizick; 12th November 2005 at 15:52.
Fizick is offline   Reply With Quote
Old 4th November 2005, 04:48   #13  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,547
cool, thanks. i'm just playing with it now.

would it be possible to allow "time" to have negative values? or would that be stupid?
__________________
sucking the life out of your videos since 2004
Mug Funky is offline   Reply With Quote
Old 4th November 2005, 17:21   #14  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
What is the most strong deblocking filter? I want to use it with mask with strong settengs, but internal deblock is too weak
Fizick is offline   Reply With Quote
Old 4th November 2005, 17:29   #15  |  Link
Manao
Registered User
 
Join Date: Jan 2002
Location: France
Posts: 2,856
What ? Too weak ? You're meaning Deblock(q=51) is too weak ? Somehow, I don't wanna know what you want to deblock
__________________
Manao is offline   Reply With Quote
Old 5th November 2005, 14:47   #16  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
Quote:
Originally Posted by Fizick
What is the most strong deblocking filter? I want to use it with mask with strong settengs, but internal deblock is too weak
With time, I've come to the conclusion that deblocking filters are not too well suited to repair artefacts from erroneous compensation. Simple as that.

In case your goal is to do something like


--> -->


then I already gave a suggestion.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 5th November 2005, 15:45   #17  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
I am not afraid to kill details in good regions, since i will use occlusion mask for it.
Fizick is offline   Reply With Quote
Old 5th November 2005, 16:04   #18  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Didée,
Thank for suggestion, but it is not real solution:
when we make compensation (forward) from previous to current frame for denoising, we can not use current frame as reference for repair .
But it may be used when we make time interplolation, however it will not so good as in your example due to time (and spatial) shift.
Fizick is offline   Reply With Quote
Old 5th November 2005, 16:29   #19  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
Quote:
Originally Posted by Fizick
Didée,
Thank for suggestion, but it is not real solution:
when we make compensation (forward) from previous to current frame for denoising, we can not use current frame as reference for repair .
Oh, we can not? Pity ... but then, what's your explanation for my example?

It's not fully clear to me at all times, if (or when) you are speaking about direct compensation (compensate neighbor frame to match the current [reference] frame), or about interpolation (create a new [artificial] intermediate frame between neighbor frame and current frame.)

For the case of direct compensation, I'm pretty sure the method can be used ... because I've it actually working in a denoiser framework that's drawing a long nose to most others (and that also delivered above example).

As opposed to the case of interpolation ...
Quote:
But it may be used when we make time interplolation, however it will not so good as in your example due to time (and spatial) shift.
... where I can't see how that method should work at all.
For the case of compensation, it does work because we're using 100% of a vector's length, and at *that* position we *do have* a reference that can be used for repairing: just the current frame. However for interpolation, at the position of, say, 50% of a vector's length, we're in free air ... *there*, there is nothing that could be used for repairing.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

Last edited by Didée; 5th November 2005 at 16:36.
Didée is offline   Reply With Quote
Old 5th November 2005, 17:38   #20  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
[QUOTE=Didée]For the case of direct compensation, I'm pretty sure the method can be used ... because I've it actually working in a denoiser framework that's drawing a long nose to most others (and that also delivered above example).
[quote]
You MAY use any method
But i think, we must use information from one frame only when direct full compensation.
What you made it is post-processing.
But if we will use this "cured" compensated frame for noise reduction, it will have exactly same pixels values in this cured regions as current frame. Strange denoising.

But we can use post-processing with repair.

I think, that we firstly need in best possible true compensation.


Quote:
Originally Posted by Didée
As opposed to the case of interpolation ...
... where I can't see how that method should work at all.
For the case of compensation, it does work because we're using 100% of a vector's length, and at *that* position we *do have* a reference that can be used for repairing: just the current frame. However for interpolation, at the position of, say, 50% of a vector's length, we're in free air ... *there*, there is nothing that could be used for repairing.
Yes, but sometimes (in some regions) we do not have any good information, so even not-compensated frame blending (locally only, with smooth occlusion mask) is good (best of availiable). It is not simple problem.
We must think and try.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:57.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.