Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
31st October 2005, 01:11 | #1 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
New toy: MVTools version with true motion compensation
EDITED: Please post here MVTools DEVELOPMENT questions only.
Use other threads for USAGE questions, for example: http://forum.doom9.org/showthread.php?t=84770 --------------------------------------------------------- I at last build new version of MVTools plugin for local motion estimation and compensation. It is experimental alpha version. The number is 0.9.10. Manao says, "Go ahead!". I am afraid, i go to some side (offside ? ) News: Added some true motion (smoothed) estimation options to MVAnalyse Added function MVFlow for per-pixel motion compensation Added function MVInter for motion interpolation (very experimental) More info: Some new parameters of MVAnalyse are mostly for true motion estimation. Some block may be not so similar to sample blocks, but motion field will be more coherent, instead of some random vectors distribution. It is important for motion interpolation. lsad: SAD limit for lambda using. Local lambda is reset to 0 for vector predictor with greater values. Default=400 (was introduced and fixed internally since v0.9.7). Larger values (about 1000) are recomended for true motion. pnew: penalty to SAD cost for new candidate vector. New vector SAD (cost) must be better than predictor on this value to be accepted. Default=0. Values about 100 are recomended for true motion. plen: penalty factor (similar to lambda) for vector length. to prevent chaotic long vectors. Default=0. Values about 100 are recomended for true motion. plevel: penalty factors (lambda, plen) level scaling mode. Default=0. Value=0 - no scaling, 1 - linear, 2 - quadratic dependance from hierarchical level size. Note, that vector length is small at lower level. MVFlow MVFlow (clip, clip "vectors", float "time", int "mode", int "idx") Do a motion compensation of the frame not by blocks (like MVCompensation), but by pixels. Motion vector for every pixel is calculated by bilinear interpolation of motion vectors of currect and neibour blocks (according to pixel position). It means that the pixels pointed by the mvs in the reference frame will be moved (flow) along the vectors to reach their places in the current frame. This flow motion compensation does not produce any blocking artifactes, but sometimes can create very strange deformed pictures . True motion estimation is strongly recomended for this function. Motion compensation may be full or partial (at intermediate time). time: percent of motion compensation (default=100.0, full compensation), define time moment between reference and current frame. mode can be either 0 ( default ), or 1. mode=0 - fetch pixels to every place of destination. It is main producing mode. mode=1 - shift pixels from every place of source (reference). It is debug (learning) mode with some empty spaces (with null intensity). It may be used for occlusion mask creation. idx works the same way as idx in MVAnalysis. MVInter MVInter (clip, clip "mvbw", clip "mvfw", float "time", int "mode", int "thres", int "idx") Motion interpolation function. Very experimental (alpha version). It is not the same as MVinterpolate function of older MVTools verion. It uses backward "mvbw" and forward "mvfw" motion vectors to create picture at some intermediate time moment between current and next frame. It uses moved blocks from both frames, some combinations of motion vectors, and masked averaging to prevent various artefactes (blockiness, occlusions). The used occlusion mask is cleaned, expanded and smoothed. True motion estimation is strongly recomended for this function. time: interpolation time position between frames (in percent, default=50.0, half-way) mode can be either 0 ( default ), or some number, currently from 1 to 15, they are mostly for learning and experimenting. . 0 - some main (best) production mode, complex interpolation method; 1, 2 - forward or backward shift ; 3, 4 - occlusion mask of forward or backward shift ; 5, 6 - forward or backward fetch; 7 - MC fetched average; 8 - static median; 9 - dynamic median; 10 - use shifted pixels if not occlusion, otherwise use fetched forward; 11 - fetch forward masked by shift backward; 12 - fetch backward masked by shift forward; 13 - time weigthed combination of 11 and 12; 14 - common occlusion mask; 15 - mode 13 mixed with simple time average by occlusion mask, it is mode=0 currently. thres: threshold for count of occlusions per block for mask binarization (default = blocksize*blocksize/4) Download link: http://www.avisynth.org/fizick/mvtools/ ATTENTION: Experimental version, do not use it for real work! But try
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. Last edited by Fizick; 29th December 2005 at 12:26. |
31st October 2005, 12:45 | #2 | Link |
brainless
Join Date: Mar 2003
Location: Germany
Posts: 3,653
|
I'll try it. especially for the Standards conversion stuff
I'd like to see a mvtools version that supports the YUY2 colorspace. May I hope for this, or is it hopeless? Cause building mocomped deinterlacing with proper chroma handling is impossible with YV12. So the current mvbob() still uses simple motion adaption for the chroma handling.
__________________
Don't forget the 'c'! Don't PM me for technical support, please. |
31st October 2005, 14:17 | #3 | Link |
interlace this!
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
|
oooh! must try this at once! thanks for your work
i'll edit this post with feedback rather than take up forum real estate... [edit] no surprise here, but sharp luma changes tend to throw this off a tad (a hand with a cigarette lighter flashing is what i'm looking at atm). mvflow looks really good though. very promising already. [edit 2] scharfi beat me to it, but there's also a rounding error on the luma plane (i ignored the chroma error by just putting "greyscale" after everything). @ scharfi: are we running the exact same script here? [edit 3] take this for now: yv12lut(yexpr="x 1 +",uexpr="x 1 +",vexpr="x 1 +",y=3,u=3,v=3)
__________________
sucking the life out of your videos since 2004 Last edited by Mug Funky; 31st October 2005 at 14:56. |
31st October 2005, 14:21 | #4 | Link |
brainless
Join Date: Mar 2003
Location: Germany
Posts: 3,653
|
possible bugs found:
- mvinter-ed frames are greenish compared to the non interpolated frames. probably a rounding error. - no chroma interpolation with mvinter?
__________________
Don't forget the 'c'! Don't PM me for technical support, please. |
31st October 2005, 15:19 | #5 | Link |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
Looking very interesting! I already had a script at hand to compare Motion against MVTools, so I plugged this in and played with it. The compensated frame is closer to the reference than Motion, although Motion produces a slightly smoother vector field (both as one might expect). MVTools produces fewer blocks, and with MVFlow I couldn't see any artefacts at all on the short clip I was playing back (I didn't try looking at stills).
I'm looking forward to playing with the (impressively many) interpolation modes. Thank you for the new version, Fizick! |
31st October 2005, 15:34 | #6 | Link |
brainless
Join Date: Mar 2003
Location: Germany
Posts: 3,653
|
this source seems to be pain for mvinter() http://tinyurl.com/7kxrb
I prepared it by bobbing an scaling to 704x576 and then applied motion interpolation.
__________________
Don't forget the 'c'! Don't PM me for technical support, please. |
31st October 2005, 17:41 | #7 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
scharfis_brain,
YUY2 wiil be very useful, when somebody implement it. But there are many parts in code where siple relation /2 for croma width and heigth is assumed I think about intermediate solution: Analyse YV12 clip, get vectors, but compensate YUY2 (or planar YV16). But thus we can not use half-pel. By the way, I forgot to say, that half-pel "npel=2" is not implemented for MVFlow and MVInter. Also I forget to say, that vector component values vx, vy must be not above 127 for MVFlow (I use 1 byte arrays for interpolation) Rounding error is quite possible. (division to 255 or 256 ?) And what do you want to say about your 704x576 clip? - no chroma interpolation with mvinter? Really? May be I forget to implement (uncomment) it? Must check it, wait. So, probably i will prepare bugfixed version soon. (without new features). Mug Funky, yes, sharp luma changes is very bad for MVTools. mg262, most modes are not disabled specially for you Manao (or somebody else), I remember you (or somebody else )say about interpolation, that all avisynth resizers is not correct for MVTools Not centered? For MVFlow, I need in interpolation of small array (vx or vy) (one value per block) to full frame. Currently I use very code of fast SimpleResize avisynth plugin by Tom Barry. How I can correct interpolated results to be consistant with MVTools? Shift it to left and top by half-block? |
31st October 2005, 19:15 | #8 | Link | |
Registered User
Join Date: Jan 2002
Location: France
Posts: 2,856
|
Quote:
In the MVTools, in hpel, it works like this : Code:
X 1 X 1 X 1 X 1 X 1 2 3 2 3 2 3 2 3 2 3 X 1 X 1 X 1 X 1 X 1 2 3 2 3 2 3 2 3 2 3
__________________
|
|
1st November 2005, 19:54 | #9 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
New version 0.9.10.1
Fixed bug with chroma and luma small changes in MVInter (rounding error). Vector interpolation in MVFlow and mask in MVInter are correct now (due to fixing bug in SimpleResize). http://www.avisynth.org/fizick/mvtools/ ToDO: pel=2 in MVFlow ?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. Last edited by Fizick; 12th November 2005 at 15:51. |
2nd November 2005, 23:31 | #10 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
I prepared occlusion mask function, but probably the best its place is MVMask.
So, i want replace boolean "showsad" parameter to integer "mode" and robably will change SAD normalization factor and gamma. By the way, MVMask has some bug in top of picture, so i will change its interpolation code to modified SimpleResize (bilinear too). |
3rd November 2005, 03:35 | #11 | Link |
interlace this!
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
|
about SAD and mvmask:
with true motion, SAD can be very high even if the motion is perfectly predicted (luma change on an object for example). is there a good (fast) metric that can tell you "motion accuracy" or "motion confidence" so we can use that for a mask? this would make interpolation much safer as we'd have a good way to tell if there's likely to be artefacts and use weighted blending in those cases. btw, i've been using mvinter to get the opposite of "film look"... it's amazing how well it works even at this early stage. [edit] also, would motion-searching an edge-mask rather than the actual clip make it more immune to luma changes? or would that just mung up other things? [edit 2] oh, btw, last night when i was doubling the framerate of some film and re-interlacing, i noticed memory usage became quite excessive after about 10000 frames worth - like 1.2 gigs of system RAM and 1.4 gigs of virtual memory while encoding with QuEnc. i can post my script, but i should "minimise" it first to eliminate unnecessary stuff.
__________________
sucking the life out of your videos since 2004 Last edited by Mug Funky; 3rd November 2005 at 05:30. |
4th November 2005, 01:25 | #12 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
Mug Funky,
Suggested occlusion measure is a good (fast) metric that can tell you "motion accuracy" or "motion confidence" so we can use that for a mask. I detect bad blocks (occlusion) due to rupture, tensile. If left neibour has vector x coordinate lesser than current block, then we have some tension, gap here (potentially empty space which will be filled by some strange pixels). I count sum of absolute gaps values for 4 neibours. After normalization and resizing we have occusion mask. New kind parameter defines kind of mask in MVMask function. Mode kind=0 creates motion mask from the motion vectors' length. kind=1 allows to build a mask of the SAD values instead of the vectors' length (like old showsad). kind=2 allows to build a occlusion mask (bad blocks due to rupture, tensile). Scpipt to blur problem (blocky) areas of compensated frame with occlusion mask: Code:
vectors = source.MVAnalyse(isb = false, lambda = 1000) compensation = source.MVCompensate(vectors) # or use MVFlow function here # prepare blured frame with some strong blur or deblock function (we use resiser for blur for example): blured = compensation.BilinearResize(width/4, height/4).BilinearResize(width,height) badmask = source.MVMask(vectors, ml=100, kind = 3) overlay(compensation,blured,mask=badmask) # or use more fast MaskedMerge function of MaskTools plugin http://www.avisynth.org/fizick/mvtools/ 2. using edge mask was discussed previously (by Manao in MVTools thread). We lost too many info for algorithm used (SAD comparing). But we can try to use external image intensity stabilizer, such as my DeFlicker (global only). May be some internal correction can be implemented later (for example by using additional cycle with some block intensity variation). But it may result in some errors. I do not know now. Threre are no ideal method, and never will be. try get clear memory leakage condition, and i (and manao ?) will try cure it.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. Last edited by Fizick; 12th November 2005 at 15:52. |
4th November 2005, 04:48 | #13 | Link |
interlace this!
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
|
cool, thanks. i'm just playing with it now.
would it be possible to allow "time" to have negative values? or would that be stupid?
__________________
sucking the life out of your videos since 2004 |
4th November 2005, 17:29 | #15 | Link |
Registered User
Join Date: Jan 2002
Location: France
Posts: 2,856
|
What ? Too weak ? You're meaning Deblock(q=51) is too weak ? Somehow, I don't wanna know what you want to deblock
__________________
|
5th November 2005, 14:47 | #16 | Link | |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Quote:
In case your goal is to do something like --> --> then I already gave a suggestion.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
|
5th November 2005, 16:04 | #18 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
Didée,
Thank for suggestion, but it is not real solution: when we make compensation (forward) from previous to current frame for denoising, we can not use current frame as reference for repair . But it may be used when we make time interplolation, however it will not so good as in your example due to time (and spatial) shift. |
5th November 2005, 16:29 | #19 | Link | ||
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Quote:
It's not fully clear to me at all times, if (or when) you are speaking about direct compensation (compensate neighbor frame to match the current [reference] frame), or about interpolation (create a new [artificial] intermediate frame between neighbor frame and current frame.) For the case of direct compensation, I'm pretty sure the method can be used ... because I've it actually working in a denoiser framework that's drawing a long nose to most others (and that also delivered above example). As opposed to the case of interpolation ... Quote:
For the case of compensation, it does work because we're using 100% of a vector's length, and at *that* position we *do have* a reference that can be used for repairing: just the current frame. However for interpolation, at the position of, say, 50% of a vector's length, we're in free air ... *there*, there is nothing that could be used for repairing.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) Last edited by Didée; 5th November 2005 at 16:36. |
||
5th November 2005, 17:38 | #20 | Link | |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
[QUOTE=Didée]For the case of direct compensation, I'm pretty sure the method can be used ... because I've it actually working in a denoiser framework that's drawing a long nose to most others (and that also delivered above example).
[quote] You MAY use any method But i think, we must use information from one frame only when direct full compensation. What you made it is post-processing. But if we will use this "cured" compensated frame for noise reduction, it will have exactly same pixels values in this cured regions as current frame. Strange denoising. But we can use post-processing with repair. I think, that we firstly need in best possible true compensation. Quote:
We must think and try.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
|
Thread Tools | Search this Thread |
Display Modes | |
|
|