Motion filter : +1st MC FPS-change script for real usage

[mg262]

FPS-change script in this post:
http://forum.doom9.org/showthread.php?p=747288#post747288
______________

This is an implementation of a filter based on some papers that I was pointed to here (http://forum.doom9.org/showthread.php?t=100791) ... (thanks MfA, Fizick!). It needs several features (like scene detection) to be added before its usable in real scripts, so just treat this version as something fun to look at/play with.

Motion, 25 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_25Oct05.dll)

(Edit: HTML documentation/quick reference (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html) is now up... but this filter is still in development, and not meant for general usage.)

Usage is very simple:

#source = ...
motion = findmotion(source, useassembly = true )
compensate(source, motion) #approximate current frame with blocks from previous frame

This script measures at 200 FPS on my P4/2400 on PAL DVD material with AVSTimer. It will slow down when I add subpixel accuracy, but should still be plenty fast!

If you want to see the motion vectors:

source = last
motion = findmotion(source, useassembly = true )
DrawMotion(source, motion) #relatively slow

Frames should be accessed from frame 0 in a linear order. Height and width must be multiples of 8. YV12 only. Assembly is iSSE.

http://people.pwf.cam.ac.uk/mg262/posts/motioncompensationbadger.png

It's taken me nearly all day to build this, so I'm rather out of it... I'm sure there are plenty of things I should mention here but I can't think clearly at present :confused: . I will collect my thoughts in the next couple of days and reply to this and other outstanding threads (as and when real-life permits!)

[Mug Funky]

you are the winrar! i was just trying to build denoisers with MVtools and thought true motion would be very cool to have (if i force mvtools to be too accurate, it gives me an exact copy of the frame to be compensated... not useful for denoising if it does the grain as well :))

i'll test it immediately!

[edit]

hehe... the seeking from frame 0 got me, but apart from that it's looking very interesting. it follows motion pretty well, and isn't sensitive to crap and spots on the screen as far as i can tell. of course it still needs work (nonlinear access, backward prediction, etc).

where are you hoping to take this engine? will there be motion interpolators made? are the motion vectors compatibly with mvtools (so we can use it's interpolators on your more accurate vectors for standards-conversion)? i see a lot of potential here.

btw, what are your thoughts on constructing the compensated image? would mesh-warping or something similar be a good idea considering the motion search is so fast?

whatever happens, this is a sterling effort! thanks for the new tool :)

[Manao]

With the mvtools, you can emulate true motion filter by raising the lambda. It'll add a penalty to the SAD when the motion vector differs from the spatial predictor. Hence, the motion field is far more coherent.

mg262 : good filter. Nice to see somebody working on ME filters.

[Mug Funky]

quick hack for halfpel:

pointresize(last.width*2,last.height*2)
motion=findmotion(last,useassembly=true)
compensate(last,motion)
reduceby2()

[edit]

how are you thinking of handling occlusion and revealing of objects? i suppose that'll come with backward prediction. doing a subtract-from-source shows crossing objects are a problem area. i'm sure you already know all this stuff as you wrote the filter :)

[Manao]

That's not hpel. Hpel would be using bilinearresize as an upsampler. [edit : mm i'm wrong][edit2 : both of us are wrong, hpel can't be emulated that way]

Backward prediction isn't possible for true motion filters ( unless two passes are done, which goes against the principle of of true motion ( the principle being : fast ) ).

[AVIL]

@Mug Funky and Manao

To emulate true motion, perhaps is preferable tweak the parameter "Search" of the MVtools. Actually i use search=3 and searchparam = 3, but I remember have tried with search=1. The goal is limit the radius of search to the maximum expectable difference of position inter-frames of the objects in motion.

[AVIL]

@mg262

An important problem I have with mvtools is the blocks it generates when not appropiate predicted block is found. This block ruins denoising. I know its impossible predict a frame if in the reference frame they are new objects or part of objetcs not visible in the previous or next frames. But i think its possible minimize the problem if the predictor uses the reference block when the predicted block not matches well.

[Didée]

Very nice tool, Clouded. Very Nice. :) :)

(However Avisynth world is turning so fast these days, I can't follow anymore ... dizzy ...)


@ Mug Funky
if i force mvtools to be too accurate, it gives me an exact copy of the frame to be compensated Now, would you please tell how you do to achieve THAT (an *exact* copy) with MVTools ??


@ Manao
That's not hpel. Hpel would be using bilinearresize as an upsampler. [edit : mm i'm wrong][edit2 : both of us are wrong, hpel can't be emulated that way] Do you mean it's not hpel "by definition"? Because, by working on doubled resolution, obviously a ~sort of~ subpixel accuracy is achieved, isn't it.

Backward prediction isn't possible for true motion filters ( unless two passes are done, which goes against the principle of of true motion ( the principle being : fast ) ). For the time being, shouldn't

reverse()
MVstuff()
reverse()

work out ... it's halving the speed when forward & backward are done both, sure. But at least one can "have it" before Clouded implements it ;)


An important problem I have with mvtools is the blocks it generates when not appropiate predicted block is found. This block ruins denoising. That's an issue indeed. Using the builtin MVDenoise, at least you can tweak SAD and other thresholds to circumvent the problem - but it's only a compromise, and not even a good one : in areas with "strong" detail, often one would like blocks with a (relatively) high SAD to still be used. Whereas in "flat" areas, a rather small SAD can already mean visible blocking ...
Lowering the pixel threshold is even more poor, since lots of the potential of MV-compensation is thrown away ...
In custom-build denoising (using MVCompensate & Co., but not MVDenoise) it gets even more difficult, with SAD, vector lenghts etc. only available through dreadful backdoors.

That's why I thought out loud (http://forum.doom9.org/showthread.php?p=724570#post724570) about a "new" deblocking filter. The already existing deblocking filters are not suited to deblock compensated frames, if they are to be used for denoising: if the deblocking is cranked up enough to catch all blocking, all noise will be killed from the compensation, too ... dead end.
(The proposed method should work, I think ... but I can't dive into it, currently.)

[mg262]

@Manao, @Mug Funky, Thank you for the kind comments :).
i'm sure you already know all this stuff as you wrote the filter :o Afraid not... it's just an implementation of standard methods. I haven't even tested/examined the algorithm (as opposed to implementation correctness) to any great degree. So feel free to throw things at me ... where are you hoping to take this engine? There are four features I had in mind before I started coding:
Scene awareness (see below)
Sub-pixel accuracy
Feature-based motion estimation similar to this (http://www.ics.ele.tue.nl/~dehaan/pdf/97_VCIP04Braspenning.pdf)*
Mesh-warping similar to this (http://www.utdallas.edu/~aria/papers/csvt01.pdf)
*(A very low/nonexistent priority after seeing results from the basic version.)

Beyond these, I'm not sure. I'm certainly not going to fiddle with the core algorithm -- looking through the papers you can see the algorithm evolving and being extensively tested over a decade, with the result that the current version is very fine-tuned.will there be motion interpolators made? are the motion vectors compatibly with mvtools (so we can use it's interpolators on your more accurate vectors for standards-conversion)? i see a lot of potential here.Certainly possible (compatibility via a translation filter). The only thing I would say is that de Haan's website (papers Fizick linked to) has a huge amount of material on true-motion motion-compensated standards conversion and deinterlacing, and IMO we should assimilate that material before diving in -- scripts/methods appropriate for MVTools may turn out to be inappropriate for this filter.

Linear access, unidirectionality:
The algorithm was AFAICS designed for standalone hardware compatibility, so the limitations (linear access, unidirectionality) make good sense. As Manao says, there is no way of doing backwards prediction without at least a 2x slowdown. OTOH, with subpixel switched off, this may often be acceptable. I was thinking of working scene by scene, scanning forwards and backwards on each scene, to minimise space usage.

In any case, the filter will need to handle scene breaks. I strongly prefer scene detection to be done once and scene information to be passed to each filter -- for separation of functionality as much as efficiency. AVISynth has no native support for this, so I will pass scene information in a clip. Seeking to the 100th frame in a scene will then cost as much as seeking from frame 0 to frame 100 at present.* (Faster seeking is possible but ugly -- more on this later.)

*Each scene behaves like a separate clip passed through the filter. I sometimes consider an Animate-like meta-filter to do this to any filter.

Last: Development is likely to be sporadic as I'm beginning to get joint fatigue from interface issues... sorry to make you wait.

Clouded

Edit: AVIL, Didée, just seen your posts...

[krieger2005]

Is this not possible to use the results (output) of this filter as an input for MVTools (for Denoising) using the MVTools-Function "mvchangecompensate"?

But for this a motion-compensated-clip have to be done before by MVTools, or not? Just an idea...

[mg262]

Motion, 26 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_26Oct05.dll)

FindMotion(source, int initialise = ..., useassembly = ..., int reset = ... )

I added a quick feature to FindMotion to facilitate testing. There is an optional argument called reset; every reset frames, the filter calculates motion without looking at the motion computed for the previous frame. So (slow) seeking is possible.

Additionally, whenever a frame is calculated without motion vectors from the previous frame being available, the filter iterates the basic algorithm initialise times (default 0); if you don't set initialise, the motion vectors will reset to 0 every reset frames. Even a fairly small value of initialise should produce results that are very close to the non-reset version.

(Detail: if reset is 50, frames 1, 51, 101, etc are computed without looking at previous frames. So seeking to frame 175 will compute 25 frames. I could have set it to reinitialise for the exact frame seeked to, but this would be nondeterministic -- and very unfriendly to Didée's Reverse() script.)
________________________________________________________________

Didée: I'm glad you like it :). As for this:
An important problem I have with mvtools is the blocks it generates when not appropiate predicted block is found. This block ruins denoising.The true-motion algorithm has a tendency to try and use contiguous (touching) blocks from the previous frame,* which should reduce this effect -- so play with the filter and see what you think. It will also be interesting to try the mesh-warping compensation (don't expect it soon!) to see if that helps. But I could at some point give you a mask of the per-block SAD if that is any use?

*AVIL: I'm not sure what you mean by reference block, but the algorithm would generally choose something that is better than the corresponding block in the previous frame.

Clouded

Edit: What about a stand-alone mask-creating BlockSAD(clip, clip) filter?

[AVIL]

@mg262

I call the reference frame to the real frame. In motion compensation we are trying to simulate this reference frame by evolving blocks of another frame, previous, next, 2 past, etc.... In this context I will call this frame the "seed" frame.

Then, I call reference block to a block in the reference frame. When the block choose from the "seed" frame don't simulate well his correspondant reference block I suggested to use the reference block instead to build the mo-comped frame.

Sorry for my poor english.

[mg262]

AVIL, understood. What you want could be done by postprocessing using the kind of filter I suggested at the end of the last post.

@all, it's just struck me that I may need to be careful about patents? Does anyone know what the situation actually is?

[Manao]

You don't sell it --> you don't care about patents. It's at least true in some countries.

[MfA]

With the mvtools, you can emulate true motion filter by raising the lambda. It'll add a penalty to the SAD when the motion vector differs from the spatial predictor. Hence, the motion field is far more coherent.Predictive searches can get trapped in local minima though ... now for coding they get it right enough of the time for it not to matter much, but for image processing it makes sense to do a little more work and use feature based matching to get extra candidates.

As for patents, if you are really worried you can always only release in source code form ... Im not aware of anyone ever being sued for contributory infringement for releasing source code.

[Fizick]

@Mg262,
You are fast code writer! :)
Probably even faster then me. :)
I will try your filter today.

IMHO, BlockSAD function will not useful. I try similar way. The better SAD is not better block for true motion.
Occlusion info is more important.

@Manao,
By the way, recently (last week) I tried to improve MVTools in two directions:
1. True motion :)
Strictly as you say above, http://forum.doom9.org/showthread.php?p=728904#post728904
I added a penalty to the SAD when the motion vector differs from the spatial predictor.
my parameter draft name is "penalty" :)
2. Motion interpolation.
It uses some (clever ?) combination of both shifted and fetched blocks (forward and backward), and temporal average of some regions from two source frames to produce output.

The speed is not fast, of course.

I have some alpha version, but my work is still not finished. I want:
1. to limit of motion vector length (at every level probably ?)
2. to use most clever blocks combination for interpolation (i try many ways).

I wanted to publish my modified version next week (if my real life give some time), after some conversation with you.
But mg262 work provoke me to publish it more quickly. :)

Will it be politically correct? :) Or you work at MVTools too now?

[Manao]

Fizick : please, go ahead. I've got no time to put my hands on the MVTools in the near future, so you're really welcome to add features to it.

The penalty you're speaking about already exist : int cost = sad + MotionDistorsion(vx, vy);

inline int MotionDistorsion(int vx, int vy)
{
int dist = SquareDifferenceNorm(predictor, vx, vy);
return (nLambda * dist) >> 8;
}> to limit of motion vector length (at every level probably ?) /* computes search boundaries */
nDxMax = nPel * (pSrcFrame->GetPlane(YPLANE)->GetExtendedWidth() - x[0] - nBlkSize);
nDyMax = nPel * (pSrcFrame->GetPlane(YPLANE)->GetExtendedHeight() - y[0] - nBlkSize);
nDxMin = -nPel * x[0];
nDyMin = -nPel * y[0];

but for image processing it makes sense to do a little more work and use feature based matching to get extra candidates.Indeed. But the main predictor I use come from hierarchal me, which tends to be rather good.

[AVIL]

@mg262

I find useful your suggested filter in scope of postprocessing or even like a wide-usage filter. IMHO appart from the two clips must be a parameter for block size, more colourspaces, and could be another parameter for distinct block-difference calculations (sum(SQR(x**2 - y**2)), by example).

I`ve gived your filter a try. I've found it more blocky and less accurate than MVtools, but I think is because of the lack of hqpel. The blockiness is especially hard at the vertical borders of the image and frequent in the edges with hard luma stepping.

I`ve subtract the mo-compend frame with the reference frame. An ideal mo-comped frame only give, as result of subtraction, an arbitrary image with the inter-frame noise. All the similarities between the subtracted image and the real one are product of blocks not accurately calculated.

This filter is ver promisory.

@Fizick

A new filter (or a new version of and old one) is always a good new. Thanks

[mg262]

AVIL,

I looked at the issue; first I tried a number of minor variations on the method to see if they had any substantial effect, but the essential behaviour was unchanged. But supersampling (as Mug Funky does) certainly helps... so I think that you are right and subpixel estimation is particularly important for this algorithm, as it changes not only the accuracy but also the average step size (which affects motion vector convergence)... so when I can I will implement that and we can see how it looks.
__
Made a minor fix+change to improve behaviour of the new parameters. Please re-download Motion, 26 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_26Oct05.dll).

[Didée]

@Mg262,
You are fast code writer! :)
Note that in this very case it's not only true, but particularily impressive...


/* bows down deeply */


/* stumbles, and falls */ :D

[mg262]

Thanks, guys... but the beauty of AVISynth (as a package) is that it does all the slow to write, reference-requiring, technology-dependent parts like opening files and rendering, so that's where most of the credit should go!

I briefly tested sub pixel (1/4 pixel) accuracy in a rough form in C, and it helps a little, but the blocking and accuracy still much worse than that of MVTools. The last time I tried MVTools was at least 18 months ago (it's pretty rare that I work with non-animated material), and I hit substantial blocking... but now, at least on the source at hand, blocking seems very rare and its fast -- this runs at 50FPS:*

mvanalyse(source)
mvcompensate(source, last)

I was pretty disappointed with the comparative performance of this method -- the papers on it certainly give the impression that as well as being super-fast and producing true motion, it gave very good results with minimal blocking. It's possible that I made some silly mistake (and I'll check through), but the vector fields look pretty sensible. My particular interest was in trying to develop this for denoising scripts, and that seems relatively pointless now. Don't take that the wrong way -- I'm quite happy to try other things with it, it's just that denoising was the target I was heading towards. So, I'm not really sure where if anywhere to go from here... thoughts?

*Motion compensated scripts (e.g. MVBob) have the reputation of being really slow... why is this?

____________________
I ran across one simple deblocking method for truemotion motion compensation which I might as well repeat here:

Obtain motion vectors, then subdivide each block into four.
For each top-left subblock:

Replace the motion vector with the median of
-- original motion vector
-- vector from block to the left
-- vector from block above

Median is computed in x and y separately. (+Optionally, you discard the vector if it doesn't much one of the three above.) Other 3/4 sub blocks work in the obvious way.

AFAICS, the effect is to take a stair step and reduce to a stair step of half the size. The method is particularly intended not to introduce new motion vectors which didn't exist before.

[Fizick]

Main aim of true motion is motion interpolation.

[mg262]

Quick note: I took a guess at the meaning of a word ("meandering") apparently defined in some papers I can't get hold of... whether or not I guessed right, it seems to help under some circumstances. So...

Motion, 28 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_28Oct05.dll)

If you really want to try quarter-pixel accuracy the slow C way, add subpixel = true and useassembly = false. It only runs at 30 FPS or so on 720 x 576, so for testing purposes I would recommend cropping down your clip until you can watch it in real time. (I probably won't have time to assemblify it for a while.)

[mg262]

Motion, 29 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_29Oct05.dll)
Bug fix for meandering and slightly faster assembly integer search code.

I've been browsing through papers and I think that at least half-pixel accuracy is needed to do anything useful (because of the half-pixel shift between fields). Quarter-pixel accuracy is preferable -- although for the assembly, I can't at present see a way to deal with quarter-pixel (without sacrificing speed) other than by coding 16 cases... ... ... . (Joy.) So I will probably fight towards that and then think about applications.

@Mug Funky, if you're reading this, anything particular you would like to see this filter do?

how are you thinking of handling occlusion and revealing of objects? i suppose that'll come with backward prediction. doing a subtract-from-source shows crossing objects are a problem area.I have read more about the problem domain now (there are downsides to diving in headfirst ;) ), so I'll try and answer this. First, it's not going to be dealt with at the motion detection stage, simply because there's no way of representing the information without complicating the datastorage format substantially. I.e. an object that has become revealed can't be sensibly represented with any motion vector. Second, depending on the application, we can try and infer this information from the motion vector field; for example we can look for discontinuities, and fallback to some simpler method (for whatever the application is).

Incidentally, I am not sure how well this kind of method will work on foreground objects in animation; the algorithm is not terribly likely to deal well with the fact that their motion is typically at a lower FPS than the motion of the main clip...

[psme]

Tried both (P4 2.66 overclocked to 3.8G):

motion = findmotion(source, useassembly = true )
compensate(source, motion)
(around 20% CPU usage)

and

mvanalyse(source)
mvcompensate(source, last)
(around 55% CPU usage)

in FFDShow->Avisynth for real time DVD playback. Both methods give smooth playback. But both methods give tons of pixelization/blocking artifact in motion area.

I guess the filter is not yet usable...

regards,

Li On

[mg262]

@psme,

Motion compensation filters aren't as far as I know intended for direct use in playback... they are intended to provide high-quality input for temporal processes, such as temporal denoising, frame rate conversion (as Fizick mentions) or deinterlacing.


@all,

Edit: scratch what was here... I have thought of a modification that may produce the backwards motion information without requiring backward tracking.

[castellandw]

I mentioned in some other post about motion compensation using phase correlation being used by video engineers, especially for broadcast standards conversion (PAL to NTSC, NTSC to PAL, etc). Mug Funky mentioned that this could be accomplished by block overlapped motion compensation. Of course, MVTools can be used for motion compensation, but Fizick recently created a Block Overlap Plugin along with a sample script to use with MVTools to create block overlap motion compensation (I tested out Fizick's block overlap plugin with MVTools and it did produce better results). Mug Funky also mentioned that phase correllation could be achieved with the DePan plugin, but also some other tricks need to be involved. Mug Funky, do you know what other tricks need to be involved for AviSynth since it's possible to achieve block overlapped motion compensation in Avisynth (Block Overlap Plugin By Fizick with MVTools) and with phase correlation (possibly Depan plugin)?

@mg262, that message thread where my link to the pdf on Snell Converter's guide to motion compensation was posted, I posted the link to that message thread on another site with big expert video engineers, and one video engineer told me that motion compensation was revolutionary in 1994 and no other technique was surpassed it. So what I'm saying is that those papers posted on that message thread, especially papers by De Haan or true motion estimation using feature correspondences don't compare to that motion compensation technique with phase correlation according to that guy.

@all, any comments on what I mentioned about motion compensation?

[mg262]

That sounds very interesting... although I'm not sure how to read this:
one video engineer told me that motion compensation was revolutionary in 1994 and no other technique was surpassed it.... most of the stuff I'm looking at is post-94 -- although maybe that's not what you mean?

I would really like to see the site you refer to if you happen to have the link at hand...

[Joel Cairo]

mg262:

Unless I'm mistaken, he's referring to the Doctor Who Restoration Team website, which is staffed by the members of that team-- some of whom are BBC engineers. You can find them at:

http://www.rtforum.co.uk/

THe thread is currently near the very bottom of their page, and it's titled "Snell Converter's Motion estimation methods said to be dated". You'll have to check it quickly, though, as topics drop off their board after a couple of weeks, and I don't believe there's an archive.

As I read the discussion though, I actually felt that our discussion was somewhat mis-represented. I don't recall anyone here saying that the Snell converter's inner workings were dated...

Anyway-- that's where you can find the discussion.

-Kevin

[mg262]

Thank you for the link:

http://www.rtforum.co.uk/read.php?id=128649
Charlie S
Snell Converter's motion estimation methods said to be dated

Guys, I'm at my wits end. Steve Bagley, if you're reading this, I posted the link to that paper on motion compensation from Snell Converters that you gave me and posted it on another website forum. The reaction I got says that the paper being from 1994 is very sad being dated because of still using blending techniques, and I think I'll post this link:

http://forum.doom9.org/showthread.php?t=100791
I find this a little confusing... it reads as if I posted it (which I didn't). Perhaps Charlie S posted a similar query on a different forum but linked here instead?

[Joel Cairo]

Well, I think this is where the misunderstanding began. There's a post just below yours in that thread (by Mfa), which makes the ironic point that the document you linked was written over 10 years ago, but that people still weren't employing the techniques that are outlined in that document. Instead, they're still using blending, etc.

The way Mfa's message was written, however, it evidently was mis-interpreted by the poster as meaning that blending, etc. was part of the method outlined in that document-- which is not correct, if I'm understanding all this properly.

So I think that's how all of this confusion got started.

-Kevin

[Fizick]

Discussion is interesting, there are many ways.
Very interesting, for example, recent bi-directionanal motion estimation and interpolation.

[castellandw]

Guys, it's a bit hard to follow you guys from one forum to another(and a bit of a headache) because the message I posted in the Doctor Who restoration team forum has been a few weeks old, and so I'm unaware where you guys are posting what unless you tell me. Yes, I posted the message in the Doctor Who restoration forum to get a second eye because I knew I could have misunderstood this message thread:
http://forum.doom9.org/showthread.php?t=100791

@Joel Cairo aka Kevin, I would have gathered that I misunderstood, especially when James Insell at the Doctor Who restoration team forum told me that no blending is being used for motion compensation. When I posted it, I assumed that when Mfa said stuff like blending is still being used, I thought he was referring to the Snell converter paper which I seem to be the first to post on the Doom9 forum:
http://forum.doom9.org/showthread.php?p=718712#post718712 (http://forum.doom9.org/showthread.php?p=718712#post718712)
Also, I was referring to Snell converter's methods being possibly dated compared to other papers posted as well which are papers by De Haan and true motion estimation using feature correspondences. Thus from there, James Insell told me that no other method has surpassed motion compensation. I didn't try to misrepresent the discussion here because I posted the link to that thread to get a second eye on the subject.(By the way, Kevin, since you mentioned discussion on the Doom9 forum being misrepresented, was there any other discussion on Snell converters apart from that link to that pdf file being discussed because I never managed to really find any?)
One more thing, I didn't post the Doctor Who restoration team forum link http://www.rtforum.co.uk here because when I asked about what they thought of PAL to NTSC conversion done on AviSynth, they said I should go to Video Editing forums, and so I had to limit my posts there with simpler explanations of techniques as best as I could when I referred to Avisynth software. So I knew it wouldn't be worth it to acknowledge the Doctor Who restoration team link if they might likely say to you guys elsewhere to go to other forums if you mention video editing software.

@mg262, I did post that Snell Converter paper first on the Doom9 forum apparently, and I assumed that you found that first from my post. I had no idea you found the paper first on Google because I didn't realize it was easily findable on Google. You're right, it does seem as if you posted as Charlie S, sorry about that, but I thought the reaction from your post was because of me posting the link to that Snell Converter paper first in the forum. I didn't post a link to my original post on Doom9 with the Snell Converter pdf link on the Doctor Who restoration team because the reaction to your post was more relevant. By the way, Mohan, you better post what you put on the Doctor Who restoration team over here as well because it'll make things very confusing for other posters looking too and as Kevin said, messages on the Doctor Who restoration team forum drop off..
Here's again the link where I originally posted the Snell Converter paper:

http://forum.doom9.org/showthread.php?p=718712#post718712 (http://forum.doom9.org/showthread.php?p=718712#post718712)

I see Kevin, has found this link from the Doctor Who restoration team:
http://www.rtforum.co.uk/read.php?id=128649
but did you guys manage to find another one I posted which has now dropped off the Doctor Who restoration team forum. Well, here's the link again on my original post involving the link to that pdf on Snell Converters involving motion compensation as well as discussion by Mug Funky on how it could be possible to do motion compensation using phase correllation:
http://forum.doom9.org/showthread.php?p=719376#post719376



Here's Fizick's post on the block overlap plugin I mentioned earlier that can be used with MVTools:
http://forum.doom9.org/showthread.php?t=101307

[castellandw]

Oh yeah, a British sci-fi show about an alien that travels through time and space in a machine that's bigger than the inside than it is on the outside is stereotypically British. I can see that coming, Mfa.

Anyway, as I mentioned in the other post I just posted, MfA, I wasn't referring only to your comments because there was mention of papers by de Haan being mentioned on that Snell Converters paper, and your comments about blending can be understood in more than one way.

[MfA]

I'd appreciate if you never speak for me again ... I don't appreciate having to go into a strange forum to clear up a gross misrepresentation of what I said only to get deleted. It sets bad blood all around, you are pissed because you are told off. They are pissed because they think I slagged Snell Wilcox and because they think the thread went way too far off topic, and my double posting probably didnt help either, and this will reflect in their opinion on Doom9 and everyone involed. Lastly I am pissed, because it is very hard to reply in any way which wont turn this into a flamewar which will probably result in me getting a warning.

[castellandw]

Mfa, they didn't delete the message thread because of your post or being pissed at you. If you noticed, that message thread was the oldest one on the board, and so it just dropped off. Nothing to do at all with any slagging to Snell-Wilcox or the thread going way off-topic. As long as everybody was discussing technical video bits on the forum, it wasn't going off-topic and I gave the link to that thread so there is no way they'd think you're just slagging off Snell-Wilcox or actually care judging by how much attention was given to that message thread. Hey, lots of people double-post on that forum, so you're not the only one. By the way, I was not speaking for you, I was trying to get a second eye from those guys at the Doctor Who restoration team forum because they're the best video engineers, especially from their restoration work they've done, I could find to give a second opinion on all the replys to that message thread on motion compensation in general. It was my misunderstanding on your post, but they are not pissed off at you or anyone at Doom9 forum. Believe me. Besides, when I mentioned Avisynth, they practically asked me to ask about AviSynth in other video editing forums until I tried to limit my posts to just regular video techniques in general. Look, as I said, you did nothing whatsoever to piss them off, and the message thread outlasted its own timespan on that forum.

[Mug Funky]

@Mug Funky, if you're reading this, anything particular you would like to see this filter do?

i think that's already been covered - mesh warping would be very cool, and it seems it's been on your list from the start. i'm really just happy to see stuff being developed so actively for avisynth (particularly by you - i think you're a contender for "most new plugins written per month". i'm in awe).

actually, a "delta" option similar to that of mvtools would be great for denoising.

i'm sure the algos you've implemented cover this to some degree, but if frame interpolation is involved, things like object acceleration and whatnot will have to be covered to some extent. i guess that would mean sampling at least 3 frames to derive the velocity/acceleration of an object anywhere in the middle. this is probably already done if i'm understanding correctly.

as far as the frame-rate issue with foreground/background in animation, don't be worried about that. not even the PhC stuff handles that properly (the paper itself says "garbage in, garbage out" when talking about something as dead-simple as telecined material... that could easily be handled in hardware by a separate hard match IVTC running before the PhC part, and i'm sure modern machines do just that). certainly i've seen plenty of stuff on TV that's gone through some kind of mocomped conversion, and it's clearly failed badly. "iron chef" is an example. fuzzy subtitles, and an ex-composite source (dot crawls) make it a difficult source. typically a medium-fast pan is handled perfectly, but any faster and it's blended. subtitles + mesh-warping aren't a good mix either. i see no shame in falling back on field-blending in lots of cases. for small areas of a frame, it's much more transparent than warpage or other artefacts.

@ castellandw:

about using depan... i said that partly in jest. though it could be done, there are severe limitations. you'd have to chop the clip into large blocks (depan will only find motion that fits within the frame), then do overlapped compensation, then split into smaller blocks, run again, then split into smaller blocks.... it'd be slow and poor quality (the overlapped blocks will not match in most cases), but would work as a proof of concept for the phase-corellation method. there's definitely many more ways to skin that cat though :). i certainly wouldn't want to script a frankenstein's monster script like that though.

[castellandw]

@Mug Funky, that sounds like it could use a recursive hashing algorithm although if it's poor quality particularly with overlapped blocks not matching, then I begin to wonder how video engineers can produce results with motion compensation using phase correllation. So every time it separates into smaller blocks, phase correlation can be performed before overlapped-block motion compensation. So, what exactly what is at least one example of a way that it can can be done to "skin that cat" of dealing with overlapping blocks not matching because I'm wondering how it's possible that video engineers produce decent results with standards converters?

By the way, Fizick and anyone involved in development with MVTools should look at this discussion here on motion compensation using phase correllation because it might prove extremely helpful.

[Mug Funky]

well, the only reason the way i described would be bad is because in depan only half the job is done - it makes no attempt to find motion vectors, rather it performs phase corellation on a whole frame to find global motion.

in a PhC unit (from reading the paper), there is a phase corellation stage that finds several (i guess) local maxima that correspond to motion-vectors, and then there's a stage where it matches the motion vectors to actual locations on the screen. this makes it far far far superior to just matching global motion on ever smaller blocks.

[mg262]

@castellandw,

Now I follow. (That paper is the first hit when Googling "motion compensation".) What I posted on that forum amounts to little more than Fizick said above; it was mainly to elicit a more detailed response from James Insell. Now the thread is gone, let's leave things be.

@Mug Funky,

actually, a "delta" option similar to that of mvtools would be great for denoising.It's trickier in this instance because each motion estimate depends on the motion estimate for the previous frame... I could create something that mimicked SelectEven.FindMotion, but I think the found motion vector quality would degrade. On the other hand, this is perfectly legit:

#source = ...
motion = findmotion(source, useassembly = true )
source
compensate(motion)
compensate(motion)

I.e., get the motion vectors once, move blocks to compensate original frame n-1 -> n, then move blocks to compensate compensated frame n-1 -> n. So we have:

original frame n-2 -> compensated frame n-1 -> doubly-compensated frame n

Will that do? I can definitely implement something else, given specifics.

By the way, using that ^^ script will make the blocking/inaccuracy worse for obvious reasons. There are a range of things which can be done to improve the motion vector quality, but I think I'm going to try and get an application out first... it makes it easier for us to check whether a particular change is actually helping. For similar reasons, I would like to have better quality motion vectors before trying mesh warping.

From Phase Correlated Motion Estimation (http://www-ise.stanford.edu/class/ee392j/projects/projects/liang_report.pdf):
Phase-correlation ME is very computationally efficient and it produces much smoother motion field with low entropy than the BM method does. Phase-correlation works better than the BM does in the cases of large scale translational motion, while BM is more suitable for predicting regular and small scale motion and multiple-object movement.

(Bear in mind that that compares phase correlation against full/logarithmic search block matching... actually I think it's some student report, so doesn't carry that much weight in itself -- but the conclusion looks very sensible to me.)

[castellandw]

So basically for phase correlation motion estimation, it's a mix of phase correllation first (for large scale motion) and then overlapped block motion compensation (for small scale motion and multiple object movements)?
(I say overlapped block motion compensation instead of block-matching motion compensation because overlapped block MC is said to be more optimal than block-matching MC:http://forum.doom9.org/showthread.php?p=719376#post719376 )

[Mug Funky]

it's a little simpler, but basically what you said.

instead of taking a block and finding a vector, phase corellation finds a load of vectors and passes them to some other algo which finds the block for that vector. it essentially works backwards. requires more stages and is conceptually more complicated than block matching, but in the end there's a fair bit less work for the computer to do.

i might have thrown you off a tad with the depan thing...

[mg262]

Minor point, but could cause confusion along the line... I want some standard terminology. Ideas on sensible names for

a) the frame we are trying to approximate

and

b) the frame from which we are extracting blocks to be used to approximate a)

, please?

I particularly ask because I've seen the word "reference" being used for both a) and b)...

[Mug Funky]

what about "source frame" and "target frame", but then these could get confused too i suppose. i'm thinking of source frame as being the one blocks are moved from, and target frame is the frame blocks are being moved to match.

for forward compensation, think of source frame as n-1 and target frame as n.

[MfA]

I use the terms forward reference and backward reference (although I use the terms exactly in reverse as they are usually used in video coding, where they use the direction of prediction rather than the direction in time to determine what's forward/backward).

[mg262]

I'm trying to forwards/backwards/previous/next as much as possible, partly because of this:
I use the terms exactly in reverse as they are usually used in video coding, where they use the direction of prediction rather than the direction in time to determine what's forward/backward
Both conventions make sense in their own way, but having both around at once can make it very confusing for the reader! Plus, much of the time you want to describe the compensation process itself in a way that is direction-independent...

Source and target are much better, but still a bit prone to confusion. I think I might go with calling a) "current"... that kind of makes sense for both directions.

[castellandw]

Minor point, but could cause confusion along the line... I want some standard terminology. Ideas on sensible names for

a) the frame we are trying to approximate

mg262, don't you think we should also use separate terminology for block portions of a frame as well because a load of vectors are being used from a block portion of a frame to approximate the (as Mug Funky put it, possibly "local maxima") vector for that block portion of the frame?

In any case, I can see that motion estimation using phase correlation uses various things such as the fourier transform to output the motion vectors used for motion compensation, but do you guys think this can be really implemented for Avisynth or is motion estimation using phase correllation and then motion compensation look like it's not gonna happen because it would be great to see how it works out on broadcast standards conversion?

[mg262]

mg262, don't you think we should also use separate terminology for block portions of a frame as well because a load of vectors are being used from a block portion of a frame to approximate the (as Mug Funky put it, possibly "local maxima") vector for that block portion of the frame?I'm sorry, I don't understand this comment.

Motion estimation using phase correlation... I don't see why wouldn't be possible in AVISynth; it just requires someone to want to code it.

[castellandw]

Quote:
"mg262, don't you think we should also use separate terminology for block portions of a frame as well because a load of vectors are being used from a block portion of a frame to approximate the (as Mug Funky put it, possibly "local maxima") vector for that block portion of the frame?"

I'm sorry, I don't understand this comment.


Well, you asked that you wanted standard terminology for "the frame we are trying to approximate", but since each motion vector approximated from phase correlation are derived from each portion (a.k.a. block) of the frame, do we need separate terminology (besides "source frame", "target frame" and "current frame") for each of those blocks of the frame to avoid even more confusion (Basically, to distinguish between discussing a whole frame and a block or portion of the frame)?

[mg262]

Unlike the backward/forwards related terminology, I can't at present see a situation in which that kind of confusion could arise*... but at the moment I'm not using phase plane correlation, so it's rather academic.

*although I'm open to be proved wrong ... you could suggest an example of a sentence which could be misconstrued with two sensible readings.

_______________________________________

Maybe this is obvious in the Development Forum, or maybe I should have said it before, but anyway: like most other temporal filters which don't explicitly state otherwise, this filter currently takes no account of interlacing. If you are using interlaced content, either deinterlace or bob first. (I do actually plan to make it treat interlaced content sensibly... but there are a couple of prerequisites.) Simply separating fields is a bad idea because the up-down jitter will mess with the true-motion algorithm.