Motion filter : +1st MC FPS-change script for real usage

[mg262]

FPS-change script in this post:
http://forum.doom9.org/showthread.php?p=747288#post747288
______________

This is an implementation of a filter based on some papers that I was pointed to here (http://forum.doom9.org/showthread.php?t=100791) ... (thanks MfA, Fizick!). It needs several features (like scene detection) to be added before its usable in real scripts, so just treat this version as something fun to look at/play with.

Motion, 25 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_25Oct05.dll)

(Edit: HTML documentation/quick reference (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html) is now up... but this filter is still in development, and not meant for general usage.)

Usage is very simple:

#source = ...
motion = findmotion(source, useassembly = true )
compensate(source, motion) #approximate current frame with blocks from previous frame

This script measures at 200 FPS on my P4/2400 on PAL DVD material with AVSTimer. It will slow down when I add subpixel accuracy, but should still be plenty fast!

If you want to see the motion vectors:

source = last
motion = findmotion(source, useassembly = true )
DrawMotion(source, motion) #relatively slow

Frames should be accessed from frame 0 in a linear order. Height and width must be multiples of 8. YV12 only. Assembly is iSSE.

http://people.pwf.cam.ac.uk/mg262/posts/motioncompensationbadger.png

It's taken me nearly all day to build this, so I'm rather out of it... I'm sure there are plenty of things I should mention here but I can't think clearly at present :confused: . I will collect my thoughts in the next couple of days and reply to this and other outstanding threads (as and when real-life permits!)

[Mug Funky]

you are the winrar! i was just trying to build denoisers with MVtools and thought true motion would be very cool to have (if i force mvtools to be too accurate, it gives me an exact copy of the frame to be compensated... not useful for denoising if it does the grain as well :))

i'll test it immediately!

[edit]

hehe... the seeking from frame 0 got me, but apart from that it's looking very interesting. it follows motion pretty well, and isn't sensitive to crap and spots on the screen as far as i can tell. of course it still needs work (nonlinear access, backward prediction, etc).

where are you hoping to take this engine? will there be motion interpolators made? are the motion vectors compatibly with mvtools (so we can use it's interpolators on your more accurate vectors for standards-conversion)? i see a lot of potential here.

btw, what are your thoughts on constructing the compensated image? would mesh-warping or something similar be a good idea considering the motion search is so fast?

whatever happens, this is a sterling effort! thanks for the new tool :)

[Manao]

With the mvtools, you can emulate true motion filter by raising the lambda. It'll add a penalty to the SAD when the motion vector differs from the spatial predictor. Hence, the motion field is far more coherent.

mg262 : good filter. Nice to see somebody working on ME filters.

[Mug Funky]

quick hack for halfpel:

pointresize(last.width*2,last.height*2)
motion=findmotion(last,useassembly=true)
compensate(last,motion)
reduceby2()

[edit]

how are you thinking of handling occlusion and revealing of objects? i suppose that'll come with backward prediction. doing a subtract-from-source shows crossing objects are a problem area. i'm sure you already know all this stuff as you wrote the filter :)

[Manao]

That's not hpel. Hpel would be using bilinearresize as an upsampler. [edit : mm i'm wrong][edit2 : both of us are wrong, hpel can't be emulated that way]

Backward prediction isn't possible for true motion filters ( unless two passes are done, which goes against the principle of of true motion ( the principle being : fast ) ).

[AVIL]

@Mug Funky and Manao

To emulate true motion, perhaps is preferable tweak the parameter "Search" of the MVtools. Actually i use search=3 and searchparam = 3, but I remember have tried with search=1. The goal is limit the radius of search to the maximum expectable difference of position inter-frames of the objects in motion.

[AVIL]

@mg262

An important problem I have with mvtools is the blocks it generates when not appropiate predicted block is found. This block ruins denoising. I know its impossible predict a frame if in the reference frame they are new objects or part of objetcs not visible in the previous or next frames. But i think its possible minimize the problem if the predictor uses the reference block when the predicted block not matches well.

[Didée]

Very nice tool, Clouded. Very Nice. :) :)

(However Avisynth world is turning so fast these days, I can't follow anymore ... dizzy ...)


@ Mug Funky
if i force mvtools to be too accurate, it gives me an exact copy of the frame to be compensated Now, would you please tell how you do to achieve THAT (an *exact* copy) with MVTools ??


@ Manao
That's not hpel. Hpel would be using bilinearresize as an upsampler. [edit : mm i'm wrong][edit2 : both of us are wrong, hpel can't be emulated that way] Do you mean it's not hpel "by definition"? Because, by working on doubled resolution, obviously a ~sort of~ subpixel accuracy is achieved, isn't it.

Backward prediction isn't possible for true motion filters ( unless two passes are done, which goes against the principle of of true motion ( the principle being : fast ) ). For the time being, shouldn't

reverse()
MVstuff()
reverse()

work out ... it's halving the speed when forward & backward are done both, sure. But at least one can "have it" before Clouded implements it ;)


An important problem I have with mvtools is the blocks it generates when not appropiate predicted block is found. This block ruins denoising. That's an issue indeed. Using the builtin MVDenoise, at least you can tweak SAD and other thresholds to circumvent the problem - but it's only a compromise, and not even a good one : in areas with "strong" detail, often one would like blocks with a (relatively) high SAD to still be used. Whereas in "flat" areas, a rather small SAD can already mean visible blocking ...
Lowering the pixel threshold is even more poor, since lots of the potential of MV-compensation is thrown away ...
In custom-build denoising (using MVCompensate & Co., but not MVDenoise) it gets even more difficult, with SAD, vector lenghts etc. only available through dreadful backdoors.

That's why I thought out loud (http://forum.doom9.org/showthread.php?p=724570#post724570) about a "new" deblocking filter. The already existing deblocking filters are not suited to deblock compensated frames, if they are to be used for denoising: if the deblocking is cranked up enough to catch all blocking, all noise will be killed from the compensation, too ... dead end.
(The proposed method should work, I think ... but I can't dive into it, currently.)

[mg262]

@Manao, @Mug Funky, Thank you for the kind comments :).
i'm sure you already know all this stuff as you wrote the filter :o Afraid not... it's just an implementation of standard methods. I haven't even tested/examined the algorithm (as opposed to implementation correctness) to any great degree. So feel free to throw things at me ... where are you hoping to take this engine? There are four features I had in mind before I started coding:
Scene awareness (see below)
Sub-pixel accuracy
Feature-based motion estimation similar to this (http://www.ics.ele.tue.nl/~dehaan/pdf/97_VCIP04Braspenning.pdf)*
Mesh-warping similar to this (http://www.utdallas.edu/~aria/papers/csvt01.pdf)
*(A very low/nonexistent priority after seeing results from the basic version.)

Beyond these, I'm not sure. I'm certainly not going to fiddle with the core algorithm -- looking through the papers you can see the algorithm evolving and being extensively tested over a decade, with the result that the current version is very fine-tuned.will there be motion interpolators made? are the motion vectors compatibly with mvtools (so we can use it's interpolators on your more accurate vectors for standards-conversion)? i see a lot of potential here.Certainly possible (compatibility via a translation filter). The only thing I would say is that de Haan's website (papers Fizick linked to) has a huge amount of material on true-motion motion-compensated standards conversion and deinterlacing, and IMO we should assimilate that material before diving in -- scripts/methods appropriate for MVTools may turn out to be inappropriate for this filter.

Linear access, unidirectionality:
The algorithm was AFAICS designed for standalone hardware compatibility, so the limitations (linear access, unidirectionality) make good sense. As Manao says, there is no way of doing backwards prediction without at least a 2x slowdown. OTOH, with subpixel switched off, this may often be acceptable. I was thinking of working scene by scene, scanning forwards and backwards on each scene, to minimise space usage.

In any case, the filter will need to handle scene breaks. I strongly prefer scene detection to be done once and scene information to be passed to each filter -- for separation of functionality as much as efficiency. AVISynth has no native support for this, so I will pass scene information in a clip. Seeking to the 100th frame in a scene will then cost as much as seeking from frame 0 to frame 100 at present.* (Faster seeking is possible but ugly -- more on this later.)

*Each scene behaves like a separate clip passed through the filter. I sometimes consider an Animate-like meta-filter to do this to any filter.

Last: Development is likely to be sporadic as I'm beginning to get joint fatigue from interface issues... sorry to make you wait.

Clouded

Edit: AVIL, Didée, just seen your posts...

[krieger2005]

Is this not possible to use the results (output) of this filter as an input for MVTools (for Denoising) using the MVTools-Function "mvchangecompensate"?

But for this a motion-compensated-clip have to be done before by MVTools, or not? Just an idea...

[mg262]

Motion, 26 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_26Oct05.dll)

FindMotion(source, int initialise = ..., useassembly = ..., int reset = ... )

I added a quick feature to FindMotion to facilitate testing. There is an optional argument called reset; every reset frames, the filter calculates motion without looking at the motion computed for the previous frame. So (slow) seeking is possible.

Additionally, whenever a frame is calculated without motion vectors from the previous frame being available, the filter iterates the basic algorithm initialise times (default 0); if you don't set initialise, the motion vectors will reset to 0 every reset frames. Even a fairly small value of initialise should produce results that are very close to the non-reset version.

(Detail: if reset is 50, frames 1, 51, 101, etc are computed without looking at previous frames. So seeking to frame 175 will compute 25 frames. I could have set it to reinitialise for the exact frame seeked to, but this would be nondeterministic -- and very unfriendly to Didée's Reverse() script.)
________________________________________________________________

Didée: I'm glad you like it :). As for this:
An important problem I have with mvtools is the blocks it generates when not appropiate predicted block is found. This block ruins denoising.The true-motion algorithm has a tendency to try and use contiguous (touching) blocks from the previous frame,* which should reduce this effect -- so play with the filter and see what you think. It will also be interesting to try the mesh-warping compensation (don't expect it soon!) to see if that helps. But I could at some point give you a mask of the per-block SAD if that is any use?

*AVIL: I'm not sure what you mean by reference block, but the algorithm would generally choose something that is better than the corresponding block in the previous frame.

Clouded

Edit: What about a stand-alone mask-creating BlockSAD(clip, clip) filter?

[AVIL]

@mg262

I call the reference frame to the real frame. In motion compensation we are trying to simulate this reference frame by evolving blocks of another frame, previous, next, 2 past, etc.... In this context I will call this frame the "seed" frame.

Then, I call reference block to a block in the reference frame. When the block choose from the "seed" frame don't simulate well his correspondant reference block I suggested to use the reference block instead to build the mo-comped frame.

Sorry for my poor english.

[mg262]

AVIL, understood. What you want could be done by postprocessing using the kind of filter I suggested at the end of the last post.

@all, it's just struck me that I may need to be careful about patents? Does anyone know what the situation actually is?

[Manao]

You don't sell it --> you don't care about patents. It's at least true in some countries.

[MfA]

With the mvtools, you can emulate true motion filter by raising the lambda. It'll add a penalty to the SAD when the motion vector differs from the spatial predictor. Hence, the motion field is far more coherent.Predictive searches can get trapped in local minima though ... now for coding they get it right enough of the time for it not to matter much, but for image processing it makes sense to do a little more work and use feature based matching to get extra candidates.

As for patents, if you are really worried you can always only release in source code form ... Im not aware of anyone ever being sued for contributory infringement for releasing source code.

[Fizick]

@Mg262,
You are fast code writer! :)
Probably even faster then me. :)
I will try your filter today.

IMHO, BlockSAD function will not useful. I try similar way. The better SAD is not better block for true motion.
Occlusion info is more important.

@Manao,
By the way, recently (last week) I tried to improve MVTools in two directions:
1. True motion :)
Strictly as you say above, http://forum.doom9.org/showthread.php?p=728904#post728904
I added a penalty to the SAD when the motion vector differs from the spatial predictor.
my parameter draft name is "penalty" :)
2. Motion interpolation.
It uses some (clever ?) combination of both shifted and fetched blocks (forward and backward), and temporal average of some regions from two source frames to produce output.

The speed is not fast, of course.

I have some alpha version, but my work is still not finished. I want:
1. to limit of motion vector length (at every level probably ?)
2. to use most clever blocks combination for interpolation (i try many ways).

I wanted to publish my modified version next week (if my real life give some time), after some conversation with you.
But mg262 work provoke me to publish it more quickly. :)

Will it be politically correct? :) Or you work at MVTools too now?

[Manao]

Fizick : please, go ahead. I've got no time to put my hands on the MVTools in the near future, so you're really welcome to add features to it.

The penalty you're speaking about already exist : int cost = sad + MotionDistorsion(vx, vy);

inline int MotionDistorsion(int vx, int vy)
{
int dist = SquareDifferenceNorm(predictor, vx, vy);
return (nLambda * dist) >> 8;
}> to limit of motion vector length (at every level probably ?) /* computes search boundaries */
nDxMax = nPel * (pSrcFrame->GetPlane(YPLANE)->GetExtendedWidth() - x[0] - nBlkSize);
nDyMax = nPel * (pSrcFrame->GetPlane(YPLANE)->GetExtendedHeight() - y[0] - nBlkSize);
nDxMin = -nPel * x[0];
nDyMin = -nPel * y[0];

but for image processing it makes sense to do a little more work and use feature based matching to get extra candidates.Indeed. But the main predictor I use come from hierarchal me, which tends to be rather good.

[AVIL]

@mg262

I find useful your suggested filter in scope of postprocessing or even like a wide-usage filter. IMHO appart from the two clips must be a parameter for block size, more colourspaces, and could be another parameter for distinct block-difference calculations (sum(SQR(x**2 - y**2)), by example).

I`ve gived your filter a try. I've found it more blocky and less accurate than MVtools, but I think is because of the lack of hqpel. The blockiness is especially hard at the vertical borders of the image and frequent in the edges with hard luma stepping.

I`ve subtract the mo-compend frame with the reference frame. An ideal mo-comped frame only give, as result of subtraction, an arbitrary image with the inter-frame noise. All the similarities between the subtracted image and the real one are product of blocks not accurately calculated.

This filter is ver promisory.

@Fizick

A new filter (or a new version of and old one) is always a good new. Thanks

[mg262]

AVIL,

I looked at the issue; first I tried a number of minor variations on the method to see if they had any substantial effect, but the essential behaviour was unchanged. But supersampling (as Mug Funky does) certainly helps... so I think that you are right and subpixel estimation is particularly important for this algorithm, as it changes not only the accuracy but also the average step size (which affects motion vector convergence)... so when I can I will implement that and we can see how it looks.
__
Made a minor fix+change to improve behaviour of the new parameters. Please re-download Motion, 26 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_26Oct05.dll).

[Didée]

@Mg262,
You are fast code writer! :)
Note that in this very case it's not only true, but particularily impressive...


/* bows down deeply */


/* stumbles, and falls */ :D

[mg262]

Thanks, guys... but the beauty of AVISynth (as a package) is that it does all the slow to write, reference-requiring, technology-dependent parts like opening files and rendering, so that's where most of the credit should go!

I briefly tested sub pixel (1/4 pixel) accuracy in a rough form in C, and it helps a little, but the blocking and accuracy still much worse than that of MVTools. The last time I tried MVTools was at least 18 months ago (it's pretty rare that I work with non-animated material), and I hit substantial blocking... but now, at least on the source at hand, blocking seems very rare and its fast -- this runs at 50FPS:*

mvanalyse(source)
mvcompensate(source, last)

I was pretty disappointed with the comparative performance of this method -- the papers on it certainly give the impression that as well as being super-fast and producing true motion, it gave very good results with minimal blocking. It's possible that I made some silly mistake (and I'll check through), but the vector fields look pretty sensible. My particular interest was in trying to develop this for denoising scripts, and that seems relatively pointless now. Don't take that the wrong way -- I'm quite happy to try other things with it, it's just that denoising was the target I was heading towards. So, I'm not really sure where if anywhere to go from here... thoughts?

*Motion compensated scripts (e.g. MVBob) have the reputation of being really slow... why is this?

____________________
I ran across one simple deblocking method for truemotion motion compensation which I might as well repeat here:

Obtain motion vectors, then subdivide each block into four.
For each top-left subblock:

Replace the motion vector with the median of
-- original motion vector
-- vector from block to the left
-- vector from block above

Median is computed in x and y separately. (+Optionally, you discard the vector if it doesn't much one of the three above.) Other 3/4 sub blocks work in the obvious way.

AFAICS, the effect is to take a stair step and reduce to a stair step of half the size. The method is particularly intended not to introduce new motion vectors which didn't exist before.

[Fizick]

Main aim of true motion is motion interpolation.

[mg262]

Quick note: I took a guess at the meaning of a word ("meandering") apparently defined in some papers I can't get hold of... whether or not I guessed right, it seems to help under some circumstances. So...

Motion, 28 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_28Oct05.dll)

If you really want to try quarter-pixel accuracy the slow C way, add subpixel = true and useassembly = false. It only runs at 30 FPS or so on 720 x 576, so for testing purposes I would recommend cropping down your clip until you can watch it in real time. (I probably won't have time to assemblify it for a while.)

[mg262]

Motion, 29 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_29Oct05.dll)
Bug fix for meandering and slightly faster assembly integer search code.

I've been browsing through papers and I think that at least half-pixel accuracy is needed to do anything useful (because of the half-pixel shift between fields). Quarter-pixel accuracy is preferable -- although for the assembly, I can't at present see a way to deal with quarter-pixel (without sacrificing speed) other than by coding 16 cases... ... ... . (Joy.) So I will probably fight towards that and then think about applications.

@Mug Funky, if you're reading this, anything particular you would like to see this filter do?

how are you thinking of handling occlusion and revealing of objects? i suppose that'll come with backward prediction. doing a subtract-from-source shows crossing objects are a problem area.I have read more about the problem domain now (there are downsides to diving in headfirst ;) ), so I'll try and answer this. First, it's not going to be dealt with at the motion detection stage, simply because there's no way of representing the information without complicating the datastorage format substantially. I.e. an object that has become revealed can't be sensibly represented with any motion vector. Second, depending on the application, we can try and infer this information from the motion vector field; for example we can look for discontinuities, and fallback to some simpler method (for whatever the application is).

Incidentally, I am not sure how well this kind of method will work on foreground objects in animation; the algorithm is not terribly likely to deal well with the fact that their motion is typically at a lower FPS than the motion of the main clip...

[psme]

Tried both (P4 2.66 overclocked to 3.8G):

motion = findmotion(source, useassembly = true )
compensate(source, motion)
(around 20% CPU usage)

and

mvanalyse(source)
mvcompensate(source, last)
(around 55% CPU usage)

in FFDShow->Avisynth for real time DVD playback. Both methods give smooth playback. But both methods give tons of pixelization/blocking artifact in motion area.

I guess the filter is not yet usable...

regards,

Li On

[mg262]

@psme,

Motion compensation filters aren't as far as I know intended for direct use in playback... they are intended to provide high-quality input for temporal processes, such as temporal denoising, frame rate conversion (as Fizick mentions) or deinterlacing.


@all,

Edit: scratch what was here... I have thought of a modification that may produce the backwards motion information without requiring backward tracking.

[castellandw]

I mentioned in some other post about motion compensation using phase correlation being used by video engineers, especially for broadcast standards conversion (PAL to NTSC, NTSC to PAL, etc). Mug Funky mentioned that this could be accomplished by block overlapped motion compensation. Of course, MVTools can be used for motion compensation, but Fizick recently created a Block Overlap Plugin along with a sample script to use with MVTools to create block overlap motion compensation (I tested out Fizick's block overlap plugin with MVTools and it did produce better results). Mug Funky also mentioned that phase correllation could be achieved with the DePan plugin, but also some other tricks need to be involved. Mug Funky, do you know what other tricks need to be involved for AviSynth since it's possible to achieve block overlapped motion compensation in Avisynth (Block Overlap Plugin By Fizick with MVTools) and with phase correlation (possibly Depan plugin)?

@mg262, that message thread where my link to the pdf on Snell Converter's guide to motion compensation was posted, I posted the link to that message thread on another site with big expert video engineers, and one video engineer told me that motion compensation was revolutionary in 1994 and no other technique was surpassed it. So what I'm saying is that those papers posted on that message thread, especially papers by De Haan or true motion estimation using feature correspondences don't compare to that motion compensation technique with phase correlation according to that guy.

@all, any comments on what I mentioned about motion compensation?

[mg262]

That sounds very interesting... although I'm not sure how to read this:
one video engineer told me that motion compensation was revolutionary in 1994 and no other technique was surpassed it.... most of the stuff I'm looking at is post-94 -- although maybe that's not what you mean?

I would really like to see the site you refer to if you happen to have the link at hand...

[Joel Cairo]

mg262:

Unless I'm mistaken, he's referring to the Doctor Who Restoration Team website, which is staffed by the members of that team-- some of whom are BBC engineers. You can find them at:

http://www.rtforum.co.uk/

THe thread is currently near the very bottom of their page, and it's titled "Snell Converter's Motion estimation methods said to be dated". You'll have to check it quickly, though, as topics drop off their board after a couple of weeks, and I don't believe there's an archive.

As I read the discussion though, I actually felt that our discussion was somewhat mis-represented. I don't recall anyone here saying that the Snell converter's inner workings were dated...

Anyway-- that's where you can find the discussion.

-Kevin

[mg262]

Thank you for the link:

http://www.rtforum.co.uk/read.php?id=128649
Charlie S
Snell Converter's motion estimation methods said to be dated

Guys, I'm at my wits end. Steve Bagley, if you're reading this, I posted the link to that paper on motion compensation from Snell Converters that you gave me and posted it on another website forum. The reaction I got says that the paper being from 1994 is very sad being dated because of still using blending techniques, and I think I'll post this link:

http://forum.doom9.org/showthread.php?t=100791
I find this a little confusing... it reads as if I posted it (which I didn't). Perhaps Charlie S posted a similar query on a different forum but linked here instead?

[Joel Cairo]

Well, I think this is where the misunderstanding began. There's a post just below yours in that thread (by Mfa), which makes the ironic point that the document you linked was written over 10 years ago, but that people still weren't employing the techniques that are outlined in that document. Instead, they're still using blending, etc.

The way Mfa's message was written, however, it evidently was mis-interpreted by the poster as meaning that blending, etc. was part of the method outlined in that document-- which is not correct, if I'm understanding all this properly.

So I think that's how all of this confusion got started.

-Kevin

[Fizick]

Discussion is interesting, there are many ways.
Very interesting, for example, recent bi-directionanal motion estimation and interpolation.

[castellandw]

Guys, it's a bit hard to follow you guys from one forum to another(and a bit of a headache) because the message I posted in the Doctor Who restoration team forum has been a few weeks old, and so I'm unaware where you guys are posting what unless you tell me. Yes, I posted the message in the Doctor Who restoration forum to get a second eye because I knew I could have misunderstood this message thread:
http://forum.doom9.org/showthread.php?t=100791

@Joel Cairo aka Kevin, I would have gathered that I misunderstood, especially when James Insell at the Doctor Who restoration team forum told me that no blending is being used for motion compensation. When I posted it, I assumed that when Mfa said stuff like blending is still being used, I thought he was referring to the Snell converter paper which I seem to be the first to post on the Doom9 forum:
http://forum.doom9.org/showthread.php?p=718712#post718712 (http://forum.doom9.org/showthread.php?p=718712#post718712)
Also, I was referring to Snell converter's methods being possibly dated compared to other papers posted as well which are papers by De Haan and true motion estimation using feature correspondences. Thus from there, James Insell told me that no other method has surpassed motion compensation. I didn't try to misrepresent the discussion here because I posted the link to that thread to get a second eye on the subject.(By the way, Kevin, since you mentioned discussion on the Doom9 forum being misrepresented, was there any other discussion on Snell converters apart from that link to that pdf file being discussed because I never managed to really find any?)
One more thing, I didn't post the Doctor Who restoration team forum link http://www.rtforum.co.uk here because when I asked about what they thought of PAL to NTSC conversion done on AviSynth, they said I should go to Video Editing forums, and so I had to limit my posts there with simpler explanations of techniques as best as I could when I referred to Avisynth software. So I knew it wouldn't be worth it to acknowledge the Doctor Who restoration team link if they might likely say to you guys elsewhere to go to other forums if you mention video editing software.

@mg262, I did post that Snell Converter paper first on the Doom9 forum apparently, and I assumed that you found that first from my post. I had no idea you found the paper first on Google because I didn't realize it was easily findable on Google. You're right, it does seem as if you posted as Charlie S, sorry about that, but I thought the reaction from your post was because of me posting the link to that Snell Converter paper first in the forum. I didn't post a link to my original post on Doom9 with the Snell Converter pdf link on the Doctor Who restoration team because the reaction to your post was more relevant. By the way, Mohan, you better post what you put on the Doctor Who restoration team over here as well because it'll make things very confusing for other posters looking too and as Kevin said, messages on the Doctor Who restoration team forum drop off..
Here's again the link where I originally posted the Snell Converter paper:

http://forum.doom9.org/showthread.php?p=718712#post718712 (http://forum.doom9.org/showthread.php?p=718712#post718712)

I see Kevin, has found this link from the Doctor Who restoration team:
http://www.rtforum.co.uk/read.php?id=128649
but did you guys manage to find another one I posted which has now dropped off the Doctor Who restoration team forum. Well, here's the link again on my original post involving the link to that pdf on Snell Converters involving motion compensation as well as discussion by Mug Funky on how it could be possible to do motion compensation using phase correllation:
http://forum.doom9.org/showthread.php?p=719376#post719376



Here's Fizick's post on the block overlap plugin I mentioned earlier that can be used with MVTools:
http://forum.doom9.org/showthread.php?t=101307

[castellandw]

Oh yeah, a British sci-fi show about an alien that travels through time and space in a machine that's bigger than the inside than it is on the outside is stereotypically British. I can see that coming, Mfa.

Anyway, as I mentioned in the other post I just posted, MfA, I wasn't referring only to your comments because there was mention of papers by de Haan being mentioned on that Snell Converters paper, and your comments about blending can be understood in more than one way.

[MfA]

I'd appreciate if you never speak for me again ... I don't appreciate having to go into a strange forum to clear up a gross misrepresentation of what I said only to get deleted. It sets bad blood all around, you are pissed because you are told off. They are pissed because they think I slagged Snell Wilcox and because they think the thread went way too far off topic, and my double posting probably didnt help either, and this will reflect in their opinion on Doom9 and everyone involed. Lastly I am pissed, because it is very hard to reply in any way which wont turn this into a flamewar which will probably result in me getting a warning.

[castellandw]

Mfa, they didn't delete the message thread because of your post or being pissed at you. If you noticed, that message thread was the oldest one on the board, and so it just dropped off. Nothing to do at all with any slagging to Snell-Wilcox or the thread going way off-topic. As long as everybody was discussing technical video bits on the forum, it wasn't going off-topic and I gave the link to that thread so there is no way they'd think you're just slagging off Snell-Wilcox or actually care judging by how much attention was given to that message thread. Hey, lots of people double-post on that forum, so you're not the only one. By the way, I was not speaking for you, I was trying to get a second eye from those guys at the Doctor Who restoration team forum because they're the best video engineers, especially from their restoration work they've done, I could find to give a second opinion on all the replys to that message thread on motion compensation in general. It was my misunderstanding on your post, but they are not pissed off at you or anyone at Doom9 forum. Believe me. Besides, when I mentioned Avisynth, they practically asked me to ask about AviSynth in other video editing forums until I tried to limit my posts to just regular video techniques in general. Look, as I said, you did nothing whatsoever to piss them off, and the message thread outlasted its own timespan on that forum.

[Mug Funky]

@Mug Funky, if you're reading this, anything particular you would like to see this filter do?

i think that's already been covered - mesh warping would be very cool, and it seems it's been on your list from the start. i'm really just happy to see stuff being developed so actively for avisynth (particularly by you - i think you're a contender for "most new plugins written per month". i'm in awe).

actually, a "delta" option similar to that of mvtools would be great for denoising.

i'm sure the algos you've implemented cover this to some degree, but if frame interpolation is involved, things like object acceleration and whatnot will have to be covered to some extent. i guess that would mean sampling at least 3 frames to derive the velocity/acceleration of an object anywhere in the middle. this is probably already done if i'm understanding correctly.

as far as the frame-rate issue with foreground/background in animation, don't be worried about that. not even the PhC stuff handles that properly (the paper itself says "garbage in, garbage out" when talking about something as dead-simple as telecined material... that could easily be handled in hardware by a separate hard match IVTC running before the PhC part, and i'm sure modern machines do just that). certainly i've seen plenty of stuff on TV that's gone through some kind of mocomped conversion, and it's clearly failed badly. "iron chef" is an example. fuzzy subtitles, and an ex-composite source (dot crawls) make it a difficult source. typically a medium-fast pan is handled perfectly, but any faster and it's blended. subtitles + mesh-warping aren't a good mix either. i see no shame in falling back on field-blending in lots of cases. for small areas of a frame, it's much more transparent than warpage or other artefacts.

@ castellandw:

about using depan... i said that partly in jest. though it could be done, there are severe limitations. you'd have to chop the clip into large blocks (depan will only find motion that fits within the frame), then do overlapped compensation, then split into smaller blocks, run again, then split into smaller blocks.... it'd be slow and poor quality (the overlapped blocks will not match in most cases), but would work as a proof of concept for the phase-corellation method. there's definitely many more ways to skin that cat though :). i certainly wouldn't want to script a frankenstein's monster script like that though.

[castellandw]

@Mug Funky, that sounds like it could use a recursive hashing algorithm although if it's poor quality particularly with overlapped blocks not matching, then I begin to wonder how video engineers can produce results with motion compensation using phase correllation. So every time it separates into smaller blocks, phase correlation can be performed before overlapped-block motion compensation. So, what exactly what is at least one example of a way that it can can be done to "skin that cat" of dealing with overlapping blocks not matching because I'm wondering how it's possible that video engineers produce decent results with standards converters?

By the way, Fizick and anyone involved in development with MVTools should look at this discussion here on motion compensation using phase correllation because it might prove extremely helpful.

[Mug Funky]

well, the only reason the way i described would be bad is because in depan only half the job is done - it makes no attempt to find motion vectors, rather it performs phase corellation on a whole frame to find global motion.

in a PhC unit (from reading the paper), there is a phase corellation stage that finds several (i guess) local maxima that correspond to motion-vectors, and then there's a stage where it matches the motion vectors to actual locations on the screen. this makes it far far far superior to just matching global motion on ever smaller blocks.

[mg262]

@castellandw,

Now I follow. (That paper is the first hit when Googling "motion compensation".) What I posted on that forum amounts to little more than Fizick said above; it was mainly to elicit a more detailed response from James Insell. Now the thread is gone, let's leave things be.

@Mug Funky,

actually, a "delta" option similar to that of mvtools would be great for denoising.It's trickier in this instance because each motion estimate depends on the motion estimate for the previous frame... I could create something that mimicked SelectEven.FindMotion, but I think the found motion vector quality would degrade. On the other hand, this is perfectly legit:

#source = ...
motion = findmotion(source, useassembly = true )
source
compensate(motion)
compensate(motion)

I.e., get the motion vectors once, move blocks to compensate original frame n-1 -> n, then move blocks to compensate compensated frame n-1 -> n. So we have:

original frame n-2 -> compensated frame n-1 -> doubly-compensated frame n

Will that do? I can definitely implement something else, given specifics.

By the way, using that ^^ script will make the blocking/inaccuracy worse for obvious reasons. There are a range of things which can be done to improve the motion vector quality, but I think I'm going to try and get an application out first... it makes it easier for us to check whether a particular change is actually helping. For similar reasons, I would like to have better quality motion vectors before trying mesh warping.

From Phase Correlated Motion Estimation (http://www-ise.stanford.edu/class/ee392j/projects/projects/liang_report.pdf):
Phase-correlation ME is very computationally efficient and it produces much smoother motion field with low entropy than the BM method does. Phase-correlation works better than the BM does in the cases of large scale translational motion, while BM is more suitable for predicting regular and small scale motion and multiple-object movement.

(Bear in mind that that compares phase correlation against full/logarithmic search block matching... actually I think it's some student report, so doesn't carry that much weight in itself -- but the conclusion looks very sensible to me.)

[castellandw]

So basically for phase correlation motion estimation, it's a mix of phase correllation first (for large scale motion) and then overlapped block motion compensation (for small scale motion and multiple object movements)?
(I say overlapped block motion compensation instead of block-matching motion compensation because overlapped block MC is said to be more optimal than block-matching MC:http://forum.doom9.org/showthread.php?p=719376#post719376 )

[Mug Funky]

it's a little simpler, but basically what you said.

instead of taking a block and finding a vector, phase corellation finds a load of vectors and passes them to some other algo which finds the block for that vector. it essentially works backwards. requires more stages and is conceptually more complicated than block matching, but in the end there's a fair bit less work for the computer to do.

i might have thrown you off a tad with the depan thing...

[mg262]

Minor point, but could cause confusion along the line... I want some standard terminology. Ideas on sensible names for

a) the frame we are trying to approximate

and

b) the frame from which we are extracting blocks to be used to approximate a)

, please?

I particularly ask because I've seen the word "reference" being used for both a) and b)...

[Mug Funky]

what about "source frame" and "target frame", but then these could get confused too i suppose. i'm thinking of source frame as being the one blocks are moved from, and target frame is the frame blocks are being moved to match.

for forward compensation, think of source frame as n-1 and target frame as n.

[MfA]

I use the terms forward reference and backward reference (although I use the terms exactly in reverse as they are usually used in video coding, where they use the direction of prediction rather than the direction in time to determine what's forward/backward).

[mg262]

I'm trying to forwards/backwards/previous/next as much as possible, partly because of this:
I use the terms exactly in reverse as they are usually used in video coding, where they use the direction of prediction rather than the direction in time to determine what's forward/backward
Both conventions make sense in their own way, but having both around at once can make it very confusing for the reader! Plus, much of the time you want to describe the compensation process itself in a way that is direction-independent...

Source and target are much better, but still a bit prone to confusion. I think I might go with calling a) "current"... that kind of makes sense for both directions.

[castellandw]

Minor point, but could cause confusion along the line... I want some standard terminology. Ideas on sensible names for

a) the frame we are trying to approximate

mg262, don't you think we should also use separate terminology for block portions of a frame as well because a load of vectors are being used from a block portion of a frame to approximate the (as Mug Funky put it, possibly "local maxima") vector for that block portion of the frame?

In any case, I can see that motion estimation using phase correlation uses various things such as the fourier transform to output the motion vectors used for motion compensation, but do you guys think this can be really implemented for Avisynth or is motion estimation using phase correllation and then motion compensation look like it's not gonna happen because it would be great to see how it works out on broadcast standards conversion?

[mg262]

mg262, don't you think we should also use separate terminology for block portions of a frame as well because a load of vectors are being used from a block portion of a frame to approximate the (as Mug Funky put it, possibly "local maxima") vector for that block portion of the frame?I'm sorry, I don't understand this comment.

Motion estimation using phase correlation... I don't see why wouldn't be possible in AVISynth; it just requires someone to want to code it.

[castellandw]

Quote:
"mg262, don't you think we should also use separate terminology for block portions of a frame as well because a load of vectors are being used from a block portion of a frame to approximate the (as Mug Funky put it, possibly "local maxima") vector for that block portion of the frame?"

I'm sorry, I don't understand this comment.


Well, you asked that you wanted standard terminology for "the frame we are trying to approximate", but since each motion vector approximated from phase correlation are derived from each portion (a.k.a. block) of the frame, do we need separate terminology (besides "source frame", "target frame" and "current frame") for each of those blocks of the frame to avoid even more confusion (Basically, to distinguish between discussing a whole frame and a block or portion of the frame)?

[mg262]

Unlike the backward/forwards related terminology, I can't at present see a situation in which that kind of confusion could arise*... but at the moment I'm not using phase plane correlation, so it's rather academic.

*although I'm open to be proved wrong ... you could suggest an example of a sentence which could be misconstrued with two sensible readings.

_______________________________________

Maybe this is obvious in the Development Forum, or maybe I should have said it before, but anyway: like most other temporal filters which don't explicitly state otherwise, this filter currently takes no account of interlacing. If you are using interlaced content, either deinterlace or bob first. (I do actually plan to make it treat interlaced content sensibly... but there are a couple of prerequisites.) Simply separating fields is a bad idea because the up-down jitter will mess with the true-motion algorithm.

[scharfis_brain]

maybe a special interlaced mode could be implemented, that takes care for the bobbing?

So it isn't fooled by static areas (which obviously are bobbing) AND creates non bobbing full sized (frame heigth, not field heigth) vector frames.

This would be a GREAT help for mocomped deinterlacing.

[mg262]

scharfis,

I'm being a bit dense atm and I'm not sure how to read and that -- do you mean that you want the filter to take bobbed input but try and be intelligent about using original scanlines rather than interpolated lines? Or do you mean that you want the filter to be/include a bobber?

[scharfis_brain]

sorry, I wrote it a bit mixed up.

I mean this:

source("blah.xxx")
assume?ff()
vectors=separatefields().analyse(interlaced=true)
compensated=somesmartbob().mvcompensate(vectors)

this means: take a fieldseparated video and do the analyse on it.
of course this analyse must take care for static areas and has to ensure that the vector movement is smooth (no bob-jitter!)
also the vector field has to have the size of the full frame (like the smart bobbed one)

[Fizick]

I do not know, what scharfis_brain told about,
but I consider (in some future) fieldbased input, half-pel interpolation,
and one pixel shift field compensation (I have it in DePan).

[scharfis_brain]

but I consider (in some future) fieldbased input, half-pel interpolation,
and one pixel shift field compensation (I have it in DePan).


but this will create wobbly vectors in static areas

[Fizick]

Half-pel assume bilinear interpolation.

Another approach (I see it in some deHaan article?) :
Withot halft-pel, simply to add penalty to SAD cost for candidate vectors with odd vertial shift.
Take full interlaced frames. (It is used for fast realtime).

[mg262]

scharfis,

what you ask is definitely possible. I was in any case thinking of having motion vectors for interlaced clips actually use distances measured in bobbed space;* it removes a lot of headache, and makes the algorithm trivially resistant to jitter. So would just be a matter of moving 8x16 blocks instead of 8x8 blocks, or equivalently duplicating each row of motion information. You can almost do that right now by writing:

Interleave(motioninformation, motioninformation).Weave()

(The only reason that wouldn't work is because motion information is currently padded to make sure it contains an even number of scanlines -- but I was planning to move it from YV12 luma to YUY2 or forthcoming Y8 anyway, which would obviate the need for padding.)

*I hope that comment is clear... poke me if it isn't. I tried to write a bit more about it but it looked like I was just making it more confusing. Key point is that still scenes will have stored motion vectors of (0,0), not (0,±0.5).

Also bear in mind that there are proper motion compensated bobbing algorithms... cf one of the papers Fizick pointed me to: http://www.ics.ele.tue.nl/~dehaan/pdf/111_IVCP_ZHAO

Edit: I have yet to find any explicit discussion of interlaced->interlaced FPS conversion. AFAICS most papers implicitly expect you to use a MC bob + a progressive MC frame rate change (both discussed extensively in the literature), then discard unneeded fields. Quite apart from the usage of interpolated values for further interpolation, I think that this misses the point completely -- if you are applying half pixel compensation, direct interlaced->interlaced FPS conversion is almost the same problem as progressive->progressive conversion, and the methods extend directly. I will expand on that later.

[That's not to imply that bob + frame rate change + discard is a bad idea in scripts, given the currently available tools.]

[scharfis_brain]

IMO one cannot go another way than

mvbobbing -> mvfpsconversion -> resize -> reinterlace

working directly with the fields won't work. You need to make it progressive.

[mg262]

To keep the argument as simple as possible, let me first ask you this: if resizing were not an issue, would you consider a one step process to be feasible?

[castellandw]

Oh yeah, the Doctor Who restoration team video engineers said the motion compensation with phase correlation was done on a full frame meaning all the 50-60 fps full progressive frames were deinterlaced from the 25-30 fps interlaced frames. So you have to make the frames progressive for motion compensation. In terms of frame rate conversion, there should be a motion compensated fps function (like MVFps in MVTools) required for anything involving motion compensation because the frame rate change occurs during the motion compensation process. To do motion compensation first with something like MVTools and then use ConvertFPS or ChangeFPS won't work properly. On the subject of bobbing,

@scharfis, on the subject of bobbing for interlaced frames involving motion compensation, you could look at section 2.2-2.3 of Snell's Guide To Motion Compensation on Pre-Processing and Motion Estimation:
http://www.snellwilcox.com/knowledgecenter/books/books/emotion.pdf
Although I'm not sure how helpful, but you should try out that Block Overlap plugin by Fizick in your mvbob function, and let's hope someone can achieve coding phase correlation.

[Joel Cairo]

castellandw:

With regard to the Doctor Who restorations, there should be no need to for the Restoration Team to recover frames from interlaced content... the episodes that they treat with motion conversion are derived from 25 fps film telerecordings, so it's my understanding that they just have special video transfers made, clean up the image, and work directly from those.

-Kevin

[castellandw]

Kevin, what are you talking about? I'm talking about regular broadcast standards conversion process (aka PAL-to-NTSC or NTSC-to-PAL) in general with the Snell Converter that the Restoration Team uses. This is nothing to do with recovering the original interlaced frames from film telerecordings like the actual VIDFIRE process.

[Joel Cairo]

Ahh-- my mistake... I lost track of the conversation.

Sorry for the confusion! :)

-Kevin

[mg262]

Oh yeah, the Doctor Who restoration team video engineers said the motion compensation with phase correlation was done on a full frame meaning all the 50-60 fps full progressive frames were deinterlaced from the 25-30 fps interlaced frames. Thank you, that is worth knowing...
So you have to make the frames progressive for motion compensation.... but that doesn't follow from it. It just means that they/the equipment they had did it that way.

[castellandw]

Well, I hope I'm explaining this right. They did mention interpolating to full frame if I remember correctly (God, I wish they didn't delete their old message threads.), but you can ask them too.

[mg262]

castellandw,

Don't worry, that's clear enough. As I said, I had the impression that that was the standard method. I just want to suggest a different way of doing it...

[castellandw]

scharfis_brain mentioned about bobbing before motion compensation took place, and it got me wondering, shouldn't we use better bobbers than just smart bob such as leakkernelbob?

[Mug Funky]

it's a chicken and egg thing.

to do a motion-compensated deinterlace (good bob) we need to start with a good bob.

[scharfis_brain]

That's why I cannot wait tritical releasing his new Tdeint2!

[mg262]

Mug Funky, scharfis,

:confused: Why do you need to start with a normal bob?

The only thing that comes to my mind is the case where you have vertical motion of 1 (or 3 or 5, etc) original scan lines per field, i.e. the case where field 0 and field 1 match pixel for pixel. In this case, you either have to look further afield temporally (which may not help and increases the risk from incorrect motion prediction), or (safer) use pure spatial upconversion.

Cf again the paper Fizick pointed us to:
http://www.ics.ele.tue.nl/~dehaan/pdf/111_IVCP_ZHAO

There is a motion-compensated deinterlace which starts ab initio.

[castellandw]

@msg262, has anyone actually implemented the GST algorithm as a plugin for AVIsynth or at least tested out the GST algorithm to see if it actually has good results? Also, do you think I should go to the Doctor Who restoration team to see if they know what kind of deinterlace technique is used before motion compensation starts in the Snell converters they use?

[MfA]

if you are applying half pixel compensation, direct interlaced->interlaced FPS conversion is almost the same problem as progressive->progressive conversion, and the methods extend directly.Performing interpolation for inherently aliased fields works even worse than for progressive frames though.

[mg262]

do you think I should go to the Doctor Who restoration team to see if they know what kind of deinterlace technique is used before motion compensation starts in the Snell converters they use?If you are curious about it, go ahead... but it's not necessary for this thread. The strength of AVISynth is that tools can be mixed and matched -- you are free to try any deinterlacer with this or any any motion compensation. If you find convincing evidence that a particular deinterlacer is very effective (for a particular task), you can present it clearly in a new thread and hope someone is interested enough to build one.

It is certainly worthwhile to know what methods are used in hardware -- if the explanation is sufficiently detailed to be replicable. But I think you should not assume that the methods used in hardware, even high-end hardware, are the "best" methods around. (The same caveat should be placed on academic material.) In the end, to choose an approach for a given task, you need to immerse yourself in the material until you understand the strengths and weaknesses of each method well enough to pick -- or construct -- the appropriate one for the task.

@msg262, has anyone actually implemented the GST algorithm as a plugin for AVIsynth or at least tested out the GST algorithm to see if it actually has good results?I don't think so, but I couldn't tell you for sure... you could :search:

On this, on phase correlation, things are only implemented when someone has an interest in implementing them -- which typically means an interest in the specific tasks for which they are useful. I'm going to steal some well chosen words:6. Some time ago i also waited when somebody will wrote filter with advanced algo.
... take C compiler, and become the talented progammer !

[mg262]

MfA, inherently aliased fields: do you mean aliased in the sense that they are aliases of each other or aliased in the sense of anti-aliasing? (I think you mean the former, but if it's the latter I'll have to think some more...)

Update on progress: I put in the half-pixel assembly, and then spent a fair while profiling, optimising, trying a lot of different things to squeeze out more speed. I will probably release a new version after putting in FindReverseMotion and ReverseCompensate functions. After that, real life will soak up most of my time, but I'm thinking of implementing one of these two (in C to start with):
-- simple (bilinear) mesh warping
-- simple (no clever occlusion handling) FPS conversion.

Simple because I like to have a baseline to test more advanced features against. Preferences?

[MfA]

The frequency of the underlying signal is unchanged compared to progressive, but the vertical sampling frequency is halfed ... so more aliasing.

[castellandw]

On this, on phase correlation, things are only implemented when someone has an interest in implementing them -- which typically means an interest in the specific tasks for which they are useful.

@msg262, so does this mean phase correlation really isn't important toward making a better motion compensation engine for right now? I'm not trying to go away from the fact that I don't want to code a plugin that does phase correlation because I'm a lesser-experienced programmer, but for now, you guys pretty much seem to have more experience in this and are much more advanced programmers in this than me. Of course, the reason I wanted to ask about what deinterlacer that the Snell converter uses for motion compensation was because I thought which deinterlacer to use did matter when motion compensation is applied to the frame.

[mg262]

Got it. I would certainly agree that

50p --MC-FPS-> 60p --discard fields-> 30i

Beats

50p --discard fields-> 25i --MC-FPS-> 30i

But our case is slightly different. Our underlying material is 25i , and the only way we can get 50p back out of it is by bobbing, i.e. by interpolating values. Now IMO the MC-FPS unit is just as capable of interpolating values as a bobber. In fact, I would argue that it is better placed because:

a) Suppose we are interpolating midway between fields B and C in this configuration:
A B C D
Now a bobber will use parts of C to fill in bits of B, and then the FPS unit will use those filled in bits to create the intermediate frame -- but really this is giving us no more information than it already gets from considering C directly.

b) The bobber will also use A to fill in B -- but the FPS unit can do this if it chooses to, simply by expanding the temporal radius. You could argue that this introduces extra inaccuracy (pulling information across 1.5 fields distance rather than 1 field distance), but
i) the information is being pulled across 1.5 fields anyway; it's just happening in two hops
ii) the bobber itself is always pulling information across 1 field's distance, whereas the MC-FPS unit can get away with looking across 0.5 fields distance.

Also note that at least some motion compensated deinterlacing methods, like the GST, can be directly applied to FPS change -- we obtain motion compensated samples from previous and next fields, apply the generalised sampling theorem to reconstruct a curve, and read the values we want off the curve. (I think there are better methods available, which allow things like e.g. lower weighting of information pulled across a larger temporal distance... but that's another matter.)

[mg262]

castellandw,
does this mean phase correlation really isn't important toward making a better motion compensation engineI honestly don't know. I'm not confident enough to call this one. If you want my gut feeling... I have a tendency to distrust maths-based methods unless I can see (theoretically or empirically) that they model the problem domain well. (IMO, there is a visible tendency for academics who are not mathematicians per se to be impressed by maths just because it looks hard.) A priori, you can also see that phase correlation will tend to mess up on small objects.

The only way to really to which method is best is to implement both methods and refine them with all the tweaks everyone has thought of and then compare them on real applications like FPS conversion. So, having implemented one method, we need to refine it and include applications before any comparison is even possible.

[scharfis_brain]

60p --MC-FPS-> 60p --discard fields-> 30i

is it meant this way or is it just a typo?

Did you mean:
50p --MC-FPS-> 60p --discard fields-> 30i

or

60p --MC-FPS-> 50p --discard fields-> 50i

[mg262]

Oops... completely right, scharfis. :o I fixed it. Thank you.

[scharfis_brain]

btw.: for standards conversion of interlaced SDTV (50i and 60i) there needs to be a deinterlacer applied.

mainly for those reasons:
1) scaling the fields between 480 and 576 lines while preserving static areas
2) doing a good motion vector search
3) getting material to replace MoComp mismatches
4) make your live easier ;)

[castellandw]

castellandw,
I honestly don't know. I'm not confident enough to call this one. If you want my gut feeling... I have a tendency to distrust maths-based methods unless I can see (theoretically or empirically) that they model the problem domain well. (IMO, there is a visible tendency for academics who are not mathematicians per se to be impressed by maths just because it looks hard.) A priori, you can also see that phase correlation will tend to mess up on small objects.

Well, it seems clear that it'll mess up on small objects, which as I understand is why phase correlation is only limited to large object movements on regular standards converters. You know, I like to give Fizick's MVFlow function a try since it works on pixels of a frame instead of blocks of a frame, but Fizick says that doing motion compensation by pixels can create very strange deformed pictures, so for now, I'll stick to anything involving blocks of frames instead of pixels.

[Mug Funky]

hmm. motion-compensating a new field for deinterlacing doesn't _necessarily_ need a bob to start with - whatever is performing the motion compensation simply needs to be aware of the half-pixel shift between fields and take that into account. MVtools doesn't do this (why should it... it's not a deinterlacer after all), so we have to feed it with a smart bobbed clip where the half-pixel shifts are minimised as much as possible. and as scharfi says, it makes things easier because you can just re-use the smartbobbed clip to fill areas where motion compensation has produced artefacts.

[edit]

@ castellandw:

mvflow still works with blocks, but it moves each pixel when compensating. it just interpolates the vector field made from the block-matching process. this can really help with angular motion and even rotation though, so it's still got an edge over moving full blocks. though for interpolation, i think forward+backward with OBMC and a blocksize of 4 would be nearly perfect as well.

[scharfis_brain]

when compensating fields directly one doesn't achieve a goodsuppirxel accuracy.

When I use an ELA based smart bob instead, I have much better subpixel precision due to the ELA interpolation. (lesser jaggieness)

[mg262]

1) scaling the fields between 480 and 576 lines while preserving static areas
2) doing a good motion vector search
3) getting material to replace MoComp mismatches
4) make your live easier2. and 3. I can certainly accept. I don't understand the "vectors=separatefields().analyse(interlaced=true)/compensated=somesmartbob().mvcompensate(vectors)" request in the context of 2, though... I would have thought you would want to do it the other way round, i.e. find the motion vectors on the bobbed clip but compensate the un-bobbed clip.

As for 1.: Once the motion compensation is in play, you simply have a vertical slice with (approx) known values at irregular intervals, and you want the values at particular points... AFAICS, the number and spacing of those points doesn't matter... so I think you can do the resampling at the same time as interpolation. (And use your favourite resampling method for the interpolation.)

4. Yours or mine :p?

Edit: I don't know how much you will be able to rely on the subpixel accuracy of the true motion algorithm, even on progressive content, anyway... if it looks like being a problem, do tell me because there are quality/speed trade-offs that can be made.

[mg262]

Motion, 3 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_03Nov05.dll)

Most of the changes are under the hood to speed things up. Half-pixel accuracy is now in assembly, and works about as fast as the integer version did before (~ 200 FPS). Half-pixel and assembly are now used by default. Bidirectional estimation/compensation is available like this:

FindMotion(from = previous) or FindMotion(from = next)
Compensate(source = previous) or Compensate(source = next)

Both default to previous. So standard compensation (moving parts of frame n-1 to replicate frame n) is still done like this:#AVISource...
Compensate(FindMotion())And reverse compensation (moving parts of frame n+1 to replicate frame n) is done like this:#AVISource...
Compensate(FindMotion(from = next), source = next)(If the argument names are confusing, suggestions for better alternatives would be welcome.)

Here is an example interleave script for motion compensated denoising:#AVISource...
Interleave(\
Compensate(FindMotion(from = previous,initialise = 4), source = previous),\
last,\
Compensate(FindMotion(from = next,initialise = 4), source = next))
#temporal smoother, radius 1
SelectEvery(3, 1)Remember to set the reset parameter in both FindMotion calls if you want to be able to seek; I typically use reset = 50, but lower values will give you faster seeking.

[castellandw]

mg262, it would be nice if you packaged a help file next time around along with the dll for all the functions in your DLL because it's gonna start to get messy to look through this long bunch of posts in this message thread.

[Mug Funky]

yeeee! will try it immediately!

[edit]

backward prediction seems to be compensating the wrong frame? try:

back=compensate(last.deleteframe(0).findmotion(reset=100,from=next))

subtract(back,last)

[mg262]

@castellandw,

I thought about recappping all the function arguments, but realised there weren't any important arguments that weren't mentioned in the above post. So:

FindMotion(clip, ...)
from = previous/next
int reset (default 0)
int initialise (default 1)*

Motion estimation usually relies on estimated motion vectors for the previous frame, but if e.g. reset = 50 then motion vectors are calculated from scratch every 50 frames. Increasing initialise increases the accuracy of vectors calculated from scratch.

*There are more arguments, but they are IMO no longer useful except possibly for debugging... just there to avoid breaking scripts for the moment.

Compensate(clip, clip motion, [source = previous/next])

DrawMotion(clip, clip motion)

Example scripts for the first two are above; view motion like this:

#source
DrawMotion(FindMotion())

#source
DrawMotion(FindMotion(from = next))

Edit: I have put together a filter recap, below, and I will update and relink with each substantial change. But I don't want to suggest that reading this is enough to jump into the discussion... the whole point of starting this thread was to obtain feedback to direct the development (and I have been given plenty of expert feedback -- thanks, guys!)

[Didée]

Regarding artefacts in compensated frames:

>> Repair( compensated, reference, 4 )

is not the solution. But it's part of it. :)

[mg262]

@Mug Funky,

I'm a bit confused... your script produces something that shows a a big difference between the two clips, but on the other hand I can't tell what it is meant to do. You can't AFAIK use backwards-estimated information for forward compensation, for the following reason: there may be blocks in frame n+1 that don't map to any block in frame n, so it's not clear what you should put in for those blocks. To make that more precise:

findmotion(reset=100,from=next)

will find, for each block in frame 30, the block in frame 31 which is most similar to it (give or take mumbling about true motion).Now

compensate(findmotion(reset=100,from=next), source = next)

will replace each block in frame 30 with the block from frame 31 which is most similar to it.

But, given the information from findmotion(reset=100,from=next), there is no way to replace blocks in frame 31 using blocks from frame 30... because we just don't have the right information. Given a block in the frame 31, we have no easy way to calculate which block from frame 30 to use.

Does that answer your question? Or have I got the wrong end of the stick completely?

Edit: there was a silly bug in the initialisation code... you won't see any difference in the results, but please grab this again anyway:
Motion, 3 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion_03Nov05.dll) (upload at 14:06)

[Mug Funky]

aaah, i missed the "source=next" thing. sorry. works a treat now :)

thanks for your work. this is looking great.

[edit]

btw, how do i shot hpel? i can't seem to see it in the options posted here (am i blind?)

[mg262]

It's on by default. You can use FindMotion(subpixel = false) to turn it off... but it's fast enough that I can't think of much reason to turn it off? (Similarly, if for some reason you want to switch off the assembly you can use useassembly = false.)

Edit: This is not documentation, in the sense that reading it isn't a proper substitute for the discussion in this thread, but it is a recap of all the filter options to save you having to look back:

Basic Usage and Filter Options
http://people.pwf.cam.ac.uk/mg262/posts/motion%20recap.txt

I agree this thread has become too long to digest... + looking back, large parts no longer apply, e.g. thoughts on implementing backwards compensation. So if no one has any objections, I may start a new thread at some point when there is a new version, and repeat the main requests, etc?

Incidentally, to let you write standards conversion scripts (your way), what do I need to implement? Just a progressive ConvertFPS function?

[castellandw]

Incidentally, to let you write standards conversion scripts (your way), what do I need to implement? Just a progressive ConvertFPS function?

Were you talking to me, mg262? If you were talking to me, a motion compensated convertfps using your motion compensation engine in your dll would be nice for a start because I probably think it's not such a good idea to do motion compensation and then frame rate conversion separately. Do you think motion compensation and convert frame rate separately might cause jerky motion? By the way, mg262, have you tried Fizick's Block Overlap plugin yet?

[scharfis_brain]

@mg262: Hmm. I seem to have contradiced myself. (I only had a few minutes before I had to leave for school)

I have the following in mind, because it 'might' be more reliable to find motion vectors:
search the motion on the separated fields, but create (stable for static areas) a vector stream, that can be applied to the smart-bobbed clip.
But only implement it, if it gives benefits over the current approach (smart-bobbing and then searching for motion)

fpsconversion itself only needs to be progressive. reinterlacing can be done easily afterwards.

[castellandw]

I have the following in mind, because it 'might' be more reliable to find motion vectors:
search the motion on the separated fields, but create (stable for static areas) a vector stream, that can be applied to the smart-bobbed clip.
But only implement it, if it gives benefits over the current approach (smart-bobbing and then searching for motion)

@scharfi, wasn't this already discussed when you mentioned this or is it something else different:

source("blah.xxx")
assume?ff()
vectors=separatefields().analyse(interlaced=true)
compensated=somesmartbob().mvcompensate(vectors)

[scharfis_brain]

yeah it was, but mg262 said he didn't understand it ?!?

[mg262]

castellandw,
have you tried Fizick's Block Overlap plugin yet
That is the third time you have said that in the last four days. I sent you an thorough reply when you first PMed me, and it hasn't changed:
Thank you for reminding me about that... it had slipped my mind. At the moment, it looks like one of the niches Motion will fill is fast motion compensation, which means that the factor of two slowdown from BlockOverlap is quite serious. Other thing is, something like this is mainly for users to try rather than developers -- the ability to mix-and-match plug-ins without having to know anything about the code is one of the great strengths of AVISynth! Of course, that doesn't preclude me from trying it and recommending it... but for the moment my main focus is on prevention rather than cure, i.e. trying to alter my plug-in so it doesn't produce as many blocks (because that's something users can't do).

I probably think it's not such a good idea to do motion compensation and then frame rate conversion separately. Do you think motion compensation and convert frame rate separately might cause jerky motion?This makes no sense. The notion of "motion compensation and then frame rate conversion separately" is meaningless.

I don't mean to be rude... but I think you need to go and absorb a lot more about a) the nature and strengths of AVISynth and b) the uses of motion compensation (which is not an end in itself, but a means to other, specific, ends). I'm not saying this based on a single comment, but rather on the general nature of your posts; there is an implied context behind messages in this thread which you are often missing. Also, second-hand comments (aka argument from authority) are not terribly useful, not because they are right/wrong, but because we need the reasoning, intuition and often technical details that underlie them. Please don't take this the wrong way... I don't mean to have a go at you. But I do think you need to learn to walk before you can run.

scharfis_brain,
Your script was very clear and I understood it; rather, I should have said that I didn't understand point 2. (higher-quality analysis possible after bobbing) coming after the first request (analyse and then bob). The situation is now clear. I think that, as you imply, we will have to implement both and see which is better.

Incidentally, it seems perfectly legal to upsample any clip (interlaced or progressive) to improve the quality of the motion vectors found. (This is just Mug Funky's supersampling again.)

[castellandw]

castellandw,

That is the third time you have said that in the last four days. I sent you an thorough reply when you first PMed me, and it hasn't changed:


Sorry, I thought when you said it didn't preclude you from trying it that you meant you might try it since the Block Overlap plugin produces fewer blocks with motion compensation. Anyway, as I mentioned, I tried out the Block Overlap plugin and seen better results with it on motion compensation, and I was just wondering if anyone tried it out to see if they got better results as well. OK, if it's more of something for users to try out still, then no problem.


This makes no sense. The notion of "motion compensation and then frame rate conversion separately" is meaningless.

I think I should try to rephrase it. What I meant here is on applying motion compensation on the video(whether it's applying MVTools or the Motion DLL) and then using a frame rate conversion function (ConvertFPS and ChangeFPS) for standards conversion. I understand that smart bob deinterlacers are used best before ChangeFPS(which drops and inserts frames) or ConvertFPS(which by default blends frames which causes some motion blur) generally on Avisynth in terms of standards conversion. However, if motion compensation is applied, then would dropping or inserting frames or motion blurring by the likes of ConvertFPS() or ChangeFPS() have any worse effect on the motion compensated video because if not, then I'd be happy to just use ConvertFPS() or ChangeFPS()? So basically, I'm just wondering if both ConvertFPS() or ChangeFPS(), without recognizing the case of motion compensated video will worsen the motion information on a motion compensated video?


I don't mean to be rude... but I think you need to go and absorb a lot more about a) the nature and strengths of AVISynth and b) the uses of motion compensation (which is not an end in itself, but a means to other, specific, ends). I'm not saying this based on a single comment, but rather on the general nature of your posts; there is an implied context behind messages in this thread which you are often missing. Also, second-hand comments (aka argument from authority) are not terribly useful, not because they are right/wrong, but because we need the reasoning, intuition and often technical details that underlie them. Please don't take this the wrong way... I don't mean to have a go at you. But I do think you need to learn to walk before you can run.

Keep in mind that I mentioned before that you guys have much more experience in this than me, and so I'm trying to learn from you guys as best as I can as well. I'm not trying to hinder the discussion in this thread, especially by attempting to be an authority on various things. I can read up and absorb on a lot of things, but it's also good to ask people to get their understanding as well. In the case of standards conversion especially by the Doctor Who restoration team who mention to me as far as I remember that frame rate is converted while the motion compensation is taking place in high-end standards converters, that's why I mention that using MVTools or the Motion DLL and then use ConvertFPS() and ChangeFPS() is probably not a good idea, but I wanted to ask for sure if the motion blurring on ConvertFPS() or the inserting and dropping of frames on ChangeFPS() won't matter? However, since it seems that I won't really get the implied context, to relieve the only need for anyone to respond which involves the point of possible argument, I'll actually just end it before beginning it as best as I can without being insulting by saying that not responding to my question is perfectly fine to be your guys' call. Without being condescending, if I get no response, which usually is very likely for me, then I'm happy to stick to asking in the Avisynth usage forum or just keep silent while looking and trying out stuff here, which I'm very used to doing within my life because I don't want to get in the way interfering and hindering other people's discussions here.

[Mug Funky]

motion estimation and interpolation is meant as a replacement for convertfps. running convertfps after compensation (compensating what?) would be counter-productive. much better to create new frames using motion interpolation than the blending in convertfps (which for lack of a better word could be called temporal interpolation, but that's an ambiguous term).

i'm totally looking forward to fast motion-compensated standards conversion in avisynth - right now i'm doing a straight smart-bob > blend > resize > re-interlace. this is good as most regular standards converters and i'm happy with it, but if there's a way to improve something i'm always up for it.

[mg262]

I've added a bit more to the documentation and converted to HTML:

http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html

[castellandw]

motion estimation and interpolation is meant as a replacement for convertfps. running convertfps after compensation (compensating what?) would be counter-productive. much better to create new frames using motion interpolation than the blending in convertfps (which for lack of a better word could be called temporal interpolation, but that's an ambiguous term).

I totally understand and agree that motion estimation and interpolation for new frames is better than blending them in convertfps. That's why I was talking about applying motion compensation and with the only usual options for frame rate conversion in Avisynth is separately using conversion functions like ConvertFPS() or ChangeFPS() in AviSynth, that's why I thought it was a good idea to have a special FPS conversion function which applies motion compensation while it changes the number of frames per second.. However, if I'm missing any context there because of what I'm saying doesn't make any sense, then I won't bother discussing it.


i'm totally looking forward to fast motion-compensated standards conversion in avisynth - right now i'm doing a straight smart-bob > blend > resize > re-interlace. this is good as most regular standards converters and i'm happy with it, but if there's a way to improve something i'm always up for it.

I'm not sure how fast you want, but scharfi-brain's mvbob (or using older versions of MVTools to use MVFPS) is the closest we seem to have to proper motion-compensated standards conversion in AviSynth right now. By using MVTools so far, I seem to have gotten much better picture and better motion although with black spots at the edges of the pictures, which as I mentioned, I managed to improve with the Block Overlap Plugin. Just using smart-bob deinterlacers alone and playing around finding the best threshold doesn't seem to minimise combing artifacts and wobbly static areas as well as any motion compensation I've used.

[mg262]

Mug Funky,

I would particularly appreciate your opinion on this when you have a moment:

Motion, 4 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_04Nov05.dll)
Contains the function SimpleWarpCompensate, usage exactly as Compensate. (I've given it an alias SWC for convenience. Simple = Bilinear, in case you're wondering.)

It's not a proper release of the function per se; that's because I've taken several shortcuts to keep the speed up, and whether or not I keep them depends on what the feedback is. On the bright side, the speed is ~150-160 FPS.

(Please ignore the 4 pixel black border for the moment. Fixing that is a simple copy/paste job... but copy/paste jobs are best left until you have finalised the code you are copying. If I trusted MSVC's optimiser, this would all be a lot easier...)

Here is the documentation link again...
http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html

[Mug Funky]

just had a look then and it's pretty cool. it very much improves the results.

occlusions look weeeiiird now :) kinda like the behaviour of light around a black hole...

i'll have to play with it a bit more though.

oh, one other thing - on very slight motion blocks are still sometimes visible. try look at a person moving very slightly to the left and you might see it.

[edit] i'm just about to try it supersampled.

[edit 2] btw, stripes on clothing really mess this thing up :). it wasn't so visible with simple block moving i guess because if the blocks look similar enough they'll compensate well, but if the vectors aren't correct, a warped compensation will really show it up. this could be a good way to tune the motion estimation, because a warp is more visible than a mis-moved block.

good work!

[edit 3]

btw, how many previous frames worth of vectors does Motion need? could seeking be improved by just having it find (and cache?) motion on the last x number of frames rather than resetting every x frames? i don't know enough about this method to know if i'm talking crap :). i'm guessing there'd be some kind of convergence after a certain number of frames, so to keep results consistent (so a random seek produces the same vectors as playing from the beginning) there could be an internal cutoff after a certain number of frames where previous data isn't considered. that may slow it down though. not sure.

[edit 4]
Just using smart-bob deinterlacers alone and playing around finding the best threshold doesn't seem to minimise combing artifacts and wobbly static areas as well as any motion compensation I've used.

auto-noise thresholding helps, but i've not been able to do that well in avisynth yet (got very close). the reason static areas wibble so is usually noise related, or simply because static areas are too sharp (it's good practice to apply a 1 pixel vertical filter if making materials for TV, but not many people do it, alas, so the extra sharpness messes with most deinterlacers). but done right, an "auto-thresholded" kernel-based deinterlace is extremely good, and suitable as input for a motion-compensated deinterlace/standards conversion (if you look close enough at some broadcast stuff, it is clearly done this way - smart bob then mocomp, with a fallback on blending if the mocomp fails. you have to look reeeally closely to tell it's not originally PAL).

[mg262]

Thanks for all the feedback :) I know what you mean about occlusions... weird ripply effect... but it bothers me a lot less on playback than blocks do.

Here's where I cut corners: no chroma warping and no bilinear interpolation. Chroma warping isn't too expensive, but I couldn't see chroma blocks at all. (... so I wanted to know whether they would jump out at you or scharfis or someone else.) Bilinear interpolation done on a per pixel basis gets pretty expensive, and I reckon the distortion due to bilinear mesh warping has a much bigger effect.

on very slight motion blocks are still sometimes visibleI think I can see it of my source but I'm not sure I can. Two possibilities. First is chroma blocks. (Could you greyscale and see if it's still there?) Second is harder to describe... imagine the (luma) picture as a height map. Moving blocks resulted in sudden luma jumps -- discontinuities -- like you had just found a cliff in the middle of the road. Bilinear mesh warping gets rid of the jumps, but the gradient can still jump -- like there's a line in the road and on the other side it suddenly slopes up at 45°. Better mesh warping is really expensive... though it may be worth it.

how many previous frames worth of vectors does Motion need? Just one frame... but it's recursive. Generating motion data for frame 130 needs motion data for frame 129. But motion data for frame 129 needs data for frame 128. And motion data for frame 128 needs data for frame 127. So without some sort of break, you end up going all the way to frame 0.could seeking be improved by just having it find (and cache?) motion on the last x number of frames rather than resetting every x frames?Definitely, over short ranges. At the moment it just caches the last frame requested. (And it doesn't access the AVISynth cache, because filters can't access the cache for their own output, at least without using kassandro's AVSRecursion. Caching its own output is not necessarily a bad thing, because motion frames are tiny compared to picture frames.) If e.g. I cache the last reset frames computed, then you'll be able to single step backwards, which is maybe the point of the request?

btw, stripes on clothing really mess this thing upMmm... I've got thin stripes on a hat and they go through okay... but they're really thin. Any ideas on how to improve it on your stuff? [At some point I'll add global motion compensation, which may help a bit.]

I'll switch on chroma warping and bilinear interpolation (or add options) and drop a new version in a bit.

BTW, auto-noise thresholding sounds pretty cool... I have thought about it in other contexts (done one way, it comes straight out of the maths for ReverseBlend)... would be interesting to hear more about it.

[mg262]

Motion, 5 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_05Nov05.dll)

SimpleWarpCompensate now has an option bilinear = true/false (default false). Leaving it off will mean it only uses existing pixel values; switching it on will cause it to average pixel values (subpixel accuracy). I think I can see a small difference, but my eyes aren't that good (especially for live-action).

Example use:
simplewarpcompensate(findmotion(initialise = 4, reset = 30), bilinear = true)

Switching it on is pretty expensive... the overall speed drops to about 100 FPS. Even if it is left off there's a small speed hit, 10 FPS or so, due to the check as to whether it's on... I can get round this by duplicating lots of code, but if none of you can see the difference then I may just scrap it.

Here is the documentation link again...
http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html

Edit: Mug Funky, I wasn't sure whether the reply to your delta request (here (http://forum.doom9.org/showthread.php?p=731060#post731060)) was sufficient for the task/sufficiently clear... I rewrote it more clearly in the docs for Compensate (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html#Compensate). Shout if you want something different...

[castellandw]

oh, one other thing - on very slight motion blocks are still sometimes visible. try look at a person moving very slightly to the left and you might see it.


@Mug Funky, have you tried applying Fizick's block overlap plugin to remove those blocks because I've tried it on MVTools and it really removes visible blocks(I even modified scharfi_brain's mvbob script with Block Overlap and the results were impressive)?

@mg262, I haven't tried out Motion yet, but I will get to it. I was really looking forward to a motion compensated progressive FPS function for Motion first though because standards conversion was my main objective out of all this.


btw, how many previous frames worth of vectors does Motion need? could seeking be improved by just having it find (and cache?) motion on the last x number of frames rather than resetting every x frames? i don't know enough about this method to know if i'm talking crap :). i'm guessing there'd be some kind of convergence after a certain number of frames, so to keep results consistent (so a random seek produces the same vectors as playing from the beginning) there could be an internal cutoff after a certain number of frames where previous data isn't considered. that may slow it down though. not sure.


Do you think there could be an option for playing with cache size?


auto-noise thresholding helps, but i've not been able to do that well in avisynth yet (got very close). the reason static areas wibble so is usually noise related, or simply because static areas are too sharp (it's good practice to apply a 1 pixel vertical filter if making materials for TV, but not many people do it, alas, so the extra sharpness messes with most deinterlacers). but done right, an "auto-thresholded" kernel-based deinterlace is extremely good, and suitable as input for a motion-compensated deinterlace/standards conversion (if you look close enough at some broadcast stuff, it is clearly done this way - smart bob then mocomp, with a fallback on blending if the mocomp fails. you have to look reeeally closely to tell it's not originally PAL).

@Mug Funky, I think you're right, the more time I play with motion compensation, despite impressive results, I notice that various artifacts(combing or wobbly static areas) could still be apparent depending on the smart bob deinterlacer you're using. An "auto-noise thresholding" kernel-based deinterlace sounds like a great idea. I understand though that one of the reasons we use a smart bob instead of just SeparateFields() is the jitter produced by separate fields which is why I'm curious if there's an alternative to using separate fields after doing a smart bob, converting the frame rate and resizing the frame. Here's a script, for example, of what I mean (no motion compensation involved) involving PAL-to-NTSC conversion:

Source(xxxx)
LeakKernelBob(order=1, threshold=4)
ConverttoYUY2(interlaced=true)
ConvertFPS(59.94)
Lanczos4Resize(720,480)
SeparateFields() <---Replacing with a better alternative
SelectEvery(4,1,2) <---Replacing with a better alternative
Weave()

As a matter of fact, I can post this in the Avisynth usage forum if you want instead.

[Fizick]

mg262,
I at least try your plugin.
1. It is super-fast (44 fps), while MVtools is 19 fps on my PC. Why ? :)
2. but accuracy is not better.
3. What is warp compensation method.?
How you compare the warped blocks? or you simply warp it later, after motion estimation? How it related to my MVFlow (pixels motion interpolation)?
4. What is license of your plugin?

More comments later.

[mg262]

1. It is super-fast (44 fps), while MVtools is 19 fps on my PC. Why ?Because the kind of algorithms you pointed me to (recursive true motion) are very fast ;) . (Because they do a very small number of comparisons per block. BTW, 44/19 FPS are low numbers... they include time spent in AVI/MPEG2source?)
2. but accuracy is not better.Accuracy is probably worse. There are tweaks to implement (e.g. feature matching), but I am putting reasonable emphasis on speed.
3. What is warp compensation method.?Warping happens after estimation. Method is identical to yours, I think (bilinear interpolation of motion vectors).
4. What is license of your plugin?I'm not sure what you mean... I don't mind what people do with the DLL short of e.g. repackaging it in commercial applications. I am not keen on communal development, but you are more than welcome to a snapshot of the source code to read, so long as you will forgive the fact that parts haven't been properly tidied yet. I had thought of PMing you and Manao to say as much, but I decided to wait until I had polished it.

[Leak]

Source(xxxx)
LeakKernelBob(order=1, threshold=4)
ConverttoYUY2(interlaced=true)
Ummm... why "interlaced=true"? After bobbing, the video really shouldn't be interlaced anymore, so I'd think that this would screw up the conversion a bit...

np: Cinematic Orchestra - All Things To All Men (feat. Roots Manuva) (Everyday)

[castellandw]

Ummm... why "interlaced=true"? After bobbing, the video really shouldn't be interlaced anymore, so I'd think that this would screw up the conversion a bit...

OK,I understand the principle of the bob deinterlacer of interpolating to full frame meaning there should be 50 full progressive frames per second in the case of PAL, but what really confused me after using the bob deinterlacer and why I used "interlaced=true" is this:
Why is separatefields() after convertfps() being used because if LeakKernelBob deinterlaced the video to 50 or 59.94 progressive full frames per second, and convertfps() converts from 50 to 59.94 or 59.94 to 50 progressive frames per second, why would separatefields() be used on a video clip that's no longer interlaced?

Leak, I tried with "interlaced=false" although I had to change to SelectEvery(0,3) to get bottom fields first for NTSC, there are some instances though where I didn't need to use ConvertToYUY2(interlaced=false) because ConvertFPS wasn't required, here's a script for example:
Source(xxxx)
LeakKernelBob(order=1, threshold=4) # or mvbob() from scharfi_brain's script
MVFPS(59.94) #From scharfi_brain's mvbob() script
Lanczos4Resize(720,480)
SeparateFields()
SelectEvery(4,0,3)
Weave()

[scharfis_brain]

separatefields() actually can be used on interlaced and progressive video.

the latter one we are doing here and throwing away half of the fields to get 60i out of 60p.

[castellandw]

separatefields() actually can be used on interlaced and progressive video.

the latter one we are doing here and throwing away half of the fields to get 60i out of 60p.

Aah, I see. So after a bob deinterlace, of course you get 2 progressive frames out of 1 interlaced frame, but after using separatefields once you converted the frame rate, you basically get 4 half-frames out of 2 progressive frames. But the thing is now, starting from a PAL source which is top fields first that, is this the field order below for every 4 half-frames after using separatefields() on 2 progressive frames?
half-frame 0: bottom fields
half-frame 1: top fields
half-frame 2: bottom fields
half-frame 3: top fields

[mg262]

castellandw,

I appreciate your desire to get a grip on this subject, but the volume of these posts is making it hard for me to sift out feedback on the filter. If you want to ask general questions about deinterlacing, a separate thread would be more appropriate...

[castellandw]

castellandw,

Please keep to the topic of the thread. If you want to ask general questions about deinterlacing, a separate thread would be more appropriate...


Oh, ok, sorry, mg262, I didn't mean to go too off-topic. By the way, mg262, what threw me off about bob deinterlacers interpolating frames was in standards conversions script with the use of Separatefields() on progressive video frames because I was under the understanding that SeparateFields() are suitable only for interlaced frames since they're field-based, and because of that, I assumed the bob deinterlacers actually just helped eliminate various combing artifacts for interlaced frames without converting to progressive frames. So now I understand that selecting even and odd half-frames out of every 4 half-frames determine the even(bottom) and odd(top) fields respectively, along with throwing away half the fields, I understand better how the re-interlacing works. I also understand that bob deinterlacers interpolate the fields of an interlaced frame to 2 progressive frames, while motion compensation uses the motion vectors on progressive frames to interpolate the motion within the frames. Of course, this goes into even more detail than this.

One of the things that still bugs me with my standards conversion is that I thought motion compensation would help eliminate jerky motion when I convert to NTSC. I'm basically trying to convert PAL MPEG-2 to NTSC MPEG-2. The original PAL MPEG-2 source has no jerky motion whatsoever when I played it back on my PC's DVD player, but the NTSC MPEG-2 output does have some jerky motion (I use Tmpgenc by the way with highest quality motion search precision). I thought using motion compensation would help eliminate the jerky motion, but I'm wondering if the jerky motion is because of the bob deinterlacer(even without motion compensation), motion compensation or an MPEG-2 encoder like TMPGEnc? (I could take this to the Avisynth Usage forum but maybe I asked it here first in case you wanted to develop the motion-compensated FPS function later on and see if the type of bob-deinterlacer or type of motion compensation would have play a role in this.)

[AVIL]

@mg262

The link to the version of november, 05 is broken for me.

[mg262]

AVIL,

Sorry about that... please try again:
Motion, 5 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_05Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

castellandw,
should be a motion compensated fps function (like MVFps in MVTools) required for anything involving motion compensation motion compensated convertfps using your motion compensation engine in your dll would be niceI thought it was a good idea to have a special FPS conversion function which applies motion compensation while it changes the number of frames per secondI was really looking forward to a motion compensated progressive FPS function for Motion first though because standards conversion was my main objective out of all this.I asked it here first in case you wanted to develop the motion-compensated FPS function later onI heard you the first time ;).

You need to appreciate that motion compensation is not a technique to be applied on its own; just using compensate(findmotion()) on a clip will always make it look worse. Cf. earlier in the thread:Motion compensation filters aren't as far as I know intended for direct use in playback... they are intended to provide high-quality input for temporal processes, such as temporal denoising, frame rate conversion (as Fizick mentions) or deinterlacing.motion estimation and interpolation is meant as a replacement for convertfps. running convertfps after compensation (compensating what?) would be counter-productive.So the notion of "motion compensation and then frame rate change" makes no sense. When people write about frame rate change in this thread it is taken for granted that we are talking about using motion compensation to do it -- because this is a thread about motion compensating filters. No, we haven't spelt it out... but there are a lot of implicit assumptions like this and there just isn't space to; that's why this is a thread in the forum for advanced topics.

By the way, mg262, what threw me off about bob deinterlacers interpolating frames was in standards conversions script with the use of Separatefields() on progressive video frames because I was under the understanding that SeparateFields() are suitable only for interlaced frames since they're field-based, and because of that, I assumed the bob deinterlacers actually just helped eliminate various combing artifacts for interlaced frames without converting to progressive frames. So now I understand that selecting even and odd half-frames out of every 4 half-frames determine the even(bottom) and odd(top) fields respectively, along with throwing away half the fields, I understand better how the re-interlacing works. I also understand that bob deinterlacers interpolate the fields of an interlaced frame to 2 progressive frames, while motion compensation uses the motion vectors on progressive frames to interpolate the motion within the frames. Of course, this goes into even more detail than this.
I am happy for you... but you didn't really need to tell me that here :D

All this material is out there, in other threads -- forum search is your friend. Plus, you really do have to use filters to understand what's going on -- what you have gained from 24 long posts in this thread and as many replies is less than what you would have gained from a little while playing with this and other filters. I thought using motion compensation would help eliminate the jerky motion, but I'm wondering if the jerky motion is because of the bob deinterlacer(even without motion compensation), motion compensation or an MPEG-2 encoder like TMPGEnc?Why don't you find out? Trying out a few judiciously chosen scripts would do it.

Go and read, play, read, play, read, play, and try and figure things out :). (And when you get stuck, start a thread on the usage forum.) I do think you need to start with simple topics, so that you understand why things like SelectEvery(4,0,3) occur. But if you want to learn more about motion compensation, reading the MVTools thread beginning to end carefully would be a pretty good start.

[mg262]

Motion, 9 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_09Nov05.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Let me quote from the documentation:

SimpleFPS(clip, clip motion, float FPS)

SimpleFPS changes the frame rate like ConvertFPS or ChangeFPS. motion should be a clip produced by FindMotion, using from = previous (default setting).

SimpleFPS works by compensating vectors across fractional distances, with no attempt to detect or deal with occlusion. At present, it compensates with half pixel accuracy. It is not meant as a usable replacement for ConvertFPS; it is a tool to help find situations which are difficult for motion compensated frame rate change to deal with.

Example usage:

#AVISource...
SimpleFPS(FindMotion(initialise = 4), 100) Although there are too many artefacts to make this usable, the results are IMO fun to look at. It is also extremely fast -- When quadrupling the frame rate, I measured it at 400 FPS, although I'm not certain about my measurement method... but if you are increasing the frame rate it should certainly be faster than simple compensation (because fewer motion vectors per output frame need to be found).

[scharfis_brain]

I wanted to try that mocomped fps change, but it immediately closes virtualdub, when I step two frames forward.

I use an AthlonXP 1600+ and tried AVS 2005-Jan-06 and 2005-May-05

[mg262]

@scharfis,
Grr... how annoying. I am sorry about that. I am looking into it, but if you have a moment, could I have the exact script and a short clip on which it fails (10 frames would be fine)? Also, I have rewritten almost the entire internals... could you try something straightforward, like a simple compensate, and see if it works?

[scharfis_brain]

the scripts I tried:
(of course they are not of any practical usage, because I let searching for motion on a combed frame, but it should NOT crash the process)


this kills VDub after 19 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
converttoyv12()
compensate(findmotion())


this kills VDub after 2 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
reduceby2()
converttoyv12()
compensate(findmotion())

this kills VDub after 2 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
converttoyv12()
SimpleFPS(FindMotion(initialise = 4), 100)


this kills VDub after 4 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
converttoyv12()
SimpleFPS(FindMotion(), 100)

[mg262]

Please try this...

Motion, 9 November 2005 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_09Nov05B.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

[Mug Funky]

awesome! the upsize->compensate->downsize technique makes this look very good indeed.

all we need is some kind of mvmask-esque filter so we can use simple blending on the bad bits, and we've got ourselves a fast, motion-compensated standards conversion for less than 500,000 bux :)

i'm not getting crashes at my end btw.

[edit]

btw, there might be a rounding issue with the fps argument - i enter "fps=60/1.001" and virtualdub reports 59.939 fps. only a slight discrepancy, but i'm sure it's easily fixed so i might as well tell you about it.

[scharfis_brain]

@Clouded: the new B version works without crashes!
what caused them? a memory leak?

[mg262]

@Mug Funky,

I was planning a FPSOcclusion filter, outputting a framerate-upconverted mask of the dubious areas. I was thinking of boosting the quality of the mocomp bit first... there's plenty to tweak... but I'm not bothered.

scharfis's scripts above gave me another idea: reduceby2, compensate/convert FPS, expandby2, and compare with the original. Large differences->dubious areas. I haven't tried this yet...

we've got ourselves a fast, motion-compensated standards conversion for less than 500,000 buxThat's insane... someone must have done this before? Even with a bobber thrown in, it doesn't take that much code. What about MotionPerfect or whatever it's called?

FPS... thanks for reminding me; I will boost the accuracy. It's just the nominal/displayed value, not the one used to perform conversion, so AssumeFPS it away until the next release. AVISynth has a (IMO mildly annoying) setup that asks for frame rate as a ratio of two integers...

@scharfis
@Clouded: the new B version works without crashes!
what caused them? a memory leak?When copying a 4x4 block that required vertical but not horizontal interpolation in assembly, I read a 4x5 block, averaged and then wrote a 4x5 block... . That code has been in there since almost the first version, but for some reason it didn't show up until recently. Thanks for catching it! (By the way, whenever and however you can induce a crash in one of my filters, whether or not the script is vaguely sensible, I would definitely like to know.)

Where do you two, or anyone else, think I should go from here? As I never (ever) standards convert, I haven't much use for this filter... don't mind where it goes as long as there's interest/feedback. Interest in this thread seems to have died down... I don't know whether it has just become too long or that motion compensation isn't directly usable or whether it's just that I was excessively rude above(?)

Its totally reworked under the hood... interface will change soon to match... so I think maybe I would like to add any more 'core' features and release as a beta with a new thread. Seem sensible?

[Mug Funky]

hmm... don't worry about apparent interest in this tool - rest assured there's a lot, but it's often a case of people can't think of anything intelligent to post and so don't post anything at all (except me of course... i'll post unintelligent stuff routinely :)).

true motion and interpolation can be used for more than standards conversion though - there's slow mo (that's a special case of standards-conversion i guess), motion blur effects, replacing bad frames or dropouts in colour (happens a lot with VHS and sometimes with beta SP as well), or slightly over-the-top scene transitions (i saw a music video where they must have hacked a long GOP mpeg file to do the scene transitions - it looked like the I-frame had been replaced with one from another scene, causing the new scene to be slowly revealed by pushing, pulling and replacing parts of the old scene).

of course there's also denoising - couple something like this with peachsmoother or a similar long-acting temporal smoother and you've got some useful and fast tools.

about the 500k standards converters... there's cheaper stuff out there, but it's way slower and not as error resilient (there's motionperfect, and the twixtor plugins for after effects, but i wouldn't use those in production because they're RGB and simply not fast, flexible or good enough for even a straight standards-conversion).

[scharfis_brain]

I hadn't a chance to try twixtor. It needs too much software around it.

Motionperfect IMO is bubble/rubber video. occlusions aren't detected/handled at all.
Not even static areas are masked. So it is useless, too.

[Mug Funky]

i think twixtor has the same deal. it's good for low motion stuff, and to it's credit it was able to detect and interpolate blinking eyes (with an area of around 8 pixels). but it just gives too much ugly meshwarp artefacting. mvflowinter and appropriate settings are already outperforming it.

i never used motionperfect, but saw a couple of demos and was unimpressed.

reason i swear by avisynth is because it's fast, flexible and extremely extensible. also it's much cheaper than after effects :). another reason is you can mask your way out of practically any artefact and you'll still end up doing more in less time than with after effects (or anything else really).

[Didée]

of course there's also denoising - couple something like this with peachsmoother or a similar long-acting temporal smoother and you've got some useful and fast tools. Yeah, try it with Peach ... and be prepared to get interesting effects, but not so much benefit over plain usage. Peach has this preroll issue, and selfmade denoisers basically work on groups of rather small sniplets ( interleave(...).denoise().selectevery() ). In other words: at the time when your compensation "ends", Peach is still in the middle of accumulating...

OT:
Next round of LTSMC() is getting shape, meanwhile. Did a complete re-write from scratch ... it's slow, but denoising and detail preservation are stunning. Biggest problem is to make a proper function out of the big mess the script currently is. :o
/OT

[708145]

Where do you two, or anyone else, think I should go from here? As I never (ever) standards convert, I haven't much use for this filter... don't mind where it goes as long as there's interest/feedback. Interest in this thread seems to have died down... I don't know whether it has just become too long or that motion compensation isn't directly usable or whether it's just that I was excessively rude above(?)


Hi mg262!

I'm definitely interested in this filter but didn't yet spend time to test it.
My target is to up 25p sources to 50p or 100p to get more fluid pans and such.
If I got it right, your filter can already do that :)

For xvid50p I'll give this a try for sure :D
Keep up the good work!

bis besser,
T0B1A5

[marcellus]

Hi,
I have a digital camera that can record little mjpeg avi's of 288 x 208 resolution at 15 fps. I have a bunch of such avi's of low quality but with a big sentimental value that I want to convert to PAL DVD.

My interest in Motion filter is obviously the framerate conversion. I already tried SimpleFPS with default settings (after I lanczosresized to 720x576 first). The outcome it's blocky especially when camera moves quickly or shakes (and I have a shaky hand :o ) or when the content of the image is confusing the algorithm -I think - (for example leaves and branches of trees in a forest - with camera moving).

Is there another way to use this filter to convert from 15 fps to 25 fps more acuratelly (with no concern for CPU time needed)? I tried SimpleWarpCompensate that IMHO gives much better results (visually) when compensating but I can't (or I didn't figure out yet a way to) use it to convert to another frame rate. Or, better, could this mesh-warping be used in SimpleFPS function?

Thanks in advance!

[mg262]

could this mesh-warping be used in SimpleFPS function?
Sure!

Motion, 10 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_10Nov05.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

I've added SimpleWarpFPS; it's used exactly like SimpleFPS. (It was on my to-do list anyway, but knowing that someone wants it pushes it to the top :))

I think you might get better results out of the new MVTools version Fizick is working on -- at least have a look at it. Also, I'd recommend trying to stabilise before applying the filter (look up DePan), and trying to use the filter before upsizing... or maybe resize halfway, apply the filter, and resize again.

@708145,

I think it has too many artefacts to be usable on real material yet, but see what you think; usage is just:

SimpleFPS(FindMotion(initialise = 4, reset = 50), 100)

@Mug Funky,

in order to find said unintelligent posts by you, I think I would need a few hours with search :) FWIW, I would certainly have dropped this filter some time ago if it hadn't been for your interest and feedback.

For denoising, I remember you saying that FFT3D worked particularly well with motion compensation...

[foxyshadis]

A friend actually came to me last week with a handful of PAL dvds that he'd bought, asking if I could do anything with them because they don't work with his DVD player (or tv? I don't know anything about that). Sure he can watch them on his laptop, but...

So I've been watching this and the mvflow thread with extra interest waiting for scharfis to work his magic on a more stable version, plus it's fascinating on its own. ;)

[Mug Funky]

aww, thank's, clouded. :)

btw, the warp-mode is working pretty good on this end. has anything changed in the ME engine? it seems to be outputting less reliable motion than the last release, but it could just be my source and the alpha nature of this plugin.

[mg262]

Motion estimation? Shouldn't have changed since 9b. I have a script that I run through avs2avi to check that I haven't broken anything, and the only time something changed was when I removed the bug scharfis found, i.e. between 9 and 9b. But if you have a clip on which it changed, then something is very wrong and I need to fix it.

I'm never sure about warping... it feels like I'm showing my junk under the carpet rather than tidying up :p. But if we are going to warp, I think maybe it's worth trying better warpers. Adding chroma warping wont hurt either.

I just converted a clip to match the refresh rate of my screen (70 FPS) and its looking nice... maybe noticeably nicer than 100 FPS.


@foxyshadis, IMO standards conversion is a lot of work! Might be possible to multiregionise the player, or else low-end PAL DVD players have dropped in price to little more than expensive DVDs... but I'm not sure your TVs can cope (hardware forum would know). He could also try TV out from the laptop?

[scharfis_brain]

@Clouded: *aaaargh8 now I know where the crashes came from: my swapfile was disabled. I dunno what caused it disabling. F***ing Windoze...

Now, since I enabled the swapfile again, the scripts are working without crashing VDub.

[mg262]

@scharfi,

Whatever the reason, I'm grateful to you for catching the bug.

@all,

Let me outline possible additions (some very speculative):

-- Newbie-friendly Documentation
Explaining more about AVISynth, motion compensation, why you would use motion compensation, etc.

-- More compensation engines
Currently there are two engines -- move blocks and simple warp.

-- More accuracy modes
Currently the motion finder and each engine can work at specified accuracy, out of pixel, halfpixel and (sometimes) general bilinear.

-- More control
I want to let the user specify e.g. warp luma with bilinear accuracy, but move chroma with halfpixel accuracy. There will be sensible defaults/presets.

-- More accurate estimation
Especially for fast moving objects.

-- Combined estimation/compensation functions
Faster than compensate(findmotion())!

-- Support for interlaced clips

-- FindMotion Cache
Support (a little) backwards seeking.

-- Occlusion Masks

-- Compensation handling occlusion


Anything else? Or any preferences out of these?*

*Though be warned some are harder to implement than others.

Clouded

[marcellus]

Hi, Clouded
Thanks for your work. I tried SimpleWarpFPS and for my little shaky movies it doesn't do the trick. I guess I will have to try, as you said, Depan and/or MVtools. As a matter of fact I tried MVTools already and it doesn't do the trick either and it's slow too. But I will try more...

Anyway, I tested SimpleWarpFPS also on some TV captures and it seems that it works pretty good on "normal" footage, at least on footage with low / normal motion.

I made an observation (studying my avi's with fast motion) and I have a suggestion to make :

To simplify things let's say I want to triple the frame rate of a clip with the frames ABC...

A----B----C will become
AMNPQBRSTUC

Now, I observed that the new generated frames M,N,P,Q are warped versions of the frame A, and the more far we go (from M to Q) the more distorted they are, Q being very distorted but still, much closer to A than to B. When I play the movie this is giving a "pulsing" effect, when it passes from Q (the very distorted version of A) to B which is undistorted.

My suggestion would be to make M and N frames by warping the A frame but P and Q warping the B frame. Or even better, a weighted warping that take into consideration both A and B.

Please note that I'm by no means a programmer so take my suggestion accordingly... :)

Anything else?
Well, tell my if I'm dreaming too much, but it would be possible to make a motion compensation based on shapes and edges? Like it would be if I would use Winmorph (http://www.debugmode.com/winmorph/) on every pair of frames, choosing the edges and their corespondance in the next frame (but made automaticly).

[danpos]

Anything else? Or any preferences out of these?*

*Though be warned some are harder to implement than others.

For a newbie like me, I guess that is perfect for now! :)

Cheers,

[Leak]

(i saw a music video where they must have hacked a long GOP mpeg file to do the scene transitions - it looked like the I-frame had been replaced with one from another scene, causing the new scene to be slowly revealed by pushing, pulling and replacing parts of the old scene).
That wouldn't happen to be the video to "Triptych Part 3" that's found on the 188 minute DVD included with Blockhead's new album "Downtown Science" on Ninja Tune? (Which I incidentally bought today... :D)

"Motion compensation artifacts done horribly right" was my first thought... the creator of the video illustrated "Ever had the feeling that the world was gone and they left you behind?" using MPEG artifacts leaving parts of the video behind... ;)

Also, they must've probably worked on reversed video as well, as a it either tore the image to shreds or repaired it over time... :)

np: Blockhead - Cherry Picker (Downtown Science)

[Leak]

A friend actually came to me last week with a handful of PAL dvds that he'd bought, asking if I could do anything with them because they don't work with his DVD player (or tv? I don't know anything about that). Sure he can watch them on his laptop, but...
How about using DGPulldown (combined with reencoding the video to the correct resolution) for this? At least if you want to get the conversion done this year... ;)

np: Blockhead - Crashing Down (Downtown Science)

[mg262]

This...A----B----C will become
AMNPQBRSTUC

Now, I observed that the new generated frames M,N,P,Q are warped versions of the frame A, and the more far we go (from M to Q) the more distorted they are, Q being very distorted but still, much closer to A than to B. When I play the movie this is giving a "pulsing" effect...... is really useful feedback. It makes perfect sense that this occurs, but my eyes aren't good enough to see it -- so I have to rely on you guys! Thank you very much.
My suggestion would be to make M and N frames by warping the A frame but P and Q warping the B frame. Or even better, a weighted warping that take into consideration both A and B.I considered the latter but abandoned it on speed grounds -- there are less time-expensive ways of improving the picture quality. OTOH, it's quicker to add and doesn't mess with my basic framework, so...

Motion, 13 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_13Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Example usage:

#toconvert = AVISource...
toconvert
fromprevious = SimpleFPS(100, FindMotion(initialise = 4))
fromnext = SimpleFPS(100, FindMotion(initialise = 4, from = next), source = next)
toconvert.CombineFPS(100, fromprevious, fromnext)

NB new argument order for frame rate conversion functions; it's now source.simple[warp]FPS(framerate,...). (IMO much more readable!) If you're using simplewarpFPS, and you are more concerned about quality than speed, I recommend using bilinear = true.Please note that I'm by no means a programmer so take my suggestion accordingly...Suggestions from non-programmers are particularly valuable. When deciding what to do (rather than how to do it), programming doesn't come into it at all... and it's easy for a programmer to e.g. not consider something because it is slow, even if it is the right solution. So please, suggest away!

[Alain2]

Hi,

Thank you mg262 for this new filter, it seems to be a very interesting work, and a fast filter if people get 44fps speed with it :)
For denoising, I remember you saying that FFT3D worked particularly well with motion compensation...
I read in the documentation the example for "motion compensated denoising", in which you place a temporal smoother radius = 1 after the introduction of motion compensated frames; Can this temporal smoother be fft3d as you seem to suggest? Is there a particular reason for a radius 1? Can you or Mug Funky points me to the post where the motion compensated + fft3d combination is discussed?

Thanks

[psme]

Hi scharfis_brain and Clouded,

Will there be a faster mvbob using this new MC filter? Or these are not related issue? I always heard that MC deinterlace is the best method for real 60i video content. Thanks in advance.

regards,

Li On

[Mug Funky]

Can you or Mug Funky points me to the post where the motion compensated + fft3d combination is discussed?
it's pretty simple. fft3d has a temporal radius of 1 in bt=3 mode.

so you do something like this:

interleave(forward_compensated, clip, back_compensated)
fft3dfilter(... , ... , ... , bt=3)
selectevery(3,1)

obviously fill the holes with your favourite settings, and use any method you prefer for getting the forward and backward compensated clips (motion or mvtools).

no particular reason to have a temporal radius of 1 - just use as many compensated frames as you need (if you're going 1x forward and 1x backward compensated frames, then temporal radius is going to be 1. it's not wise to have a temporal radius larger than the distance you're compensating).

[mg262]

Alain2,

Good to hear from you again! Mug Funky has answered your question, but you might also find it useful to open up the documentation and scroll down to the heading Example.

psme,

I think that interest is definitely running in this direction... but it needs me to write a robust frame rate conversion function. Or else an effective mask showing where the compensation is working well and where it isn't. On that note...

@all,
For a mask showing how effective the motion compensation during frame rate compensation is, what about trying the (absolute) difference between forwards and backwards compensation? I.e. between these two things:

SimpleFPS(100, FindMotion(initialise = 4))
SimpleFPS(100, FindMotion(initialise = 4, from = next), source = next)

I've had a quick look at it and it seems interesting. I'm going to try coding something else as well... but let me know what you think of this one. Even if it messes up, it would be useful to know why it messes up...

[Mug Funky]

i liked the idea in the mvtools thread about getting a more traditional metric from a full compensation (time=100 or whatever), and compensating that mask to the desired time value.

but difference between 2 compensations seems pretty straightforward too. but it doesn't allow for the possibility that both compensations are wrong. this shouldn't come up too often though (generally with occlusion type situations when one compensation fails the other succeeds in the same areas and vice versa).

[mg262]

i liked the idea in the mvtools thread about getting a more traditional metric from a full compensation (time=100 or whatever), and compensating that mask to the desired time value.Sounds good... seems possible with the existing functions? If not, tell me what is needed. Maybe it would be useful to have some kind of difference function other than just Overlay(mode = "difference")?

I added a quick function to help when playing with masks:

Motion, 14 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_14Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

BlendFPS -- works more or less like ConvertFPS, but supports YV12 -- also should deal with arbitrary colour spaces and arbitrary frame rates.* Unlike ConvertFPS, it will average more than two frames (mainly applies when decreasing FPS by a large factor)... so it might be useful for motion blur.

*It ain't 100% clear what the "correct" weights for general blend conversion are; I've picked values that have some physical motivation. If they don't look right, let me know.

Don't worry if mask-related scripts become slow... there's plenty I can do to speed them up. But "premature optimisation is the root of all evil" ;).

Last thing: the DLL seems to have shrunk considerably without any very clear explanation... all my standard tests seem to be working okay, but please let me know if anything unusual happens.

[krieger2005]

I had a problem with your plugin and want tell it... maybe it is a bug. I use your filter of 10Nov05. I used it in this way:
# last, in high motion precise, motion builder
motionVB = FindMotion(last, reset=300, initialise=20, iterate=6, from=previous)
motionCB = SWC(last,motionVB, source=previous, bilinear=true)
motionVF = FindMotion(last, reset=300, initialise=20, iterate=6, from=next)
motionCF = SWC(last,motionVB, source=next, bilinear=true)

interleave(motionCB,last,motionCF)
Despot(mscene=10, pwidth=260, minpts=20, maxpts=260, pheight=160, p1=23,p2=10,mthres=16, merode=19, color=true, fitluma=true)
Despot(mscene=10, pwidth=140, maxpts=50, minpts=9, pheight=140, p1=22,p2=10,mthres=12, merode=18, color=true, fitluma=true)
SelectEvery(3,1)
I tried it and it works well, but on a special situation VDub crashed (closed without any message). I looked at this position (it was frame 4232 but however). It is a scene where a static picture was shifted... like a panorama-picture, which was shifted from left-to-right side. The shift was very fast...

I looked at the code and the problem seems to be in the FindMotion-Function. Especialy if the variable "iterate" decrease the position, when VDub crashes, is later. if "itearate=1" VDub does not crash on these scene...

If youwant these clip a have to manage the upload but maybe you does not need it.

Thanks for development

[mg262]

krieger2005,

thanks for your message!
motionVF = FindMotion(last, reset=300, initialise=20, iterate=6, from=next)
motionCF = SWC(last,motionVB, source=next, bilinear=true)There is a typo here -- I think you mean motionVF. I'm not sure whether that caused the crash though -- I will look into it.

[krieger2005]

thanks for showing of the typo... too many changes in the script, as allways

I must corrent what i said above... Not the function FundMotion make the probelm but SWC, because if i let show the Vectors with "DrawMotion" on the original-clip it works ok. When i put out the compensated frames made by SWC VDub close.
However, it seems to be something with the "iterate"-Vaiable. Also the failure is when "source=previous" and "source=next". With "bilinear=false" the failure occur also.

[marcellus]

Hi, Clouded
Speaking of Motion... you are moving realy fast! :)
With the new function CombineFPS things are looking FAR better. :thanks: for your work!

In the week-end I had time to play with DePan and I didn't got good enough results, until I discovered Deshaker for VirtualDub (shame on me discovering it so late, I'm more accustomed to Avisynth than VDub processing). AntiFlicker and Deshaker do wonders with my clips, putting them in much better shape for a FPS conversion.

And now you come with this CombineFPS which works much better than anything else I tried (even on un-deshaked clips), and it's fast too. Thanks a lot! I'm begining to hope I will get where I want.

[scharfis_brain]

@clouded: thanks for blendfps.
May I suggest a boolean parameter that lets the user decide the blending mode:

option 1: behaviour like convertfps:
- never blend more that two frame together when doing framerate downconversions
- duplicate frames except for the frame to frame transition with extreme framerate upconversions.
- but fps paramter unrestriced

option 2:
- blend as many frames as needed together when the user defines a framerate downconversion.
- make a slow travelling blend weigth movement with extreme framerate upconversions. (so one can simulate slowmotion with blending)

[mg262]

krieger2005,

There was a typo in the warp code (copy/paste bug as ever... h in place of v) that caused a slight inaccuracy in the warping; luckily for me, you found a case where this caused it to access off-screen. Thank you very much!

I replaced the DLL...
Motion, 14 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_14Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Edit: I'm not sure the server has the new file... give it a few minutes
... it should be there now.

And, please, anyone using warping, download this version. I added more safety checks to the warping which may slow it down somewhat... but they are needed in some unusual situations. Please tell me if the slowdown is large.

marcellus,

I'm glad it's working well! It is only because I have incisive feedback from yourself and others that I can improve things. So, if you spot other reasons/circumstances where it doesn't work well, please tell me!

By the way, I think it is possible (but fiddly) to import Deshaker into AVISynth if needed; at least, I saw a thread on this once.

scharfis_brain,

Thanks for the suggestion. I will certainly put it in, when I have thought about it a little and thought of sensible parameter names (I dislike mode = 1, mode = 2, etc). Edit: what about a variable "aperture" for motion blur?

Grab Motion, 14 November 2005 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_14Nov05C.dll) and try BlendFPS(float targetframerate[, float "aperture" = 1]). Increasing aperture will blend more frames; decreasing aperture will blend fewer.

[krieger2005]

Okey,
i tested the new version (not the c-version) and the old from 10Nov. Here my speed-results for 2 clips with 1000 frames:

#14Nov05:
#ca. 37-42 fps, 26 sec / ca. 37-40 fps, 26 sec
#10Nov05:
#ca 37-40 fps, 25 sec / ca. 36-40 fps, 26 sec

I have a P4 3000MHZ, HP enabled. The new version of the motion-filter does now not quit Vdub. thanks.

I would say, that if there is a speed drop, than not a big one, since both times are equal for a 1000 frame-clip, so i guess for a larger clip it would be the same...

But still VDub close. But this is one thing of my avs-scripts... they are too complex. But maybe you can add something in, so that your filter stop to allocate memory when a maximum is arrived because i had situations, where vdub take 2GB (swap-file).

BTW. I think also it would be great to use a global motion/MV-Format... Fizicks' MVFlow and your Vector-Analyzer would, i think, work great together.

Can you add (but i guess you have it) the scene-change-detection in your to-do-list?

great work.

[mg262]

Thank you for checking it.
maybe you can add something in, so that your filter stop to allocate memory when a maximum is arrived because i had situations, where vdub take 2GB (swap-file).I'm not allocating memory at all; I'm relying on AVISynth to do it for me. There is a mechanism to provide hints to AVISynth about what information should be kept, and I will try that to see if it helps. Could you give me a script that causes large memory usage?

As far as I know, the MVFlow???? functions and my warp functions do exactly the same thing (bilinear mesh compensation)... so there's not much point in mixing and matching.

Scene detection: will do. Also see here (http://forum.doom9.org/showthread.php?p=736198#post736198).

[krieger2005]

Here is such a script:
f=4
#analyze1 = last.FFT3DFilter(sigma=6).BilinearResize(m4(width*f), m4(height*f))
analyze1 = last.BilinearResize(m4(width*f), m4(height*f))
analyze2 = last.BilinearResize(m4(width*f), m4(height*f))
motionVB = FindMotion(analyze1, reset=0, initialise=20, iterate=6, from=previous)
motionCB = SWC(analyze2,motionVB, source=previous, bilinear=true).LanczosResize(width,height)
motionVF = FindMotion(analyze1, reset=0, initialise=20, iterate=6, from=next)
motionCF = SWC(analyze2,motionVF, source=next, bilinear=true).LanczosResize(width,height)
interleave(motionCB,last,motionCF)
temporalsoften(1,5,5,10,2)
SelectEvery(3,1)
i know that fact that with such supersampling one must have much memory, but i thought that there would be an error-message, so i thought you allocate the memory internal.


As far as I know, the MVFlow???? functions and my warp functions do exactly the same thing (bilinear mesh compensation)... so there's not much point in mixing and matching.
I don't know anything... i'm a noob :) The results looks just a little bit different. Maybe because of different motion-informations given by two different Algorithms...

Your idea for scene-change-detection-extra-filter is, i think, good, since very much filters need such an information (as you written in that thread). Do you want make it based on motion-informations made by, for example, "FindMotion" or a simple "luma-difference"-thing? But... i will see it when you release it :)

[mg262]

Careful when supersampling... you are effectively making everything move much faster, and I don't know how well the filter will cope. At least, please check vs the non-supersampled version. Also, as it says in the docs, iterate isn't really meant for general use... you are welcome to try it, but be aware that I haven't checked its effectiveness yet.

I've read all of Fizick's thread, but not all the documentation updates... so I will go and read up when I have a bit of time.

Scene detection: The idea is to have a standard scene representation (a scene type) and many different scene-detection filters which all output that type. Certainly SAD (= luma-difference?) and SSD will be in there.

I am dragging my feet on this because once a representation is shared between many filters, it becomes much harder to change -- so I need to get close to the right answer first time.*

*The "much harder" is for the case where all the filters are mine... the alternative is much worse. This is precisely what you see in formats like AVI... they can't be changed because they are too widely used, so you get all sorts of hacks being put in place and the situation ends up in a total mess. Same goes for AVISynth 2.5. Simulating associative arrays, etc. helps a bit but is far from a real solution.

[Fizick]

Stupid question:
What is SSD ?

[Manao]

SSD: Sum of Square Differences
SAD: Sum of Absolute Differences

[mg262]

I intend to allow auto-generation of motion data if motion isn't specified. So these

Compensate(FindMotion(from = next,initialise = 4), source = next)
SimpleFPS(100, FindMotion(initialise = 4))
SimpleWarpFPS(100, FindMotion(initialise = 4, from = next), from = next)

will become

Compensate(source = next)
SimpleFPS(100)
SimpleWarpFPS(100, from = next)

But first, FindMotion needs better defaults...

In particular: shall I set reset or not? And what kind of value?

[krieger2005]

For my taste i can't see any differences when i set reset. I set it mostly to 100-300. I think a value about 150 should work good enough for the most content and give a fast startpoint.

Just an other question. Since the filter compensate just a frame before/after to an actual frame there does not exists enough informations to compensate the picture correctly. For example on a left-shifted scene the filter can't reconstruct the left side. I am right till now?
Now it would be nice, if these informations could get from somewhere. Since this is a complex search in most cases a simple assumtation could be made:

A B C D E

is are 5 frames. If the frame C should get compensated then B could be used. For all unknown content frame E could be used. On the other side D and A can be used.

Is there a possibility to get something like that or is this all nonsense?

krieger2005

[foxyshadis]

Of course once you have a scenechange mechanism, you can truncate reset there everytime.

Is there planned to be any support for global luminance changes? Maybe there already is and it just causes a lot of extra hiccups or something.

krieger: That's what CombineFPS does at the moment (though clouded is working on a less cumbersome method). The 'not enough information' areas are called occulsions, and you're right, that is generally the best way to deal with them.

I've been testing out 120 fps on a few sources, mostly out of curiosity and to see whether I could break it. Sometimes it looks ghosty and ripply as hell, especially on fast motion (it gets trippppy), but at 120 it's not that big a deal, and the times when it looks amazingly smooth make up for that.

[mg262]

@krieger2005,

Setting lower values of reset will give you faster seeking -- but at a cost of either time or quality. If initialise is high, quality won't suffer -- but processing time increases by about (initialise/reset*100)%... so with your settings of 20 and 150, you get a 13% increase. I tend to use initialise = 4 and reset = 50, but I haven't yet run enough checks to be sure whether 4 is high enough (20 seems rather high).

The rest of what you said is definitely not nonsense -- in fact it contains the spirit of the methods used in some papers I've read. Adapting such things to warping is a little more work, though (but definitely possible).

@foxyshadis,

I'm definitely going to reset on scene changes! Global luminance changes... atm only fades come to mind; could you give me better examples?

@all

I'm going to quote something from the docs...[SimpleFPS] is not meant as a usable replacement for ConvertFPS; it is a tool to help find situations which are difficult for motion compensated frame rate change to deal with.FPS change is a battle I have not yet begun to fight... think of this as reconnaissance.* Complicated strategies can break things as well as fix them... so you need a simple benchmark that is not clever in any way to check them against. That's what SimpleFPS is for. It is also a speed reference; it may also be useful in your scripts (e.g. with masking)... but I'm not suggesting it as a ConvertFPS replacement.

*(And the two functions, SimpleWarpFPS and CombineFPS, I built on request because they were easily added.)

Edit: but it suits my purposes very well that you're playing with it :cool: ... because then you'll spot when other methods break things!

[foxyshadis]

Fades are really 95% of global changes... otherwise, brief flashes, drops, or auto contract correction.

One of the weird things I notice (I have a specific very regular panning clip I like for this) is that using combinefps, occusions will leave a rippling pattern in their wake that extends well beyond the passing of the object. Usually on the second original frame after the occulsion it settles down, rather than the first or even before. (Maybe it could use confidence weighting for mvs to decide whether forward or backward gets the most weight?)

Also many objects seem to move via a trippy blendfps, the blendweight changes but the object also... shrinks and melts toward the new position in big chunks. I'm not sure how to describe it. Protrusions (tops of logs) and thin objects suffer from it the most, I guess because there isn't enough... meat? to grab on to.

And unfortunately it doesn't seem to work too well with solid-color areas, when checking out the vectors it often seems to get confused and point to similar nearby spots instead. I can understand that as a limitation of the area matching, so I'm not bitching, just letting people know even for animation with well-defined motion it's to e used cautiously.

Oh, and concerning your earlier comment about mvflow, I noticed almost exactly the same artifacts in a clip in both methods, though not as pronounced in motion, so your algorithms are definitely very similar.

Last question, is seach-radius configurable? Or would that be silly for this application?

[mg262]

This is really useful stuff, foxyshadis. :thanks:

Fades first: I can't see a way to tweak the filter, but you could detect the fade in script (e.g. search for many consecutive blends), and treat it sensibly. E.g. luma-equalise the fade, motion detect on that, and compensate the original. Or: fade-compensated interleave, like this (0 0)(0 1)(1 2)(2 3)(3 4)… with each () pair luma-equalised, FindMotion with reset = 1, high initialise, and SelectOdd to get MVs. Similarly for drops (n.b. Mug Funky's point on using Compensate to fix them) and flashes. Auto-contrast... :scared:. I'll keep thinking about these.

The method always tries the motion vector from the previous frame, which is why the ripple can persist. There's no search radius, but increasing iterate has a similar effect... but e.g. iterate = 4 -> FindMotion is 4x slower.

Maybe it could use confidence weighting for mvs to decide whether forward or backward gets the most weight?'Confidence' is a better term than I had -- I shall steal it shamelessly :D. I was thinking of this:

CombineFPS(forwards, backwards, fmask, bmask, fallback)

using a weighted combination of forwards and backwards, with weights depending both on temporal closeness to original and the two confidence masks; if both masks were low, it would blend in fallback as well. That should also deal with screen edges and small/thin objects. It won't fix flat areas... I have some ideas on that and on faster methods, but that's for later.

For now, the right thing to do is to play with confidence masks until we build up intuition about them. After I finish tidying, I will throw out a few masks (maybe 2 filters + 2 scripts) and rely on yous to tell me which ones work well for what kind of content :).

[mg262]

I'm soon going to put in the new interface, but I wanted opinions on a couple of things first. I'll illustrate it with FPS conversion (compensation will look almost identical).

SimpleFPS and SimpleWarpFPS are going to be replaced* with one function FPS, that can be used like this:

FPS(100, move) #works like SimpleFPS(100,...)
FPS(100, warp) #works like SimpleWarpFPS(100,...)
FPS(100, warp(bilinear)) #works like SimpleWarpFPS(100,bilinear = true,...)
FPS(100) # default mode = move, so works like SimpleFPS(100,...)

*well, I'll leave them around for a bit so as not to break scripts.

Here's the full argument list

(source) clip
float targetFPS
mode = move/warp/warp(bilinear)/... [default = move]
chromamode = move/warp/warp(bilinear)/... [default = same as mode]
clip motion [default = autogenerated]
source = previous/next [default = previous]

I hope that's clear. If motion isn't specified, it will be auto-found from the source; otherwise, you can use FindMotion(...) as before OR you can pass another clip of the same size as the source, and motion will be found from that. So a more complicated example would be this:

denoised = input.SpatialSmooth(...)
input
FPS(100, mode = warp(bilinear), chromamode = move, motion = denoised)

This applies warp-compensation to the luma plane with bilinear accuracy, but constructs a chroma plane by moving blocks; motion is automatically found from the clip denoised.

[Mode arguments can be passed through functions... so CombineFPS(100, ???) will call FPS(100, ???, source = previous) and FPS(100, ???, source = next) and amalgamate the results. So you will be able to write e.g. CombineFPS(100, warp)... a bit more compact than the current version! Or, I may add the option source = both.]

How's it sound? In particular, I'd welcome any thoughts on sensible argument order -- targetFPS + mode should be probably be first, but after that I'm not sure...

[marcellus]

I'm soon going to put in the new interface
Great news! :cool:
How's it sound?
Changes are looking good.
One little thing:
warp(bilinear) looks weird when I read the script, like bilinear is a clip.
Maybe warp_bilinear or something like that would be better. Or you could keep bilinear=true with no effect when mode is 'move'.

I hope you will keep 'iterate' as a parameter, I obtain better results when I increase it, the result pays the speed decrease. And maybe you could 'unveil' more parameters (regarding especially the motion search), in general I have no problem with a lot of parameters as long they are well explained in the documentation.

I'd welcome any thoughts on sensible argument order
Personally I like to use in scripts named arguments like:
FPS(targetFPS=100, mode=warp)
instead of
FPS(100,warp)
because I have a poor memory so when I read the script after half a year the first version looks more intuitive, so I can remember what I was doing. But the natural order seems IMHO:
FPS(clip,targetFPS,mode,chromamode,source,motion)

[foxyshadis]

The parenthesis are kind of weird... "warpbilinear" would work as well. As for argument order it seems great now, the last two could swing either way. Source=both sounds more coherent overall than a separate combinefps function, even if they'd be exactly the same (ie, source=both just calls source=next/last).

The main thing I can see is that maybe it should be MotionFPS; I'm sure there's no plain "FPS" now but it does seem like namespace pollution. =p

[mg262]

Warp is currently a function... syntax like this:

Warp(accuracy = pixel, moveaccuracy = halfpixel)
[each accuracy can be set to pixel/halfpixel/bilinear... but I may add more modes]

So you could write
FPS(100, mode = warp(accuracy = bilinear))

Is it less confusing like that?

More advanced modes will need the function syntax [e.g. mode = rigidwarp(rigidity = 3.5)] so it should probably stay in for consistency -- and it allows finer control from script... but I could also add lots of constants like warppixelpixel, warppixelhalfpixel, warppixelbilinear, (and maybe warppixel, warphalfpixel, warpbilinear) for those that prefer them?

I felt names like SimpleWarpCompensate were getting out of control, so I've been trying to push the other way. I wasn't sure about FPS either... but if I use MotionFPS then I should use MotionCompensate for consistency... and that seems too long to inflict on people. Plus, the syntax for FPS is such that it is unlikely to clash with another FPS function (though I can't see what one would be). Maybe I could add both FPS and MotionFPS, Compensate and MotionCompensate... :confused:

Namespace issues worry me a lot... but OTOH that comes from the bit of my brain that works on properly-typed extensively structured languages, and often such intuitions turn out to be v. inappropriate for AVISynth. I don't know... :confused: :( .

[Fizick]

May be I am wrong, but you can not use mode=warp, it must be mode="warp", i.e.string.

[marcellus]

I felt names like SimpleWarpCompensate were getting out of control, so I've been trying to push the other way.
As long the function does it's job I have no reason to complain for function names being too long, as long they are intuitive (we all do copy/paste anyway). A clear and detailed documentation is more important. Having multiple names for same function might be confusing though, IMHO.


So you could write
FPS(100, mode = warp(accuracy = bilinear))

Is it less confusing like that?

More advanced modes will need the function syntax [e.g. mode = rigidwarp(rigidity = 3.5)] so it should probably stay in for consistency -- and it allows finer control from script... but I could also add lots of constants like warppixelpixel, warppixelhalfpixel, warppixelbilinear, (and maybe warppixel, warphalfpixel, warpbilinear) for those that prefer them?
Wouldn't be better, for example:
FPS(100, mode = warp,accuracy = bilinear,rigidity=3.5,subpixel =true)
and explain every parameter and their relationship in the documentation? And when, somebody writes by mistake:
FPS(100, mode = move,accuracy = bilinear)
the accuracy parameter should be ignored because has no signification in mode "move" (or throw an error message).
That's the kind of synthax I'm used to.
(Previewing my post I noticed that while I was writing Fizick made a post too and I think he's right.)

[Fizick]

IMHO, FPS() or WARP() is not appropriarte names (too general).
(Do you remember recent V.C.Mohan's plugin with too general name "Denoise" ?)

But by the way, first alpha versions of MVTools were named "Motion" (Motion.dll") :)

[foxyshadis]

I see! I was confused, I thought mode= was always a string and that was just a weird way to put it. Now it makes a lot more sense. (Certainly "profiles" would be a great idea, deen proves their popularity, but they can come later if it's a hassle.)

I forgot again that names can overlap. I don't think there's enough of a standard in avisynth to say "it has to be prefixed" or vice versa, but in some way it should identify its source (like MDeblock vs. just Deblock).

[mg262]

I don't use strings because AVISynth doesn't copy them properly -- so certain planned features would require leaking memory or something else unpleasant. Profiles -- definitely! I was thinking of defining HQ and LQ as arguments to be used anywhere... like this

FPS(100, HQ)
FPS(100, warp(HQ))
FPS(100, mode = HQ, chromamode = LQ)

I have taken all the remarks on name conflicts on board... it's just that something like
MotionFPS(100, mode = motionwarp(accuracy = motionbilinear, moveaccuracy = motionhalfpixel), from = motionprevious)
seems to me to become completely unreadable.

At root, it comes down to the fact that the language doesn't have implicit or explicit namespaces or (productive) types,* and I at least find it very difficult to work without these. Type-based function overloading definitely isn't the right mechanism for dealing with namespace conflicts, but in the absence of a proper mechanism, it's better than most alternatives. (I.e.: I'm not so worried about using warp because it doesn't take a clip as an argument -- so it won't conflict with other functions.) And, many natural names have already been taken... consider Invert, Deblock, RemoveGrain, DeGrain, UnBlend, DeBlend, RemoveBlend; although in theory it could cause trouble, I haven't seen it happen often if ever. [Warpsharp is the only exception that comes to mind.]

Having been heavily criticised (not here) for making things too inaccessible, I'm trying to make things as un-intimidating for newbies as possible... and making things short is one of the main ways to do this.

*There is a pseudo-namespace feature, in that you can write DLLName_functionname(...).

Edit: I forgot to say anything about iterate. I'll certainly leave it in if you want it... but I haven't yet put in the tweaks that might make it more effective. (Plus I think of this as a faster, lower quality alternative to MVTools... ) I'd like to have a way to set reset, iterate, etc in the new syntax... my preferred method would be this:

FPS(100,..., motion = auto(iterate = 2),...)

i.e. have auto be a function just like warp, with overrideable defaults.

[tsp]

I don't use strings because AVISynth doesn't copy them properly -- so certain planned features would require leaking memory or something else unpleasant. Profiles -- definitely! I was thinking of defining HQ and LQ as arguments to be used anywhere... like this
mg262: Did you use env->SaveString(const char* s, int length = -1); when using strings? or when does the memory leak happend?

[mg262]

No... but as far as I know that doesn't deallocate until everything is unloaded. It's not that I have a resource problem due to losing a few hundred bytes due to leakage; I just find the idea of things persisting beyond their appropriate lifetime... deeply unpleasant. It makes me want to wash my hands ;).

[tsp]

you could say that by calling SaveString you link the string with the IScriptEnvironment so it is okay that it is first destroyed when the last reference to it disappear :) In fact most avisynth filters that handles strings uses SaveString meaning if you uses them with ScriptClip or FrameEvaluate they will allocate memory for the string argument for every frame because the constructor is called each frame and the memory is first freed when the IScriptEnvironment is destroyed. I don't know if that is what you mean with memory leak but it only happends when using ScriptClip or FrameEvaluate?

[mg262]

tsp,

:( I'm sorry, I've given you the wrong impression. I haven't measured a leak. I'm not concerned about resources. I don't like this for aesthetic reasons, just as I don't like global variables, or typecasting pointers to ints, or goto statements, or non-constant static variables for aesthetic reasons. Working in these ways... just makes me feel horrible inside :( .

(Please don't let me give the wrong impression. I'm not trying to be rude or criticise the way anyone else chooses to work. @all, do as you see fit :) .)

Edit: I missed this too
FPS(100, mode = warp,accuracy = bilinear,rigidity=3.5,subpixel =true)The thing about this is that it doesn't let you pass the mode around easily. So script functions might want to e.g. call FPS twice (once with previous, once with next), mask the results in some way, and then combine them. Currently they can take mode as an argument and pass it into both calls... that becomes much, much messier & harder if the mode info. is scattered across many parameters.

[mg262]

1. You're all right, MotionFPS it is.

2. Two alternatives for mode = warp(accuracy = bilinear) syntax have been suggested:
-- Add all the arguments (accuracy, moveaccuracy, rigidity, chromaaccuracy, chromamoveaccuracy, chromarigidity etc) to every function

A simple script that e.g. generates forwards and backwards motion (with a specified mode), creates masks and combines motion will need a line like "accuracy = default(accuracy,...)" for each argument.
An editor like AVSEdit will end up displaying every argument for every mode, which becomes unreadable.
It's harder to write newbie-friendly docs, because they have to be split up into lots of cases.

-- Use strings instead (leaving aside my distaste for these...)

Unless I write a proper parser, the syntax will be more restrictive than AVISynth.
Numeric arguments (e.g. rigidity = 3.5) will need to be stuffed into strings and extracted again (ugly).
Scripts will have to write things like "warp[" + mode+"]" rather than warp(mode).

So I'm sticking with the current syntax. BUT

3. There will be plenty of presets
...so no one has to delve into this syntax unless they want to try something advanced.

Just for fun, the graph below shows relative time/frame of each mode. (Warp has two arguments because it can often move a block without stretching it; the second argument tells it what interpolation to use when doing this.) You can see that moving things with half-pixel accuracy is just as fast as moving them with pixel accuracy... so there are just 3 sensible modes at present. I was thinking move, warpfast and warp? (And bilinear move-accuracy will give moveslow and warpslow.) Any better ideas?

http://people.pwf.cam.ac.uk/mg262/posts/Motion/FPS 100 times.PNG

Apart from CombineFPS/both, it's ready to release once I have names for everything. [BTW, I might use "method" instead of "mode"... I think it's more descriptive.]

[communist]

I'm mostly testing this plugin with bobbed DV PAL footage that I wish to slowdown extremely - works good for some and doesnt for some other footage (some parts are slowed down constantly while others 'stutter'). But so far looks very promising - Thanks for this great plugin :)

I can post some problematic scripts / clips if needed but gotta clean it all up a bit ;)

[Velocity 7]

Seems a bit quiet in this thread.

I've read through the entire thing, but could someone remind me if there was anything in regard to motion compensated deinterlacing (as opposed to motion compensated frame conversion)? For example, 30i -> 60p (for video game consoles outputting 60 fps, but squashing to 30i, or for example MiniDV cameras recording in 30i for 60 fps content, etc., see http://www.100fps.com for camera examples)*.

* - just posting as a reminder for those who might not know what the hell I'm talking about

[Mug Funky]

sure. if you can compensate frames, you can use them for deinterlacing.

basically you use a regular smart bobber (leakkernelbob works well for this purpose and is fast), then compensate 1 field onto the other, and use the compensation to fill the missing lines in the original field. then (the slow part) you make sure no artefacts get through.

i think mvbob has been modified already to use the latest mvtools, and i'm sure it can be further modified to support clouded's motion filter (which i prefer conceptually, but it needs tweaking to avoid peculiar artefacts. it's the same deal with fizick's improved mvtools really).

in short, if you can compensate you can deinterlace.

[Velocity 7]

What are the actual parameters to Compensate(), though? The documentation API is kind of off... (e.g., can't just use Compenaste() alone)

[mg262]

I'm sorry about that... I updated the documentation to the new filter version before I should have. The old documentation is still there at

Old Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion_old.html)

I haven't finished testing the new version to my satisfaction -- but I've become rather tied up in other things, so it will be a while before I can finish that. If you want to try it anyway, here it is:

Motion, 28 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_28Nov05.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Old scripts should still work, if you make one change: rename Compensate to SimpleCompensate. [Or alternatively, name the motion argument of Compensate, i.e. source.Compensate(motion =...).]

I haven't abandoned this project, but it may be a bit before there are substantial updates.

I will write proper replies to some of the above shortly...

Edit: On the standard mode (i.e. not warp), any idea which kind of errors you're mainly seeing?

A. Incorrect motion estimation
The wrong motion vector is predicted and so the wrong block is chosen.

B. Occlusion/Revelation
The "right" block simply doesn't exist in the source frame because e.g. one object is moving in front of another.

Type A errors are hard to fix because there's not much leeway to play with the motion estimation algorithm (except by increasing iterate). Type B errors can be prevented by e.g. making use of the previous/next frame or bidirectional compensation; they can also be 'cured' by e.g. using masks to fall back to blending.

[mg262]

A script function (for use with the new version):

CombineBlendFPS changes the frame rate by combining forward and backwards compensation (like CombineFPS), but when the luma difference between the two is large, it gradually falls back to blending (like ConvertFPS).

Arguments:
input,
FPS,
mode -- move or warpfast or warp,
compdiff -- luma difference giving pure compensation,
blenddiff -- luma difference giving pure blending,
show -- set to true to show where blending is being used,

example usage:
CombineBlendFPS(100, warpfast)

A luma difference halfway between compdiff and blenddiff will give a 50:50 mix of compensation (i.e. CombineFPS) and blending (i.e. ConvertFPS). I recommend playing with compdiff and blenddiff ... lowering them will remove more motion artefacts, but increase blending, and vice versa.

function CombineBlendFPS(clip input, float FPS,\
int "mode", int "compdiff", int "blenddiff", bool "show")
{
mode = default(mode, move)
compdiff = default(compdiff, 8)
blenddiff = default(blenddiff, 14)
show = default(show, false)

input
p=MotionFPS(FPS, mode)
n=MotionFPS(FPS, mode, source = next)
b=BlendFPS(FPS)
c=CombineFPS(FPS, p, n)
s=c.greyscale().Levels(0, 1, 128, 0, 255)

yv12subtract (p,n,tol=compdiff)
yv12lut("255 " + string(blenddiff-compdiff)+" / x *")
mask=fity2uv()

return maskedmerge(c,show?s:b,mask,u=3,v=3)
}I would be grateful if someone who is good with scripts checks that, especially the chroma processing. (Basic idea is to construct a mask of the luma difference, copy the mask to the chroma planes, and use a masked merge.)

This isn't the end of the story... I don't think luma differences are that reliable. But feedback on this will be useful for future development.

Edit: in addition to other stupidities I pasted in the version with the FPS hardwired to 70...

[Didée]

Seems okay to me, so far. Replacing the "levels()" with "YV12Lut()" would be a tad faster, and the "r2 = ReduceBy2() + YtoUV()" combo could be done more simply by "r2 = FitY2UV()".

Working with plain luma differences is somewhat like tieing up shoes while wearing thick mittens, indeed ;)
It gets better when building a relation between the luma difference and the local detail level - but doing so, things get slower and slower: checks, more checks, even more checks ...

[mg262]

Didée,

Thank you very much! I've changed it as suggested, and also tweaked the defaults.
It gets better when building a relation between the luma difference and the local detail level - but doing so, things get slower and slower: checks, more checks, even more checks ...I'm hoping that something more intelligent can be pulled out of the motion vectors... for example looking for discontinuities or for sudden changes from one frame to the next. But this script was lower-brainpower effort :).

[Didée]

yv12lut("255" + string(blenddiff-compdiff)+"/ x *") must be
yv12lut("255 " + string(blenddiff-compdiff)+" / x *")
^ ^ ;)

[mg262]

Rrrr... I don't seem to be very good at scripts. Thank you!

I can speed this all up more with filterlets (e.g. scale, scaleandmerge, scalediffandemerge) if anyone finds it useful. + other FPS scripts will appear...

[mg262]

I've tried implementing a MVFlowFPS-like method, and I was very impressed with the results. IIRC Fizick spent a lot of time and effort looking at artefact protection, and it shows. (Thank you, Fizick!)

Motion, 6 December 05 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_06Dec05B.zip)
MotionProtectedFPS script (http://people.pwf.cam.ac.uk/mg262/posts/Motion/MotionProtectedFPS.avsi)

MaskTools is needed (but you should have it anyway ;)). Simple usage is just

MotionProtectedFPS(100)

This runs reasonably fast (AVSTimer measured it at 70 FPS).

You might get slightly higher quality by increasing iterate:

MotionProtectedFPS(100, iterate = 4)

(Runs at 60 FPS on my machine). Increasing protection may decrease the number of artefacts, at the cost of increased jerkiness -- but the default works pretty well, so don't increase it for the sake of it. The other arguments probably don't need fiddling with.

[ONLY INTERESTING TO SCRIPTERS]
I haven't documented SumStretchFPSMask yet, but it just creates an occlusion mask using the same method as Manao/Fizick. I've tried to write the script in such a way that it is easily readable by scripters (even the spacing :p).

Mug Funky, scharfis, Didée (if you aren't too buried in other things), and anyone else interested in this stuff: if you have a moment try looking at the intermediates, especially maskp, maskn, maskn.invert(), and the output of CombineFPS. All these masks are small (1/64th the original, plus padding up to be legal resolution), so you need to use the bilinearresize/crop combo (see script) to look at them usefully.

The core of Fizick's method relies on using both temporal information like CombineFPS (i.e. if you are trying to construct a frame at time 63.1, you want to use frame 63 much more than frame 64) and also occlusion information (where frame 63 is useless, use frame 64 anyway). Try single-stepping maskedmerge(p, white_clip, ??) with each mask -- or maybe all 4 at once reduced -- and it should become pretty obvious what is going on. [But poke me if anything is unclear.]

Now: because the masks are so small, operations on them are essentially free. So you can try expands, inflates, etc -- also you can replace bilinearresize -- to your hearts content, to try and build better occlusion/combination masks. I have faith in you guys :) [/ONLY INTERESTING TO SCRIPTERS]

[scharfis_brain]

I get a problem with unmatchable objects, that are getting blended - obviously.
Their chroma isn't in place!

try this video: http://scharfisbrain.no-ip.info/doom9.avi

the script I used:

avisource("doom9.avi")
assumetff()
tdeint(mode=1,type=3,link=0,mthreshl=7)
converttoyv12()
Motionprotectedfps(framerate*4)

you'll notice, that the yellow moving flag has wrong chroma

[Didée]

In the last line of MotionProtectedFPS( ), write

maskedmerge(p, n, last, U=3,V=3 )

?

[scharfis_brain]

tried that. didn't make things better

[mg262]

Leaving aside the stupid bug (thank you, Didée!) I'm not sure I have got the right end of the stick... is it this thing (circled in red)?

Edit: picture removed.

That's a frame from the source, i.e. from

directshowsource("doom9.avi")
assumetff()
tdeint(mode=1,type=3,link=0,mthreshl=7)
converttoyv12()

[I can try to sort out use of AVISource if that makes a difference.]

[scharfis_brain]

Directshowsource messes up the chroma here.

please use AVISource and ensure that the codec delivers the native YUY2 of the source video.

The source has no temporal chroma problem.

but the video after motion interpolation looks like in your screenshot

[mg262]

I'm really sorry I keep making so many silly script bugs... as well as U=3,V=3 change, there needs to be a FitY2UV applied to the mask before the MaskedMerge (because the mask is just a luma mask). I've updated the script. I hope that fixes it (it seems to at my end)...

[scharfis_brain]

this simple script

avisource("doom9.avi")
assumetff()
#securedeint(6,1,2)
bob()
converttoyv12()
Motionprotectedfps(50)

stops interpolating new frames after frame 376 and shows the first frame unti the end of the video.

EDIT: seems to be an 1000/1001 thing again, cause interpolating to framerate*n works without problems.

then I have a question, how to avoid those jumpy texture movements by one pixel? is there some sore of subpixel accuracy?

mode = default(mode, warp(bilinear,halfpixel))

didn't show any improvement

[mg262]

Safety check became over-eager... now fixed:

Motion, 6 December 05 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_06Dec05B.zip)
MotionProtectedFPS script (http://people.pwf.cam.ac.uk/mg262/posts/Motion/MotionProtectedFPS.avsi)

Thank you for all the testing, scharfis.

Edit: try warp(bilinear, bilinear). [The second argument is used in the case where the motion of the four corners is the same, so the block can just be copied. But, general bilinear translation is still in C, so for the moment it's pretty slow.] If that doesn't work, I will have to think much harder about how to change the algorithms; qpel estimation may be necessary, although it's a nightmare to implement.

Part of the problem I have is that my eye for non-animated material is not very good at all -- could you give me a frame number where this effect is particularly distracting?

[foxyshadis]

While editing the vfr page on avisyth, I've had a bit of a revelation: Wouldn't this be the perfect filter to replace decimate modes 1 & 3? Mode 1 translates 24 fps segments to 30 by stretching 4 frames into 5 by blending, and mode 3 compresses 30 fps into 24 the same way. If someone wants a constant framerate video, motion compensation would be the best compromise between visual stability and compression efficiency. =D

The problem is that something would have to developed to detect the segments. Of course it would be fairly easy to make a bare-bones conditional function to do it, it just wouldn't be as fast or as featureful as Decimate/TDecimate. Still, I think I should give it a shot.

[Mug Funky]

@ clouded: thanks heaps for this. i've got a head full of bad audio at the moment, but i'll give this a shot on a suitable source ASAP.

@ foxyshadis: it'd be awesome if you could pull this off (especially without artefacts), as detecting different framerates and treating them accordingly is something not available easily. and depending on how stable the framerate conversion is, it might even be able to replace speed-up to PAL (this is good if one wants to re-use audio).

[edit]

while waiting for a compile to finish, i gave this a try on some 60i stuff. it's awesome! and fast! thanks heaps :)

you don't have any problem with my using this in production now do you? this could be quite useful for DVD extras (they're very often 60i), and this thing runs almost as fast as vanilla kernelbob+convertfps+resize...

[scharfis_brain]

For Standards Conversion I suggest you to use this deinerlacing script:


function securedeint(clip i, int "th", int "l", int "mode")
{ th=default(th,6)
l = default(l,1)
mode=default(mode,2)

function staticmask(clip e, int th, int l)
{e0=e.motionmask(thy1=0,thy2=th,thc1=0,thc2=th,thSD=255,u=3,v=3).temporalsoften(6,1,1)
e1=e0.trim(1,0)
e2=e0.trim(2,0)
e3=e0.trim(3,0)
e4=e0.trim(4,0)
e5=e0.trim(5,0)
e6=e0.trim(6,0)
e7=e0.trim(7,0)
ea0=logic(e0,e1,"OR")
ea1=logic(e2,e3,"OR")
ea2=logic(e4,e5,"OR")
ea3=logic(e6,e7,"OR")
eb0=logic(ea0,ea1,"OR")
eb1=logic(ea2,ea3,"OR")
ec=logic(eb0,eb1,"OR")
(l==0) ? ec .duplicateframe(0).duplicateframe(0).duplicateframe(0) :\
(l==1) ? eb0.duplicateframe(0) :\
(l==2) ? ea0 : e0
}

i.converttoyv12(interlaced=true).separatefields()
e=selecteven().staticmask(th, l)
o=selectodd().staticmask(th, l)
mi=interleave(e,o)
m=mi.bob()
#interleave(mi.weave().tomsmocomp(-1,0,0), mi.doubleweave().selectodd().tomsmocomp(-1,0,0))
m=merge(m.trim(1,0),m,0.5).binarize(upper=false,u=3,v=3)


eedi = getparity(i) ? i.separatefields().eedi2(field=3) : i.separatefields().eedi2(field=2)
toms = interleave(i.tomsmocomp(-1,0,0), i.doubleweave().selectodd().tomsmocomp(-1,0,0))
krnl = i.leakkernelbob(order=(getparity(i) ? 1 : 0), threshold=0)
dumb = i.bob(0,1)

b = (mode==0) ? dumb : (mode==1) ? krnl : (mode==2) ? toms : eedi

d=i.doubleweave()

overlay(d,b,mask=m)
#interleave(min,m,mex)
getparity(i) ? assumetff() : assumebff()
}


I practically has no artifacts caused by the motionmask (holes)
And it is pretty fast.

Parameters:
th - deinterlacing threshold 6 seems to be a good value and is default

l - temporal radius of the combmask:
0 -> 16+2 fields are used for motion masking
1 -> 8+2 fields (default)
2 -> 4+2 fields
any other value -> 2+2 fields

mode - interpolation type
0 - bicubic
1 - kernel
2 - tomsmocomp's ELA (default)
3 - triticals EEDI2 (slooooow)

[mg262]

you don't have any problem with my using this in production now do you? this could be quite useful for DVD extras (they're very often 60i),Of course not -- it's free software, you can use it for what you like! If I can make specific changes to make it better for the task,* I'm happy to. But, for the love of God, make sure it's good enough for the task and doesn't cause excessive weird glitches before you use it...

*i.e. it is really useful to know what kind of footage it fails on, but that doesn't always translate to a concrete change I can make to the thing. Plus, some specific changes are a lot of work to implement or won't work well with the motion algorithm (e.g. different block sizes). But, time willing, I'll enjoy trying to get it to run better if you can think how, esp. if it's going to be used on "real" stuff. If you want, if it ever reaches the point where you can use it heavily, you could always talk your company into making a subscription to a children's charity (http://www.bridgestobelarus.org.uk/) I'm a trustee of... but you don't have to; I'll be pleased just to have it working well.

and this thing runs almost as fast as vanilla kernelbob+convertfps+resize... :cool: It's not running as fast as it can, yet, mainly because I left everything possible in script to let you all play with it. Currently it's about three times as slow as (convertto... + convertFPS + convertto...) -- 70 FPS vs 200 FPS here. If we reach a point where you're all happy with the script, I can start filterising bits... I'd like to hit 100 FPS. (NB BlendFPS is also pretty fast, so you can use it in script for additional artefact protection.)

I'm unlikely to make many changes to this for a fair while -- hitting the script-tweakable mark was the milestone I was aiming at. But there are a few things I have in mind further down the line. First is a couple of new compensation modes, which split the blocks into 4x4 blocks before compensation (I talked about this above)... should decrease artefacts a bit. Second is inbuilt GMC, to track zooms, etc better. Third if I have the energy is proper warping, not bilinear... though this will be very slow. Last is lots of speedups!

[Mug Funky]

hehe... charitable donations, huh? we're not above such things (though it'll probably be me that does the donating...).

i've tried the script on 1 clip so far, but it had a fair range to play with. it was one of the "group discussion" extras from Otogi Zoshi. it's ex-composite (ew), has fairly high contrast, some pans, some fast hand-gestures (these tend to be hell for mocomp, but your script handles them flawlessly), and some clips out of the actual show (ie animated).

your script seems to fall back on blending for the anime bits, and mocomp everything else. very nice - just the behaviour i like. i'll have to throw something more challenging at it :)

btw, i tried it on another source - a mix of 25p and 50i, with NTSC as it's final destination (yes, i make stuff for PAL land, but sometimes PAL land produces something that NTSC land might like), and it seems it blends progressive most of the time, but sometimes mocomps it. i imagine that's a matter of tweaking the script... of course it doesn't bother me at all, as IMHO progressive PAL should be blended to NTSC otherwise you get a disconcerting "jerky-slidy" look that probably nobody notices but me...

one thing i saw was that scenechanges weren't always being caught, but seeing as the transitions were over 1 field it was barely visible (certainly not catastrophic due to the robustness of the motion-algo you're using).

anyhoo, very nice stuff. maybe i'll send you a copy of the first disc i use this on (haven't used it on anything that's going out the door just yet).

[mg262]

Minor issue: the script as it currently stands upsamples the 1/64-size mask and then removes the padding, like this.
bilinearresize(width*8, height*8)
crop(0, 0, input.width(), input.height())
This is a bad idea on my part because the resamplers will use padding values to interpolate at the right hand and bottom edges of the frame. Unfortunately the obvious solution, namely
bilinearresize(input.width(), input.height(),
\0, 0, input.width()/8, input.height()/8)
doesn't solve things. From testing and from glancing at the resampling code, I think that it is accessing values from the original frame base outside the cropped region. (So, I think the manual may be wrong when it states that this is exactly equivalent to crop + resize).

There isn't a really satisfactory solution to this until Y8 is available [and I'm very much coding everything with Y8 in mind]. For bilinear resampling, I can work around it by copying the padding values from the nearest pixel values... and I will release a version that does that soon. That will be close enough to the right answer for bicubic, etc to be indistinguishable -- actually, the way the masks are used, its unlikely that the current method is indistinguishable from the correct one, but it should of course be fixed anyway!

Sorry about this...

Proper replie(s) also coming soon.

[mg262]

Padding copied from border pixels to work around crop/resize issue (see last post):

Motion, 9 December 05 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_09Dec05.zip)
MotionProtectedFPS script (http://people.pwf.cam.ac.uk/mg262/posts/Motion/MotionProtectedFPS.avsi) (unchanged)
Edit:Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Edit: Scene changes aren't being dealt with at all... IMO detecting + dealing with them properly takes much more work than a AverageLumaDifference, and sorting it out will take a couple of weeks at some point. (Although I'll do it in such a way that I can use it for all my filters at once.)

But in the meantime, you need something to work with... how about this: a postprocessing filter that detects scene breaks using a dumb-as-hell method (AverageLumaDifference), and duplicates frames to avoid interpolation-across-scene-breaks? So it would look something like this:

interpolated = MotionProtectedFPS(original, FPS, ...)
FixUpFPSA(original, FPS, interpolated, lumathreshold)

? (I assume duplicating frames is preferable to blending across scene breaks?)

[Mug Funky]

that's an interesting idea. i think kassandro has a filter that does that (improvesceneswitch?)

though i can imagine disturbing jerkiness on flashes and quick pans using this method. i can settle for blended scenechanges though - they'll only last 1 field anyway :)... even with the slight warping that the current script gives, it is nearly impossible to actually see at playback speed on an interlaced screen.

[scharfis_brain]

I have a much better Idea for stable scene detection!

Initial step:
- create a motionmask
motionmask(thy1=noise,thy2=noise+1,thc1=noise,thc2=noise+1,thscd=255,u=3,v=3)
- noise is a variable to pass noisy clips
- make a logical AND link between two temporal adjacent Motionmasks
logic(last,last.trim(1,0),"AND",u=3,v=3)

this ANDed motionmask will now show completely filled on scene changes, while normal motion just shows the contoures of moving objects.

Now the decision step:
- calculate the averageluma of that ANDed motionmask.
it has to be factor plus [offset[/i] times greater than the avglume of two predecessing and two following frames.

factor needs to be about 5 to 20
offset about 20.

the offset avoids, that in low motion scenes nearly every frame gets detected as scene change. and the factor defines the minium factor in luma between normal and scene changed motionmask.

ahhh, I finally found my code snipplet:
function scd(clip i, int "factor", int "offset")
{ global fac=default(factor,5)
global off=default(offset,25)

global msk0=i.duplicateframe(0).reduceby2().reduceby2().motionmask(thsd=255,y=3,u=1,v=1)
global msk1=msk0.trim(1,0)
global msk2=msk0.trim(2,0)

global blk=i #blankclip(i)
global wht=i.subtitle("scenechange") #blk.invert()

f1=scriptclip(i,"sc")
f2=f1.frameevaluate("sc=( (a2 > (a1*fac+off)) && (a2 > (a3*fac+off)) ) ? wht : blk")
f3=f2.frameevaluate("a3=averageluma(msk2)")
f4=f3.frameevaluate("a2=averageluma(msk1)")
f4.frameevaluate("a1=averageluma(msk0)")
}

[Mug Funky]

yo. i'm getting crashes, but haven't isolated the problem to anything specific yet.

just running mpeg2source and motionprotectedfps(50) to bump a 25p source up to 50p, then playing in mplayer to gauge CPU usage (awesome... realtime and averaged 60% CPU use for PAL DVB source).

only problem is it crashed after about a minute of playback. memory use was a constant 170 megs (about the same in VM too).

i'm running an athlon 64 3500+ venice, at a very cruisy 2200 MHz, with a gig of DDR and winXP (SP2, not x64 unfortunately).

performance is awesome other than that though. really fast and only artefacts in certain cases (i might clip some stuff out where artefacts occur).

[edit]

this could be my source. it crashes _around_ the 7000 frame mark (that's 3500 source frames). i'll take a look at it on another clip.

[mg262]

Damn damn damn. I tried that on a longer clip than I had previously tested + it crashed at the 74000 frame mark. Really sorry about this; CPU is largely tied up with a long encode but I will track it down ASAP.

EDit: rounnding error of 1/16 of a motiion vector... causes read outside frame in some unusual circumstances. I'll fix it tomor row when i have enough cpu to have speech-rec running again.

[mg262]

Motion, 12 December 05 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_12Dec05.zip)
MotionProtectedFPS script (http://people.pwf.cam.ac.uk/mg262/posts/Motion/MotionProtectedFPS.avsi)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

I think this should have fixed it -- if not, could you send me a clip on which it fails? FWIW, the motion algorithm converge is pretty fast so if you have an error you can generally trim to about 200 input-side frames before an error without making it disappear.

Also: for reasons that escape me I had set warp to be the same as warpfast... I've fixed it, but also updated the MotionProtectedFPS script to use warpfast as default so that the behaviour is unchanged. If you use the old script with the new DLL, you may see a slowdown (but perhaps better quality... or it may look more blurry due to subpixel compensation). There shouldn't be slowdowns for any other reason... if you see them, please tell me.

scharfis_brain: that sounds pretty robust, but also pretty expensive? I will test it later when I have a bit more free time...

Edit: I've been meaning to say that while FindMotion more or less requires frames to be accessed in linear order, none of the other functions do (so long as they don't implicitly called FindMotion). So, you should be able to request more than one frame from them in parallel using tsp's SetMTMode (in any mode). Unfortunately I only have a single-threaded single-processor box so I can't test this... but I would be very interested in the results if anyone does.

[Fizick]

I've tried implementing a MVFlowFPS-like method, and I was very impressed with the results. IIRC Fizick spent a lot of time and effort looking at artefact protection, and it shows. (Thank you, Fizick!)
...


If you use MVTools source (GPL), your source must be availible too.

[mg262]

If you use MVTools source (GPL)I certainly haven't; not one line of it. What I used was your idea of blending two occlusion masks to create another mask... and a great idea it is, too. As it happens, it's actually implemented in the MotionProtectedFPS script rather than in Motion, so that the scripters can play with it.

FWIW, your implementation scales up the two small masks (you call them something like MaskB and MaskF) and then averages the results... you could probably manage it a little faster by averaging them first and then scaling up the result, (which is what the MotionProtectedFPS script does, by calling CombineFPS and then BilinearResize).

As before, you are more than welcome to the latest source of Motion to read and to pick out ideas from, if you would like?

Incidentally, Merry Christmas all!

[krieger2005]

i have a question how to use this thing because i have here a problem. I use this script:
encode=false
c=last
vf1b=FindMotion(c, from=previous, reset= encode ? 0 : 20, initialise=4)
cf1X=Compensate(c, warp,source=previous, motion=vf1b)
cf1b=cf1X.trim(1,0)


now the problem: It looks, that the compensated Clip cf1X does not fit to the frame of the clip c but the clip cf1b does. But cf1b have the same artifacts like c, so the compensation is not usefull.

So the general problem is, that the "normal compensated" clip seems to be one frame before the original.

Where is the problem?

EDIT: OK, i now remove for the call of the clip cf1X the source=previous and now cf1X give me the right compensated frame, BUT: it has the same artifacts as the original frame (but the frames before and after does not have this artifact). I want to remove these artifacts through the motion-compensation...

[mg262]

I can't reproduce the problem... it may be that the true motion estimation algorithm is not working with some kind of source. (Although removing source = previous shouldn't make any difference. I completely rewrote the relevant code at one point, and although I regression tested it, it may be that some glitch slipped through...)

Would you mind uploading a clip on which both problems occur? I'm particularly keen to see a section where removing source=previous makes a difference. The complete script would also be useful.

The motion algorithm converges pretty fast, so you should probably find that trimming out 50 or 100 frames before the problem frame will give you a clip that reproduces the issue. Although any size download is fine by me...

[krieger2005]

I uploaded a short snip to the net. You can get it here (http://rapidshare.de/files/9847113/tst.avi.html)... Sorry for the length but a have only dial-in-connection.
Ok, here the exactly script i use. I used to return closures. Enable one of them...
LoadPlugIn("..\AVS\X\Motion_12Dec05.dll")

avisource("tst.avi")

encode=false

c=last

vf1b=FindMotion(c, from=previous, reset= encode ? 0 : 20, initialise=4)
cf1b=Compensate(c, warp,source=previous, motion=vf1b)

# return 1
return interleave(cf1b.trim(1,0).subtitle("first"),c,cf1b)

# return 2
return interleave(c,cf1b.trim(1,0))

when you enable "return 1" you can see first the trimmed frame, then the original clip and then the original-compensated. Please look at frame 261. Between the trimmed and original clip there is only minimal differences. But the original compensated jumps and looks like the frame before...

Now enable "return 2". Look for example at frame 152. There are much spots. These spots i want remove with despot-filter... But they are also in the compensation. Even better you can see it on frame 178 and so on...

[mg262]

The truemotion algorithm is becoming confused because the foreground object is very fast moving around frame 84 of the input (~frame 261 of your script with return 1). Unfortunately, this means that the compensation isn't working properly. This is a known issue with this kind of recursive algorithm (where the motion found for each frame depends on the motion found for the previous frame), and I've seen a suggested solution, but when I tested it it was pretty useless. I am sorry not to be more helpful. EDIT: Try adding iterate = 4 to FindMotion... results look much better.

For what it's worth, Trim(1, 0) is never necessary -- the compensated clip is just cf1b. In other words, frame 30 of cf1b *should* look like frame 30 of c, when all is working well.

When I try your script with return 1, and look at frames
165 (marked "first")
166
167

then frame 166 is much closer to frame 167 than either is to frame 165, as it should be. It easier to see the behaviour of the script using this return:

p=c.trim(1, 1)+c
interleave(p.subtitle("1. Previous"),cf1b.subtitle("2. Compensated"),c.subtitle("3. Current"))

So here 1 and 3 are consecutive source frames. 2. is made up of bits of 1., but *should* look like 3.
When estimation/compensation isn't working well, 2. will probably look like 1. For the purposes you are using this for, namely preprocessing for despot, that's not a catastrophe... the idea behind fast motion compensation with temporal filters is that while it is not always perfect, it should nearly always be better than using the filter without MC. In other words, we are replacing 1. with 2., and 2. is nearly always better than 1.

I tried to look for the other problem using this addition to your script:

other =Compensate(c, warp, motion=vf1b)
subtract(cf1b,other)
Levels(128 - 15, 1, 128 + 15, 16, 235)

but I couldn't reproduce the issue where omitting source = previous made a difference.

[milh31]

Great work mg262.

[krieger2005]

The truemotion algorithm is becoming confused because the foreground object is very fast moving around frame 84 of the input (~frame 261 of your script with return 1). Unfortunately, this means that the compensation isn't working properly.
But the predicted Vectors look ok? But... humen eyes ;)



I am sorry not to be more helpful.
Is it possible to add/get an option or a mask, which show how good parts of the frames could be compensated by the filter?
EDIT: Try adding iterate = 4 to FindMotion... results look much better.
Thanks, i will try this.

For the purposes you are using this for, namely preprocessing for despot, that's not a catastrophe... the idea behind fast motion compensation with temporal filters is that while it is not always perfect, it should nearly always be better than using the filter without MC. Despotting in general (not only with despot), for my optinion, is dangerous. So i would say, that when one don't know, that a filter can't compensate the frame good enough it is a catastrophe... But this is surely not the problem of your plugin. It make what it can...

but I couldn't reproduce the issue where omitting source = previous made a difference.
Sorry.. this was a mistake by myself.

Thanks for your help

[Backwoods]

In the motion.htm, you have MotionCompensatedFPS() listed as a function. Do you mean MotionProtectedFPS()?

[Gargantula]

I have the latest (I think) Motion_12Dec05.dll file installed but, I get the error that Avisynth doesn't find a function named MotionProtectedFPS. Is this the right file? Do I need another file?

Where can I find the latest version of the file? Maybe the first post should be edited to reflect the latest version.

[foxyshadis]

The first link in the first post is: http://forum.doom9.org/showthread.php?p=747288#post747288
It's a script you have to put into your avisynth plugins folder.

[Gargantula]

Got it. Thanks! How come the file for download is an older version? I got the Motion_12Dec05.dll file from a pack.

[Terka]

is this filter still in development or finished?

[Adub]

I am pretty sure that MVFlowFPS1/2 have surpassed this.

[Fizick]

"Motion" plugin is greatly opimized by speed.
IMO, it is mostly for realtime processing (not coding).
The algo is similar to used in Philps TV (may be not latest).
It is posiible to continue its development if you can (source code is open).
(me not)

[cweb]

The first link in the first post is: http://forum.doom9.org/showthread.php?p=747288#post747288
It's a script you have to put into your avisynth plugins folder.
The files are no longer on the server... can they be uploaded somewhere please..

[NormanBates]

@cweb:
Hallo,
I found some files on my hdd and I have uploaded the files Motion_12Dec05.dll and MotionProtectedFPS.avsi for you:
http://rapidshare.com/files/91877862/Motion_12Dec05.zip.html

[cweb]

@cweb:
Hallo,
I found some files on my hdd and I have uploaded the files Motion_12Dec05.dll and MotionProtectedFPS.avsi for you:
http://rapidshare.com/files/91877862/Motion_12Dec05.zip.html

thanks!

[Terka]

Could this plugin be optimized for multi-core processors? (for realtime processing in ffdshow)

[Adub]

Have you tried using MT?

[Terka]

i was not able to run it under ffdshow :(

[scharfis_brain]

what was the problem?
I am running it right now under ffdshow.

I use the following setup:

loadplugin("f:\x\Motion_12Dec05.dll")
fps=100 #last.framerate()*4
last.CombineFPS(fps, last. MotionFPS(fps, warpfast, move), last.MotionFPS(fps, source = next, warpfast, move))


[x] Buffer back/ahead: [10][10]

it looks really awesome on 100Hz driven CRT.
also it is nice on a 60 (or 75) Hz driven LCD (you have to change fps=.. for this).

[Terka]

scharfi, thanks for reply.
i was already able to run it. The new thing i didnot know is
[x] Buffer back/ahead: [10][10]
used
loadplugin("c:\ProgramFiles\AviSynth\plugins\Motion_12Dec05.dll")
motionprotectedfps(50)
or
MotionProtectedFPS(50, iterate = 4) #higher quality

[scharfis_brain]

if the source resolution is small enough, (like 640x360) I can use motionprotectedfps(fps,iterate=4), too.

but generally I use combinefps(...). So it is able to interpolate even 720p30 contents to 100Hz

[Selur]

anyone got a link to the latest Motion.dll (Motion_12Dec05.dll I guess)+ Documentation? -> sorry found it over at: warpenterprises