Motion filter : +1st MC FPS-change script for real usage

[scharfis_brain]

btw.: for standards conversion of interlaced SDTV (50i and 60i) there needs to be a deinterlacer applied.

mainly for those reasons:
1) scaling the fields between 480 and 576 lines while preserving static areas
2) doing a good motion vector search
3) getting material to replace MoComp mismatches
4) make your live easier

[castellandw]

Originally Posted by mg262

castellandw,
I honestly don't know. I'm not confident enough to call this one. If you want my gut feeling... I have a tendency to distrust maths-based methods unless I can see (theoretically or empirically) that they model the problem domain well. (IMO, there is a visible tendency for academics who are not mathematicians per se to be impressed by maths just because it looks hard.) A priori, you can also see that phase correlation will tend to mess up on small objects.

Well, it seems clear that it'll mess up on small objects, which as I understand is why phase correlation is only limited to large object movements on regular standards converters. You know, I like to give Fizick's MVFlow function a try since it works on pixels of a frame instead of blocks of a frame, but Fizick says that doing motion compensation by pixels can create very strange deformed pictures, so for now, I'll stick to anything involving blocks of frames instead of pixels.

[Mug Funky]

hmm. motion-compensating a new field for deinterlacing doesn't _necessarily_ need a bob to start with - whatever is performing the motion compensation simply needs to be aware of the half-pixel shift between fields and take that into account. MVtools doesn't do this (why should it... it's not a deinterlacer after all), so we have to feed it with a smart bobbed clip where the half-pixel shifts are minimised as much as possible. and as scharfi says, it makes things easier because you can just re-use the smartbobbed clip to fill areas where motion compensation has produced artefacts.

[edit]

@ castellandw:

mvflow still works with blocks, but it moves each pixel when compensating. it just interpolates the vector field made from the block-matching process. this can really help with angular motion and even rotation though, so it's still got an edge over moving full blocks. though for interpolation, i think forward+backward with OBMC and a blocksize of 4 would be nearly perfect as well.

[scharfis_brain]

when compensating fields directly one doesn't achieve a goodsuppirxel accuracy.

When I use an ELA based smart bob instead, I have much better subpixel precision due to the ELA interpolation. (lesser jaggieness)

[mg262]

1) scaling the fields between 480 and 576 lines while preserving static areas
2) doing a good motion vector search
3) getting material to replace MoComp mismatches
4) make your live easier

2. and 3. I can certainly accept. I don't understand the "vectors=separatefields().analyse(interlaced=true)/compensated=somesmartbob().mvcompensate(vectors)" request in the context of 2, though... I would have thought you would want to do it the other way round, i.e. find the motion vectors on the bobbed clip but compensate the un-bobbed clip.

As for 1.: Once the motion compensation is in play, you simply have a vertical slice with (approx) known values at irregular intervals, and you want the values at particular points... AFAICS, the number and spacing of those points doesn't matter... so I think you can do the resampling at the same time as interpolation. (And use your favourite resampling method for the interpolation.)

4. Yours or mine ?

Edit: I don't know how much you will be able to rely on the subpixel accuracy of the true motion algorithm, even on progressive content, anyway... if it looks like being a problem, do tell me because there are quality/speed trade-offs that can be made.

[mg262]

Motion, 3 November 2005

Most of the changes are under the hood to speed things up. Half-pixel accuracy is now in assembly, and works about as fast as the integer version did before (~ 200 FPS). Half-pixel and assembly are now used by default. Bidirectional estimation/compensation is available like this:

FindMotion(from = previous) or FindMotion(from = next)
Compensate(source = previous) or Compensate(source = next)

Both default to previous. So standard compensation (moving parts of frame n-1 to replicate frame n) is still done like this:

#AVISource...
Compensate(FindMotion())

And reverse compensation (moving parts of frame n+1 to replicate frame n) is done like this:

#AVISource...
Compensate(FindMotion(from = next), source = next)

(If the argument names are confusing, suggestions for better alternatives would be welcome.)

Here is an example interleave script for motion compensated denoising:

#AVISource...
Interleave(\
Compensate(FindMotion(from = previous,initialise = 4), source = previous),\
last,\
Compensate(FindMotion(from = next,initialise = 4), source = next))
#temporal smoother, radius 1
SelectEvery(3, 1)

Remember to set the reset parameter in both FindMotion calls if you want to be able to seek; I typically use reset = 50, but lower values will give you faster seeking.

[castellandw]

mg262, it would be nice if you packaged a help file next time around along with the dll for all the functions in your DLL because it's gonna start to get messy to look through this long bunch of posts in this message thread.

[Mug Funky]

yeeee! will try it immediately!

[edit]

backward prediction seems to be compensating the wrong frame? try:

back=compensate(last.deleteframe(0).findmotion(reset=100,from=next))

subtract(back,last)

[mg262]

@castellandw,

I thought about recappping all the function arguments, but realised there weren't any important arguments that weren't mentioned in the above post. So:

FindMotion(clip, ...)
from = previous/next
int reset (default 0)
int initialise (default 1)*

Motion estimation usually relies on estimated motion vectors for the previous frame, but if e.g. reset = 50 then motion vectors are calculated from scratch every 50 frames. Increasing initialise increases the accuracy of vectors calculated from scratch.

*There are more arguments, but they are IMO no longer useful except possibly for debugging... just there to avoid breaking scripts for the moment.

Compensate(clip, clip motion, [source = previous/next])

DrawMotion(clip, clip motion)

Example scripts for the first two are above; view motion like this:

#source
DrawMotion(FindMotion())

#source
DrawMotion(FindMotion(from = next))

Edit: I have put together a filter recap, below, and I will update and relink with each substantial change. But I don't want to suggest that reading this is enough to jump into the discussion... the whole point of starting this thread was to obtain feedback to direct the development (and I have been given plenty of expert feedback -- thanks, guys!)

[Didée]

Regarding artefacts in compensated frames:

>> Repair( compensated, reference, 4 )

is not the solution. But it's part of it.

[mg262]

@Mug Funky,

I'm a bit confused... your script produces something that shows a a big difference between the two clips, but on the other hand I can't tell what it is meant to do. You can't AFAIK use backwards-estimated information for forward compensation, for the following reason: there may be blocks in frame n+1 that don't map to any block in frame n, so it's not clear what you should put in for those blocks. To make that more precise:

findmotion(reset=100,from=next)

will find, for each block in frame 30, the block in frame 31 which is most similar to it (give or take mumbling about true motion).Now

compensate(findmotion(reset=100,from=next), source = next)

will replace each block in frame 30 with the block from frame 31 which is most similar to it.

But, given the information from findmotion(reset=100,from=next), there is no way to replace blocks in frame 31 using blocks from frame 30... because we just don't have the right information. Given a block in the frame 31, we have no easy way to calculate which block from frame 30 to use.

Does that answer your question? Or have I got the wrong end of the stick completely?

Edit: there was a silly bug in the initialisation code... you won't see any difference in the results, but please grab this again anyway:
Motion, 3 November 2005 (upload at 14:06)

[Mug Funky]

aaah, i missed the "source=next" thing. sorry. works a treat now

thanks for your work. this is looking great.

[edit]

btw, how do i shot hpel? i can't seem to see it in the options posted here (am i blind?)

[mg262]

It's on by default. You can use FindMotion(subpixel = false) to turn it off... but it's fast enough that I can't think of much reason to turn it off? (Similarly, if for some reason you want to switch off the assembly you can use useassembly = false.)

Edit: This is not documentation, in the sense that reading it isn't a proper substitute for the discussion in this thread, but it is a recap of all the filter options to save you having to look back:

Basic Usage and Filter Options
http://people.pwf.cam.ac.uk/mg262/po...on%20recap.txt

I agree this thread has become too long to digest... + looking back, large parts no longer apply, e.g. thoughts on implementing backwards compensation. So if no one has any objections, I may start a new thread at some point when there is a new version, and repeat the main requests, etc?

Incidentally, to let you write standards conversion scripts (your way), what do I need to implement? Just a progressive ConvertFPS function?

[castellandw]

Originally Posted by mg262

Incidentally, to let you write standards conversion scripts (your way), what do I need to implement? Just a progressive ConvertFPS function?

Were you talking to me, mg262? If you were talking to me, a motion compensated convertfps using your motion compensation engine in your dll would be nice for a start because I probably think it's not such a good idea to do motion compensation and then frame rate conversion separately. Do you think motion compensation and convert frame rate separately might cause jerky motion? By the way, mg262, have you tried Fizick's Block Overlap plugin yet?

[scharfis_brain]

@mg262: Hmm. I seem to have contradiced myself. (I only had a few minutes before I had to leave for school)

I have the following in mind, because it 'might' be more reliable to find motion vectors:
search the motion on the separated fields, but create (stable for static areas) a vector stream, that can be applied to the smart-bobbed clip.
But only implement it, if it gives benefits over the current approach (smart-bobbing and then searching for motion)

fpsconversion itself only needs to be progressive. reinterlacing can be done easily afterwards.

[castellandw]

Originally Posted by scharfis_brain

I have the following in mind, because it 'might' be more reliable to find motion vectors:
search the motion on the separated fields, but create (stable for static areas) a vector stream, that can be applied to the smart-bobbed clip.
But only implement it, if it gives benefits over the current approach (smart-bobbing and then searching for motion)

@scharfi, wasn't this already discussed when you mentioned this or is it something else different:

source("blah.xxx")
assume?ff()
vectors=separatefields().analyse(interlaced=true)
compensated=somesmartbob().mvcompensate(vectors)

[scharfis_brain]

yeah it was, but mg262 said he didn't understand it ?!?

[mg262]

castellandw,

Originally Posted by castellandw

have you tried Fizick's Block Overlap plugin yet

That is the third time you have said that in the last four days. I sent you an thorough reply when you first PMed me, and it hasn't changed:

Thank you for reminding me about that... it had slipped my mind. At the moment, it looks like one of the niches Motion will fill is fast motion compensation, which means that the factor of two slowdown from BlockOverlap is quite serious. Other thing is, something like this is mainly for users to try rather than developers -- the ability to mix-and-match plug-ins without having to know anything about the code is one of the great strengths of AVISynth! Of course, that doesn't preclude me from trying it and recommending it... but for the moment my main focus is on prevention rather than cure, i.e. trying to alter my plug-in so it doesn't produce as many blocks (because that's something users can't do).

Originally Posted by castellandw

I probably think it's not such a good idea to do motion compensation and then frame rate conversion separately. Do you think motion compensation and convert frame rate separately might cause jerky motion?

This makes no sense. The notion of "motion compensation and then frame rate conversion separately" is meaningless.

I don't mean to be rude... but I think you need to go and absorb a lot more about a) the nature and strengths of AVISynth and b) the uses of motion compensation (which is not an end in itself, but a means to other, specific, ends). I'm not saying this based on a single comment, but rather on the general nature of your posts; there is an implied context behind messages in this thread which you are often missing. Also, second-hand comments (aka argument from authority) are not terribly useful, not because they are right/wrong, but because we need the reasoning, intuition and often technical details that underlie them. Please don't take this the wrong way... I don't mean to have a go at you. But I do think you need to learn to walk before you can run.

scharfis_brain,
Your script was very clear and I understood it; rather, I should have said that I didn't understand point 2. (higher-quality analysis possible after bobbing) coming after the first request (analyse and then bob). The situation is now clear. I think that, as you imply, we will have to implement both and see which is better.

Incidentally, it seems perfectly legal to upsample any clip (interlaced or progressive) to improve the quality of the motion vectors found. (This is just Mug Funky's supersampling again.)

[castellandw]

Originally Posted by mg262

castellandw,

That is the third time you have said that in the last four days. I sent you an thorough reply when you first PMed me, and it hasn't changed:

Sorry, I thought when you said it didn't preclude you from trying it that you meant you might try it since the Block Overlap plugin produces fewer blocks with motion compensation. Anyway, as I mentioned, I tried out the Block Overlap plugin and seen better results with it on motion compensation, and I was just wondering if anyone tried it out to see if they got better results as well. OK, if it's more of something for users to try out still, then no problem.

Originally Posted by mg262

This makes no sense. The notion of "motion compensation and then frame rate conversion separately" is meaningless.

I think I should try to rephrase it. What I meant here is on applying motion compensation on the video(whether it's applying MVTools or the Motion DLL) and then using a frame rate conversion function (ConvertFPS and ChangeFPS) for standards conversion. I understand that smart bob deinterlacers are used best before ChangeFPS(which drops and inserts frames) or ConvertFPS(which by default blends frames which causes some motion blur) generally on Avisynth in terms of standards conversion. However, if motion compensation is applied, then would dropping or inserting frames or motion blurring by the likes of ConvertFPS() or ChangeFPS() have any worse effect on the motion compensated video because if not, then I'd be happy to just use ConvertFPS() or ChangeFPS()? So basically, I'm just wondering if both ConvertFPS() or ChangeFPS(), without recognizing the case of motion compensated video will worsen the motion information on a motion compensated video?

Originally Posted by mg262

I don't mean to be rude... but I think you need to go and absorb a lot more about a) the nature and strengths of AVISynth and b) the uses of motion compensation (which is not an end in itself, but a means to other, specific, ends). I'm not saying this based on a single comment, but rather on the general nature of your posts; there is an implied context behind messages in this thread which you are often missing. Also, second-hand comments (aka argument from authority) are not terribly useful, not because they are right/wrong, but because we need the reasoning, intuition and often technical details that underlie them. Please don't take this the wrong way... I don't mean to have a go at you. But I do think you need to learn to walk before you can run.

Keep in mind that I mentioned before that you guys have much more experience in this than me, and so I'm trying to learn from you guys as best as I can as well. I'm not trying to hinder the discussion in this thread, especially by attempting to be an authority on various things. I can read up and absorb on a lot of things, but it's also good to ask people to get their understanding as well. In the case of standards conversion especially by the Doctor Who restoration team who mention to me as far as I remember that frame rate is converted while the motion compensation is taking place in high-end standards converters, that's why I mention that using MVTools or the Motion DLL and then use ConvertFPS() and ChangeFPS() is probably not a good idea, but I wanted to ask for sure if the motion blurring on ConvertFPS() or the inserting and dropping of frames on ChangeFPS() won't matter? However, since it seems that I won't really get the implied context, to relieve the only need for anyone to respond which involves the point of possible argument, I'll actually just end it before beginning it as best as I can without being insulting by saying that not responding to my question is perfectly fine to be your guys' call. Without being condescending, if I get no response, which usually is very likely for me, then I'm happy to stick to asking in the Avisynth usage forum or just keep silent while looking and trying out stuff here, which I'm very used to doing within my life because I don't want to get in the way interfering and hindering other people's discussions here.

[Mug Funky]

motion estimation and interpolation is meant as a replacement for convertfps. running convertfps after compensation (compensating what?) would be counter-productive. much better to create new frames using motion interpolation than the blending in convertfps (which for lack of a better word could be called temporal interpolation, but that's an ambiguous term).

i'm totally looking forward to fast motion-compensated standards conversion in avisynth - right now i'm doing a straight smart-bob > blend > resize > re-interlace. this is good as most regular standards converters and i'm happy with it, but if there's a way to improve something i'm always up for it.