Motion filter : +1st MC FPS-change script for real usage

[mg262]

I've added a bit more to the documentation and converted to HTML:

http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html

[castellandw]

motion estimation and interpolation is meant as a replacement for convertfps. running convertfps after compensation (compensating what?) would be counter-productive. much better to create new frames using motion interpolation than the blending in convertfps (which for lack of a better word could be called temporal interpolation, but that's an ambiguous term).

I totally understand and agree that motion estimation and interpolation for new frames is better than blending them in convertfps. That's why I was talking about applying motion compensation and with the only usual options for frame rate conversion in Avisynth is separately using conversion functions like ConvertFPS() or ChangeFPS() in AviSynth, that's why I thought it was a good idea to have a special FPS conversion function which applies motion compensation while it changes the number of frames per second.. However, if I'm missing any context there because of what I'm saying doesn't make any sense, then I won't bother discussing it.


i'm totally looking forward to fast motion-compensated standards conversion in avisynth - right now i'm doing a straight smart-bob > blend > resize > re-interlace. this is good as most regular standards converters and i'm happy with it, but if there's a way to improve something i'm always up for it.

I'm not sure how fast you want, but scharfi-brain's mvbob (or using older versions of MVTools to use MVFPS) is the closest we seem to have to proper motion-compensated standards conversion in AviSynth right now. By using MVTools so far, I seem to have gotten much better picture and better motion although with black spots at the edges of the pictures, which as I mentioned, I managed to improve with the Block Overlap Plugin. Just using smart-bob deinterlacers alone and playing around finding the best threshold doesn't seem to minimise combing artifacts and wobbly static areas as well as any motion compensation I've used.

[mg262]

Mug Funky,

I would particularly appreciate your opinion on this when you have a moment:

Motion, 4 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_04Nov05.dll)
Contains the function SimpleWarpCompensate, usage exactly as Compensate. (I've given it an alias SWC for convenience. Simple = Bilinear, in case you're wondering.)

It's not a proper release of the function per se; that's because I've taken several shortcuts to keep the speed up, and whether or not I keep them depends on what the feedback is. On the bright side, the speed is ~150-160 FPS.

(Please ignore the 4 pixel black border for the moment. Fixing that is a simple copy/paste job... but copy/paste jobs are best left until you have finalised the code you are copying. If I trusted MSVC's optimiser, this would all be a lot easier...)

Here is the documentation link again...
http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html

[Mug Funky]

just had a look then and it's pretty cool. it very much improves the results.

occlusions look weeeiiird now :) kinda like the behaviour of light around a black hole...

i'll have to play with it a bit more though.

oh, one other thing - on very slight motion blocks are still sometimes visible. try look at a person moving very slightly to the left and you might see it.

[edit] i'm just about to try it supersampled.

[edit 2] btw, stripes on clothing really mess this thing up :). it wasn't so visible with simple block moving i guess because if the blocks look similar enough they'll compensate well, but if the vectors aren't correct, a warped compensation will really show it up. this could be a good way to tune the motion estimation, because a warp is more visible than a mis-moved block.

good work!

[edit 3]

btw, how many previous frames worth of vectors does Motion need? could seeking be improved by just having it find (and cache?) motion on the last x number of frames rather than resetting every x frames? i don't know enough about this method to know if i'm talking crap :). i'm guessing there'd be some kind of convergence after a certain number of frames, so to keep results consistent (so a random seek produces the same vectors as playing from the beginning) there could be an internal cutoff after a certain number of frames where previous data isn't considered. that may slow it down though. not sure.

[edit 4]
Just using smart-bob deinterlacers alone and playing around finding the best threshold doesn't seem to minimise combing artifacts and wobbly static areas as well as any motion compensation I've used.

auto-noise thresholding helps, but i've not been able to do that well in avisynth yet (got very close). the reason static areas wibble so is usually noise related, or simply because static areas are too sharp (it's good practice to apply a 1 pixel vertical filter if making materials for TV, but not many people do it, alas, so the extra sharpness messes with most deinterlacers). but done right, an "auto-thresholded" kernel-based deinterlace is extremely good, and suitable as input for a motion-compensated deinterlace/standards conversion (if you look close enough at some broadcast stuff, it is clearly done this way - smart bob then mocomp, with a fallback on blending if the mocomp fails. you have to look reeeally closely to tell it's not originally PAL).

[mg262]

Thanks for all the feedback :) I know what you mean about occlusions... weird ripply effect... but it bothers me a lot less on playback than blocks do.

Here's where I cut corners: no chroma warping and no bilinear interpolation. Chroma warping isn't too expensive, but I couldn't see chroma blocks at all. (... so I wanted to know whether they would jump out at you or scharfis or someone else.) Bilinear interpolation done on a per pixel basis gets pretty expensive, and I reckon the distortion due to bilinear mesh warping has a much bigger effect.

on very slight motion blocks are still sometimes visibleI think I can see it of my source but I'm not sure I can. Two possibilities. First is chroma blocks. (Could you greyscale and see if it's still there?) Second is harder to describe... imagine the (luma) picture as a height map. Moving blocks resulted in sudden luma jumps -- discontinuities -- like you had just found a cliff in the middle of the road. Bilinear mesh warping gets rid of the jumps, but the gradient can still jump -- like there's a line in the road and on the other side it suddenly slopes up at 45°. Better mesh warping is really expensive... though it may be worth it.

how many previous frames worth of vectors does Motion need? Just one frame... but it's recursive. Generating motion data for frame 130 needs motion data for frame 129. But motion data for frame 129 needs data for frame 128. And motion data for frame 128 needs data for frame 127. So without some sort of break, you end up going all the way to frame 0.could seeking be improved by just having it find (and cache?) motion on the last x number of frames rather than resetting every x frames?Definitely, over short ranges. At the moment it just caches the last frame requested. (And it doesn't access the AVISynth cache, because filters can't access the cache for their own output, at least without using kassandro's AVSRecursion. Caching its own output is not necessarily a bad thing, because motion frames are tiny compared to picture frames.) If e.g. I cache the last reset frames computed, then you'll be able to single step backwards, which is maybe the point of the request?

btw, stripes on clothing really mess this thing upMmm... I've got thin stripes on a hat and they go through okay... but they're really thin. Any ideas on how to improve it on your stuff? [At some point I'll add global motion compensation, which may help a bit.]

I'll switch on chroma warping and bilinear interpolation (or add options) and drop a new version in a bit.

BTW, auto-noise thresholding sounds pretty cool... I have thought about it in other contexts (done one way, it comes straight out of the maths for ReverseBlend)... would be interesting to hear more about it.

[mg262]

Motion, 5 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_05Nov05.dll)

SimpleWarpCompensate now has an option bilinear = true/false (default false). Leaving it off will mean it only uses existing pixel values; switching it on will cause it to average pixel values (subpixel accuracy). I think I can see a small difference, but my eyes aren't that good (especially for live-action).

Example use:
simplewarpcompensate(findmotion(initialise = 4, reset = 30), bilinear = true)

Switching it on is pretty expensive... the overall speed drops to about 100 FPS. Even if it is left off there's a small speed hit, 10 FPS or so, due to the check as to whether it's on... I can get round this by duplicating lots of code, but if none of you can see the difference then I may just scrap it.

Here is the documentation link again...
http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html

Edit: Mug Funky, I wasn't sure whether the reply to your delta request (here (http://forum.doom9.org/showthread.php?p=731060#post731060)) was sufficient for the task/sufficiently clear... I rewrote it more clearly in the docs for Compensate (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html#Compensate). Shout if you want something different...

[castellandw]

oh, one other thing - on very slight motion blocks are still sometimes visible. try look at a person moving very slightly to the left and you might see it.


@Mug Funky, have you tried applying Fizick's block overlap plugin to remove those blocks because I've tried it on MVTools and it really removes visible blocks(I even modified scharfi_brain's mvbob script with Block Overlap and the results were impressive)?

@mg262, I haven't tried out Motion yet, but I will get to it. I was really looking forward to a motion compensated progressive FPS function for Motion first though because standards conversion was my main objective out of all this.


btw, how many previous frames worth of vectors does Motion need? could seeking be improved by just having it find (and cache?) motion on the last x number of frames rather than resetting every x frames? i don't know enough about this method to know if i'm talking crap :). i'm guessing there'd be some kind of convergence after a certain number of frames, so to keep results consistent (so a random seek produces the same vectors as playing from the beginning) there could be an internal cutoff after a certain number of frames where previous data isn't considered. that may slow it down though. not sure.


Do you think there could be an option for playing with cache size?


auto-noise thresholding helps, but i've not been able to do that well in avisynth yet (got very close). the reason static areas wibble so is usually noise related, or simply because static areas are too sharp (it's good practice to apply a 1 pixel vertical filter if making materials for TV, but not many people do it, alas, so the extra sharpness messes with most deinterlacers). but done right, an "auto-thresholded" kernel-based deinterlace is extremely good, and suitable as input for a motion-compensated deinterlace/standards conversion (if you look close enough at some broadcast stuff, it is clearly done this way - smart bob then mocomp, with a fallback on blending if the mocomp fails. you have to look reeeally closely to tell it's not originally PAL).

@Mug Funky, I think you're right, the more time I play with motion compensation, despite impressive results, I notice that various artifacts(combing or wobbly static areas) could still be apparent depending on the smart bob deinterlacer you're using. An "auto-noise thresholding" kernel-based deinterlace sounds like a great idea. I understand though that one of the reasons we use a smart bob instead of just SeparateFields() is the jitter produced by separate fields which is why I'm curious if there's an alternative to using separate fields after doing a smart bob, converting the frame rate and resizing the frame. Here's a script, for example, of what I mean (no motion compensation involved) involving PAL-to-NTSC conversion:

Source(xxxx)
LeakKernelBob(order=1, threshold=4)
ConverttoYUY2(interlaced=true)
ConvertFPS(59.94)
Lanczos4Resize(720,480)
SeparateFields() <---Replacing with a better alternative
SelectEvery(4,1,2) <---Replacing with a better alternative
Weave()

As a matter of fact, I can post this in the Avisynth usage forum if you want instead.

[Fizick]

mg262,
I at least try your plugin.
1. It is super-fast (44 fps), while MVtools is 19 fps on my PC. Why ? :)
2. but accuracy is not better.
3. What is warp compensation method.?
How you compare the warped blocks? or you simply warp it later, after motion estimation? How it related to my MVFlow (pixels motion interpolation)?
4. What is license of your plugin?

More comments later.

[mg262]

1. It is super-fast (44 fps), while MVtools is 19 fps on my PC. Why ?Because the kind of algorithms you pointed me to (recursive true motion) are very fast ;) . (Because they do a very small number of comparisons per block. BTW, 44/19 FPS are low numbers... they include time spent in AVI/MPEG2source?)
2. but accuracy is not better.Accuracy is probably worse. There are tweaks to implement (e.g. feature matching), but I am putting reasonable emphasis on speed.
3. What is warp compensation method.?Warping happens after estimation. Method is identical to yours, I think (bilinear interpolation of motion vectors).
4. What is license of your plugin?I'm not sure what you mean... I don't mind what people do with the DLL short of e.g. repackaging it in commercial applications. I am not keen on communal development, but you are more than welcome to a snapshot of the source code to read, so long as you will forgive the fact that parts haven't been properly tidied yet. I had thought of PMing you and Manao to say as much, but I decided to wait until I had polished it.

[Leak]

Source(xxxx)
LeakKernelBob(order=1, threshold=4)
ConverttoYUY2(interlaced=true)
Ummm... why "interlaced=true"? After bobbing, the video really shouldn't be interlaced anymore, so I'd think that this would screw up the conversion a bit...

np: Cinematic Orchestra - All Things To All Men (feat. Roots Manuva) (Everyday)

[castellandw]

Ummm... why "interlaced=true"? After bobbing, the video really shouldn't be interlaced anymore, so I'd think that this would screw up the conversion a bit...

OK,I understand the principle of the bob deinterlacer of interpolating to full frame meaning there should be 50 full progressive frames per second in the case of PAL, but what really confused me after using the bob deinterlacer and why I used "interlaced=true" is this:
Why is separatefields() after convertfps() being used because if LeakKernelBob deinterlaced the video to 50 or 59.94 progressive full frames per second, and convertfps() converts from 50 to 59.94 or 59.94 to 50 progressive frames per second, why would separatefields() be used on a video clip that's no longer interlaced?

Leak, I tried with "interlaced=false" although I had to change to SelectEvery(0,3) to get bottom fields first for NTSC, there are some instances though where I didn't need to use ConvertToYUY2(interlaced=false) because ConvertFPS wasn't required, here's a script for example:
Source(xxxx)
LeakKernelBob(order=1, threshold=4) # or mvbob() from scharfi_brain's script
MVFPS(59.94) #From scharfi_brain's mvbob() script
Lanczos4Resize(720,480)
SeparateFields()
SelectEvery(4,0,3)
Weave()

[scharfis_brain]

separatefields() actually can be used on interlaced and progressive video.

the latter one we are doing here and throwing away half of the fields to get 60i out of 60p.

[castellandw]

separatefields() actually can be used on interlaced and progressive video.

the latter one we are doing here and throwing away half of the fields to get 60i out of 60p.

Aah, I see. So after a bob deinterlace, of course you get 2 progressive frames out of 1 interlaced frame, but after using separatefields once you converted the frame rate, you basically get 4 half-frames out of 2 progressive frames. But the thing is now, starting from a PAL source which is top fields first that, is this the field order below for every 4 half-frames after using separatefields() on 2 progressive frames?
half-frame 0: bottom fields
half-frame 1: top fields
half-frame 2: bottom fields
half-frame 3: top fields

[mg262]

castellandw,

I appreciate your desire to get a grip on this subject, but the volume of these posts is making it hard for me to sift out feedback on the filter. If you want to ask general questions about deinterlacing, a separate thread would be more appropriate...

[castellandw]

castellandw,

Please keep to the topic of the thread. If you want to ask general questions about deinterlacing, a separate thread would be more appropriate...


Oh, ok, sorry, mg262, I didn't mean to go too off-topic. By the way, mg262, what threw me off about bob deinterlacers interpolating frames was in standards conversions script with the use of Separatefields() on progressive video frames because I was under the understanding that SeparateFields() are suitable only for interlaced frames since they're field-based, and because of that, I assumed the bob deinterlacers actually just helped eliminate various combing artifacts for interlaced frames without converting to progressive frames. So now I understand that selecting even and odd half-frames out of every 4 half-frames determine the even(bottom) and odd(top) fields respectively, along with throwing away half the fields, I understand better how the re-interlacing works. I also understand that bob deinterlacers interpolate the fields of an interlaced frame to 2 progressive frames, while motion compensation uses the motion vectors on progressive frames to interpolate the motion within the frames. Of course, this goes into even more detail than this.

One of the things that still bugs me with my standards conversion is that I thought motion compensation would help eliminate jerky motion when I convert to NTSC. I'm basically trying to convert PAL MPEG-2 to NTSC MPEG-2. The original PAL MPEG-2 source has no jerky motion whatsoever when I played it back on my PC's DVD player, but the NTSC MPEG-2 output does have some jerky motion (I use Tmpgenc by the way with highest quality motion search precision). I thought using motion compensation would help eliminate the jerky motion, but I'm wondering if the jerky motion is because of the bob deinterlacer(even without motion compensation), motion compensation or an MPEG-2 encoder like TMPGEnc? (I could take this to the Avisynth Usage forum but maybe I asked it here first in case you wanted to develop the motion-compensated FPS function later on and see if the type of bob-deinterlacer or type of motion compensation would have play a role in this.)

[AVIL]

@mg262

The link to the version of november, 05 is broken for me.

[mg262]

AVIL,

Sorry about that... please try again:
Motion, 5 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_05Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

castellandw,
should be a motion compensated fps function (like MVFps in MVTools) required for anything involving motion compensation motion compensated convertfps using your motion compensation engine in your dll would be niceI thought it was a good idea to have a special FPS conversion function which applies motion compensation while it changes the number of frames per secondI was really looking forward to a motion compensated progressive FPS function for Motion first though because standards conversion was my main objective out of all this.I asked it here first in case you wanted to develop the motion-compensated FPS function later onI heard you the first time ;).

You need to appreciate that motion compensation is not a technique to be applied on its own; just using compensate(findmotion()) on a clip will always make it look worse. Cf. earlier in the thread:Motion compensation filters aren't as far as I know intended for direct use in playback... they are intended to provide high-quality input for temporal processes, such as temporal denoising, frame rate conversion (as Fizick mentions) or deinterlacing.motion estimation and interpolation is meant as a replacement for convertfps. running convertfps after compensation (compensating what?) would be counter-productive.So the notion of "motion compensation and then frame rate change" makes no sense. When people write about frame rate change in this thread it is taken for granted that we are talking about using motion compensation to do it -- because this is a thread about motion compensating filters. No, we haven't spelt it out... but there are a lot of implicit assumptions like this and there just isn't space to; that's why this is a thread in the forum for advanced topics.

By the way, mg262, what threw me off about bob deinterlacers interpolating frames was in standards conversions script with the use of Separatefields() on progressive video frames because I was under the understanding that SeparateFields() are suitable only for interlaced frames since they're field-based, and because of that, I assumed the bob deinterlacers actually just helped eliminate various combing artifacts for interlaced frames without converting to progressive frames. So now I understand that selecting even and odd half-frames out of every 4 half-frames determine the even(bottom) and odd(top) fields respectively, along with throwing away half the fields, I understand better how the re-interlacing works. I also understand that bob deinterlacers interpolate the fields of an interlaced frame to 2 progressive frames, while motion compensation uses the motion vectors on progressive frames to interpolate the motion within the frames. Of course, this goes into even more detail than this.
I am happy for you... but you didn't really need to tell me that here :D

All this material is out there, in other threads -- forum search is your friend. Plus, you really do have to use filters to understand what's going on -- what you have gained from 24 long posts in this thread and as many replies is less than what you would have gained from a little while playing with this and other filters. I thought using motion compensation would help eliminate the jerky motion, but I'm wondering if the jerky motion is because of the bob deinterlacer(even without motion compensation), motion compensation or an MPEG-2 encoder like TMPGEnc?Why don't you find out? Trying out a few judiciously chosen scripts would do it.

Go and read, play, read, play, read, play, and try and figure things out :). (And when you get stuck, start a thread on the usage forum.) I do think you need to start with simple topics, so that you understand why things like SelectEvery(4,0,3) occur. But if you want to learn more about motion compensation, reading the MVTools thread beginning to end carefully would be a pretty good start.

[mg262]

Motion, 9 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_09Nov05.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Let me quote from the documentation:

SimpleFPS(clip, clip motion, float FPS)

SimpleFPS changes the frame rate like ConvertFPS or ChangeFPS. motion should be a clip produced by FindMotion, using from = previous (default setting).

SimpleFPS works by compensating vectors across fractional distances, with no attempt to detect or deal with occlusion. At present, it compensates with half pixel accuracy. It is not meant as a usable replacement for ConvertFPS; it is a tool to help find situations which are difficult for motion compensated frame rate change to deal with.

Example usage:

#AVISource...
SimpleFPS(FindMotion(initialise = 4), 100) Although there are too many artefacts to make this usable, the results are IMO fun to look at. It is also extremely fast -- When quadrupling the frame rate, I measured it at 400 FPS, although I'm not certain about my measurement method... but if you are increasing the frame rate it should certainly be faster than simple compensation (because fewer motion vectors per output frame need to be found).

[scharfis_brain]

I wanted to try that mocomped fps change, but it immediately closes virtualdub, when I step two frames forward.

I use an AthlonXP 1600+ and tried AVS 2005-Jan-06 and 2005-May-05

[mg262]

@scharfis,
Grr... how annoying. I am sorry about that. I am looking into it, but if you have a moment, could I have the exact script and a short clip on which it fails (10 frames would be fine)? Also, I have rewritten almost the entire internals... could you try something straightforward, like a simple compensate, and see if it works?

[scharfis_brain]

the scripts I tried:
(of course they are not of any practical usage, because I let searching for motion on a combed frame, but it should NOT crash the process)


this kills VDub after 19 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
converttoyv12()
compensate(findmotion())


this kills VDub after 2 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
reduceby2()
converttoyv12()
compensate(findmotion())

this kills VDub after 2 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
converttoyv12()
SimpleFPS(FindMotion(initialise = 4), 100)


this kills VDub after 4 frames.

#640x480; 29,97fps; YUY2; Interlaced; MJPEG
avisource("doom9.avi") #the asian sports-girl clip
converttoyv12()
SimpleFPS(FindMotion(), 100)

[mg262]

Please try this...

Motion, 9 November 2005 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_09Nov05B.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

[Mug Funky]

awesome! the upsize->compensate->downsize technique makes this look very good indeed.

all we need is some kind of mvmask-esque filter so we can use simple blending on the bad bits, and we've got ourselves a fast, motion-compensated standards conversion for less than 500,000 bux :)

i'm not getting crashes at my end btw.

[edit]

btw, there might be a rounding issue with the fps argument - i enter "fps=60/1.001" and virtualdub reports 59.939 fps. only a slight discrepancy, but i'm sure it's easily fixed so i might as well tell you about it.

[scharfis_brain]

@Clouded: the new B version works without crashes!
what caused them? a memory leak?

[mg262]

@Mug Funky,

I was planning a FPSOcclusion filter, outputting a framerate-upconverted mask of the dubious areas. I was thinking of boosting the quality of the mocomp bit first... there's plenty to tweak... but I'm not bothered.

scharfis's scripts above gave me another idea: reduceby2, compensate/convert FPS, expandby2, and compare with the original. Large differences->dubious areas. I haven't tried this yet...

we've got ourselves a fast, motion-compensated standards conversion for less than 500,000 buxThat's insane... someone must have done this before? Even with a bobber thrown in, it doesn't take that much code. What about MotionPerfect or whatever it's called?

FPS... thanks for reminding me; I will boost the accuracy. It's just the nominal/displayed value, not the one used to perform conversion, so AssumeFPS it away until the next release. AVISynth has a (IMO mildly annoying) setup that asks for frame rate as a ratio of two integers...

@scharfis
@Clouded: the new B version works without crashes!
what caused them? a memory leak?When copying a 4x4 block that required vertical but not horizontal interpolation in assembly, I read a 4x5 block, averaged and then wrote a 4x5 block... . That code has been in there since almost the first version, but for some reason it didn't show up until recently. Thanks for catching it! (By the way, whenever and however you can induce a crash in one of my filters, whether or not the script is vaguely sensible, I would definitely like to know.)

Where do you two, or anyone else, think I should go from here? As I never (ever) standards convert, I haven't much use for this filter... don't mind where it goes as long as there's interest/feedback. Interest in this thread seems to have died down... I don't know whether it has just become too long or that motion compensation isn't directly usable or whether it's just that I was excessively rude above(?)

Its totally reworked under the hood... interface will change soon to match... so I think maybe I would like to add any more 'core' features and release as a beta with a new thread. Seem sensible?

[Mug Funky]

hmm... don't worry about apparent interest in this tool - rest assured there's a lot, but it's often a case of people can't think of anything intelligent to post and so don't post anything at all (except me of course... i'll post unintelligent stuff routinely :)).

true motion and interpolation can be used for more than standards conversion though - there's slow mo (that's a special case of standards-conversion i guess), motion blur effects, replacing bad frames or dropouts in colour (happens a lot with VHS and sometimes with beta SP as well), or slightly over-the-top scene transitions (i saw a music video where they must have hacked a long GOP mpeg file to do the scene transitions - it looked like the I-frame had been replaced with one from another scene, causing the new scene to be slowly revealed by pushing, pulling and replacing parts of the old scene).

of course there's also denoising - couple something like this with peachsmoother or a similar long-acting temporal smoother and you've got some useful and fast tools.

about the 500k standards converters... there's cheaper stuff out there, but it's way slower and not as error resilient (there's motionperfect, and the twixtor plugins for after effects, but i wouldn't use those in production because they're RGB and simply not fast, flexible or good enough for even a straight standards-conversion).

[scharfis_brain]

I hadn't a chance to try twixtor. It needs too much software around it.

Motionperfect IMO is bubble/rubber video. occlusions aren't detected/handled at all.
Not even static areas are masked. So it is useless, too.

[Mug Funky]

i think twixtor has the same deal. it's good for low motion stuff, and to it's credit it was able to detect and interpolate blinking eyes (with an area of around 8 pixels). but it just gives too much ugly meshwarp artefacting. mvflowinter and appropriate settings are already outperforming it.

i never used motionperfect, but saw a couple of demos and was unimpressed.

reason i swear by avisynth is because it's fast, flexible and extremely extensible. also it's much cheaper than after effects :). another reason is you can mask your way out of practically any artefact and you'll still end up doing more in less time than with after effects (or anything else really).

[Didée]

of course there's also denoising - couple something like this with peachsmoother or a similar long-acting temporal smoother and you've got some useful and fast tools. Yeah, try it with Peach ... and be prepared to get interesting effects, but not so much benefit over plain usage. Peach has this preroll issue, and selfmade denoisers basically work on groups of rather small sniplets ( interleave(...).denoise().selectevery() ). In other words: at the time when your compensation "ends", Peach is still in the middle of accumulating...

OT:
Next round of LTSMC() is getting shape, meanwhile. Did a complete re-write from scratch ... it's slow, but denoising and detail preservation are stunning. Biggest problem is to make a proper function out of the big mess the script currently is. :o
/OT

[708145]

Where do you two, or anyone else, think I should go from here? As I never (ever) standards convert, I haven't much use for this filter... don't mind where it goes as long as there's interest/feedback. Interest in this thread seems to have died down... I don't know whether it has just become too long or that motion compensation isn't directly usable or whether it's just that I was excessively rude above(?)


Hi mg262!

I'm definitely interested in this filter but didn't yet spend time to test it.
My target is to up 25p sources to 50p or 100p to get more fluid pans and such.
If I got it right, your filter can already do that :)

For xvid50p I'll give this a try for sure :D
Keep up the good work!

bis besser,
T0B1A5

[marcellus]

Hi,
I have a digital camera that can record little mjpeg avi's of 288 x 208 resolution at 15 fps. I have a bunch of such avi's of low quality but with a big sentimental value that I want to convert to PAL DVD.

My interest in Motion filter is obviously the framerate conversion. I already tried SimpleFPS with default settings (after I lanczosresized to 720x576 first). The outcome it's blocky especially when camera moves quickly or shakes (and I have a shaky hand :o ) or when the content of the image is confusing the algorithm -I think - (for example leaves and branches of trees in a forest - with camera moving).

Is there another way to use this filter to convert from 15 fps to 25 fps more acuratelly (with no concern for CPU time needed)? I tried SimpleWarpCompensate that IMHO gives much better results (visually) when compensating but I can't (or I didn't figure out yet a way to) use it to convert to another frame rate. Or, better, could this mesh-warping be used in SimpleFPS function?

Thanks in advance!

[mg262]

could this mesh-warping be used in SimpleFPS function?
Sure!

Motion, 10 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_10Nov05.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

I've added SimpleWarpFPS; it's used exactly like SimpleFPS. (It was on my to-do list anyway, but knowing that someone wants it pushes it to the top :))

I think you might get better results out of the new MVTools version Fizick is working on -- at least have a look at it. Also, I'd recommend trying to stabilise before applying the filter (look up DePan), and trying to use the filter before upsizing... or maybe resize halfway, apply the filter, and resize again.

@708145,

I think it has too many artefacts to be usable on real material yet, but see what you think; usage is just:

SimpleFPS(FindMotion(initialise = 4, reset = 50), 100)

@Mug Funky,

in order to find said unintelligent posts by you, I think I would need a few hours with search :) FWIW, I would certainly have dropped this filter some time ago if it hadn't been for your interest and feedback.

For denoising, I remember you saying that FFT3D worked particularly well with motion compensation...

[foxyshadis]

A friend actually came to me last week with a handful of PAL dvds that he'd bought, asking if I could do anything with them because they don't work with his DVD player (or tv? I don't know anything about that). Sure he can watch them on his laptop, but...

So I've been watching this and the mvflow thread with extra interest waiting for scharfis to work his magic on a more stable version, plus it's fascinating on its own. ;)

[Mug Funky]

aww, thank's, clouded. :)

btw, the warp-mode is working pretty good on this end. has anything changed in the ME engine? it seems to be outputting less reliable motion than the last release, but it could just be my source and the alpha nature of this plugin.

[mg262]

Motion estimation? Shouldn't have changed since 9b. I have a script that I run through avs2avi to check that I haven't broken anything, and the only time something changed was when I removed the bug scharfis found, i.e. between 9 and 9b. But if you have a clip on which it changed, then something is very wrong and I need to fix it.

I'm never sure about warping... it feels like I'm showing my junk under the carpet rather than tidying up :p. But if we are going to warp, I think maybe it's worth trying better warpers. Adding chroma warping wont hurt either.

I just converted a clip to match the refresh rate of my screen (70 FPS) and its looking nice... maybe noticeably nicer than 100 FPS.


@foxyshadis, IMO standards conversion is a lot of work! Might be possible to multiregionise the player, or else low-end PAL DVD players have dropped in price to little more than expensive DVDs... but I'm not sure your TVs can cope (hardware forum would know). He could also try TV out from the laptop?

[scharfis_brain]

@Clouded: *aaaargh8 now I know where the crashes came from: my swapfile was disabled. I dunno what caused it disabling. F***ing Windoze...

Now, since I enabled the swapfile again, the scripts are working without crashing VDub.

[mg262]

@scharfi,

Whatever the reason, I'm grateful to you for catching the bug.

@all,

Let me outline possible additions (some very speculative):

-- Newbie-friendly Documentation
Explaining more about AVISynth, motion compensation, why you would use motion compensation, etc.

-- More compensation engines
Currently there are two engines -- move blocks and simple warp.

-- More accuracy modes
Currently the motion finder and each engine can work at specified accuracy, out of pixel, halfpixel and (sometimes) general bilinear.

-- More control
I want to let the user specify e.g. warp luma with bilinear accuracy, but move chroma with halfpixel accuracy. There will be sensible defaults/presets.

-- More accurate estimation
Especially for fast moving objects.

-- Combined estimation/compensation functions
Faster than compensate(findmotion())!

-- Support for interlaced clips

-- FindMotion Cache
Support (a little) backwards seeking.

-- Occlusion Masks

-- Compensation handling occlusion


Anything else? Or any preferences out of these?*

*Though be warned some are harder to implement than others.

Clouded

[marcellus]

Hi, Clouded
Thanks for your work. I tried SimpleWarpFPS and for my little shaky movies it doesn't do the trick. I guess I will have to try, as you said, Depan and/or MVtools. As a matter of fact I tried MVTools already and it doesn't do the trick either and it's slow too. But I will try more...

Anyway, I tested SimpleWarpFPS also on some TV captures and it seems that it works pretty good on "normal" footage, at least on footage with low / normal motion.

I made an observation (studying my avi's with fast motion) and I have a suggestion to make :

To simplify things let's say I want to triple the frame rate of a clip with the frames ABC...

A----B----C will become
AMNPQBRSTUC

Now, I observed that the new generated frames M,N,P,Q are warped versions of the frame A, and the more far we go (from M to Q) the more distorted they are, Q being very distorted but still, much closer to A than to B. When I play the movie this is giving a "pulsing" effect, when it passes from Q (the very distorted version of A) to B which is undistorted.

My suggestion would be to make M and N frames by warping the A frame but P and Q warping the B frame. Or even better, a weighted warping that take into consideration both A and B.

Please note that I'm by no means a programmer so take my suggestion accordingly... :)

Anything else?
Well, tell my if I'm dreaming too much, but it would be possible to make a motion compensation based on shapes and edges? Like it would be if I would use Winmorph (http://www.debugmode.com/winmorph/) on every pair of frames, choosing the edges and their corespondance in the next frame (but made automaticly).

[danpos]

Anything else? Or any preferences out of these?*

*Though be warned some are harder to implement than others.

For a newbie like me, I guess that is perfect for now! :)

Cheers,

[Leak]

(i saw a music video where they must have hacked a long GOP mpeg file to do the scene transitions - it looked like the I-frame had been replaced with one from another scene, causing the new scene to be slowly revealed by pushing, pulling and replacing parts of the old scene).
That wouldn't happen to be the video to "Triptych Part 3" that's found on the 188 minute DVD included with Blockhead's new album "Downtown Science" on Ninja Tune? (Which I incidentally bought today... :D)

"Motion compensation artifacts done horribly right" was my first thought... the creator of the video illustrated "Ever had the feeling that the world was gone and they left you behind?" using MPEG artifacts leaving parts of the video behind... ;)

Also, they must've probably worked on reversed video as well, as a it either tore the image to shreds or repaired it over time... :)

np: Blockhead - Cherry Picker (Downtown Science)

[Leak]

A friend actually came to me last week with a handful of PAL dvds that he'd bought, asking if I could do anything with them because they don't work with his DVD player (or tv? I don't know anything about that). Sure he can watch them on his laptop, but...
How about using DGPulldown (combined with reencoding the video to the correct resolution) for this? At least if you want to get the conversion done this year... ;)

np: Blockhead - Crashing Down (Downtown Science)

[mg262]

This...A----B----C will become
AMNPQBRSTUC

Now, I observed that the new generated frames M,N,P,Q are warped versions of the frame A, and the more far we go (from M to Q) the more distorted they are, Q being very distorted but still, much closer to A than to B. When I play the movie this is giving a "pulsing" effect...... is really useful feedback. It makes perfect sense that this occurs, but my eyes aren't good enough to see it -- so I have to rely on you guys! Thank you very much.
My suggestion would be to make M and N frames by warping the A frame but P and Q warping the B frame. Or even better, a weighted warping that take into consideration both A and B.I considered the latter but abandoned it on speed grounds -- there are less time-expensive ways of improving the picture quality. OTOH, it's quicker to add and doesn't mess with my basic framework, so...

Motion, 13 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_13Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Example usage:

#toconvert = AVISource...
toconvert
fromprevious = SimpleFPS(100, FindMotion(initialise = 4))
fromnext = SimpleFPS(100, FindMotion(initialise = 4, from = next), source = next)
toconvert.CombineFPS(100, fromprevious, fromnext)

NB new argument order for frame rate conversion functions; it's now source.simple[warp]FPS(framerate,...). (IMO much more readable!) If you're using simplewarpFPS, and you are more concerned about quality than speed, I recommend using bilinear = true.Please note that I'm by no means a programmer so take my suggestion accordingly...Suggestions from non-programmers are particularly valuable. When deciding what to do (rather than how to do it), programming doesn't come into it at all... and it's easy for a programmer to e.g. not consider something because it is slow, even if it is the right solution. So please, suggest away!

[Alain2]

Hi,

Thank you mg262 for this new filter, it seems to be a very interesting work, and a fast filter if people get 44fps speed with it :)
For denoising, I remember you saying that FFT3D worked particularly well with motion compensation...
I read in the documentation the example for "motion compensated denoising", in which you place a temporal smoother radius = 1 after the introduction of motion compensated frames; Can this temporal smoother be fft3d as you seem to suggest? Is there a particular reason for a radius 1? Can you or Mug Funky points me to the post where the motion compensated + fft3d combination is discussed?

Thanks

[psme]

Hi scharfis_brain and Clouded,

Will there be a faster mvbob using this new MC filter? Or these are not related issue? I always heard that MC deinterlace is the best method for real 60i video content. Thanks in advance.

regards,

Li On

[Mug Funky]

Can you or Mug Funky points me to the post where the motion compensated + fft3d combination is discussed?
it's pretty simple. fft3d has a temporal radius of 1 in bt=3 mode.

so you do something like this:

interleave(forward_compensated, clip, back_compensated)
fft3dfilter(... , ... , ... , bt=3)
selectevery(3,1)

obviously fill the holes with your favourite settings, and use any method you prefer for getting the forward and backward compensated clips (motion or mvtools).

no particular reason to have a temporal radius of 1 - just use as many compensated frames as you need (if you're going 1x forward and 1x backward compensated frames, then temporal radius is going to be 1. it's not wise to have a temporal radius larger than the distance you're compensating).

[mg262]

Alain2,

Good to hear from you again! Mug Funky has answered your question, but you might also find it useful to open up the documentation and scroll down to the heading Example.

psme,

I think that interest is definitely running in this direction... but it needs me to write a robust frame rate conversion function. Or else an effective mask showing where the compensation is working well and where it isn't. On that note...

@all,
For a mask showing how effective the motion compensation during frame rate compensation is, what about trying the (absolute) difference between forwards and backwards compensation? I.e. between these two things:

SimpleFPS(100, FindMotion(initialise = 4))
SimpleFPS(100, FindMotion(initialise = 4, from = next), source = next)

I've had a quick look at it and it seems interesting. I'm going to try coding something else as well... but let me know what you think of this one. Even if it messes up, it would be useful to know why it messes up...

[Mug Funky]

i liked the idea in the mvtools thread about getting a more traditional metric from a full compensation (time=100 or whatever), and compensating that mask to the desired time value.

but difference between 2 compensations seems pretty straightforward too. but it doesn't allow for the possibility that both compensations are wrong. this shouldn't come up too often though (generally with occlusion type situations when one compensation fails the other succeeds in the same areas and vice versa).

[mg262]

i liked the idea in the mvtools thread about getting a more traditional metric from a full compensation (time=100 or whatever), and compensating that mask to the desired time value.Sounds good... seems possible with the existing functions? If not, tell me what is needed. Maybe it would be useful to have some kind of difference function other than just Overlay(mode = "difference")?

I added a quick function to help when playing with masks:

Motion, 14 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_14Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

BlendFPS -- works more or less like ConvertFPS, but supports YV12 -- also should deal with arbitrary colour spaces and arbitrary frame rates.* Unlike ConvertFPS, it will average more than two frames (mainly applies when decreasing FPS by a large factor)... so it might be useful for motion blur.

*It ain't 100% clear what the "correct" weights for general blend conversion are; I've picked values that have some physical motivation. If they don't look right, let me know.

Don't worry if mask-related scripts become slow... there's plenty I can do to speed them up. But "premature optimisation is the root of all evil" ;).

Last thing: the DLL seems to have shrunk considerably without any very clear explanation... all my standard tests seem to be working okay, but please let me know if anything unusual happens.

[krieger2005]

I had a problem with your plugin and want tell it... maybe it is a bug. I use your filter of 10Nov05. I used it in this way:
# last, in high motion precise, motion builder
motionVB = FindMotion(last, reset=300, initialise=20, iterate=6, from=previous)
motionCB = SWC(last,motionVB, source=previous, bilinear=true)
motionVF = FindMotion(last, reset=300, initialise=20, iterate=6, from=next)
motionCF = SWC(last,motionVB, source=next, bilinear=true)

interleave(motionCB,last,motionCF)
Despot(mscene=10, pwidth=260, minpts=20, maxpts=260, pheight=160, p1=23,p2=10,mthres=16, merode=19, color=true, fitluma=true)
Despot(mscene=10, pwidth=140, maxpts=50, minpts=9, pheight=140, p1=22,p2=10,mthres=12, merode=18, color=true, fitluma=true)
SelectEvery(3,1)
I tried it and it works well, but on a special situation VDub crashed (closed without any message). I looked at this position (it was frame 4232 but however). It is a scene where a static picture was shifted... like a panorama-picture, which was shifted from left-to-right side. The shift was very fast...

I looked at the code and the problem seems to be in the FindMotion-Function. Especialy if the variable "iterate" decrease the position, when VDub crashes, is later. if "itearate=1" VDub does not crash on these scene...

If youwant these clip a have to manage the upload but maybe you does not need it.

Thanks for development

[mg262]

krieger2005,

thanks for your message!
motionVF = FindMotion(last, reset=300, initialise=20, iterate=6, from=next)
motionCF = SWC(last,motionVB, source=next, bilinear=true)There is a typo here -- I think you mean motionVF. I'm not sure whether that caused the crash though -- I will look into it.