Animation frames don't line up perfectly (was: Help me judge the stability ...)

[mg262]

I would be grateful for opinions on how stable the output of this global motion compensator is (it's just four seconds per clip, so it should be fairly quick to look at).

Input (original)
http://people.pwf.cam.ac.uk/mg262/posts/Background/mountain_source_xvid.avi

Input (on enlarged background for easy comparison*)
http://people.pwf.cam.ac.uk/mg262/posts/Background/mountain_source_onpaper_xvid.avi

Output
http://people.pwf.cam.ac.uk/mg262/posts/Background/mountain_abs15_compensated_bilinear_xvid.avi

All compressed using Xvid at Quant 1. If the files are too large (at ~1.3 MB) for anyone I will make smaller versions.

*The output is at a higher resolution than the input because otherwise the motion compensation would push the picture "off the edge of the screen". So I encoded the input on a background of the same size to make the comparison easier.

I'm posting this because some indirect reasoning suggests the output isn't completely stable, but I can't see the problems by eye. Any opinions would be really appreciated!

Edit: I added a less blurry version of the output...
http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_abs15_compensated_power_xvid.avi
... but the file is proportionately larger (~1.8 MB).

[movax]

I *think* I saw some instability in there, but I'm not quite sure. Sorry, in the same boat you are mg262. :(

[mg262]

No problem -- thanks for looking!

[Revgen]

The the borders on the output video tend to shift side-to-side horizontaly while the input doesn't.

[Marius-the-Mad]

@ mg262: It seems to be unstable. Not to some extreme extent, but it's visible.

Take a look at the hill behind the trees, the one to the left. I played it looped and when the selection reaches the top of the image and the clip is being repeated, there is visible instability.

But - it's unnoticable (to me) on the first and even third look, unless someone pays attention to that particular area and not the picture as a whole.

<edit> I tested the sharp version.

[mg262]

Thanks for looking!

@Revgen,

The shifting of the borders happens because the original clip jitters left-to-right (easier to spot if you play full-screen and keep your eye on an object near the right-hand side), and when this is motion compensated it causes the borders to jitters -- i.e. if the borders didn't, the clip itself would. I think.

@Marius-the-Mad,

So, the frame-to-frame instability is invisible, but the cumulative effect is visible? That makes sense. (I have a way of fixing it, but it makes the filter about 10 times slower :( ) Thanks!

Edit: Just for fun, here's what it looks like when used as a denoiser.

http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_d40_sidebyside_xvid.avi

(I'm not asking for comments on it; it's just for interest -- though do feel free to reply if you have any questions.) For what it's worth, the denoised version compresses to just under half the size of the original using Xvid at quant 1.

[videoFred]

I would be grateful for opinions on how stable the output of this global motion compensator is.

And how did you stabilised it? :confused:

Fred.

[mg262]

Using a plug-in I wrote...

Not sure whether that's the answer you're looking for?

[videoFred]

Using a plug-in I wrote...

Not sure whether that's the answer you're looking for?

Yes, of cource ;)

Are you testing it, not ready yet?

Fred.

[mg262]

A little testing, some calibration of default parameters and a lot of dithering over whether it's good enough to release.


Edit:
(Bear in mind that this is in some ways a toy problem before reading this.)
Supersampling increases the compressibility even more...

Original (http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_source_cropped_xvid.avi) -- 1338 kB

Denoised (http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_abs15_d40_recreated_power_xvid.avi) -- 656 kB

Supersampled Denoised (http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_abs15_d40_ss2recreated_power_xvid.avi) -- 496 kB

Is Xvid at quant 1 (with B-frames at quant 2) a legitimate way of checking compressibility?

[movax]

I think Q2 would be better, since Q1 really just saturates the codec. For anime settings, IIRC, I think I have Q2 w/ Q3 b-frames. (I usually just set and forget them).

[communist]

I'd rather use quant 2 (with (4 for b-frames) - quality wise q1 and q2 are almost at the same level but filesize differs quite a bit.

/edit looks like someone was faster - anyway the defaults for 1stpass from 2pass mode should work pretty good.

[Marius-the-Mad]

So, the frame-to-frame instability is invisible, but the cumulative effect is visible?I'm not sure. I tried playing it without looping and now I can see the same shimmering in that place. I think frame-to-frame stability needs an improvement, but said instability is rather unnoticable if one doesn't look directly at that particular place in the appropriate moment. :)

[mg262]

Thank you - that's very clear.

...now I have to figure out how to improve the results when I can't see the glitches.

http://people.pwf.cam.ac.uk/mg262/posts/knifesmiley.gif

[mg262]

movax, communist: thank you. I think I will go with 1st pass defaults. Should I switch on 'Full quality first pass?'

Edit:
With that option switched off, these are the results:

Original (http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_source_cropped_xvidp1.avi) -- 368 kB

Denoised (http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalf_abs15_recreated_power_xvidp1.avi) -- 244 kB (66%)

Supersampled Denoised (http://people.pwf.cam.ac.uk/mg262/posts/Background/mountainhalfss2_abs15_recreated_power_xvidp1.avi) -- 182 kB (49%)

[mg262]

Well, I found the answer: the frames can't be made to line up perfectly, with any motion vector. That's a tall claim, so here's something to substantiate it (incidentally, this can be used to check global motion vectors produced by any method)...

Download this (script plus filter plus 2-frame AVI) 285 kB file:
http://people.pwf.cam.ac.uk/mg262/posts/Background/animatemotion.zip

Extract it somewhere and open the script in VirtualDub(Mod); you should see something like this...

http://people.pwf.cam.ac.uk/mg262/posts/Background/animatemotion.jpg

Now go to frame 1275 (use control+G). This is the motion vector estimated by the plug-in. If you go backwards or forwards by single frames (use cursor keys), the x of the motion vector will be changed slowly; if you go backwards or forwards by exactly 50 frames (click on the slider in VirtualDub or use Alt+cursor keys) the y will be changed slowly. So using the "Magnified Difference" pane, you can check the motion vector yourself. I think the filter does about as well as can be done, although obviously I'm biased ;)

It may be that changing the resampling method will produce better results... I'm going to play with this.

Any thoughts, agreement, disagreement welcome!

[movax]

movax, communist: thank you. I think I will go with 1st pass defaults. Should I switch on 'Full quality first pass?'

Yeah, without that, you don't get a q2 1stpass sitting around on your HDD to compare the 2nd pass against.

[Mug Funky]

oooh, more motion-compensation tools? i'm interested :)

care to share some info on what this plugin's doing? like OBMC? mesh-warping? this could quite obviously be a good addition to the avisynth toolbox.

[mg262]

Glad you found this thread! It's just global motion compensation, but with three features:

1. It takes masks
2. It deals with subpixel stuff properly
3. The sensitivity to different luma-differences is specifiable

So you mask out the foreground objects, and line up and stitch together the background objects. (This is part of Cel Background, as you've probably guessed by now.) It also has scene detection in, and all the functions take a scene argument and work sensibly (i.e. without crossing scene boundaries). Hopefully I will add features to cope properly with zooms and with parallax (i.e. bits of background moving at different rates).

The mask code isn't in assembly yet, so I was just testing the subpixel stuff... although there is now enough in place to make it work as a stabiliser, so I think I'll release that shortly. It's a nuisance that frames don't quite "stack up" perfectly, but I think this is still the way to go; OBMC, mesh-warping estimate many more parameters, so the error/parameter will be much higher. Very subtle warping from a grid mesh might help...

Stacking results are looking better since I fixed a bug, so I'll post some new screenshots soon.

Edit: stabiliser here...
http://forum.doom9.org/showthread.php?t=100396