Good VirtualDub or Avisynth deshaker for this job?

[fvisagie]

Hi All,

I'm looking for the simplest way to deshake a handycam video file. The main challenges I'm facing with this file are:

1. The file contains multiple segments. Some stable ones shot in landscape mode from a tripod, and shaky hand-held ones shot at high zoom levels. I would prefer not to apply deshaking to segments that don't need it. As far as I know (most) VirtualDub filters can't be applied to multiple range selections. It seems that selectively applying deshaking would require multiple passes, intermediate files or chained frameservers in VirtualDub, or multiple trims in Avisynth. I do favour VirtualDub's linear editing , but would make the trims in Avisynth if a professional result requires it.

2. The source file is in the YV12 colourspace and I would like to avoid colourspace conversions as far as possible. Again, I understand that VirtualDub filters operate in the RGB32 colourspace only.

Given these challenges, which VirtualDub or Avisynth image stabiliser/deshaker would you recommend for this job?

Thanks,
Francois

[martin53]

Hi @all,

I'd like to drop some notes about my experiences with deshaking, and propose some methods that I hope might find a friend or two.

Probably the deshaker among free software is Gunnar Thalin's DeShaker for VirtualDub.
Please read the manuals, the Wiki or various posts, or the attached scripts, on how to use DeShaker from AviSynth.

However, even a good tool gives better results if used with brain.

Deshaking divides up into
1) motion estimation
2) motion compensation
3) maybe border handling

DeShaker uses an 'intuitive' algorithm for motion estimation, that differs from the algorithms used for e.g. the MPEG codecs. Most important, it only estimates motion for blocks that are clear enough to be tracked, and leaves the rest of the frame 'unestimated'.

Long time ago, I contacted Gunnar Thalin when I wanted to deshake a scene with an aircraft takeoff. In that scene, DeShaker's motion estimation jumped between the aircraft and the ground, as the aircraft was hiding and revealing some observers on the ground.
At that time, we did not share the same opinion if deshaking should really freeze some part of the image or try to smoothen the average speed of all objects. Today, I would support the opinion, that no motion for the background is better than slow panning especially for playback on all those liquid crystal displays. When the eye follows objects on these displays, details smear on the retina - that is the neccessary drawback of all non-flickering displays.

But the example still shows that different situations require different approaches.

1) For the motion estimation, I prefer MVTools2's MDepan(). See attached DeShakerPrep.txt.
MDepan() writes a log file that DeShaker can read. The motion estimation is similar to MPEG methods and works well even with bad source material. My example is slower than DeShaker, but you can download the manual and speed it up.

2) For the motion compensation, DeShaker has the advantage that it uses the complete motion history over other approaches that average only some previous/future frames. On how to use it inside of AviSynth, see DeShakerExec.txt. Inpand() is part of the MaskTools and needs separate download. You might need to adjust the path strings. Not all Deshaker options are configurable here; e.g. I never found adaptive zoom useful.
A one pass alternative is DepanStabilize() from the DePan plugin. My example DeShake.txt also needs the Undot plugin. All tools can be found via the well-known AviSynth sites.

3) Finally, you might like less obvious black borders. All my examples are in RGBA mode and provide a mask to be overlayed over a background. If you do not need that, you may strip them down which will also make them faster.

See my next post for more on this topic.

[martin53]

Deshaking neccessarily moves the content of frames and thus one of two options, or a mix of them, must be chosen: Either zoom in, or fill the empty space with something.


DeShaker provides the option to use previous/future frames, DepanStabilize can mirror the frame outside its edge etc.

This post introduces an AviSynth filter that uses 'all frames from the beginning of the clip'.

One preliminary note: the filter is not fast, and it forces you to run the motion compensation pass from clip start to clip end - it definitely violates AviSynths random access paradigm!

The filter functions that were described in the previous post prepare not only a motion compensated frame, but one in RGBA mode with a suitable mask. This means, they can be used as an overlay to anything, and where the border is, 'anything' will shine through.

The attached filter function FillBG() saves the current frame and uses it as the background for the next frame, and so on infinitely.
Because that looked better, the saved background is blurred each time, dissolving slowly into indistinguishable colour.

It does not yet look perfect. My future plans include to use larger dimensions than the clip dimensions for that file, and motion compensate it, before it is used as background. But in comparison to the original tools' options, it is already more unobstrusive, I feel.

The filter provides a zoom factor, so zooming with good quality can be used to balance between borders and loss of content.

The attached file contains an example for the effect: you can open it as an avs script and watch a small point move around, leaving a dissolving trail.

[martin53]

Originally Posted by fvisagie

1. The file contains multiple segments. ...
2. I would like to avoid colourspace conversions

1. If you do not use zoom, then you mainly lose time by deshaking the whole clip. But honestly: probably you will end up in dividing the clip into parts that need/don't need processing, so you could do that right away.

2. Agreed, and DepanStabilize might offer that. But if you read my post about overlaying, you might find that some colorspace conversion allows you to do things that are just impossible in the original colorspace. Because in the end, most content is displayed on displays working with RGB, and therefore incapable of displaying zero/maximum luma with saturated colors, I would suggest you give it a try and judge the result.

[fvisagie]

Thanks for your suggestions, Martin. Looking more closely, it turned out very little of the clip does not contain unstable footage, so I simply ran all of it through the deshaker. I was more than thrilled with the results!

First I evaluated both DePanStabilize() and DeShaker, against my footage. From my limited experience and knowledge it seems that they offer different strengths. For this footage, I obtained vastly better results with less hassling with DeShaker than with DePanStabilize(). Specifically, it was able to stabilise a greater range of movement (with very acceptable zoom), and its edge filling was markedly superior to DePanStabilize's. The latter would create fairly noticeable line artifacts along the moved frame's previous borders.

Thanks also for your contributions to this topic. I trust you'll receive some knowledgeable and valuable feedback.

[johnmeyer]

Originally Posted by martin53

This post introduces an AviSynth filter that uses 'all frames from the beginning of the clip'. ...

The attached filter function FillBG() saves the current frame and uses it as the background for the next frame, and so on infinitely. ...

Do you have any video that shows the results?

I use DePanStabilize in my version of a script developed by "VideoFred" for compensating motion in old amateur film transfers, but the border handling is a major issue. I have written extensively about Deshaker, so I know that pretty well, but haven't figured out a smooth way to incorporate it into this film restoration script. The stabilization in DepanStabilize actually works fine in this particular application; it's just the border handling I want to fix.

I have to scratch my head some more to see if I can figure out why your FillBg script has to read all the frames in order to work. It seems like there should be some way to simply average two or three adjacent frames and use those.