Deshaker by Gunnar Thalin

[texmex99]

Just discovered Deshaker and it worked miracles on some handheld ski footage (bit hard to hold the camera still while skiing downhill). I am, however, having problems with another clip that I was hoping to "stabilize". This is a shot from a "fixed" camera (sitting on a deck rail - no tripod available) of the moon rising over the mountains. The clip is 15 minutes long, but will be compressed to about 7 seconds for "effect". Unfortunately, the camera was on the deck, and so was the beer, and whenever someone stepped out on the deck for a beer, the camera shook ever so slightly. I figured Deshaker would have no problems with this clip, but I can't figure out how to adjust the settings to make it work. The frame is mostly black, other than a few lights in fixed positions, car headlights moving through the frame, and then the moon rising. I assume that the easiest way for Deshaker to do its magic is to have it lock on to one of the "fixed" features in the frame, but I can't figure out how to tell it what to lock on to. I've tried isolating several parts of the frame, but it won't "lock on" - I guess it assumes that the black is background and the lights are "moving objects". Anyway, any suggestions from the pros here on how to deshake this scene? I've attached a frame with the "fixed" lights circled - those are the features I need to "lock on" to (i.e. they shouldn't move). Any suggestions would be appreciated! Thanks!

[guth]

If you'd like to send me a short problematic sample clip, I'll have a look at it.

[texmex99]

If you'd like to send me a short problematic sample clip, I'll have a look at it.
That would be great - only I'm on vacation at the moment. I'll post a clip as soon as I get back. Thanks so much for this great tool!

[wonkey_monkey]

I second that - deshaker is fantastic. I do have one question though...

In the documentation, it says:

The motion smoothness calculation is...based on frame count

But for the default of 400 frames, the video doesn't come out as smooth as I would expect. Is it actually measured in frames, or is it a %age (so 400% would be 4 frames)?

David

[Terka]

when a motion is fast (+long shutter speed) -the deshaked picture is blurred. is it able to add some motion deblur to this pictures? (like focus magic)

[wonkey_monkey]

Guth, are you still working on deshaker? In addition to my previous question, is there any chance that a future version might include perspective correction? :)

I ask because some stabilised videos suffer from a slight warping if the input is shakey enough (as my time lapse videos can be, and I've seen the same effect on normal video). If deshaker asked for the FOV of the images, it could correct in a more realistic way, rotating the input images in three dimensions rather than just shifting horizontally and vertically.

David

[WarpEnterprises]

I just saw that there is a new Version 2.2 (http://www.guthspot.se/video/deshaker.htm).

@guth: I have two questions:
(1) You wrote "Improved handling of new scenes ... allows pass 2 to take appropriate action".
Would you mind giving some details what this action is? Is there a forced "jump" in the resulting deshaking shifts or the like?

(2) regarding the new "Remember discarded areas" option: how is this related to the "blocks OK" percentage?

[guth]

Sorry for not replying. I didn't get any notifications of any activity in this thread.

But for the default of 400 frames, the video doesn't come out as smooth as I would expect. Is it actually measured in frames, or is it a %age (so 400% would be 4 frames)?

It's based on frame count. It doesn't mean that it equals the number of frames. What I mean is just that if you have video with more fps (as interlaced video in a way has), you should increase the value to get the same result. If 400 isn't smooth enough for you, just increase it. :)

when a motion is fast (+long shutter speed) -the deshaked picture is blurred. is it able to add some motion deblur to this pictures? (like focus magic)

Focus magic looks pretty cool, but it doesn't seem to be able to correct motion blur automatically, you have to manually measure the blur direction and length. That's hard to do automatically. And I'm sure the actual correction isn't easy to do either :). I doubt it will happen.

...is there any chance that a future version might include perspective correction?

Hmmmmm..... I'll have to think about that one.... real hard... :)

(1) You wrote "Improved handling of new scenes ... allows pass 2 to take appropriate action".
Would you mind giving some details what this action is? Is there a forced "jump" in the resulting deshaking shifts or the like?

It does this:
1. The end of scene 1 and beginning of scene 2 are basically placed at "zero correction". And there's no smoothing over the scene cut, so if scene 1 ends by panning to the right, scene 2 can start by panning to the left, abruptly.
2. It doesn't attempt to use frames from another scene when using previous and future frames to fill in the border.

(2) regarding the new "Remember discarded areas" option: how is this related to the "blocks OK" percentage?

I'm not sure I understand your question. Basically, what I do is use the pattern of good/bad blocks for one frame as a starting point for the next frame. This usually results in that it is able to continue locking to the background when large objects enter the scene.

[Terka]

yes to remove the motion blur - the amount of motion is needed to know. and deshaker has this information-direction and amount. thats why im asking about removing motion blur.

[guth]

It's not quite that simple, I'm afraid. The motion blur occurs *within* each frame not *between* them. Matching between two motion-blurry frames will be extremely unreliable so you wouldn't be able to get any good information about direction, and specially not about the amount of motion blur, since it can be much different from the amount *between* the frames (the shutter speed doesn't always equal the "frame speed").

[Fjord]

Gunnar, I have a request for the Use previous and future frames to fill in borders option. Could you add an option to start pass2 output immediately without waiting for Future frames to be read, and to continue until all frames of the input file are output?

It would make life alot easier if the output file frames matched the input frames exactly, with audio in sync. I know how to resync the audio via VDub's audio interlace dialog, but having the frames shifted 30 frames later in the AVI isn't too handy.

Could you process the first n (30) input frames with the number of future/previous frames starting at 0, and incrementing to n (30) by input frame 30. By that I mean to process input frame 1 (the second frame of the input) as if the user had specified only 1 previous/future frames. At the third frame you would have 2 previous frames in the buffer, and use only those to fill borders.

Similarly, to decrement the number of future frames at the end, to enable all of the input frames to be processed, although with fewer previous/future frames for the last few frames.

I usually have at least a second of leader and tail on my videos, so I wouldn't be bothered by degraded border fill in the first few and last few frames. But having the Deshake'd video a frame-for-frame match with the original would definitely make some editing steps easier, with respect to original timecode for example.

Does that make sense to you?

Edit: Perhaps the easiest programming solution would be to initialize the the entire previous frames buffer with copies of the first input frame, and at the end to append copies of the last input frame to the future frames buffer, until the last input frame is finally output.

[Fjord]

I have a Canon S3 IS digicam, which records 640x480 30.000 fps progressive MJPEG video. When I Deshake these video clips, with 30 previous/future fill frames, the text message in the first output frames says "Deshaker: output is delayed 30 frames to collect future frames; Audio should be delayed 999 ms to maintain audio/video sync". I understand that message, but it should say 1000 ms rather than 999, since the input video is 30.000 fps. 999 would be right if the video was 29.97 fps. If I use a delay of 999 ms, my audio starts 45 samples before the video frame. 45/44100 = 0.001 = 1 ms. If I use a delay of 1000 ms I get perfect sync.

Does Deshaker internally use an NTSC framerate of 29.97 even if the input clip is 30.000? (VDub's file information dialog does show correctly "640x480, 30.000 fps (33333 us)" for the input file.

[Terka]

Guth thanks for reply

[wonkey_monkey]

At the third frame you would have 2 previous frames in the buffer, and use only those to fill borders.

You still wouldn't have any future frames - Deshaker can't use future frames until VirtualDub has played them, so the choice is either to delay the output, or not use any future frames.

The only solution would be to store all frames during pass 1 for use during pass 2.

David

[guth]

Fjord, about your first question David is absolutely correct. The video simply has to be delayed. Otherwise Deshaker won't be able to access the future frames.

As for your second question, you are correct. It should say 1000, but due to a rounding error it says 999. I'll fix it in the next release. I doubt many people will be able to detect a 1ms out of sync, though. (Btw, 29.97 will actually give you 1001ms, not 999.)

[Fjord]

Thanks David and Gunnar, now I understand the problem with future frames. I guess I'll have to figure out a script to pad the input video with 30 frames, and then to delete the first 30 "delay" frames from the output file. Has anybody worked out a script for that?

As for the 999 vs 1000....you're absolutely right. Nobody could hear a 1 ms lag. I frequently synchronize clips by matching waveforms on the timeline, so I notice it by eye.

By the way, Deshaker is fantastic. You're fantastic, Guth.

[smok3]

I'll have to figure out a script to pad the input video with 30 frames, and then to delete the first 30 "delay" frames from the output file.
both can be done with avisynth (to pad, check 'BlankClip', to trim check 'Trim' :))

[wonkey_monkey]

Gunnar, have you thought any more about adding true 3D correction to Deshaker? I have a sequence of shots that would look great, except I used a wide angle lens.

I've done similar stuff in the past if you think you might need any assistance :D

David

PS Have you any idea why Deshaker would have such a problem with these two images?

http://img14.imgspot.com/u/07/161/15/deshakerfails.jpg

Even if I force it to accept all blocks, or increase the search, Deshaker draws tails in the wrong direction. This pair of images doesn't show much more more movement than any other pair in the sequence.

David

[guth]

Yeah, I thought about it for two minutes, until my head exploded. :)
I'm afraid it would mean a lot of hard work. More than I'm up for atm. But we'll see...

As for your two images, I can't see them.

[wonkey_monkey]

Yeah, I thought about it for two minutes, until my head exploded. :)
I'm afraid it would mean a lot of hard work. More than I'm up for atm. But we'll see...

Fair enough :) I might give it a try myself, though I've only just figured out how to write AviSynth filters.

As for your two images, I can't see them.

Yeah, they kept getting deleted from the image hosts, presumably cos they were too popular... oh well. I'll dig them out again and upload them somewhere safe:

http://horman.net/trouble1.jpg
+
http://horman.net/trouble2.jpg
=
http://horman.net/deshaker.jpg

The originals are 4x larger in both dimensions, but the scale doesn't seem to matter as Deshaker still fails badly. I can manually rotate the first image to be closer so Deshaker succeeds, but I've seen it match much worse pairs before without any problem.

David

[guth]

I've never seen this behaviour before. For some reason deshaker finds a completely incorrect initial match and can't "get out of it". But if you limit the initial search range to 5% or so, it works.

[wonkey_monkey]

I've never seen this behaviour before. For some reason deshaker finds a completely incorrect initial match and can't "get out of it". But if you limit the initial search range to 5% or so, it works.

Phew, good to know I'm not just going crazy.

Guth, would you mind sharing the "secret" of how Deshaker smooths the pan/rotate/zoom variables it records in the log? From what I can see, they are all relative, i.e., they measure the difference between one frame and the previous frame. Do you translate these to "absolute" rotations/translations and average them over time, or am I missing a nicer way of doing it?

I'm having a tough time visualising the process in my head (or on a bit of paper).

David

[WarpEnterprises]

guth described it here in this post: http://forum.doom9.org/showthread.php?p=448933#post448933

[guth]

Indeed, but I'm not sure that post answered everything you asked.
You are correct in that I translate the values to "absolute" values, simply by adding them together (for zoom I add the logarithm of the zoom factors). And then I smooth that curve as described in my other post. Then I "move" each frame according to the difference between the smooth curve and the original curve. Simple as that... :)

[wonkey_monkey]

Thanks guth - just what I wanted to know! Your earlier post make it sound a little more complex...

David

[wonkey_monkey]

Here's a gif showing the kind of distortion that can be fixed by using 3D correction (Deshaker on the left, my AVS plugin on the right - the difference is subtle):

http://www.channeltv.co.uk/deshakers.gif

The plugin's still in development. It relies on Deshaker's log file and won't be anywhere near as comprehensive, but it solves my problem :)

guth, a while ago I ran Deshaker on almost 40 minutes of video. As I was still experimenting, I set very high motion smoothness parameters, and the calculations at the start of pass 2 took an extremely long time, so I assume that the speed is dependent on the amount of smoothing, as well as the number of frames in the clip. How do you smooth the variables?

David

[guth]

Nice, it still shakes a lot but the perspective distortion seems to be a lot better. To make it perfect I guess you have to do something about pass 1 too. It can only take panning, rotation and zoom into account today, so it will get confused by perspectives.

The smoothing process is described in the post that warp linked to above. The equation systems take longer time to solve when the smoothness is high, because each frame will then have a bigger impact on frames far away in the past or future. And since the equation systems are solved numerically, it takes a while for these low frequency changes to propagate over the equations, so to speak. Also, if you use max. correction limits, they can be very time consuming.

[wonkey_monkey]

Nice, it still shakes a lot but the perspective distortion seems to be a lot better.

Yes, there is some trickiness to be solved when it comes to rotating first around one axis, then around another. I might be doing the z-axis rotation in the wrong place too...

Some residual shakiness is to be expected anyway, as the frames weren't taken from the same position (I was walking along the top of a damn with my finger held on the camera's shutter release).

The smoothing process is described in the post that warp linked to above. The equation systems take longer time to solve when the smoothness is high,

I was a little confused by that first post, and the mention of equation systems... but after your recent post, I just did this (taking rotation as an example):

* Add each rotation from the logfile sequentially so that it becomes an "absolute" figure for that frame.

* Smooth these absolute numbers out - essentially a blur. (a sequence of 0,1,1,0 might become 0.333,0.666,0.666,0.333 for example)

* Use the difference between the smoothed and absolute figures and translate the images.

But Deshaker does something a lot more complex, by the sound of things...? I get similar results though.

At the moment my "blurring" process is just a simple average over a specified number of frames, but applied multiple times (even just 3) this can approximate a Gaussian, err, thing (I want to say blur). This can be performed in a time that is only dependent on the number of frames, completely independent of the amount of smoothness. But I haven't considered how the maximum correction limits would mess things up, since I've never used them :D

David

[guth]

My algorithm might sound a little complex, but it doesn't require a lot of coding. Blurring probably works pretty well too. One difference is that I try to minimize the *squared* correction amounts in order to minimize the worst cases, whereas your approach seems more linear. And you don't really have much control over the process, for example to fix max. corrections. But that might be fixable, I don't know. Most importantly, though, my approach just feels more mathematically correct to me. :)
Or maybe I'm just jealous I didn't come up with such an easy solution. :)

[Fizick]

I know also about mathematically correct Kalman filtering, but I did not implement it in DePan....
:)

[Undead Sega]

http://www.channeltv.co.uk/deshakers.gif

why do these distortions happen?

and what do you call them also?

[mustardman]

My guess would be the effect is caused by the distortions present at the edges of the lens. The distortion becomes worse the further away from the center of the image the 'target' is.
For a steady lens, this is of course not noticeable. However, if the lens moves around, and then the image is stabilised later, the distortion becomes very obvious (as you can see).
Think 'pincusion' effect with wide angle still (and video) cameras.

Is is particulary bad with cheaper lens, and is even present with more expensive lens. If you really don't want it, buy a 'hollywood' movie camera lens... but be prepared to walk away 20k poorer!

MM

PS: One way to minimise this effect with a "cheap" lens is to zoom in to the subject. Less of the glass is used, and the area in use is closer to the center. Only problem, if the camera is shaking to begin with, zooming in is going to make it a whole lot worse!

[scharfis_brain]

AFAIK there was a solution for AVISynth to undo the distortions of wide angle lenses...
But I don't know its name anymore...

EDIT: http://forum.doom9.org/showthread.php?t=127432&highlight=barrel+distortion

One need to undo the distortion before deshaking.
If the fish-eye look is desired one may re-apply the distortion after deshaking again.

[Undead Sega]

well i remember using 2d3 SteadyMove and i get these weird distortions on my old Hi8 camcorder with just the stock lens with the settings are ranked up high to get the best stablized video.

[guth]

Actually, this effect doesn't come from the fact that wide angle lenses often are imperfect, but rather from the nature of wide angles.

Most wide angle lenses are usually either rectilinear or fisheye. I guess it's pretty obvious that fisheye lenses will give strange results when deshaking, but the fact is that even perfectly rectilinear lenses will look a bit weird (which that sequence above demonstrates). Here's my attempt at explaining why, in a hopefully easy to understand way... :)

A rectilinear lens will make straight lines in the world appear straight on the image too. Now, imagine you're taking a close wide-angle picture of a chessboard. All squares will get the same size on the image even though the outer squares are further from the camera than the squares in the middle. This essentially means that objects are magnified when they are close to the edge. When you turn the camera to point at an outer square it will get smaller on the image.

And deshaker doesn't like it when objects change size when you pan the camera... :mad:

[Undead Sega]

And deshaker doesn't like it when objects change size when you pan the camera...

i see, is this something you are going to or are wanting to work on?

[guth]

I have considered it, but to be honest I don't really think it will happen. Not anytime soon anyway. Sorry...

[Undead Sega]

oh, thats a real shame, it hasnt received an update for a very long time now, i do hope u can get to it one day, i will look forward to it.

and, since speaking to the actual creator of the filter :D, about the borders, i assume u have watched the video sample from YUVSoft and its comparisions to yours?

can your filter literally recover the missing edges in the black borders?

[scharfis_brain]

the video of YUVSoft is a LIE!
they didn't set up deshaker properly just to present their own product as the better one.

[2Bdecided]

Deshaker is lovely, but how can I speed it up?

I assumed this would have been asked before, but I can't see a list of which options really speed it up or slow it down in the five pages here.

Cheers,
David.

[guth]

i assume u have watched the video sample from YUVSoft and its comparisions to yours?

can your filter literally recover the missing edges in the black borders?
I hadn't watched that video before. How funny. :)
First they steal the name "deshaker", and now this.

Yes, deshaker can recover missing edges (by taking video data from past and future frames).


Deshaker is lovely, but how can I speed it up?
Thanks! I recommend you search for the words "fast" and "slow" here:
http://www.guthspot.se/video/deshaker.htm

[Undead Sega]

Deshaker is lovely, but how can I speed it up?

Thanks! I recommend you search for the words "fast" and "slow" here:
http://www.guthspot.se/video/deshaker.htm

i think this is something new you could work on, meaning adding support for Multithread and MultiCore CPUs? that would be so great if it took advantage of it.

and when it uses previous frames to recover missing edges, does it literally look for wat was suppose to be there?

possible to show an example?

[scharfis_brain]

Undead Sega: would you PLEASE read the authors site carefully before asking for obvious things?

[guth]

I think I'll have to agree with you, scharfis. :rolleyes:

It already does multithreading and as for recovering missing edges, please read my site and/or do some testing yourself.

[2Bdecided]

Thanks! I recommend you search for the words "fast" and "slow" here:
http://www.guthspot.se/video/deshaker.htmThank you.


I have read it (the web page), and used it (deshaker!), but still have a "kind of" question about edges. I'm not sure how to phrase this. Obviously everything in the frame is processed with motion estimation and compensation. However, it seems that the areas out of frame are not. Here's an example of what I'm trying to say...

Let's say the area at the top of the frame is first revealed in original frame 100, but was needed for compensated frames 80-99 too. During those frames, it looks like the static top of frame 100 is copied to the top of frame 80-99. Let's say the existing top of frames 80-99 is moving left to right. It looks like there's no attempt to make the part pasted from frame 100 move left to right - it just sits there.

Now I know you don't actually know how it should be moving, because it's not there in those frames(!) - however, would it not be reasonable to guess that if the parts near it are moving left to right, then the part copied from frame 100 should move left to right too? It might not be spot on, but it might be closer to reality that not moving it. I guess it comes with its own complications...!


Not a criticism (because deshaker is amazing) - just a question/suggestion.

Cheers,
David.

[scharfis_brain]

deshaker 'only' applies global motion compensation to the frames. This means that only the whole frame is being moved.
This is also done with previous and future frames.
But every movement that does not equal to the whole frames movement - I call it differential movement - will most likely produce some weird artifacts at the compensated borders.
Of course, one could write algorithms that will do local motion compensation on the frames that are being used form border filling. but that would slow down the process even more.

[guth]

Exactly. Thanks scharfis!

This usually means that the background will be moved correctly when taken from another frame, but if there are objects moving in the borders, they tend to look a little weird.

And btw, I have absolutely no problem with questions/suggestions or even criticism. It's actually even slightly ok to ask things that's already on my web page.
:readguid::readguid::readguid: ;)

[Undead Sega]

would this be a good example of what your talking about?

[guth]

would this be a good example of what your talking about?
Yes. But you don't seem to be using "soft borders". You might want to try that.

[Undead Sega]

also, here is the actual video of using the Deshaker program.

it includes a before and after processing:

http://www.wikiupload.com/download_page.php?id=35945

P.S. i dont know if anyone has seen such shaky camera work either.