Deshaker by Gunnar Thalin

[Undead Sega]

@ Gunnar, even though ow great this filter works for me and many people out there, is there any improvements u would want to consider or is there more work that can be done on your filter?

[guth]

I'm always open for suggestions.
(But I'm a lazy man, so it would have to be something truly awesome! :))

[Undead Sega]

(But I'm a lazy man, so it would have to be something truly awesome! )

Now that doesnt suprise me, hahahaha, but i praise you for admiting it rather than someone else slowly trying to get it out from you :d

anyways, wat im meaning is, the way it is extrapolating to fill the borders, the current state of it, is there any thing more u can do about it? or did u just got it to a certain stage and thought "this looks good enough" ? Not being rude of course.

[guth]

If you mean the extrapolation setting specifically, I think it's quite perfect looking, but very slow.
But if you mean using past and future frames to fill the borders, it could probably be improved a bit. It might happen some day, but then again it might not.

[Undead Sega]

Now, with you saying that it could improve abit, what would that be exactly may i ask? If u dont mind telling me :D

[guth]

Maybe doing some kind of local motion compensation that scharfis_brain talked about earlier. I don't know.
But it's kind of impossible to know what's outside a frame at the specific time that frame was shot. Guessing by looking at other frames will simply always look a bit strange under some circumstances.

[Undead Sega]

i see, so if u were to impliment motion compensation, what difference could we probably see? as opposed to filling black borders with previous frames, or even with the combination.

also, i have read somewhere the super resolution is a very good algorithm for extrapolation.

[guth]

I would obviously still have to take the image data from past and future frames (cause that's where the missing info is).
Seriously, you need to relax a little. I'm most probably not going to implement anything like that. Mostly because I don't think it would look that much better.
As for "super resolution"... believe me, no matter how awesome it is it will never be able to extrapolate a missing head from someone's body. :)

[Undead Sega]

As for "super resolution"... believe me, no matter how awesome it is it will never be able to extrapolate a missing head from someone's body.

oh im very aware of that, but i am just seeing if this filter can seriously be improved or become any better, its just i cannot find the right words at the moment to pass my exact words on. i know it can.

[Dr.D]

No doubt deshaker is a great tool, big thanks to the author.
Filling borders done very smart too, great feature.
But, I believe, there are people (like me) who are crazy about any artefacts, what filled borders definitely are, regardless of how smart this done. So, I can sacrifize of resolution, cut off borders and have an artefacts free picture.
Here is the problem: how to know how much I should cut off? Now I do some not easy calculations based on 2nd pass parameters, using even trigonometry :), and still not sure that no artefacts left.
So, the following feature will be very appreciated. For 2nd pass I'd like to see a mode (let say "cut off borders") with only one option - resolution for output file, and let Deshaker choose optimal parameters for that.

[guth]

If you select the edge compensation "fixed zoom", deshaker will add just enough zoom to completely eliminate the borders.
I have also made an "adaptive+fixed zoom" that starts by adding the usual adaptive zoom, and then adds a smaller fixed zoom to completely eliminate the borders. This feature hasn't been released yet, though.

[Dr.D]

If you select the edge compensation "fixed zoom", deshaker will add just enough zoom to completely eliminate the borders.
I have also made an "adaptive+fixed zoom" that starts by adding the usual adaptive zoom, and then adds a smaller fixed zoom to completely eliminate the borders. This feature hasn't been released yet, though.
Right, but it's not what I'm looking for. May be I've explained not good.
1. I don't need zoom, I need crop. Let say 720x480 -> 632x432.
2. This crop should comletely cut off borders. So, I need to be sure that there are no borders in the cropped clip.

Another explanation. Draw imaginery frame 632x432 inside a clip 720x480. Then do deshaking with no edge compensation and no filling borders.
In the output clip I'll see floating black frame. I need that this frame never touch the imaginery internal frame. Of course, with maximum deshaking possible.

Did you see the proDAD Mercalli plugin? There is parameter 'strength' there, you can slide this parameter and visually see what internal rectangle will be untouched. Not exactly, but close to what I'm looking for. I need opposite - set output size and let deshaker calculate strength, or whatever.

[JK1974]

Hi,

the problem is the mixture between correction limits and the motion smoothness settings. You can set the correction limits to a calculated value, e.g. if you create a border of 16x16 pixels for a PAL video (720x576), the x and y limit normally should not exceed 2%. But: This only counts for heavy movements in x or y direction. So, if just small corrections are necessary, low values like this might be enough.
But: You don´t want sudden jumps again by settings low motion smoothness settings, and then you might jump out of the border you have set above.

For getting this in a right way, I just can recommend to search for a scene with a heavy shift in one of the directions and try to set the settings accordingly - that´s how I did it, using AVISynth for cropping the image and replacing it against black borders.
BTW.: I don´t correct zooms and have a low value for rotation correction. Futhermore, I "just" edited PAL video for a long time where you have an overscan on every TV set, so even a little jump outside the 16x16 border was not visible on TV.

With my new AVCHD with 1920x1080, I have to reconsider what to do as a 32x32 border is quite to big and FullHD TVs don´t seem to have an overscan anymore.
So it would be great to be able to test this new "adaptive+fixed zoom" feature.

[guth]

Right, but it's not what I'm looking for. May be I've explained not good.
1. I don't need zoom, I need crop. Let say 720x480 -> 632x432.
2. This crop should comletely cut off borders. So, I need to be sure that there are no borders in the cropped clip.

If you use "fixed zoom", you can also adjust the output size so that you get practically no zoom. But it will be hard to get no zoom *at all* this way. Is that really a problem?

[Undead Sega]

hey, im back, how is everyone?

i have a small problem and question i would like to share and hope if someone here can help me with.

in a few shots i have made, like vox pops, interviews or just a single person on camera that occupies probably more or less 2/3 of the frame, are done handheld and is pretty shaky, therefore of course i used Deshaker, which ahs worked for me in alot of occasions however on this matter, it does stabilze the footage but not what i imagine. Because the background is quite static and the object or person is the one moving, like their body or hand hand gestures etc, it seems that Deshaker is picking the motion from the person to help stabilze the image.

i was wondering if there may be a way around this? i would really appreciate it, as i do not know exactly what to do and dont want to spend too much time experimenting on the footage as we all know the process it takes ages.

[WarpEnterprises]

Did you try the pass1-option "Ignore pixel Inside"?

[Undead Sega]

Umm no...i just looked at it now, and i see u have to fill in a few numbers, what does this option do exactly?

[Undead Sega]

also, how do u work it out? the video has people on either side of the screen, and because of the non moving background, the motion is picked up from the people's movement, therefore in a deshaked video clip, say if a person either moved/nodded their head, or waved their hands around, the camera movement will look like as if it was moving with it.

not to forget to mention, the person would appear on either side of the frame, and not just one side throughout.

does anyone get what i mean by this?

[2Bdecided]

does anyone get what i mean by this?Yes, you mean you need to buy a tripod. ;)

If there's a part of the frame where there's always background, you can force deshaker to concentrate on that.

Otherwise, I don't have a suggestion - I've not been able to solve this one other than by ignoring the centre of the frame when the problem is in the centre.

Cheers,
David.

[2Bdecided]

I asked a question a year ago...

http://forum.doom9.org/showthread.php?p=1107877#post1107877

And received a reply in the next two posts, including...

This usually means that the background will be moved correctly when taken from another frame, but if there are objects moving in the borders, they tend to look a little weird.Do I have to enable something somewhere to make this work?

What I'm seeing is that when there's up/down wobble that's being removed, then while the frame contents are moving left > right, the added border contents are not moving left > right. If the border contents are from a future frame, they will sit there, non-moving, until that frame is reached, and then obviously they'll start to move. This means the border contents don't pan, even though the rest of the frame does.

Does this explanation make sense? Is it what is supposed to happen?

Cheers,
David.

[guth]

does anyone get what i mean by this?
You can change the "ignore pixels" setting when the person moves. You don't have to run pass 1 in one go.

But if there's enough static background (must be static!) to stabilize on, you shouldn't need to use that setting. Try setting "discard motion of blocks that move > X pixels in wrong direction" to something very low, like "0.5". And also, try enabling "deep analysis". Also, make sure "remember discarded areas to next frame" is enabled.

[guth]

Does this explanation make sense? Is it what is supposed to happen?
No, it's not what's supposed to happen. The copied data of a frame should be moved so that static background (or the area that is matched on) is aligned correctly. If you look at the frame in this post:

http://forum.doom9.org/showthread.php?p=1108164#post1108164

You can see that the copied background is aligned as it should be. (But the moving person is not.)
If you still feel something is not as it should be, please email me an example.

[Undead Sega]

it helped a tiny bit, one noticable shot i know was not the same like with the default settings, however it felt it was very much like how it was shot.

to me awareness now, i find Deshaker quite slow for a 10 minute footage, would speed be soemthing of an improvement for later installments? :D

[shindou]

I kept wondering about something that was asked here a while ago - about deblurring individual frames to improve the deshaked result.

(My first idea was to use motion estimation for that, but long ago Gunnar mentioned it was too unreliable for the task.)

After a lot of googling and little success, I found this paper, High-quality Motion Deblurring from a Single Image (by Qi Shan, Jiaya Jia and Aseem Agarwala):
http://www.cse.cuhk.edu.hk/~leojia/projects/motion_deblurring/index.html

So in theory, this should be possible. The only problem is that this method (best so far it seems) takes about 40 seconds per frame... And I'm not sure if it is able to handle non-linear blur paths. Any ideas?

[mustardman]

I read the paper about motion deblurring, and was pretty impressed, but didn't understand much of the maths! I have yet to actually try the executable on some digital photos (shindou - you've done this on video frames?).

I don't quite see how the "non-blind" version would be much good for anything in the real world, only to show to an academic audience that the processing algorithm works. The real impressive work is the ability to go in "blind" and work back to a clear image from there.

@shindou: The paper claims the processor is capable of non-linear blur paths (page 6) - to be tested...

I know exactly what kind of problem you are experiencing, as I have deshaked some footage that was shot with a slow shutter (1/50th sec), and although the actual objects in the frame 'deshaked' to stationary, they exhibited a very weird looking (and entirely expected) variable blur halo around them (stationary objects and moving objects alike)!

My suggestion would be to batch process (overnight at 40 sec per frame!) the footage with this deblur tool (providing clear but shaky video) and then 'Gunnar' deshake from there.

I would be very interested in your results. I used to have the time to play with this stuff (about 4 years ago), but just don't any more :(

Another thing that interests me greatly is tools to perform deblurring that has been caused by focussing problems - especially doing this "blind". Any pointers?

Cheers,
MM

[shindou]

I haven't touched a single line of source-code, I'm just giving out ideas. :P
I said "per frame" because a frame would take the same time to process an individual image.

Just as you said, that's exactly the "problem" I've been experiencing. I'd call it a "minor issue" though, since deshaked videos are so much better than the originals. I love this plugin, now my videos have actually become intelligible. :D

It only seems to me that automated deshaking + deblurring would bring videos to the best achievable form. The non-technically-inclined user may actually expect deshaking to include deblurring since blur is also an undesired byproduct of shake.

And I agree with you, the process should be totally blind to be useful, more so for video.

I haven't tested the executable either. It may be my ignorance, but I would have done so if I knew any easy, automated way (I'm batch-processing a ton of videos) to extract frames from a video, process them (with this deblurring method or anything else, say ImageMagick (http://www.imagemagick.org)) and put the processed images back into a video stream.

NOTE: I know VirtualDub[Mod] can save a video stream as a sequence of images. Can it load a sequence of images as a video stream though?

I'd gladly try to use this method for video (even if it takes 40 sec per frame) in combination with DeShaker if possible. Like you, if I had the time, I'd work it up myself. Ideally, I think, this method of deblurring would need acceleration (probably on the GPU), otherwise any video, even short ones, would take an unbearably long time. For instance, 60 seconds of video at 30 fps would take a minimum of 20 hours to deblur.

Unfortunately I don't know any method for focal blur removal, I'm not really an expert in image/video processing. :/
The paper may contain references to works specific for this, but implementations tend to appear only years after papers are released. :/
Good luck!

[mustardman]

Not having actually tried the algorithm, but from the notes in the paper on how their executable runs, I get the impression it is very greedy on memory. That aside though, I would guess that smaller images would process faster. That is, the difference between a field of video (720x288 in PAL land) and a digital photo of 3000x2000 must have some bearing on happenings! It may also affect how good it performs too.

If anyone has tried this, I would love to hear all about it! (unfortunately, I won't have time for at least 6 months :( :( )

I think virtualdub [no-mod] can import pictures as frames, at least the older ones could. Never tried exporting, though I think it does that as well.

I think you mean 30spf, not 30fps!

Cheers,
MM

[shindou]

I mean 30 fps actually. 60 seconds of video running at 30 frames per second gives 1800 frames. If each frame takes 40 seconds to deblur, it takes a total of 72000 seconds, which is 20 hours.

[mustardman]

Cool. I see it now...

[loekverhees]

If I apply the 'fixed zoom' option, will Deshaker apply only one zoom value for the entire video (so if some 3 seconds part of a 2 hour movie is extremely shaky, a huge zoom value will be applied for the entire movie) or will it detect scenes for example and apply different fixed zoom values for the different scenes?

[guth]

Version 2.4 uses different zoom values for each individual detected scene. Older versions use the same zoom value for the entire clip.

[wonkey_monkey]

Hi Gunnar,

Something I noticed a long time ago is that Deshaker takes longer and longer to do the actual "Deshaking..." part with higher values of motion smoothness. For example (10000 frames):

Motion smoothness=100, <1s
Motion smoothness=1000, 11s
Motion smoothness=10000, >200s

I assume these values decide how many frames' worth of parameters either side are averaged to decide how to move that frame, and that it might be a Gaussian distribution to give a smooth effect. Have you considered optimizing this step? For instance, you can approximate a Gaussian "blur" of parameters like this with multiple simple box blurs, which will only take longer as the number of frames increases, not (or not much) with the width of the "blur".

I know most people won't find this a problem but if you do want to use a high value of motion smoothness, it can take a while.

David

[guth]

To generate the smooth paths I solve some large equation systems numerically, and they converge slower with higher smoothness values. I haven't gotten many complaints about it, but maybe I can do it some other way. We'll see...

[guth]

Actually, your times seem a lot higher than what's normal. Maybe you're using too low max. correction limit values? They can have a huge effect on deshaking times.

[wonkey_monkey]

I ran a default first pass (with a 1% search range), then I altered all four of the motion smoothness parameters (horizontal, vertical, rotation, zoom) to 100, 1000, or 10000 in turn.

If you remember, a long time ago I was playing around with writing a perspective-correcting deshaker, which used deshaker.log as it's input. For each frame, all I did was average the values for a number of frames on either side, then subtract that from the original value for that frame to get the parameter for adjusting the image. It seemed to come out pretty similar to Deshaker (although I should say it was a bit of a guess, I didn't know if it would work), but I didn't take it very far.

The beauty of it is that a) if the average of a parameter for frame x is calculated from by the total from p[x-w] to p[x+w], divided by 2w+1, the next average (for x+1) can be calculated by taking that total (pre-division), and subtracting p[x-w] and adding p[x+w+1]. Only the calculation for the first frame is dependent on the size of w - all other frames only need one subtraction and one addition.

And b) if you perform this averaging 3 or more times, you have a good approximation to a Gaussian distribution, which makes for a nice smooth curve.

As I said, I don't doubt that no-one else has been bothered by this, because who really needs to smooth over 10000 frames (except me ;) )?

David

[guth]

Did the video come out looking ok with the 10000 smoothness? Cause using adaptive zoom with that high smoothness, usually doesn't look good at all. If you turn off the adaptive zoom it should be faster.
I just ran a test with 10000 frames and 10000 smoothness, and it took 10 seconds to deshake. But I don't doubt it can take very long with certain clips/settings.

As for your smoothing idea, I do remember it, and I'm still not sure I like it. I don't think I want a Gaussian distribution. My algorithm tries to minimize the squared corrections (ie tries extra hard to minimize the biggest corrections) in order to minimize the ugly borders. Your algorithm doesn't even bother looking at how big the corrections become. Maybe the difference in practice is small, but still... :)

[wonkey_monkey]

I didn't actually watch the video, I was just watching how look it took to deshake :)

Deshaker is obviously more sophisticated than I assumed - not that I thought it was unsophisticated! - my averaging looked enough like Deshaker's output on my test video that I assumed it was similar. That project wasn't about the smoothing anyway, but about the correction of wide-angle frames, which benefit from being rotated in three dimensions rather than panned in two.

David

[WarpEnterprises]

It's not so easy to use the adaptive zoom correctly.
I can't think of many situations where you have to "deshake" the real zoom, so although "edge compensation adaptive zoom" is the default, maybe "adaptive zoom only" will be the more likely choice.
Then the motion smoothness for zoom must indeed be quite high (~5000) because it should resemble a slow real zoom.
But the motion smoothness for panning and rotation can be much lower (~500), only those do the real deshaking.
The lowest "nice" values can be found by leaving the zoom value 0 and using "edge compensation none", then increasing the pan/rot values until the shake is gone.
Then use the high zoom value and "adaptive zoom only".
That way only ONE of the four values is high, which makes it much faster and leaves more "space" for soft zooming.

Maybe the caption is a little bit misleading, too:
It looks like the difference between "Adaptive zoom (some borders)" and "Adaptive zoom only" is only how much borders are in the result which in fact is NOT the difference (as the manual correctly tells).

[guth]

David...
I was just trying (hard) to come up with a reason for using my algorithm instead of yours :)


Warp...
"Adaptive zoom only (some borders)" is too long, it doesn't fit. And anyway, I never meant for Deshaker to be user friendly ;)
I'll think about it, though...

As for "nice" settings, what's nice for one person might not be nice for another. I, for example, almost always use "edge compensation: none", but do use zoom smoothing (unless there's no zooming in the clip). Almost the exact opposite of you :)

[WarpEnterprises]

guth, I have one more question:
is your internal timescale/unit for motion smoothness the same for pan and zoom?
I mean, if there is a clip which shakes in X-direction with the same frequency as another clip which "shakes" in the zoom-value would I have to apply the same motion smoothness values to get similar results?

[guth]

Panning and zooming don't share the same unit, so you can't really compare them. It's like asking if a minute is longer than a meter.
Another answer is that the numbers I use internally during smoothing to represent panning is 'number of pixels', and for zoom 'the natural logarithm of the zoom factor'. Based on this, I'd say that a zoom unit would "seem" quite a lot bigger than a pixel unit, but how much depends on the size (in pixels) of a frame. That is, if they were comparable at all :)
I haven't really thought much about this. It's an interesting question... sort of...

[loekverhees]

Version 2.4 uses different zoom values for each individual detected scene. Older versions use the same zoom value for the entire clip.Ok, thanks for the answer.

[WarpEnterprises]

@guth: interesting! But I was concerned of the time dimension - if I understand correctly higher smoothness values make the resulting video shake less AND shake slower (make the resulting amplitude smaller AND the frequency lower). That way pan and zoom would be more comparable, at least the frequency. But maybe I'm simplifying things the wrong way...

[guth]

I'm not sure I understand what you're asking. Yes, higher smoothness values makes the video shake less and slower.
But you can't really separate the frequency from the amount during the smoothing. And since it's hard to compare the amount, it's hard to compare the overall effect (including frequency).
My algorithm doesn't really correspond to a lowpass filter thingy with a certain cutoff frequency, or at least I don't think so. :)

[BabaG]

just tried the deshaker on some old digitized film i have and it worked great! i'd like to try running it as a batch. i know it's not the recommended procedure but i think i could narrow down my tasks that way since the default setting worked so well on my first jitter test.

being new to this, i'm very much unclear on how to set up a batch test using the jobs function. could someone post a step-by-step on how to use jobs to execute deshaker on several files? i'd very much appreciate it.

the thing that has me most confused is how i'd run the first pass since this seems very much a manual operation the way i tested it.

thanks,
BabaG

[shindou]

just tried the deshaker on some old digitized film i have and it worked great! i'd like to try running it as a batch. i know it's not the recommended procedure but i think i could narrow down my tasks that way since the default setting worked so well on my first jitter test.

being new to this, i'm very much unclear on how to set up a batch test using the jobs function. could someone post a step-by-step on how to use jobs to execute deshaker on several files? i'd very much appreciate it.

the thing that has me most confused is how i'd run the first pass since this seems very much a manual operation the way i tested it.

thanks,
BabaG

First of all, welcome to the wonderful world of Deshaker!

Second, and sadly, I have never heard of any practical way of queueing up Deshaker jobs in batch. I've made my own scripts which would, in fact, generate the VirtualDubMod job file directly. Writing them was a trial and error job. They surely won't work for any video - just for those I take with my cam, a Canon PowerShot 720 IS. I also needed to make a variety of similar scripts just to handle slightly different movie formats. Mine will probably not work for you unmodified, unless you have the same camera. Moreover, it makes some assumptions on file naming which may add to the hassle and solve nothing for you after all.

Third, and contrary to what I did (I just needed a quick fix for 270 files), applying Deshaker in batch might not be a good idea. Several of my movies did not deshake well, even after I carefully adjusted "my own default parameters".

If you have the time, I suggest you read up the whole thread to grab a few ideas on how to find the best parameters for each of your movies, manually. If you have even more time and expertise, or a nice programmer friend with plenty of time (not the case usually), make your own scripts.

Currently I think the problem of batch deshaking is just one of several deficiencies in Deshaker (e.g. an easy, intuitive way of telling it to disconsider a specific object or other part of the image). Nonetheless, Deshaker is the best and we'd love an open-source version, which could then prompt a solution more quickly.

[guth]

could someone post a step-by-step on how to use jobs to execute deshaker on several files? i'd very much appreciate it.
Well, it's not super convenient (and not really recommended), but you can do it like this:


Setup everything as you usually do for pass 1 processing.

In the job control window, use "Edit/Process directory" to add your clips.

Setup everything as you usually do for pass 2 processing.

In the job control window, use "Edit/Process directory" to add your clips again.

In the job control window, move the clips so that you have order: clip1pass1, clip1pass2, clip2pass1, clip2pass2 etc. You need to do this since the log files will be overwritten at each pass1.

[BabaG]

thanks guth! i'll have a look at this.

BabaG

[Diamenz]

hey all. first of all guth i'd like to say how great of a plug in this is for shaky video. it really works wonders for me. i own a flip hd mino which is really bad for moving around while recording, especially when walking. unfortunately i don't know the exact rolling shutter %. the default 89% seems to work pretty well, but i emailed flip and asked so hopefully they'll reply with an answer. if they do, i'll be sure to post it here for everyone.

i have a few questions for you guth... i'm not very educated on video like a lot of the people here, but i also don't consider myself a complete newbie.

for motion smoothness, what would be general ideal settings for someone that wants to stabilize, but at the same time not lose sharpness. i don't know if it's me, but it seems the higher i set smoothness the blurrier my video becomes. or perhaps it's another setting that's causing blurriness? it's not too bad, but i'm a quality nut and it's bugging me. is the cost of stabilizing video slight loss of quality?

actually that's my only question now that i think about it. thanks!

i guess i should also mention i use no edge compensation and uncheck "ignore pixels darker that 15% brightness". everything else is left at default besides a full scale and using 'all pixels' under first pass. my source video is 1280 x 720 29.97 progressive.

[guth]

for motion smoothness, what would be general ideal settings for someone that wants to stabilize, but at the same time not lose sharpness. i don't know if it's me, but it seems the higher i set smoothness the blurrier my video becomes.
There's always a slight loss of sharpness due to the resampling process, but that's unavoidable and not dependent on the smoothness settings.

But I suspect that what you're seeing is motion blur in the frames. Deshaker can only remove unwanted motion between frames, not within frames. So if you have a slow shutter speed (slower than around 1/300 sec) there's a high risk you'll get motion blur. When you have motion both within and between frames, motion blur often looks nice, but when you remove the motion between frames, it looks bad. And in a way it might look slightly worse with higher smoothness settings.