Deshaker by Gunnar Thalin - Page 9

guth · 29th September 2008, 17:20

If you select the edge compensation "fixed zoom", deshaker will add just enough zoom to completely eliminate the borders.
I have also made an "adaptive+fixed zoom" that starts by adding the usual adaptive zoom, and then adds a smaller fixed zoom to completely eliminate the borders. This feature hasn't been released yet, though.

Dr.D · 30th September 2008, 08:09

Quote:

Originally Posted by guth

If you select the edge compensation "fixed zoom", deshaker will add just enough zoom to completely eliminate the borders.
I have also made an "adaptive+fixed zoom" that starts by adding the usual adaptive zoom, and then adds a smaller fixed zoom to completely eliminate the borders. This feature hasn't been released yet, though.

Right, but it's not what I'm looking for. May be I've explained not good.
1. I don't need zoom, I need crop. Let say 720x480 -> 632x432.
2. This crop should comletely cut off borders. So, I need to be sure that there are no borders in the cropped clip.

Another explanation. Draw imaginery frame 632x432 inside a clip 720x480. Then do deshaking with no edge compensation and no filling borders.
In the output clip I'll see floating black frame. I need that this frame never touch the imaginery internal frame. Of course, with maximum deshaking possible.

Did you see the proDAD Mercalli plugin? There is parameter 'strength' there, you can slide this parameter and visually see what internal rectangle will be untouched. Not exactly, but close to what I'm looking for. I need opposite - set output size and let deshaker calculate strength, or whatever.

JK1974 · 30th September 2008, 09:37

Hi,

the problem is the mixture between correction limits and the motion smoothness settings. You can set the correction limits to a calculated value, e.g. if you create a border of 16x16 pixels for a PAL video (720x576), the x and y limit normally should not exceed 2%. But: This only counts for heavy movements in x or y direction. So, if just small corrections are necessary, low values like this might be enough.
But: You don´t want sudden jumps again by settings low motion smoothness settings, and then you might jump out of the border you have set above.

For getting this in a right way, I just can recommend to search for a scene with a heavy shift in one of the directions and try to set the settings accordingly - that´s how I did it, using AVISynth for cropping the image and replacing it against black borders.
BTW.: I don´t correct zooms and have a low value for rotation correction. Futhermore, I "just" edited PAL video for a long time where you have an overscan on every TV set, so even a little jump outside the 16x16 border was not visible on TV.

With my new AVCHD with 1920x1080, I have to reconsider what to do as a 32x32 border is quite to big and FullHD TVs don´t seem to have an overscan anymore.
So it would be great to be able to test this new "adaptive+fixed zoom" feature.

guth · 30th September 2008, 17:25

Quote:

Originally Posted by Dr.D

Right, but it's not what I'm looking for. May be I've explained not good.
1. I don't need zoom, I need crop. Let say 720x480 -> 632x432.
2. This crop should comletely cut off borders. So, I need to be sure that there are no borders in the cropped clip.

If you use "fixed zoom", you can also adjust the output size so that you get practically no zoom. But it will be hard to get no zoom *at all* this way. Is that really a problem?

Undead Sega · 30th March 2009, 19:52

hey, im back, how is everyone?

i have a small problem and question i would like to share and hope if someone here can help me with.

in a few shots i have made, like vox pops, interviews or just a single person on camera that occupies probably more or less 2/3 of the frame, are done handheld and is pretty shaky, therefore of course i used Deshaker, which ahs worked for me in alot of occasions however on this matter, it does stabilze the footage but not what i imagine. Because the background is quite static and the object or person is the one moving, like their body or hand hand gestures etc, it seems that Deshaker is picking the motion from the person to help stabilze the image.

i was wondering if there may be a way around this? i would really appreciate it, as i do not know exactly what to do and dont want to spend too much time experimenting on the footage as we all know the process it takes ages.

WarpEnterprises · 30th March 2009, 20:46

Did you try the pass1-option "Ignore pixel Inside"?

Undead Sega · 30th March 2009, 21:11

Umm no...i just looked at it now, and i see u have to fill in a few numbers, what does this option do exactly?

Undead Sega · 31st March 2009, 14:13

also, how do u work it out? the video has people on either side of the screen, and because of the non moving background, the motion is picked up from the people's movement, therefore in a deshaked video clip, say if a person either moved/nodded their head, or waved their hands around, the camera movement will look like as if it was moving with it.

not to forget to mention, the person would appear on either side of the frame, and not just one side throughout.

does anyone get what i mean by this?

2Bdecided · 31st March 2009, 14:26

Quote:

Originally Posted by Undead Sega

does anyone get what i mean by this?

Yes, you mean you need to buy a tripod.

If there's a part of the frame where there's always background, you can force deshaker to concentrate on that.

Otherwise, I don't have a suggestion - I've not been able to solve this one other than by ignoring the centre of the frame when the problem is in the centre.

Cheers,
David.

2Bdecided · 31st March 2009, 14:36

I asked a question a year ago...

http://forum.doom9.org/showthread.ph...77#post1107877

And received a reply in the next two posts, including...

Quote:

Originally Posted by guth

This usually means that the background will be moved correctly when taken from another frame, but if there are objects moving in the borders, they tend to look a little weird.

Do I have to enable something somewhere to make this work?

What I'm seeing is that when there's up/down wobble that's being removed, then while the frame contents are moving left > right, the added border contents are not moving left > right. If the border contents are from a future frame, they will sit there, non-moving, until that frame is reached, and then obviously they'll start to move. This means the border contents don't pan, even though the rest of the frame does.

Does this explanation make sense? Is it what is supposed to happen?

Cheers,
David.

guth · 31st March 2009, 16:23

Quote:

Originally Posted by Undead Sega

does anyone get what i mean by this?

You can change the "ignore pixels" setting when the person moves. You don't have to run pass 1 in one go.

But if there's enough static background (must be static!) to stabilize on, you shouldn't need to use that setting. Try setting "discard motion of blocks that move > X pixels in wrong direction" to something very low, like "0.5". And also, try enabling "deep analysis". Also, make sure "remember discarded areas to next frame" is enabled.

guth · 31st March 2009, 16:31

Quote:

Originally Posted by 2Bdecided

Does this explanation make sense? Is it what is supposed to happen?

No, it's not what's supposed to happen. The copied data of a frame should be moved so that static background (or the area that is matched on) is aligned correctly. If you look at the frame in this post:

http://forum.doom9.org/showthread.ph...64#post1108164

You can see that the copied background is aligned as it should be. (But the moving person is not.)
If you still feel something is not as it should be, please email me an example.

Undead Sega · 2nd April 2009, 21:02

it helped a tiny bit, one noticable shot i know was not the same like with the default settings, however it felt it was very much like how it was shot.

to me awareness now, i find Deshaker quite slow for a 10 minute footage, would speed be soemthing of an improvement for later installments?

shindou · 22nd July 2009, 15:24

I kept wondering about something that was asked here a while ago - about deblurring individual frames to improve the deshaked result.

(My first idea was to use motion estimation for that, but long ago Gunnar mentioned it was too unreliable for the task.)

After a lot of googling and little success, I found this paper, High-quality Motion Deblurring from a Single Image (by Qi Shan, Jiaya Jia and Aseem Agarwala):
http://www.cse.cuhk.edu.hk/~leojia/p...ing/index.html

So in theory, this should be possible. The only problem is that this method (best so far it seems) takes about 40 seconds per frame... And I'm not sure if it is able to handle non-linear blur paths. Any ideas?

mustardman · 5th August 2009, 10:47

I read the paper about motion deblurring, and was pretty impressed, but didn't understand much of the maths! I have yet to actually try the executable on some digital photos (shindou - you've done this on video frames?).

I don't quite see how the "non-blind" version would be much good for anything in the real world, only to show to an academic audience that the processing algorithm works. The real impressive work is the ability to go in "blind" and work back to a clear image from there.

@shindou: The paper claims the processor is capable of non-linear blur paths (page 6) - to be tested...

I know exactly what kind of problem you are experiencing, as I have deshaked some footage that was shot with a slow shutter (1/50th sec), and although the actual objects in the frame 'deshaked' to stationary, they exhibited a very weird looking (and entirely expected) variable blur halo around them (stationary objects and moving objects alike)!

My suggestion would be to batch process (overnight at 40 sec per frame!) the footage with this deblur tool (providing clear but shaky video) and then 'Gunnar' deshake from there.

I would be very interested in your results. I used to have the time to play with this stuff (about 4 years ago), but just don't any more

Another thing that interests me greatly is tools to perform deblurring that has been caused by focussing problems - especially doing this "blind". Any pointers?

Cheers,
MM

shindou · 18th August 2009, 18:02

I haven't touched a single line of source-code, I'm just giving out ideas. :P
I said "per frame" because a frame would take the same time to process an individual image.

Just as you said, that's exactly the "problem" I've been experiencing. I'd call it a "minor issue" though, since deshaked videos are so much better than the originals. I love this plugin, now my videos have actually become intelligible.

It only seems to me that automated deshaking + deblurring would bring videos to the best achievable form. The non-technically-inclined user may actually expect deshaking to include deblurring since blur is also an undesired byproduct of shake.

And I agree with you, the process should be totally blind to be useful, more so for video.

I haven't tested the executable either. It may be my ignorance, but I would have done so if I knew any easy, automated way (I'm batch-processing a ton of videos) to extract frames from a video, process them (with this deblurring method or anything else, say ImageMagick) and put the processed images back into a video stream.

NOTE: I know VirtualDub[Mod] can save a video stream as a sequence of images. Can it load a sequence of images as a video stream though?

I'd gladly try to use this method for video (even if it takes 40 sec per frame) in combination with DeShaker if possible. Like you, if I had the time, I'd work it up myself. Ideally, I think, this method of deblurring would need acceleration (probably on the GPU), otherwise any video, even short ones, would take an unbearably long time. For instance, 60 seconds of video at 30 fps would take a minimum of 20 hours to deblur.

Unfortunately I don't know any method for focal blur removal, I'm not really an expert in image/video processing. :/
The paper may contain references to works specific for this, but implementations tend to appear only years after papers are released. :/
Good luck!

mustardman · 19th August 2009, 12:56

Not having actually tried the algorithm, but from the notes in the paper on how their executable runs, I get the impression it is very greedy on memory. That aside though, I would guess that smaller images would process faster. That is, the difference between a field of video (720x288 in PAL land) and a digital photo of 3000x2000 must have some bearing on happenings! It may also affect how good it performs too.

If anyone has tried this, I would love to hear all about it! (unfortunately, I won't have time for at least 6 months

)

I think virtualdub [no-mod] can import pictures as frames, at least the older ones could. Never tried exporting, though I think it does that as well.

I think you mean 30spf, not 30fps!

Cheers,
MM

shindou · 19th August 2009, 23:17

I mean 30 fps actually. 60 seconds of video running at 30 frames per second gives 1800 frames. If each frame takes 40 seconds to deblur, it takes a total of 72000 seconds, which is 20 hours.

mustardman · 20th August 2009, 10:08

Cool. I see it now...

loekverhees · 22nd September 2009, 19:07

If I apply the 'fixed zoom' option, will Deshaker apply only one zoom value for the entire video (so if some 3 seconds part of a 2 hour movie is extremely shaky, a huge zoom value will be applied for the entire movie) or will it detect scenes for example and apply different fixed zoom values for the different scenes?

29th September 2008, 17:20	#161 \| Link
guth Registered User Join Date: Apr 2003 Location: Uppsala, Sweden Posts: 157	If you select the edge compensation "fixed zoom", deshaker will add just enough zoom to completely eliminate the borders. I have also made an "adaptive+fixed zoom" that starts by adding the usual adaptive zoom, and then adds a smaller fixed zoom to completely eliminate the borders. This feature hasn't been released yet, though.

30th September 2008, 09:37	#163 \| Link
JK1974 Registered User Join Date: Mar 2005 Posts: 89	Hi, the problem is the mixture between correction limits and the motion smoothness settings. You can set the correction limits to a calculated value, e.g. if you create a border of 16x16 pixels for a PAL video (720x576), the x and y limit normally should not exceed 2%. But: This only counts for heavy movements in x or y direction. So, if just small corrections are necessary, low values like this might be enough. But: You don´t want sudden jumps again by settings low motion smoothness settings, and then you might jump out of the border you have set above. For getting this in a right way, I just can recommend to search for a scene with a heavy shift in one of the directions and try to set the settings accordingly - that´s how I did it, using AVISynth for cropping the image and replacing it against black borders. BTW.: I don´t correct zooms and have a low value for rotation correction. Futhermore, I "just" edited PAL video for a long time where you have an overscan on every TV set, so even a little jump outside the 16x16 border was not visible on TV. With my new AVCHD with 1920x1080, I have to reconsider what to do as a 32x32 border is quite to big and FullHD TVs don´t seem to have an overscan anymore. So it would be great to be able to test this new "adaptive+fixed zoom" feature.

30th March 2009, 19:52	#165 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	hey, im back, how is everyone? i have a small problem and question i would like to share and hope if someone here can help me with. in a few shots i have made, like vox pops, interviews or just a single person on camera that occupies probably more or less 2/3 of the frame, are done handheld and is pretty shaky, therefore of course i used Deshaker, which ahs worked for me in alot of occasions however on this matter, it does stabilze the footage but not what i imagine. Because the background is quite static and the object or person is the one moving, like their body or hand hand gestures etc, it seems that Deshaker is picking the motion from the person to help stabilze the image. i was wondering if there may be a way around this? i would really appreciate it, as i do not know exactly what to do and dont want to spend too much time experimenting on the footage as we all know the process it takes ages.

30th March 2009, 20:46	#166 \| Link
WarpEnterprises C64 Join Date: Apr 2002 Location: Austria Posts: 830	Did you try the pass1-option "Ignore pixel Inside"?

30th March 2009, 21:11	#167 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	Umm no...i just looked at it now, and i see u have to fill in a few numbers, what does this option do exactly?

31st March 2009, 14:13	#168 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	also, how do u work it out? the video has people on either side of the screen, and because of the non moving background, the motion is picked up from the people's movement, therefore in a deshaked video clip, say if a person either moved/nodded their head, or waved their hands around, the camera movement will look like as if it was moving with it. not to forget to mention, the person would appear on either side of the frame, and not just one side throughout. does anyone get what i mean by this?

2nd April 2009, 21:02	#173 \| Link
Undead Sega Registered User Join Date: Oct 2007 Posts: 713	it helped a tiny bit, one noticable shot i know was not the same like with the default settings, however it felt it was very much like how it was shot. to me awareness now, i find Deshaker quite slow for a 10 minute footage, would speed be soemthing of an improvement for later installments?

22nd July 2009, 15:24	#174 \| Link
shindou Registered User Join Date: Jul 2009 Posts: 4	I kept wondering about something that was asked here a while ago - about deblurring individual frames to improve the deshaked result. (My first idea was to use motion estimation for that, but long ago Gunnar mentioned it was too unreliable for the task.) After a lot of googling and little success, I found this paper, High-quality Motion Deblurring from a Single Image (by Qi Shan, Jiaya Jia and Aseem Agarwala): http://www.cse.cuhk.edu.hk/~leojia/p...ing/index.html So in theory, this should be possible. The only problem is that this method (best so far it seems) takes about 40 seconds per frame... And I'm not sure if it is able to handle non-linear blur paths. Any ideas?

5th August 2009, 10:47	#175 \| Link
mustardman Registered User Join Date: Dec 2002 Location: Australia Posts: 264	I read the paper about motion deblurring, and was pretty impressed, but didn't understand much of the maths! I have yet to actually try the executable on some digital photos (shindou - you've done this on video frames?). I don't quite see how the "non-blind" version would be much good for anything in the real world, only to show to an academic audience that the processing algorithm works. The real impressive work is the ability to go in "blind" and work back to a clear image from there. @shindou: The paper claims the processor is capable of non-linear blur paths (page 6) - to be tested... I know exactly what kind of problem you are experiencing, as I have deshaked some footage that was shot with a slow shutter (1/50th sec), and although the actual objects in the frame 'deshaked' to stationary, they exhibited a very weird looking (and entirely expected) variable blur halo around them (stationary objects and moving objects alike)! My suggestion would be to batch process (overnight at 40 sec per frame!) the footage with this deblur tool (providing clear but shaky video) and then 'Gunnar' deshake from there. I would be very interested in your results. I used to have the time to play with this stuff (about 4 years ago), but just don't any more Another thing that interests me greatly is tools to perform deblurring that has been caused by focussing problems - especially doing this "blind". Any pointers? Cheers, MM

18th August 2009, 18:02	#176 \| Link
shindou Registered User Join Date: Jul 2009 Posts: 4	I haven't touched a single line of source-code, I'm just giving out ideas. :P I said "per frame" because a frame would take the same time to process an individual image. Just as you said, that's exactly the "problem" I've been experiencing. I'd call it a "minor issue" though, since deshaked videos are so much better than the originals. I love this plugin, now my videos have actually become intelligible. It only seems to me that automated deshaking + deblurring would bring videos to the best achievable form. The non-technically-inclined user may actually expect deshaking to include deblurring since blur is also an undesired byproduct of shake. And I agree with you, the process should be totally blind to be useful, more so for video. I haven't tested the executable either. It may be my ignorance, but I would have done so if I knew any easy, automated way (I'm batch-processing a ton of videos) to extract frames from a video, process them (with this deblurring method or anything else, say ImageMagick) and put the processed images back into a video stream. NOTE: I know VirtualDub[Mod] can save a video stream as a sequence of images. Can it load a sequence of images as a video stream though? I'd gladly try to use this method for video (even if it takes 40 sec per frame) in combination with DeShaker if possible. Like you, if I had the time, I'd work it up myself. Ideally, I think, this method of deblurring would need acceleration (probably on the GPU), otherwise any video, even short ones, would take an unbearably long time. For instance, 60 seconds of video at 30 fps would take a minimum of 20 hours to deblur. Unfortunately I don't know any method for focal blur removal, I'm not really an expert in image/video processing. :/ The paper may contain references to works specific for this, but implementations tend to appear only years after papers are released. :/ Good luck!

19th August 2009, 12:56	#177 \| Link
mustardman Registered User Join Date: Dec 2002 Location: Australia Posts: 264	Not having actually tried the algorithm, but from the notes in the paper on how their executable runs, I get the impression it is very greedy on memory. That aside though, I would guess that smaller images would process faster. That is, the difference between a field of video (720x288 in PAL land) and a digital photo of 3000x2000 must have some bearing on happenings! It may also affect how good it performs too. If anyone has tried this, I would love to hear all about it! (unfortunately, I won't have time for at least 6 months ) I think virtualdub [no-mod] can import pictures as frames, at least the older ones could. Never tried exporting, though I think it does that as well. I think you mean 30spf, not 30fps! Cheers, MM

19th August 2009, 23:17	#178 \| Link
shindou Registered User Join Date: Jul 2009 Posts: 4	I mean 30 fps actually. 60 seconds of video running at 30 frames per second gives 1800 frames. If each frame takes 40 seconds to deblur, it takes a total of 72000 seconds, which is 20 hours.

20th August 2009, 10:08	#179 \| Link
mustardman Registered User Join Date: Dec 2002 Location: Australia Posts: 264	Cool. I see it now...

22nd September 2009, 19:07	#180 \| Link
loekverhees Registered User Join Date: Sep 2005 Location: Holland Posts: 86	If I apply the 'fixed zoom' option, will Deshaker apply only one zoom value for the entire video (so if some 3 seconds part of a 2 hour movie is extremely shaky, a huge zoom value will be applied for the entire movie) or will it detect scenes for example and apply different fixed zoom values for the different scenes?