Deshaker3D - experimental 3D image stabiliser (VDub Deshaker required)

[wonkey_monkey]

Deshaker3D download (http://horman.net/deshaker3d0.1.zip)

Well, I've been threatening to do this for years, but I've finally gone and done it.

This Avisynth filter can be used in place of pass 2 of guth's VirtualDub filter, Deshaker (http://forum.doom9.org/showthread.php?t=49597).

It implements deshaking in 3D (treating each frame as a plane in 3D space) which eliminates (or at least mitigates) the distortion that occurs when using a purely 2D process (as Deshaker does):

http://horman.net/deshaker3d.gif

YouTube video (https://www.youtube.com/watch?v=fB0Hemeh5mk)

Calling the filter requires a log file created by Deshaker pass 1 (probably to be found at C:\Users\[username]\AppData\Local\Deshaker\Deshaker.log).

Parameters are as follows:


deshaker3d(clip, int fov, string log_file, int x, int y, int z)

# fov - horizontal FOV of the image (non-constant FOV clips may not work well)
# log_file - path to Deshaker pass 1 log file
# x - amount of horizontal smoothing
# y - amount of vertical smoothing
# z - amount of rotational smoothing


Input clips must be converted to RGB32.

Omitting the y and z parameters will cause all smoothing to use the x parameter. If you don't want a particular kind of smoothing, set to 0.

Smoothing parameter values don't match those used by Deshaker, but higher numbers=smoother. The maximum effective value is approximately num_frames/2.

Deshaker3D doesn't implement a lot of Deshaker's functionality for now, but may implement more in future.

[johnmeyer]

This looks very interesting. Does it retain the Deshaker rolling shutter "fix?"

[cretindesalpes]

Nice. No source code?

[wonkey_monkey]

This looks very interesting. Does it retain the Deshaker rolling shutter "fix?"'

Not yet. It might not be too hard to do though.

Nice. No source code?

Maybe one day, when I can pare down my image library a bit. It's currently a mess of files scattered across my hard drive.

[Reel.Deel]

Thanks, nice to see another stabilization plugin for Avisynth. I quickly played around with this but did not get to work right. Output seems similar to the input :confused:. What are the defaults (or recommended values) for "fov" and "x"? I know VDub Deshaker requires RGB32 but is there any possibly of adding support for YV12 in Deshaker3D?

[wonkey_monkey]

I can add "support" in the sense of internally calling ConvertToRGB32 when required, and calling ConvertToYV12 at the end ;)

FOV for a "normal" camera would be somewhere around 45-60 degrees. x depends on how much smoothing you want - you should certainly see some difference with x=100. If not, can you post your script and your Deshaker logfile?

[Reel.Deel]

I can add "support" in the sense of internally calling ConvertToRGB32 when required, and calling ConvertToYV12 at the end ;)

No worries, I can add support for it myself :)


FOV for a "normal" camera would be somewhere around 45-60 degrees. x depends on how much smoothing you want - you should certainly see some difference with x=100.

Yeah I was using too low of a value for "fov", setting it to 50 and x=100 I can definitely see a difference.

[Undead Sega]

This is mighty impressive!

Has there been any developments on this???

[wonkey_monkey]

This is mighty impressive!

Thanks!

Has there been any developments on this???

Nope...

[Undead Sega]

Why not?

I have not seen anything like this anywhere, so this must be a first right???

[shekh]

Hi, I am actually making similar thing as a VirtualDub filter.

I have thoughts about rolling shutter compensation. This thing is quite subtle and Deshaker motion estimation does not seem to be accurate with wide angle material. So I have doubts whether it is worth the attempt. Ideally it needs dedicated motion estimation which can handle camera model, but this is so much more work to do.

[shekh]

Hi, I am actually making similar thing as a VirtualDub filter.


Just begin to post about it here: http://forum.doom9.org/showthread.php?p=1747268#post1747268

[StainlessS]

Very cool, is the sourcecode available anywhere yet?


++ on that, Pretty Please David :)

Oops, Yeshua was addressing Shekh, I was addressing David.
Yeshua, Shekh posted link to source in earlier Shekh linked thread.

[shekh]

I know this is a quite old post but shekh i am having the same issue as you, have you find to make it work.

By the same issue you mean you also attempt to implement deshaking algorithm?
The estimation is not big problem. What I tried and it was good:
step 1: (same as 2d case) collect motion vectors grid
step 2: find rotation quaternion that resolves motion vectors (It is trivially derived by gradient descent method)

[TCmullet]

Need a 2d (or 3d) deshaker for Avisynth. (Will use Vdub as front end for Avisynth output, but other than that, no Vdub involvement feeding Avisynth.) Thank you for any reference. I've not been able to find anything.

[wonkey_monkey]

DePan?

http://avisynth.org.ru/depan/depan.html

[shekh]

Just saw posts 13, 14 are abused with garbage links.

[MODERATOR EDIT]
Original posts 13, 14 deleted by spam

[nji]

Hi shekh.

I would like to ask if there is an up-to-date version of your plugin.
(Only the plugin and the needed files for ini etc. for use in VDub2)

I use DeShaker for a long time, but had to make the experience
that with large FOV it produces "wobbling" results (due to the
lack of taking perspective into account) that make the result
worse than the shaking version.

In Deshaker - as a first step - a simple option for FOV
(wide-angle, normal, tele) would be fantastic.
But ... sadly ... DeShaker isn't improved anymore.

As far as I understand the matter, BEFORE phase 1 in DeShaker
there would have to be the perspective correction (panotools etc.?)
and after that to calculate the motion vectors etc.

But maybe your plugin works well enough already?

Greetings! :-)

[wonkey_monkey]

There's no update. It's on my list now I have a much faster remapper but it's a low priority. There isn't really any sensible pre-correction you can do, because Deshaker always treats the input as a 2D image.

[nji]

Hi wonkey_monkey.

Thankyou for your reply.

Actually I posted to the right addressee (shekh), but into the wrong thread.
I meant the VDub plugin (https://forum.doom9.org/showthread.php?t=172886).
I'm sorry.

[johnmeyer]

While it is far from free, Mercalli is an amazing stabilizer. Try out the demo.

[Rob105]

Calling the filter requires a log file created by Deshaker pass 1

https://i.postimg.cc/h4zxj8mr/1.jpg

All that Deshaker log file has for that frame is
Move X = 2.82
Move Y = -1.31
Rotate= 0.080
Zoom = 0.99837

How are you able to make such complex geometrical distortions as seen on the left side of the image, namely line from lower left to upper right corner is not straight.

[nji]

Framecount mismatch
Calling the filter a dialog pops up:
"Log file doesn't have enough frames".

The steps:
Testfile: https://c.gmx.net/@1155842887640945833/TRFrX0XNjbnXtQYaQy66eg
Run VD2 filter DeShaker, 1st pass produces logfile
Run script:

DirectShowSource("testDeshaker3D.mp4")

Subtitle(string(Framecount()))

clip = ConvertToRGB32()

deshaker3d(clip, fov=100, log_file="Deshaker.log", x=100)

Dialog pops up.

If I enter 1 additional line at head of log file,
say at... ehm... first pos, like:

0 skipped # 0.00 0.00


then deshaker3d is satisfied.

What is going wrong here?

First:
How many frames does the movie actually contain?
VD2 says 508 (frame line from 0 to 508, while 508 is blind (= no frame)).
Deshaker's produces difference lines, that is framecount-1.
The actual log file contains 507 lines (1 to 507).
This is consistent.
But AVS says there are 509 frames.

A mismatch of 1 frame is desaster for deshaking.

Any idea anyone?

AviSynthPlus_3.7.5_20250420_xp @ Win7

[wonkey_monkey]

DirectShowSource wouldn't process the clip properly for me. Initially the first frame was grey, and subsequent frames were all identical. Going back to frame 0 showed the same identical frame. It did say 508 frames though, not 509.

Likely a DirectShowSource decoding issue returning the wrong number of frames in some cases. Try ffmpegsource2 instead. Or better still, BestSource2 (https://forum.doom9.org/showthread.php?p=2000872).

[nji]

It's strange that DirectShowSource doesn't process the frames correctly for you...
DirectShowSource should't be as bad, should it?

I never had any problems with it. All frames show fine.

(BTW All 3 movies I checked have the same mismatch effect)

If I get it right the source filters you recommend have prebuilded only 64 bit.
And your DeShaker3d is only 32 bit binary (no source)
Looks like cul-de-sac.

I checked: Deshaker, part1, on AVS DirectShowSource input:
All identical, just last line is "missing".
So adding it will be my workaround.

[nji]

Got a second strange effect.

If deshake3d the movie from above (and some other movies I testet)
and do "border filling" with Feedback filter (https://forum.doom9.org/showthread.php?t=182910):

... see start of script above ...

stab = deshaker3d(clip, fov=100, log_file="Deshaker.log", x=100)

Layer(stab.FeedbackInput(), stab).FeedbackOutput()



there are black lines at the border of the transformed frames.
https://c.gmx.net/@1155842887640945833/nFsGf-7DF8CafdvnUch6CA


Of course I checked... they aren't in the frames originally.

A bug of deshaker3D or of feedback filter?

[wonkey_monkey]

Try calling Unpremultiply (http://avisynth.nl/index.php/Unpremultiply) on the unshaken clip before calling Layer.

[nji]

All the same.

The script in it's entirety's beauty:

DirectShowSource("testDeshaker3D.mp4")

clip = ConvertToRGB32()

Unpremultiply(clip)

stab = deshaker3d(clip, fov=100, log_file="Deshaker.log", x=100)

Layer(stab.FeedbackInput(), stab).FeedbackOutput()

As I provided movie, script etc. not just for fun:

Do you confirm the effect?
Does anybody confirm the effect?
(Is there anybody out there?)

[wonkey_monkey]

Unpremultiply(clip)

This runs the filter but doesn't assign the result to anything. Also, you have to unpremultiply the stabilised clip, not the input clip.

DirectShowSource("testDeshaker3D.mp4")

clip = ConvertToRGB32()

stab = deshaker3d(clip, fov=100, log_file="Deshaker.log", x=100).Unpremultiply

Layer(stab.FeedbackInput(), stab).FeedbackOutput()

[nji]

Unpremultiply(clip)

This runs the filter but doesn't assign the result to anything. Also, you have to unpremultiply the stabilised clip, not the input clip.

Oh... right...
(Now you know I don't use AVS so often (so far) ;)

Thank you!
Now the borders of the layed-in transformed frame are as soft as in your example. (However: Still percetible grayish; esp. if the outer area gets "older".)

May I ask for a hint what was wrong with the movie to be "unpremultiplied" (whatever that may be)?

[wonkey_monkey]

https://en.wikipedia.org/wiki/Alpha_compositing#Straight_versus_premultiplied

[nji]

OK, got it, thank you.

So I wonder:

Is it uncommon to have premultiplied data?
From orig movie, or from DirectShowSource().

If not there maybe should be a hint in Feedback() (https://forum.doom9.org/showthread.php?t=182910) to have data
unpremultiplied before using the Layer trick...

BTW
I wonder why your filter doesn't seem to be much more popular...

As there is only just ONE other filter to account 3D:
"lens transform (https://forum.doom9.org/showthread.php?t=172886)" for VirtualDub2.

Doing "planar stabilizing" with Deshaker only always produces magic mushroom effect;
most inacceptable for larger FoV, shaking and/or smoothing.
Stabilizing small trembling maybe acceptable... maybe....

[wonkey_monkey]

The premultiplication is introduced by Deshaker3D, as it remaps the (default) white alpha channel. The eventual future version will includes options to account for it.

[nji]

So I wanted to check once for all which 3D stabilizer to use now.

Compared the only two 3D stabilizers at this time (using some 70° FoV movies):

Deshaker3D (DS3D) vs. lens transform (https://forum.doom9.org/showthread.php?t=172886) (LT)

Analysing: both are based on Deshaker (https://www.guthspot.se/video/deshaker.htm)'s 1st pass.
Sources: both don't provide.
Multithread: both do.

Their differences (important bad ones marked red):

Binaries: DS3D: only 32bit, LT: both 32/64bit
Speed: DS3D is abt. 11% slower.
Stabilizing algs: different, obviously.
Interpolation: DS3D: cubic, LT: linear
Border filling: DS3D: content from former frames ditched (partwise non-matching older content); also sometimes accumulation slight canvas lines
LT: medium quality inpaint. In general more prominent than DS3D's filling. Crashes at extreme shaking (needs deactivation around those pos.)
Zoom correction: DS3D: none, even if analysis data have been calculated based on zoom, LT: yes
Stab quality: Difficult to say if one is better.
If logo: Removal necessary for both before analysis. DS3D: needs additional removal.
Etc. : DS3D: No frame positioning in editor possible: crashes (by design).
Etc. : DS3D: Color error: slightly darker, more blue-ish

Quintessence for me: Will use lens transform (having to accept the blurring (lin. interpol.) and sub-optimal border filling.) :(

EDIT:
Due to more test cases I had to alter the table very much