Lens Transform tool (work in progress)

[redfordxx]

Sounds good. A few notes:
I dont understand how defish uses fov parameter.
Uses only a b c

Similarly, defish does not tell you the new fov for resulting image.

The defish adjustment is 12 black pixel in the corner.
So the resulting fov will be pretty much the same...or 1% change of the PTGui FoV.

So you should crop or scale the result in such way that pixels at positions (0,h/2) and (w,h/2) remain in same place.
I do not follow.
I thought (w/2, h/2) should remain in the center.

[shekh]

It is, because the input and output projections are assumed to be fisheye and rectilinear.

You mean in this fisheye model the "amount of barrel" is connected to fov? ok


if the aim is to find a projection where 3D rotation can be eliminated by changing the projection, translating the image in 2D, and then changing the projection back, then no such projection exists.

Just for fun: what if we allow 3 passes (one for each axis) :)

[shekh]

I thought (w/2, h/2) should remain in the center.

This is not what I wrote.

In this example fov is reduced (and no way to know how much).
https://i.postimg.cc/ydN9J4XR/defish.png

[wonkey_monkey]

By default, Defish keeps the centre of the image at the same scale. That means some cropping. This can be overridden by setting the "scaling" parameter to "fitx", "fity", or "fitxy".

[wonkey_monkey]

Just for fun: what if we allow 3 passes (one for each axis) :)

Z is just a rotation of the image, so it doesn't need a "pass", but otherwise that might work. It's not very "pretty," though, when the image could be manipulated in one pass in 3D (to generate a 2D mapping).

[nji]

...

Having vaguely followed this rather meandering thread

... yes, that's pretty true :-)

I can only speak for myself ... of course.

And I used orig Deshaker since some time to my very satisfaction.
Actually get kind of "addicted" to it ...
as a calm camera is a relieve when watching.

However I made the observation that if the movie has been taken
by a lens with large fov then the result of orig Deshaker
sometimes is worse than the shaky original movie (like on LSD etc. :))
Clearly that is because Deshaker assumes/ works 2D only.

So the simple question was if it would be possible to take
into account the 3D origin of the movie somehow for large fovs.

Your DeShaker3D obviously does it although it has some severe
drawbacks (if I get it right)
https://forum.doom9.org/showthread.php?p=1895158#post1895158

I thought that Lens Transform would do better.
Lens Transform is also for the specific distortions of lenses,
but personally I'm not about enhancing of a device I own,
but about older movies where I only can tell roughly fov.
And with that indoor movies (family feasts etc.)
obviously taken with larger fov (but not fisheye),
Lens Transform does't do good either.
It does good for fisheye ... to my experience.
I cannot tell what's the reason for that.
Is it "by design" as to the fisheye projection?
(And if ... could it be improved by offering for "normal lenses" too?)
Or is it because panning and tilting (redfordxx called it rotation x and y)
have more effects the larger the fov?

All in all ... just a simple question from my side:

How do I deshake large fov (not fisheye) movies?
If not possible with up to now:
Which improvements could be done ...
(at Lens Transform ... as the alternatives do not do (?))

[shekh]

nji
"doesn't do good" is void discussion unless you demonstrate specific shots and results.
In pure math adding fisheye is more complex so "flat" lens can't be worse. But the perceived quality can be ruined by many factors which are subject to research.
Are you trying to find real software for the job? Have you tried Mercalli or anything else?

[nji]

"doesn't do good" is void discussion unless you demonstrate specific shots and results.


I think it is well known what kind of "wobbling", "LSD-effect" you get when deshaking large fov movies.


In pure math adding fisheye is more complex so "flat" lens can't be worse.


I do not agree.
It is not: The more complex the better.
It is: The more suitable the better,


But the perceived quality can be ruined by many factors which are subject to research.


Don't believe that either.
I'm pretty sure it's simple projection math (I sadly am not able to).
Research ... in the 17th century perhaps :)


Are you trying to find real software for the job? Have you tried Mercalli or anything else?

What do you mean by: "real software"?
Only software that costs LOTS of money shall be taken serious?

[poisondeathray]

I think it is well known what kind of "wobbling", "LSD-effect" you get when deshaking large fov movies.


Did you mean rolling shutter artifacts ?

Or that the "wobbling" was introduced by the compensatory stabilization method ?

What kind of "wobbling?" exactly ? There are different types .

Or some combination?

[nji]

Did you mean rolling shutter artifacts ?

Or that the "wobbling" was introduced by the compensatory stabilization method ?

What kind of "wobbling?" exactly ? There are different types .

Or some combination?

Try to deshake a global shutter large fov movie...
to see what this thread is about.

[poisondeathray]

Try to deshake a global shutter large fov movie...
to see what this thread is about.


This thread is about lens correction.

After you "undo" the lens distortion - how is this any different than other scenarios?

I'm simply asking you to clarify what you mean by "wobbling"

[nji]

Yes, you are right, the thread is about lens correction (tool).

And we're discussing how to make it useable not only
for well known lenses but also if only known of a large fov.
For in that case all deshaker do bad.

It's difficult for me to describe what the effect looks like
that I called "wobbling".
Actually "wobbling" seems not to be the right English word for it.
(I don't know a fitting word).
It's more like "kneads".
And it seems to be caused when tilting and panning
a large fov movie.
Why don't you have a look for yourself as I suggested?

[redfordxx]

If the wobbling is panning and tilting the camera... or as I said rotation around x and y axis...

Can someone stronger than me in math help me?

When I rotate camera precisely around x axis...
...does quad help me with symetric transform (zoom bottom or top of the image) to compensate completely...or there is some curve involved?

[redfordxx]

In this example fov is reduced (and no way to know how much).
https://i.postimg.cc/ydN9J4XR/defish.png
I guess PTGui would tell you... there is final FoV of your panorama ... and your panorama can be only one image.

[poisondeathray]

Yes, you are right, the thread is about lens correction (tool).

And we're discussing how to make it useable not only
for well known lenses but also if only known of a large fov.
For in that case all deshaker do bad.

It's difficult for me to describe what the effect looks like
that I called "wobbling".
Actually "wobbling" seems not to be the right English word for it.
(I don't know a fitting word).
It's more like "kneads".
And it seems to be caused when tilting and panning
a large fov movie.
Why don't you have a look for yourself as I suggested?


I would, but I don't have any of those types of cameras anymore. I'm trying to think back - maybe DV, wide angle lens. Maybe you have something different, because I don't remember having any "wobbling" problem with those. But like you say maybe that was the wrong wording or description.

It doesn't make logical sense that you say it "works" with something more extreme like fisheye, but not something less extreme like a wide angle lens. If what you're describing is related to the lens distortion, shouldn't a more extreme distortion cause more problems ?


How do you expect shekh or any developer to improve any of these tools , or make them "more useable" - without any footage, settings, or at the very least an accurate description?

A video is worth a thousand words. Find some footage without people or personally identifiable elements. Or find footage from same or similar camera / lens somewhere else that demonstrates the issue clearly

When I want something fixed or improved, I go out of my way to provide footage, description, upload everything in a neat package. I want the developer to do as less work as possible, so that he or she would be more inclined to help, instead of having to do the "legwork" themselves.

[nji]

If the wobbling is panning and tilting the camera... or as I said rotation around x and y axis...


Slowly please!
That's my concretization of the effect I am adressing here.
(I'm no native speaker so my words my not always hit.
And "wobbling" maybe was even wrong, as "wobbling" seems to be the same than "shaking".
I tried to find a better fitting word, it that was (see above) "kneads within".)
This effect occurs - to my observation - allways if you 2D deshake a large fov movie. (I wonder how often I wrote this expression in this thread)


Can someone stronger than me in math help me?


To say it explicitly: That won't be me.
As I don't know enough either.
And the answers (kindly by shekh and wonkey_monkey) gave me ever more questions.


When I rotate camera precisely around x axis...
...does quad help me with symetric transform (zoom bottom or top of the image) to compensate completely...or there is some curve involved?
quad is Avisynth I don't know anything about (& different sub forum).
But if it does simply resize lines wrt to a reference line, then it will be correct only for fov = 0.
(No math, but just simple imagination :))

Sorry, but I'm at a loss like you. Can't help any better.
Tried to contribute by describing an effect that possibly could be enhanced.
Ran against walls.

[wonkey_monkey]

A transformation with quad is equivalent to the 3D rotation which would put the corners in the same positions (for a rectilinear view, anyway).

[redfordxx]

A transformation with quad is equivalent to the 3D rotation which would put the corners in the same positions (for a rectilinear view, anyway).

Thank you.

Can you help me please one more?

When I want to simulate rotation around x and y...not z axis.

I have
d ...direction of combined rotation (0..360) degrees
a ...amount of rotation > 0 degrees
w,h ...dimension of video.

How will I calculate the 4 points for quad?

[poisondeathray]

When I want to simulate rotation around x and y...not z axis.

I have
d ...direction of combined rotation (0..360) degrees
a ...amount of rotation > 0 degrees
w,h ...dimension of video.

How will I calculate the 4 points for quad?

maybe this old post will help or get you started ? the math is there with cosine , sine
https://forum.doom9.org/showthread.php?t=167797

the avisynth part was taken from that post (but the older version, not the revised version)
https://stackoverflow.com/questions/40186787/rotate-an-image-around-its-x-axis

[redfordxx]

maybe this old post will help or get you started ? the math is there with cosine , sine
https://forum.doom9.org/showthread.php?t=167797

the avisynth part was taken from that post (but the older version, not the revised version)
https://stackoverflow.com/questions/40186787/rotate-an-image-around-its-x-axis
This seem to be rotation around the objects axis.
I need rotation around camera axis.

[poisondeathray]

This seem to be rotation around the objects axis.
I need rotation around camera axis.

Maybe you can make some rough assumptions like focal length, etc... just to make the representation work ?


Camera rotation is really pan (camera y-axis rot) and tilt (camera x-axis rotation) .

It will also depend on the camera and lens characteristics, focal length, sensor size to determine the camera rotation axis of your particular shot. In short, you need to do a 3d camera solve to determine the relationship of the camera to the scene contents.

A single set of variables like r,a,w,h won't be able to manipulate this for all types of cameras setups, lens and shots ; or it won't give you similar results across different cameras.

You have a single 2D "plane" representation with all objects, far and close on that 2D plane. You're going to get problem with parallax and perspective in many types of shots and camera movements

The proper way to do it is 3d camera mapping. You track the shot with a 3d tracker and create a virtual projection camera. You can project your video onto geometry and use a 2nd camera to adjust the camera move. There are free ways to do this in blender (not as good as commercial methods) . And there are old free 3d trackers like voodoo. The benefit of voodoo is it fires a bunch of trackpoints like hundreds (in native blender you have to set the point locations manually), but it's quite a dated program and 32bit.

You might be able to look at the export data and coordinates an "massage" it to plug into deshaker, deshaker3d or quad (it can export blender scripts and plain text files) . It gives camera position (x,y,z) , axis (3 coordinate , 3 axis = 9 values) , focal length, field of view data

[redfordxx]

OK, I see it is much more complex than I thought.

Now, I spent few hour on rotation clip, but it is notworking as it is supposed to, I think.
I post it probably later somewhere, so maybe someone finds the problem.

[poisondeathray]

OK, I see it is much more complex than I thought.

Now, I spent few hour on rotation clip, but it is notworking as it is supposed to, I think.
I post it probably later somewhere, so maybe someone finds the problem.

You would have to make some assumptions about camera distance to a "target" plane ; the effect of camera rotation on the "2.5d" plane would look quite different depending on different distances for a given camera/lens setup. Up close , the sheer effect on the 2.5d plane would be greater (greater angles for a given degree of rotation). A far shot would have less effect, the end result approaches something closer to translation x,y

But you're not necessarily looking for mathematically correct optics, are you ? You're only looking for a closer "match" as determined by deshaker %ok blocks . ie. It's not necessary that your "label" of "x,y" degree rotation is accurate, only that it "looks" more similar to the previous frame

[redfordxx]

As I presented i the warp thread, I calculated focal distance from FoV which I know.

I don't know what is "2.5d"...

[poisondeathray]

I don't know what is "2.5d"...


"2.5D" is the term used to describe "flat" 2D planes in 3D space. It's not "2D" , but not quite "3D" either... hence the name "2.5D"

[nji]

Can please somebody explain what the arrow-buttons (beside the edit fields) are for?
No matter what I do I don't see any effect when pressing...

Thanks in advance!

[shekh]

Can please somebody explain what the arrow-buttons (beside the edit fields) are for?
No matter what I do I don't see any effect when pressing...

Thanks in advance!

Move the mouse left/right while holding. Bigger arrows are for faster changes.

[nji]

Ooops.

Thank you for that one.

I don't know if I ever found that out (maybe by occasion)
by myself. As I haven't ever seen a button used like that.
(Wait... there is ONE in AviDemux. But that "wheel control"
is quite obvious, just optical).

Still let me remark, shekh, that your plugin does very well,
producing the best results for my wide-angle movies.
If you ever start coding again... this one would deserve
a 2nd version. :-)

Greetings!

[nji]

I did a comparison to Deshaker3D (https://forum.doom9.org/showthread.php?p=2025810#post2025810)