Lens Transform tool (work in progress)

[shekh]

This is plugin for VirtualDub.
It combines all of the following:
* Remove lens distortion using calibrated profile. Can use Adobe fisheye lens profile (used in Lightroom etc).
* Stabilize camera motion. For analysis I just use Deshaker log file.
* Apply arbitrary animated rotation (3d), zoom, barrel/pincushion.
* Crop/extend image size.
Rolling shutter correction would be nice, but atm it is not finished.

The image transformation may combine up to all these steps: convert from fisheye to rectilinear, remove skew, apply 3d rotation/zoom, apply barrell, convert from rectilinear back to fisheye. In all cases it is a single bilinear lookup directly from source (no intermediate bitmaps are rasterized inbetween). In other words it must be high quality.

There is no automatic path creation like found in Deshaker. Instead I make reasonable UI to do whatever path adjustments manually (animate rotation/zoom). Performance is quite good for interaction. Also there is manual shake correction for cases where Deshaker fails.

Right now I am satisfied with every aspect except skew correction, it is quite usable.
But before publishing I better polish something about lens profiles.
Anyone interested?

Example screenshot:
https://sourceforge.net/p/vdfiltermod/wiki/LensTransform/attachment/sample_screen.jpg

to be continued

[Reel.Deel]

Sounds interesting. I have a couple of cameras that could use some lens corrections.

Can a mod please approve the attachment.

[shekh]

The tough part is getting good profile for each camera. I have calibrated mine myself - quite challenging process to do at home, especially for water.
It is possible to pick a profile from the adobe library but this way you apply wrong correction for the center offset, as it varies per unit.
If you have some grid shots and profiles, throw it at me so I can test compatibility.

[shekh]

What amount of skew distortion is tolerable (there should be animated gif):

Left picture is sequence shot while knocking the camera to get some rolling shutter skew effect, most noticeable on frame 3 (bottom part skewed left).
Right picture is the attempt to align frames as much as possible, eliminating both shake and skew. The analysis by Deshaker failed - I matched frames manually.

https://sourceforge.net/p/vdfiltermod/wiki/LensTransform/attachment/extreme_align.g

Btw the filter was setup to output result in original (distorted, fisheye) space, to ease comparison.

Not perfect but this result convinced me of two things:
1) it is very hard to ruin material while camera is handheld - the skew is quite soft over all frames I have inspected - I never got a frame where camera is changing direction (the shots are 48 fps).

2) the sufficient de-skew algorithm is very simple and fast (the part which does pixel projection, analysis is whole different story).

[n01d]

Very promising, it would be great to have avisynth's deshaker3d features inside vdub, can't wait to try it out!

[shekh]

Demo: canyon (https://youtu.be/LmHmAInj4Iw)
Project source here (https://drive.google.com/open?id=0B8BT09xR2LmadnBhZWNIc3lKY3c) includes source video (75Mb) and plugin setup.

How to get started:
1) download and unpack executable here (https://drive.google.com/open?id=0B8BT09xR2LmaRTNGcWpCWEpkYnM)
It includes VirtualDub, plugins, and sample lens profiles.

2) download and unpack canyon project.

3) launch VirtualDub, use File->Load project, point to canyon.vdproject

Suggestion for some fun:
open Lens Transform filter configuration, activate preview, click and drag big square directional button in the "Rotate xy" row.
If it does not react make sure current frame is 0 (it has keyframe in this project).

[shekh]

The plugin actually has 3 filters:

Lens Transform is the main thing.

Lens Test is much simpler, made to inspect lens profiles whether they work as expected.
It can do just simple forward/reverse distortion without any rotation etc. This is style of transform useful for camera tracking and cg composition (but I am not doing this).

Repair is simple border filling, same purpose as Deshaker` extrapolate pixels.
Repair uses alpha channel to determine what to fill and Lens Transform in turn uses alpha channel to signal good/bad pixels.

About lens profiles
The most accessible way to get one AFAIK is to use Adobe Lens Profile Creator free utility.
To calibrate, just open checkerboard image on flat display, take about 9 shots at different angles, then put them in the profile creator and it will automatically determine distortion, fov, and lens center.
At the moment I only read "fisheye" model (there is also "rectilinear" model which I have ignored).
I made profile this way using ordinary 23" monitor. I also made water profiles, for this purpose monitor is not an option :)

[shekh]

Made some basic manual: https://sourceforge.net/p/vdfiltermod/wiki/LensTransform/

[otalp]

shekh!

This is the most amazing tool for stabilizing video clips I've stumbled upon. Great work I must say.

Thanks a thousand times for sharing!

[shekh]

shekh!

This is the most amazing tool for stabilizing video clips I've stumbled upon. Great work I must say.

Thanks a thousand times for sharing!

Thanks, I had more progress with it but was lazy to post any update due to lack of feedback.
Have you any success at using it? :)

[b2kguga]

Excellent work. How you managed to make the preview without controls ? I´m trying to grab the handle of the preview window without using EnumWindowsProc.

[shekh]

Thanks!

"How you managed to make the preview without controls"

It is without controls because I modified VirtualDub source to make this. Also see my reply in other thread :)

[b2kguga]

Ok...Btw...i simply loved the filter. have you considered using the output data from BCC or Mochapro insetad only relying on Deshaker?

[shekh]

I have internal motion analyzer in progress (but I really need to get back to it). I dont even know what is BCC and Mochapro. What is useful out there?
It all started from playing with syntheyes demo, I use its lens profiles but not camera track (demo does not export it and I dont think I need it anyway).

[b2kguga]

Mochapro is the best professional tracking tool out there. It is used to remove objects from the scene, stabilize, lens correction, etc. http://www.imagineersystems.com/products/mocha-pro/

BCC - BCC Motion Key (Boris FX Motion key) is also a very good motion tracker. https://web.borisfx.com/helpdocs/?page_id=951

Both export their data on a regular txt file, from where you can use them on Sony Vegas, for example. Although the result of both are very accurated, the problem of those two apps is that they are very slow and the configuration is a complete headache. The good is that the resultant txt file is somewhat similar to deshaker, therefore, it maybe ported to get a better result.

[shekh]

From my experience with Syntheyes (I assume all motion trackers are same or more difficult to use) there is huge gap between hollywood and consumer material.
I want to repair shots which are poor quality by professional standards, so I suppose no software is trained to help on them. Or difficult anyway. With syntheyes I had to eyeball misaligned trackers all the time. This is so much manual intervention that doing full manual tracking looks not bad too.
If Mocha etc. can export motion tracking data, this is maybe useful for someone in VFX, but I have doubts I can make with these data something better than Mocha itself?
Anyway my goal is to create easy workflow to do it all in one place, quick and happy.

[b2kguga]

I totally agree with you. Professional apps are a hell to configure. About mocha tracks, in fact the tracker can be improved. Their results, i mean. For example, we can get the data from the tracker to completely remove a object without the needs of using mochapro itself. Removing a logo (or other static objects), for example, in Mocha is very hard.

Your idea of making a tracker be simple and fast is really awesome. This can be used not only for stabilization, but also for object removing as well (Static or moving object). One thing to be considered is maybe implement a scene detector in order to gain accuracy on the tracking/stabilization. On this way, whenever a scene is detected, another track data is performed for the next scene.

The 2 filters i saw for vdub that are suitable for object remover (i.e. tracking/stabilizing) are yours and stabbo. I could give a help, but i´m still learning about Vdub Api while building my own version of LogoAway. When i have enough knowledge on how exactly Vdub Apis can be used i can try helping you on stabilization and object remover techniques.

[b2kguga]

Shekh, i was trying your vdub version with the Lab plugin. The Lab plugin was made by Neiromaster, but If you have the source it can be improved a little bit more. For example, it can use a limitation of Hue (from CieLCH) to check for difference ranges. For example, whenever you set a image to use more blue or red etc, it will increase the amount of smooth "spots" of Blue/Yellow/Red/Green. The "spots" can be removed simply making check for a Hue difference range.
Like this: when you are increasing the Blueish tone (a and b factors from Lab) , whenever the pixel Hue varies in, let´s say...10 degrees (user can define the range), the conversion from RGB to Lab stops at that pixel. Or...instead stopping the conversion, the Hue can be fixed inside that range and all that will change is the Luma or the a/b factors without changing the hue anymore.
This probably will remove the "spots" that appears when you increase too much the tone from yellow/blue/red/green used in Lab colorspaces.

[shekh]

I dont understand most part of what are you saying :)
But if you want to try you can get source here: https://sourceforge.net/projects/vdfiltermod/files/plugins/Neiromaster%20lab/1.1/LabColorCorr.rar/download

However by inspecting source I conclude it has nothing to do with CieLab.
I made "6-axis color" filter based on same colorspace as "lab" filter, it is basically some YCbCr.

[poisondeathray]

shekh - any ideas on how to address the rs skew and wobble ? That is one area that is lacking in free/open source tools. That's a large part of what prevents deshaker or any stablilizer from working properly. Skew can sometimes be handled by commercial software/plugins fairly well, but the wobble from vibrations (multiple vector directions) isn't handled as effectively

Nice location in the canyon clip BTW, where is it ?

[shekh]

This is göynük canyon, turkey.

For skew I have some ideas in todo list. It is quite easy to match frames to each other using 2 sets of parameters instead of 1 (same as deshaker).
Then it is quite easy to try full compensation matching all frames to single reference frame, this is suitable for shots without motion. This should certainly work, but I am not sure RS is an issue at all with such shots.
The trouble is to build a path when camera moves. I certainly cannot just accumulate differences stacking one frame after another: this works for no-skew mode but with skew included the growing error tears frame apart.
I tried deshaker approach without success (either I did it wrong or did not understand the result), also tried filtered average as immediate reference (same here).
Another idea is to collect frames with lowest skew and use them as references.
First I need better motion data. For this I started to make analyzer
Second problem is to understand what happens. This is hard with real footage. I dont see something simple like skewed pillar, instead the shot just looks "jelly" but I cant tell for each frame where it is wrong.
To some extent I train the math on synthetic RS-skewed shots. This way I can have perfect result to compare with. But I need to go deeper and prepare realistic synthetic shot with motion and shaking similar to reality, and this is damn long task to prepare :)

For wobble I have no ideas. I suppose it requires different level of enginery (structure from motion etc).
I chose as milestone some piece of footage which has really bad shake but still no wobble.

Btw maybe by the time I finish that, they come up with practical global shutter sensor? :)

[djonline]

shekh, b2kguga. I was used Synthyes too, it works fine on CCD camera (my example https://www.youtube.com/watch?v=wsrs28aUAKw and https://www.youtube.com/watch?v=bxQ8AkyK70k and https://www.youtube.com/watch?v=NTYdAVV71p8) , but it completely unusable on consumer CMOS camera due Rolling Shutter. There is no way to fully eleminate them, so better result for CMOS is to use Warp Stabilizer in Adobe Premiere.

I dont 'like shaked edges in this example of Canyone, it is not usable for me. But I still use deshaker for little shaked video to eleminate small shake, coz Warp is still very slow.

[nji]

Hi shekh.

I posted I question to your plugin in another (wrong) thread

https://forum.doom9.org/showthread.php?p=1884379#post1884379

Greetings.

[shekh]

Hi shekh.

I would like to ask if there is an up-to-date version of your plugin.
(Only the plugin and the needed files for ini etc. for use in VDub2)

I use DeShaker for a long time, but had to make the experience
that with large FOV it produces "wobbling" results (due to the
lack of taking perspective into account) that make the result
worse than the shaking version.

In Deshaker - as a first step - a simple option for FOV
(wide-angle, normal, tele) would be fantastic.
But ... sadly ... DeShaker isn't improved anymore.

As far as I understand the matter, BEFORE phase 1 in DeShaker
there would have to be the perspective correction (panotools etc.?)
and after that to calculate the motion vectors etc.

But maybe your plugin works well enough already?

Greetings! :-)

Hi,
actually the published version is the last one. Any further improvement is in awful "maybe resume later" state.

That said, the last version can do something, I believe, better than deshaker wrt perspective and distortion.
If you really want to use it as is but don't understand the GUI, I can guide you :) Post your sample.

[nji]

Hi shekh.

Thank you for your response and your offer for help.
I'll try on my own first.
And if nothing more helps ... I could read your posted instruction ;-)

Only one thing about the "published version" you mentioned.
It is a collection of a complete (and old) VD version and some other things.
Would it be possible for you to do only the needed files for your tool/ filter in one package/ zip, so that one can easily integrate it into an existing VD2?
That would be nice.

BTW: Strange that there is so few interest in that wonderful quality improvement of deshaking (And ever more the lens correction). As most shaky videos will be taken on a middle FOV, and not FOV=0 (= standard DeShaker without perspective accounting).

[nji]

[...]

Btw maybe by the time I finish that, they come up with practical global shutter sensor? :)

Please don't forget that most videos VD manipulates are from the past (at least for me it's the case). And all they are taken by "previous" optics/ sensor types.

[wonkey_monkey]

Please don't forget that most videos VD manipulates are from the past

I'm curious to know who's processing videos from the future.

[nji]

Hi,
actually the published version is the last one. Any further improvement is in awful "maybe resume later" state.

That said, the last version can do something, I believe, better than deshaker wrt perspective and distortion.
If you really want to use it as is but don't understand the GUI, I can guide you :) Post your sample.

Hi,

I recently tried (had to :)) your plugin on a wide angle movie.

I stripped the nescessary files to the minimum (see attachment).
Is that complete?

Does your alg do "3D calculation", or "simply" performs DeShaker's log?

Does the rolling shutter/ "skew" work?

[shekh]

Hi,

I recently tried (had to :)) your plugin on a wide angle movie.

I stripped the nescessary files to the minimum (see attachment).
Is that complete?

Does your alg do "3D calculation", or "simply" performs DeShaker's log?

Does the rolling shutter/ "skew" work?

Please note that this project is somewhat buried.
Can't see attachment. What you planned to show?

I saw your posts about it in other places, so I try to summarize replies

The "transform" rebuilds true projection to different direction and with different lens (but from same focal point).
By true projection I mean, if you had actual real life shot of the same scene it would match exactly.
For this to work it needs accurate lens description with FOV and distortion curve.
In this way it does true 3d calculation. Did you mean something else?
(also the amount of rotation for all axes is expressed in units of rotation not pixels)

Deshaker log was used as a guide to derive motion but not as direct instruction to move pixels.

Rolling shutter: no, I don't think it worked

[nji]

Hi shekh!

Actually I don't understand why you don't see
why nobody can see the attachment at the moment.
Being not logged-in it says (for everbody I guess):
"Attachments Pending Approval".

The case I'm interested in is that I don't have detailed
information about the lens the movie has been taken with.
(Except a vague value of FOV).
And even if I would know, first there would have to build
a description as you wrote.
If I understand it right (do I?) without that information
your plugin is useless for all its functionality?

Next: I do not understand how Deshaker's log could be
of any use for your plugin. As its analysis was done
without knowing the specific lens characteristics.
"... as a guide ..." ???

Rolling shutter ("skew") doesn't work?
But there is an active check-box for it ...

[shekh]

The goal of this plugin was use with known camera which you can calibrate and test.
Some program may help you to extract lens params from movie. Don't know much on this.

The principle how deshaker log was used here: suppose deshaker recorded 100px move to the left. By comparing pixels and how they are projected from world you can translate this into radians.
Together this is not accurate enough but is 'move in right direction'.

[nji]

So, is there a way to "feed" your plugin just with the FOV value for a movie?
With "neutral values" for the other distortion.
And if ... how?

[shekh]

try to add your new profile like below


[lens]
name = "my test"
hfov = 90

[nji]

... suppose deshaker recorded 100px move to the left. By comparing pixels and how they are projected from world you can translate this into radians.
Together this is not accurate enough but is 'move in right direction'.

You calculate which kind of angle movement (wrt to FOV)
would cause a Deshaker's 2D movement.
Then you take that angle movement and transform it back to 2D.
Do I get it right?

How does this compare to Avisynth Deshaker3D approach?

And how does it compare to Deshaker solely?
Identical for FOV = 0; and better results than Deshaker with larger FOV?

[shekh]

nji

It is really boring to explain application of deshaker log because there is nothing to say, it is just basic trig and the method is also just a proof of concept (better log is needed).
Deshaker3D is bare minimum algorithm which is also included in lens transform (minimum math is so basic you can't avoid it).
Looks like I made this for you to help understand the place of FOV (old video): https://www.youtube.com/watch?v=JpKHxfC28mQ

[nji]

Sorry having asked.

[nji]

Is there a way to let not only the borders "repair" but also dubbing-ons (like logos etc.)?

[shekh]

"repair" just fills based on alpha channel, so you need some way to "erase" alpha where you need. Something like push logo to alpha channel. I don't have such filter but it is really simple. Do you have a use-case?
Btw, I believe delogo filters solve this better so what is your intent?

[nji]

My intent is simply to prevent a dubbed-on date jumping around
when the movie is deshaken.

In Deshaker you can mark arbitrary areas by a special color
which is recognized by Deshaker and repaired.
But - and this is important - Deshaker does much better than
just delogo. As it takes neighboured frames for the filling
of the areas.

EDIT:
I suspected that repair would do something similar.
But that was wrong thinking, as repair gets only the alpha from lens transform.
Conclusion:
Do delogo before lens transform.

[shekh]

Ok, I assumed such workflow but it is not ready.

[nji]

I'm referencing an idea of redfordxx
https://forum.doom9.org/showthread.php?p=1894961#post1894961
to enhance analysis phase of orig Deshaker.

On my attempts to understand the projection
shekh uses in his filter for accounting FOV...
and with my idea that it would be better to
account FOV BEFORE analysing phase...
https://forum.doom9.org/showthread.php?t=177335
...
I would like to propose to use exactly the "simple trig"
that shekh is using for his phase two
for pre analysis.

Example
If you have a movie with large FOV (say 90++)
project it "flat" (simple trig).
==> Will make it a pincushion und use that for analysis phase.

Both ideas should give more and better (wrt FOV) matchings.

RFC

[redfordxx]

I looked at LensTransform. It uses deshaker data gathered before transform and aplies them to video after transform? Is that correct?
EDIT: sorry, I missed thde point that log is bare guidance.

[shekh]

I'll try to explain better:
given: log data describes movement
assume movement applies to image center exactly
find 3d rotation for entire projection which results in given movement of the center pixel.
Hope it makes sense now :)

edit:
Again, more precise way which I did not finish:
given: array of motion vectors
find 3d rotation which minimizes all(most) vectors

[redfordxx]

I again gave try to the lens profile and handedited it until I saw any change in the frame.

I ended up with two findings:
If I have perspectivemodel...no change...only fisheyemodel.

If I change the profile, I have to close the the VDub for changes to take effect.

Can you please confirm/explain/deny it?

[shekh]

Yes, I wrote in first posts that only fisheye adobe profile is implemented.
Restart vdub: not really, it should reload presets every now and then. Open/close filter dialog, open/close Video->Filters dialog.

[nji]

Yes, I wrote in first posts that only fisheye adobe profile is implemented...

In a former post you adviced trying your filter for improving large FOV effects.
https://forum.doom9.org/showthread.php?p=1895232#post1895232

Does this hold only for fisheye lenses?
Or will it be an improvement for "normal" (rectingular) lenses too
(in comparison to orig Deshaker)?
EDIT
Might there be an enchancement to Lens Transform
so it does better for "normal" lenses?

[shekh]

For perfect rectilinear lens all you need to know is just fov parameter, there is nothing else to describe it.
Adobe profile creator defined 2 lens models but I implemented only "fisheye" because this is the one that fits hero etc.
Perspective model can be implemented too in theory, also profiles from other software can be implemented (like ptgui), or embedded gopro profiles, and so on.. as long as diagonal fov does not approach 180deg, at which point it is impossible to project it in rectilinear way.
In my filter I actually don't use any formula (like polinomial) but instead I use lookup table for distortion so it is basically profile/formula independent (but applying some 'standard' profiles requires knowledge how to interpret it and test samples - this is quite some work ).

Difference between Adobe "fisheye" model and "perspective" model is subtle, it is just the curvature formula which tighter fits actual lens. Same lens can be described by both models but you'll see more curvature errors in corners.

"Perspective" model assumes very little radial distortion. Small distortion is more difficult to test because, well, you dont see it :)
If you have it and want support, lets do this: post your 9 grid shots (full sensor) as required by lens creator and some exterior shots of very straight objects.

[redfordxx]

For perfect rectilinear lens all you need to know is just fov parameter, there is nothing else to describe it.

So... when I get the parameters from PTGui:
FoV, a, b, c

I will defish.dll with a,b,c...

Then I use lens transform with FoV value, it sould be correct?

[shekh]

So... when I get the parameters from PTGui:
FoV, a, b, c

I will defish.dll with a,b,c...

Then I use lens transform with FoV value, it sould be correct?

Sounds good. A few notes:
I dont understand how defish uses fov parameter. It cant be both the angle and control distortion... Well, with a b c the fov is not used at all.
Similarly, defish does not tell you the new fov for resulting image.
So you should crop or scale the result in such way that pixels at positions (0,h/2) and (w,h/2) remain in same place. Otherwise you need to calculate the new fov yourself.

Link to defish for convenience
https://forum.doom9.org/showthread.php?p=1375626#post1375626

[wonkey_monkey]

I dont understand how defish uses fov parameter. It cant be both the angle and control distortion...

It is, because the input and output projections are assumed to be fisheye and rectilinear.

Having vaguely followed this rather meandering thread, I feel it's worth pointing out again that if the aim is to find a projection where 3D rotation can be eliminated by changing the projection, translating the image in 2D, and then changing the projection back, then no such projection exists.