FQSharp plugin for deconvolving camera movement blur

[vcmohan]

I have searched deconvolution, restoration and the writeup of motion compensating plugins and could not find a plugin that does what I have developed. In case I am wrong and that already such a plugin exists then, this may be treated as a variant.

Due to movement of camera during the time the shutter is open, the image is blurred linearly and exhibits this over the entire image. Blur due to objects (or subjects moving eg: racing car, sporting events) also blurs but is limited to that parrticular object. Blur due to lens being out of focus blurs circularly but it will vary if objects are at different distances from camera.

This FQSharp plugin operates in freq domain and requires fftw3 dll to be in the path. The linear blur and its orientation are estimated, the inverse filter designed is applied.

Below are some images shot by me during my European trip. On top is original and below is the output. The fields are required to be separated for this process and so the squashed appearance of images.
1
http://img406.imageshack.us/img406/8237/fqsharp1uq5.th.jpg (http://img406.imageshack.us/my.php?image=fqsharp1uq5.jpg)
2
http://img150.imageshack.us/img150/2884/fqsharp2wz4.th.jpg (http://img150.imageshack.us/my.php?image=fqsharp2wz4.jpg)
3
http://img440.imageshack.us/img440/8641/fqsharp3lp5.th.jpg (http://img440.imageshack.us/my.php?image=fqsharp3lp5.jpg)
4
http://img171.imageshack.us/img171/7822/fqsharp4yr3.th.jpg (http://img171.imageshack.us/my.php?image=fqsharp4yr3.jpg)
5
http://img171.imageshack.us/img171/3958/fqsharp6bf2.th.jpg (http://img171.imageshack.us/my.php?image=fqsharp6bf2.jpg)

Request cooments.

[Dark-Cracker]

hum seems very interesting.
does this plugin could be used on video not taken from a car ?

Keep up the good work.

Bye.

[Pookie]

Another unique plugin. Thanks for being so smart ;)

[Fizick]

some time ago I tried create a similar... But results is not great- i do not see much difference from simple oversharping....
Same as with your filter. Sorry.
It is hard task.

[foxyshadis]

What is the maximum radius? Or is that what maxblur sets?

This is a photoshop plugin that does such a thing:
http://www.reindeergraphics.com/index.php?option=com_content&task=view&id=212&Itemid=158
but it had issues back when I tried it (also: slooow).

Also this is one of my favorite papers on the topic:
http://people.csail.mit.edu/fergus/papers/deblur_fergus.pdf
(maybe just because of all the pretty pictures, I'm shallow that way.)

Interesting to see avisynth getting such a tool. I won't have much need for it - particularly with the strong haloing - but I know a few things that might benefit. Hope to test it soon, it'd be worth it if it helps.

[Terka]

imho this could be useful to add to deshaker

[vcmohan]

could be used on video not taken from a car ?

Yes. This plugin can be used wherever the camera moves during the shutter open time. It can be hand shaking. If there is no movement the plugin just returns the original.
What is the maximum radius? Or is that what maxblur sets?
Camera motion is a linear blur. Maxblur is the maximum half width of this blur. Its not radius.

but it had issues back when I tried it (also: slooow).
Well. this plugin is not that slow. Try it and find out the speed. My VGA (or is it called AGP?) card is, I suspect faulty as my system freezes when I try to stream fast.

[wonkey_monkey]

Could you post some examples that have more motion blur? Those you've posted seem to have hardly any...

David

[scharfis_brain]

I tried this plugion and it did only produce oversharpened results, like Fizick mentioned.

I couldn't see any motion blur to be removed or at least reduced.

[*.mp4 guy]

In the last shot there is noticible motion blur that it removes, I think the reason it looks oversharpened in some shots is because of the haloing in the source.

[vcmohan]

Could you post some examples that have more motion blur?
I am using the footage I have shot for the test. I have shot some footage while travelling in TGV train in France and it has large blur. The difficulty for using images with large motion blur is 1. The foreground objects ( trees) of the image have large motion blur and really they appear as a smooth blob. Therefore it is difficult to autodetect the blur. 2.Most of the other images do not have sufficient quantity of objects (edges) to be amenable for the auto detection. 3. The size of the frame is not enough for getting enough statistics.

However I propose to do a synthetic blur on a suitable image and test. I will comeback on it. Also meanwhile I will be modifying the FQSharp plugin so as to get some more user input.

[cwk]

Howdy VCMohan,

Thank you for your plug-ins, I have enjoyed working with them. What would you (and others) think about the possibilities of using this to "enhance resolution" of old VHS captures?

[vcmohan]

Howdy VCMohan,

Thank you for your plug-ins, I have enjoyed working with them. What would you (and others) think about the possibilities of using this to "enhance resolution" of old VHS captures?

I have no idea. However please try and if it improves well and good. Otherwise dont use it.

[krieger2005]

The dowloadlink of the page is linked wrong. Can you correct it. Thanks.

[vcmohan]

I will check it. But I am revamping it and possibly take a couple of weeks. Please use after the modifications.

[wonkey_monkey]

http://avisynth.org/vcmohan/FQSharp/FQSharp.zip

David

[vcmohan]

I have taken many many weeks for this.

I have been trying on and off with the frequency domain inversion with the ultimate aim being a type of blind deconvolution. Didee in his masterly exposition in thread (http://forum.doom9.org/forumdisplay.php?f=69) has shown why for video images it is a futile excercise. Inspite of it, being a little stubborn and possibly having a try to reinvent wheel mind set, I have done some experiments and reached a stage at which I feel need of sharing and getting advice.

To enable me to test my progress I needed some functions. These are
1 FQBlur that creates the specified motion or out of focus blur in a window.
2 FQRestore that restores back by inversion, given the exact blur
3.FQVisual that aids in visually determining the blur charecteristics.
4 The main tool still a long way off and is under experimentation to "automatically" restore.

There are several ifs and constraints under which these may work. There are also several artifacts and severe ringing effects that need to be overcome. The inherent instability and how to overcome it requires intervention.

It is true that this will (may) never become a production tool, and only those who like me having no other work to do, can spend time, will attempt to use these tools.

However I will be happy if even one person tries these functions out. I have completely revamped FQSharp and included these functions. In my plugins page I have given detailed write ups as well as some displays showing problems. Included below is a display produced by FQVisual.

Inputs for displays on left have out of focus blur, while those on right have motion blur. The radius or main semi dimension blur values are 2, 4, 8 and 16. The second row on right has blur exactly along width axis and the bottom one is along height axis.

cepstral displays in top row have a quefrency enlarged panel also. The line displays are for values along x axis, y axis, southeast and southwest. These two lines x val to be multiplied by 1.414 to get corresponding values with lines along x or y axis.
http://img85.imageshack.us/img85/5954/fqvisual0jp0.th.jpg (http://img85.imageshack.us/my.php?image=fqvisual0jp0.jpg)
The problem to automate are evident even under the ideal test conditions. Small blur expression are too close to origin where generally high values are present. In place of a peak or trough a small plateau or change in gradient may only be manifest.

However with naked eye, it looks possible to estmate type and dimensions of blur. For some reason The values I get on these displays are double of the real input values.

Next I need to test with actual camera images having such blur.

I would like to get comments on this work

[mikeytown2]

Not sure if you've seen these 2, u might be able to get some ideas...
http://refocus.sourceforge.net/
http://refocus-it.sourceforge.net/

Article on the subject
http://www.linux.com/feature/124567

[vcmohan]

http://refocus.sourceforge.net/
http://refocus-it.sourceforge.net/]

Yes. I have seen these. Both act with the same principle. They try to optimize a matrix Gaussian or circular to get an acceptable level of correction through large number of iterations. On the other hand my approach is through freq domain. I am trying to see whether I can automatically detect the blur vector. I am running into severe problems here. While visually one can feel or arrive at this vector, I am unable to find a criterion for automating it.

I also have problems in eliminating severe edge effects and ringing while inverting it. I stopped working on this for some time now. I will once again have a look at it.
Thanks for interest shown.

[Terka]

could be possible to get the blur direction using mvtools or deshaker? as far i know there are more application doing this for still images eg. astroart or focus magic or many others.
on http://ieeexplore.ieee.org are many interesting papers but cant open them.
http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/97/26750/01193031.pdf?tp=&isnumber=&arnumber=1193031
keeping the fingers crossed to find the solution!
look at
http://people.csail.mit.edu/fergus/research/deblur.html

[*.mp4 guy]

Alright, heres an idea. its frequency based, since I'm assuming thats what you have the most experience with.

initialization: X=0

step1: perform fft on an arbitrarily sized square, or squares.

step2: determine which frequency axis has less energy, horizontal or vertical.

step3: rotate window by (45/ (X += 1)) (45) degrees, perform fft

step4: repeat step2.

step5: rotate window by (45/ (X += 1)) (22.5) degrees, in relation to the two previous windows axis that had the least energy.

step6+: repeat step 5 untill desired precision is reached.

There are fourier transforms that can be used on arbitrarily shaped windows, which could be used. or you could rotate the image with a high tap sinc kernel (which should introduce small enough error). To reduce the resampline error, you could supersample the image with a specific kernel, then do all of the window rotations, this way most of the frequency distortion comes from the resampling step, which will be the same for all fft's.

You could perform temporal averaging, or median filtering to remove anomalous results, and use multiple windows within the same frame, in a motion search like patter for similar added robustness.

for all I know you've already got something that works better, but it looks like a good idea.

[vcmohan]

a. If I understood correctly the procedure is for identifying the direction of motion blur. I can better do that by using the output I get using FQVisual function. However the problem is if the blur and main lineaments of the image are parallel or subparallel. I have also seen in many images various other features produce alignments in Freq domain that will confuse.
Apart from this the inversion process for blurs greater than a couple of pixels or so produce unacceptable levels of artifacts.
May be its all a wild goose chase

[*.mp4 guy]

I thought you probabably had it handled pretty well, but it was worth a shot.

[vcmohan]

[QUOTE=*.mp4 guy;1176298t it was worth a shot.[/QUOTE]
Using your suggestion I rotated (using Spinner) the input by 45 degrees and checked with FQVisual. In those cases where the motion is along any of the axii, it brought out the motion vector along 45degree line. However in the process, the end points are no longer discernable. The vector is now a gradually fading long line. Possibly the interpolation matrix has destroyed the contrast.

Thanks once again for the suggestion.