Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 9th March 2009, 12:49   #1  |  Link
Terka
Registered User
 
Join Date: Jan 2005
Location: cz
Posts: 704
rolling shutter

is it possible to fix rolling shutter (d)effect?
some camcorder have it.
http://cpressvideo.superhosting.cz/s...9/camileo3.mp4
Terka is offline   Reply With Quote
Old 11th March 2009, 05:02   #2  |  Link
morsa
the dumbest
 
Join Date: Oct 2002
Location: Malvinas
Posts: 494
It is and it is not.Better ask manufacturers to waste a couple more pennies on sensors and choose to have Global shutter on CMOS...

(or at least a so fast readout that RS becomes almost invisible)
morsa is offline   Reply With Quote
Old 11th March 2009, 11:46   #3  |  Link
Terka
Registered User
 
Join Date: Jan 2005
Location: cz
Posts: 704
yes, i know. ;( i meant avisynth way.
Terka is offline   Reply With Quote
Old 11th March 2009, 12:35   #4  |  Link
2Bdecided
Registered User
 
Join Date: Dec 2002
Location: UK
Posts: 1,673
VirtualDub deshaker claims to try - it doesn't help with my camcorder though.

Cheers,
David.
2Bdecided is offline   Reply With Quote
Old 17th April 2010, 18:46   #5  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
There is actually a recent plugin for Nuke by The Foundry that now corrects video with Rolling Shutter issues:

http://www.thefoundry.co.uk/pkg_over...1-FB380B2119EF

but it is bloody expensive, if only someone took the concept of this and made it as a avisynth filter, can it be done? and wat would they need? a .dll of the sfotware or something?
Undead Sega is offline   Reply With Quote
Old 17th April 2010, 20:50   #6  |  Link
scharfis_brain
brainless
 
scharfis_brain's Avatar
 
Join Date: Mar 2003
Location: Germany
Posts: 3,653
have just tried to search for "rolling shutter" in the forums?

actually there seems to be a solution:
http://forum.doom9.org/showthread.ph...er#post1374487
__________________
Don't forget the 'c'!

Don't PM me for technical support, please.
scharfis_brain is offline   Reply With Quote
Old 17th April 2010, 23:53   #7  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
that seems interesting, but the user claims "it works. Sometimes." Im assuming that it would only fix the typical panning effects of rolling shutter, abit like from Deshaker, if you know wat i mean. Also, i dont seem to bee is anyone has tried it out?
Undead Sega is offline   Reply With Quote
Old 18th April 2010, 04:51   #8  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
It works pretty well, actually, except that it's a memory hog, somewhat unstable, extremely slow, etc. But it's a proof-of-concept, and gives acceptable results on small videos (and crashes on large ones). The good part is that it corrects all rolling shutter motion artifacts, both global and local, much like the Foundry plugin. Someone just needs to implement it as a subfunction of MVTools with optimizations, something which I am currently not capable of. Here is the latest version, though I have not messed with it in months:

Code:
#   RollAway filter by um3k/Justin Phillips
#   Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
#   "Amount" is the same as that used for DeShaker. The formula can be found here:
#   http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
#   "Segh" is the height of the segments the image is split into. It should be an even number, and a dividend of the clip height.
#   This script is compatible with SetMTMode.
#   Use SetMemoryMax in your script to increase speed. Set it to a fairly high number that is reasonable for you computer's capabilities.
#
#   Alpha 2

function RollAway_alpha2(clip c, float amount, int segh, bool "test", bool "flip", int "pel", int "blk", int "ovr", int "thSAD")
{
    Assert(c.height%segh == 0, "'segh' must be a dividend of clip height")
    ctime = amount/2.0
    segs = c.height/segh
    den = segs-1
    num = den
    test = Default(test, false)
    flip = Default(flip, false)
    pel = Default(pel, 1)
    blk = Default(blk, 16)
    ovr = Default(ovr, 12)
    thSAD = Default(thSAD, 100)
    super = c.MSuper(pel=pel)
    fv = MAnalyse(super, blksize=blk, isb = false, overlap=ovr)
    bv = MAnalyse(super, blksize=blk, isb = true, overlap=ovr)
    
    DeRoll(c, super, fv, bv, ctime, den, segh, t=test, flip=flip)
}

function DeRoll(clip c, clip super, clip fv, clip bv, float ctime, int den, int segh, int "segn", clip "stack", bool "t", bool "flip")
{
    stackh  = Defined(stack) ? stack.height
    \   :   0
    segn    = Defined(segn) ? segn+1
    \   :   0
    num     = abs(den-segn*2)
    fclp      = (flip==true) ? c.MFlow(super, bv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
    \   :   c.MFlow(super, fv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, 1)
    bclp     = (flip==true) ? c.MFlow(super, fv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, 1)
    \   :   c.MFlow(super, bv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
    nseg    = stackh > c.height/2 ? bclp
    \   :   fclp
    nseg = (t==true) ? nseg.Subtitle("num=" + String(num) + "  den=" + String(den) + "  segn=" + String(segn) + "  ctime=" + String(ctime*float(num)/float(den)))
    \   :   nseg
    stack   = Defined(stack) ? StackVertical(stack, nseg)
    \   :   nseg
    segn = segn    return stack.height == c.height ? stack
    \   :   DeRoll(c, super, fv, bv, ctime, den, segh, segn, stack, t, flip)
}
It basically motion compensates the frame numerous times, cuts up the frames, and recombines them into a new one. Needless to say, it is quite slow. Try it on a video downscaled to SD resolution, HD is practically guaranteed to crash.

I'd planned to put together a demo video, but, well, haven't.

Also, as a side note, it does not fully correct very fast motion, due to a limitation in MVTools. There's not really anything I can do about that.
um3k is offline   Reply With Quote
Old 20th April 2010, 00:11   #9  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
i would pay good money to make or see this happen (and also as an independant program based on the Avisynth filters)
Undead Sega is offline   Reply With Quote
Old 1st May 2010, 19:13   #10  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
okay, i see no responses to that, hahaha, what i would like to mention is that many of the cameras sold today are unfortunately implemented with a CMOS chip, which will result giving the rolling shutter artifact, my problem is i want to shoot stuff using my HD CMOS camera and i would want to correct it as well for various good reasons (like tracking and adding visuals etc.), but after from what you're saying, it doesnt seem to be practical whatsoever?
Undead Sega is offline   Reply With Quote
Old 10th May 2010, 04:09   #11  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
ummm, anyone at all may i ask?

any samples or clips showing this in action?

i would also like to mention, with the foundry's rollingshutter plugin, apprantly not only it fixes the skews in the frame but it can also only align those that are skewed keeping those that werent in motion unaffected.
Undead Sega is offline   Reply With Quote
Old 10th May 2010, 14:55   #12  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
you could always shoot on the D20 or D21...

failing that, shooting around the defect is pretty easy - many a feature film have been shot with rolling shutter. the trick is not to move the camera too fast, and avoid flashing lights as they'll show up halfway down a frame and give you computer style "screen tearing".

i'd love to see the foundry plug in action, but i reckon it probably does the same as the avisynth script above - foundry have a large number of plugs based on their mo-comping engine.
__________________
sucking the life out of your videos since 2004
Mug Funky is offline   Reply With Quote
Old 11th May 2010, 20:45   #13  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
Well, for many many others like myself, we shoot on a Canon HV camera which is a great camercorder but at the same time it is unfortunate that is does have a CMOS chip implemented, and if we're honest there will be sojme or many times where fast movement of the camera has to come in, and this relates to what kind fo style the person is wanting to shoot, even steady movements of the camera is still affected because u still have movements happening anyways which all gets affected by rolling shutter anyways.

I think there should be a dedicated thread for this rolling shutter filter, just like TGMC and NNEDI etc. therfore people might be able to try it out?
Undead Sega is offline   Reply With Quote
Old 17th June 2010, 09:18   #14  |  Link
PitifulInsect
Registered User
 
Join Date: Nov 2008
Location: Sydney, Australia
Posts: 26
Roller

CAVEAT: Until a couple of days ago, I had never played with this, so I act all knowledgable but if I'm spouting any BS, please pull me up on it.
---
Hey there rollers!

Two presentations came to my attention in the last two days about how to fix rolling shutter issues - they're being presented at a conference this week so it's up-to-date stuff. I got interested, so googled and found this thread. I had a play with um3k/Justin Phillips' script. Examples and an annotated version in a moment, but a couple of comments:
  • There are two distinct classes of problem: When the motion is slower/smoother than the framerate, and when it's faster/shakier.
  • If the motion between frames is smooth, then temporally you don't really have motion aliasing, and Deshaker or a script like um3k's can be enough.
  • If the motion between frames is not smooth (and especially if it's periodic) then your video is temporally aliased, and no simple approach can handle it. (Think of the shake frequencies being above the Nyquist sampling of the frame-rate if that means anything to you)
  • So smooth panning can be cleaned up, but fast shaking, especially the "jellycam" examples where the camera is attached to a mechanically-vibrating platform like a car, are unrecoverable (by any means available to mortals).

The first paper, "Rectifying Rolling Shutter Video from Hand-held Devices", (Forssen and Ringaby, Linkoping University, Sweden, http://www.cvl.isy.liu.se/research/rs-dataset/0382.pdf), shows the ultimate way to solve the simple cases (smooth motion) efficiently, by tracking interest points (not using optical flow) and doing fancy stuff for the regularisation - read the paper if you're interested. Requires a static scene (no moving objects).

The second paper "Removing Rolling Shutter Wobble", (Baker, Bennett, Kang, and Szeliski, Microsoft Corporation, http://research.microsoft.com/pubs/121490/0198.pdf), actually uses temporal super-resolution in order to do fix the faster-than-framerate wobble sequences! Fancy stuff. The "simple" version is robust but requires a static scene (no moving objects), and there's a more-fancy version but it's not robust and it's pretty computationally expensive.

I learned:
Results can be very good, but they still aren't perfect with complex and expensive state-of-the-art techniques from the top experts in the field!
I also learned more about the problems involved, about super-resolution generally, etc, but the take-home message is that if you camera is shaking fast or vibrating, you're screwed. Don't let your camera shake or vibrate - it's never good anyway. If it's just panning - use deshaker, you won't immediately find better. The script suffers issues (so would deshaker I think - I haven't used it), but it can help.

So here's an annotated version of the script and results on the original video:
Code:
#
# The MVTools2 plugin is used by this script.  You'll find it at:
# http://avisynth.org.ru/mvtools/mvtools2.html
#
LoadPlugin("mvtools2.dll")


#
# Pixels begone!  Doing this at high-res is too annoying
# (I'm working off an 800-pixel-high laptop screen here!)
#
DirectShowSource("camileo3.mp4").ReduceBy2()


#   RollAway filter by um3k/Justin Phillips
#   Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
#   "Amount" is the same as that used for DeShaker. The formula can be found here:
#   http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
#   "Segh" is the height of the segments the image is split into. It should be an even number, and a dividend of the clip height.
#   This script is compatible with SetMTMode.
#   Use SetMemoryMax in your script to increase speed. Set it to a fairly high number that is reasonable for you computer's capabilities.
#
#   Alpha 2
function RollAway_alpha2(clip c, float amount, int segh, bool "test", bool "flip", int "pel", int "blk", int "ovr")
{
    #
    # Just syntactic housekeeping:  Define default values and set up some paramters.
    #
    Assert(c.height%segh == 0, "'segh' must be a dividend of clip height")
    ctime = amount/2.0
    segs = c.height/segh
    den = segs-1
    num = den
    test = Default(test, false)
    flip = Default(flip, false)
    pel = Default(pel, 1)
    blk = Default(blk, 16)
    ovr = Default(ovr, 12)

    #
    # Just uses MVTools - See http://avisynth.org.ru/mvtools/mvtools2.html
    # MSuper heirarchically decomposes the clip, while MAnalyse prepares
    # the motion vectors.
    #
    super = c.MSuper(pel=pel)
    fv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = false)
    bv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = true )

    #
    # Call the method below
    #
    DeRoll(c, super, fv, bv, ctime, den, segh, t=test, flip=flip)
}

#
# This function calls itself recursively to cut up the clip into vertical
# segments, then assemble them together.  Very clever!
# Each segment is created using MFlow
#
function DeRoll(clip c, clip super, clip fv, clip bv, float ctime, int den, int segh, int "segn", clip "stack", bool "t", bool "flip")
{
    #
    # More variable preparation - note especially that this is where the segment number 'segn' is increased.
    #
    stackh = Defined(stack) ? stack.height : 0
    segn   = Defined(segn) ? segn + 1 : 0
    num    = abs(den-segn*2)

    #
    # "flip" allows you to switch the forward and backward clips - I can't
    # figure out why exactly.  I think that it's for field parity for
    # interlaced videos.
    #
    time=ctime * float(num) / float(den)
    fclp = (flip==true) ? \
        c.MFlow(super, bv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1) : \
        c.MFlow(super, fv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1,  1)
    bclp = (flip==true) ? \
        c.MFlow(super, fv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1,  1) : \
        c.MFlow(super, bv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
    #
    # Figure out which segment of the stack we're at (past halfway?), and
    # whether we should be using a segment forward-interpolated from the last
    # frame, or back-interpolated from the next frame.
    #
    nseg = stackh > c.height/2 ? bclp : fclp

    #
    # Just add subtitles to explain what's going on for debugging
    #
    nseg = (t==true) ? \
        nseg.Subtitle("segn=" + String(segn) + "(num=" + String(num) + "  den=" + String(den) + ")  time=" + String(time)) : \
        nseg

    #
    # Build the stack - either begin it, or add the current segment to it.
    #
    stack = Defined(stack) ? StackVertical(stack, nseg) :   nseg

    #
    # Is the stack complete?  return the stacked image.  No?  Call recursively
    # to add the next segment.
    #
    return stack.height == c.height ? \
        stack : \
        DeRoll(c, super, fv, bv, ctime, den, segh, segn, stack, t, flip)
}

#
# Now just call the functions
#
StackHorizontal( \
    last.Subtitle("Original", x=Width()/2, align=8), \
    RollAway_alpha2(86.0, 2, test=false, flip=false, pel=2, blk=8, ovr=4).Subtitle("De-rolled", x=Width()/2, align=8) \
)
I couldn't find a single static frame to demonstrate the fact (it's more of a temporal matter), but for the wobble, the script just doesn't make a difference.
Worse, you sometimes get little artefacts from imperfect optical flow.
Worse still, is that small-artefacts aside, fast motion will sometimes kill the optical flow entirely, and then you get sadness all-around.



For smooth motion, you get strange effects during acceleration, but during the motion it more-or-less works.


So there you are - a whole bunch of info in one go. I hope that it's about right because as I said - I'm new to this so someone might know more. If you're reading and you know more - post an answer!

Last edited by Guest; 18th June 2010 at 00:23.
PitifulInsect is offline   Reply With Quote
Old 20th June 2010, 12:16   #15  |  Link
cretindesalpes
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
 
cretindesalpes's Avatar
 
Join Date: Feb 2009
Location: No support in PM
Posts: 712
I tried to improve the speed and memory usage of the script. I ended up modifying the MVTools2 to let MFlow() accept the motion compensation time parameter as a clip, the time being defined for each pixel instead of a single scalar for the whole clip.

The visual quality hasn't changed but it resulted in a x10 speed up improvement over the original script (tested on terka's example camileo3.mp4 with segh=4 and pel=4). And now that the memory usage has become fair, it is possible to SetMTMode() it on HD material, giving an impressive x40 speed boost on a quad core CPU!

I also tried the same thing on MFlowInter() but the results were disappointing. It generates much more artifacts than a simple MFlow(). However I kept it in the script so you can try to play with it and find better parameters.

You can find the modified MVTools2 in this package. And the updated script below:

Code:
#   RollAway filter by um3k/Justin Phillips/Firesledge
#   Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
#   "Amount" is the same as that used for DeShaker. The formula can be found here:
#   http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
#   This script is compatible with SetMTMode.
#
#   Alpha 2 mod
function RollAway_alpha2mod (clip c, float amount, bool "flip", int "pel", int "blk", int "ovr", bool "inter")
{
    #
    # Just syntactic housekeeping:  Define default values and set up some paramters.
    #
    flip  = Default(flip,  false)
    inter = Default(inter, false)
    pel   = Default(pel,       1)
    blk   = Default(blk,      16)
    ovr   = Default(ovr,      12)

    #
    # We'll split the picture at a height multiple of 4, and this split
    # line is used as the reference time (no motion interpolation here).
    # Therefore we need that the time function equals 0 at this line :
    # t = amount * abs (ofs - y), with ofs close to 0.5 and y in
    # the range 0 to 1.
    # MFlowInter() timeclip formula is a bit different.
    #
    w = c.Width ()
    h = c.Height ()
    sy = 4 * Round (h / 8)	# Split line
    ofs = Float (sy) / Float (h)

    OFS = String (ofs)
    AMT = String (amount)
    expr_norm  = OFS + " y - abs " + AMT + " * 2.56 *"
    expr_inter = OFS + " y - " + AMT + " * 2.56 * y " + OFS + " >= 256 0 ? +"
    expr = (inter) ? expr_inter : expr_norm

    #
    # Just uses MVTools - See http://avisynth.org.ru/mvtools/mvtools2.html
    # MSuper heirarchically decomposes the clip, while MAnalyse prepares
    # the motion vectors.
    #
    super = c.MSuper(pel=pel)
    fv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = false)
    bv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = true )

    #
    # "flip" allows you to switch the forward and backward clips - I can't
    # figure out why exactly.  I think that it's for field parity for
    # interlaced videos.
    #
    timeclip = c.mt_lutspa (relative=true, expr=expr, y=3, u=3, v=3)
    fclp =
\     (inter) ? c.MFlowInter (super, bv, fv, tclip=timeclip)
\   : (flip ) ? c.MFlow(super, bv, tclip=timeclip).SelectEvery(1, -1)
\   :           c.MFlow(super, fv, tclip=timeclip).SelectEvery(1,  1)
    bclp =
\     (inter) ? fclp.SelectEvery(1, -1)
\   : (flip ) ? c.MFlow(super, fv, tclip=timeclip).SelectEvery(1,  1)
\   :           c.MFlow(super, bv, tclip=timeclip).SelectEvery(1, -1)

    #
    # Figure out which segment of the stack we're at (past halfway?), and
    # whether we should be using a segment forward-interpolated from the last
    # frame, or back-interpolated from the next frame.
    #
    top = fclp.Crop (0, 0, w, sy)
    bot = bclp.Crop (0, sy, w, h - sy)
    stack = StackVertical (top, bot)

    return (stack)
}

Last edited by cretindesalpes; 20th June 2010 at 12:22. Reason: Typo
cretindesalpes is offline   Reply With Quote
Old 20th June 2010, 14:18   #16  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
PitifulInsect: Good job, thanks. It's been a while since I worked on any of my Avisynth functions, it kind of comes in bursts. The "flip" parameter was created to deal with an oddball video from, I believe, an iPhone. The rolling shutter effect seemed to be reversed and required special treatment.

cretindesalpes: Wow! Awesome! This is exactly the sort of thing I was hoping someone would do, as I lack the programming ability to modify MVTools. I'm gonna give it a try and see how it does.
um3k is offline   Reply With Quote
Old 21st June 2010, 16:22   #17  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
Ok, cretindesalpes, I gave it a try. It works pretty well, but there is one major downfall, and that is the 8-bit limited color depth of Avisynth. This results in stair-stepping the time clip, which translates to discontinuous motion in the interpolated image. I think the next step in the developing of this filter would be to port it completely into a new MVTools function, which would allow the use of a smooth gradient and optimized speed and memory usage. The problem, of course, is my lack of programming skills. If someone would take up this task, they would earn the thanks of a great many camcorder users.
um3k is offline   Reply With Quote
Old 24th June 2010, 11:46   #18  |  Link
Terka
Registered User
 
Join Date: Jan 2005
Location: cz
Posts: 704
have an idea how deal with RS:
1.calibrate a camera for different movements speeds.
for every speed got frames: good, distorted. calculate motion vectors for their comparsion.
2.estimate the speed using global motion compensation (depan).
3.use mcompensate (using vectors from 1)
Terka is offline   Reply With Quote
Old 25th June 2010, 04:55   #19  |  Link
Undead Sega
Registered User
 
Join Date: Oct 2007
Posts: 713
Quote:
Originally Posted by Terka View Post
have an idea how deal with RS:
1.calibrate a camera for different movements speeds.
for every speed got frames: good, distorted. calculate motion vectors for their comparsion.
2.estimate the speed using global motion compensation (depan).
3.use mcompensate (using vectors from 1)
Possible if you can elaborate on this for those who are not totally knowlegable to this language (I understand abit but sometimes I do have trouble )

Almost sounds like a solution to Rolling Shutter
Undead Sega is offline   Reply With Quote
Old 28th June 2010, 15:30   #20  |  Link
PitifulInsect
Registered User
 
Join Date: Nov 2008
Location: Sydney, Australia
Posts: 26
Quote:
Originally Posted by Terka View Post
have an idea how deal with RS:[...]
No, Terka, that would not help at all.

Rolling shutter causes a very simple problem, but it is very difficult to remove it because you can't know enough about the scene to undo it. The Microsoft-paper approach is maybe the closest approximation possible.

Lets say that you had a very simple scene with a vertical line in it:
Code:
+-----------+
|     |     |
|     |     |
|     |     |
+-----------+
If the camera is moving at a constant speed, then you can "simply" interpolate back and forth in time. Um3k and cretindesalpes have the script which can solve this perfectly:
Code:
+-----------+      +-----------+          +-----------+
|       /   |      |     /     |          |     |     |
|      /    |      |    /      |    ->    |     |     |
|     /     |      |   /       |          |     |     |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02             solution is
                                      difference of positions
But when the camera moves in any non-linear way (which is all the time, for example acceleration before and after even the simple motion as above), the problem is hard. Um3k and cretindesalpes' script will not solve the problem because the interpolation is only linear (though it will help a little, so it is probably still the best option available):
Code:
+-----------+      +-----------+          +-----------+
|     |     |      |     |     |          |     |     |
|     |     |      |    /      |    ->    |     /     |
|     |     |      |  /        |          |    /      |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02         difference of positions
                (imagine a curling         doesn't work
                  line like a 'j')
You can imagine that the problem is even worse when the camera is moving up and down also.

To solve the problem, you need to know the camera position at every instant in time to know where/when to compensate the data from. If you have the position only at every frame, and you try to use that, then your solution is even worse than above:
Code:
+-----------+      +-----------+          +-----------+
|     |     |      |     |     |          |      \    |
|     |     |      |    /      |    ->    |      |    |
|     |     |      |  /        |          |     /     |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02           global per-frame
                (imagine a curling         motion is not
                line like a 'j' as            enough
                camera accelerates)
If the camera shakes a lot, the problem is worse and you get "jellycam" (search Youtube):
Code:
+-----------+      +-----------+          +-----------+
|     |     |      |      \    |          |           |
|     |     |      |     /     |    ->    |     ?     |
|     |     |      |     \     |          |           |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02
                  (camera shakes)
And of course this problem is the worst because then the video is motion-blurred, and then knowing the position to compensate to is not possible (I don't mean hard, I mean that it is mathematically not possible at all).

If one day we have optical flow methods good enough to measure position at every *line* of a video, then perhaps a per-line compensated approach would be possible, but even then, only if the video has enough detail! Such optical-flow technology does not exist though, and there would not be enough detail in most videos anyway to make this possible. The Microsoft approach measures what information is available, and uses temporal super-resolution methods to discover the rest, which is very clever, but again: They must be able to measure motion, so their method breaks (as does every other known method) when the scene has moving objects which pollute the flow-field.

So if your camera has a rolling shutter, don't shake it about!
PitifulInsect is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:20.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.