rolling shutter

[Terka]

is it possible to fix rolling shutter (d)effect?
some camcorder have it.
http://cpressvideo.superhosting.cz/s...9/camileo3.mp4

[morsa]

It is and it is not.Better ask manufacturers to waste a couple more pennies on sensors and choose to have Global shutter on CMOS...

(or at least a so fast readout that RS becomes almost invisible)

[Terka]

yes, i know. ;( i meant avisynth way.

[2Bdecided]

VirtualDub deshaker claims to try - it doesn't help with my camcorder though.

Cheers,
David.

[Undead Sega]

There is actually a recent plugin for Nuke by The Foundry that now corrects video with Rolling Shutter issues:

http://www.thefoundry.co.uk/pkg_over...1-FB380B2119EF

but it is bloody expensive, if only someone took the concept of this and made it as a avisynth filter, can it be done? and wat would they need? a .dll of the sfotware or something?

[scharfis_brain]

have just tried to search for "rolling shutter" in the forums?

actually there seems to be a solution:
http://forum.doom9.org/showthread.ph...er#post1374487

[Undead Sega]

that seems interesting, but the user claims "it works. Sometimes." Im assuming that it would only fix the typical panning effects of rolling shutter, abit like from Deshaker, if you know wat i mean. Also, i dont seem to bee is anyone has tried it out?

[um3k]

It works pretty well, actually, except that it's a memory hog, somewhat unstable, extremely slow, etc. But it's a proof-of-concept, and gives acceptable results on small videos (and crashes on large ones). The good part is that it corrects all rolling shutter motion artifacts, both global and local, much like the Foundry plugin. Someone just needs to implement it as a subfunction of MVTools with optimizations, something which I am currently not capable of. Here is the latest version, though I have not messed with it in months:

#   RollAway filter by um3k/Justin Phillips
#   Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
#   "Amount" is the same as that used for DeShaker. The formula can be found here:
#   http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
#   "Segh" is the height of the segments the image is split into. It should be an even number, and a dividend of the clip height.
#   This script is compatible with SetMTMode.
#   Use SetMemoryMax in your script to increase speed. Set it to a fairly high number that is reasonable for you computer's capabilities.
#
#   Alpha 2

function RollAway_alpha2(clip c, float amount, int segh, bool "test", bool "flip", int "pel", int "blk", int "ovr", int "thSAD")
{
    Assert(c.height%segh == 0, "'segh' must be a dividend of clip height")
    ctime = amount/2.0
    segs = c.height/segh
    den = segs-1
    num = den
    test = Default(test, false)
    flip = Default(flip, false)
    pel = Default(pel, 1)
    blk = Default(blk, 16)
    ovr = Default(ovr, 12)
    thSAD = Default(thSAD, 100)
    super = c.MSuper(pel=pel)
    fv = MAnalyse(super, blksize=blk, isb = false, overlap=ovr)
    bv = MAnalyse(super, blksize=blk, isb = true, overlap=ovr)
    
    DeRoll(c, super, fv, bv, ctime, den, segh, t=test, flip=flip)
}

function DeRoll(clip c, clip super, clip fv, clip bv, float ctime, int den, int segh, int "segn", clip "stack", bool "t", bool "flip")
{
    stackh  = Defined(stack) ? stack.height
    \   :   0
    segn    = Defined(segn) ? segn+1
    \   :   0
    num     = abs(den-segn*2)
    fclp      = (flip==true) ? c.MFlow(super, bv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
    \   :   c.MFlow(super, fv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, 1)
    bclp     = (flip==true) ? c.MFlow(super, fv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, 1)
    \   :   c.MFlow(super, bv, time=ctime*float(num)/float(den)).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
    nseg    = stackh > c.height/2 ? bclp
    \   :   fclp
    nseg = (t==true) ? nseg.Subtitle("num=" + String(num) + "  den=" + String(den) + "  segn=" + String(segn) + "  ctime=" + String(ctime*float(num)/float(den)))
    \   :   nseg
    stack   = Defined(stack) ? StackVertical(stack, nseg)
    \   :   nseg
    segn = segn    return stack.height == c.height ? stack
    \   :   DeRoll(c, super, fv, bv, ctime, den, segh, segn, stack, t, flip)
}

It basically motion compensates the frame numerous times, cuts up the frames, and recombines them into a new one. Needless to say, it is quite slow. Try it on a video downscaled to SD resolution, HD is practically guaranteed to crash.

I'd planned to put together a demo video, but, well, haven't.

Also, as a side note, it does not fully correct very fast motion, due to a limitation in MVTools. There's not really anything I can do about that.

[Undead Sega]

i would pay good money to make or see this happen (and also as an independant program based on the Avisynth filters)

[Undead Sega]

okay, i see no responses to that, hahaha, what i would like to mention is that many of the cameras sold today are unfortunately implemented with a CMOS chip, which will result giving the rolling shutter artifact, my problem is i want to shoot stuff using my HD CMOS camera and i would want to correct it as well for various good reasons (like tracking and adding visuals etc.), but after from what you're saying, it doesnt seem to be practical whatsoever?

[Undead Sega]

ummm, anyone at all may i ask?

any samples or clips showing this in action?

i would also like to mention, with the foundry's rollingshutter plugin, apprantly not only it fixes the skews in the frame but it can also only align those that are skewed keeping those that werent in motion unaffected.

[Mug Funky]

you could always shoot on the D20 or D21...

failing that, shooting around the defect is pretty easy - many a feature film have been shot with rolling shutter. the trick is not to move the camera too fast, and avoid flashing lights as they'll show up halfway down a frame and give you computer style "screen tearing".

i'd love to see the foundry plug in action, but i reckon it probably does the same as the avisynth script above - foundry have a large number of plugs based on their mo-comping engine.

[Undead Sega]

Well, for many many others like myself, we shoot on a Canon HV camera which is a great camercorder but at the same time it is unfortunate that is does have a CMOS chip implemented, and if we're honest there will be sojme or many times where fast movement of the camera has to come in, and this relates to what kind fo style the person is wanting to shoot, even steady movements of the camera is still affected because u still have movements happening anyways which all gets affected by rolling shutter anyways.

I think there should be a dedicated thread for this rolling shutter filter, just like TGMC and NNEDI etc. therfore people might be able to try it out?

[PitifulInsect]

CAVEAT: Until a couple of days ago, I had never played with this, so I act all knowledgable but if I'm spouting any BS, please pull me up on it.
---
Hey there rollers!

Two presentations came to my attention in the last two days about how to fix rolling shutter issues - they're being presented at a conference this week so it's up-to-date stuff. I got interested, so googled and found this thread. I had a play with um3k/Justin Phillips' script. Examples and an annotated version in a moment, but a couple of comments:

• There are two distinct classes of problem: When the motion is slower/smoother than the framerate, and when it's faster/shakier.
• If the motion between frames is smooth, then temporally you don't really have motion aliasing, and Deshaker or a script like um3k's can be enough.
• If the motion between frames is not smooth (and especially if it's periodic) then your video is temporally aliased, and no simple approach can handle it. (Think of the shake frequencies being above the Nyquist sampling of the frame-rate if that means anything to you)
• So smooth panning can be cleaned up, but fast shaking, especially the "jellycam" examples where the camera is attached to a mechanically-vibrating platform like a car, are unrecoverable (by any means available to mortals).

The first paper, "Rectifying Rolling Shutter Video from Hand-held Devices", (Forssen and Ringaby, Linkoping University, Sweden, http://www.cvl.isy.liu.se/research/rs-dataset/0382.pdf), shows the ultimate way to solve the simple cases (smooth motion) efficiently, by tracking interest points (not using optical flow) and doing fancy stuff for the regularisation - read the paper if you're interested. Requires a static scene (no moving objects).

The second paper "Removing Rolling Shutter Wobble", (Baker, Bennett, Kang, and Szeliski, Microsoft Corporation, http://research.microsoft.com/pubs/121490/0198.pdf), actually uses temporal super-resolution in order to do fix the faster-than-framerate wobble sequences! Fancy stuff. The "simple" version is robust but requires a static scene (no moving objects), and there's a more-fancy version but it's not robust and it's pretty computationally expensive.

I learned:
Results can be very good, but they still aren't perfect with complex and expensive state-of-the-art techniques from the top experts in the field!
I also learned more about the problems involved, about super-resolution generally, etc, but the take-home message is that if you camera is shaking fast or vibrating, you're screwed. Don't let your camera shake or vibrate - it's never good anyway. If it's just panning - use deshaker, you won't immediately find better. The script suffers issues (so would deshaker I think - I haven't used it), but it can help.

So here's an annotated version of the script and results on the original video:

#
# The MVTools2 plugin is used by this script.  You'll find it at:
# http://avisynth.org.ru/mvtools/mvtools2.html
#
LoadPlugin("mvtools2.dll")


#
# Pixels begone!  Doing this at high-res is too annoying
# (I'm working off an 800-pixel-high laptop screen here!)
#
DirectShowSource("camileo3.mp4").ReduceBy2()


#   RollAway filter by um3k/Justin Phillips
#   Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
#   "Amount" is the same as that used for DeShaker. The formula can be found here:
#   http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
#   "Segh" is the height of the segments the image is split into. It should be an even number, and a dividend of the clip height.
#   This script is compatible with SetMTMode.
#   Use SetMemoryMax in your script to increase speed. Set it to a fairly high number that is reasonable for you computer's capabilities.
#
#   Alpha 2
function RollAway_alpha2(clip c, float amount, int segh, bool "test", bool "flip", int "pel", int "blk", int "ovr")
{
    #
    # Just syntactic housekeeping:  Define default values and set up some paramters.
    #
    Assert(c.height%segh == 0, "'segh' must be a dividend of clip height")
    ctime = amount/2.0
    segs = c.height/segh
    den = segs-1
    num = den
    test = Default(test, false)
    flip = Default(flip, false)
    pel = Default(pel, 1)
    blk = Default(blk, 16)
    ovr = Default(ovr, 12)

    #
    # Just uses MVTools - See http://avisynth.org.ru/mvtools/mvtools2.html
    # MSuper heirarchically decomposes the clip, while MAnalyse prepares
    # the motion vectors.
    #
    super = c.MSuper(pel=pel)
    fv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = false)
    bv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = true )

    #
    # Call the method below
    #
    DeRoll(c, super, fv, bv, ctime, den, segh, t=test, flip=flip)
}

#
# This function calls itself recursively to cut up the clip into vertical
# segments, then assemble them together.  Very clever!
# Each segment is created using MFlow
#
function DeRoll(clip c, clip super, clip fv, clip bv, float ctime, int den, int segh, int "segn", clip "stack", bool "t", bool "flip")
{
    #
    # More variable preparation - note especially that this is where the segment number 'segn' is increased.
    #
    stackh = Defined(stack) ? stack.height : 0
    segn   = Defined(segn) ? segn + 1 : 0
    num    = abs(den-segn*2)

    #
    # "flip" allows you to switch the forward and backward clips - I can't
    # figure out why exactly.  I think that it's for field parity for
    # interlaced videos.
    #
    time=ctime * float(num) / float(den)
    fclp = (flip==true) ? \
        c.MFlow(super, bv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1) : \
        c.MFlow(super, fv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1,  1)
    bclp = (flip==true) ? \
        c.MFlow(super, fv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1,  1) : \
        c.MFlow(super, bv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
    #
    # Figure out which segment of the stack we're at (past halfway?), and
    # whether we should be using a segment forward-interpolated from the last
    # frame, or back-interpolated from the next frame.
    #
    nseg = stackh > c.height/2 ? bclp : fclp

    #
    # Just add subtitles to explain what's going on for debugging
    #
    nseg = (t==true) ? \
        nseg.Subtitle("segn=" + String(segn) + "(num=" + String(num) + "  den=" + String(den) + ")  time=" + String(time)) : \
        nseg

    #
    # Build the stack - either begin it, or add the current segment to it.
    #
    stack = Defined(stack) ? StackVertical(stack, nseg) :   nseg

    #
    # Is the stack complete?  return the stacked image.  No?  Call recursively
    # to add the next segment.
    #
    return stack.height == c.height ? \
        stack : \
        DeRoll(c, super, fv, bv, ctime, den, segh, segn, stack, t, flip)
}

#
# Now just call the functions
#
StackHorizontal( \
    last.Subtitle("Original", x=Width()/2, align=8), \
    RollAway_alpha2(86.0, 2, test=false, flip=false, pel=2, blk=8, ovr=4).Subtitle("De-rolled", x=Width()/2, align=8) \
)

I couldn't find a single static frame to demonstrate the fact (it's more of a temporal matter), but for the wobble, the script just doesn't make a difference.
Worse, you sometimes get little artefacts from imperfect optical flow.
Worse still, is that small-artefacts aside, fast motion will sometimes kill the optical flow entirely, and then you get sadness all-around.



For smooth motion, you get strange effects during acceleration, but during the motion it more-or-less works.


So there you are - a whole bunch of info in one go. I hope that it's about right because as I said - I'm new to this so someone might know more. If you're reading and you know more - post an answer!

[cretindesalpes]

I tried to improve the speed and memory usage of the script. I ended up modifying the MVTools2 to let MFlow() accept the motion compensation time parameter as a clip, the time being defined for each pixel instead of a single scalar for the whole clip.

The visual quality hasn't changed but it resulted in a x10 speed up improvement over the original script (tested on terka's example camileo3.mp4 with segh=4 and pel=4). And now that the memory usage has become fair, it is possible to SetMTMode() it on HD material, giving an impressive x40 speed boost on a quad core CPU!

I also tried the same thing on MFlowInter() but the results were disappointing. It generates much more artifacts than a simple MFlow(). However I kept it in the script so you can try to play with it and find better parameters.

You can find the modified MVTools2 in this package. And the updated script below:

#   RollAway filter by um3k/Justin Phillips/Firesledge
#   Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
#   "Amount" is the same as that used for DeShaker. The formula can be found here:
#   http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
#   This script is compatible with SetMTMode.
#
#   Alpha 2 mod
function RollAway_alpha2mod (clip c, float amount, bool "flip", int "pel", int "blk", int "ovr", bool "inter")
{
    #
    # Just syntactic housekeeping:  Define default values and set up some paramters.
    #
    flip  = Default(flip,  false)
    inter = Default(inter, false)
    pel   = Default(pel,       1)
    blk   = Default(blk,      16)
    ovr   = Default(ovr,      12)

    #
    # We'll split the picture at a height multiple of 4, and this split
    # line is used as the reference time (no motion interpolation here).
    # Therefore we need that the time function equals 0 at this line :
    # t = amount * abs (ofs - y), with ofs close to 0.5 and y in
    # the range 0 to 1.
    # MFlowInter() timeclip formula is a bit different.
    #
    w = c.Width ()
    h = c.Height ()
    sy = 4 * Round (h / 8)	# Split line
    ofs = Float (sy) / Float (h)

    OFS = String (ofs)
    AMT = String (amount)
    expr_norm  = OFS + " y - abs " + AMT + " * 2.56 *"
    expr_inter = OFS + " y - " + AMT + " * 2.56 * y " + OFS + " >= 256 0 ? +"
    expr = (inter) ? expr_inter : expr_norm

    #
    # Just uses MVTools - See http://avisynth.org.ru/mvtools/mvtools2.html
    # MSuper heirarchically decomposes the clip, while MAnalyse prepares
    # the motion vectors.
    #
    super = c.MSuper(pel=pel)
    fv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = false)
    bv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = true )

    #
    # "flip" allows you to switch the forward and backward clips - I can't
    # figure out why exactly.  I think that it's for field parity for
    # interlaced videos.
    #
    timeclip = c.mt_lutspa (relative=true, expr=expr, y=3, u=3, v=3)
    fclp =
\     (inter) ? c.MFlowInter (super, bv, fv, tclip=timeclip)
\   : (flip ) ? c.MFlow(super, bv, tclip=timeclip).SelectEvery(1, -1)
\   :           c.MFlow(super, fv, tclip=timeclip).SelectEvery(1,  1)
    bclp =
\     (inter) ? fclp.SelectEvery(1, -1)
\   : (flip ) ? c.MFlow(super, fv, tclip=timeclip).SelectEvery(1,  1)
\   :           c.MFlow(super, bv, tclip=timeclip).SelectEvery(1, -1)

    #
    # Figure out which segment of the stack we're at (past halfway?), and
    # whether we should be using a segment forward-interpolated from the last
    # frame, or back-interpolated from the next frame.
    #
    top = fclp.Crop (0, 0, w, sy)
    bot = bclp.Crop (0, sy, w, h - sy)
    stack = StackVertical (top, bot)

    return (stack)
}

[um3k]

PitifulInsect: Good job, thanks. It's been a while since I worked on any of my Avisynth functions, it kind of comes in bursts. The "flip" parameter was created to deal with an oddball video from, I believe, an iPhone. The rolling shutter effect seemed to be reversed and required special treatment.

cretindesalpes: Wow! Awesome! This is exactly the sort of thing I was hoping someone would do, as I lack the programming ability to modify MVTools. I'm gonna give it a try and see how it does.

[um3k]

Ok, cretindesalpes, I gave it a try. It works pretty well, but there is one major downfall, and that is the 8-bit limited color depth of Avisynth. This results in stair-stepping the time clip, which translates to discontinuous motion in the interpolated image. I think the next step in the developing of this filter would be to port it completely into a new MVTools function, which would allow the use of a smooth gradient and optimized speed and memory usage. The problem, of course, is my lack of programming skills. If someone would take up this task, they would earn the thanks of a great many camcorder users.

[Terka]

have an idea how deal with RS:
1.calibrate a camera for different movements speeds.
for every speed got frames: good, distorted. calculate motion vectors for their comparsion.
2.estimate the speed using global motion compensation (depan).
3.use mcompensate (using vectors from 1)

[Undead Sega]

Originally Posted by Terka

have an idea how deal with RS:
1.calibrate a camera for different movements speeds.
for every speed got frames: good, distorted. calculate motion vectors for their comparsion.
2.estimate the speed using global motion compensation (depan).
3.use mcompensate (using vectors from 1)

Possible if you can elaborate on this for those who are not totally knowlegable to this language (I understand abit but sometimes I do have trouble )

Almost sounds like a solution to Rolling Shutter

[PitifulInsect]

Originally Posted by Terka

have an idea how deal with RS:[...]

No, Terka, that would not help at all.

Rolling shutter causes a very simple problem, but it is very difficult to remove it because you can't know enough about the scene to undo it. The Microsoft-paper approach is maybe the closest approximation possible.

Lets say that you had a very simple scene with a vertical line in it:

+-----------+
|     |     |
|     |     |
|     |     |
+-----------+

If the camera is moving at a constant speed, then you can "simply" interpolate back and forth in time. Um3k and cretindesalpes have the script which can solve this perfectly:

+-----------+      +-----------+          +-----------+
|       /   |      |     /     |          |     |     |
|      /    |      |    /      |    ->    |     |     |
|     /     |      |   /       |          |     |     |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02             solution is
                                      difference of positions

But when the camera moves in any non-linear way (which is all the time, for example acceleration before and after even the simple motion as above), the problem is hard. Um3k and cretindesalpes' script will not solve the problem because the interpolation is only linear (though it will help a little, so it is probably still the best option available):

+-----------+      +-----------+          +-----------+
|     |     |      |     |     |          |     |     |
|     |     |      |    /      |    ->    |     /     |
|     |     |      |  /        |          |    /      |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02         difference of positions
                (imagine a curling         doesn't work
                  line like a 'j')

You can imagine that the problem is even worse when the camera is moving up and down also.

To solve the problem, you need to know the camera position at every instant in time to know where/when to compensate the data from. If you have the position only at every frame, and you try to use that, then your solution is even worse than above:

+-----------+      +-----------+          +-----------+
|     |     |      |     |     |          |      \    |
|     |     |      |    /      |    ->    |      |    |
|     |     |      |  /        |          |     /     |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02           global per-frame
                (imagine a curling         motion is not
                line like a 'j' as            enough
                camera accelerates)

If the camera shakes a lot, the problem is worse and you get "jellycam" (search Youtube):

+-----------+      +-----------+          +-----------+
|     |     |      |      \    |          |           |
|     |     |      |     /     |    ->    |     ?     |
|     |     |      |     \     |          |           |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02
                  (camera shakes)

And of course this problem is the worst because then the video is motion-blurred, and then knowing the position to compensate to is not possible (I don't mean hard, I mean that it is mathematically not possible at all).

If one day we have optical flow methods good enough to measure position at every *line* of a video, then perhaps a per-line compensated approach would be possible, but even then, only if the video has enough detail! Such optical-flow technology does not exist though, and there would not be enough detail in most videos anyway to make this possible. The Microsoft approach measures what information is available, and uses temporal super-resolution methods to discover the rest, which is very clever, but again: They must be able to measure motion, so their method breaks (as does every other known method) when the scene has moving objects which pollute the flow-field.

So if your camera has a rolling shutter, don't shake it about!