CAVEAT: Until a couple of days ago, I had never played with this, so I act all knowledgable but if I'm spouting any BS, please pull me up on it.
---
Hey there rollers!
Two presentations came to my attention in the last two days about how to fix rolling shutter issues - they're being presented at a conference this week so it's up-to-date stuff. I got interested, so googled and found this thread. I had a play with um3k/Justin Phillips' script. Examples and an annotated version in a moment, but a couple of comments:
- There are two distinct classes of problem: When the motion is slower/smoother than the framerate, and when it's faster/shakier.
- If the motion between frames is smooth, then temporally you don't really have motion aliasing, and Deshaker or a script like um3k's can be enough.
- If the motion between frames is not smooth (and especially if it's periodic) then your video is temporally aliased, and no simple approach can handle it. (Think of the shake frequencies being above the Nyquist sampling of the frame-rate if that means anything to you)
- So smooth panning can be cleaned up, but fast shaking, especially the "jellycam" examples where the camera is attached to a mechanically-vibrating platform like a car, are unrecoverable (by any means available to mortals).
The first paper, "Rectifying Rolling Shutter Video from Hand-held Devices", (Forssen and Ringaby, Linkoping University, Sweden,
http://www.cvl.isy.liu.se/research/rs-dataset/0382.pdf), shows the ultimate way to solve the simple cases (smooth motion) efficiently, by tracking interest points (not using optical flow) and doing fancy stuff for the regularisation - read the paper if you're interested. Requires a static scene (no moving objects).
The second paper "Removing Rolling Shutter Wobble", (Baker, Bennett, Kang, and Szeliski, Microsoft Corporation,
http://research.microsoft.com/pubs/121490/0198.pdf), actually uses temporal super-resolution in order to do fix the faster-than-framerate wobble sequences! Fancy stuff. The "simple" version is robust but requires a static scene (no moving objects), and there's a more-fancy version but it's not robust and it's pretty computationally expensive.
I learned:
Results can be very good, but they still aren't perfect with complex and expensive state-of-the-art techniques from the top experts in the field!
I also learned more about the problems involved, about super-resolution generally, etc, but the take-home message is that if you camera is shaking fast or vibrating, you're screwed. Don't let your camera shake or vibrate - it's never good anyway. If it's just panning - use deshaker, you won't immediately find better. The script suffers issues (so would deshaker I think - I haven't used it), but it can help.
So here's an annotated version of the script and results on the original video:
Code:
#
# The MVTools2 plugin is used by this script. You'll find it at:
# http://avisynth.org.ru/mvtools/mvtools2.html
#
LoadPlugin("mvtools2.dll")
#
# Pixels begone! Doing this at high-res is too annoying
# (I'm working off an 800-pixel-high laptop screen here!)
#
DirectShowSource("camileo3.mp4").ReduceBy2()
# RollAway filter by um3k/Justin Phillips
# Reduces rolling shutter artifacts on video from CMOS-chipped camcorders/DSLRS
# "Amount" is the same as that used for DeShaker. The formula can be found here:
# http://www.guthspot.se/video/deshaker.htm#rolling shutter setting
# "Segh" is the height of the segments the image is split into. It should be an even number, and a dividend of the clip height.
# This script is compatible with SetMTMode.
# Use SetMemoryMax in your script to increase speed. Set it to a fairly high number that is reasonable for you computer's capabilities.
#
# Alpha 2
function RollAway_alpha2(clip c, float amount, int segh, bool "test", bool "flip", int "pel", int "blk", int "ovr")
{
#
# Just syntactic housekeeping: Define default values and set up some paramters.
#
Assert(c.height%segh == 0, "'segh' must be a dividend of clip height")
ctime = amount/2.0
segs = c.height/segh
den = segs-1
num = den
test = Default(test, false)
flip = Default(flip, false)
pel = Default(pel, 1)
blk = Default(blk, 16)
ovr = Default(ovr, 12)
#
# Just uses MVTools - See http://avisynth.org.ru/mvtools/mvtools2.html
# MSuper heirarchically decomposes the clip, while MAnalyse prepares
# the motion vectors.
#
super = c.MSuper(pel=pel)
fv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = false)
bv = MAnalyse(super, blksize=blk, overlap=ovr, truemotion=true, searchparam=8, isb = true )
#
# Call the method below
#
DeRoll(c, super, fv, bv, ctime, den, segh, t=test, flip=flip)
}
#
# This function calls itself recursively to cut up the clip into vertical
# segments, then assemble them together. Very clever!
# Each segment is created using MFlow
#
function DeRoll(clip c, clip super, clip fv, clip bv, float ctime, int den, int segh, int "segn", clip "stack", bool "t", bool "flip")
{
#
# More variable preparation - note especially that this is where the segment number 'segn' is increased.
#
stackh = Defined(stack) ? stack.height : 0
segn = Defined(segn) ? segn + 1 : 0
num = abs(den-segn*2)
#
# "flip" allows you to switch the forward and backward clips - I can't
# figure out why exactly. I think that it's for field parity for
# interlaced videos.
#
time=ctime * float(num) / float(den)
fclp = (flip==true) ? \
c.MFlow(super, bv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1) : \
c.MFlow(super, fv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, 1)
bclp = (flip==true) ? \
c.MFlow(super, fv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, 1) : \
c.MFlow(super, bv, time=time).Crop(0, segn*segh, -0, segh).SelectEvery(1, -1)
#
# Figure out which segment of the stack we're at (past halfway?), and
# whether we should be using a segment forward-interpolated from the last
# frame, or back-interpolated from the next frame.
#
nseg = stackh > c.height/2 ? bclp : fclp
#
# Just add subtitles to explain what's going on for debugging
#
nseg = (t==true) ? \
nseg.Subtitle("segn=" + String(segn) + "(num=" + String(num) + " den=" + String(den) + ") time=" + String(time)) : \
nseg
#
# Build the stack - either begin it, or add the current segment to it.
#
stack = Defined(stack) ? StackVertical(stack, nseg) : nseg
#
# Is the stack complete? return the stacked image. No? Call recursively
# to add the next segment.
#
return stack.height == c.height ? \
stack : \
DeRoll(c, super, fv, bv, ctime, den, segh, segn, stack, t, flip)
}
#
# Now just call the functions
#
StackHorizontal( \
last.Subtitle("Original", x=Width()/2, align=8), \
RollAway_alpha2(86.0, 2, test=false, flip=false, pel=2, blk=8, ovr=4).Subtitle("De-rolled", x=Width()/2, align=8) \
)
I couldn't find a single static frame to demonstrate the fact (it's more of a temporal matter), but for the wobble, the script just doesn't make a difference.
Worse, you sometimes get little artefacts from imperfect optical flow.
Worse still, is that small-artefacts aside, fast motion will sometimes kill the optical flow entirely, and then you get sadness all-around.
For smooth motion, you get strange effects during acceleration, but during the motion it more-or-less works.
So there you are - a whole bunch of info in one go. I hope that it's about right because as I said - I'm new to this so someone might know more. If you're reading and you know more - post an answer!