Thread: rolling shutter
View Single Post
Old 28th June 2010, 15:30   #20  |  Link
PitifulInsect
Registered User
 
Join Date: Nov 2008
Location: Sydney, Australia
Posts: 26
Quote:
Originally Posted by Terka View Post
have an idea how deal with RS:[...]
No, Terka, that would not help at all.

Rolling shutter causes a very simple problem, but it is very difficult to remove it because you can't know enough about the scene to undo it. The Microsoft-paper approach is maybe the closest approximation possible.

Lets say that you had a very simple scene with a vertical line in it:
Code:
+-----------+
|     |     |
|     |     |
|     |     |
+-----------+
If the camera is moving at a constant speed, then you can "simply" interpolate back and forth in time. Um3k and cretindesalpes have the script which can solve this perfectly:
Code:
+-----------+      +-----------+          +-----------+
|       /   |      |     /     |          |     |     |
|      /    |      |    /      |    ->    |     |     |
|     /     |      |   /       |          |     |     |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02             solution is
                                      difference of positions
But when the camera moves in any non-linear way (which is all the time, for example acceleration before and after even the simple motion as above), the problem is hard. Um3k and cretindesalpes' script will not solve the problem because the interpolation is only linear (though it will help a little, so it is probably still the best option available):
Code:
+-----------+      +-----------+          +-----------+
|     |     |      |     |     |          |     |     |
|     |     |      |    /      |    ->    |     /     |
|     |     |      |  /        |          |    /      |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02         difference of positions
                (imagine a curling         doesn't work
                  line like a 'j')
You can imagine that the problem is even worse when the camera is moving up and down also.

To solve the problem, you need to know the camera position at every instant in time to know where/when to compensate the data from. If you have the position only at every frame, and you try to use that, then your solution is even worse than above:
Code:
+-----------+      +-----------+          +-----------+
|     |     |      |     |     |          |      \    |
|     |     |      |    /      |    ->    |      |    |
|     |     |      |  /        |          |     /     |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02           global per-frame
                (imagine a curling         motion is not
                line like a 'j' as            enough
                camera accelerates)
If the camera shakes a lot, the problem is worse and you get "jellycam" (search Youtube):
Code:
+-----------+      +-----------+          +-----------+
|     |     |      |      \    |          |           |
|     |     |      |     /     |    ->    |     ?     |
|     |     |      |     \     |          |           |
+-----------+      +-----------+          +-----------+
  frame 01            frame 02
                  (camera shakes)
And of course this problem is the worst because then the video is motion-blurred, and then knowing the position to compensate to is not possible (I don't mean hard, I mean that it is mathematically not possible at all).

If one day we have optical flow methods good enough to measure position at every *line* of a video, then perhaps a per-line compensated approach would be possible, but even then, only if the video has enough detail! Such optical-flow technology does not exist though, and there would not be enough detail in most videos anyway to make this possible. The Microsoft approach measures what information is available, and uses temporal super-resolution methods to discover the rest, which is very clever, but again: They must be able to measure motion, so their method breaks (as does every other known method) when the scene has moving objects which pollute the flow-field.

So if your camera has a rolling shutter, don't shake it about!
PitifulInsect is offline   Reply With Quote