Trading resolution for framerate.

[Seedmanc]

This will be yet another theorizing and speculation post, so bear with me.

Say, would it be possible to take a high-resolution video, say, 2560x1440p @ 30fps and turn it into a 1280x720p but 60fps without the usual MFlowFPS/SVP stuff? I've got this idea while thinking about the way (de)interlacing and Superresolution work.

First, some kinds of interlacing (probably the original TV kind) work by storing two fields of a frame, each 1/2 in height but at double framerate. Thus, when doing deinterlacing you usually have a choice either to keep resolution by blending fields together while sacrificing framerate, or keep the framerate by interpolating them to full height, but that essentially means upscaling (losing resolution). What if there was a way to do that with progressive video?
Second, you might know about the Superresolution (http://staff.utia.cas.cz/sroubekf/papers/TIP_07.pdf) filter which upscales video based on motion analysis, using data from adjacent frames to construct new pixels for the upscaled output. What if there was a way to do the opposite process, reconstructing frames from excessive pixels?

In essence, I think that hi-res video provides higher fidelity of motion, which is also a feature of high framerate. Think of a low-res video being watched on a large screen, even at high framerate the motion would seem jerky as the individual pixels switch on and off. Thus, both spatial and temporal resolutions and connected and must be interchangeable.

Unfortunately I can't produce any solution to these questions yet. I have a feeling it must be about separating video into even and odd rows/colums of pixels and messing with that but the idea eludes me. What do you make of this?

[vivan]

Thus, when doing deinterlacing you usually have a choice either to keep resolution by blending fields together while sacrificing framerate, or keep the framerate by interpolating them to full height, but that essentially means upscaling (losing resolution).Third option is motion compensated deinterlacing - upscaling and then bringing details from adjustment frames. That's why QTGMC is actually a real superres filter.

Essentially that sort of deinterlacing is a motion interoplation, it just has more reference points to work with (not just adjustment frames, but also adjustment pixels and this is why it has better quilaty (still there're motion interpolation artifacts).

[Seedmanc]

Oh right thanks for reminding me about MC in application to deinterlacing, I keep forgetting about that method. Almost made a release with the dumb blend instead.

But what about my idea? Is it only applicable for pixel-level motion (where the distance traveled per frame is comparable to the pixel size difference between original and downscale)?

[kuchikirukia]

What gave you the idea you could turn 30fps into 60 without interpolation? You have 30, not 60. 30, not 60. 30, not 60. 30. not 60.

[FranceBB]

@seedmanc... Uhm... so... let's assume you have a video which is NTSC interlaced (not telecined). There are three main different ways to deinterlace it: the"normal" one which outputs a 30p without duplicates, the blending, which merges fields together, and the Bob, which outputs a 60p. Basically, you are saying that you would like to "Bob" deinterlace, using only the Even fields first to make an half resolution frame, and then the odd fields to make another half resolution frame, am I correct?
So, what you are looking for is an half resolution bobbing, am I right?

[wonkey_monkey]

So, what you are looking for is an half resolution bobbing, am I right?

That's just separatefields(). The result will judder slightly, as it'll display the even lines and the odd lines in the same positions, instead of offset by 1 row from each other. Bobbing shifts (bobs) the separated fields up and down to avoid this.

[raffriff42]

you might know about the Superresolution (http://staff.utia.cas.cz/sroubekf/papers/TIP_07.pdf) filter which upscales video based on motion analysis, using data from adjacent frames to construct new pixels for the upscaled output. What if there was a way to do the opposite process, reconstructing frames from excessive pixels?MVTools (http://avisynth.nl/index.php/MVTools) does exactly as you say: interpolates new "in-between" frames, using motion information as a guide.

For example, check out DoubleRate (http://forum.doom9.org/showthread.php?p=1789149#post1789149) in the ongoing "slo-mo" thread (http://forum.doom9.org/showthread.php?t=174089)

[Seedmanc]

FranceBB, davidhorman, yeah and that works but only for interlaced video. If I apply that to progressive video, regardless of resolution the even frames will be just duplicates of the odd ones.

raffriff42, that's the common method of doing that, I was wondering about somehow making use of excess pixels in a hi-res video to produce new frames instead of the way mvtools do that. It might seem strange, but I explained my motivation in the OP-post, the part about motion fidelity being a product of both framerate and resolution.

[johnmeyer]

I've known a lot of Russian mathematicians, and they have training beyond anything I received in engineering school and therefore can often think of things that would never occur to me.

However ...

I completely fail to see any relationship or correlation between spatial and temporal information. Even with infinite resolution, there is no way whatsoever to predict where pixels will move next. Having more pixels (i.e., a super-high resolution video) doesn't change that basic fact.

Motion estimation can use past motion to attempt to predict the future, but like all such tools (including multivariate regression, which attempts to predict the future based on past data points), it fails completely when the next move fails to follow the past pattern.

[kuchikirukia]

I completely fail to see any relationship or correlation between spatial and temporal information. Even with infinite resolution, there is no way whatsoever to predict where pixels will move next. Having more pixels (i.e., a super-high resolution video) doesn't change that basic fact.

This. One has nothing to do with another. If you take a picture of a kid with a 20MP camera that won't tell you if he'll have grandchildren some day.
Pixels can't see into the future.

[Bloax]

I completely fail to see any relationship or correlation between spatial and temporal information. Even with infinite resolution, there is no way whatsoever to predict where pixels will move next. Having more pixels (i.e., a super-high resolution video) doesn't change that basic fact.
The only correlation that exists is that on quite a number of video cameras, you can record in a lower resolution at a higher framerate.

The reason this is possible, however, is because it splits up the frame "grabbing" process from the different "pixel" sensors in the camera into a sequence of "first grab the frame with this half" and "then just a little bit after that, grab the frame with the other half", not because it splits the full input of the sensor into different times.

Obviously a very simplified explanation, but that's not a bad thing in this situation.