miamicanes
15th December 2010, 01:48
Suppose I have two videos that were encoded from the same 720p24 source (source not available to me):
* a 640x360 copy that's 24fps, blurry, and has poor color fidelity
* a 1280x720 copy that's 12fps, decently-sharp, and has temporal artifacts from hell when played back because whomever did it basically just threw away every other frame and called it a day.
Both have an average bitrate of around 1mbit/second (360p24 is WMV, 720p12 is h.264).
Using Avisynth and a plugin like MVflowFPS, is there any way to:
a) normalize the two and associate each 12fps frame with a 24fps frame (keeping in mind that even if both were encoded from the same 720p24 source, things like frame-drops and slightly different cropping, not to mention different encoding algorithms, might cause them to drift slightly with respect to one another over time). The expected result would be identifying one specific 12fps and one specific 24fps frame as the starting point (that both are regarded as the same moment in time), with subsequent 12fps frames correlating to roughly every other 24fps frame.
b) analyze the 24fps video to derive the authoritative motion vectors
c) interpolate the 12fps video using the motion vectors derived from the 24fps video, and intelligently blending the 12fps and 24fps source frames with a mask derived from those motion vectors to obtain the new intermediate 720p24 frames.
d) Go back over the whole thing and downsample it from 720p24 to 480p24 to further smooth out any quirks that might have resulted from the interpolation.
In theory, it seems like this would only be about a half step beyond what MVflowFPS already does (the only really NEW part being the normalization to account for the fact that the two are "mostly" equivalent temporally, but might have an occasional frame that diverges from that ideal), but I've never seen anything written about actually DOING something like this.
In terms of use case, it's one of those scenarios that would never occur in real video production(*) (where you have the source), but is common in real-world web video where you, as the end user, might have two versions you can download that are both slaughtered for the sake of meeting a bitrate target... one that's HD, but low-framerate, and one that's bit-starved SD with poor color & slightly-blocky detail, but high-framerate. Something like this would enable you to combine both into one that's better than the sum of its parts (and, if everyone downloading the videos ends up downloading both and using twice the bandwidth, might eventually lead the site to realize it would be better off if it just encoded proper 720p30 in the first place).
(*) OK, actually, I *can* think of a real-world scenario where this could be very handy. For various reasons having to do with light and physics, 300fps video isn't happening anytime soon. Unfortunately, 300fps is the least common denominator of 50 and 60. HOWEVER, suppose someone were to make a video camera that shoots normal 1280x720 frames at 60fps, but squeezes in an occasional underexposed frame with poor color depth and effective resolution of ~640x360 in between 60fps frames at a time that happens to more or less align with the instant a 50fps frame would have been taken.
Using a scheme like I described above in post-production, you could then take those underexposed, low-res (possibly even monochrome) frames and use them to interpolate the 50fps frame that SHOULD have been there from the adjacent 60fps frames. Suddenly, shooting hybrid video that can be edited for native 50 AND 60fps playback (as long as you only cut on a point where both 50 and 60fps video fall at the same instant) becomes feasible. The 50fps version wouldn't look as good as the 60fps version, but it would probably still be nicer to watch than 60fps butchered into a 50fps mess of judder. ;-)
* a 640x360 copy that's 24fps, blurry, and has poor color fidelity
* a 1280x720 copy that's 12fps, decently-sharp, and has temporal artifacts from hell when played back because whomever did it basically just threw away every other frame and called it a day.
Both have an average bitrate of around 1mbit/second (360p24 is WMV, 720p12 is h.264).
Using Avisynth and a plugin like MVflowFPS, is there any way to:
a) normalize the two and associate each 12fps frame with a 24fps frame (keeping in mind that even if both were encoded from the same 720p24 source, things like frame-drops and slightly different cropping, not to mention different encoding algorithms, might cause them to drift slightly with respect to one another over time). The expected result would be identifying one specific 12fps and one specific 24fps frame as the starting point (that both are regarded as the same moment in time), with subsequent 12fps frames correlating to roughly every other 24fps frame.
b) analyze the 24fps video to derive the authoritative motion vectors
c) interpolate the 12fps video using the motion vectors derived from the 24fps video, and intelligently blending the 12fps and 24fps source frames with a mask derived from those motion vectors to obtain the new intermediate 720p24 frames.
d) Go back over the whole thing and downsample it from 720p24 to 480p24 to further smooth out any quirks that might have resulted from the interpolation.
In theory, it seems like this would only be about a half step beyond what MVflowFPS already does (the only really NEW part being the normalization to account for the fact that the two are "mostly" equivalent temporally, but might have an occasional frame that diverges from that ideal), but I've never seen anything written about actually DOING something like this.
In terms of use case, it's one of those scenarios that would never occur in real video production(*) (where you have the source), but is common in real-world web video where you, as the end user, might have two versions you can download that are both slaughtered for the sake of meeting a bitrate target... one that's HD, but low-framerate, and one that's bit-starved SD with poor color & slightly-blocky detail, but high-framerate. Something like this would enable you to combine both into one that's better than the sum of its parts (and, if everyone downloading the videos ends up downloading both and using twice the bandwidth, might eventually lead the site to realize it would be better off if it just encoded proper 720p30 in the first place).
(*) OK, actually, I *can* think of a real-world scenario where this could be very handy. For various reasons having to do with light and physics, 300fps video isn't happening anytime soon. Unfortunately, 300fps is the least common denominator of 50 and 60. HOWEVER, suppose someone were to make a video camera that shoots normal 1280x720 frames at 60fps, but squeezes in an occasional underexposed frame with poor color depth and effective resolution of ~640x360 in between 60fps frames at a time that happens to more or less align with the instant a 50fps frame would have been taken.
Using a scheme like I described above in post-production, you could then take those underexposed, low-res (possibly even monochrome) frames and use them to interpolate the 50fps frame that SHOULD have been there from the adjacent 60fps frames. Suddenly, shooting hybrid video that can be edited for native 50 AND 60fps playback (as long as you only cut on a point where both 50 and 60fps video fall at the same instant) becomes feasible. The 50fps version wouldn't look as good as the 60fps version, but it would probably still be nicer to watch than 60fps butchered into a 50fps mess of judder. ;-)