Video with no V-Sync

[Velocity 7]

Yes, I'm back and here for some more mayhem.

Okay, so I have some footage from ANUBIS: Zone of the Enders, and the game is supposed to be 60 fps. However, sometimes when there is too much on-screen, the game slows down and starts rendering "incomplete" frames; any idea how to deal with this and get rid of these "incomplete" frames (replacing them with a whole, complete frame)?

http://webhome.idirect.com/~tihhhuhv/sample_1.avi

P.S. What are good (not best mind you, since asking for the "best" usually results in trouble) ways of dealing with static + moving deinterlacing, other than maybe TDeint?

[mg262]

P.S. What are good (not best mind you, since asking for the "best" usually results in trouble) ways of dealing with static + moving deinterlacing, other than maybe TDeint?

See here: http://forum.doom9.org/showthread.php?p=690725#post690725

[Velocity 7]

Thanks.

Anyways besides that small note, maybe there might be some way to merge the "incomplete" frames with other frames, as seen here:
http://webhome.idirect.com/~tihhhuhv/pic0-1.JPG

Anyways, when the game slows down, it doesn't cut back from 60 fps straight to 30 fps; the frame rate is variable (just like in my MGS3 thread), and will go from 60 to maybe 58, 45, and so on. As a result these "incomplete" frames are drawn. It's very likely you people will find this in computer-output videos as well (say, outputting UT2004 video through S-Video is likely to have this result).

Or, maybe a theoretical possibility:

1. Make note of non-interlaced sections of frame (call it the inner frame)
2. Separate the fields of the entire frame to fields A and B
3. Find what parts of fields A and B match up with the inner frame and create
the "outer frame"
4. Merge together an image
5. Interpolate the "outer frame" sections


Using Photoshop I've come up with an idea of what this method should be doing except for step #5, which I haven't a clue how to do. >_>

http://webhome.idirect.com/~tihhhuhv/pic0-2.JPG

You can clearly tell where the "non-interlaced" section is in this picture.

[scharfis_brain]

Those incomplete Frames are a result of disabled VSync.

[Velocity 7]

I guess that's a better way of putting it.

[Velocity 7]

That makes more sense... question is, is there any script or plugin out there that can properly handle this?

This is venturing into the Development thread likely, but some pseudo-code to consider:max_vertical_size = size of frame vertically in pixels // for NTSC this is 480 usually

frame_A = interlaced section of frame before the non-interlaced section
frame_B = non-interlaced section of frame
frame_C = interlaced section of frame after the non-interlaced section

non_A = start position of non-interlaced section vertically, from 0 to max_vertical_size
non_B = end position of non-interlaced section vertically, from 0 to max_vertical_size

field_A = first field of frame
field_B = second field of frame

final_frame = final frame to put out

open frame
do 100% interlace check, report as x from 0 to 100

if (x = 100),
do dumb bob
end thread
else if (0 < x < 100),
generate frame_B, non_A, and non_B
separate into two fields, called field_A and field_B

// checking field_A for continuity
if field_A continuous from 0 to non_B // yes, 0 to non_B
write section (0 to non_A) into frame_A
else if field_A continuous from non_A to max_vertical_size
write section (non_B to max_vertical_size) into frame_C

// checking field_B for continuity
if field_B continuous from 0 to non_B
write section (0 to non_A) into frame_A
else if field_B continuous from non_A to max_vertical_size
write section (non_B to max_vertical_size) into frame_C

final_frame = 2x(frame_A) + frame_B + 2x(frame_C)
// doubling vertical size of frame_A and frame_C since the two are still in the
// field format that was presented

duplicate final_frame twice as output // instead of dumb bob
end thread
else
final_frame = current frame
duplicate final_frame twice as output // very nice for 30fps in 60fps
end thread

Questions for this code:


Will this start causing stuttering in motion?
Is there any way to interpolate frame_A and frame_C more accurately, based on frame_B? (to create a more progressive image, obviously)
What if, in order to complete a progressive image properly, requires the frame before or after the one being processed? (Say, if you have frames A, B, and C, and frame B is the strange v-sync'ed one, what if frame A and C might have the sections that would match? Is this even possible?)
Big problem! What if frame_A or frame_C is empty?! (Is this possible?)
Anyone out there who is willing to write this code in actual, practical form? (Putting this to the AVISynth development forum? I'm not an experienced programmer, and never dealt with scripting AVISynth or coding plugins for AVISynth at all...)

[Velocity 7]

That makes more sense... question is, is there any script or plugin out there that can properly handle this?

This is venturing into the Development thread likely, but some pseudo-code to consider:max_vertical_size = size of frame vertically in pixels // for NTSC this is 480 usually

frame_A = interlaced section of frame before the non-interlaced section
frame_B = non-interlaced section of frame
frame_C = interlaced section of frame after the non-interlaced section

non_A = start position of non-interlaced section vertically, from 0 to max_vertical_size
non_B = end position of non-interlaced section vertically, from 0 to max_vertical_size

field_A = first field of frame
field_B = second field of frame

final_frame = final frame to put out

open frame
do 100% interlace check, report as x from 0 to 100

if (x = 100),
do dumb bob
end thread
else if (0 < x < 100),
generate frame_B, non_A, and non_B
separate into two fields, called field_A and field_B

// checking field_A for continuity
if field_A continuous from 0 to non_B // yes, 0 to non_B
write section (0 to non_A) into frame_A
else if field_A continuous from non_A to max_vertical_size
write section (non_B to max_vertical_size) into frame_C

// checking field_B for continuity
if field_B continuous from 0 to non_B
write section (0 to non_A) into frame_A
else if field_B continuous from non_A to max_vertical_size
write section (non_B to max_vertical_size) into frame_C

final_frame = 2x(frame_A) + frame_B + 2x(frame_C)
// doubling vertical size of frame_A and frame_C since the two are still in the
// field format that was presented

duplicate final_frame twice as output // instead of dumb bob
end thread
else
final_frame = current frame
duplicate final_frame twice as output // very nice for 30fps in 60fps
end thread

Questions for this code:


Will this start causing stuttering in motion?
Is there any way to interpolate frame_A and frame_C more accurately, based on frame_B? (to create a more progressive image, obviously)
What if, in order to complete a progressive image properly, requires the frame before or after the one being processed? (Say, if you have frames A, B, and C, and frame B is the strange v-sync'ed one, what if frame A and C might have the sections that would match? Is this even possible?)
Big problem! What if frame_A or frame_C is empty?! (Is this possible?)
Anyone out there who is willing to write this code in actual, practical form? (Putting this to the AVISynth development forum? I'm not an experienced programmer, and never dealt with scripting AVISynth or coding plugins for AVISynth at all...)

[Velocity 7]

That makes more sense... question is, is there any script or plugin out there that can properly handle this?

This is venturing into the Development thread likely, but some pseudo-code to consider:max_vertical_size = size of frame vertically in pixels // for NTSC this is 480 usually

frame_A = interlaced section of frame before the non-interlaced section
frame_B = non-interlaced section of frame
frame_C = interlaced section of frame after the non-interlaced section

non_A = start position of non-interlaced section vertically, from 0 to max_vertical_size
non_B = end position of non-interlaced section vertically, from 0 to max_vertical_size

field_A = first field of frame
field_B = second field of frame

final_frame = final frame to put out

open frame
do 100% interlace check, report as x from 0 to 100

if (x = 100),
do dumb bob
end thread
else if (0 < x < 100),
generate frame_B, non_A, and non_B
separate into two fields, called field_A and field_B

// checking field_A for continuity
if field_A continuous from 0 to non_B // yes, 0 to non_B
write section (0 to non_A) into frame_A
else if field_A continuous from non_A to max_vertical_size
write section (non_B to max_vertical_size) into frame_C

// checking field_B for continuity
if field_B continuous from 0 to non_B
write section (0 to non_A) into frame_A
else if field_B continuous from non_A to max_vertical_size
write section (non_B to max_vertical_size) into frame_C

final_frame = 2x(frame_A) + frame_B + 2x(frame_C)
// doubling vertical size of frame_A and frame_C since the two are still in the
// field format that was presented

duplicate final_frame twice as output // instead of dumb bob
end thread
else
final_frame = current frame
duplicate final_frame twice as output // very nice for 30fps in 60fps
end thread

Questions for this code:


Will this start causing stuttering in motion?
Is there any way to interpolate frame_A and frame_C more accurately, based on frame_B? (to create a more progressive image, obviously)
What if, in order to complete a progressive image properly, requires the frame before or after the one being processed? (Say, if you have frames A, B, and C, and frame B is the strange v-sync'ed one, what if frame A and C might have the sections that would match? Is this even possible?)
Big problem! What if frame_A or frame_C is empty?! (Is this possible?)
Anyone out there who is willing to write this code in actual, practical form? (Putting this to the AVISynth development forum? I'm not an experienced programmer, and never dealt with scripting AVISynth or coding plugins for AVISynth at all...)

[scharfis_brain]

IMO. trying to restore this will be unsuccesful.

imagine this seqence of fields:

(A) (1/4A + 3/4B) (C) (1/2D + 1/2E) (E) (F) (3/4F + 1/4G) (H)

I made it random by intention.

so you can see, that fields 2,4 and 7 ore incomplete ones. there is missing image data.

one *could* try to detect it by partial field matching and try to do some kind of motion interpolation, but this process will be horrible slow and results aren't guaranteed.

[mic]

The term *theoretically* was used... That's very much what this is:... :D

If the prob is bad frames (or more accurately fields :)), and you have ample fps, then I think I'd personally try removing the bad frames in a way that wasn't too noticable in the overall flow of the video. I don't know if this would look better then the bad fields (would have to try a test and see), and if desired, frames could be inserted to bring back to full 60 fps.

To remove the frames with bad fields, I'd try an adaptive ivt as used in V/Dub. I know it isn't possible to ivt non-telecined material, but by the same logic, telicine is impossible too. :D I'd set the fps to 29.97 or 30 prior to ivt, and while working on avi -- it can be set back easily enough when done.

To increase fps to compensate for ivt I think one method would be to use Vegas, which will generate new frames based on before and after. The other way I'd try would be to telecine the ivt'd video. I don't know which would look better, if either was an improvement over ivt'd video. Vegas wouldn't really know where to insert, yet the adaptive ivt wouldn't always correspond to telecine pattern.

But again, just theory, and the way I think I'd approach this to see if anything could be done. It should just take a small bit of time to find out...

[Velocity 7]

OT: I should have checked up on this thread and continued it from here, but what the hell. (http://forum.doom9.org/showthread.php?t=87080)

Hmm, still bugs me how I could handle this... I really wouldn't like having portions of a frame showing up like that.

[mg262]

I don't mean to be overly negative, but my feeling is that it would be very difficult to fix this up with existing filters. At the least, it seems to me somewhat similar to so-called "peculiar blending", which (as far as I know) can't be treated effectively in AVISynth at present. But I could well be wrong!

[Trixter]

Here's a different viewpoint: What's wrong with just keeping it interlaced? The output of the game is interlaced 30i, you dump to 30i, and you play back 30i -- the end result is completely transparent and nothing is lost. Why the massive effort to try to create "perfect" 60p material out of 30i material for which there is not always new information in every field?

Maybe I'm missing the original intention -- are you trying to "fix" the output of the game, filling in parts that weren't rendered during gameplay? In that case, you're trying to generate "tween" frames based on the previous and next frame, and I've used motion-synthesis products like Twixtor (not great) and Realviz Retimer (better) to do that, but those products cost 4 figures and are used in production environments -- not exactly in the realm of avisynth (yet).

[mic]

I've used motion-synthesis products like Twixtor (not great) and Realviz Retimer (better) to do that, but those products cost 4 figures and are used in production environments -- not exactly in the realm of avisynth (yet).
Vegas has had this since 4, though not in realm of software mentioned I'm sure.

[Trixter]

Vegas doesn't have motion-vector synthesis in any released version. I think you're confusing it's integrated field blending for interframe motion tweening.

[mic]

Vegas doesn't have motion-vector synthesis in any released version. I think you're confusing it's integrated field blending for interframe motion tweening.
I think you're right on motion-vector. Vegas uses a simpler method to: "generate "tween" frames based on the previous and next frame" -- more along the lines of advanced morphing algos. with vectors only for FX I think. Using supersampling the prog increases accuracy generating additional temporary frames for calculations. At any rate, it does "generate "tween" frames." :) And it doesn't do this: "...in realm of software mentioned I'm sure." :)

[Trixter]

No, you're still confused -- Vegas tweens only in the motion controls. I'm talking about something else completely: Tweening the source video itself.

What products like ReTimer, etc. do is they analyze each frame of video and, compared to the previous frame, attempt to determine which areas are moving. If you have this kind of motion data, you can generate inbetween frames that contain images that appear to have been recorded at a higher rate, but are in fact synthesized using the motion data. It's very similar to MPEG encoding, actually, except that MPEG determines motions on a block-by-block basis, while products like ReTimer, MotionPerfect, etc. try to determine the motion of each PIXEL. This is why they take so long to work (hours if not days depending on what you are doing). Some of them are cheap (MotionPerfect) and others are expensive because they allow you to do more things with the motion, like slowly bend it, or *fix* it if the program didn't guess properly in certain sections.

If you want to see for yourself what I'm talking about, go to dynapel.com and download a trial of MotionPerfect. Take a short 10-second 320x240 clip with a low framerate, like 12 or 15fps, and run it through MotionPerfect at 30fps, then look at the output. You'll see that, unless the motion in the original clip is extremely drastic, the clip will have smooth motion as if it were originally recorded at that speed.

To do things like this in avisynth, I believe there is a new set of plugins that deal with motion vectors (something like MVxxx?) but don't recall the names or locations, sorry.

[Velocity 7]

How about dropping frames that have these anomalies? And creating a variable framerate video as a result? The game is slowing down anyway at the points mentioned.

[scharfis_brain]

Forget about ANY attempt to fix this problem, please!
Just Encode interlaced MPEG2 and author a DVD afterwards.

There are rows of teared fields. This means if you are dropping them all, you'll get a slide show.

Also, I don't know of a stable algorthim to detect thes teared fields.