Recreating missing frames

[Floatingshed]

I hope this doesn't violate rule 6...

A friend has captured a VHS recording and binned the tape.
It is an off-air recording he made in the 1980's. He proudly showed me what he had done and it is terrible.

The tape was most certainly PAL 25fps (BBC 1 1983) but his capture is 23.976 fps. Stepping through it I can clearly see that the frame change has been achieved by dropping one frame every second. This causes a very jarring movement issue.

Someone please give me some pointers to re-creating the missing frames...

Thanks.

[ronnylov]

Here is a thread describing how to replace bad frames with interpolated frames: http://forum.doom9.org/showthread.php?t=161154
Maybe some similar method could work?

[Floatingshed]

Hmm, interesting reading but I'm simply not clever enough to expand it for my case.
I don't have any "bad" frames, they are missing. How can a filter act upon something that is missing?
I wonder if it is possible to insert a blank frame where one is missing, (the missing frames do seem to be in a regular pattern), somehow mark it as "bad" and have mvtools re-create a frame based on the previous and next...

Sometimes I wish I had a brain...
Somebody please help. Thanks.

[Didée]

Not all hope is lost. The problem has been solved before.

The most-cruicial point is to identify the points where a frame is missing. Having those, the rest is relatively easy: double the framerate with dups, replace (or maskmerge) flow-interpolated frames at the identified positions, finally decimate down to target framerate.

I've tackled the problem a few times in the past, each time with a different evaluation strategy.

Here (http://forum.doom9.org/showthread.php?p=879309#post879309) is an older reference of an even more complicated case (interlaced source, which required to interpolate two independent fields instead of only one progressive frame.)

At later times, one time I fiddled a method with manual frame-differencing instead of ScriptClip'ing, like e.g. this here (http://forum.doom9.org/showthread.php?p=1409324#post1409324).
The latest one, a few months ago, used a more simple method by (ab)using SCSelect() to identify the jump positions.

I'll have to browse my script archive, there should be something that fits.

(Note - you probably cannot use the linked scripts out-of-the-box. They aren't generalized, and need adjustment for your case of 23.976<>25 fps.)

[laserfan]

I don't have any "bad" frames, they are missing. How can a filter act upon something that is missing?

Sometimes I wish I had a brain...
A "filter" is just a term that's thrown around--these guys are talking about Avisynth scripts that in many cases work magic.

If you upload a small sample clip of the program somewhere e.g. mediafire, often times that will inspire one of the experts here to suggest a repair method.

[johnmeyer]

I used some of the great code that Didée developed in the second link he provided above, and used it to do something similar to what you are trying to do. My attempts are recorded in this recent thread:

Automatically fix dups followed (eventually) by drops (http://forum.doom9.org/showthread.php?p=1510780#post1510780)

Perhaps some of what is discussed in that thread will be useful to you.

[Floatingshed]

All this looks very promising, thanks.

[mike20021969]

I hope this doesn't violate rule 6...

A friend has captured a VHS recording and binned the tape.


Don't count on it :eek:

This guy no longer has his DVD's...
http://forum.doom9.org/showthread.php?t=162180

[Floatingshed]

Having studied and played with the various scripts and ideas linked here I am still struggling with this. Any further help would be much appreciated. Thanks.

[Floatingshed]

I've been playing with a manual fix, just to acquaint myself with the techniques required here.
I wrote a macro for AvsP which will insert a blank frame where I see a large jump. Then using mugfunky's "morph" I can create a good frame in its place and the results are excellent. I'm rather baffled though how I get back to the original framerate... Other than speeding up using Assumefps. Remember this is a 25fps recording captured with a card set to 23.976(!) so one frame every second has been dropped.
Then there's the audio, of course there is now a one frame pause in the audio every second.
Obviously doing this manually is not really an option, the clip is 50 minutes long and every second needs work!

I tried to adapt johnmeyer's fix dup & drops script but it seemed very hit and miss. The metrics were quite distinctly different when a frame was missing (dropping from 1.3456 to 0.87299 (etc)) but the mask didn't always reflect this. However I was stumbling about in the dark so I'm sure it was my fault.

@Didee Did you manage to find anything useful in your script scrap book?

Thanks.

[jmac698]

This is similar to something to I want to write, but I'm repairing semi-regular dropped frames by the capture card. Can you post a video clip to train on?

[Didée]

@Didee Did you manage to find anything useful
*cough* Frankly, I didn't even look yet. I realized there is one difficult point in your case: the long pattern cycle (1 out of 24), due to 25->24 decimation. In those "usual" cases I've dealt with, it's rather short cycles as (1 in 4) or (1 in 5), when 30i/p was falsely IVTC'ed, or framedropped to PAL.

The difficulty is this: the raw detection of missing frames is never really safe, but somewhat vague. And if you have a confidence of say 90%, then the remaining 10% can absolutely ruin your day. Therefore, it's strongly indicated to make "justification" for each frame with its "in-pattern neighbors". (To avoid false positives as well as missed drops.) This is okay for short cycles, looking 9 or 11 frames back & ahead doesn't put much stress on the chain as a whole. But I've a hunch that things might change when you need to look 50 frames back & ahead, and then follow-up the whole shebang with TDecimate, trying to dec from 25(50) to 23.976 ....

"Code visualisation":
The code that actually worked for the short-cycle cases, would need to be adjusted to sth similar like the following, for the long-cycle case:

(Not necessarily 'TemporalSoften', but that's the basic idea.)
interleave( selectevery(24, 0).temporalsoften(2,...), \
selectevery(24, 1).temporalsoften(2,...), \
selectevery(24, 2).temporalsoften(2,...), \
selectevery(24, 3).temporalsoften(2,...), \
selectevery(24, 4).temporalsoften(2,...), \
selectevery(24, 5).temporalsoften(2,...), \
selectevery(24, 6).temporalsoften(2,...), \
selectevery(24, 7).temporalsoften(2,...), \
selectevery(24, 8).temporalsoften(2,...), \
selectevery(24, 9).temporalsoften(2,...), \
selectevery(24,10).temporalsoften(2,...), \
selectevery(24,11).temporalsoften(2,...), \
selectevery(24,12).temporalsoften(2,...), \
selectevery(24,13).temporalsoften(2,...), \
selectevery(24,14).temporalsoften(2,...), \
selectevery(24,15).temporalsoften(2,...), \
selectevery(24,16).temporalsoften(2,...), \
selectevery(24,17).temporalsoften(2,...), \
selectevery(24,18).temporalsoften(2,...), \
selectevery(24,19).temporalsoften(2,...), \
selectevery(24,20).temporalsoften(2,...), \
selectevery(24,21).temporalsoften(2,...), \
selectevery(24,22).temporalsoften(2,...), \
selectevery(24,23).temporalsoften(2,...) )

Technically correct, but it doesn't feel savy. Its a pretty big window, and TDecimate is required afterwards, which would impose another (big) search window on the already-window'ed construct. Making long-range correlations via scripting is not a joyride in Avisynth.
(A two-pass scenario should drastically reduce the frame-reading/caching pressure, though.)

It would be good to first make a closer inspecion of the source. There's a fair chance that the drop pattern is fixed 'n static (if no "smart" decimator was used, but a fixed operation that behaved like ChangeFPS or similar). In that case, all this "smart reckognition" would not be needed at all, and a fixed replacing pattern could be used instead.

[Floatingshed]

Here is a sample of the clip in question...

http://www.mediafire.com/?aux674sdba5s18m

All help gratefully received.

[MatLz]

The sample isn't long enough to identify the pattern.
It should be at least 2997 frames. (if it was real 24/1.001 fps, 960 frames should have been enough but it's 2997/125 fps....weird...).

[Floatingshed]

My apologies.
A longer sample here: http://www.mediafire.com/?84pqi1ggww7kql2

[MatLz]

Theory seems to work.
But that would be easier to work since the begining of your movie.
Sometimes, there is no motion, so difficult to see missing frames :

avisource("source.avi")
a=stackvertical(trim(0,2996),trim(2997,5993))
b=stackvertical(trim(5994,8990),trim(8991,11987))
stackhorizontal(a,b)

This code will help you to identify the missing frames.
If you can't see one on a part, you will certainly see in an other.

Personaly I would write something like :
8, 31, 55, 78, 102, 125, 148, 172, 195, 219.....nnnn (first frames of example2.avi)
where missing frames follow the number (so the interpolation will be between 8 and 9, 31 and 32 .... n and n+1)

After that, a clever function is needed to apply on the ranges which begin by a mod%2997 frame an interpolation of the missing frames on the pattern found.

I actually don't have the time (and maybe not the brain power) to write such a function but some avisynth gods are here... :)

What you need to do first is identify the pattern since the begining with the code I gave you.

[Floatingshed]

Working from the beginning of the file is difficult as it begins with several seconds of VHS freeze frame, followed by an animated title sequence! Neither are very helpful for spotting frame errors.

I don't understand the relevance of frames 2996/7 or indeed the purpose of the code. I must be very dim, please explain further.

[MatLz]

In your case, 25fps has been decimated to 23.976fps (2997/125).

The ratio between these two fps is 25/(2997/125)
which gives : 3125/2997

That means 3125frames at 25fps are equal in duration to 2997frames at 23.976fps.

You can NOT reduce this fraction to other small integers.
That means the ROUNDINGS in the calculations done during the decimation has ONLY been finished when atteining the 3125th frame of the 25fps source, giving the 2997th frame of the 23.976fps output.

The decimation, after that, BEGINS a new cycle of the pattern.



I assume :) the vhs freeze frames are not part of the movie, so you can trim() them.
But after that, don't touch it !

As I wrote, the stacks horizontal/vertical are here to help you to find the missing frames in the same relative frame position in the pattern but in an other cycle....so if the animated title is part of the movie, keep it and watch to an other stack.

My english is pure crap :)

[Floatingshed]

This is all a little above my head, but thanks. I will try to come to terms with it.
Your English is fine, a great deal better than my French (assuming you are French!).
Thanks for your help. However I have no idea how to apply it practically in a script.

[johnmeyer]

I modified the script that I linked to above. The first modification was to use the excellent detection logic in TDecimate, which is made available via CFrameDiff. I generated a detection function using the metrics from this function.

I then used TDecimate to go from the frame-doubled 48 fps down to the desired 25 fps.

The script is shown below, and the output file is available here:

Smoothed 25 fps result (http://dl.dropbox.com/u/1561578/Smoothed.avi)

A few notes:

1. There is no scene detection logic, so you get blends at scene changes.

2. I didn't use 48000/1001 for the MFlowFPS. I just didn't have time to figure out whether this was going to give me "fractional" frames. The whole 23.976 -> 25 fps conversion, therefore, is probably not exactly correct.

3. The TDecimate line is what makes this script slow.

4. You will want to set "showdot" to false. I left it set to true so you can see which frames have been synthesized.

The script seems to get 95% of the drops.

# Based on script created by Didée
# Modified by John Meyer on August 16, 2011
#
# Create interpolated frames a 2x original frame rate using MVTools2
# Detect jumps
# Create white mask at each jump point; black mask for all other frames
# Use mask to "choose" between original video, or motion estimated video
# Decimate to go from 48 to 25 fps.

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

#Threshold for detecting jumps.
JumpThresh = 1.5
showdot = true # true for troubleshooting; otherwise, false

global source=AVISource( "E:\frameserver.avi" ).ConvertToYV12.killaudio().assumefps(24)

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

super = showdot ? source.subtitle("***").MSuper(pel=2) : source.MSuper(pel=2)
bvec = MAnalyse(super, overlap=4, isb = true, search=4, dct=5)
fvec = MAnalyse(super, overlap=4, isb = false, search=4, dct=5)
double = source.MFlowFps(super, bvec, fvec, num=48, den=1, blend=false)

#Remove comment from ShowMetrics, and change "return final" to "return test" to look at metrics in order to determine proper JumpThresh
#test=source.ScriptClip("Subtitle(String( Pow( CFrameDiff(denoise=true),2) /
# \ CFrameDiff(selectevery(1,-1),denoise=true) / CFrameDiff(selectevery(1,1),denoise=true) ))")


#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask2(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate
RequestLinear
final=tdecimate(display=false,mode=1,cycleR=23,cycle=48) # Decimate from 48 to 25 fps

return final.assumefps(25).ConvertToRGB()
#return stackvertical(source,final)
#return test


function GenerateMask2 (clip c)
{
MyMask=c.ScriptClip("""
\(Pow( CFrameDiff(denoise=true),2) / CFrameDiff(selectevery(1,-1),denoise=true) /
\ CFrameDiff(selectevery(1,1),denoise=true) ) > JumpThresh
\ ? WhiteFrame : BlackFrame """)
return MyMask
}

[Gavino]

JumpThresh = 1.5
global source= ...
global BlackFrame = ...
global WhiteFrame = ...
...
function GenerateMask2 (clip c)
{
MyMask=c.ScriptClip("""
\(Pow( CFrameDiff(denoise=true),2) / CFrameDiff(selectevery(1,-1),denoise=true) /
\ CFrameDiff(selectevery(1,1),denoise=true) ) > JumpThresh
\ ? WhiteFrame : BlackFrame """)
return MyMask
}
JumpThresh needs to be global.
Conversely, 'source' does not.
(Also, the first '\' is unnecessary and looks distinctly odd - a continuation of an empty line.)

[Didée]

Another technical nitpick: The Black|White frame creation really should be

global BlackFrame = BlankClip( source, Color_yuv=$000000 )
global WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF )

to get true-black and true-white for the mask.
(Else, you get "TV levels" black (Y=16) and white (Y=235), and all the merging would be only fractional: the not-affected dups get ~6% of the interpolated frames merged in, and the actually used interpolations keep ~8% of the underlying dup.)

Also when trying to run the script, after a short time, RequestLinear is bailing out for me with "internal error: frame not cached". Tried it with clim=100, and that seemed to work without err'ing out. (clim usually should be a bit larger than 2 times tdecimate's "cycle")

[johnmeyer]

Thanks for both comments. I'll try them out in a day when I get back to my computer.

[Floatingshed]

@JohnMeyer Many thanks for your hard work. This seems to work very well. I've only tried it on a short clip so far but it certainly looks promising.

[johnmeyer]

Glad it is working for you. If you have any problems, try incorporating Gavino's and Didée's suggestions. I would definitely recommend you make the color change (Color_yuv) that Didée recommended.

[Gavino]

Glad it is working for you. If you have any problems, try incorporating Gavino's and Didée's suggestions. I would definitely recommend you make the color change (Color_yuv) that Didée recommended.
Actually, I was wrong to say JumpThresh needs to be global - it works as is, since JumpThresh is declared at outer script level, which is where the run-time script is executed.

So in fact, BlackFrame and WhiteFrame needn't be global either.

[SSH4]

after some crashes on hard script on Avisinth MT with BlankClip on some last lines but not crashed with mt_lut i finally migrate to mt_lut. and mt_lut probably faster a bit on yv12...

BlackFrame = mt_lut( source, y=0, u=1,v=1 )
WhiteFrame = mt_lut( source, y=-255, u=1,v=1 )

[johnmeyer]

Here's the script with most of the changes suggested by Didée and Gavino. I did try to improve the logic that detects the jumps, but while each "improvement" helped the script detect some gaps it was missing, at the same time each improvement caused the script to now miss other jumps that it previously detected. So, in the end, I kept the same code I previously created.

I still like using CFrameDiff better than YDifferenceFromPrevious (as I did in a similar script I posted a few months back) or using MaskTools as Didée did in the script that was the inspiration for what I posted here. None of them come close to doing a "perfect" job, but CFrameDiff was developed by someone who spent a lot of time (a lot more than I've spent) trying to determine whether two frames are similar. Therefore, I have more "faith" in its ability to detect similarities (or in this case, differences) than logic I might develop.

If someone ever wanted to do this "correctly," I think that the motion detection logic would have to be modified so that it works differently for different "types" of motion. That word "types" would have to be defined, of course, and I am not entirely sure what those types would be. I do know that on this small test clip, the detection logic works perfectly when the camera is panning across a static scene, but has trouble when the camera is fixed and "objects" are moving in random directions across the scene. It also has problems with scenes with almost no motion (but of course if there is truly no motion, then would a missing frame even matter, other than sound sync?).

# Based on script created by Didée
# Modified by John Meyer on August 19, 2011
#
# Create interpolated frames at 2x original frame rate using MVTools2.
# Detect jumps using TDecimate's internal logic.
# Create white mask at each jump point; black mask for all other frames.
# Use mask to "choose" between original video, or motion estimated video.
# Decimate to go from 48 to 25 fps. This gets from wrong 23.976 frame rate to original 25 fps.
# Decimate logic gets rid of most similar frames in each group of 48 frames.
# This will first eliminate perfect dups created by 2x operation, but then
# will delete any other similar frames. What this means is that if you
# choose JumpThresh that is too low, this script will create interpolated
# frames at places where it doesn't need to. Then, some other location in the
# group of 48 that has relatively low motion will have a good frame decimated.
# This will introduce a new "jump," but hopefully one that isn't as noticeable.
# If JumpThresh is chosen correctly, and if GenerateMask logic perfectly detects
# only the places where a frame is really missing, the script will work perfectly.
#
# You can try using other missing frame algorithms by creating your own GenerateMask
# function. I have previously posted logic that uses YDifferenceFromPrevious instead
# of the CFrameDiff function from TIVTC. Didée has shown in the past how
# to detect large jumps using MaskTools. Take your pick: YMMV.

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

#Threshold for detecting jumps.
JumpThresh = 1.5
showdot = true # true for troubleshooting; otherwise, false

source=AVISource( "E:\frameserver.avi" ).ConvertToYV12.killaudio().assumefps(24)

BlackFrame = BlankClip( source, Color_yuv=$000000 )
WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF )

super = showdot ? source.subtitle("***").MSuper(pel=2) : source.MSuper(pel=2)
bvec = MAnalyse(super, overlap=4, isb = true, search=4, dct=5)
fvec = MAnalyse(super, overlap=4, isb = false, search=4, dct=5)
double = source.MFlowFps(super, bvec, fvec, num=48, den=1, blend=false)

#Remove comment from ShowMetrics, and change "return final" to "return test" to
#look at metrics in order to determine proper JumpThresh
#test=source.ScriptClip("Subtitle(String( Pow( CFrameDiff(denoise=true),2) /
# \ CFrameDiff(selectevery(1,-1),denoise=true) / CFrameDiff(selectevery(1,1),denoise=true) ))")

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)
interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate
RequestLinear(clim=100)
final=tdecimate(display=false,mode=1,cycleR=23,cycle=48) # Set "cycle" to 2x original frame rate.
# Set "cycleR" so correct framerate (25 fps) remains.

return final.assumefps(25).ConvertToRGB()
#return stackvertical(source,final)
#return test


function GenerateMask (clip c)
{
MyMask=c.ScriptClip("""
\ (Pow( CFrameDiff(denoise=true),2) / CFrameDiff(selectevery(1,-1),denoise=true) /
\ CFrameDiff(selectevery(1,1),denoise=true) ) > JumpThresh
\ ? WhiteFrame : BlackFrame """)
return MyMask
}

[Floatingshed]

Many thanks to everyone who has spent time on this for me.
Work has taken me away for the last few days so I haven't been able (frustratingly) to play with it very much.
Home tomorrow, then the fun begins!