Controlled/selective interpolation

[ceth]

Hello,

I'm looking for a way to achieve "controlled interpolation". What I mean by "controlled" is the script would only interpolate pre-determined frames from an inputed list.

For exemple:

original sample has frames 1-2-3-4-5-6-7-8-9-10

inputed list:
4
8
9

outputed sample: 1-2-3-4-i1-5-6-7-8-i2-9-i3-10 where i1, i2, i3 are intermediate frames interpolated from original frames 4&5, 8&9 and 9&10 respectively, and the other frames are the original frames that remain untouched.

Do you see a way to do this ?


edit: from what I've seen ConditionalReader should help me in what I want to do. Gonna try with it.

[johnmeyer]

I'm pretty sure this has been discussed before, but I don't have a link to the previous thread.

One way to do it is to double the framerate by interpolating every other frame, and then use TDeciminate, which can accept a list of frames to decimate, to eliminate all the interpolated frames except at the places where you want to insert an interpolated frame.

[ceth]

Hi john,

Nice idea, though:
- I don't understand how to provide TDecimate an external list of frames to decimate :o
- the sample already has dups to decimate (uneven telecine on which I ran srestore, so now I have uneven dups), which will make complicated the creation of the list of frames to decimate if I go the double framerate + TDecimate route


It sounds to me it would be easier with ConditionalReader (+ some Gscript I guess) but I don't understand how to make it work.

For exemple, I'm trying this simple script to check an early version of my script:

aviSource("mysample.avi")
myvar = false
myvar ? grayscale() : last
ConditionalReader("manual_dup_list.txt", "myvar", true)

I was expecting the video to become greyscale every frame that is set to true in my manual_dup_list.txt file but it doesn't work. The manual_dup_list.txt is made correctly as I see the value going to true on the image (from ConditionalReader show=true), but it is not going greyscale.

What I am doing wrong ? Shall I use ScriptClip ? I've tried a bit with it but still can't get the simple above script to work.

[Gavino]

myvar ? grayscale() : last
This line has to be inside ScriptClip() so that the value of myvar is tested on every frame (at run-time) instead of just once (at compile-time).

ScriptClip("myvar ? grayscale() : last")

However, I'm not sure that your original problem can be easily solved using ConditionalReader/ScriptClip, since ScriptClip cannot change the number of frames in a clip.

[johnmeyer]

The "ovr" parameter in Tdecimate accepts the name of a file that contains the list of frames to decimate.

[ceth]

This line has to be inside ScriptClip() so that the value of myvar is tested on every frame (at run-time) instead of just once (at compile-time).

ScriptClip("myvar ? grayscale() : last")
Ah thanks, it works now with ScriptClip and removing the myvar = false declaration. :) Pleased I've learnt more about this :)

However, I'm not sure that your original problem can be easily solved using ConditionalReader/ScriptClip, since ScriptClip cannot change the number of frames in a clip.
Yes, I already have had a look at your conditional delete script here (http://forum.doom9.org/showthread.php?t=163107), but I cannot understand where does the source go in this script :confused: (I'm a bit sick today and have hard time to focus on anything). I guess this at-compile-time analysis could help in doing what I want to do.

I have about 95% of evenly spread dups (every 5 frames) and the rest more or less randomly spread. So the options I may have to explore are:

- delete 1 out of 5 frames (so the 95% of evenly spread dups) and replace the remaining dups with interpolated frames. But doing so means I will delete "good" (not dup) frames with the inital decimation. I'd end with a 23.976 video but I'm not sure the result will be perfectly smooth.
- delete only real dups from those that are evenly spread, then for the remaining dups I would continuously check whether it should be decimated or replaced by an interpolated frame in order to keep a constant framerate for audio sync purpose.
- replace all dups by interpolated frames.

I have made some bargraph to help visualize the problem I try to solve (click the thin white lines to open, then click again (twice) to zoom to fullscreen) :

frame 0 to 5k:
http://img11.hostingpics.net/thumbs/mini_3709280to5k.png (http://www.hostingpics.net/viewer.php?id=3709280to5k.png)

frame 5k to 10k:
http://img11.hostingpics.net/thumbs/mini_5953705kto10k.png (http://www.hostingpics.net/viewer.php?id=5953705kto10k.png)

frame 30k to 35k
http://img11.hostingpics.net/thumbs/mini_27830230kto35k.png (http://www.hostingpics.net/viewer.php?id=27830230kto35k.png)

Those bargraph show the distance in frames between each dup after the use of srestore.

The "ovr" parameter in Tdecimate accepts the name of a file that contains the list of frames to decimate.
Thank you. I had seen this ovr parameter but missed it was accepting a file as input. I'll try this too :)

[johnmeyer]

The problem you initially described was that you wanted to interpolate frames from a predetermined list. However, the description in your latest post gets more to the source of your probelm and says that you want to interpolate frames at locations where there are duplicates. This problem was "solved" by MugFunky many years ago. Search this forum for "filldrops". If your source is interlaced, search instead for the variation of that script which I produced called "filldropsI". My version of the script uses the later, better version of MVTools (MVTools2). It should work on progressive material as well.

Finally, if you have a duplicate frame problem, you may find that simply interpolating the duplicate frames doesn't entirely solve the problem because duplicates are often caused by the capture program dropping a frame, and then to keep from losing audio sync, it then duplicates a later frame. Unless the drop and duplicate frames are immediate adjacent, you end up with a video stream that has a fairly noticeable jump where the frame was dropped followed, several frames later, by the duplicate frames. The Filldrops script will beautifully handle fixing the duplicates, but you will be left with jumps. With the help of several people, I came up with a script that somewhat handles this problem:

Automatically fix dups followed (eventually) by drops (http://forum.doom9.org/showthread.php?p=1510813#post1510813)

I just used a variation of this script earlier this week, and found that it worked well, but that sometimes is left a few duplicates. So, what I did (and this worked really well) was to add a "filldropsI" statement at the very end to make sure that absolutely no drops were left.

One of the things I would love to do -- and maybe one of the really great programmers in this forum could make a suggestion -- would be to somehow extract motion vector information from MVTools2 and use that to determine the places where a big jump (missing frame) happens. The script I linked to above uses a moving average based on YDifference values to detect the jump and it is only about 95% effective.

[Gavino]

I already have had a look at your conditional delete script here (http://forum.doom9.org/showthread.php?t=163107), but I cannot understand where does the source go in this script :confused:
What that script does is simply declare a function called DeleteFrames().
To use it, you have to add your source filter and a call to the function.
So the whole script might look like this:
... # function declaration (from other thread)
AviSource("myexample.avi")
DeleteFrames("YDifferenceFromPrevious < 0.1 && current_frame > 0")
# just an example, condition to use depends on your requirements

[ceth]

The problem you initially described was that you wanted to interpolate frames from a predetermined list. However, the description in your latest post gets more to the source of your probelm and says that you want to interpolate frames at locations where there are duplicates. This problem was "solved" by MugFunky many years ago. Search this forum for "filldrops". If your source is interlaced, search instead for the variation of that script which I produced called "filldropsI". My version of the script uses the later, better version of MVTools (MVTools2). It should work on progressive material as well.
Yes I've already seen this topic about filldrops and your variation of it for interlaced content. However the problem here is not to replace all dups by interpolated frames (see below)

Finally, if you have a duplicate frame problem, you may find that simply interpolating the duplicate frames doesn't entirely solve the problem because duplicates are often caused by the capture program dropping a frame, and then to keep from losing audio sync, it then duplicates a later frame. Unless the drop and duplicate frames are immediate adjacent, you end up with a video stream that has a fairly noticeable jump where the frame was dropped followed, several frames later, by the duplicate frames. The Filldrops script will beautifully handle fixing the duplicates, but you will be left with jumps. With the help of several people, I came up with a script that somewhat handles this problem:

Automatically fix dups followed (eventually) by drops (http://forum.doom9.org/showthread.php?p=1510813#post1510813)

I just used a variation of this script earlier this week, and found that it worked well, but that sometimes is left a few duplicates. So, what I did (and this worked really well) was to add a "filldropsI" statement at the very end to make sure that absolutely no drops were left.
I'll try to make it clear about the whole problem I'm facing:

The original video is a 29.97 telecined movie (it has double ghost frames every 5 frames).
My goal is to make it 25fps (by going back to 23.976 then speedup to 25).

-My 1st step is to use srestore: srestore(omode="pp3")

Doing so I end with a video that has the same number of frames, same duration, same framerate. This is the step I am now. All the srestore pass has changed is that each set of 2 ghosted frames has been replaced by 2 rebuilt frames (that are identical). So I have now the same video but with dups instead of ghosted frames. The problem is, has mentionned previously, that those dups are not evenly spread.


From this observation, I see 2 possible causes:
1°) I did not use srestore properly (possible, as I'm noob to this)
2°) there is no problem with my use of srestore and the "problem" comes from the telecine that was achieved to create this video


By having a closer look at the distance between each consecutive dup sets (which is the same that was previously separating ghosted image sets, as sretore only replaced them with dups), you can see that when dups are not separated by the regular 5 frames, the addition of the distance between those consecutive dup sets is always (almost always ?) a multiple of 5.

For exemple, when a dup set is 3 frames after the previous set, the next set will almost always be 7 frames away.
It can even be spread over more frames. For exemple at frames 2000 of my 1st bargraph (frame 0 to 5k) you can see this distance distribution: 5 5 5 9 3 5 6 5 5 5 5 5 5 5 5 7 5 5 5 etc.

And 9+3+6+7 = 25, a multiple of 5.


Due to this I concluded the problem was not coming from my use of srestore but from a telecine that seemingly was adaptative: instead of having a constant 5 frames distance this distance can vary but seems always compensated to keep the average 5 frames distance (maybe Mug Funky will jump in to confirm this kind of hard telecine exists, as this is his work if I understood correctly).

The problems come when trying to revert this kind of telecine, which is what I'm trying to do.

As said, the vast majority of dup sets (= ghosted images sets, if we refer to the video I'm starting from) is evenly spread (more than 95% are separated by 5 frames). So if I just decimate 1 frame out of 5 the result will be almost good: 95% of the dups are eliminated, the framerate can be set to 23.976 (or 25), and there will be no audio sync issue. However, due to the small portion of dups not evenly spread it will not be perfect: those dups would still be there and I'd also have decimated good(?) (non-dup) frames.

My first thought was I could just delete all dups. Afterall, if the variable gap between dups (ghosts) is always quickly compensated to keep an average 5 frames gap, I should be able to keep the audio sync'ed when removing all the dups. But no, it didn't work. My audio was more and more desync'ed over the run of the video.

Maybe this is because of the way I get the dup list: I'm using the output file of the 1st pass of Dedup (DupMC), so maybe I'm getting "false positives" due to uniform color ranges here and there in the video (?), or if the video have real dups here and there (not sure if this is possible in a movie). It may help if srestore could output a log file giving the frames that were found to be ghosts but it doesn't seem to have this option (or I missed it). Also srestore has a threshold parameter that I haven't tried to play with.

One of the things I would love to do -- and maybe one of the really great programmers in this forum could make a suggestion -- would be to somehow extract motion vector information from MVTools2 and use that to determine the places where a big jump (missing frame) happens. The script I linked to above uses a moving average based on YDifference values to detect the jump and it is only about 95% effective.
I was recently also thinking about something similar to what you seem to describe here. Are you talking about a "motion repair" tool ?

For exemple let say we have a video with a dot moving from left to right, and for some reason the dot at frame x is not where it should be given its global motion on this video, and the script detect that frame so you can replace it by an interpolation between previous and next frame where the dot is at the right place. Is this what you are referring to ? (with more complex frames of course, I simplified for this exemple)

[ceth]

What that script does is simply declare a function called DeleteFrames().
To use it, you have to add your source filter and a call to the function.
So the whole script might look like this:
... # function declaration (from other thread)
AviSource("myexample.avi")
DeleteFrames("YDifferenceFromPrevious < 0.1 && current_frame > 0")
# just an example, condition to use depends on your requirements

I feel stupid. Indeed I missed the function line at the top as it was before the block of comments. Now I understand why I had this "{" missing error when I tried a raw copy of your script :D (as I just copied what was BELOW the comments... :p)

How would I proceed if I wanted to use a boolean value returned by conditionalreader as the condition for your DeleteFrames function ? I tried a bit but only got some 'unknown "myvar" ' errors.
Also if ScriptClip is required with conditionalreader, I guess things are different when evaluation is done at compile-time rather than run-time ? Or maybe is it just not possible to use conditionalreader in a at-compile-time environment as with your script ?

I have the feeling Gscript could simplify my trial and error tests. I'm currently using openoffice to build my frame number lists depending on what I want to test and that's a real pain if I want to test on a small range in the middle of the video, or if shifts occur due to a decimation pass. I easily get lost in what I'm doing.

[ceth]

I'll quote pbristow solution from the interframe thread:

First, a good way to avoid confusion is to number your frames 0, not from 1, as that's what avisynth assumes. For simplicity's sake, I'm going to assume/pretend that in your example there's a frame at the beginning of the input called frame 0, but which you don't want to include in the output.

The way to approach the task is to just generate extra in-between frames with InterFrame as usual, and then select the frames you want from that. Don't worry about time/CPU being wasted producing frames you don't want: Avisynth is cleverly designed to only generate frames that are actually required for the eventual output. So for example, if I say:


Interframe()
SelectEvery(999, 2, 4, 6, 8, 9, 10, 12, 14, 16, 17, 18, 19, 20)


Then avisynth looks at the SelectEvery function first, requests the frames it needs Interframe, and interframe either passes through an original frame or generates a new one as required. It never generates a frame you haven't asked for.

The result will be your "1-2-3-4-i1-5-6-7-8-i2-9-i3-10" sequence. All you need to do is double the frame numbers you want to keep from the original, and insert odd numbers in between for the ones you want interpolated by InterFrame.

SelectEvery() is a quick-n-dirty way of selecting a fixed sequence of frames if your source clip is short enough (up to a thousand frames, say): You just use a number bigger than the input clip length as the first argument, so that it doesn't start looping. There are lots of other ways to select frames, depending what you need: SelectRangeEvery (new in version 2.5); Decimate; and various things recently created by StainlessS.

Thanks pbristow. This is a similar solution to johnmeyer one, except you use selectevery to keep what must be kept instead of decimating what shouldn't be kept.
I have to say I find more transparent to proceed by determining what to keep rather than what to delete.

However, like you mentionned using SelectEvery only works for small clips due to the length limitation for the argument list. I was not able to use my full list without getting a "too long parameter list" error. The best I was able to do was processing about one or two thousands frames as you pointed.

I was wondering if I could bypass this limitation by using conditionalreader to provide SelectEvery the list of image to keep from an external file ?

[johnmeyer]

BTW, in the future, it would be good to post a short clip. You say your are trying to recover 24p film that has been to 29.97 interlaced. If it was telecined in some reasonably normal way, then SRestore may not be the correct tool and instead a straigthforward inverse telecine is all that is needed. You can then either motion interpolate up to 25 or just change the frame rate, without adding any frames, and adjust the audio to match. The latter is what is done for most 24p sound movies, but I'm not sure from your explanation whether you want to use MFLowFPS (or similar technique) to add frames that weren't there, or if you want to simply speed up the recovered 24 fps original frames so that they play at a 25 fps rate.

If this is normal telecined material, I'd just do something like this:

tfm()
tdecimate()
assumefps(25)I'd then use my sound editing program to change the audio speed to match.

[Gavino]

How would I proceed if I wanted to use a boolean value returned by conditionalreader as the condition for your DeleteFrames function ?
I think this should work (untested, and abusing the run-time environment even more):
aviSource("mysample.avi")
ConditionalReader("manual_dup_list.txt", "myvar")
DeleteFrames("AverageLuma()>=0 && myvar")
The "AverageLuma()>= 0" part is a hack to force a frame to be read from ConditionalReader in order to set myvar. It can be replaced by any run-time function expression that returns 'true'.

[pbristow]

I was wondering if I could bypass this limitation by using conditionalreader to provide SelectEvery the list of image to keep from an external file ?

The phrase "sledghammer to crack a walnut" leaps irresistably to mind... :)

If you *really* want to drive things from a pre-written file, then I recommend looking at StainlessS's FrameSelect filter, which is designed to do exactly that job:
http://forum.doom9.org/showthread.php?t=164497

*BUT* if the criteria for what frames should or shouldn't be interpolated are being determined from the input video, then that may not be the best approach. So the question is, what *are* your criteria for deciding which frames to interpolate? Once we know that, we can (hopefully) figure out the best way to go about selecting frames according to those criteria.

[ceth]

BTW, in the future, it would be good to post a short clip. You say your are trying to recover 24p film that has been to 29.97 interlaced. If it was telecined in some reasonably normal way, then SRestore may not be the correct tool and instead a straigthforward inverse telecine is all that is needed. You can then either motion interpolate up to 25 or just change the frame rate, without adding any frames, and adjust the audio to match. The latter is what is done for most 24p sound movies, but I'm not sure from your explanation whether you want to use MFLowFPS (or similar technique) to add frames that weren't there, or if you want to simply speed up the recovered 24 fps original frames so that they play at a 25 fps rate.

If this is normal telecined material, I'd just do something like this:

tfm()
tdecimate()
assumefps(25)I'd then use my sound editing program to change the audio speed to match.

I went the srestore path because that's the only thing I got working.
My 29.97 clip is not interlaced but progressive. Using TFM + TDecimate was one of the first things I tried but never got anywhere with it (maybe because I misused it, idk).
I first tried it with the clip as is (progressive). As it was not working I then tried again by adding SeparateFields to the script, but still fields were not matched.

I assumed it was telecined because that's a movie and because of the doubleblend every 5 frames. So I thought it was telecined then encoded progressive.
But maybe I'm wrong and I'm facing a convertFPS (or changeFPS ?). That would explain why only srestore worked (?)

I have also tried a bit with restoreFPS (with fps paramater at 24/1.001) but went nowhere with it so far. I can't make the phase detection function work, so I tried to manually play with the phase parameter, going from 0 to 1 by step of 0.1 but never got a good result. I also tried to trim up to 5 frames but it didn't help (with phase=0). Finally using a fps parameter of 25 (and phase=0) didn't produce a good result neither.

So far, only srestore managed to perfectly reverse the blends. But resulting in the uneven spread I'm mentionning since the beginning of this thread. I have looked a bit closer at those detected blends that are not evenly spread (= out of that "every 5 frames" scheme) and it **seems** to only (mainly?) be in ranges with very low motion. As soon as there is a decent motion it **seems** the every 5 frames scheme is respected. So maybe playing with srestore threshold parameter could get it good.

In the end I still can't say if this 29.97 comes from a telecine that was encoded progressive or from a fps change using blending. So any idea to try to figure out the right thing to do is welcome.

@Gavino your solution to combine your DeleteFrames function with ConditionalReader works :)

@pbristow my criteria are to remove the blends from the original and to obtain a smooth motion result with a constant framerate so I have no audio sync problem. Considering the point I am now with srestore (which may not be the right filter to use, or could be set for a better result), I can only do some trial and errors to check what gives the best result.

[johnmeyer]

Without a clip, I can't really know what's going on and therefore can't help any more. Sorry.