interpolate 18fps file to 24fps?

[BabaG]

i'm looking to preserve the 10bit depth of my input file as well as avoid blending scene-cut frames. i tried FrameRateConverter first but it seemed to turn the frames to mush, at least the way i used it, by interpolating every frame.

then i poked around and found that i could double every third frame with ChangeFPS. that worked but impacts the fluidity of motion.

then i found InterpolateDoubles, which replaced the doubled frames with blended or interpolated ones. that seems to work better than FrameRateConverter as it doesn't apply to every frame.

none of this preserves the depth of the original, though, and there are lots of instances where a scene cut is blended or interpolated.

here's what i have so far:

FFMPEGSource2("input18fpsProResHQ.mov", atrack=1)
z_ConvertFormat(pixel_type="YUV420P8")
threads=9
SetMemoryMax(24576)
ChangeFPS(24, 1)                            # Adds duplicate frames
InterpolateDoubles()
Prefetch(9)

i've used the z_Convert line in other scripts with P16 but had to change it to P8 here or the script would not load into Virtualdub.

is there a way around any of this?

thanks,
babag

[DTL]

You can set bitdepth to 16 for processing and convert back to 8bit at the end if monitoring with VirtualDub is required.

May try simple ConvertBits(8) or ConvertToYV12()

After final tuning of the script delete end conversion to 8bit before feeding to 10bit encoder.

[Frank62]

ChangeFPS(24, 1) # Adds duplicate frames
InterpolateDoubles()

Your approach will produce sometimes slower speed, sometimes original speed, even if the stuttering double frames are interpolated.

[johnmeyer]

frank62's approach is what I use. Duplicate frames (or fields, if your result will be interlaced) is the only method which will produce consistent results. Any optical flow (e.g., MVTools2) approach will introduce a huge number of artifacts (I can provide examples) and will also completely alter the "feel" of the original (i.e., it will not longer feel like film). Frame blending will kill details and soften the image. If you duplicate frames, you will definitely get judder when the camera quickly pans horizontally, but that was true of the original. Even 24 fps "Hollywood" films have this artifact and a surprising number of big-screen epics were filmed by DPs who didn't follow the rules for horizontal pans and ended up with juddery/jerky pans.

I have used MVTools2 (and several other approaches, all of which are commented out in my master film restoration script), but only when I absolutely had to, such as this transfer of an early 1940s Flint Michigan parade. If you play all the way through you'll see artifact after artifact, e.g., morphing when people emerge in the background as the float goes by; flag poles that break and then re-form; and worst of all, the antlers on stuffed moose at the 1:38 mark ...

1940 Flint Michigan Parade

[wonkey_monkey]

who didn't follow the rules for horizontal pans and ended up with juddery/jerky pans.

What rule do you follow to eliminate juddery/jerky pans?

[kolak]

If you pan with camera slowly, then pan 2 x slower

[johnmeyer]

Originally Posted by wonkey_monkey

What rule do you follow to eliminate juddery/jerky pans?

Read this blog entry:

Cinematography Tip: Capture Smooth Panning Shots at 24fps

[Frank62]

Originally Posted by johnmeyer

frank62's approach is what I use. Duplicate frames (or fields, if your result will be interlaced) is the only method which will produce consistent results.

It was not my approach, I only quoted it. I am of different opinion, but not to what you said.

What you wrote is undoubtfully true. There is no 100% artefact-free, blend-free way to change framerates and keep speed, no way. AI ways sometimes seem the best, sometimes even FramerateConverter, sometimes Hardware solutions, but they are all far from being perfect - the more motion and the bigger the moving objects, the worse. Still.

So frame doubling - although it is jerky, and i don't like that at all - still seems to be the best compromise, also because it does no damage at all and is fully reversable. And is and had been widely used with really valuable material, also in former times. Yes.

But the thread starter used both:
1. Frame doubling from 18 to 24 fps.
2. InterpolateDoubles, which is done via FramerateConverter, which will use optical flow (or can it do also blending? I don't know).
That was the situation.
When you double SOME frames to get from 18 to 24, then you will get something like this:

ABCDEFGHIJKLMNOPQR
AABCDDEFGGHIJJKLMMNOPPQR

After this using InterpolateDoubles will replace the second As, Ds, Gs a. s. o. with interpolations of AB, DE, GH a. s. o.
The other frames will be left untouched. So the result will not feel as jerky as after only frame doubling, but still jerky, changing speed all the time, no matter if you smooth it a bit by InterpolateDoubles.

So IF you accept artefacts and not-so-perfect optical flow it would be better to double ALL frames to get to 36fps and then use FrameRateConverter to get to the final 24fps.

[johnmeyer]

Originally Posted by Frank62

So IF you accept artefacts and not-so-perfect optical flow it would be better to double ALL frames to get to 36fps and then use FrameRateConverter to get to the final 24fps.

It would be interesting to try this. I've never done it myself. I'd have to see the results before I could rank it to the other alternatives you described.

It would certainly be worth trying this approach.

[BabaG]

interesting. last time i looked there hadn't been much response to this. now all this!

@frank62
what is the command you would suggest to double the 18fps to 36fps via interpolation? i'd love to give it a try.

edit:
i guess, after thinking about it, i have a couple of more questions. first, would you do these steps as separate scripts, generating a 36fps file and then generating a 24fps file from it? or, would these processes all be folded into a single script? and, second, i already asked about the command to get to 36fps but what is the command to downconvert that to 24fps. i'll be playing with FrameRateConverter in the meantime.

and here's a new one, i just tried using FrameRateConverter to downconvert a test from 36 to 24fps and got an avisynth error that 24fps is too small a number and that i should use at least 24.0006fps. when i tried that i got an error that it was the wrong type. i tried again with a value of 25fps, which i don't want, and it's working. does this function not work at 24fps?

thanks all,
babag

[poisondeathray]

Originally Posted by BabaG

i'm looking to preserve the 10bit depth of my input file as well as avoid blending scene-cut frames. i tried FrameRateConverter first but it seemed to turn the frames to mush, at least the way i used it, by interpolating every frame.

By default it doubles the frame rate , so it interpolates "between every frame"

Assuming you're ok with possible interpolation artifacts , FrameRateConverterMIX (the mod by Dogway) supports higher bit depths, and you can replace it for your experiments with different methods to keep HBD
https://github.com/Dogway/Avisynth-S...verterMIX.avsi

Use NewNum=24, NewDen=1 for the smoothest conversion cadence-wise for a direct conversion. If you introduce duplicates before it will be less smooth

FrameRateConverterMix(24,1,output="flow")

Many of the interpolators can hit any arbitrary frame rate by using fraction time points between frames, like 0.33, 0.17 . it doesn't have to be doubles or perfectly @ 0.5 between 2 frames, or framerates by powers of 2. Even if an interpolator couldn't, you would go 18 *4 = 72 . Take every 3rd frame 72/3 =24. So sample is evenly spaced in time and motion is perfect cadence wise

Look at the advanced options like artifact masking, skipping, and blending - because the results are filtered by default. Some of the "mush" artifacts might have been due to interpolation, or perhaps not ideal mask or blending settings. If you want to disable everything, and want just pure interpolation use output="flow". Then examine, and add back or adjust back the other options like thresholds for masks. The beta versions and dogway's version can use RIFE instead of Mvtools2 for the interpolation engine. RIFE is generally better but much slower

Some types of scenes/content don't interpolate well, period. There are newer machine learning algorithms that score higher than RIFE, but they still fail on some types of content

[BabaG]

wow. thanks pdr. will definitely switch to the Dogway version when i get done with the current test. great info here.

i'm very interested in RIFE and have been looking into it but there's no way that i have the skills to set it up. at this point, any documentation i've seen is for people who already know what they're doing with this stuff.

edit:
while i am interested in smooth motion, i'm also interested in trying to preserve the crispness of the original image. so far, when i upconvert, that's when i get mush, albeit smoother mush. for my subject matter, so far, i'm pretty happy with the doubling the third frame and then interpolating it so i get a crisp-crisp-crisp-mush-crisp-crisp-crisp-mush-crisp-crisp-crisp-mush cadence. the type of motion, if any, in the images i'm doing this on is such that it seems to mask any hiccupping cadences. i will be trying the things you've suggested, though.

thanks again,
babag

[poisondeathray]

Originally Posted by BabaG

i'm very interested in RIFE and have been looking into it but there's no way that i have the skills to set it up. at this point, any documentation i've seen is for people who already know what they're doing with this stuff.

Dogway to the rescue again. Look at the .avsi for explanations. You can use RIFE=1 or 2 for FrameRateConverterMIX

## @ RIFE - Uses RIFE to double the framerate of the frame blending clip used to mask artifacts, reducing the need for frame blending.
## 0 disables RIFE and uses only frame blending. 1 runs RIFE once but you will still notice frame blending. 2 runs RIFE twice for better quality.
## You can lower MaskThr to see more of the RIFE clip.
## More about the RIFE implementation: https://forum.doom9.org/showthread.php?t=174793&page=25

Also, RIFEwrap is an easy-to-use wrapper function which does all the intermediate steps like RGB float conversion and back. It's found in Dogway's ResizersPack.avsi
https://github.com/Dogway/Avisynth-S...izersPack.avsi

eg.

RIFEWrap(FrameNum=24, FrameDen=1, model=9)

Models 9 and higher (RIFE v4) are faster and can hit arbitrary frame rates (like those 0.17 between frames mentioned above) . The older ones can do inbetween, so 2x, 4x, 8x, etc.... But the older ones can be better on some types of content

One thing about the default avs version RIFE is the built in scene change detection isn't great for the avs version, and actually off by default (it's "On" for the the 2 functions above). You often have to adjust the threshold. If you're going to use RIFE, you 're going to need the dependencies in this link
https://github.com/Asd-g/AviSynthPlus-RIFE

And if you have decently fast Nvidia card with lots of Tensor cores, vapoursynth versions of RIFE are significantly faster. The Vulkan/NCNN RIFE versions can be used by most modern GPUs

[BabaG]

calling it a day but decided to try FrameRateConverterMix before quitting. when i set the NewNum value to 24 i got the following error:

Avisynth open failure:
ConvertFPS: New frame rate too small. Must be greater than 24.0006 Increase or use 'zone='

it then references line 258 in the FrameRateConverterMIX.avsi file.

thanks,
babag

[Frank62]

Originally Posted by BabaG

what is the command you would suggest to double the 18fps to 36fps via interpolation? i'd love to give it a try.

Simply

v=interleave(v,v)

...
i guess, after thinking about it, i have a couple of more questions. first, would you do these steps as separate scripts, generating a 36fps file and then generating a 24fps file from it? or, would these processes all be folded into a single script?

The power of frameserver-scripting IS, that you do NOT need to write in-between-files.

...
and got an avisynth error that 24fps is too small a number and that i should use at least 24.0006fps.

Oops. I didn't know that limitation, I never used FramerateConverter. Others will know better.

Originally Posted by poisondeathray

If you introduce duplicates before it will be less smooth

I am not 100% sure about that, but I think you are right.

[poisondeathray]

Originally Posted by BabaG

calling it a day but decided to try FrameRateConverterMix before quitting. when i set the NewNum value to 24 i got the following error:

Avisynth open failure:
ConvertFPS: New frame rate too small. Must be greater than 24.0006 Increase or use 'zone='

it then references line 258 in the FrameRateConverterMIX.avsi file.

thanks,
babag

I don't get that error message from 18 to 24 by default, or using RIFE as the engine. Post your full script

[BabaG]

pdr,

do you get the error going from 36 or 72 to 24? i'm wondering if it's better at raising rather than lowering the frame rate. when i first started playing with this, i think i went from 18 to 24 also and did not get the error.

babag

[poisondeathray]

Originally Posted by BabaG

pdr,

do you get the error going from 36 or 72 to 24? i'm wondering if it's better at raising rather than lowering the frame rate. when i first started playing with this, i think i went from 18 to 24 also and did not get the error.

babag

For 36 or 72 (e.g using duplicates) , then FrameRateConverter(NewNum=24, NewDen=1) works for me without error messages. But the output is clearly less smooth

[BabaG]

hmm. ok. moving on.

some success with FrameRateConverter bumping up to 72fps with 'flow' option and stepping back down to 24fps with SelectEvery(3, 1).

wanted to try the Dogway recommendation of FrameRateConverterMIX to preserve HBD but getting error that:

There is no function named 'ex_blend'.

anybody know where this lives so i can add it?

thanks,
babag

[poisondeathray]

Originally Posted by BabaG

wanted to try the Dogway recommendation of FrameRateConverterMIX to preserve HBD but getting error that:

There is no function named 'ex_blend'.

In Dogway's GradePack.avsi

https://github.com/Dogway/Avisynth-S...GradePack.avsi

The dependencies are listed in FrameRateConverterMIX.avsi

## Dependencies: > FrameRateConverter.dll
##                 ExTools
##                 MVTools2
##                 MaskTools2
##                 RgTools
##                 GradePack
##                 RIFE              (for preset="RIFE")
##                    TransformsPack (for preset="RIFE")