I need some help with interpolation videos, lots of detail in post

[poisondeathray]

Originally Posted by bradwiggo

Did you cut the youtube video to the same size as the one I made? I will try cutting it and then go through frame by frame with the youtube one and my one on a different monitor.

Yes; as mentioned earlier, I aligned them temporally (using trim , so the start frame is the same) , and spatially by resized them. (so they are the same dimensions). This enables you to stack them or view in different tabs superimposed

Another way you can do this frame by frame on a single 720p monitor is split screen and in avspmod (e.g. left half YT, right half jm_fps) , but put the reverse in tab 2 (e.g. left half jm_fps, right half YT). So you could advance frame by frame with arrow keys and swap views by hitting number keys (1 for tab 1, 2 for tab 2)

It's important to just "watch" it normally for motion issues, but having them superimposed in different tabs and looking frame by frame reveals problems each version too. For example there are worse problems in YT version at the end zooming away from the tower. But overall, the motion "smoothness" is the same, and overall the motion artifacts are very similar; you can tell that some mvtools2 derivative was used for sure because of this

[bradwiggo]

Originally Posted by poisondeathray

Yes; as mentioned earlier, I aligned them temporally (using trim , so the start frame is the same) , and spatially by resized them. (so they are the same dimensions). This enables you to stack them or view in different tabs superimposed

Another way you can do this frame by frame on a single 720p monitor is split screen and in avspmod (e.g. left half YT, right half jm_fps) , but put the reverse in tab 2 (e.g. left half jm_fps, right half YT). So you could advance frame by frame with arrow keys and swap views by hitting number keys (1 for tab 1, 2 for tab 2)

It's important to just "watch" it normally for motion issues, but having them superimposed in different tabs and looking frame by frame reveals problems each version too. For example there are worse problems in YT version at the end zooming away from the tower. But overall, the motion "smoothness" is the same, and overall the motion artifacts are very similar; you can tell that some mvtools2 derivative was used for sure because of this

Something I have just noticed is the two clips in the compare.mp4 video you made are slightly different colours. I used xnView MP (my default image viewer) to look at a screenshot I took of the video. I got the RGB colour value of the same pixel on each video, and the top one was RGB(197, 167, 241, 255), and the bottom one was RGB(201, 169, 251, 255). Did you download the youtube video directly from youtube using a downloader, if so do you think this colour difference may be down to youtube's compression?

[poisondeathray]

Originally Posted by bradwiggo

Something I have just noticed is the two clips in the compare.mp4 video you made are slightly different colours. I used xnView MP (my default image viewer) to look at a screenshot I took of the video. I got the RGB colour value of the same pixel on each video, and the top one was RGB(197, 167, 241, 255), and the bottom one was RGB(201, 169, 251, 255). Did you download the youtube video directly from youtube using a downloader, if so do you think this colour difference may be down to youtube's compression?

Yes, a downloader.

It's probably compression differences. But you're not using exactly the same source as they did either, and there are multiple generation (rounding) differences

For example, even different studio release versions from the same blu-ray in the same year can have different colors.

[bradwiggo]

Originally Posted by poisondeathray

Yes, a downloader.

It's probably compression differences. But you're not using exactly the same source as they did either, and there are multiple generation (rounding) differences

For example, even different studio release versions from the same blu-ray in the same year can have different colors.

There is also a download link in the description I think. Does interpolation to 60fps cause problems as it was originally 24fps? In the interpolation script it said fps = 25, and I though that might have caused the original stuttering, so I later changed it to 24fps, but the output video was the same, does it automatically detect the framerate?

[poisondeathray]

Originally Posted by bradwiggo

There is also a download link in the description I think. Does interpolation to 60fps cause problems as it was originally 24fps? In the interpolation script it said fps = 25, and I though that might have caused the original stuttering, so I later changed it to 24fps, but the output video was the same, does it automatically detect the framerate?

For that script, you enter the desired framerate (59.94) . Other types of scripts have slightly different syntax. For example, SVPFlow uses a multiplier for numerator and denominator

FFVideoSource("test1.mkv")
jm_fps(59.94)

So if FFVideoSource is loading the video correctly at 23.976, jm_fps will interpolate that to 59.94

It interpolates from the fps avisynth "thinks" the source is. That is partially determined by the source filter

So if you have a source filter that returns an "off" frame rate, you could possibly get the wrong results

You can use info() to "see" what avisynth "thinks" the framerate is, with the video only . It will print out an overlay on top. Some source filters have problems with some sources or containers, and you might have to make adjustments . That might have caused some of the issues you were seeing earlier and in the videohelp thread . (But another difference for sure, is the scenechange blending vs. duplicates)

eg.

FFVideoSource("test1.mkv")
Info()

When you align clips, or compare them in different tabs, or stack them - it's very easy to see something is "off" right away . Or that they match.

[bradwiggo]

Originally Posted by poisondeathray

For that script, you enter the desired framerate (59.94)

FFVideoSource("test1.mkv")
jm_fps(59.94)

So if FFVideoSource is loading the video correctly at 23.976, jm_fps will interpolate that to 59.94

It interpolates from the fps avisynth "thinks" the source is. That is partially determined by the source filter

So if you have a source filter that returns an "off" frame rate, you could possibly get the wrong results

You can use info() to "see" what avisynth "thinks" the framerate is, with the video only . It will print out an overlay on top. Some source filters have problems with some sources or containers, and you might have to make adjustments . That might have caused some of the issues you were seeing earlier and in the videohelp thread . (But another difference for sure, is the scenechange blending vs. duplicates)

eg.

FFVideoSource("test1.mkv")
Info()

When you align clips, or compare them in different tabs, or stack them - it's very easy to see something is "off" right away . Or that they match.

Is that defining the framerate for this script:

# Motion Protected FPS converter script by johnmeyer from Doom9
# Slightly modified interface by manolito
# Requires MVTools V2 and RemoveGrain
# Also needs fftw3.dll in the System32 or SysWOW64 folder for Dct values other than 0


function jm_fps(clip source, float "fps", int "BlkSize", int "Dct")
{
fps = default(fps, 25.000)
fps_num = int(fps * 1000)
fps_den = 1000
BlkSize = default(BlkSize, 16)
Dct = default(Dct, 0)

prefiltered = RemoveGrain(source, 22)
super = MSuper(source, hpad = 16, vpad = 16, levels = 1, sharp = 1, rfilter = 4) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad = 16, vpad = 16, sharp = 1, rfilter = 4) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward = MAnalyse(superfilt, isb = false, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = true)#, ml = 200, mask = 2)

return out
}

Also, what does jm mean? Is it just a variable that we set to certain values?

[poisondeathray]

Originally Posted by bradwiggo

Is that defining the framerate for this script:

We probably posted at the same time; have a look at the post above it explains some things

The function is defined in the stuff below in that script. You have to call the function to use it in the script. jm_fps(something) . If I used jm_fps(120) it would interpolate to 120 fps

You could have "named" the function anything like jm_fps2 or brads_function() or whatever

Also, what does jm mean? Is it just a variable that we set to certain values?

"jm" is for our buddy John Meyer . He posted a few times in this thread. That manolito version is a variation on one he posted years ago (by you guessed it, manolito)

[bradwiggo]

Originally Posted by poisondeathray

We probably posted at the same time; have a look at the post above it explains some things

The function is defined in the stuff below in that script. You have to call the function to use it in the script. jm_fps(something) . If I used jm_fps(120) it would interpolate to 120 fps

You could have "named" the function anything like jm_fps2 or brads_function() or whatever





"jm" is for our buddy John Meyer . He posted a few times in this thread. That manolito version is a variation on one he posted years ago (by you guessed it, manolito)

# Motion Protected FPS converter script by johnmeyer from Doom9
# Slightly modified interface by manolito
# Requires MVTools V2 and RemoveGrain
# Also needs fftw3.dll in the System32 or SysWOW64 folder for Dct values other than 0
LoadPlugin("C:\Users\bradw\Downloads\masktools2-v2.2.17\x86\masktools2.dll")
LoadPlugin("C:\Users\bradw\Downloads\mvtools-v2.5.11.22\mvtools2.dll")
LoadPlugin("C:\Users\bradw\Downloads\RgTools-0.97\x86\RgTools.dll")
LoadPlugin("C:\Users\bradw\Downloads\GRunT101\GRunT.dll")
LoadPlugin("C:\Users\bradw\Downloads\MeGUI-2836-32\tools\avisynth_plugin\svpflow1.dll")
LoadPlugin("C:\Users\bradw\Downloads\MeGUI-2836-32\tools\avisynth_plugin\svpflow2.dll")
PluginPath = "C:\Users\bradw\Downloads\MeGUI-2836-32\tools\lsmash\"
LoadPlugin(PluginPath+"LSMASHSource.dll")

Function jm_fps(clip source, float "fps", int "BlkSize", int "Dct") {
    fps = default(fps, 23.976)
    fps_num = int(fps * 1000)
    fps_den = 1000
    BlkSize = default(BlkSize, 16)
    Dct = default(Dct, 0)
    prefiltered = RemoveGrain(source, 22)
    super = MSuper(source, hpad = 16, vpad = 16, levels = 1, sharp = 1, rfilter = 4) # one level is enough for MRecalculate
    superfilt = MSuper(prefiltered, hpad = 16, vpad = 16, sharp = 1, rfilter = 4)    # all levels for MAnalyse
    backward = MAnalyse(superfilt, isb = true, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
    forward = MAnalyse(superfilt, isb = false, blksize = BlkSize, overlap = 4, search = 3, dct = Dct)
    forward_re = MRecalculate(super, forward, blksize = 8, overlap = 2, thSAD = 100)
    backward_re = MRecalculate(super, backward, blksize = 8, overlap = 2, thSAD = 100)
    out = MFlowFps(source, super, backward_re, forward_re, num = fps_num, den = fps_den, blend = true)#, ml = 200, mask = 2)
    return out
}

DCT     = 0    # EDIT: 1 is SLOW
BLKSIZE = 16

c=LSMASHVideoSource("C:\Users\bradw\Documents\source.mkv")

Return jm_fps(c,fps=c.FrameRate*2,blkSize=BLKSIZE,dct=DCT)

That is my current script, is that right, as it is slightly different to the one you posted. You had the number in blue as 25, I set it to 23.976 as I believe that is the framerate of the source.

If I used jm_fps(120) it would interpolate to 120 fps

If you wanted to do that would you change the text in red to that?

[poisondeathray]

Originally Posted by bradwiggo

<snip>

That is my current script, is that right, as it is slightly different to the one you posted. You had the number in blue as 25, I set it to 23.976 as I believe that is the framerate of the source.



If you wanted to do that would you change the text in red to that?

Not the way you have it set up in that script; the blue actually does nothing - it's just a default value. As explained in the post above, the source FPS is determined by what avisynth "thinks" it is from the source filter . Or you can manually override it with AssumeFPS(something) . You have to do that sometimes if the source filter isn't returning a proper value

The final fps is in the return line "fps=c.FrameRate*2"

"c" refers to the source video as loaded by LSmash in that script

So if LSMash returned the proper 23.976 fps for the source, it would be 23.976*2 or 47.952. If it "thought" the video was 30.0 fps, you would get 30.0*2 or 60.0fps



So if you wanted output 120.0 FPS for the script as you have it now, you would just change the last line to

Return jm_fps(c,fps=120, blkSize=BLKSIZE,dct=DCT)

But again , the interpolation also partially depends on source framerate (or what avisynth "thinks" the source framerate is). If it's not loading correctly , you can get off results

[bradwiggo]

Originally Posted by poisondeathray

Not the way you have it set up in that script; the blue actually does nothing - it's just a default value. As explained in the post above, the source FPS is determined by what avisynth "thinks" it is from the source filter . Or you can manually override it with AssumeFPS(something) . You have to do that sometimes if the source filter isn't returning a proper value

The final fps is in the return line "fps=c.FrameRate*2"

"c" refers to the source video as loaded by LSmash in that script

So if LSMash returned the proper 23.976 fps for the source, it would be 23.976*2 or 47.952. If it "thought" the video was 30.0 fps, you would get 30.0*2 or 60.0fps



So if you wanted output 120.0 FPS for the script as you have it now, you would just change the last line to

Return jm_fps(c,fps=120, blkSize=BLKSIZE,dct=DCT)

But again , the interpolation also partially depends on source framerate (or what avisynth "thinks" the source framerate is). If it's not loading correctly , you can get off results

So it should be interpolating my videos to (almost) 48 fps? I have checking using the Info() script, and it is running at 47.9520. The youtube video is running at 59.9401 fps (60000/1001), if I want to interpolate to that framerate do I just use:

Return jm_fps(c,fps=59.9401, blkSize=BLKSIZE,dct=DCT)

[manolito]

@ poisondeathray

That manolito version is a variation on one he posted years ago (by you guessed it, manolito)

Actually this is just a year and a half old. It started when I stumbled over this post:
https://forum.doom9.org/showthread.p...25#post1788725

This script did beat everything I had tried before, so I stuck with it and advertised it a little bit.

As mentioned earlier, only the very last line is changed (highlighted in red) ; to blend=true, and the masking disabled (rest of line commented out)

Are you sure that you are disabling masking by commenting out the "ml" and "mask" parameters? According to the documentation the defaults are 2 for "mask" and 100 for "ml", so commenting out these params will just change "ml" from 200 to 100.


Cheers
manolito

[johnmeyer]

Yes, apparently "jm" refers to the settings I posted in that thread manolito linked to. For the video that was the topic of that post, my script did indeed produce fewer artifacts than other scripts and as a result, in subsequent threads, it became somewhat of a benchmark. I was flattered but I can assure you that I did nothing more than spend a few hours "twiddling the knobs" in all of the MVTools2 functions (MAnalyze, MFlowFPS, etc.) until I got the best result for that video.

I've been doing this for a long time, having first used the old MotionPerfect to create slow motion more than fifteen years ago, and since having used just about every motion estimation program around. My conclusion from this experience? The technology simply can't handle some situations. I listed a few of these earlier in this thread.

The only solution is to look for ways to detect when the ME is failing, and use some other method for that frame, or via masking, for part of that frame. MysteryX spent a huge amount of time tackling this problem and made some progress, but IMHO (through no fault of his) his script still doesn't yet work in 99% of all scenes, or anything close to that.

I've already stated why this particular animated video is going to be particularly difficult to smooth out without making it look worse. My mantra when doing video restoration is similar to the doctor's mantra: "first of all, do no harm." If the cure is worse than the disease, then you should just forget about it. Which, of course, has been my advice on this one from the beginning.

[bradwiggo]

Originally Posted by johnmeyer

Yes, apparently "jm" refers to the settings I posted in that thread manolito linked to. For the video that was the topic of that post, my script did indeed produce fewer artifacts than other scripts and as a result, in subsequent threads, it became somewhat of a benchmark. I was flattered but I can assure you that I did nothing more than spend a few hours "twiddling the knobs" in all of the MVTools2 functions (MAnalyze, MFlowFPS, etc.) until I got the best result for that video.

I've been doing this for a long time, having first used the old MotionPerfect to create slow motion more than fifteen years ago, and since having used just about every motion estimation program around. The technology simply can't handle some situations (I already listed a few of these earlier in this thread). The only solution is to look for ways to detect when the ME is failing, and use some other method for that frame, or via masking, for part of that frame. MysteryX spent a huge amount of time tackling this problem and made some progress, but IMHO (through no fault of his) still didn't quite create something that will work in 99% of all scenes.

I've already stated why this particular animated video is going to be particularly difficult to smooth out without making it look worse. My mantra when doing video restoration is similar to the doctor's mantra: "first of all, do no harm." If the cure is worse than the disease, then you should just forget about it. Which, of course, has been my advice on this one from the beginning.

I don't think it looks that bad, the youtube video looks quite good, not too many artifacts. As I said in one of my other comments, I am not necessarily trying or expecting to get the whole film to look as good as that video, I am just trying to reproduce that video.

[poisondeathray]

Originally Posted by bradwiggo

So it should be interpolating my videos to (almost) 48 fps? I have checking using the Info() script, and it is running at 47.9520. The youtube video is running at 59.9401 fps (60000/1001), if I want to interpolate to that framerate do I just use:

Return jm_fps(c,fps=59.9401, blkSize=BLKSIZE,dct=DCT)

I think it was suggested to use 2x earlier just to examine the effect of frame doubling (every second frame synthesized). Yes you would use 59.94 (the exact rate should be 60000/1001, but it's not going to make a difference here; technically 59.94 or 59.9401 are approximations. 60000/1001 is exact)

Originally Posted by manolito

Are you sure that you are disabling masking by commenting out the "ml" and "mask" parameters? According to the documentation the defaults are 2 for "mask" and 100 for "ml", so commenting out these params will just change "ml" from 200 to 100.

You're right - it's just the default settings for mask, not disabled

[bradwiggo]

Originally Posted by poisondeathray

I think it was suggested to use 2x earlier just to examine the effect of frame doubling (every second frame synthesized). Yes you would use 59.94 (the exact rate should be 60000/1001, but it's not going to make a difference here; technically 59.94 or 59.9401 are approximations. 60000/1001 is exact)




You're right - it's just the default settings for mask, not disabled

Do I need to disable mask?

If I wanted to reproduce the youtube video, should I use the fps = 59.94.

Also, can I write is as 60000/1001 in order to be as precise as possible?

[poisondeathray]

I already said the difference between those framerates are not going to make a difference here. If you do the math, over the course of a 3 hour movie , the difference between 59.94 vs 59.9401 vs 60000/1001 is not even 1 frame. So you will end up getting the same results either way

There are only minor differences if you use that script. Some of the difference might be from multi generation compression artifacts too. The only major difference is blend=true for the scene change. You can preview the results and adjust the settings, eg. blocksize etc... Some frames will be better some worse. You don't have to encode anything. Preview things before you waste time encoding the whole thing. What I'm saying is you can answer these questions easily yourself . Just open it up in avspmod, preview, change settings, rinse , repeat

Or maybe you want to adjust the settings per scene.

Or if you want better results, then do some manual masking along with layers and different settings

[bradwiggo]

Originally Posted by poisondeathray

I already said the difference between those framerates are not going to make a difference here. If you do the math, over the course of a 3 hour movie , the difference between 59.94 vs 59.9401 vs 60000/1001 is not even 1 frame. So you will end up getting the same results either way

There are only minor differences if you use that script. Some of the difference might be from multi generation compression artifacts too. The only major difference is blend=true for the scene change. You can preview the results and adjust the settings, eg. blocksize etc... Some frames will be better some worse. You don't have to encode anything. Preview things before you waste time encoding the whole thing. What I'm saying is you can answer these questions easily yourself . Just open it up in avspmod, preview, change settings, rinse , repeat

Or maybe you want to adjust the settings per scene.

Or if you want better results, then do some manual masking along with layers and different settings

I had changed the CRF settings a while ago in meGUI, I has the crf at 6 and the preset set to slower. I noticed this today and changed it back to 9.5 and medium, and then interpolated a small (2 minute long) video. The interpolation approximately doubled the file size. I then started to interpolate a larger (1 hour) file, and megui was reporting that the output file would be almost 4 times the size of the original. Why is this the case, as I am using the exact same settings for each of those two?

Also, as seen as I am interpolating 24fps to 60fps, does that mean the file size of the output should be at least 2.5 times that of the input to stop quality loss, or does it not necessarily work like that?

[poisondeathray]

Originally Posted by bradwiggo

I had changed the CRF settings a while ago in meGUI, I has the crf at 6 and the preset set to slower. I noticed this today and changed it back to 9.5 and medium, and then interpolated a small (2 minute long) video. The interpolation approximately doubled the file size. I then started to interpolate a larger (1 hour) file, and megui was reporting that the output file would be almost 4 times the size of the original. Why is this the case, as I am using the exact same settings for each of those two?

Also, as seen as I am interpolating 24fps to 60fps, does that mean the file size of the output should be at least 2.5 times that of the input to stop quality loss, or does it not necessarily work like that?

CRF doesn't necessarily work like that, and strictly speaking it's not a measure of "quality" ; it's just a rate control method

2.5x the number of frames generated by interpolation usually does not result in 2.5x the bitrate at a given CRF (everything else the same) ; because a) the relationship is not linear and b) the interpolated frames are usually more blurry and lower quality. When you do get much higher bitrate at a CRF value, it's usually indirect evidence that your have more interpolation artifacts than normal. Ugly edge morphing artifacts tend to "eat up" more bitrate, than a clean smooth interpolation. ie. The less differences between frames, the less bitrate at a given CRF value; roughly speaking - it's the differences between frames that are stored. The more differences, the more bitrate

[bradwiggo]

Originally Posted by poisondeathray

CRF doesn't necessarily work like that, and strictly speaking it's not a measure of "quality" ; it's just a rate control method

2.5x the number of frames generated by interpolation usually does not result in 2.5x the bitrate at a given CRF (everything else the same) ; because a) the relationship is not linear and b) the interpolated frames are usually more blurry and lower quality. When you do get much higher bitrate at a CRF value, it's usually indirect evidence that your have more interpolation artifacts than normal. Ugly edge morphing artifacts tend to "eat up" more bitrate, than a clean smooth interpolation

So my interpolated video will have loads of artefacts most likely? How should I fix this, as it has never done this before?

[poisondeathray]

Originally Posted by bradwiggo

So my interpolated video will have loads of artefacts most likely? How should I fix this, as it has never done this before?

It's just an educated guess. A clean interpolation will result in a much lower ratio. There can be other explanations too, maybe you messed up the settings, or it's not passing the correct commandline etc...

Take a look at the results first. Look frame by frame. It's the inbetween frames that have problems like edge morphing artifacts. A character might have 3 legs when walking , 3 arms when waving. (its the inbetween frame that isn't calculated cleanly). Because there are large differences, the bitrate will shoot up at a given CRF. But a clean interpolation where everything looks "normal" will have much lower bitrate

It's a matter of perspective or opinion; "is the glass half full or half empty" . For example you though the YT video was good. Many people here did not like the artifacts . Some people would rather have a duplicate frame or blend or other options instead of the ugly artifacts. It's a matter of opinion or what type of scenario you're working with

I think this was already discussed in some earlier posts - With current technology you can't fix these problems easily. They require a lot of manual work, masking, motion tracking, guiding motion estimation, compositing layers , patching areas. It's not like photoshoping each frame, but there is a steep learning curve on composting techniques tips and tricks. And it's tedious and takes a lot of time.

The MysteryX FrameRateConverter script is probably the closest thing that attempts to mask out problems or at least have options to address some of the issues automatically. Realistically it's far from perfect too.

There was a link referring to Nvidia AI - that looks very promising if you can believe demos