SD transfer in really bad shape - attempting to restore progressive frames

[johnny_b]

This was recorded from broadcast, but looks like someone filmed it off a screen that was wobbling (check out the distortion in the background towards the end of the clip). Oddly I didn't see any duplicate frames when I used separatefields, although I could be doing it wrong, but since this movie is from 1948 it's undoubtedly originally 24fps and not 29.97fps. I've tried a few deinterlacers but still end up with combing. Also I think some frames have been dropped (look at the ice truck around 12 seconds in).

Anyway, hoping for some help to get this as close as possible back to its original frames, and maybe if needed interpolate the frames that are missing.

Thanks!

https://www.mediafire.com/file/9gjnvjhni1kga0p/sample.mkv/file

[poisondeathray]

It was "normal" telecine, but has been vertically resized in a progressive manner before pulldown was removed. This causes fields to be distorted, so it's unlikely you'll be able to get clean results with "standard" methods. There are some "reverse" resizers like debicubic, debilinear that might help a bit if you apply before IVTC, but I doubt results will be very clean

If the cadence is set all the way through (not something like broadcast or post edits), one way is to interpolate /optical flow to interpolate "over" the combed frames using selectevery and interleave

But interpolation can produce many types of artifacts, it's source dependent. Sometimes you can get decent results, other times they are terrible

Vapoursynth can produce cleaner results , if you use rife to peform the interpolation. MVtools tends to have more artifacts (but you might be able to tweak some settings to get slightly better results)

In this example, RIFE is much cleaner on interpolated frames. e.g. typical mvtools2 artifacts

mvtools2
https://i.postimg.cc/BbXRLhzj/avs-mvtools2-interp000338.png
rife
https://i.postimg.cc/RVJjvDf8/vpy-rife-interp000338.png

videos
https://www.mediafire.com/file/eicj45l0aqdl8sv/interpolation,mvtools2,rife.zip/file


avs

#source
selectevery(5,0,1,2,4)
src=last

src
sup = MSuper()
bv = MAnalyse(sup, isb=true, delta=2)
fv = MAnalyse(sup, isb=false, delta=2)
interpolated = MFlowInter(sup, bv, fv, time=50.0, ml=100).DuplicateFrame(0)
replace = interpolated.SelectEvery(4,2) # replacement frames

DeleteEvery(4,2)

InterleaveEvery(replace, 4,2)



vpy

import vapoursynth as vs
core = vs.core
from vsrife import RIFE as RF3

clip = #source

a = core.std.SelectEvery(clip, cycle=5, offsets=[0])
b = core.std.SelectEvery(clip, cycle=5, offsets=[1])
c = core.std.SelectEvery(clip, cycle=5, offsets=[2])#
d = core.std.SelectEvery(clip, cycle=5, offsets=[3])#
e = core.std.SelectEvery(clip, cycle=5, offsets=[4])

int = core.std.Interleave(clips=[b,e])
r24 = core.resize.Point(int, format=vs.RGBS, matrix_in_s="170m")
r24 = core.rife.RIFE(r24, model=1)
r = core.std.SelectEvery(r24, cycle=4, offsets=[1])
r = core.resize.Point(r, format=vs.YUV420P8, matrix_s="170m")

intf = core.std.Interleave(clips=[a,b,r,e])
intf.set_output()

[johnny_b]

This is great, thank you. Would like to try RIFE but running into issues - apologies for my lack of knowledge here. I usually use StaxRip to encode and have been trying to edit in vsedit and load the script in StaxRip with no luck. So far I can't get StaxRip to notice that I've installed vsrife so it keeps crashing when I load the script. Any suggestions? I'm open to encoding with something else if that's easier but although I know a bit of Python I'm completely out of my element with VapourSynth.


import os, sys
import vapoursynth as vs
core = vs.get_core()

core.std.LoadPlugin(r"C:\PATH\ffms2.dll", altsearchpath=True)
clip = core.ffms2.Source(r"V:\PATH.mkv", cachefile = r"V:\PATH_t00_temp\temp.ffindex")
from vsrife import RIFE as RF3

a = core.std.SelectEvery(clip, cycle=5, offsets=[0])
b = core.std.SelectEvery(clip, cycle=5, offsets=[1])
c = core.std.SelectEvery(clip, cycle=5, offsets=[2])#
d = core.std.SelectEvery(clip, cycle=5, offsets=[3])#
e = core.std.SelectEvery(clip, cycle=5, offsets=[4])

int = core.std.Interleave(clips=[b,e])
r24 = core.resize.Point(int, format=vs.RGBS, matrix_in_s="170m")
r24 = core.rife.RIFE(r24, model=1)
r = core.std.SelectEvery(r24, cycle=4, offsets=[1])
r = core.resize.Point(r, format=vs.YUV420P8, matrix_s="170m")

intf = core.std.Interleave(clips=[a,b,r,e])
intf.set_output()

[poisondeathray]

Sorry, that script uses the plugin rife that uses vulkan.

RIFE.dll

https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan

You need to download the model(s) that you are using as well in the release link. In this example it was 2.4 which is a good generic model


Ignore the "from vsrife import RIFE as RF3" for that, unless you want to use other models than 2.4 or 3.1

There are 2 vapoursynth RIFE implementations, and various models between them

The other version uses CUDA and is signifcantly faster if you have fairly quick recent Nvidia card. But you have to load older versions of vs-rife to access the older models. The newest version is limited to 4.0
https://github.com/HolyWu/vs-rife

I access the older models by importing and mixing/matching other versions . v2.3 and 2.4 are the "best" generic models for general use on live action content

If you wanted to use the CUDA version , the way I organize it is 2 folders, old and new
from vsrife import RIFE as RF3
from vsrife4 import RIFE as RF4

core.rife.RIFE refers to the vulkan version
But RF3 refers to the old vs-rife, and RF4 to the newest version that runs models 4.x . You just change the model in the parameter

Here is example of some calls for 2x interpolations. I use rxx to denote which model is being used

#vulkan
#r24 = core.rife.RIFE(clip , model=1)
#ranime = core.rife.RIFE(clip , model=2)

#cuda
r18 = RF3(clip, model_ver=1.8)
r23 = RF3(clip, model_ver=2.3)
r24 = RF3(clip, model_ver=2.4)
r31 = RF3(clip, model_ver=3.1)
r35 = RF3(clip, model_ver=3.5)
r38 = RF3(clip, model_ver=3.8)
r40 = RF4(clip)

[kedautinh12]

I hope rife will port to avisynth :D

[Boulder]

Apologies for borrowing the thread, but would the VS method apply also to this? Similar issues I think - only this time the material was shot with a PAL DV camera (16:9 apparently) and somehow edited and rendered in NTSC so that it was resized while it was still interlaced. The original shots are long lost so this final result is the only thing left.

https://drive.google.com/file/d/1U1sTrYMjwdc8lVKAMehBbLsmlVxEzPRl/view?usp=sharing

I've only managed to tune it by a small amount with this:
TDecimate(cycle=6).AssumeFPS(25)
Santiag(1,1,nns=4)
Then resizing to widescreen 1:1 like 1280x720.

[johnny_b]

Thanks again for this. I managed to get this working finally - ended up encoding via ffmpeg in Vapoursynth Editor 2. This works really well for the section I uploaded and fairly well for the whole movie, although there is still some minor combing later in the movie. Might call it 'good enough' unless you have some suggestions for eliminating the remaining combing.

[poisondeathray]

Apologies for borrowing the thread, but would the VS method apply also to this? Similar issues I think - only this time the material was shot with a PAL DV camera (16:9 apparently) and somehow edited and rendered in NTSC so that it was resized while it was still interlaced. The original shots are long lost so this final result is the only thing left.

https://drive.google.com/file/d/1U1sTrYMjwdc8lVKAMehBbLsmlVxEzPRl/view?usp=sharing

I've only managed to tune it by a small amount with this:
TDecimate(cycle=6).AssumeFPS(25)
Santiag(1,1,nns=4)
Then resizing to widescreen 1:1 like 1280x720.

Not with that VS method - because there are no "good" fields or frames to interpolate from. Every field has been vertically resized. In order to use synthesize/interpolation to replace "bad" frames, you need 2 good(clean) "goal posts" or "end points" to interpolate from.

In johnny_b's case, it started as standard 3:2 film pulldown - 3 progressive (clean), 2 combed - and was vertically resized in a progressive manner. So those 3 progressive clean frames are still "clean"; only the combed frames are the problem, so you have to reconstruct that missing frame from adjacent "good" frames




Thanks again for this. I managed to get this working finally - ended up encoding via ffmpeg in Vapoursynth Editor 2. This works really well for the section I uploaded and fairly well for the whole movie, although there is still some minor combing later in the movie. Might call it 'good enough' unless you have some suggestions for eliminating the remaining combing.

It depends what the "minor combing" is... If the "minor combing" is the same artifact (vertical resize, before pulldown removal) and appears later in the movie, then it has gone out of phase. You'd then be interpolating the wrong frames

If it's from some other decombing related filters first before interpolation, otherwise you will contaminate the interpolation

You might be able to write a detection script more specific to those combed frames to make it adaptive. That current version is just a set pattern

[Boulder]

Not with that VS method - because there are no "good" fields or frames to interpolate from. Every field has been vertically resized. In order to use synthesize/interpolation to replace "bad" frames, you need 2 good(clean) "goal posts" or "end points" to interpolate from.

In johnny_b's case, it started as standard 3:2 film pulldown - 3 progressive (clean), 2 combed - and was vertically resized in a progressive manner. So those 3 progressive clean frames are still "clean"; only the combed frames are the problem, so you have to reconstruct that missing frame from adjacent "good" frames

Ok, cheers :) Do you have any hints on what to try or just leave it as it is? I know that some severe blur would work on it to hide the artifacts more, but it's not a very nice solution.