RIFE / custom scene cuts list

[elcoyote]

Hello,

I'm trying to build a framerate conversion script in which I specify a list of frames corresponding to scene changes instead of relying on automatic scene detection.

Why do this? Because I want to be able to review/fine tune the cut detection in an NLE (Davinci in my case) to make sure the scene cuts list is perfect instead of guessing threshold values and processing the clip many times.

I built that script that:
- trims the source clip in segments
- process the segment
- re-splice the segments into an output clip.

When I try to run it, VSEdit hangs and crash. Same with vspipe.

Here's the script:
import os
import sys
import vapoursynth as vs

core = vs.core

scene_cut_frames = [0,241,347,372,445,529,2304]


source = r'/test_clip.mov'
clip = core.lsmas.LibavSMASHSource(source)
clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s="709")

segments = []

for i in range(len(scene_cut_frames) - 1):
start_frame = scene_cut_frames[i]
end_frame = scene_cut_frames[i + 1] - 1
segment = core.std.Trim(clip, start_frame, end_frame)
# Process each segment here, for example:
segment = core.rife.RIFE(segment, model=45, fps_num=30, fps_den=1)
segments.append(segment)

# Join segments using Splice
output_clip = core.std.Splice(segments)

# Resize and set output
output_clip = core.resize.Bicubic(output_clip, format=vs.YUV422P10, matrix_s="709")
output_clip.set_output()

Is there anything obvious I am doing wrong? If I substitute the RIFE function with something else (like a resize). It works fine.

Specs:
AMD VegaII on Mac Intel
Vapoursynth R65
VapourSynth-RIFE-ncnn-Vulkan (https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan) library

Thanks!

[_Al_]

Guess,
maybe rife cannot be executed at once many times, without previous deleting, garbage collecting, maybe putting it in a function would help.

But what about this, working with ranges instead, because naturally that filtering is within a range:

import vapoursynth as vs
core = vs.core

scene_cut_frames = [0, 241, 347, 372, 445, 529, 2304]
source = r'/test_clip.mov'
clip = core.lsmas.LibavSMASHSource(source)

ranges = zip(scene_cut_frames, scene_cut_frames[1:])
ranges = [range(*_range) for _range in ranges] + [range(scene_cut_frames[-1], clip.num_frames)]
# ranges = [range(0, 241), range(241, 347), ... ]

class Clip_segments:
def __init__(self, clip, ranges, model=45, fps_num=30, fps_den=1):
self.clip = clip
self.ranges = ranges
self.model = model
self.fps_num = fps_num
self.fps_den = fps_den
self.range = ranges[0]
self.segment = self.get_segment()

def get_segment(self):
return core.rife.RIFE(self.clip[self.range.start:self.range.stop],
model=self.model,
fps_num=self.fps_num,
fps_den=self.fps_den)

def process(self, n):
if n not in self.range:
for _range in self.ranges:
if n in _range:
self.range = _range
self.segment = self.get_segment()
break
return self.segment[n-self.range.start]


clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s="709")
segments = Clip_segments(clip, ranges, model=45, fps_num=30, fps_den=1)
output = clip.std.FrameEval(segments.process)
output_clip = core.resize.Bicubic(output, format=vs.YUV422P10, matrix_s="709")
output.set_output()

[_Al_]

Oh, I neglected basic fact that frame rate is changed, so that script above would not work.

This might be the case for an unknown attribute clip (exaggerated) length in FrameEval is fed by a clip generator and when there is no more clips an error is thrown.

[_Al_]

I came up with this, is it too much? I tried that in vspipe and it works, encoder throws that planned error, and ends encoding, stream is fine. Not testing RIFE though, don't know what it will do.
It is strictly linear, no seeking is possible, it will just return next frame in any way to the end.
import vapoursynth as vs
core = vs.core

scene_cut_frames = [0, 241, 347, 372, 445, 529, 2304]
source = r'/test_clip.mov'
clip = core.lsmas.LibavSMASHSource(source)
clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s="709")
ranges = zip(scene_cut_frames, scene_cut_frames[1:])
ranges = [range(*_range) for _range in ranges] + [range(scene_cut_frames[-1], clip.num_frames)]
# ranges = [range(0, 241), range(241, 347), ... ]

def segment_generator():
for _range in ranges:
segment = clip[_range.start:_range.stop]
segment = filters(segment)
yield segment


def filters(clip):
return core.rife.RIFE(clip, model=45, fps_num=30, fps_den=1)


class NoMoreSources(Exception):
error_instance = [False]

def __init__(self, err, total_frames):
super().__init__(err)
self.total_frames = total_frames
self.log_end()

def log_end(self):
# it only gets printed once (or logged), even if more frames is requested
if self.error_instance[0]==False:
print(f'No more sources, total frames: {self.total_frames}')
self.error_instance[0]=True

class Clip_handler:
def __init__(self, segment_generator):
self.segment_generator = segment_generator()
self.load_new_clip()
self.frame_num = -1
self.total_length = 0

def load_new_clip(self):
try:
self.clip = next(self.segment_generator)
except StopIteration:
self.clip = None
self.frame_num = -1

def fetch_frame(self, n):
self.frame_num += 1
if self.clip is not None and self.frame_num == self.clip.num_frames:
self.load_new_clip()
if self.clip is None:
raise NoMoreSources(f'No more sources', self.total_length)
self.total_length += 1
return self.clip[self.frame_num]


attr_clip = clip.std.BlankClip(length=1_000_000)
clip_handler = Clip_handler(segment_generator)
out = attr_clip.std.FrameEval(clip_handler.fetch_frame)

out = core.std.AssumeFPS(out, fpsnum=30, fpsden=1)
out = core.resize.Bicubic(out, format=vs.YUV422P10, matrix_s="709")
out.set_output()

[Selur]

Maybe another approach?
Instead of chunking the source, wouldn't it be better to modify the '_SceneChangeNext ' (and maybe '_SceneChangePrev') value for each frame?
_SceneChangeNext = 1 for the last frame in scene
_SceneChangePrev = 1 for the first frame in a scene
Reading http://www.vapoursynth.com/doc/functions/video/modifyframe.html, I expect:

sChangePrev = [0, 100, 200, 300] # just examples
sChangeNext = [99, 199, 299] # just examples
def set_sceneChange(n, f):
fout = f.copy()
if n in sChangePrev:
fout.props['_SceneChangePrev'] = True
fout.props['_SceneChangeNext'] = False
elif n in sChangeNext:
fout.props['_SceneChangePrev'] = False
fout.props['_SceneChangeNext'] = True
else:
fout.props['_SceneChangeNext'] = False
fout.props['_SceneChangePrev'] = False
return fout
clip = core.std.ModifyFrame(clip=clip, clips=clip, selector=set_sceneChange)
to do the job. (it should be fast and avoid the whole splitting and joining)


Cu Selur

[elcoyote]

I came up with this, is it too much? I tried that in vspipe and it works, encoder throws that planned error, and ends encoding, stream is fine. Not testing RIFE though, don't know what it will do.
It is strictly linear, no seeking is possible, it will just return next frame in any way to the end.

I just tested your code and it seems to work for a few frames and then crash. My guess is that using the RIFE function inside of a loop is creating problems. But it's an improvement over what I was doing. I will retain the idea for other instances where RIFE is not used.

Thanks!

[elcoyote]

Maybe another approach?
Instead of chunking the source, wouldn't it be better to modify the '_SceneChangeNext ' (and maybe '_SceneChangePrev') value for each frame?


Yeah, this is what I've been trying yesterday. I was missing the "f = f.copy()" function and getting errors, but your sample code made me see the missing step.

Does RIFE actually need the "_SceneChangePrev" flag? In RIFE's plugin source code here (https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan/blob/master/RIFE/plugin.cpp), line 101, it seems to only use the "_SceneChangeNext".

For the record, here's my working code:
import vapoursynth as vs

core = vs.core

scene_change_list = [10, 47, 240]

source = r'/test.mov'
clip = core.lsmas.LibavSMASHSource(source)

clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s="709")
clip = core.std.SetFrameProps(clip, _SceneChangeNext=0)

def modify_frame(n, f):
if n in scene_change_list:
f = f.copy()
f.props["_SceneChangeNext"] = 1
return f

clip = core.std.ModifyFrame(clip=clip, clips=clip, selector=modify_frame)
clip = core.rife.RIFE(clip, model=41, fps_num=30, fps_den=1, sc=True)
clip = core.text.FrameProps(clip)

clip.set_output()

Thanks for the help

[Selur]

Does RIFE actually need the "_SceneChangePrev"
I don't know which is why I wrote "(and maybe '_SceneChangePrev'')" ;)
If it works without it: No :D

Cu Selur