Aborting ModifyFrame or FrameEval

[_Al_]

Is there a way to abort ModifyFrame function and modify output clip length to when aborted?
This would be for a situation with unknown length of clips if fed from a sort of frame generator. Example below creates an attribute clip for ModifyFrame where its clip length is exaggerated when passed to ModifyFrame. Then if no more sources just black frames are added.

import vapoursynth as vs
from vapoursynth import core
import os 

DIRECTORY = r'D:\sources'

def get_clips():
    for filename in os.listdir(DIRECTORY):
        path = os.path.join(DIRECTORY, filename)
        clip = core.lsmas.LibavSMASHSource(path)
        yield clip

class Clip_handler:
    def __init__(self, get_clips):
        self.clip_generator = get_clips()
        self.load_new_clip()
        self.frame_num = -1
        self.total_length = 0

    def load_new_clip(self):
        try:                  self.clip = next(self.clip_generator)
        except StopIteration: self.clip = None
        self.frame_num = 0

    def fetch_frame(self, n, f):
        self.frame_num += 1
        if self.clip is not None and self.frame_num == self.clip.num_frames:
            self.load_new_clip()
            if self.clip is None:
                pass
                #need to abort here with total_length
        self.total_length += 1
        return self.clip.get_frame(self.frame_num) if self.clip is not None else  f.copy()

placeholder_clip = core.std.BlankClip(width=1920, height=1080, format=vs.YUV420P8, length=500000)
placeholder_clip = core.std.AssumeFPS(placeholder_clip, fpsnum=60000, fpsden=1001)
clip_handler = Clip_handler(get_clips)
joined_clips = core.std.ModifyFrame(clip=placeholder_clip, clips=placeholder_clip, selector=clip_handler.fetch_frame)
joined_clips.set_output()

So in that concrete example, is it possible to abort ModifyFrame when StopIteration happens and return correct clip length to joined_clips?
Or is there another way how to approach it? All is needed for an encoder is to feed it with frames in linear fashion. No seeking is needed.

[Myrsloik]

If you don't do temporal filtering you could set a frame property like "Stop=1" when you run out of source material. As the last filter before output use FrameEval() to throw an error if the property is set.

Something like that may work. You can't change the total number of frames after graph construction but you can at least make vspipe error out and abort if that's how you output things. Note that this also works if you use the vfw api directly where the error will be clearly reported.

[_Al_]

wow, that really works, thank you

class Stop_error(Exception):
    pass
.
.
.
    def fetch_frame(self, n, clip):
        self.frame_num += 1
        if self.clip is not None and self.frame_num == self.clip.num_frames:
            self.load_new_clip()
        if self.clip is None:
            if is_API4: clip = clip.std.SetFrameProps(stop=1)
            else:       clip = clip.std.SetFrameProp(prop='stop', intval=1)
            return clip
        return self.clip[self.frame_num]

def is_stop(n, clip):
    if 'stop' in clip.get_frame(n).props:
        raise Stop_error('No more sources')
    return clip

import functools    
placeholder_clip = core.std.BlankClip(width=WIDTH, height=HEIGHT, format=FORMAT, length= LENGTH)
placeholder_clip = core.std.AssumeFPS(placeholder_clip, fpsnum = FPSNUM, fpsden = FPSDEN)
clip_handler = Clip_handler(get_clips)
joined_clips = placeholder_clip.std.FrameEval(functools.partial(clip_handler.fetch_frame, clip=placeholder_clip))
try:
    joined_clips = joined_clips.std.FrameEval(functools.partial(is_stop, clip=joined_clips))
except Stop_error as e:
    pass
joined_clips.set_output()

when using vspipe cmd line to encode video, x264 will throw error "No more sources" and aborts encoding when at the end of sources, and prints stats as usually. That "*.264" video is playable. Mp4Box muxes that video and it seems to work.

[_Al_]

It works ok, so far, but while loading, many, many files, vspipe tends to throw occasionally an error and encoding aborts. If for example frames are just requested at the end of script it is ok, previewer even shows frames without error, but vspipe would end encoding.
Is it possible using vspipe command somehow ignore errors, like skip a frame or just silently skip the whole clip. Any woo-doo that would help.

F:\show\SLIDESHOW>"vspipe.exe" --y4m "F:\\show\\SLIDESHOW\\slideshow.py" -  | 
"F:\\prg\\vapoursynth R60 portable\\tools\\x264.exe" --demuxer y4m --crf 18 --vbv-maxrate 30000 --vbv-bufsize 30000 --keyint 60 --tune film
--colorprim bt709 --transfer bt709 --colormatrix bt709 --output "F:\\show\\SLIDESHOW\\output.264" -
y4m [info]: 640x360p 0:0 @ 25/1 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
x264 [info]: profile High, level 4.1
bad header magic (39303720 <=>  709)), 1799.00 kb/s
y4m [error]: bad header magic (20200a0d <=>
   )
x264 [info]: frame I:3607  Avg QP:15.70  size: 28521
x264 [info]: frame P:67305 Avg QP:19.25  size: 11812
x264 [info]: frame B:47027 Avg QP:20.78  size:  3458
x264 [info]: consecutive B-frames: 34.9% 32.1% 11.1% 21.9%
x264 [info]: mb I  I16..4: 12.8% 64.0% 23.3%
x264 [info]: mb P  I16..4:  1.8% 14.4%  3.4%  P16..4: 30.9% 19.2% 11.1%  0.0%  0.0%    skip:19.1%
x264 [info]: mb B  I16..4:  0.1%  0.8%  0.3%  B16..8: 29.3%  7.7%  2.4%  direct: 7.4%  skip:51.9%  L0:37.5% L1:44.7% BI:17.8%
x264 [info]: 8x8 transform intra:71.1% inter:65.1%
x264 [info]: coded y,uvDC,uvAC intra: 80.0% 88.3% 75.5% inter: 32.5% 41.2% 19.1%
x264 [info]: i16 v,h,dc,p: 28% 14%  6% 53%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 17% 18%  6%  8%  7%  9%  7%  9%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 19% 12%  6% 11%  9% 11%  6%  7%
x264 [info]: i8c dc,h,v,p: 55% 19% 16% 10%
x264 [info]: Weighted P-Frames: Y:13.3% UV:7.2%
x264 [info]: ref P L0: 62.0% 14.0% 15.9%  7.5%  0.6%
x264 [info]: ref B L0: 83.0% 15.1%  1.9%
x264 [info]: ref B L1: 96.2%  3.8%
x264 [info]: kb/s:1798.44

encoded 117939 frames, 63.69 fps, 1798.44 kb/s
Error: fwrite() call failed when writing frame: 117939, plane: 0, errno: 32
Output 117948 frames in 1852.37 seconds (63.67 fps)

Again, while previewing or requesting frames, it does not throw errors. Only when encoding via vspipe.

Or is there a way to load a clip, buffer all of the frames to ram, or just some chunk of frames, when done, output it for vspipe? If it fails buffering, rest of clip could be just skipped , ....and proceed to load other clip etc. But basically I do not know why it aborts, because just requesting or viewing frames is ok.