Need a plugin to delete frames dynamically in 1 pass

[flossy_cake]

Hello, is something like this possible?

Inside a ScriptClip I am writing a frame property called "DeleteMe" which I am setting to 1 to signal the frame should be deleted, and 0 otherwise.

ScriptClip(last,
\ """

if (something){
return last.propSet("DeleteMe", 1) }
else {
return last.propSet("DeleteMe", 0) }

\ """ , after_frame=true, local=false)


So eg. for frame 123 the property might be 0 (meaning don't delete) and for frame 456 it could be 1 (meaning I want to delete it). etc.

Is it possible to make a dll plugin which can be called after the ScriptClip to delete a frame and its audio whenever the "DeleteMe" prop is 1?

So inside the dll it would need to have some conditional that reads the frame prop, and if it's 1, then delete that frame and its audio (note: audio cannot be modified inside ScriptClips, that's why I'm having this problem actually).

Is this possible, or do I have to do some big convoluted multipass thing writing the properties to text file in pass 1 and forcing evaluation with RT_ForceProcess() or something like that which I am trying to avoid.

I definitely can't do multipass as it's for realtime usage.

I would much rather solve it inside a dll that basically does the same thing Trim() does but dynamically based on the frame prop.

Thank you

:thanks:

PS. StainlessS if you are reading this your inbox is full - I was going to ask you.

ping

[flossy_cake]

After a few hours with ChatGPT I got this. No idea if it's hallucinated rubbish cause I caught it giving me rubbish a few times with variables that never get used anywhere and a GetAudio() that doesn't actually skip ahead in the audio chunk when a duplicate frame is deleted.

This one is looking the most promising of the 3 candidates though:


#include <avisynth.h>
#include <cstdint>

class LazySyncSkip : public GenericVideoFilter {
private:

inline int64_t frame_to_sample(int n) const {
return (int64_t)((double)n * vi.audio_samples_per_second *
vi.fps_denominator / vi.fps_numerator + 0.5);
}

int FindNextValidFrame(int start, IScriptEnvironment* env) {
int src = start;

while (src < vi.num_frames) {
PVideoFrame f = child->GetFrame(src, env);

const AVSMap* props = env->getFramePropsRO(f);
int err = 0;
int64_t val = props ? env->propGetInt(props, "DeleteMe", 0, &err) : 0;

if (!err && val == 1) {
src++;
continue;
}

return src;
}

return vi.num_frames - 1; // fallback
}

public:

LazySyncSkip(PClip child, IScriptEnvironment* env)
: GenericVideoFilter(child)
{}

PVideoFrame __stdcall GetFrame(int n, IScriptEnvironment* env) override {

int src = FindNextValidFrame(n, env);

return child->GetFrame(src, env);
}

void __stdcall GetAudio(void* buf, int64_t start, int64_t count, IScriptEnvironment* env) override {
if (!vi.HasAudio() || count <= 0)
return;

uint8_t* out = (uint8_t*)buf;
int bps = vi.BytesPerAudioSample() * vi.AudioChannels();

int64_t end = start + count;

for (int64_t pos = start; pos < end; ) {

// Convert audio position → approximate frame index
int frame = (int)(pos * vi.fps_numerator /
(vi.audio_samples_per_second * vi.fps_denominator));

if (frame >= vi.num_frames)
frame = vi.num_frames - 1;

int src_frame = FindNextValidFrame(frame, env);

// Map back to audio time of that frame
int64_t src_audio = frame_to_sample(src_frame);

int64_t next_frame = src_frame + 1;
if (next_frame >= vi.num_frames)
next_frame = vi.num_frames - 1;

int64_t next_audio = frame_to_sample(next_frame);

int64_t chunk = next_audio - src_audio;
if (chunk <= 0)
chunk = 1;

if (pos + chunk > end)
chunk = end - pos;

child->GetAudio(out + (pos - start) * bps,
src_audio,
chunk,
env);

pos += chunk;
}
}

int __stdcall SetCacheHints(int cachehints, int frame_range) override {
if (cachehints == CACHE_GET_MTMODE)
return MT_NICE_FILTER;
return 0;
}
};

// Factory
AVSValue __cdecl Create_LazySyncSkip(AVSValue args, void*, IScriptEnvironment* env) {
return new LazySyncSkip(args[0].AsClip(), env);
}

extern "C" __declspec(dllexport)
const char* __stdcall AvisynthPluginInit3(IScriptEnvironment* env) {
env->AddFunction("LazySyncSkip", "c", Create_LazySyncSkip, 0);
return "LazySyncSkip (timeline-preserving DeleteMe skip)";
}



Would be great if an actual plugin developer like @StainlessS could look at it and advise if it's not rubbish before I get stuck into trying to build and test it tomorrow.

The use case is decimation of duplicates from 24p (you know the ones, that typically end up due to orphan fields in the source around scenechanges). This means the audio has to be decimated too to keep audio in sync. I tested chopping out a single 41ms frame of audio with Trim() and I can't really hear any distracting skip in the audio so it should be ok.

Obviously the best solution would be a 2 pass job with duplicate removal in the first pass and then TimeStretch() the audio to fit the new duration, but that's not an option for my use case scenario which is for realtime screening, so I need this dll to skip to the next frame for both video and audio when the dupe is detected in the frame prop.

[StainlessS]

You can post in forum, I dont much like getting PM unless totally necessary.

I'm about to run out of the door, but will look further later.

I think that exactly what you want is not really possible, on fileserve start, the output frame count is already fixed.
Gavino says such somewhere on site.
I'll try see if I can find thread when I return, and sober up,
there might have been a solution where end frames shortage replaced by BLACK frames (requiring trim off).

Suggest you take a read of Prune() thread, where there is some script to delete (or Keep) frames complying with some condition.

Also, what I will be looking for is (in D9 search),
Search "SelectRanges", Show results as Posts, User name "StainlessS".
Probable working answer in that lot somewhere.

EDIT: None of the possible solutions that I may have been aware of, use frame properties, they are not something that
I've ever tried using. (old dog, new tricks thingy)

EDIT: Prune():- https://forum.doom9.org/showthread.php?t=162446&highlight=selectranges

EDIT: A solution using Prune() can also avoid 'cracks' in audio where frame/frames removed.

EDIT: Somehting I just found.
The basic problem here is that ScriptClip cannot change the number of frames, so the last frame is repeated up to the length of the original clip. You can get round this by doing the processing outside ScriptClip, something like:
current_frame = 0
GScript("
while (current_frame < 80 && vid.AverageLuma() < 17) {
current_frame = current_frame + 1
}
")
vid.Trim(current_frame, 0)
This is a bit of a hack, since AverageLuma is meant to be used only inside run-time filters like ScriptClip - it works by setting current_frame directly, 'fooling' AverageLuma into working normally.

If you need to remove blank frames from the end as well as the beginning, add something similar working back from the end, or make the code into a suitable function that you apply also to vid.Reverse().

EDIT: "Need a plugin to delete frames dynamically in 1 pass "
Auto 2 pass is quite easy, create frames file in pass 1, edit in pass 2.
Pass1: pass1 script to write frames file, followed by RT_ForceProcess(), to force script execution, (or alternatively TWriteAVI::ForceProcessAVI()).
Pass2: edit = SelectRanges() or RejectRanges() scripts.

RT_ForceProcess(clip, bool "Video"=True,bool "Audio"=false,Bool "Debug"=True)
If Video (default true), Force read every frame of clip from 0 to FrameCount-1.
If Audio (default false), Force read every sample of clip from 0 to NumberOfSamples-1.
Useful where a clip outputs some kind of file for use in a second filter, this function would in such a case
forcibly read and therefore write the file, so that it may be available to other filters or to a second pass of the filter that
initially wrote the file.
Debug, default true. Write Progress to DebugView.
Returns only after completed force reading of video frames and/or audio samples.
Returns 0.

TWriteAVI::ForceProcessAVI

ForceProcessAVI(clip c,bool "Debug"=True)
Force Process clip c, ie read from first to last frame, for TWriteAVI writes the AVI file (Video + Audio) without having to play clip.
Debug, default True, sends some progress info to DebugView (google).


EDIT: Maybe also see FrameSel [Does NOT handle audio]:- https://forum.doom9.org/showthread.php?t=167971&highlight=framesel
EDIT: SelectRanges() and RejectRanges():- https://forum.doom9.org/showthread.php?t=167971&highlight=framesel

[wonkey_monkey]

After a few hours with ChatGPT I got this. No idea if it's hallucinated rubbish

It is, or at least it won't do what you want. It doesn't keep track of how many frames should already have been deleted up to the current frame.

As StainlessS says, a plugin has to confirm the number of output frames when it's instantiated, so either it'll have to keep the number of frames the same (and pad with black at the end), or you'll have to specify the number of deleted frames when you call it - because it's probably not practical to scan the entire clip first to make a map and count the number of deleted frames.

This is a drawback of frame properties - the video frame has to be generated before you can access them. Which is fair enough for dynamic properties that a filter might create, like per frame min/max pixel values, but for static ones (like "DeleteMe") it's a big time-waster.

For the same reason this wouldn't work very well as a random access filter.

One idea: write the DeleteMe property to a dummy blank clip, 1x1 Y8, that can be rapidly scanned by the plugin to remap the frame numbers of the real clip. Maybe even bypass frame properties altogether - return a white pixel frame as part of the dummy clip if it's to be deleted, or black if it's to be kept.

[johnmeyer]

I do have a few scripts that will delete arbitrary frames in two passes, although lots of people know how to do that, probably including yourself. However, if you don't know and it if would be useful, I'll be glad to post one of them.

[flossy_cake]

Thanks for the feedback, much appreciated.

ChatGPT was initially implementing a solution where all frames were preprocessed before playback to figure out which frames had the DeleteMe property, and then removing those frames, and the output clip ends up with a lower frame count. But this would take a very long time to preprocess every frame before playback. Outputting a different frame count in a dll does seem valid though, after all that's what TDecimate does (and TDecimate does it in realtime without preprocessing every frame either, but it knows in advance what the modified framecount will be due to its Cycle & CycleR values being specified by the user in advance).

In the 2nd and 3rd solutions by ChatGPT I asked it to avoid the preprocessing issue and its solution was to keep the same number of frames but just "lazily" skip ahead, i.e jump to the next frame when encountering a DeleteMe. But its solution does seem wrong like wonkey_monkey says - it's not keeping track of how many frames it's skipped ahead, so if frame 100 has a DeleteMe and got skipped over, frame 101 would get shown instead, then the next frame after that is 101 without a DeleteMe, and so frame 101 ends up getting shown twice. So some kind of remap array of which frames are cumulatively skipped over needs to be kept track of between function calls perhaps inside the FindNextValidFrame(). The even harder part seems to be processing the audio samples to keep it in sync.

My current thinking is that a realtime 1-pass solution might still be possible, but it would probably result in some strange side effects like even though the framecount is static, the mapping between source frames and output frames is dynamic and depends on how many DeleteMe's have been encountered thus far during playback. And it might need some kind of seeking detection to reset the mapping on seeks.

btw the DeleteMe property is the result of quite a bit of processing on the main clip inside the ScriptClip, so putting it in a 1x1 clip I don't think would speed anything up, at least not for a 1-pass solution.



EDIT: Somehting I just found.
The basic problem here is that ScriptClip cannot change the number of frames, so the last frame is repeated up to the length of the original clip. You can get round this by doing the processing outside ScriptClip, something like:
current_frame = 0
GScript("
while (current_frame < 80 && vid.AverageLuma() < 17) {
current_frame = current_frame + 1
}
")
vid.Trim(current_frame, 0)
This is a bit of a hack, since AverageLuma is meant to be used only inside run-time filters like ScriptClip - it works by setting current_frame directly, 'fooling' AverageLuma into working normally.



Before getting help from ChatGPT I was doing many experiments inside ScriptClip with Trim() and manipulating current_frame, but no matter what I did, the audio coming out of the ScriptClip is always the same as the clip being iterated on. So unless it trims the audio as well, it could never be a 1-pass solution .

However what Gavino wrote above seems like it might open up some interesting possibilities, will need to test some things out to see what is accessible from outside the ScriptClip, since Trim()ing outside a ScriptClip is the only way to Trim the audio as well. But since Trimming outside a ScriptClip reduces frame count, that seems to imply the whole clip must be processed in advance because at the end of the day Avisynth is CFR only.

[flossy_cake]

Suggest you take a read of Prune() thread, where there is some script to delete (or Keep) frames complying with some condition.

Prune is great but I would still need to preprocess every frame in advance to gather the info to tell Prune which frames to prune.

If I call Prune or Trim inside a ScriptClip I'm back to the issue of ScriptClip being hardcoded to output audio from the source clip being iterated on.

What I need is a runtime-compatible Prune or Trim, but I don't think it's possible without modifying Avisynth.dll to remove ScriptClip's hardcoded audio limitation.

[flossy_cake]

Seem to have a working prototype in Visual Studio, so I'm now convinced this is possible.

The basic logic goes like this:

Start the skipahead counter at 0

If DeleteMe==1 on frame number <current+skipahead>, increment skipahead by 1

Output frame: <current_frame+skipahead>

Reset skipahead to 0 on seek (if current_frame differs from previous_frame by +/- 1 second of frames)

Output audio: start of chunk + (skipahead * samples_per_frame)

Test pattern - half dupes

InFrame DeleteMe Skipahead OutFrame
0 0 0
1 1 1 2
2 2 4
3 1 3 6
4 4 8
5 1 5 10
6 5 11
7 1 5 12
8 5 13
9 1 5 14

Test pattern - third dupes

InFrame DeleteMe Skipahead OutFrame
0 1 1 1
1 1 2
2 2 4
3 1 2 5
4 3 7
5 3 8
6 1 4 10
7 4 11
8 4 12
9 1 4 13


Have tested this by automating a DeleteMe once per second with if (Fmod(float(current_frame), 24.0) == 0 ){ PropSet("DeleteMe", 1) } and the audio stays in sync, and the number of duplicate frames left over at the end is equal to the skipahead count. Reset on seek behaviour seems to be working too according to debug props, and seeking to less than 1 second before end = no dupes at end of clip, and vice versa.

Issues:

1. Cannot handle strings of consecutive dupes. Well it handles them without breaking anything or going out of sync, but it doesn't skip all of them due to only doing a single lookahead:

Test pattern - dupe strings

InFrame DeleteMe Skipahead OutFrame
0 0 0
1 1 1 2 (wrong)
2 1 1 3
3 1 4
4 2 6 (wrong)
5 1 3 8
6 1 4 10
7 1 4 11
8 4 12
9 1 4 13


So the DeleteMes need to be spaced at least 1 nondupe apart for it to get them all. Easily solvable I would say, with a while loop that looks ahead n frames until no more DeleteMe is found, then increment skipahead by n instead of 1.

2. On a sine wave audio like from ColorBars() there is an audible click when skipping over a dupe due to the skipping of audio samples. On real world content there is no audible click and it's almost imperceptible. If there is music with a sustained pitch note during the audio skip, then I can just hear it (kind of like a single 41ms buffer overrun in a DAW). Maybe it's possible to do some improvement there with fading the audio level in over the duration of the frame.

3. The skipahead is cumulative, so over time it could potentially end up looking quite far into the future at the value of DeleteMe, and this value is the result of many upstream filters. I'm not sure this is going to play nice with the frame cache. Also, if the DeleteMe is calculated based on several previous frames leading up to it (which in my case it is) then Preroll() will probably be needed on the ScriptClip that sets the DeleteMe.

Cautiously optimistic.

[wonkey_monkey]

Reset skipahead to 0 on seek (if current_frame differs from previous_frame by +/- 1 second of frames)

That doesn't sound right.

[flossy_cake]

That doesn't sound right.

Well, I guess it depends what a seek event is supposed to represent.

If a seek means "go to this absolute time code" then I would want to reset skipahead to 0 to make sure I'm seeking to that timecode, not timecode+skipahead frames. For example, suppose a large number of skipahead frames have accumulated, then I want to seek to 00:00:00. If skipahead is not reset to 0, then I would actually end up seeking a large number of frames after 00:00:00.

On the other hand, if seeking is relative rather than absolute (like, "take me back 5 seconds in time relative to the current outframe") I would want it to NOT reset skipahead so that my seeks are correct relative to the current visible outframe.

I need to do a whole day of testing tomorrow then I'll post the solution if it still plays nice with my other big multithreaded CPU intensive scripts

[StainlessS]

I know that you want a real time single pass script, but maybe below of use to some people where real time not a issue, see 2 pass script near the end.
{I dont know how to achieve your requirement, you are in uncharted territory, good luck :) }

From here:- https://forum.doom9.org/showthread.php?p=1980401#post1980401
OOoops, the PruneGenny.avsi thingy was probably not best flexible for you needs, is intended only for use with Prune as writes output clip index 0 too.

Maybe better,

# MakeframesGenny.avsi # General Purpose conditional frames file writer. # https://forum.doom9.org/showthread.php?p=1980401#post1980401

Function MakeframesKeepFrames(clip c, string fileName, string condition, bool "Fast") {
# Conditional KEEP frames Command File generator
# MUST call with correct colorspace for condition eg YDifferenceFromPrevious requires a Planar colorspace.
# May require additional condition for end frames for eg YDifferenceFromPrevious and YDifferenceToNext.
# Any-DifferenceFromPrevious,
# Gives 0 for frame zero and so you may want to add to condition " || current_frame==0" to include frame 0.
# Any-DifferenceToNext,
# Gives 0 for last frame and so you may want to add to condition " || current_frame==framecount-1 " to include last frame.

c=(Default(Fast,true))?c.AssumeFPS(250.0):c # Fast as we can if Fast = true (Default true)
WriteFileIf(c, fileName, condition, "current_frame", append=false)
}

Function MakeframesDeleteFrames(clip c, string fileName, string condition, bool "Fast") {
# Conditional DELETE frames Command File generator
# MUST call with correct colorspace for condition eg YDifferenceFromPrevious requires a Planar colorspace.
# Is EXACT opposite of MakeframesKeepFrames() for same conditional.
# NOTE, any additional end frame condition as used in FramesKeepFrames, will also be inverted
# eg " || current_frame==0", frame 0 would be deleted.

c=(Default(Fast,true))?c.AssumeFPS(250.0):c # Fast as we can if Fast = true (Default true)
condition = "!(" + condition + ")"
WriteFileIf(c, fileName, condition, "current_frame", append=false)
}

###########################
# Below :HINT ON USE:
/* ################
Avisource("In.avi").ConvertToYV12() # Planar Required for AverageLuma and YDifferenceFromPrevious
##### pick one of below [can replace 0.1 with whatever threshold you like]
#MakeframesKeepFrames(".\cmd.txt","AverageLuma>=0.1")
#MakeframesDeleteFrames(".\cmd.txt","AverageLuma>=0.1")
##### Or with additional conditional for end frame [YDifferenceFromPrevious ALWAYS 0.0 on frame 0, YDifferenceToNext ALWAYS 0.0 on last frame]
#MakeframesKeepFrames(".\cmd.txt","YDifferenceFromPrevious>=0.1 || current_frame==0") # Always keeps first frame 0 (else delete)
#MakeframesDeleteFrames(".\cmd.txt","YDifferenceFromPrevious>=0.1 || current_frame==0") # Always deletes first frame 0 (else keep)
#MakeframesKeepFrames(".\cmd.txt","YDifferenceToNext>=0.1 || current_frame==framecount-1") # Always keeps last frame (else delete)
#MakeframesDeleteFrames(".\cmd.txt","YDifferenceToNext>=0.1 || current_frame==framecount-1") # Always deletes last frame (else keep)
Return last

##### And Pass2
Avisource("In.avi") #.ConvertToYV12() # Planar Required for AverageLuma and YDifferenceFromPrevious NOT REQUIRED HERE
Return FrameSel(cmd=".\cmd.txt")
################ */


Test

Colorbars.ConvertToYV12.KillAudio
F = Last.BlankClip(length=1)
A = Trim(0,-100) # 100 frames
A++F++A # frame 100 is black
SSS="""
Y=AverageLuma
RT_Subtitle("%d] y=%f",current_frame,Y)
"""
#ScriptClip(SSS) return last

RT_FileDelete(".\cmd.txt")
MakeframesKeepFrames(".\cmd.txt","AverageLuma < 17.0") # for above, writes Only Frame 100 (We are Writing the FrameNo we are going to Delete). [reverse Logic, fewer frames to write]

Return last


so can use for your case, eg

MakeFramesKeepFrames(Frames,"AverageLuma() < 17")

As the Pass1 MakeSomeKindOfFramesFile(Frames=Frames)

So can use simple frame file it produces with SelectRanges(), RejectRanges(), or FrameSel().

In your case we would use RejectRanges(cmd=Frames), to delete the frames in frames file.

So, 2 pass script

SourceSource("...")
Frames=".\Frames.txt"
RT_FileDelete(Frames) # kill existing

# Pass 1
MakeFramesKeepFrames(Frames,"AverageLuma() < 17") # write framenos we will delete

RT_ForceProcess() # Force creation of Frames file using above code, scans complete file returning only on completion where Frames file will exist.

# Pass 2
RejectRanges(cmd=Frames) # Reject frames in Frames File

Return last

[flossy_cake]

I did it!

Babby's first plugin
https://github.com/flossy83/SkipFrames

https://i.imgur.com/LsJw6nJ.png

Have done a fair bit of testing under various single/multithreaded light/heavy CPU load scenarios.

It has some logic to detect nonsequential access and err on the side of caution (wait for n sequential frames again before making further decisions).

Singlethreaded seems always fine, never had any issues with that in testing.

Multithreaded + RequestLinear() or Preroll(24) afterwards seems fine too, but RequestLinear() itself might not be bulletproof, not sure. See readme.md for clock test scenario and see if you can break it with heavy CPU load, multithreading and lots of random seeking around :devil:

Multithreading on its own with a high CPU load is where problems can potentially occur. Worst case scenario should be that it will simply not skip the frame if there was nonsequential access. I thought this is better than allowing illogical things to happen. It defaults to MT_SERIALIZED and that mustn't be changed.

Was this vibe coded? At first, yes. ChatGPT was telling me some nonsense so I switched to Grok and asked it for a basic skeleton code and went from there. Grok seems quite good, like for example asking it how to synchronise the audio to frame number, and how to set the multithreading mode and which mode is needed for a plugin of this nature. I presume there are probably better coding-specific AIs out there though.

[StainlessS]

Just a little info,
This,

# flag every 24th frame to be skipped
if (Fmod(float(current_frame+1), 24.0) == 0){
return c.PropSet("SkipMe", 1)
}
else { return c }

could be this [ uses % mod operator :- http://avisynth.nl/index.php/Operators ]

# flag every 24th frame to be skipped
if ((current_frame+1) % 24 == 0){
return c.PropSet("SkipMe", 1)
}
else { return c }

[flossy_cake]

Cool and good, I'll use that from now on cause it's so much neater

[StainlessS]

Additional, Fmod() is a function (slower), whereas % operator is (I think) done via the script parser, ie executed in-line.