Need help "perfecting" script to delete drops and dups

[johnmeyer]

Originally Posted by zorr

You need Pinterf's version of MVTools, v2.2.7 or later.

I'm definitely using his compile, but I'll have to wait until I'm back on the office computer to check the version number. I'm pretty sure I updated it to something fairly recent.

However, I still think the problem with the merge with white not producing an identical frame lies with the mt_merge plugin, not with MVTools2.

I've been tied up with other things today, but during driving time I thought about this script and am slowly coming to the conclusion that as good as Didée's idea is, it may actually be at the heart of the problem.

The issue is that you have to keep the video the same length, which his idea brilliantly handles. Put another way, for each drop you fill, you have to find a duplicate (or near duplicate) to delete.

Dups removed must equal new frames added at drop points.

I have really good detection logic (he said modestly) that finds both drops and dups. The trick of the whole thing is to have an approach which 100% guarantees that the number of drops filled exactly equals the number of dups removed. If the drops and dups are sparse, and if the Cycle size is large enough, this Didée'-inspired script is going to work most of the time, but if you have lots of drops and if the Cycle size is small, it will fail a lot.

So, I am contemplating rethinking the whole idea and coming up with something that eliminates TDecimate from the script. Unfortunately, this is likely to lead to a convoluted workflow, where I export my dup and drop metrics, dump them into a spreadsheet, and then use spreadsheet formulas and VBA to come up with a list of dups and drops which I'll feed back into an AVISynth script. Unfortunately, I don't know if there is any plugin which will take such a list because I don't think you can just arbitrarily add a few frames here or subtract a few frames somewhere else. That isn't how AVISynth works.

Anyway, I'm going to keep trying for a few more days. There needs to be a solution to this problem.

[StainlessS]

Originally Posted by poisondeathray

For different bit depths blackclip(souce).expr should be faster than source.mt_lut; both will "flexible" in terms of bitdepth and correct on inverting
expr("0", "x", "x") for copy u,v and expr("0", "range_half", "range_half") for half uv at source bitdepth

I'm kinda reluctant to use Expr() for anything in my scripts directory, If I need to test something under 2.60 std, then I gotta remove all of those scripts so as not to cause multiple problems.

[stephen22]

Originally Posted by johnmeyer

mt_merge does not provide identical video to the original when merged with pure white.

Never was quite sure why you need to use a mask at all. Why can't you just get your drop seeking routine to choose between Source and a (single) MFlowInter interpolation clip? Interleave with Source and do your TDecimate.

[StainlessS]

However, I still think the problem with the merge with white not producing an identical frame lies with the mt_merge plugin, not with MVTools2.

mt_merge does not provide identical video to the original when merged with pure white.

The problem is that it aint PURE White, rgb $FFFFFF arrives at Y = 235dec $EB hex, and RGB black $000000 at 16 dec, $10 hex,
so you dont get all of it when you want it all, and do get some of it when you want none.
Nothing wrong with mt_merge, problem is in script [as pointed out (I think) by several people in several threads].

[zorr]

Originally Posted by johnmeyer

However, I still think the problem with the merge with white not producing an identical frame lies with the mt_merge plugin, not with MVTools2.

Sorry, meant to say MaskTools, not MVTools. Pinterf makes too many plugins.

[johnmeyer]

Originally Posted by stephen22

Never was quite sure why you need to use a mask at all. Why can't you just get your drop seeking routine to choose between Source and a (single) MFlowInter interpolation clip? Interleave with Source and do your TDecimate.

I have had the same question a few times as I've been working on this. Didée's code from the "Inverse of Decimate" thread used a mask to choose the clip. If I'd coded this from scratch I would have used ConditionalFilter, or something similar. Perhaps I will make that change now because it would make it simpler to get to the bottom of my problems if the exact dups created by doubling the number of rames source video would cause TDecimate to return a dup metric of 0.00. This would make it simpler to do the right thing with the "almost" dups in the actual video that need to be decimated.

[StainlessS]

Originally Posted by zorr

Pinterf makes too many plugins.

Yep, he just wants to be popular, very needy, sad really.

[johnmeyer]

Originally Posted by StainlessS

The problem is that it aint PURE White, rgb $FFFFFF arrives at Y = 235dec $EB hex, and RGB black $000000 at 16 dec, $10 hex,
so you dont get all of it when you want it all, and do get some of it when you want none.
Nothing wrong with mt_merge, problem is in script [as pointed out (I think) by several people in several threads].

Yes, poinsondeathray and others have pointed this out, but I still haven't been able to get the proper level. I posted earlier my attempt, using Levels(), to get the right result.

[StainlessS]

Levels(16,1.0,235,0,255,coring=false)

should do it I think. [Must have the coring thingy]

[johnmeyer]

Originally Posted by poisondeathray

For the mask part, if you wanted to use levels, it would be this for 8bit YUV

BlackFrame = BlankClip( source, Color=$000000 ).levels(16,1,235,0,255,false)
WhiteFrame = BlankClip( source, Color=$FFFFFF ).levels(16,1,235,16,235,false)

As I speculated in my earlier post, I had it backwards. This fixes the problem. Thanks! (Thanks to StainlessS also).

[stephen22]

Originally Posted by johnmeyer

Perhaps I will make that change now

Doesn't it just mean replacing MFlowfps with MFlowInter, Blackframe with Source, WhiteFrame with the MFlowInter clip, juggling the interleave and forgetting the merge_mt?

It seems to me there's no way to get round the cycle difficulty, and that the problem is intrinsic in the method, not down to TDecimate, which is just doing what it's paid for. Whatever method you use for removing the 50% of frames with the least change, it will have to be regional, otherwise (apart from the time taken!) sequences with lots of movement will inevitably steal frames from sequences with very little movement. Regions have boundaries and dupe/drop pairs will inevitably straddle those boundaries at times.

You'd have to devise a very sophisticated algorithm which would seek to identify the dupe nearest to a drop and link the two in some way, marking the dupe so it's no longer available - not too hard in a normal programming environment but I suspect way outside the constraints imposed by Avisynth.

Simplest perhaps to just repeat the whole procedure as necessary - lengthy perhaps, but getting rid of the mask must save a bit of time. It's a very fine script as it is.

[johnmeyer]

Originally Posted by stephen22

Doesn't it just mean replacing MFlowfps with MFlowInter, Blackframe with Source, WhiteFrame with the MFlowInter clip, juggling the interleave and forgetting the merge_mt?

It seems to me there's no way to get round the cycle difficulty, and that the problem is intrinsic in the method, not down to TDecimate, which is just doing what it's paid for. Whatever method you use for removing the 50% of frames with the least change, it will have to be regional, otherwise (apart from the time taken!) sequences with lots of movement will inevitably steal frames from sequences with very little movement. Regions have boundaries and dupe/drop pairs will inevitably straddle those boundaries at times.

You'd have to devise a very sophisticated algorithm which would seek to identify the dupe nearest to a drop and link the two in some way, marking the dupe so it's no longer available - not too hard in a normal programming environment but I suspect way outside the constraints imposed by Avisynth.

Simplest perhaps to just repeat the whole procedure as necessary - lengthy perhaps, but getting rid of the mask must save a bit of time. It's a very fine script as it is.

You are MUCH smarter than I am because it took me a few years to realize what you just wrote. Your analysis is 100.0% correct, and very clearly written.

Which brings me to my current idea, but one which I can't quite figure out to do. Here it is.

My next approach will be to export the drop matching metrics from my detection function and also export dup matching metrics, either from TDecimate or from my own matching logic. I'll then put those metrics into an Excel spreadsheet. In that sheet I can use a combination of formulas and VBA (much easier than coding a plugin in C), to look backwards and forwards to match the nearest dup with its corresponding drop and use those matches to create a column which shows at which frames a motion-estimated frame must be inserted, and also which frames must be decimated because they are dups. The logic will ensure that dups=drops.

Then, in AVISynth, I'll create the intermediate frames by doubling the frame rate and then using SelectOdd() to choose only the intermediate frames. This will give me a stream from which I can insert frames to bridge the drop.

However, this is where I hit a dead end. I need to have some way in which I can insert frames arbitrarily and can decimate frames arbitrarily. I don't think AVISynth provides a mechanism for this even though the total number of frames will, in the end, be the same.

I can write a script in my NLE, Vegas Pro, that will take the original video along with the "in-between frames" stream (which is the same length as the original video, but shifted in time by half a frame) and perform the substitutions and decimation, but very few people reading this will be able to use such a thing because most people don't own Sony Vegas Pro.

I know that I can do arbitrary decimation using MultiDecimate because I do that all the time with my shutterless projector film transfer system that I invented fifteen years ago and which I have posted about several times in this forum. However, while it can easily do arbitrary decimation from a list of frame numbers, I don't think it permits me to arbitrarily insert frames.

So, bottom line, the approach I adopted from Didée years ago is fundamentally flawed, and I am now thinking that the way in which AVISynth works will make it impossible for me to do this task completely from within the AVISynth environment.

To do so will require me to

• Figure out how to make the number of dups removed exactly equal the number of drops inserted;
• Insert synthesized video frames at arbitrary locations in the video;
• Delete a "duplicate" frame within some set distance from each drop;

To make this work, the drops must be the "controlling" metric, and they must drive the dup decimation.

I need to build up the desire to keep going on this. My head hurts right now from banging it against the wall. I may re-visit the idea Zorr had in your Dropped frames thread where I run TDecimate three times with Cycle, 2*Cycle, and 3*Cycle and then "vote" by using Median() to make the decision on decimation based on which two provides the same result. However, as I think through that, if there are lots of drops and dups, you get overlap between the drops and the dups, and the voting may get confused.

[VoodooFX]

Originally Posted by johnmeyer

As I speculated in my earlier post, I had it backwards. This fixes the problem. Thanks! (Thanks to StainlessS also).

This should be faster than poisondeathray's example:

BlackFrame = BlankClip( source, Color_yuv=$000000, pixel_type="Y8" )
WhiteFrame = BlankClip( source, Color_yuv=$FFFFFF, pixel_type="Y8" )

[stephen22]

The writer of that script weren't no slouch.

I'm very inexperienced with Avisynth so be kind if I'm talking nonsense.

If you export your data (how do you do this?) and can externally generate a text file that has a code for each frame (normal/delete/insert_after) that can be read by ConditionalReader, might it be possible to build up the output file in one pass? You would have to have at least one variable available from one frame to the next (is this possible?) which kept tabs on how out of step you were when choosing a frame (increment after delete, decrement after insert) from the clip (normal or interpolated) specified by the imported code. Something like

Code
normal__________selectevery(source,1,stepcode)

delete__________increment stepcode; selectevery(source,1,stepcode)

insert_after______decrement stepcode; selectevery(interp,1,stepcode)

I've probably got the logic wrong but is the general idea viable?

[johnmeyer]

Originally Posted by stephen22

If you export your data (how do you do this?) and can externally generate a text file that has a code for each frame (normal/delete/insert_after) that can be read by ConditionalReader, might it be possible to build up the output file in one pass?

The key limitation of AVISynth is that it has to know, at every step in the script, the number of frames in the video file. You can use functions which, as an example, double the frames (like MFlowFPS), but I don't think there is a way to insert a frame here, and then a frame there, at arbitrary points.

[stephen22]

Don't follow. The number of frames in the output is the same as the input and doesn't change. There's no insertion or deletion: each output frame is specified according to the algorithm based on information from the text file which your clever spreadsheet program has compiled.

It might look something like

Outputfile=Scriptclip(source,"stepcode=stepcode+framecode"+ chr(13) + \
"framecode < 1 ? selectevery(source,1,stepcode) : selectevery(interp,1,stepcode)")

ConditionalReader("spreadsheet.txt","framecode",false)

The framecode would be -1 = insert_after, 0 = normal, 1 = delete (dupe). Stepcode (how many frames ahead the required frame will be) would have been initialised as 0.

Your spreadsheet text file would look something like

Type Int
Default 0
1 0
2 0
3 -1
4 0
5 1
etc

I don't know whether I've got the syntax right, or indeed whether this would be allowed. If you can't use a variable like this, the "stepcode" value could be computed by your spreadsheet program and buried in the codes in the text.

And of course your spreadsheet would have to provide the text file in the right format.....with the right values!

[VoodooFX]

Why you need Excel for dupe detection?

[johnmeyer]

Originally Posted by VoodooFX

Why you need Excel for dupe detection?

I don't need Excel for detection; AVISynth YDifference can do that quite well. What I need it for is to solve the problem of deciding which "dup" to delete.

Let me go back to the statement of the problem:

• The script must first detect the location of where a frame has been dropped;
• It must then look for a duplicate that has been added to keep audio in sync. This usually happens within a few frames of the drop
• It must do both these things so that dups=drops in order that the video length stay the same.
• It must then insert a motion estimated frame at each drop location and must also delete all the duplicate frames

My script works really well at detecting dups and drops, but it fails to match some dups with the drops because any decimation algorithm based on cycles will inevitably have a drop that happens in one cycle, but the dup doesn't happen until the next cycle.

By putting the metrics into an Excel spreadsheet, the decimation can be done without using cycles. The idea is to start with the first drop and then scan the next "n" frames for a metric that indicates a dup, and mark that for deletion. That will take a little work, but I know I can do it because I've done something very similar before.

What I don't know is how to use the metrics I create with the spreadsheet to reassemble the video in AVISynth, with motion estimated frames inserted at the drop points, and duplicates removed at the locations with identical frames. I can script it in Vegas Pro, but that won't help people here.

I have a long plane ride at the end of the week and I may use that time to see how far I can get with this. However, I'm stuck at the moment doing it all with AVISynth until and unless I get a few hints or ideas.

BTW, one reason I stuck with the original approach for so long is not only does it work for many drop/dup pairs, but also if the detection logic flags a "drop" that really isn't a drop, the visual penalty is very small. Any approach must gracefully handle false detection, given that perfect detection of every drop is not possible.

[stephen22]

There 'tis done. Finally cracked the diabolical ConditionalReader syntax (phew!)

source=AVISource( "D:\Video\wares-28-09-2021-17-35-08.avi" ).trim(5000,6000)

stepcode=0

super = MSuper(source)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)
interp = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70).subtitle("**")

final=Scriptclip(source,"stepcode=stepcode+framecode"+ chr(13) + \
"framecode < 1 ? selectevery(source,1,stepcode) : selectevery(interp,1,stepcode)")
final2=ConditionalReader(final, "spreadsheet.txt", "framecode", true)

return final2

It'll need tweaking when you sort out your spreadsheet program - for instance I think Insert_after should be insert_before and the interp file may need a nudge.

With the text file example in my previous post, the output would be


Frame 0=source frame 0, stepcode=0
Frame 1=source frame 1
Frame 2=source frame 2
Frame 3=interp frame 2, stepcode= -1
Frame 4=source frame 3, stepcode= -1
Frame 5=source frame 5, stepcode=0
etc

(Must be reset with F2 if you fiddle with the timeline or the value of stepcode is corrupted)

[stephen22]

That does work but only if dupes and drops are guaranteed to alternate, which doesn't usually happen. Better approach is to generate a text file which specifies the actual frame number required, indicating whether it's from the original clip or an interpolation. A good way of doing this is by by making interpolated frames negative, so 127 means frame 127 from the source clip and -127 means frame 127 from the interpolation clip.

So this is the method, based on John's brilliant insights - it works pretty well!

1. Generate a file of the ydiff data: the mfile from Multidecimate will do for this.

loadplugin ("D:\Videos\Avisynth\plugins\MultiDecimate.dll")
avisource ("D:\Video\SONY_DVD_RECORDER_VOLUME\ambass test.avi").converttoyuy2
multidecimate (pass=1) 

#***run video analysis pass then exit VD immediately***

2. Feed the resulting mfile.txt data into a program which will generate a text file as described above (I have called the generated file "spreadsheet.txt")

3. Put the file in the same folder as this avs script, and run the script.

loadplugin("C:\program files (x86)\avisynth\plugins\mvtools2.dll")

source=avisource ("D:\Video\SONY_DVD_RECORDER_VOLUME\ambass test.avi")
source2=scriptclip(source,"subtitle (string(current_frame))")

super = MSuper(source)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)
interp = MFlowInter(source,super, backward_vectors, forward_vectors, time=50, ml=70).duplicateframe(0)
interp2=scriptclip(interp,"subtitle (string(current_frame) + chr(42))")

final=Scriptclip(source2,"step = (framecode>0)? framecode - current_frame : 0 - (framecode + current_frame)" + chr(13) + \
"framecode >= 0 ? selectevery(source2,1,step) : selectevery(interp2,1,step)")

return stackvertical(ConditionalReader(final, "spreadsheet.txt",  "framecode", true),source2)

Writing the program for step 2 is the key, and free from the shackles of Avisynth scripting, the world is your oyster! My own version works as follows:

1. Set thresholds for dupe, drop and scene change.

2. Divide the clip into scenes, which will be processed individually

3. In each scene, score each frame by comparing with the average of the 16 frames surrounding it.

4. Identify and mark the drops

5. For each drop, mark the first dupe among the 8 surrounding frames. If none is found, mark the frame which has the lowest score.

6. Build up the output file, adding frame numbers to the list sequentially starting from 0. If a frame is marked as a dupe, skip its frame number. Put in frames marked as drops twice, the first time negative (to specify an interpolated frame). Save as a text file, in the format required by ConditionalReader (frame number followed by data.)

Here is a ragged clip captured by an HDMI to PAL converter, and the "spreadsheet" text file generated from its mfile data by my program. The Avisynth script above will display the clip before and after processing for comparison, with the frame number, aesterisked if interpolated, in the top left hand corner.
https://drive.google.com/file/d/1kzy...ew?usp=sharing

There's enormous scope here for skilful programming - for instance a sudden increase in movement within a scene can be a problem, creating runs of spurious "drops" which will in turn generate spurious "dupes" which will be removed and introduce new dropped frames. I thought I might try to identify this by comparing the 8 framescores before a frame with the 8 after, and perhaps starting a new "scene" if there is a large increase. It might also be useful to automatically adjust the dupe and drop thresholds for individual scenes according to the amount of movement. There's all sorts of problem-solving to be done. All of which is easy in a normal programming environment.

Unfortunately my version is programmed on an ancient computer - the Atari ST (in fact an ST emulator for Windows called Steem.) If anybody's interested I could supply the necessary files to run the program, which I would have to make a bit more user-friendly. But maybe if there's sufficient interest and demand, programmers more skilful than I could write exe files that everyone could use easily.