Trying to remove duplicate frames in a pattern

[chilledinsanity]

I'm trying to convert film that was encoded at 25fps back to its original framerate as it's loaded with duplicate frames. It's progressive, not interlaced. I've looked at the footage and it has a consistent pattern:

ABBABBB

A = every 22nd frame is duplicated (making the 23rd frame a copy)
B = every 24th frame is duplicated (making the 25th frame a copy)

I want to remove every duplicate frame based on that pattern, but I have no idea how to go about this. Does anyone know what command I should use for this with ffmpeg or avisynth?

[StainlessS]

So are you saying that,
Each A is 23 frames
Each B is 25 frames
so total cycle len = 2A + 5B = 2*23 + 5*25 = 171
and remove count = 7

?

[Sharc]

@chilledinsanity
I don't quite understand your pattern. Is it possibly Euro-pulldown (24p -> 25i)?
Upload a sample of the source (few seconds, it sould include a full pattern cycle) and you may get better help.

[StainlessS]

Maybe something like this (Assuming I interpreted your description OK)

MakeDelDupPattern.avs

# MakeDelDupPattern.avs

Colorbars.ShowFrameNumber
ORG=Last # Orig Video + Audio

OFFSET = 0 # Where to start pattern (which frame)

ALen = 23
BLen = 25
Pat = "ABBABBB"

Prelude = (OFFSET==0) ? Last.BlankClip(Length=0) : Trim(0,OFFSET-1)
Last = (OFFSET==0) ? Last : Trim(OFFSET,0)

str = MakeDelDupPattern(Pat, ALen, BLen) # "DeleteEvery(171,22,47,72,95,120,145,170)"

Subtitle(str)

Fixed = Eval(str) # Delete dups

Prelude = Prelude.AssumeFPS(Fixed) # Make Prelude same FrameRate as Fixed, otherwise Splice fail (MakeDelDupPattern changes Framerate as per DeleteEvery)

Prelude + Fixed

Return Last

Function MakeDelDupPattern(String Pattern,Int ALen,Int BLen,Int "CLen",Int "DLen") { # https://forum.doom9.org/showthread.php?t=181628
# Make Eval string to remove last frame from each A, B, C or D sequence. Patterns like "ABBCDABBCBD"
# Requires DeleteEvery from ApplyEvery:- http://avisynth.nl/index.php?title=ApplyEvery&redirect=no#DeleteEvery
# With x64 version from Groucho2004:- https://forum.doom9.org/showthread.php?t=173259&highlight=ApplyEvery
CLen = Default(CLen,0) # 'C' and 'D' in pattern are optional, but must have their Len provided if used.
DLen = Default(DLen,0)
Off=0
RS=""
For(i=1,Pattern.StrLen) {
Char = Pattern.MidStr(i,1)
Assert(Char=="A" || Char=="B" || Char=="C" || Char=="D","MakeDelDupPattern: Pattern can contain 'A', 'B', 'C' or 'D'Only")
Len = (Char=="A") ? ALen : (Char=="B") ? BLen : (Char=="C") ? CLen : DLen
Assert(0 < Len,"MakeDelDupPattern: " + Char + "Len cannot be 0 ('" + Char + "' used in Pattern)")
Off = Off + Len
RS = RS + String(Off-1,",%.0f")
}
Return String(Off,"DeleteEvery(%.0f") + RS + ")"
}


Requires DeleteEvery from ApplyEvery plugin:- http://avisynth.nl/index.php?title=ApplyEvery&redirect=no#DeleteEvery
Above link point to https://web.archive.org/web/20191106173119if_/http://www.iol.ie/%7Eschubert/gas/ApplyEvery031_32_64.7z
My ISP seem to block webarchive and so I cannot download the plugin, can somebody post it somewhere else please.
EDIT: OK, I downloaded from Groucho2004 stuff:- https://forum.doom9.org/showthread.php?t=173259&highlight=ApplyEvery

https://i.postimg.cc/nVRTMnKC/Every-00.jpg (https://postimages.org/)

EDIT: Also now allows for 'C' and 'D' in pattern.

[chilledinsanity]

StainlessS: I feel stupid writing this, but I get the impression you solved my problem entirely, but I'm too clueless to know how to implement it (in my defense, I did post in the newbies section). How do I plugin the code you provided to point at the video file? I assumed I saved it as .avs, but was lost after that. I was able to load the ApplyEvery plugin successfully in a script for what it's worth.

If there's a guide on this, by all means link me to it, I tried doing a search for that also, but was coming up dry for how to implement custom code like this.

[StainlessS]

There is no guide, I wrote this for your problem.
Use like so, [req Avs+, If you're not using AVS+, say so, just needs wrap the Entire MakeDelDupPattern Function in GSCript(""" ... """) ]

chilledinsanity.avs

Avisource("...") # Whatever your source is

### User Config ###
OFFSET = 0 # Where to start pattern (which frame)
ALen = 23 # "A = every 22nd frame is duplicated (making the 23rd frame a copy)"
BLen = 25 # "B = every 24th frame is duplicated (making the 25th frame a copy)"
Pat = "ABBABBB" # "and it has a consistent pattern "ABBABBB" "
### End User Config ###

Prelude = (OFFSET==0) ? Last.BlankClip(Length=0) : Trim(0,OFFSET-1)
Last = (OFFSET==0) ? Last : Trim(OFFSET,0)

str = MakeDelDupPattern(Pat, ALen, BLen) # str = "DeleteEvery(171,22,47,72,95,120,145,170)"

#Subtitle(str) # Un-comment to Show DeleteEvery() string

Fixed = Eval(str) # Delete dups

Prelude = Prelude.AssumeFPS(Fixed) # Make Prelude same FrameRate as Fixed, otherwise Splice fail (MakeDelDupPattern changes Framerate as per DeleteEvery)

Prelude + Fixed

Return Last

# GSCript(""" # UnComment if not AVS+, where will require GScript plugin
Function MakeDelDupPattern(String Pattern,Int ALen,Int BLen,Int "CLen",Int "DLen") { # https://forum.doom9.org/showthread.php?t=181628
# Make Eval string to remove last frame from each A, B, C or D sequence. Patterns like "ABBCDABBCBD"
# Requires DeleteEvery from ApplyEvery:- http://avisynth.nl/index.php?title=ApplyEvery&redirect=no#DeleteEvery
# With x64 version from Groucho2004:- https://forum.doom9.org/showthread.php?t=173259&highlight=ApplyEvery
CLen = Default(CLen,0) # 'C' and 'D' in pattern are optional, but must have their Len provided if used.
DLen = Default(DLen,0)
Off=0
RS=""
For(i=1,Pattern.StrLen) {
Char = Pattern.MidStr(i,1)
Assert(Char=="A" || Char=="B" || Char=="C" || Char=="D","MakeDelDupPattern: Pattern can contain 'A', 'B', 'C' or 'D'Only")
Len = (Char=="A") ? ALen : (Char=="B") ? BLen : (Char=="C") ? CLen : DLen
Assert(0 < Len,"MakeDelDupPattern: " + Char + "Len cannot be 0 ('" + Char + "' used in Pattern)")
Off = Off + Len
RS = RS + String(Off-1,",%.0f")
}
Return String(Off,"DeleteEvery(%.0f") + RS + ")"
}
# """) # UnComment if not AVS+, where will require GScript plugin


EDIT:
Or provided the pattern starts at frame 0 and always exactly like your problem then like so

chilledinsanity_2.avs

Avisource("...") # Whatever your source is
DeleteEvery(171,22,47,72,95,120,145,170)

The function just creates the above DeleteEvery() as a string given YOUR ALen, Blen and Pattern.
You still need the ApplyEvery plugin for the DeleteEvery filter.

[chilledinsanity]

StainlessS: Thanks, the "DeleteEvery" command made this vastly easier for me to understand (and worked, sort of).

Unfortunately it looks like the pattern is even more complex. While that command reduced the VAST number of the repeats, it still left some in, I suspect because when trying to convert back to 23.976, any variance other than knowing exactly which frames to decimate means duplicates invariable occur. So while that pattern I posted accounts for the vast majority of frames, it won't account for duplicates than show up every couple hundred frames or so because of rounding down to 3 decimal points.

In other words: every once in a great while it will be ABBABB or ABBBABBB as opposed to ABBABBB every single time. I haven't been able to determine what the (much longer) pattern is for that. I may have to accept defeat on this one.

[manono]

I understood what you wrote differently than the others - 2 duplicate frames in every 25-frame cycle.

You should do what Sharc suggested and post a sample.

[StainlessS]

Not to worry, maybe the function will come in useful for something else one day.
I'll leave it to Sharc and Manono when you post your sample, they should sort you out.

EDIT: Mediafire.com file host is a popular choice.