Repeated frame removal - 1 in every 23 frames

[mariner]

Greetings.

This one is strange: the BD appears to have been encoded with one repeated frame inserted for every 23 frames, resulting in very jerky motion. Frame rate is 23.976 progressive.

How can one remove the repeated frame, ideally without "re-encoding", or perhaps even in real time playback? And what to do with the audio and subtitles?

The BD is Zang Ao Duo Ji, a Japanese/Chinese animation.

Many thanks and best regards.

Sample
Sample

[Guest]

You can't fix it without re-encoding. You can run an Avisynth script for re-encoding or for real-time playback:

TDecimate(cycleR=1,cycle=23)

You'll end up with a weird frame rate, though.

[mariner]

Originally Posted by neuron2

You can't fix it without re-encoding. You can run an Avisynth script for re-encoding or for real-time playback:

TDecimate(cycleR=1,cycle=23)

You'll end up with a weird frame rate, though.

Thanks for the kind reply, neuron2.

A few questions:

1. Does it start from frame 1? Or does it identify the repeated frame?

2. What frame rate do you suggest to use for reencoding the decimated video stream? Same as the original, 23.976?

3. What software should I use to speed up/resample the audio stream to match the new video stream? Do I do 24/23 speed up? Can it be done keeping the same pitch?

4. Same for subtitle?

5. Is this due to some obscure anime cadence confusing the splitter? Or is this a genuine error in the encoding encoding?

Many thanks and best regards.

[Guest]

Originally Posted by mariner

1. Does it start from frame 1? Or does it identify the repeated frame?

It identifies the repeated frames, usually very successfully.

2. What frame rate do you suggest to use for reencoding the decimated video stream? Same as the original, 23.976?

It depends on your targeted device. That would be a good choice, as it remains close to the original.

3. What software should I use to speed up/resample the audio stream to match the new video stream? Do I do 24/23 speed up? Can it be done keeping the same pitch?

You can do it simply in Avisynth as follows:

vid=dgsource("mastiff.dgi")
aud=nicac3source("mastiff PID 1101 3_2ch 48KHz 448Kbps DELAY -25ms.ac3",2).delayaudio(-0.025)
audiodub(vid,aud)
tdecimate(cycle=23,cycleR=1)
assumefps(23.976,sync_audio=true)

That will not correct the pitch but maybe it is good enough.

If you want to correct the pitch, look at TimeStretch(), although I could not make it work for changing the decimated 22.934 to 23.976. Maybe an audio expert will chime in on that. Alternatively, you can use an external application like Audacity to change the audio rate without changing the pitch.

4. Same for subtitle?

Sorry, I've no experience there. Try the Subtitles forum, I've seen threads there about resampling.

5. Is this due to some obscure anime cadence confusing the splitter? Or is this a genuine error in the encoding encoding?

It's in the encoding, as I see it with my tools, which do not use a splitter.

[mariner]

Thanks again, neuron2.

Got the following error message:

Scripte error: there is no function name "dgsource"

Created the following .avs file and played in MPC. What else is amiss?

vid=dgsource("mastiff.dgi")
aud=nicac3source("mastiff PID 1101 3_2ch 48KHz 448Kbps DELAY -25ms.ac3",2).delayaudio(-0.025)
audiodub(vid,aud)
tdecimate(cycle=23,cycleR=1)
assumefps(23.976,sync_audio=true)

[Guest]

dgsource() is a source filter available in the DGDecNV package.

You may use any other source filter that you have access to.

[mariner]

Thanks neuron2.

1. Copied the contents of DGDecNV2042 to the plugins folder of Avisynth and got the following error message:

DGSource : Cannot open index file!

2. Does DGDecNV2042 work with ATI graphics card? If not. what are the alternatives?

3. Should the cycle parameter in tdecimate be set to 24 instead of 23?

Many thanks and best regards.

[Guest]

Originally Posted by mariner

1. Copied the contents of DGDecNV2042 to the plugins folder of Avisynth and got the following error message:

DGSource : Cannot open index file!

Well, did you make an index file? Did you read the quick start guide and learn how everything works? Anyway, don't bother with that now because you need an nVidia card.

2. Does DGDecNV2042 work with ATI graphics card? If not. what are the alternatives?

Of course it won't work with ATI! You have to start reading things like documentation, tool websites, and existing threads. The alternative as I said is to use a different source filter. Maybe DirectShowSource() or FFMPEGSource().

3. Should the cycle parameter in tdecimate be set to 24 instead of 23?

You said the problem was to remove one in 23, so the cycle is 23.

[Keiyakusha]

Have you considered to interpolate every 23th frame instead of removing it?
Benefits: no need to mess with audio, framerate will remain standard

But you have to ask someone else to provide exact script for that.

[poisondeathray]

Originally Posted by Keiyakusha

Have you considered to interpolate every 23th frame instead of removing it?
Benefits: no need to mess with audio, framerate will remain standard

just to add:

negatives: possible interpoloation artifacts (edge morphing).

recommended a frame accurate source filter for this type of work (otherwise frames get mixed up and interpolated incorrectedly)

You can encoded to an I-frame lossless intermedate with directshowsource if you have to

But you have to ask someone else to provide exact script for that.

If it's constant then you can use applyevery.dll with mflowinter from mvtools2

e.g these are duplicates:
5,6
29,30
53,54

(This code was from gavino's help on another project)

sup = MSuper()
bv = MAnalyse(sup, isb=true, delta=2)
fv = MAnalyse(sup, isb=false, delta=2)
interpolated = MFlowInter(sup, bv, fv, time=50.0, ml=100).DuplicateFrame(0)
replace = interpolated.SelectEvery(24, 6) # replacement frames

DeleteEvery(24, 6) # delete bad frames
InterleaveEvery(replace, 24, 6) # replace deleted frames

So the offset is 6 (1st dupe occurs there), and it's every 24 (that's why 24, 6 is used)



If the duplicates are not in a fixed pattern, you can use filldrops() or related functions (there are a few modified ones floating around for mvtools2), which calculate based on dupe detection (you can adjust the threshold value and check it with the debug mode)

[mariner]

Thanks neuron2. Few more questions:

Video

1. The pattern seems to be 23 distinct frames followoed by 1 repeated frame, so tdecimate(cycle=24,cycleR=1)?

2. What parameters do you suggest for re-encoding the decimated video, for 22GB target file size to fit in a BD-25?
Here's the mediainfo:

General
ID                                       : 0 (0x0)
Complete name                            : D:\BDMV\STREAM\00007.m2ts
Format                                   : BDAV
Format/Info                              : Blu-ray Video
File size                                : 19.5 GiB
Duration                                 : 1h 34mn
Overall bit rate mode                    : Variable
Overall bit rate                         : 29.5 Mbps
Maximum Overall bit rate                 : 48.0 Mbps

Video
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 3 frames
Format settings, GOP                     : M=3, N=12
Codec ID                                 : 27
Duration                                 : 1h 34mn
Bit rate mode                            : Variable
Maximum bit rate                         : 35.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 23.976 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio

3. It's not clear how the -25ms delay helps when the decimated clip is ~4% (4 minutes) shorter. Or is the audio sync issue dealt with by the sync_audio=true parameter?

4. The AC3 audio is 6 channels, but can't seem to get nicac3source to output more than 2 channels. Both aud=nicac3source("mastiff.ac3",6) and ("mastiff.ac3") result in error.

5. TimeStretch(tempo = 100*25.0/(24000.0/1001.0)) seems to work on the sample. Need to test with the full clip.

6. Need help with audio source filters that work on the m2ts clip instead of demuxed audio files. Not having much success with FFaudiosource and DirectShowSource.

The clip has two 6 channel audio streams : 640kbps TrueHD/AC3 and 448kbps AC3.

Audio #1
ID                                       : 4352 (0x1100)
Menu ID                                  : 1 (0x1)
Format                                   : TrueHD / AC-3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Muxing mode                              : Stream extension
Codec ID                                 : 131
Duration                                 : 1h 30mn
Bit rate mode                            : Variable / Constant
Bit rate                                 : Unknown / 640 Kbps
Maximum bit rate                         : 4 947 Kbps / 640 Kbps
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits

Audio #2
ID                                       : 4353 (0x1101)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 1h 30mn
Bit rate mode                            : Constant
Bit rate                                 : 448 Kbps
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 291 MiB (1%)

Many thanks and best regards.

[Guest]

Yes, those tdecimate values would be correct values for 23 frames followed by one duplicate.

Can't help with your encoding question, I rarely do such things.

Audio delay affects the starting offset of the audio so it doesn't matter if frames are removed. Note that the first frame cannot be removed because it cannot be a duplicate. The tdecimate() by itself will not change the audio sync. That would only be an issue if you did an AssumeFPS() to get back to a standard frame rate.

I suggest reposting your other audio questions in appropriate threads/forums. I don't know the answers to them.

[Didée]

Did anyone realize that always 2 frames after the duplicate frame there is one frame missing? - It's a "A B C D D E,,G H" sequence. It almost seems like some form of IVTC has been performed, but the decimator has dropped out of phase, hence has removed good nearby frames instead of the IVTC-duplicates.

[Guest]

Originally Posted by Didée

Did anyone realize that always 2 frames after the duplicate frame there is one frame missing?

Nope, didn't notice that. I believe you though.

[Didée]

In the given sample it's not exactly obvious to see, but it can be tracked. In the very beginning there is 12fps animation. Stepping single frames, one can see that every 24 frames there is one instance with 3 frames of one motion state, followed by 1 frame of one motion state. In the long running dog sequence its hard to see in motion, but the missing frame can be tracked by the grass at the bottom, there's always a skip one frame after the dup. It would be more obvious in the end scene, scrolling mountains at 24fps - but the scene is too short to notice it easily.

In any case, just deleting the dup frame and forcing the 23fps to 24fps again will not result in fluid motion. (Well, whatever "fluid" may mean in case of animation.) After deleting the dups, you still have the skips ...

What would be needed to be done is: shift the frame right of the dup to the left, and place something interpolated into its position - either a simple blending, or a motion interpolation. Motion interpolation is somewhat problemtic with animation, though.

[mariner]

Greetings Keiyakusha, poisondeathray and Didée.

Many thanks for all the suggestions, and in particular Didée's brilliant observation.

Didée is right, of course. The attached screen cap of 8 frame running dog sequence shows clearly the repeated (blue) and removed (red) frames.

Re-encoding seems to be in order. Detailed instructions appreciated.

Many thanks and best regards.

[Didée]

# LoadPlugin("MVTools2.dll")

dgsource("mastiff.dgi")

o     = last
o4    = o.pointresize(width+16,height+16,-4,-4,width+16,height+16)
o8    = o.pointresize(width+16,height+16,-8,-8,width+16,height+16)
o12   = o.pointresize(width+16,height+16,-12,-12,width+16,height+16)

d2p   = mt_lutxy(o,o.selectevery(1,-1),"x y - abs 128 *").greyscale()
dup   = mt_lutf(d2p,d2p,mode="avg",yexpr="x").mt_lut("x 2 < 255 0 ?")
skip  = dup.selectevery(1,-1)

sup   = o.msuper(pel=1)
bv    = sup.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv    = sup.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter = o.mblockfps(sup,bv,fv,num=0,mode=3).selectodd()

sup4  = o4.msuper(pel=1)
bv4   = sup4.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv4   = sup4.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter4 = o4.mblockfps(sup4,bv4,fv4,num=0,mode=3).selectodd().crop(4,4,-12,-12,true)

sup8  = o8.msuper(pel=1)
bv8   = sup8.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv8   = sup8.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter8 = o8.mblockfps(sup8,bv8,fv8,num=0,mode=3).selectodd().crop(8,8,-8,-8,true)

sup12 = o12.msuper(pel=1)
bv12  = sup12.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv12  = sup12.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter12 = o12.mblockfps(sup12,bv12,fv12,num=0,mode=3).selectodd().crop(12,12,-4,-4,true)

inter  = merge(merge(inter12,inter8),merge(inter4,inter))

conditionalfilter(dup,o,o.selectevery(1,1),"averageluma","<","2")
conditionalfilter(skip,last,inter,"averageluma","<","2")

return(last)

Try and see if/how it works out.

If you watch the single frames, you'll certainly find that there are errors in the motion-interpolated frames. In particular in the running dog scene, motioninterpolation is not very successful. In the mountains scene, OTOH, the interpolation is fairly okay.

However, do not only judge by the quality of single frames. Make some shorter encoding tests, and watch the result in motion. Lots of errors that seem obvious in a freeze frame, may go by almost un-noticed during normal playback. Important is that there is something that represents the expected motion state.
_____

Technical note:
The script implements averaging of 4 off-setted MBlockFPS. That's not fast (but no drama here because it is used only for 1 frame out of 24 anyway), but avoids the swirling-artifacts of MFlowFPS, and reduces the blocking of normal MBlockFPS. (IF it occurs, it's split up into micro-blocking, with the result appearing more "blurry", rather than "blocky".)

[mariner]

Thanks for the script, Didée.

1. Can you suggest another source filter (must it be frame accurate?) for my non NV system?
2. Can you help with getting audio to work? This script result in error.

a=FFaudioSource("mastiff.m2ts",track=2)

Many thanks and best regards.

Edit:
1. Audio track 2 playable by enabling raw codec in FFdshow audio decoder.
2. Audio track 1 not playable, error message: "FFaudioSource : audio stream not seekable."

[Guest]

Originally Posted by mariner

This script result in error.

Tell us what the error was.

[mariner]

Originally Posted by neuron2

Tell us what the error was.

How do I get audio track 1 to play?