Repeated frame removal - 1 in every 23 frames

[Guest]

Please provide the entire script.

Also, I don't know if ffaudiosource() may have a problem with TrueHD audio, but if so, you can convert that to something usable with EAC3TO.

[mariner]

Originally Posted by neuron2

Please provide the entire script.

Also, I don't know if ffaudiosource() may have a problem with TrueHD audio, but if so, you can convert that to something usable with EAC3TO.

Greetings neuron2.

This scruipt works fro track=2, not track=1.

v=DirectShowSource("dog1.m2ts",audio=false)
a=FFaudioSource("dog1.m2ts",track=2)
audiodub(v,a)

[mariner]

Originally Posted by Didée

# LoadPlugin("MVTools2.dll")

dgsource("mastiff.dgi")

o     = last
o4    = o.pointresize(width+16,height+16,-4,-4,width+16,height+16)
o8    = o.pointresize(width+16,height+16,-8,-8,width+16,height+16)
o12   = o.pointresize(width+16,height+16,-12,-12,width+16,height+16)

d2p   = mt_lutxy(o,o.selectevery(1,-1),"x y - abs 128 *").greyscale()
dup   = mt_lutf(d2p,d2p,mode="avg",yexpr="x").mt_lut("x 2 < 255 0 ?")
skip  = dup.selectevery(1,-1)

sup   = o.msuper(pel=1)
bv    = sup.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv    = sup.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter = o.mblockfps(sup,bv,fv,num=0,mode=3).selectodd()

sup4  = o4.msuper(pel=1)
bv4   = sup4.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv4   = sup4.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter4 = o4.mblockfps(sup4,bv4,fv4,num=0,mode=3).selectodd().crop(4,4,-12,-12,true)

sup8  = o8.msuper(pel=1)
bv8   = sup8.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv8   = sup8.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter8 = o8.mblockfps(sup8,bv8,fv8,num=0,mode=3).selectodd().crop(8,8,-8,-8,true)

sup12 = o12.msuper(pel=1)
bv12  = sup12.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv12  = sup12.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter12 = o12.mblockfps(sup12,bv12,fv12,num=0,mode=3).selectodd().crop(12,12,-4,-4,true)

inter  = merge(merge(inter12,inter8),merge(inter4,inter))

conditionalfilter(dup,o,o.selectevery(1,1),"averageluma","<","2")
conditionalfilter(skip,last,inter,"averageluma","<","2")

return(last)

Try and see if/how it works out.

If you watch the single frames, you'll certainly find that there are errors in the motion-interpolated frames. In particular in the running dog scene, motioninterpolation is not very successful. In the mountains scene, OTOH, the interpolation is fairly okay.

However, do not only judge by the quality of single frames. Make some shorter encoding tests, and watch the result in motion. Lots of errors that seem obvious in a freeze frame, may go by almost un-noticed during normal playback. Important is that there is something that represents the expected motion state.
_____

Technical note:
The script implements averaging of 4 off-setted MBlockFPS. That's not fast (but no drama here because it is used only for 1 frame out of 24 anyway), but avoids the swirling-artifacts of MFlowFPS, and reduces the blocking of normal MBlockFPS. (IF it occurs, it's split up into micro-blocking, with the result appearing more "blurry", rather than "blocky".)

Thanks Didee.

Did not get too far with this script:

First, "Script error: there is no function named "mt-lutxy".
After loading "mt_masktools-25.dll"(is "mt_masktools-26.dll" needed?),
this error "MAnalyse: cannot load FFTW3.dll" appeared.
How should I proceed?

LoadPlugin("E:\data\tibetdog\didee\MVTools2.dll")
LoadPlugin("E:\data\tibetdog\didee\mt_masktools-25.dll")
DirectShowSource("dog1.m2ts",audio=false)

o     = last
o4    = o.pointresize(width+16,height+16,-4,-4,width+16,height+16)
o8    = o.pointresize(width+16,height+16,-8,-8,width+16,height+16)
o12   = o.pointresize(width+16,height+16,-12,-12,width+16,height+16)

d2p   = mt_lutxy(o,o.selectevery(1,-1),"x y - abs 128 *").greyscale()
dup   = mt_lutf(d2p,d2p,mode="avg",yexpr="x").mt_lut("x 2 < 255 0 ?")
skip  = dup.selectevery(1,-1)

sup   = o.msuper(pel=1)
bv    = sup.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv    = sup.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter = o.mblockfps(sup,bv,fv,num=0,mode=3).selectodd()

sup4  = o4.msuper(pel=1)
bv4   = sup4.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv4   = sup4.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter4 = o4.mblockfps(sup4,bv4,fv4,num=0,mode=3).selectodd().crop(4,4,-12,-12,true)

sup8  = o8.msuper(pel=1)
bv8   = sup8.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv8   = sup8.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter8 = o8.mblockfps(sup8,bv8,fv8,num=0,mode=3).selectodd().crop(8,8,-8,-8,true)

sup12 = o12.msuper(pel=1)
bv12  = sup12.manalyse(isb=true, blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
fv12  = sup12.manalyse(isb=false,blksize=16,overlap=0,search=5,searchparam=6,DCT=5)
inter12 = o12.mblockfps(sup12,bv12,fv12,num=0,mode=3).selectodd().crop(12,12,-4,-4,true)

inter  = merge(merge(inter12,inter8),merge(inter4,inter))

[Guest]

Originally Posted by mariner

This scruipt works for track=2, not track=1.

If it's like mastiff.m2ts, then track 1 is TrueHD. As I said, convert it to something usable with EAC3TO.

[Didée]

masktools-25.dll is correct if you use Avisynth v2.5.x. masktools-26.dll is correct if you use Avisynth v2.6.x

Oops, I wasn't aware that MVTools requires the fftw3 library. Seems it is required when a DCT mode other than zero is used.

You grab the fftw3 DLL from here, and unpack the DLL into ...

... \windows\system32 - IF you run 32-bit Windows, or into
... \windows\syswow64 - IF you run 64-bit Windows.

[mariner]

Originally Posted by neuron2

If it's like mastiff.m2ts, then track 1 is TrueHD. As I said, convert it to something usable with EAC3TO.

Good to get confirmation that FFaudioSource does not load TrueHD track. Thanks neuron2.

[Guest]

I'm not confirming that! I'm saying if it is the case, and your experience suggests it, then you can work around it by transcoding it with EAC3TO.

[mariner]

Originally Posted by Didée

masktools-25.dll is correct if you use Avisynth v2.5.x. masktools-26.dll is correct if you use Avisynth v2.6.x

Oops, I wasn't aware that MVTools requires the fftw3 library. Seems it is required when a DCT mode other than zero is used.

You grab the fftw3 DLL from here, and unpack the DLL into ...

... \windows\system32 - IF you run 32-bit Windows, or into
... \windows\syswow64 - IF you run 64-bit Windows.

Thanks Didée.

1. The inserted frame, especially the dog, looks blurry.

2. Can't get smooth motion during playback. My Q8400 system had to pause to generate the new frame. Will do a re-encode to test.

3. What would you suggest using, FFVideoSource or DSS2?

4. How about using this 2-pass script to encode the 94min video? Suggestions for tweaking?

x264 --bitrate 26000 --preset veryslow --tune animation --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --pass 1 -o 00007.264 00007.avs
x264 --bitrate 26000 --preset veryslow --tune animation --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --pass 2 -o 00007.264 00007.avs

5. Is --fps 24000/1001 needed?

6. x264 displays the number of frames encoded, but not of the original file. How does one get that info or get x264 to display that?

Many thanks and best regards.

[mariner]

Originally Posted by mariner

Thanks Didée.

3. What would you suggest using, FFVideoSource or DSS2?

4. How about using this 2-pass script to encode the 94min video? Suggestions for tweaking?

x264 --bitrate 26000 --preset veryslow --tune animation --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --pass 1 -o 00007.264 00007.avs
x264 --bitrate 26000 --preset veryslow --tune animation --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --pass 2 -o 00007.264 00007.avs

5. Is --fps 24000/1001 needed?

6. x264 displays the number of frames encoded, but not of the original file. How does one get that info or get x264 to display that?

Many thanks and best regards.

Greetings Didee.

1. Did you get a chance to take a look at these questions? Would appreciate some advice here.

2. Could you kindly modify the algorithm to accommodate this sequence shown in cut.m2ts:

1,1a, 2,2a, 3,3a, ....11,11a,11b, 12
13,13a, 14,14a, 15,15a,......23,23a,23b, 24

The current routine seems to remove frame 11a and insert a new frame between frames 11b and 12, while it should actually be removing 11b and insert one between 12 and 13.

It would be better still if frame 12 can be repeated for this type of sequence.

3. x264 log reports different number of frames encoded when different source filters are used:
DS=DSS2+1=FFVS+2
Why?

4. DS also seems to exhibit some strange out of sequence behavior : frame 13 is replaced by 11, in 2 out of 3 sequences (see cut_ds.m2ts). What's peculiar is this seems to happen during encoding, but not during playback of original file.
How did this happen?

Many thanks and best regards.

cut.m2ts
cut_ds.m2ts

[Didée]

I hear you. However, it's a tricky problem.
The filterchain is laid out for 24fps sections. That's the "easy" part, and that's where motion stutter is mostly noticeable. (Panning scenes in particular).

For 12fps sections, doing the "correct" thing is a different ballgame. Instead of looking for single dups, it would be required to look for triples, and the interpolation part should be omitted completely.

I hoped to eventually get by with the slightly suboptimal handling of 12fps sections. - Getting everything together is difficult. Doing extended dup detection via script is cumbersome, and it is coarse. (E.g.: can't tell if some small frame differences are due to small artifacts [is-a-dup], or due to small mouth-movement [is-NOT-a-dup].) Filters like TDecimate have much more sophisticated evaluation methods for dup detection ... but are laid out to just do the decimation, and won't allow nicely to be used for frame-shuffling like it is required here.

The easiest thing would be to set up one routine for 12fps and one for 24fps, then let the user splice everything together manually.
A trim()+trim()+trim()+trim() orgy, that is.

Perhaps I can have another look at it this weekend ... but no guarantee.

[mariner]

Originally Posted by Didée

I hear you. However, it's a tricky problem.
The filterchain is laid out for 24fps sections. That's the "easy" part, and that's where motion stutter is mostly noticeable. (Panning scenes in particular).

For 12fps sections, doing the "correct" thing is a different ballgame. Instead of looking for single dups, it would be required to look for triples, and the interpolation part should be omitted completely.

I hoped to eventually get by with the slightly suboptimal handling of 12fps sections. - Getting everything together is difficult. Doing extended dup detection via script is cumbersome, and it is coarse. (E.g.: can't tell if some small frame differences are due to small artifacts [is-a-dup], or due to small mouth-movement [is-NOT-a-dup].) Filters like TDecimate have much more sophisticated evaluation methods for dup detection ... but are laid out to just do the decimation, and won't allow nicely to be used for frame-shuffling like it is required here.

The easiest thing would be to set up one routine for 12fps and one for 24fps, then let the user splice everything together manually.
A trim()+trim()+trim()+trim() orgy, that is.

Perhaps I can have another look at it this weekend ... but no guarantee.

Greetings and many thanks Didee.
You're right about how panning scenes killed the film.
I've enclosed examples of these that have not been effectively handled by the algorithm,
and hope perhaps you can make some improvements as well.
Best regards.

pan1.m2ts
pan2.m2ts

[Didée]

Sorry, but ATM I can do nothing. My PC is knocked-out, the power supply blew up on Saturday. Don't have suited sparepart, and RMA will take (too many) days ...

[mariner]

Sorry to hear that, Didée. Please take your time.

Did you use neuron2's source filter to test the clips? If so can you kindly let me know the number of frames encoded reported by x264 to be used as a reference?

Which source filter would you suggest? DirectShowSource, FFvideoSource or DSS2? Or something else?

Many thanks and best regards.

[Didée]

Here we go again.

In the first set of samples, there was an offset of 2 frames between false-dup and missing-frame. In this second set, there is only 1 frame offset.
The script was made for the 1st case, and of course won't work correctly in the 2nd case.

What ~seems~ to have happened to the source is this: there was a telecined stream at either 30p or 60p stage, and the decimation was not done with a "smart" decimator, but rather by using a static/fixed removal pattern. Which of course will fail when the source's telecine pattern isn't 100% constant.

This would a) fully explain the issue of dups and missing frames, but it als raises b) a bigger problem: Though the samples are only showing cases of 1 or 2 frames offset, it is to expect that there could be ANY offset, from zero up to 23 frames, between the dups and the missing-frames. And that's a problem indeed. This means you cannot just look for duplicates and conclude the missing frames from the dups' positions. Instead, you have to identify the positions of missing frames by themselves. Yeeha! Reckognizing dup-frames is relatively easy, but reckognizing missing frames is very difficult.

I fear there will be no magic-script-solution for this problem. Well, I have done something similar in the past, but it was for "natural" content instead of Anime (natural content is much easier in respect to skip detection), and the deal was only about short 5-frames-intervals (from 30>25 decimation) instead of 24-frames intervals like here.
(Reckognition of missing frames in any case is lousy and whacky, so you need confirmation of pattern-distanced neighbors to help making decisions. Having to use -10-5+5+10 frames (20 frame window) is bad enough, but using -48-24+24+48 neighbors (100 frame window) smells like the whole shebang going BOOM.

It seems I'm giving up at this point.

If anybody has better ideas: speak loud.

[mariner]

Greetings Didée. Many thanks for taking a detailed look at the clips.

I think you're right. While the flying eagle sequence in pan1.m2ts follows the 2 frame offset pattern, the beginning of the clip and pan2.m2ts does not reveal any clue. 1 frame offset seems like the best fit.

Did you get a chance to look at cut.m2ts? It seems using the DirectShowSource filter with you script results in out of sequence behavior. Any explanation?

Many thanks and best regards.