Source filter behavior for progressive NTSC @23.976 with pulldown flags

[poisondeathray]

Your DSS2Mod version looks right, maybe compare it with my version:
https://www.sendspace.com/file/if6r6r


Thanks , it's the same. (and I tried avss_x64.dll for x64 version)

What lav version are you using ? splitter and decoder ?


In one previous post you mentioned your source filter call:
DSS2("title00 (1).mkv", preroll=X)

I hope you did not really set "preroll=X". Use 15, and do specify the fps value...


The "X" was to denote different values were attempted

If you check the later post about the full movie I used fps=23.976 and preroll=15 (And I tried without preroll and 0 and other values like 24,30 ) to see if it would help


And yes, a longer clip which shows the issues would be nice.


The first stretch of big problems occurs ~ 52min in and lasts about 7 minutes. I tried cutting a 10min section, enclosing the problem area but it wouldn't reproduce on the cut section . I think it has to be from the beginning . I'm trying to avoid having to upload 2-3GB

Also I I'll include audio this time




Maybe it's time to add MKV support to DGIndex/MPEG2Source.

+1 .

That would be great if you have free time

[poisondeathray]

If anyone else wants to take a look at the longer sample, PM me.

Unfortunately the sample is massive. I couldn't get the first long bad section to replicate unless it was cut from near the beginning. But OTOH it's easier to see the ffms2 sync issues with a longer sample



The dss2mod massive dropped/duplicated frame sections issue looks related to dss2mod+lav decoder (none or CPU), or dss2mod+lav(cuvid,nvidia). dss2mod+QS seemed ok at first, but it just shifted the drop/dupe section to other sections instead (but not as long or as bad as lav(cpu or cuvid), at least with a quick glance). Also, dss2mod+ffdshow doesn't seem have that issue for mpeg2 decoding . Other decoders like mainconcept also have the same long stretch of bad (the full 7 min or other long stretches weren't included in this sample). DirectShowSource doesn't have that particular issue with any decoder (which is weird) . But all DS versions still have combing in some scenes, and usually a grey frame at the end

haali + lav (cpu or cuvid) had same issue, so it's probably not lav splitter . I tried older versions of lav too, lav threads=1

ffms2 had 31 extra frames in this longer sample, not surprisingly it was about that much out of sync near the end. (And more than double that at the end of the full movie)

If anyone can't see the combing with DS/DSS2 or ffms2 , let me know and I can point where those sections are

[FranceBB]

Maybe it's time to add MKV support to DGIndex/MPEG2Source.




+1 .

That would be great if you have free time

+2.
Even though we can just demux the file and use DGIndex, it would be great to have a new version of DGIndex. ^_^

[manolito]

@ poisondeathray

Played with your uploaded source for a while, this is what I got so far...

The combed frames in the output are not caused by the source filters, they are already present in the source MKV. I suppose that they were already in the VOBs on the source DVD. MakeMKV does not reencode anything, and removing copy protection should not introduce combed frames either.

Otherwise I could reproduce the duplicated frames issue on two different computers. It still puzzles me how this can happen. Correct output for a long time, then suddenly dupes and dropped frames. Some kind of buffer overflow problems?

I tried a few different MPEG2 decoders to make sure that LAV Filters was not to blame, but no real result. With an old Cyberlink decoder and also the old DScaler decoder the sections with the dupes started at a different point in time, but they were there.

So the gold standard for such 23.976 progressive NTSC sources with pulldown flags really seems to be using MPEG2Source (honoring pulldown flags) followed by a proper IVTC.

The only other method I found which does come close to it is this one:
1. Use DSS2Mod forcing 29.97. This will result in a progressive 29.97 output with tons of dupes.
2. Instead of using standard IVTC just use "TDecimate(mode=7, rate=23.976)".

The Aliens clip comes out almost perfectly with this setting. I also tried FDecimate and FDecimate2 (by StainlessS), but the TDecimate command worked better.

Looks like we need to wait for DG to update DGIndex to work with MKV sources...


Cheers
manolito

[poisondeathray]

The combed frames in the output are not caused by the source filters, they are already present in the source MKV. I suppose that they were already in the VOBs on the source DVD. MakeMKV does not reencode anything, and removing copy protection should not introduce combed frames either.


Yes.

But the MPEG2Source method using TIVTC does not have them in any of the sections. Recall I purposely set TFM(pp=0) to reveal any problems (in case it was deinterlacing the combed frames and hiding the problems) . I believe underlying issue is it was something like 99.8% film according to DGIndex, so if you had "forced film" in DGIndex you'd get the same combed frames. But this is very common for soft pulldown DVD's . The majority are probably around 99.5 or so. The implication here is that you need to return fields, in order to field match. So if DSS2 at 29.97 (or any source filter) is not returning fields , it has no hope of doing it properly in terms of the combing (unless you do some sort filtering afterwards, which will always produce worse) . But combing isn't as bad as the jerky playback or sync issues. You could argue that combing in a few scenes is watchable even if not ideal .



Otherwise I could reproduce the duplicated frames issue on two different computers. It still puzzles me how this can happen. Correct output for a long time, then suddenly dupes and dropped frames. Some kind of buffer overflow problems?

That sample only had ~3 minutes of dupes at the end , but that stretch of problems on the full movie actually lasted 7 minutes. And then recovered, and a few more stretches of dupes/drops that were about a few min long later on. But I tried the x64 version as well in case it was some memory issue , but same issue. Very flaky behaviour



I tried a few different MPEG2 decoders to make sure that LAV Filters was not to blame, but no real result. With an old Cyberlink decoder and also the old DScaler decoder the sections with the dupes started at a different point in time, but they were there.


Yes, they get shifted with different decoders, but not sure if you missed it but ffdshow seemed to work ok with dss2mod when set to libavcodec, even on the full movie (only the combing and customary missing frame at the end) .

So do you expect users to play with directshow configurations and decoder settings? Person A might have different settings than Person B. I usually have mine set on CUVID. ffdshow is hardly used by anyone. So ffdshow might worked here in this particular example, but I've seen problems with ffdshow in the past too in this scenario... way to inconsistent




The only other method I found which does come close to it is this one:
1. Use DSS2Mod forcing 29.97. This will result in a progressive 29.97 output with tons of dupes.
2. Instead of using standard IVTC just use "TDecimate(mode=7, rate=23.976)".

The Aliens clip comes out almost perfectly with this setting. I also tried FDecimate and FDecimate2 (by StainlessS), but the TDecimate command worked better.


whaaat ? Not even close, jerky in sections.



And did you get a chance to look at ffms2 versions? Hopeless, right? Unusable because of the progressive sync issues and added frames. Or did you find a "magical" version ?



To reiterate - this is quite common with DVD/MPEG2 when using Directshow, FFMS2 . Not just these sorts of problems, but others as well. I'm not making this up or blowing it out of proportion. It might be "new" to you because you're in PAL land, but this is common knowledge for NTSC avisynth users . You have a consistent reliable proven method (actually 2, if you count dgsource), and then "the others" which are unreliable , prone to many issues. Again - I would avoid even having those as options for DVD/MPEG2 sources for a GUI - grey they out or plaster some warning signs

[manolito]

whaaat ? Not even close, jerky in sections.

Which sections exactly?
https://we.tl/t-9aaHGd12Fo

I think this conversion looks pretty good...


Cheers
manolito

[poisondeathray]

Which sections exactly?
https://we.tl/t-9aaHGd12Fo

I think this conversion looks pretty good...


Cheers
manolito


Reallly ? look in the first 5-6 minutes , it should be very obvious, every scene has dropped/duplicated frames and jumps in motion.

What's weird is looks closer to normal after that. I got much choppier results initially , throughout that the whole sample. But then I switched to lav CPU (instead of lav CUVID), then I got the same choppy sections as you. But in the full movie, choppy sections reappear later too even with lav CPU. The combing might be considered minor to some people, but the choppiness makes it unusable . Maybe you didn't look at the first 5-6 minutes ?

[manolito]

Maybe you didn't look at the first 5-6 minutes ?

Yes, this could be it. I was somehow concentrating on the last third of the clip where the dupes showed up in the older conversions. Need to have another look.. :o

[manono]

But some folks here are adamant that this method is evil and only MPEG2Source can handle such clips. And I want proof of this...

Do you think we're 'adamant' because we're fanboys or something similar? We're adamant because it's the only way that works reliably. It seems to me that you're the adamant one, trying to fit a square peg into a round hole. Sure, you're trying come up with a method to make AVSToDVD work well with all sources.


To reiterate - this is quite common with DVD/MPEG2 when using Directshow, FFMS2 . Not just these sorts of problems, but others as well. I'm not making this up or blowing it out of proportion. It might be "new" to you because you're in PAL land, but this is common knowledge for NTSC avisynth users . You have a consistent reliable proven method (actually 2, if you count dgsource), and then "the others" which are unreliable , prone to many issues. Again - I would avoid even having those as options for DVD/MPEG2 sources for a GUI - grey they out or plaster some warning signs

I suspect pdr spent hours of his own time creating his samples, testing and making his long posts explaining the results of his tests. Neither I nor pdr (I don't think) even use AVSToDVD. It's not just you but many of the source filter developers that reside in PAL countries and haven't had all that much experience with NTSC material. And their filters just don't work well with difficult NTSC VOBs/MPGs. It's a worthy goal to help the inexperienced make better encodes. Please pay attention and learn.

[manolito]

Do you think we're 'adamant' because we're fanboys or something similar? We're adamant because it's the only way that works reliably. It seems to me that you're the adamant one, trying to fit a square peg into a round hole. Sure, you're trying come up with a method to make AVSToDVD work well with all sources.


Well, I started this thread because I wanted to learn about the different source filter behavior with these progressive NTSC sources with soft pulldown flags. But basically all the answers I got were like "Forget it, just use MPEG2Source, all the other source filters just don't work with these sources, case closed". And these kind of answers do trigger a certain stubbornness in me... :devil:

Can it really be true that after the DVD format has been around for so many years noone except DG was able to write a source filter which could handle such sources?

Thanks to pdr I got a nice source file to play with, and yes, I have exhausted all options with DSS2Mod so far. The reason is that DirectShow based source filters do not honor pulldown flags, and for sources which are less than 100% film there is no other way than adding dupes or drop frames to achieve the target rate.

But I am not a person who gives up that easily. We still have ffms2 to test. I am aware that pdr mentioned that while ffms2 can honor pulldown flags with the rffmode parameter, this option was buggy. But since I believe only in things I have tested myself, I put his uploaded file on my test bench...

I used the FFMS2 C-plugin 1140+101 from 2016, the AVStoDVD script was straightforward:
LoadCPlugin("C:\Program Files (x86)\AVStoDVD\Lib\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Audio = FFAudioSource("C:\Download\title00 (1).mkv", track=-1)
Video = FFVideoSource("C:\Download\title00 (1).mkv", track=-1, rffmode=1, seekmode=0)

Video = Video.ConvertToYV12()
Video = Video.AssumeTFF()
Video = Video.TFM().TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)

The AssumeTFF() call was a shot from the hip, I did not take the time to check the field order. Downmixing audio to stereo was just a means to get the final result size below 2 GB.

Download the result here:
https://we.tl/t-ojt0XCVLs5

Any objections? Audio sync is perfect, and I could not detect any jerkiness due to duplicated or dropped frames.


Cheers
manolito

[manono]

And these kind of answers do trigger a certain stubbornness in me... :devil:
And your earlier claim that I was (or we were) promoting some kind of an agenda pissed me off no end. My only agenda is having something that works reliably. So I kept quiet for a few days lest I write something I might later regret. Besides, I know your goal is worthy and your heart's in the right place. And I can't do, and have no interest in doing, the kinds of things pdr did for you.

Don't you think you should make available your source, the title00 (1).mkv?

[videoh]

Can it really be true that after the DVD format has been around for so many years noone except DG was able to write a source filter which could handle such sources? There are several gotchas and corner cases for implementing correct and robust field pulldown, especially with frame accurate random access (seeking). It has to be implemented carefully. Many of the source filter developers are in Europe and they may therefore not be fully motivated for NTSC things, or may not have the needed source samples for proper testing. It's hard to say. But the source code for MPEG2Source contains the solution and it is open source so in my opinion there's no excuse for having a buggy pulldown implementation. The original implementation was by jackei of DVD2AVI fame, by the way, and DG enhanced it for robust random access. So there's no need to succumb to DG derangement syndrome, at least in this case. ;)

[poisondeathray]

Any objections? Audio sync is perfect, and I could not detect any jerkiness due to duplicated or dropped frames.




It's better and the other ffms2 versions and in sync

But there is significantly more aliasing and combing, even though you didn't set pp=0. (For testing purposes, I always use pp=0 , so it doesn't deinterlace and it's easier to see the problems). Check the first few minutes again, about 1/3-1/4 of the frames in the first few minutes are combed. It's easy to see with the spaceship coming in. In the other sections, significantly more combing introduced in other scenes than the directshow methods. Some of the problems might be partially hidden by your resizing and encoding. Preview the script without resizing and pp=0 to see it clearly. When using TIVTC, you should set the field order explicitly if not using DG source filters (DG stuff passes the info, others usually do not. For example, it's TFF here, but avisynth assumes BFF otherwise - but it didn't help with the problems in this case) . There is a duplicate in the same "cockpit up" section (along with combing), there might be more duplicates but I didn't check closely

So this ffms2_r1140+101-avs+vsp is significantly worse than directshowsource with ffdshow . That latter option would be second best to the DG options, with only minor aliasing/combing.

Cheers

[manolito]

Don't you think you should make available your source, the title00 (1).mkv?

pdr sent me the link via PM, see here:
https://forum.doom9.org/showthread.php?p=1850845#post1850845


@ poisondeathray
When using TIVTC, you should set the field order explicitly if not using DG source filters (DG stuff passes the info, others usually do not. For example, it's TFF here, but avisynth assumes BFF otherwise - but it didn't help with the problems in this case)

I did have "AssumeTFF()" right before the TFM call in my script. TFM defaults to "order=-1", which means it takes the AviSynth field order. So my "AssumeTFF()" should have taken care of it.


@ videoh
The original implementation was by jackei of DVD2AVI fame, by the way, and DG enhanced it for robust random access. So there's no need to succumb to DG derangement syndrome, at least in this case.

jackei was a little bit before my time at doom9. And I don't think that I showed any symptoms of the "DG derangement syndrome" here, when I said that obviously noone except you was able to write a working source filter for such sources, then I was merely expressing my appreciation for your work.
Anything else would certainly have minimized our chances for this:
Maybe it's time to add MKV support to DGIndex/MPEG2Source.


Cheers
manolito

[videoh]

jackei was a little bit before my time at doom9. And I don't think that I showed any symptoms of the "DG derangement syndrome" here, when I said that obviously noone except you was able to write a working source filter for such sources, then I was merely expressing my appreciation for your work. Thank you for your reply. The remark wasn't directed at you but rather a little dig at the cabal (they know who they are -- Fiona's ex flying monkeys) that delight in trolling and provoking me. If I was as bad a coder as they would have you believe, probably I wouldn't have gotten the pulldown working correctly in DGIndex, and I wouldn't have 10,000 users of DGDecNV, etc.

Just gonna finish a few things with my CUDASynth experiment, then I'll get started on MKV for DGMPGDec. Stay well.

[manolito]

:thanks:

[videoh]

You're welcome. Starting on MKV tomorrow...

[FranceBB]

I wouldn't have 10,000 users of DGDecNV

Including one of my colleagues that uses it at work.

Starting on MKV tomorrow...

About that...
This is really appreciated, but since you are willing to develop DGIndex (and DGDecNV) even further, it would be really useful for professional usage to have a proper support for XDCAM-HD422 files in the future (Container: .mxf Video: MPEG-2 50 Mbit/s 4:2:2 planar (yv16) 8bit, closed GOP M=3 N=12 Audio: CH.1-2 DolbyE 5.1 CH.3-4 DolbyE 5.1 CH.5-6 PCM Stereo Downmix CH.7-8 PCM Stereo Downmix + Ancillary data like timecode).

[manolito]

Sorry it might look a little bit like I must always have the last word, but I cannot have this post
https://forum.doom9.org/showthread.php?p=1851171#post1851171
stand unchallenged.

Here is the converted clip using the "gold standard"
https://we.tl/t-o57QJNLXz2

Using this AVS script:
Import("C:\Program Files (x86)\AVStoDVD\Lib\A2DSource.avsi")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\DGIndex\DGDecode.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Video = MPEG2Source("D:\title00 (1)_1.d2v")
Audio = A2DAudioSource("C:\Download\title00 (1)_2_EN.ac3", CacheFolder="D:")

Video = Video.ConvertToYV12(interlaced=true)
Video = Video.TFM(d2v="D:\title00 (1)_1.d2v").TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)
Prefetch(4)


And this is the result using ffms2 with the "rffmode=1" parameter:
https://we.tl/t-beNb9tWTPx

Using the following script:
LoadCPlugin("C:\Program Files (x86)\AVStoDVD\Lib\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Audio = FFAudioSource("C:\Download\title00 (1).mkv", track=-1)
Video = FFVideoSource("C:\Download\title00 (1).mkv", track=-1, rffmode=1, seekmode=0)

Video = Video.ConvertToYV12()
Video = Video.AssumeTFF()
Video = Video.TFM().TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)


Now please compare the results and tell me if you can detect any differences in quality...

Do not test by stepping through frames, play the clip in a player software instead. PDR insisted that the ffms2 output showed more aliasing and combed frames, I could not confirm this. I prefer using real world scenarios, and here field matching is never 100% accurate, so there is a reason that TDecimate uses postprocessing by default.


I showed the two clips to a friend who works for a German public broadcaster in a senior position. He is not involved in the technical aspects, he manges he content side. But he certainly has a very trained eye when it comes to video content. I showed both clips to him, he could not see any quality differences, and he would have approved both clips for broadcasting right away.


Cheers
manoito

[hello_hello]

I haven't read the whole thread thoroughly, or looked at manolito's samples yet because they're downloading at dialup speed, but I have a question.....

Has anyone been using the ffms2 frame rate conversion (fpsnum/fpsden) and the rffmode=1 option interchangeably? It's just that I recall a mention of combing problems.

rffmode=1 enables CFR mode itself. If there's no repeat field flags it'll output an error when rffmode=1, otherwise it should output 29.970fps. No need for frame rate conversion too. In fact the way I remember it, frame rate conversion doesn't work when rffmode=1, but that could be wrong.

And when it's a job for rffmode=1, you can't use frame rate conversion instead, because I'm pretty sure ffms2 outputs the average frame rate by default (when there's both soft and hard telecined sections), so it can mess with the audio sync and I doubt frame rate conversion will help. Plus if you used frame rate conversion for a source that had some hard telecined sections and set it to 23.976fps, I assume ffms2 will drop frames in the 29.970fps sections as required without giving a thought to field matching. Although what probably happens is.... ffms2 starts off at the average frame rate and drops frames from the entire video to get the frame rate down to 23.976fps, so without rffmode=1 it'd be dropping frames from the soft telecined sections too..... I think.

Anyone know why rffmode=1 isn't the default for mpeg2, at least? I've sometimes wondered if it should be.

[manolito]

From my (limited) experience "rffmode=1" is pretty much the same as "Honor Pulldown Flags" in DGIndex. As opposed to DGIndex it will throw an error if the source has no pulldown flags, and it cannot do tricks like detect and fix field order transitions. To IVTC the output you can use the normal "TFM().TDecimate()", but there is no way to teach TFM about the pulldown flags like "TFM(d2v="my d2v file"). So it is not as reliable as DGIndex / DGDecode, but for sources which are less than 100% film it works pretty well for me.

Cheers
manolito

[hello_hello]

Something in the help file wording doesn't seem quite right to me.

rffmode

- **0**: Ignore all flags (the default mode).
- **1**: Honor all pulldown flags.
- **2**: Equivalent to DVD2AVI's "force film" mode.

Also note that "force film" is mostly useless and only here for completeness' sake, since if your source really is safe to force film on, using mode 0 will have the exact same effect while being considerably more efficient.

I wonder what the definition of "really is safe to force film" is, because if a source is mostly soft-telecined, with just a small section or two hard telecined, won't modes 0 and 2 output different frame rates? The average frame rate for mode=0 and, I assume 23.976fps for mode 2.

The same would happen for soft telecined sources with small sections of interlaced video. I understand "force film" to mean "the studio promo at the beginning is interlaced and I don't care if it's converted to 23.976fps by dropping frames", or something like that, but wouldn't mode=0 output the wrong (average) frame rate in that situation?

Although while I was typing it occurred to me it's possibly better to output a timecodes file instead and give it to x264 to use for VFR encoding. And now I've thought about it, ffms2 no doubt writes the timecodes file while it's indexing, whereas I've been indexing with DGIndex, running a first pass with TIVTC to create the timecodes file, then a second pass for encoding. Now I'm wondering why I'd use DGIndex in preference to ffms2. :)

Sorry manolito. I realise you do a lot of encoding for DVD and VFR is no good for you..... I was thinking as I was typing..... but....

Is there a tool for applying pulldown to existing video? Well... I'm aware of DGPulldown although I've never used it, but it appears to only convert from one constant frame rate to another. What if it could accept a timecodes file to determine which fields to repeat, so then you could use ffms2 mode=0, output a timecodes file, apply the pulldown after encoding and bypass having to convert the frame rate and applying IVTC etc.

It sounds like a good idea in my head....

[poisondeathray]

And this is the result using ffms2 with the "rffmode=1" parameter:
https://we.tl/t-beNb9tWTPx

Using the following script:

LoadCPlugin("C:\Program Files (x86)\AVStoDVD\Lib\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Audio = FFAudioSource("C:\Download\title00 (1).mkv", track=-1)
Video = FFVideoSource("C:\Download\title00 (1).mkv", track=-1, rffmode=1, seekmode=0)

Video = Video.ConvertToYV12()
Video = Video.AssumeTFF()
Video = Video.TFM().TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)



How is that any different than the last one in post #60 ? The script is the same. Was the .dll different ? Do I really have to download it?

The poor results were already confirmed and I reproduced the issues locally with the same .dll. Some sections had ~ 30% combing, and that' s with post processing enabled (PP wasn't set to 0 in that script) - so those passed through in your encode (TFM didn't detect, didn't apply post processing on them).


Do not test by stepping through frames, play the clip in a player software instead. PDR insisted that the ffms2 output showed more aliasing and combed frames, I could not confirm this. I prefer using real world scenarios, and here field matching is never 100% accurate, so there is a reason that TDecimate uses postprocessing by default.

I showed the two clips to a friend who works for a German public broadcaster in a senior position. He is not involved in the technical aspects, he manges he content side. But he certainly has a very trained eye when it comes to video content. I showed both clips to him, he could not see any quality differences, and he would have approved both clips for broadcasting right away.



Don't tell me how to test it. Step through it and see the combing issues. Some people have terrible eyesight or cannot see these issues. If you blink, you might miss some sections.

Maybe your method of viewing is flawed ? Maybe your player is deinterlacing ?

Just because you or your so called expert can't see it , does not mean it's not there. Some sections had ~ 30% combing! It's really obvious. Usually content managers don't do this low level QC stuff. But people in senior positions are ultimately responsible. If that' s how he does QC and he misses stuff this obvious - he really should be fired .

We are testing the effectiveness of a source filter +/- field matching ... not post processing. We just want the original progressive frames back. You don't add x,y,z variables to convolute your results. You don't add lossy encoding on top of it. That's why you look at PNG images not JPG. Why not resize it to UHD? Why not use pointresize or bicubicresize ? Why don't you just add QTGMC or some deinterlacer or denoiser afterwards ? Maybe vinverse? Answer to the rhetorical questions : Because other variables can pollute your results and it's not appropriate for what you set out to test . When you are looking at this , why are you even converting to PAL and resizing ? I know you're doing it for a specific scenario, but it just adds other variables . You want to strip out all the other extraneous variables to test exactly what you set out to test - It's called the "scientific method."

I seriously can't tell if you're pulling my leg. The combing and aliasing are so obvious, even in a media player. Some sections are fairly clean with only a few frames combed, but that spaceship coming in section has ~30% bad frames. Even if you had just forced film without field match , or used directshow , you only have <0.5% combing total! Recall the film% in the log

[Groucho2004]

Maybe your method of viewing is flawed ? Maybe your player is deinterlacing ?Or the decoder could be de-interlacing. LAV filters have de-interlace options and also ffdshow if I recall correctly.

[poisondeathray]

Or the decoder could be de-interlacing. LAV filters have de-interlace options and also ffdshow if I recall correctly.

I asked about that earlier, he said it was disabled for LAV when doing the directshow testing .

Presumably he didn't enable it for playback if he was using a directshow based media player

[StainlessS]

In ExBlend thread, one user was wanting to deblend a clip whose blending originated in DirectShow (doing some kind of de-interlacing),
easiest way was to not use DirectShow, blending gone.

[Groucho2004]

I asked about that earlier, he said it was disabled for LAV when doing the directshow testing .Ah, OK. I did not read the entire thread.

[hello_hello]

manolito, I've only just started downloading the "gold standard" sample so I haven't compared the two yet, but the ffms2_rffmode_1 sample definitely has lots of combing. Frame #14631.

https://s33.postimg.cc/bn435p1cv/ffms2_rffmode_1.mpg_snapshot.jpg

[hello_hello]

I don't understand the "why" but I think I discovered the "what". I tested with poisondeathray's sample (he PM'd me a link) after remuxing the MKV as a TS file.

After opening the source with both DGIndex and ffms2, Info() shows TFF for DGIndex and assumed TFF for FFMS2.

I tried the following and it was obvious DGIndex and FFMS2 were repeating different fields.

FFVideoSource("E:\title00 (1).mkv", threads=1, rffmode=1)
SeparateFields()
A=last
B=A.SelectOdd().Subtitle("ffms2 odd")
C=A.SelectEven().Subtitle("ffms2 even")
mpeg2source("D:\title00 (1).d2v")
SeparateFields()
D=last
E=D.SelectOdd().Subtitle("DGIndex odd")
F=D.SelectEven().Subtitle("DGIndex even")
G=StackVertical(B,E)
H=StackVertical(C,F)
StackHorizontal(G,H)

AssumeBFF() for FFMS2 makes it repeat the same fields as DGIndex
Or instead of AssumeBFF(), you can tell TFM to Assume BFF and the result is the same after IVTC.

Best as I can tell using the ShowDiff function I borrowed from jagabo at VideoHelp, aside from what might be some decoding differences (which I've decided to call deblocking for the moment) the following outputs the same video.

FFVideoSource("E:\title00 (1).mkv", threads=1, rffmode=1)
AssumeBFF()
TFM().TDecimate()
A=last
mpeg2source("D:\title00 (1).d2v")
TFM().TDecimate()
B=last
ShowDiff(A,B)

As does this:

FFVideoSource("E:\title00 (1).mkv", threads=1, rffmode=1)
TFM(order=0).TDecimate()
A=last
mpeg2source("D:\title00 (1).d2v")
TFM().TDecimate()
B=last
ShowDiff(A,B)

Frame #14654 courtesy of ShowDiff() without and with AssumeBFF.

https://s33.postimg.cc/nw9saj0xb/title00_1_a.jpg

https://s33.postimg.cc/bueegdjz3/title00_1_b.jpg

function ShowDiff(clip Clip1, clip Clip2, bool "Amp", bool "Show", bool "Comp")
{
Amp = default(Amp, false)
Show = default(Show, false)
Comp = default(Comp, false)

N1 = Clip1.levels(96, 1.0, 160, 96, 160).greyscale()
N2 = Clip1.subtract(Clip2)
N3 = (Amp) ? N2.levels(124, 1.0, 131, 0, 255) : N2
N4 = (Show) ? N3.merge(N1) : N3
N5 = (Comp) ? Clip1.compare(Clip2, show_graph=true) : N4

return N5
}

[poisondeathray]

Nice detective work hello_hello . I can confirm it works. And the field order really is BFF with that ffms2 CPlugin version if you check with separatefields(). In hindsight we should have double checked and gone back to basics

Not sure about the minor decoding differences but those are negligible . I would call that an alternative working solution, at least on this disc .

Now why don't the other ffms2 versions work ?

I guess l33tmeatwad's version is technically the "newest" , although it's not on the official github page. But it works .

ffms2_r1140+101-avs+vsp => ok
FFMS2_2.32.1_MSVC l33tmeatwad => ok
ffms2-2.23.1-msvc => wrong framecount/sync issue
ffms2-2.23-clang => wrong framecount/sync issue
ffms2000-test8 => ok

[manolito]

Wow, this is getting exciting... :D

I can confirm that using a working ffms2 version and specifying BFF results in a perfect output stream (with rffmode=1). Nice...

I did stick with the MKV container as the source. When I extracted the video stream from the MKV and loaded it into DGIndex the info was that this stream was TFF.

What does this mean? When using ffms2 with rffmode=1 do I have to reverse the field order which is reported by DGIndex, or does ffms2 always expect BFF? I would like to get a universal solution which works with each and every MKV source created with MakeMKV. Unfortunately I do not own any NTSC DVDs with progressive 23.976 soft pulldowned content to test this...


Cheers
manolito

[hello_hello]

Now why don't the other ffms2 versions work ?

I guess l33tmeatwad's version is technically the "newest" , although it's not on the official github page. But it works .

ffms2_r1140+101-avs+vsp => ok
FFMS2_2.32.1_MSVC l33tmeatwad => ok
ffms2-2.23.1-msvc => wrong framecount/sync issue
ffms2-2.23-clang => wrong framecount/sync issue
ffms2000-test8 => ok

I wasn't clever enough to think about ffms2 versions and now I've looked I'm not 100% sure which one I was using.

MeGUI loads ffms2 from it's tools folder, and it's some flavour of version 2.23.1, but all I know for sure is the dll is dated 2016/12/29, the copying text file still says version 2.22 and the link within is this one. https://github./FFMS/ffms2

The Avisynth auto-loading folder has a dll dated 2017/05/24. I normally make sure any dlls MeGUI might load and the versions in the auto-loading folder are the same, but I think I put ffms2000 in the auto-loading folder for testing and forgot about it.

Fortunately, a quick check shows they both output the same number of frames for the sample when rffmode=1 (98912). The index files they create aren't the same size.

I'm pretty sure I left MeGUI's load plugin line in the script so I was probably testing with ffms 2.23.1. I'm just not sure which flavour it is.

I did stick with the MKV container as the source.

I did the same for ffms2. I only remuxed as a TS file for DGIndex.

[hello_hello]

What does this mean? When using ffms2 with rffmode=1 do I have to reverse the field order which is reported by DGIndex, or does ffms2 always expect BFF? I would like to get a universal solution which works with each and every MKV source created with MakeMKV. Unfortunately I do not own any NTSC DVDs with progressive 23.976 soft pulldowned content to test this...

I had one thought, although it's a long shot.....
MKV has an element for specifying field order. Unfortunately I've deleted poisondeathray's source file, but if someone still has it, it might be worth checking to see if the element it present and if it's correct. You can check with the MKVToolNix Header Editor.

None of the MKVs I checked contained the element, and after remuxing a vob file with MKVToolNix it wasn't automatically created, but I'm not sure which program was used to create poisondeathray's MKV.

Chances are FFMS2 doesn't look for that element even if it exists, but it's easy enough to eliminate it as the cause.

[manolito]

Good idea, but unfortunately no luck...

The MKV was created with MakeMKV, and the latest MKVToolNix says that there is no field order element in the header.

Neither MediaInfo (in debug mode) nor MKVInfo give any information about the field order. (I did not try ffprobe so far)

When you extract the video track from the MKV and feed it to DGIndex then you will get the info "TFF" when you play or preview the track. But when you create the D2V file and the log then the field order row is empty. To be absolutely certain you probably have to use the old AssumeXFF().SeparateFields() method and step through the fields.

What I found out in the meantime is that ffms2 with the rffmode=1 parameter outputs the opposite field order compared to MPEG2Source (with Honor Pulldown Flags). I tested this by slowing down a progressive PAL clip to 23.976 and then manually adding pulldown flags with DGPulldown. Here you can specify the field order, and when I selected TFF then ffms2 (with rffmode=1) delivered BFF and vice versa. Weird... I wonder what happens with a source which has field order transitions.


Cheers
manolito