Source filter behavior for progressive NTSC @23.976 with pulldown flags

[manolito]

Please forgive me if this question is stupid or has been discussed before, but I live in PAL land... :o

My source is a progressive NTSC clip @23.976 fps with standard 3:2 pulldown flags. I want to convert this clip to PAL. My arsenal of AviSynth source filters consists of DGIndex/DGDecode, DSS2Mod and DirectShowSource (LSmash does not run on my old computer).

Now I found that all of these source filters treat this source differently:
1: Mpeg2Source has "Honor Pulldown Flags" set by default. It correctly decodes to (pseudo-) interlaced 29.97 output. If I want the original 23.976 progressive output then I have to change the field operation to "Force Film".

2: FFMpegSource does not honor pulldown flags by default (There is an RFFMode option which can change this). It decodes to the original progressive 23.976 output.

I have no problem with the behavior of these two source filters, but DirectShowSource and also DSS2Mod do it differently.

3: On the first look DSS2Mod (DirectShowSource has the same behavior) seems to honor the pulldown flags, it decodes to 29.97. But unlike Mpeg2Source the output shows no combing. It looks progressive, but with tons of duplicate frames. So in reality the pulldown flags are not honored at all. This is not what anybody would want. To get the original 23.976 output without dupes I need to add "fps=23.976" to the source filter call.

Is there a way to bring the DSS2Mod behavior in line with the other source filters? Any (hidden) option to either honor pulldown flags or to force film?


Cheers
manolito

[manono]

Just use MPEG2Source after having created the D2V using DGIndex. Not what you asked, but it's the best way to handle MPG/VOB sources.

"My source is a progressive NTSC clip @23.976 fps with standard 3:2 pulldown flags."

If it's already 100% film (check the bottom of the D2V file), just make the D2V using the 'Forced Film' option and your video is already 23.976fps. An easy edit of the D2V will turn it into a Forced Film D2V. If it's less than 100% film but still not pure video, make the D2V using the default "Honor Pulldown Flags" and use TIVTC like so:

TFM(D2V="Movie.d2v")
TDecimate()

The effect will be (pretty much) to treat the soft pulldown parts as if they had been Forced Film, and to IVTC the rest.

Again, not what you asked but why try and force an inferior source filter?

[manolito]

Thanks manono,

this is how I already do it most of the time. Mpeg2Source has the additional benefit that it can automatically detect and repair field order transitions.

My question was mainly for AVStoDVD. Here users can choose between DSS2Mod, FFmpegSource and Mpeg2Source, and it would be very nice if it was possible to treat all these source filters identically without ruining the result. And most AVStoDVD users are not experienced, they cannot be expected to be able to change DGIndex options depending on the source.

MrC did change the procedure recently (after having to listen to my nagging way too long), and the current behavior is like this:

MediaInfo reports 23.976 progressive and 3:2 pulldown for such sources. AVStoDVD now evaluates the pulldown report and sets the source frame rate to 29.97 whenever pulldown is reported. Additionally it adds IVTC to the AVS script (TFM().TDecimate() ) for such sources.

This is correct for Mpeg2Source (Honor Pulldown Flags is the default). But for FFmpegSource and DSS2Mod the decoded result has no repeated fields, it is progressive with lots of dupes. But still applying IVTC seems to work nicely. TFM won't find any fields to match, but TDecimate reliably removes all the dupes so the end result is the original 23.976 progressive clip. Not very elegant, but the same procedure works for all the possible source filters.


Cheers
manolito

[manono]

I didn't understand much of that and have never used AvsToDVD. But if you're looking for a one-size-fits-all approach to MPG/DVD sources, then the script I showed earlier works well. Just make all D2Vs using Honor Pulldown Flags followed by:

TFM(D2V="Movie.d2v")
TDecimate()

You won't have the mistakes sometimes made by the field matching when IVTCing everything and, if it's all soft pulldown, it's nearly (not quite) as fast as using Forced Film to make the D2V to begin with.

If you're looking for a one-size-fits-all approach for all sources, then I don't know what to tell you and maybe someone else can answer your questions.

[FranceBB]

My source is a progressive NTSC clip @23.976 fps with standard 3:2 pulldown flags. I want to convert this clip to PAL.

I have a sample which is 23.976 with 3:2 pulldown in my network drive to play with.
This is what I would do to convert it to PAL:

#Index video and audio 29.970fps progressive with dups
video=DGDecode_MPEG2Source("I:\Production\RAW\test.d2v")
audio=FFAudioSource("I:\Production\RAW\audio T80 2_0ch 224Kbps DELAY 0ms.ac3")
AudioDub(video, audio)

#Decimate to original 23.976 progressive
tfm(mode=1,pp=5,slow=2,micmatching=2,clip2=tdeint(mode=2,type=3))
Tdecimate(mode=2, rate=23.976)

#Resize from 720x480 to 720x576 using NNEDI
nnedi3_rpow2(cshift="Spline64ResizeMT", rfactor=2, fwidth=720, fheight=576, nsize=4, nns=4, qual=1, etype=0, pscrn=2, threads=0, csresize=true, mpeg2=true, threads_rs=0, logicalCores_rs=true, MaxPhysCore_rs=true, SetAffinity_rs=false, opt=3)

#Blending to 50fps and interlacing to 25i
ConvertFPS(50)
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()

Some people just don't like blending.
I don't think it's actually that bad, especially in motion and I do it all the time at work, but if you prefer a different approach, you can just do a speed up with pitch adjustment and encode the progressive 25fps file as interlaced:

#Speed up 4% with pitch adjustment
ResampleAudio(48000)
AssumeFPS(25, 1, true)
SSRC(48000)

As to the DirectShowSource behaviour, I rarely use it. I mainly use FFMpegSource2 for general purpose contents and DGIndex for old MPEG-2 files.

[Cary Knoop]

Some people just don't like blending.
I don't think it's actually that bad, especially in motion and I do it all the time at work, but if you prefer a different approach, you can just do a speed up with pitch adjustment and encode the progressive 25fps file as interlaced:

I am one of them (who does not like blending).

Also, it is uncommon in a professional setting, 24p (or 23.976p) is always speed converted to 25p.
If it isn't I personally would call it a sloppy job.

[FranceBB]

23.976p is always speed converted to 25p.
If it isn't I personally would call it a sloppy job.

If the content is a tv series or a movie that needs to be aired from start to end on a linear channel, then yes, absolutely, I do speed up with pitch adjustment, but whenever I have to use a few scenes of a series/movie to cover something in the news, I just blend (due to people saying "chop chop", journalists pushing and me being lazy).

[manolito]

TFM(D2V="Movie.d2v")
TDecimate()


Thanks manono for bringing up the d2v parameter for TFM. I will definitely lobby MrC to add it to AVStoDVD.

Otherwise AVStoDVD tries to be as universal as possible, and DGIndex / DGDecode cannot always be used for sources with pulldown flags. An example is a NTSC DVD title. The recommended procedure is to rip the title to MKV using MakeMKV, because this way all the chapters and subtitles are retained. When this MKV is fed to AVStoDVD only DSS2Mod or FFmpegSource can be used as source filters.


@ FranceBB
Thanks for the suggestions. I believe though that you really should insert "TFM(d2v="{your D2V file}") before the TDecimate call, because your source will have repeated fields (with the default "Honor Pulldown Flags").


A little OT, but just the other day I discovered an old thread at VideoHelp with an alternative method to convert progressive 23.976 to progressive 25. This comes from gavino, it gives perfect results, no speedup required, the slight blending is almost invisible. It looks like this:
f1=ChangeFPS(24)
f2=Trim(0,-1).AssumeFPS(24) + Trim(1,0).ChangeFPS(24)
Film=Merge(f1,f2)
p1=ChangeFPS(Film,25)
p2=Trim(Film,0,-1).AssumeFPS(25) + Trim(Film,1,0).ChangeFPS(25)
Pal=Merge(p1,p2)
Return Pal

The original thread is here:
https://forum.videohelp.com/threads/322430-Frame-rate-conversions-How-to-blend-select-frames-in-AVISynth

I prefer this method over using 23.976 -> 25 pulldown because the output stays progressive, audio does not need to be touched, and I do not get motion judder. The problem with PAL Speedup is that pitch correction only sounds good when using highest quality commercial tools (like iZotope RX). The AviSynth TimeStretch plugin can sound awful for music.


Cheers
manolito

[Cary Knoop]

The problem with PAL Speedup is that pitch correction only sounds good when using highest quality commercial tools (like iZotope RX).
I think that is nonsense.

Use SoX, which is free.

sox <in> <out> tempo 1.0427

Does the job just fine.

http://sox.sourceforge.net/

[manolito]

Have you tried this with music content? With string layers? Didn't you notice the flanging effect?
Why do you think that the pros spend big bucks for pitch changing tools?

[wonkey_monkey]

Also, it is uncommon in a professional setting, 24p (or 23.976p) is always speed converted to 25p.
If it isn't I personally would call it a sloppy job.

Torchwood: Miracle Day as shown on the BBC is the only example I can think of. I can only imagine it's because they either didn't want to change the running time (by a whole 2.5 minutes) or didn't want to pitch shift the voices, which everyone would be too familiar with as characters from a previously 25p production.

[Cary Knoop]

Have you tried this with music content? With string layers? Didn't you notice the flanging effect?
Why do you think that the pros spend big bucks for pitch changing tools?

Here is a sound clip and a time stretched Sample-A and Sample-B.

https://www.dropbox.com/sh/2hofgu1nnff3cgv/AAC7XG2AZbiU02l0GPIj1ay0a?dl=0

Can you hear which one is the Rx6 and which one is the SoX? And do you think any of them is inferior?

[Cary Knoop]

Torchwood: Miracle Day as shown on the BBC is the only example I can think of. I can only imagine it's because they either didn't want to change the running time (by a whole 2.5 minutes) or didn't want to pitch shift the voices, which everyone would be too familiar with as characters from a previously 25p production.
You can avoid shifting the pitch.

[wonkey_monkey]

You can avoid shifting the pitch.

You can, but the BBC still typically don't.

[Cary Knoop]

You can, but they still typically don't.
They typically do!

Pitch shifting is far more noticeable than time stretching.

[Cary Knoop]

Torchwood: Miracle Day as shown on the BBC is the only example I can think of. I can only imagine it's because they either didn't want to change the running time (by a whole 2.5 minutes) or didn't want to pitch shift the voices, which everyone would be too familiar with as characters from a previously 25p production.
According to IMDB Torchwood was recorded in 25p.
What is it you think they have not done?

https://www.imdb.com/title/tt0485301/technical?ref_=tt_dt_spec

[poisondeathray]

(And you can probably avoid doing anything ; 99.999% of PAL DVD players play NTSC discs just fine)

DSS2 /mod or any directshow derivative will be at risk of whatever the system has installed and whatever settings are set. If the directshow filter is set to deinterlace it will deinterlace, degrading the image . Maybe that's why there is no combing. Can you be sure it' s not actually doing other stuff ? Not really, unless you personally setup and configured your filters directly. If some general user doesn't know what's going on or anything about this stuff - I would make the smarter choices for them if I was making a GUI - and make choosing the bad options more difficult or impossible . DSS2mod tends to drop a frame at the end too . Really inconsistent . Avoid.

[wonkey_monkey]

According to IMDB Torchwood was recorded in 25p.
What is it you think they have not done?

https://www.imdb.com/title/tt0485301/technical?ref_=tt_dt_spec

Series 1 and 2, and the Children of Earth miniseries were all shot 25p in the UK as a UK production.

Miracle Day (series 4) was largely a US production shot at 23.976fps. When the BBC broadcast it, frames/fields were blended to make it 25p instead of doing the usual PAL speed-up. Audio was presumably untouched.

The BBC don't show many US imports these days (apart from films), but when they do they are usually PAL sped-up with audio at a higher pitch.

Channel 4, on the other hand, have in the past shown both Stargate and Enterprise with (not very good) pitch correction but I don't think they've used any correction lately with, for example, Agents of Shield.

[Cary Knoop]

Miracle Day (series 4) was largely a US production shot at 23.976fps. When the BBC broadcast it, frames/fields were blended to make it 25p instead of doing the usual PAL speed-up.

While it appears they blend frames they prefer others not to do it:

"Speed change is the preferred method of converting from 24fps (including 23.976fps) to 25fps. Due attention must be given to the audio. "

Source: http://dpp-assets.s3.amazonaws.com/wp-content/uploads/specs/bbc/TechnicalDeliveryStandardsBBCFile.pdf (Page 9).

The BBC don't show many US imports these days (apart from films), but when they do they are usually PAL sped-up with audio at a higher pitch.

I assume you know this for a fact (that the pitch is higher).
Then shame on the BBC, when you speed up the audio I think you should really compensate for the pitch shift.

[wonkey_monkey]

But it does cause artefacts, no matter how good the resampling. And decent resampling hasn't been around all that long, so plain speed-up has been the de facto standard for years, and no-one really notices anyway. For someone like the BBC it's probably far preferable to stick with a simple, guaranteed clean method that everyone can be told to stick to than mess around and get it wrong like Channel 4 has been known to do - both Stargate and Enterprise never sounded great, and they once played out a film with a horribly mangled soundtrack because something had gone wrong with pitch-shifting somewhere along the line.

[Cary Knoop]

But it does cause artefacts, no matter how good the resampling. And decent resampling hasn't been around all that long, so plain speed-up has been the de facto standard for years, and no-one really notices anyway. For someone like the BBC it's probably far preferable to stick with a simple, guaranteed clean method that everyone can be told to stick to than mess around and get it wrong like Channel 4 has been known to do - both Stargate and Enterprise never sounded great, and they once played out a film with a horribly mangled soundtrack because something had gone wrong with pitch-shifting somewhere along the line.
A 4% pitch shift is definitely noticeable while those suggested artifacts, well do you hear any artifacts in the comparison below?
https://forum.doom9.org/showthread.php?p=1850449#post1850449

[manolito]

You really cheated on this one. Your source file is not demanding at all, just about any pitch correction software handles this one.

What about this source?
It is the intro from the Yellowjackets Greenhouse track. Ripped from the original CD (no intermediate MP3). One of the most excellent analog recordings ever by Jan Erik Kongshaug.
Download here:
https://www.zeta-uploader.com/1143320492

I do not own iZotope RX, the two conversions are by SoX and the integrated TimeStretch plugin from AviSynth 2.61 Alpha.

For the first 30 seconds they are tolerable but after that listening becomes really painful for both pitch corrected versions. Absolutely unusable.


Cheers
manolito

[Cary Knoop]

You really cheated on this one.
If that is the way you think you can have a respectful discussion you have it wrong.

[manolito]

(
DSS2 /mod or any directshow derivative will be at risk of whatever the system has installed and whatever settings are set. If the directshow filter is set to deinterlace it will deinterlace, degrading the image . Maybe that's why there is no combing. Can you be sure it' s not actually doing other stuff ? Not really, unless you personally setup and configured your filters directly. If some general user doesn't know what's going on or anything about this stuff - I would make the smarter choices for them if I was making a GUI - and make choosing the bad options more difficult or impossible . DSS2mod tends to drop a frame at the end too . Really inconsistent . Avoid.

This may be your personal experience - mine is totally different... :devil:

I use DSS2Mod together with LAV Filters. I set it up using all the default settings, so there is no deinterlacing whatsoever. And still the 23.976 clip with pulldown flags gets decoded to 29.97, but not with the repeated fields, instead it did ignore the RFF flags and duplicated frames to reach the 29.97 frame rate. I have no idea if this would be different when using ffdshow instead of LAV Filters, but ffdshow is not an option for me.

And your advice to avoid DSS2Mod because it might skip a frame at the end or maybe at the beginning is totally misguided. Nobody cares about a skipped frame at the end as long as there are no decoding artifacts and no audio sync problems. And from my experience DSS2Mod (with a reasonable Prefetch value) is much more reliable than ffms2.


Cheers
manolito

[manolito]

If that is the way you think you can have a respectful discussion you have it wrong.

You started being disrespectful by telling me that my experience with free pitch correction tools was nonsense.
https://forum.doom9.org/showthread.php?p=1850424#post1850424

[FranceBB]

believe though that you really should insert TFM before the TDecimate call, because your source will have repeated fields (with the default "Honor Pulldown Flags").

True. Fixed:

tfm(mode=1,pp=5,slow=2,micmatching=2,clip2=tdeint(mode=2,type=3))

I should stop replying at 2am in the morning xD

just the other day I discovered an old thread at VideoHelp with an alternative method to convert progressive 23.976 to progressive 25. This comes from gavino, it gives perfect results, no speedup required, the slight blending is almost invisible.

If I understand correctly, that's because it actually blends 1 frame from 23.976 (24) to 25, making a 25fps progressive.
That's a different approach, but it can be done, it's not a "big deal".
Ideally, whenever I have to blend, I always blend to 50fps and then divide in fields to get a truly interlaced 25i.
Blending just 1 frame every second actually looks... kinda... weird to me.
I don't know how to explain that, but it's like... you get a single frame with something that "doesn't feel right" and my mind seems to be focussed on that.
When I blend to 50 and divide in fields, though, it's like having a truly interlaced 25i content and since there are "many" blended frames, my mind doesn't notice.
I don't know how to explain this, but anyway the method you found is a way to do it and if it looks good to you, go for it.

As to the speed-up with pitch adjustment, there are many ways to do it instead of using Avisynth; some are actually better for some contents, some are better for others, but it's still about compromises.
What I can say, however, it's that even with professional tools like the Sony DP600 I have at work, a speed up with pitch adjustment it's still gonna be noticeable if you compare it with the original version, but hey, there's nothing we can do about it.
The point is that it's always gonna be different, no matter what you use.
Sure, you can fine-tune it, but it's still gonna be noticeable.
Anyway, for many people, it's gonna be fine.
Honestly, in my whole career (it's not long, though xD) I've never received a complaint by a user at home that didn't like a speed-up.

So, in the end, it's all about compromises.
You still have to convert from a frame-rate to another and there's always going to be a catch:

- do you wanna use blending? It's gonna look a bit odd in some scene-changes.
- do you wanna use the speed-up + pitch adjustment? It's gonna be noticeable by someone and it's gonna sound slightly different.
- do you wanna use motion interpolation? It may introduce artifacts.
- do you wanna duplicate frames? It's gonna stutter.

In other words, it doesn't matter how much time you spend trying to figure out a perfect method, it's still gonna have some "issues", the point is trying to figure out which issue looks better for you. ;)

[poisondeathray]

This may be your personal experience - mine is totally different... :devil:


Says the guy with limited experience, living in PAL land asking for help with NTSC discs :D :devil:

It's not just my personal experience - those are the facts . Directshow is inconsistent. It's dependent on how the user has it configured. User A might have it configured differently that user B.


I use DSS2Mod together with LAV Filters. I set it up using all the default settings, so there is no deinterlacing whatsoever. And still the 23.976 clip with pulldown flags gets decoded to 29.97, but not with the repeated fields, instead it did ignore the RFF flags and duplicated frames to reach the 29.97 frame rate. I have no idea if this would be different when using ffdshow instead of LAV Filters, but ffdshow is not an option for me.

And did you check the final output closely with tdecimate or decimate ? :D because I bet you are in for a surprise. Check the motion

Sure you might have some knowlege of setting up directshow filters , so you might be predisposed to only some of the problems - but what about Joe Public? This is in the context that you brought up - ie. some general user using a GUI like avs2dvd. What if he had some filters activated? Things like deinterlacing, denoising, saturation, color correction? There are other directshow decoding filters too. In your small world you were only thinking of LAV and ffdshow, but there are others.

In LAV alone, you might have MPEG2 decoding set to HW CUVID, or quicksync, or DXVA, on CPU . They don't necessarily give the same results. The inconsistency is a deal breaker for many people . The only consistent thing about Directshow is it's consistently inconsistent :)



And your advice to avoid DSS2Mod because it might skip a frame at the end or maybe at the beginning is totally misguided.

But that was only one of the many reasons :) . That issue affects basically everything, even outside of the NTSC MPEG2 DVD soft telecine scenario - it frequently occurs in many other scenarios and video types


Nobody cares about a skipped frame at the end as long as there are no decoding artifacts and no audio sync problems.


That's about as ignorant as saying something like: people don't care about audio pitch shift issues as long as it's in sync. Sounds close enough right ?? :D :devil:

Maybe you don't care, but there are other people DO care about these problems. And I' m not just talking dropped frames, occasional green frames (rare, but usually only at the beginning or end), but the other potential issues as well.

How about jerky playback for one ? Frequently there jumps in motion when using dss2mod or any directshow derivative with tdecimate when used in this NTSC MPEG2/DVD soft telecine scenario. It just chooses the wrong frame so you get the incorrect duplicated frames sometimes, and dropped wrong frame others. This is not even talking about non linear seeking (regardless of preroll). This is complete frame by frame linear seeking, linear encode, no other filters. The encodes are jerky and botched. It's easy to reproduce, regardless of CPU/GPU HW directshow setting. Let me know if you want any samples to chew on :D But I bet the one you have now already shows the problems if you look closely




And from my experience DSS2Mod (with a reasonable Prefetch value) is much more reliable than ffms2.

Actually, neither are particularly reliable for soft telecine NTSC MPEG2 from DVD . ffms2 tends to have fewer jerky spots, but still there are sections that are jerky. I really suggest you look closer. MKV container actually makes it worse for some issues. MPEG2 in MKV is known a bad combo prone to issues

You have a proven, reliable and consistent method with DGIndex. Instead you suggest using buggy, unreliable methods? I don't see the point. It's one thing to make flawed stuff for yourself, but to suggest others to do so too is just wrong.

Cheers

[wonkey_monkey]

A 4% pitch shift is definitely noticeable while those suggested artifacts, well do you hear any artifacts in the comparison below?

A 4% pitch shift is not noticeable if you're not doing an A/B test - okay, if you're very familiar with the speaker, it might be, but usually not. Just as the 4% change in video speed isn't noticeable.

And it's guaranteed to work perfectly every time, with every input, which are extremely valuable attributes to have.

[SeeMoreDigital]

For anyone interested...

I've used both TSmuxer GUI and UsEac3to to remove 3:2 pull-down from all of my old 'movie' NTSC DVD sources and some Blu-ray disc sources.

UsEac3to is also pretty good at changing the speed of audio streams too ;)

[Sharc]

A 4% pitch shift is not noticeable if you're not doing an A/B test ….
Hmm …. how about people who have "absolute pitch"?
https://en.wikipedia.org/wiki/Absolute_pitch

[Katie Boundary]

If it's less than 100% film but still not pure video, make the D2V using the default "Honor Pulldown Flags" and use TIVTC like so:

TFM(D2V="Movie.d2v")
TDecimate()

If the content was originally 100.00% film, and the small amounts of non-film content are the result of an incorrectly applied pulldown pattern, then I'd specify this:

TFM(mode=0,pp=0,d2v="movie.d2v",micmatching=0).TDecimate()

It will improve accuracy (negligibly) and save a few CPU cycles.

blah blah speedup, blah blah pitch shifting, blah blah blending

Or you could rescale to 720x576, separate fields, duplicate one field from every 24th frame, rearrange the fields into a sensible order using Selectevery, and weave them back together. Example:

assumefps(24).spline144resize(720,576).separatefields()

selectevery(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,22,25,24,27,26,29,28,31,30,33,32,35,34,37,36,39,38,41,40,43,42,45,44,47,46,47)

Weave()

[manolito]

Thanks everybody, but my goal is not to apply manual tweaks to achieve optimal results. Instead I need a method which works under a One-click-Software like AVStoDVD for all the available source filters (MPEG2Source, DSS2Mod, ffms2 and DirectShowSource).

I think I know quite well how to do this for sane sources which have a regular pulldown pattern.

From the DGIndex manual for the "Ignore Pulldown" option:
However, because repeated fields intended for display are ignored and not displayed, the resulting frame rate may differ from the source frame rate. It may even vary throughout the clip, due to irregular patterns of pulldown flags. If the pulldown is irregular, use of this option will cause the audio-video sync to change at different parts of the clip, and most likely sync will not be acceptable

What do I do with sources like these? I do not have any such source files with irregular pulldown patterns to test. So if anyone could upload such a source I would be really grateful...


Cheers
manolito

[mp3dom]

You need to use TFM specifying the d2v parameter. TFM will take the rff indication inside the d2v file and will field-match the other parts to still match the frame rate.
Irregular pulldown is quite common if the ivtc is made by the encoder, because it's conservative, so in case of any doubt, it will simply encode the frame as is without using rff flags.

[manolito]

Yes, this is the preferred and safest method, but more than often you don't have a d2v file because the source is in a different container. For converting DVD titles the easiest way is to use MakeMKV which removes copy protection, repacks the VOBs into an MKV container while retaining all chapters and subs.

I know that poisondeathray thinks that "MPEG2 in MKV is known a bad combo prone to issues", but in my experience he is pretty much the only person who says this. MakeMKV has been around for a while now, and the current versions do a perfect job.

Again my request: Could anyone upload a short clip which has this "irregular pulldown", so I can experiment with it using other source filters than DGIndex / DGDecode?


Cheers
manolito

[videoh]

http://rationalqm.us/misc/IrregularPulldown.ts

It's a mix of hard and soft pulldown, so the repeat flags are irregular. Is that what you are looking for?

[poisondeathray]

I know that poisondeathray thinks that "MPEG2 in MKV is known a bad combo prone to issues", but in my experience he is pretty much the only person who says this. MakeMKV has been around for a while now, and the current versions do a perfect job.



MKV is great if you just keep it like that to watch or archive. To be clear, the problems occur when you use it with avisynth as DVD/MPEG2 in MKV with source filters.

Problem(s) go away when you demux it and use standard dgindex method. Many of the old timers know about this. I know manono knows about it. When people upload DVD samples in MKV for examination in various threads, people know to demux it and use dgindex because of these known issues. I'm surprised you don't know about it, because you're old timer too :)

I would say it occurs more than "occasionally", but less than "very frequent" . Frequent enough to never use directshow or ffms2 in this scenario. This DVD/MPEG2 in MKV combo causes a reproducible flaky behaviour with avisynth source filters. Problems such as mixed up frames, jerky decimation/duplication, combing on progressive sources that you otherwise wouldn't get (even with pp=0 in tfm with using mpeg2source/tivtc) .

[videoh]

I would say it occurs more than "occasionally", but less than "very frequent" . Frequent enough to never use directshow or ffms2 in this scenario. This DVD/MPEG2 in MKV combo causes a reproducible flaky behaviour with avisynth source filters. Problems such as mixed up frames, jerky decimation/duplication, combing on progressive sources that you otherwise wouldn't get (even with pp=0 in tfm with using mpeg2source/tivtc) . Just curious, poisondeathray. Do these issues also arise when DGDecNV is used, the point being that DGDecNV opens MKV directly without demuxing? If so, it could be something to be investigated.

[wonkey_monkey]

I thought MPEG2 caused those problems with DSS and FFMS2 in any container.

[manono]

I know that poisondeathray thinks that "MPEG2 in MKV is known a bad combo prone to issues", but in my experience he is pretty much the only person who says this.

Nope, I agree with him 100%. When someone makes available a sample from a DVD in an MKV container, all it does is make me mad because I have to first extract the M2V before then running it through DGIndex so I can use MPEG2Source on it. Making an MKV out of it screws it up royally. You just can't work with it much of the time.

When using a DVD as a source, one should use a proper decrypter.

[poisondeathray]

Just curious, poisondeathray. Do these issues also arise when DGDecNV is used, the point being that DGDecNV opens MKV directly without demuxing? If so, it could be something to be investigated.

Not sure, it occurred so frequently in the past that I know many people automatically demux the mkv by knee jerk reflex . It might be worth investigating


I thought MPEG2 caused those problems with DSS and FFMS2 in any container.
Yes, but not necessarily "generic" MPEG2 .

MPEG2 from other sources (not DVD) in other containers do not necessarily have these problems . For example, MPEG2 cameras (XDCAM in MP4 or MXF work fine with ffms2)

But in the case of the DVD/MPEG2 MKV - You can demux it and it will still have at least some of the problems when using ffms2/directshow. (Ignoring the additional problems caused by DSS for now, we are just taking about field matching or frame order)




Another thing is there might have been improvements to the mkv container spec and/or makemkv , so it might be worth revisiting. manolito said "current versions" of makemkv. Maybe some special something was applied in the last few years. I doubt it. I can rerip some discs done a few years ago but I don't think it will make a difference. I know mkv's made by mkvtoolnix v26 (today) still exhibit this issue when used with ffms2 or directshow (regardless of the workarounds like threads=1 for ffms2, or dss2mod with prefetch any value)

[wonkey_monkey]

Nope, I agree with him 100%. When someone makes available a sample from a DVD in an MKV container, all it does is make me mad because I have to first extract the M2V before then running it through DGIndex so I can use MPEG2Source on it. Making an MKV out of it screws it up royally. You just can't work with it much of the time.


I see I get picture errors when I try. It carries on for a bit, then halts. Initial loading and scrubbing seems fine, though.

DGIndexNV, however, has no such issues.

[manolito]

http://rationalqm.us/misc/IrregularPulldown.ts

It's a mix of hard and soft pulldown, so the repeat flags are irregular. Is that what you are looking for?

Thanks for uploading this clip...

But no, this is not what I am looking for, I want a clip where MediaInfo reports 23.976 progressive AND 3:2 pulldown. Like the clips I made myself from progressive PAL sources which I slowed down to 23.976 and then applied pulldown (either by HCenc or DGPullown). These self-made clips in an MKV container show no (or almost no) problems using ffms2 or DSS2Mod as the source filter. But some folks here are adamant that this method is evil and only MPEG2Source can handle such clips. And I want proof of this...

The clip you uploaded is interesting, though, and I did play with it a little bit. All the following is from the perspective of a AVStoDVD user who will not have any expertise on such sources.

Loading the TS into AVStoDVD prompts the user to index it with DGIndex. Did this, it then shows up in the input window as 29.97 progressive. Of course this is MediaInfo's fault, but in AVStoDVD MediaInfo is all we got. I did not even try to start a conversion with these settings, but the average user would, and he would get a very ugly result. Adding IVTC manually to the script solved it of course, but again an average AVStoDVD user would never be able to do this.

Next thing I tried was to repack the TS to MKV. To my surprise the current version of mkvmerge failed to do this. It could not detect a video stream in this TS file. Weird...
FFmpeg was able to do this, and feeding the resulting MKV to MediaInfo now showed that it was 29.97 Interlaced BFF. Loading it into AVStoDVd was no problem, the default was DSS2Mod as the source filter, interlaced encoding set by default. Which is a good choice for DVD output. Of course there was also the option to apply deinterlacing.

So AVStoDVD users would be able to get a decent conversion result from this clip after repacking the clip into an MKV container. Only the second best choice, but very watchable.


Cheers
manolito

[manolito]

Nope, I agree with him 100%. When someone makes available a sample from a DVD in an MKV container, all it does is make me mad because I have to first extract the M2V before then running it through DGIndex so I can use MPEG2Source on it. Making an MKV out of it screws it up royally. You just can't work with it much of the time.

When using a DVD as a source, one should use a proper decrypter.

All this is because you guys are so damned preoccupied with using MPEG2Source (and nothing else) when it comes to MPEG2 sources. And are you really saying that MakeMKV is not a proper decrypter? Think again...

Can you please try for a minute to put yourself into the shoes of an average user who knows nothing about all these inner workings of the different source formats. Of course all you guys know (and I know, too, I have been around for a while) how to manually demux a DVD title, then extract the chapters using a different tool, and then extract the subs with yet another tool. But there are other users who will use one-click tools like AVStoDVD, and they expect a decent conversion result even for difficult sources.

Now back to the original issue:
In AVStoDVD using MakeMKV to convert DVD titles is by far the most convenient method. This means that MPE2Source cannot be used, but I think that this is no big problem, even with soft telecined NTSC sources. During the last couple of days I did make many such conversions using DSS2Mod and ffms2, and they all came out pretty good...

For such sources MediaInfo reports Progressive 23.976 AND 3:2 Pulldown. DSS2MOD and ffms2 both ignore the pulldown flags (can be changed in ffms2 by using the RFFMode=1 parameter). For DSS2Mod I use DSS2("my_source", fps=23.976, preroll=15). The fps parameter forces the frame rate, if the source has issues then you may get some duplicated or dropped frames. But this is only noticeable if you step through the frames. And this makes sure that there will be no audio sync problems.

I believe that this method does what most users want. It may not be perfect, but IMO it is "Good Enough".

And if any of you guys can provide a source clip where this method fails miserably, go ahead and upload such a clip.


Cheers
manolito

[poisondeathray]

I want a clip where MediaInfo reports 23.976 progressive AND 3:2 pulldown. Like the clips I made myself from progressive PAL sources which I slowed down to 23.976 and then applied pulldown (either by HCenc or DGPullown). These self-made clips in an MKV container show no (or almost no) problems using ffms2 or DSS2Mod as the source filter. But some folks here are adamant that this method is evil and only MPEG2Source can handle such clips. And I want proof of this...



Here you go, fresh re-rip using newest makemkv (in case it did something new and magical the last few years), newest mkvtoolnix to strip everything else out and cut a 5min30sec sample. (from the beginning in case there were issues caused by cutting the in point)
http://www.mediafire.com/file/ke3pjoubowoxlod/title00+%281%29.mkv


Original frame rate : 23.976 (24000/1001) FPS
Standard : Component
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Scan order : 2:3 Pulldown


I made sure to use single treaded, no prefetch. DSS2Mod/Directshowsource/any DS derivative fails miserably (jerky motion). Completely unusable with lav splitter, lav decoder (cpu, hw or otherwise), regardless of settings, regardless of DSS2 preroll, tested on multiple configurations and computers.

DSS2("title00 (1).mkv", preroll=X)
AssumeTFF().TDecimate()
#AssumeTFF().TFM(pp=0).TDecimate() #Doesn't matter if you include TFM, still jerky
#AssumeTFF().Decimate() #Doesn't matter if you use Decimate instead of TDecimate, still jerky


FFMS2 does better here than directshow derivatives, but notice combing in a few frames ~ 2:54-2:55 as the ship goes up. It does not matter if you demux to ES, or leave it in MKV. Tried multiple ffms2 versions.

FFVideoSource("title00 (1).mkv", threads=1)


If you try FFMS2 with rffmode=1 (so it honor flag), with TIVTC, it becomes jerky in some parts and lots of combing (remember we set PP=0 to disable post processing to check)


The "gold standard" . Demuxed mkv, then DGIndex using Honor flags. Notice I put PP=0 to disable post processing and comb detection deinterlacing. Notice no combing where ffms2 had problems. (AssumeTFF() is redundant because MPEG2Source passes that info, but I left it in to be complete to compare to DSS)

MPEG2Source("title00 (1)_track1_eng.d2v", cpu=0)
AssumeTFF().TFM(pp=0).TDecimate()





It does not matter if other valid decrypting methods are used (e.g. anydvd, dvd decrypter, dvd shrink, etc...this is not a decryption issue). There are many examples over the years in various threads. If I re-rip a random DVD , you will see some problems , quite frequently when using DSS or FFMS2. (But not "always" - or maybe I missed some errors that were there; it occurs so frequently , so repeatedly, that many people just don't waste time using FFMS2 and especially DSS, for this DVD/MPEG2 scenario). Granted, FFMS2 wasn't as badly affected in this specific example, but it can be, where frames are actually misplaced or mixed up in order. Why is FFMS2 prone too? It uses an index... My understanding is that the difference is because indexing is different ; DGIndex does something with bytes, but the ffms2 index isn't as robust. I'm sure videoh or myrsloik can clarify the exact details if you're interested.

https://forum.doom9.org/showthread.php?p=1835524#post1835524
Myrsloik:

Basically FFMS2 makes a lot of assumptions like timecodes are correct and that the libavformat demuxers can always seek accurately. Other source filters like the d2v using ones don't and instead more or less index which specific bytes are needed by each frame (or something thereabout).


Eitherway, the DGIndex method is the most consistent for DVD/MPEG2. By far. Directshow is prone to a few issues (this is well known), and ffms2 might be slightly better but not immune to problems. I haven't even gone into other issues using filters. There are sometimes compounding errors when you have filters that require non linear access. Things such as temporal filters. ffms2 can be slightly improved in that regard in general (more robust seeking) if you use seekmode=0 (slower). Also some of the other ffms2 issues can be from MKV container (that's why I always test ES too), because it makes some assumptions about timecodes. Well sometimes the MKV timecodes are just buggy and maybe that's why, not sure, but this has been shown to improve some of the problems with samples in other threads over the years; but not all the problems everytime.






Can you please try for a minute to put yourself into the shoes of an average user who knows nothing about all these inner workings of the different source formats. Of course all you guys know (and I know, too, I have been around for a while) how to manually demux a DVD title, then extract the chapters using a different tool, and then extract the subs with yet another tool. But there are other users who will use one-click tools like AVStoDVD, and they expect a decent conversion result even for difficult sources.

Now back to the original issue:
In AVStoDVD using MakeMKV to convert DVD titles is by far the most convenient method. This means that MPE2Source cannot be used, but I think that this is no big problem, even with soft telecined NTSC sources. During the last couple of days I did make many such conversions using DSS2Mod and ffms2, and they all came out pretty good...

For such sources MediaInfo reports Progressive 23.976 AND 3:2 Pulldown. DSS2MOD and ffms2 both ignore the pulldown flags (can be changed in ffms2 by using the RFFMode=1 parameter). For DSS2Mod I use DSS2("my_source", fps=23.976, preroll=15). The fps parameter forces the frame rate, if the source has issues then you may get some duplicated or dropped frames. But this is only noticeable if you step through the frames. And this makes sure that there will be no audio sync problems.

I believe that this method does what most users want. It may not be perfect, but IMO it is "Good Enough".

And if any of you guys can provide a source clip where this method fails miserably, go ahead and upload such a clip.


But why can't AVS2DVD be coded to demux the MKV and use DGIndex? Other GUIs can if I'm not mistaken. I think megui can

[poisondeathray]

For DSS2Mod I use DSS2("my_source", fps=23.976, preroll=15). The fps parameter forces the frame rate, if the source has issues then you may get some duplicated or dropped frames. But this is only noticeable if you step through the frames. And this makes sure that there will be no audio sync problems.


aha ! fps=23.976 fixes the jerky issues . Still combing in the cockpit up scene, but that' s minor compared to jerky

EDIT: spoke too soon... it completely messes up other sections in the larger movie, where every 2nd frame is dropped, and duplicated. I'll try to see if I can isolate an example


I cannot reproduce on a cut sample for some reason, only the whole movie. Even a 10 min sample is not enough to produce the issue. When loading the whole movie with dss2("full.mkv" , fps=23.976, preroll=15), there are sections that have ~ every 2nd frame dropped and replaced with a duplicate frames. It's like the errors just get pushed later into the movie for several scenes. But then other sections recover perfectly, then later on others have it again. It affects both x86 avisynth classic or mt (in single thread mode) , or avs+ x64 (in single thread mode) . Affects different lav settings (CPU, HW etc..). Very flaky behaviour.

ffms2 has 64 extra frames. The running time is ~2.6 seconds longer and out of sync if you use FFAudioSource/FFVideoSource with audiodub. Progressive sync issue, simple shift won't fix. It's not immediately clear where the extra frames are inserted, some here some there. (not an easy fix) . Some jerky frames (fwd/back) too (even when going way back past a few GOP's then "sneaking" up on the section). It might be those jerky frames are the inserted frames.

Both are unusable on the whole movie

But DirectShowSource("full.mkv", fps=23.976) doesn't have those dropped/duplicated frames sections. A grey frame at the end. But that might be semi- usable with only minor issues if you did a straight linear encode, no filters.

[manolito]

aha ! fps=23.976 fixes the jerky issues . Still combing in the cockpit up scene, but that' s minor compared to jerky

Maybe you should have read some of my earlier posts... :devil:

From the opening post of this thread:
3: On the first look DSS2Mod (DirectShowSource has the same behavior) seems to honor the pulldown flags, it decodes to 29.97. But unlike Mpeg2Source the output shows no combing. It looks progressive, but with tons of duplicate frames. So in reality the pulldown flags are not honored at all. This is not what anybody would want. To get the original 23.976 output without dupes I need to add "fps=23.976" to the source filter call.

DSS2Mod ignores all pulldown flags, using "fps=23.976" forces the output frame rate (CFR) by duplicating or dropping frames. But this does not explain the combed frames you are getting occasionally. DSS2Mod certainly did not add them, and in my tests I cannot reproduce this.

I converted your uploaded clip using both DSS2Mod and ffms2, and both conversions came out nicely. Download here:
https://www.zeta-uploader.com/1333956887

I could not detect any combing, and because I resized vertically this would have looked especially ugly. Neither did I see any dupes or dropped frames, for me these conversions look perfect.

EDIT: spoke too soon... it completely messes up other sections in the larger movie, where every 2nd frame is dropped, and duplicated. I'll try to see if I can isolate an example

This looks fishy, especially if you say that this does not happen with DirectShowSource. I use DSS2Mod almost daily with captured HEVC sources in an MKV container, and these are full 2 hour movies. Never saw any issues like these. I suspect that something with your DSS2Mod installation is problematic.

Are you using the original forclip avss_26.dll? Do not use avss.dll under AVS 2.60 or AVS+. Also there is an unofficial 64-bit version floating around, don't...


Some of my ffms2 experiences:
The versions do matter, newer versions seem to be less reliable. The ones I prefer are 2.23.1 from Github and the C-Plugin version by qyot27 ffms2_r1140+101-avs+vsp.7z. Newer versions of the C-Plugin seem to have issues.

Basically ffms2 with the default parameters also ignores the pulldown flags, but the output for such soft telecined sources will be at 23.976 fps. Still it is recommended to force CFR 23.976 fps because you never know which issues the source might have. For ffms2 this is done by adding the fpsnum and fpsden parameters to the call. But this can also introduce problems. I won't go into the details here, you can find this in the ffms2-C-Plugin thread. What fixed it for me is to generally not use fpsnum and fpsden, but instead use "ChangeFPS(23.976)" right after the source filter call. The ffms2 conversion in my uploaded file was done this way.

My conclusion:
DSS2Mod (maybe even DirectShowSource) and FFmpegSource can handle 23.976 progressive NTSC sources with 3:2 pulldown just fine. You just need to have an understanding how these source filters work. Dealing with hybrid sources with a mixture of soft and hard telecined sources, possibly adding real 29.97 interlaced video is a different story. No source filter will handle these cases automatically.


Cheers
manolito

[poisondeathray]

I could not detect any combing, and because I resized vertically this would have looked especially ugly.

Look more closely at the scene where the cockit goes up. Both your encodes have it in a few frames



Neither did I see any dupes or dropped frames, for me these conversions look perfect.

There is on the full movie, and ffms2 is out of sync.

I'll try to reproduce it on a larger cut file, but I suspect it's going to have to be very large


This looks fishy, especially if you say that this does not happen with DirectShowSource.

I agree, super fishy. I would have expected the same behaviour on linear seeking. I'll try to debug it tonight


I use DSS2Mod almost daily with captured HEVC sources in an MKV container, and these are full 2 hour movies. Never saw any issues like these. I suspect that something with your DSS2Mod installation is problematic.


But did you try a 2hr soft pulldown DVD/MPEG2 source ripped with makemkv ?



Are you using the original forclip avss_26.dll? Do not use avss.dll under AVS 2.60 or AVS+. Also there is an unofficial 64-bit version floating around, don't...

Yes, avss_26.dll, the original forclip . The version is 2.0.0.13

Can you post a link to the exact one if that is not the correct one


Some of my ffms2 experiences:
The versions do matter, newer versions seem to be less reliable. The ones I prefer are 2.23.1 from Github and the C-Plugin version by qyot27 ffms2_r1140+101-avs+vsp.7z. Newer versions of the C-Plugin seem to have issues.

I tried 2.23.1 from Github, and a few others, but not the recent c-plugins. I deleted the index each time. Progressively out of sync, too many frames



Generally ffms2 with the default parameters also ignores the pulldown flags, but the output for such soft telecined sources will be at 23.976 fps.


Because rffmode=0 by default. If set rffmode=1 it's supposed to honor flags (But it's buggy) .


Still it is recommended to force CFR 23.976 fps because you never know which issues the source might have. Generally for ffms2 this is done by adding the fpsnum and fpsden parameters to the call. But this can also introduce problems. I won't go into the details here, you can find this in the ffms2-C-Plugin thread. What fixed it for me is to generally not use fpsnum and fpsden, but instead use "ChangeFPS(23.976)" right after the source filter call. The ffms2 conversion in my uploaded file was done this way.


Maybe for other types buggy source files in general, but if it's ripped correctly, DVD should be CFR.

If the source read at 23.976 CFR already, that is a no-op. The versions of ffms2 I used read it at 23.976 CFR

You can check with info() for what frame rates are internally, and ffinfo() for the CFR/VFR times


My conclusion:
DSS2Mod (maybe even DirectShowSource) and FFmpegSource can handle 23.976 progressive NTSC sources with 3:2 pulldown just fine.

But are you basing that "conclusion" on a single small test clip without audio ? A bit premature maybe ? So how was the sync? :D

eg. In your encodes, you have 7914 frames in your ffms2 version 7910 frames in your dss2mod version. Do you think that's ok? Well that's what I'm seeing too on the small test too. Is it not plausible that on longer video, there are more frames ? Because that's what I saw on the full movie with 64 extra frames, therefore progressively out of sync. Not an easy fix where all 64 frames are located nicely at the end or beginning.

My conclusion is there are plenty of problems with DSS2Mod and FFMS2 with DVD/MPEG2 sources. Many problems in the past. Many problems now. DirectShow had fewer problems here (go figure!) but still some issues like combing and grey frame at the end. It might be my dss2mod configuration is messed up, I'll look into it

I might have to upload a very long clip for you to see the problems.

[videoh]

Maybe it's time to add MKV support to DGIndex/MPEG2Source.

[manolito]

Your DSS2Mod version looks right, maybe compare it with my version:
https://www.sendspace.com/file/if6r6r

In one previous post you mentioned your source filter call:
DSS2("title00 (1).mkv", preroll=X)

I hope you did not really set "preroll=X". Use 15, and do specify the fps value...

And yes, a longer clip which shows the issues would be nice.

Cheers
manolito

[manolito]

Maybe it's time to add MKV support to DGIndex/MPEG2Source.

Hey, this would be really great... :D

[poisondeathray]

Your DSS2Mod version looks right, maybe compare it with my version:
https://www.sendspace.com/file/if6r6r


Thanks , it's the same. (and I tried avss_x64.dll for x64 version)

What lav version are you using ? splitter and decoder ?


In one previous post you mentioned your source filter call:
DSS2("title00 (1).mkv", preroll=X)

I hope you did not really set "preroll=X". Use 15, and do specify the fps value...


The "X" was to denote different values were attempted

If you check the later post about the full movie I used fps=23.976 and preroll=15 (And I tried without preroll and 0 and other values like 24,30 ) to see if it would help


And yes, a longer clip which shows the issues would be nice.


The first stretch of big problems occurs ~ 52min in and lasts about 7 minutes. I tried cutting a 10min section, enclosing the problem area but it wouldn't reproduce on the cut section . I think it has to be from the beginning . I'm trying to avoid having to upload 2-3GB

Also I I'll include audio this time




Maybe it's time to add MKV support to DGIndex/MPEG2Source.

+1 .

That would be great if you have free time

[poisondeathray]

If anyone else wants to take a look at the longer sample, PM me.

Unfortunately the sample is massive. I couldn't get the first long bad section to replicate unless it was cut from near the beginning. But OTOH it's easier to see the ffms2 sync issues with a longer sample



The dss2mod massive dropped/duplicated frame sections issue looks related to dss2mod+lav decoder (none or CPU), or dss2mod+lav(cuvid,nvidia). dss2mod+QS seemed ok at first, but it just shifted the drop/dupe section to other sections instead (but not as long or as bad as lav(cpu or cuvid), at least with a quick glance). Also, dss2mod+ffdshow doesn't seem have that issue for mpeg2 decoding . Other decoders like mainconcept also have the same long stretch of bad (the full 7 min or other long stretches weren't included in this sample). DirectShowSource doesn't have that particular issue with any decoder (which is weird) . But all DS versions still have combing in some scenes, and usually a grey frame at the end

haali + lav (cpu or cuvid) had same issue, so it's probably not lav splitter . I tried older versions of lav too, lav threads=1

ffms2 had 31 extra frames in this longer sample, not surprisingly it was about that much out of sync near the end. (And more than double that at the end of the full movie)

If anyone can't see the combing with DS/DSS2 or ffms2 , let me know and I can point where those sections are

[FranceBB]

Maybe it's time to add MKV support to DGIndex/MPEG2Source.




+1 .

That would be great if you have free time

+2.
Even though we can just demux the file and use DGIndex, it would be great to have a new version of DGIndex. ^_^

[manolito]

@ poisondeathray

Played with your uploaded source for a while, this is what I got so far...

The combed frames in the output are not caused by the source filters, they are already present in the source MKV. I suppose that they were already in the VOBs on the source DVD. MakeMKV does not reencode anything, and removing copy protection should not introduce combed frames either.

Otherwise I could reproduce the duplicated frames issue on two different computers. It still puzzles me how this can happen. Correct output for a long time, then suddenly dupes and dropped frames. Some kind of buffer overflow problems?

I tried a few different MPEG2 decoders to make sure that LAV Filters was not to blame, but no real result. With an old Cyberlink decoder and also the old DScaler decoder the sections with the dupes started at a different point in time, but they were there.

So the gold standard for such 23.976 progressive NTSC sources with pulldown flags really seems to be using MPEG2Source (honoring pulldown flags) followed by a proper IVTC.

The only other method I found which does come close to it is this one:
1. Use DSS2Mod forcing 29.97. This will result in a progressive 29.97 output with tons of dupes.
2. Instead of using standard IVTC just use "TDecimate(mode=7, rate=23.976)".

The Aliens clip comes out almost perfectly with this setting. I also tried FDecimate and FDecimate2 (by StainlessS), but the TDecimate command worked better.

Looks like we need to wait for DG to update DGIndex to work with MKV sources...


Cheers
manolito

[poisondeathray]

The combed frames in the output are not caused by the source filters, they are already present in the source MKV. I suppose that they were already in the VOBs on the source DVD. MakeMKV does not reencode anything, and removing copy protection should not introduce combed frames either.


Yes.

But the MPEG2Source method using TIVTC does not have them in any of the sections. Recall I purposely set TFM(pp=0) to reveal any problems (in case it was deinterlacing the combed frames and hiding the problems) . I believe underlying issue is it was something like 99.8% film according to DGIndex, so if you had "forced film" in DGIndex you'd get the same combed frames. But this is very common for soft pulldown DVD's . The majority are probably around 99.5 or so. The implication here is that you need to return fields, in order to field match. So if DSS2 at 29.97 (or any source filter) is not returning fields , it has no hope of doing it properly in terms of the combing (unless you do some sort filtering afterwards, which will always produce worse) . But combing isn't as bad as the jerky playback or sync issues. You could argue that combing in a few scenes is watchable even if not ideal .



Otherwise I could reproduce the duplicated frames issue on two different computers. It still puzzles me how this can happen. Correct output for a long time, then suddenly dupes and dropped frames. Some kind of buffer overflow problems?

That sample only had ~3 minutes of dupes at the end , but that stretch of problems on the full movie actually lasted 7 minutes. And then recovered, and a few more stretches of dupes/drops that were about a few min long later on. But I tried the x64 version as well in case it was some memory issue , but same issue. Very flaky behaviour



I tried a few different MPEG2 decoders to make sure that LAV Filters was not to blame, but no real result. With an old Cyberlink decoder and also the old DScaler decoder the sections with the dupes started at a different point in time, but they were there.


Yes, they get shifted with different decoders, but not sure if you missed it but ffdshow seemed to work ok with dss2mod when set to libavcodec, even on the full movie (only the combing and customary missing frame at the end) .

So do you expect users to play with directshow configurations and decoder settings? Person A might have different settings than Person B. I usually have mine set on CUVID. ffdshow is hardly used by anyone. So ffdshow might worked here in this particular example, but I've seen problems with ffdshow in the past too in this scenario... way to inconsistent




The only other method I found which does come close to it is this one:
1. Use DSS2Mod forcing 29.97. This will result in a progressive 29.97 output with tons of dupes.
2. Instead of using standard IVTC just use "TDecimate(mode=7, rate=23.976)".

The Aliens clip comes out almost perfectly with this setting. I also tried FDecimate and FDecimate2 (by StainlessS), but the TDecimate command worked better.


whaaat ? Not even close, jerky in sections.



And did you get a chance to look at ffms2 versions? Hopeless, right? Unusable because of the progressive sync issues and added frames. Or did you find a "magical" version ?



To reiterate - this is quite common with DVD/MPEG2 when using Directshow, FFMS2 . Not just these sorts of problems, but others as well. I'm not making this up or blowing it out of proportion. It might be "new" to you because you're in PAL land, but this is common knowledge for NTSC avisynth users . You have a consistent reliable proven method (actually 2, if you count dgsource), and then "the others" which are unreliable , prone to many issues. Again - I would avoid even having those as options for DVD/MPEG2 sources for a GUI - grey they out or plaster some warning signs

[manolito]

whaaat ? Not even close, jerky in sections.

Which sections exactly?
https://we.tl/t-9aaHGd12Fo

I think this conversion looks pretty good...


Cheers
manolito

[poisondeathray]

Which sections exactly?
https://we.tl/t-9aaHGd12Fo

I think this conversion looks pretty good...


Cheers
manolito


Reallly ? look in the first 5-6 minutes , it should be very obvious, every scene has dropped/duplicated frames and jumps in motion.

What's weird is looks closer to normal after that. I got much choppier results initially , throughout that the whole sample. But then I switched to lav CPU (instead of lav CUVID), then I got the same choppy sections as you. But in the full movie, choppy sections reappear later too even with lav CPU. The combing might be considered minor to some people, but the choppiness makes it unusable . Maybe you didn't look at the first 5-6 minutes ?

[manolito]

Maybe you didn't look at the first 5-6 minutes ?

Yes, this could be it. I was somehow concentrating on the last third of the clip where the dupes showed up in the older conversions. Need to have another look.. :o

[manono]

But some folks here are adamant that this method is evil and only MPEG2Source can handle such clips. And I want proof of this...

Do you think we're 'adamant' because we're fanboys or something similar? We're adamant because it's the only way that works reliably. It seems to me that you're the adamant one, trying to fit a square peg into a round hole. Sure, you're trying come up with a method to make AVSToDVD work well with all sources.


To reiterate - this is quite common with DVD/MPEG2 when using Directshow, FFMS2 . Not just these sorts of problems, but others as well. I'm not making this up or blowing it out of proportion. It might be "new" to you because you're in PAL land, but this is common knowledge for NTSC avisynth users . You have a consistent reliable proven method (actually 2, if you count dgsource), and then "the others" which are unreliable , prone to many issues. Again - I would avoid even having those as options for DVD/MPEG2 sources for a GUI - grey they out or plaster some warning signs

I suspect pdr spent hours of his own time creating his samples, testing and making his long posts explaining the results of his tests. Neither I nor pdr (I don't think) even use AVSToDVD. It's not just you but many of the source filter developers that reside in PAL countries and haven't had all that much experience with NTSC material. And their filters just don't work well with difficult NTSC VOBs/MPGs. It's a worthy goal to help the inexperienced make better encodes. Please pay attention and learn.

[manolito]

Do you think we're 'adamant' because we're fanboys or something similar? We're adamant because it's the only way that works reliably. It seems to me that you're the adamant one, trying to fit a square peg into a round hole. Sure, you're trying come up with a method to make AVSToDVD work well with all sources.


Well, I started this thread because I wanted to learn about the different source filter behavior with these progressive NTSC sources with soft pulldown flags. But basically all the answers I got were like "Forget it, just use MPEG2Source, all the other source filters just don't work with these sources, case closed". And these kind of answers do trigger a certain stubbornness in me... :devil:

Can it really be true that after the DVD format has been around for so many years noone except DG was able to write a source filter which could handle such sources?

Thanks to pdr I got a nice source file to play with, and yes, I have exhausted all options with DSS2Mod so far. The reason is that DirectShow based source filters do not honor pulldown flags, and for sources which are less than 100% film there is no other way than adding dupes or drop frames to achieve the target rate.

But I am not a person who gives up that easily. We still have ffms2 to test. I am aware that pdr mentioned that while ffms2 can honor pulldown flags with the rffmode parameter, this option was buggy. But since I believe only in things I have tested myself, I put his uploaded file on my test bench...

I used the FFMS2 C-plugin 1140+101 from 2016, the AVStoDVD script was straightforward:
LoadCPlugin("C:\Program Files (x86)\AVStoDVD\Lib\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Audio = FFAudioSource("C:\Download\title00 (1).mkv", track=-1)
Video = FFVideoSource("C:\Download\title00 (1).mkv", track=-1, rffmode=1, seekmode=0)

Video = Video.ConvertToYV12()
Video = Video.AssumeTFF()
Video = Video.TFM().TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)

The AssumeTFF() call was a shot from the hip, I did not take the time to check the field order. Downmixing audio to stereo was just a means to get the final result size below 2 GB.

Download the result here:
https://we.tl/t-ojt0XCVLs5

Any objections? Audio sync is perfect, and I could not detect any jerkiness due to duplicated or dropped frames.


Cheers
manolito

[manono]

And these kind of answers do trigger a certain stubbornness in me... :devil:
And your earlier claim that I was (or we were) promoting some kind of an agenda pissed me off no end. My only agenda is having something that works reliably. So I kept quiet for a few days lest I write something I might later regret. Besides, I know your goal is worthy and your heart's in the right place. And I can't do, and have no interest in doing, the kinds of things pdr did for you.

Don't you think you should make available your source, the title00 (1).mkv?

[videoh]

Can it really be true that after the DVD format has been around for so many years noone except DG was able to write a source filter which could handle such sources? There are several gotchas and corner cases for implementing correct and robust field pulldown, especially with frame accurate random access (seeking). It has to be implemented carefully. Many of the source filter developers are in Europe and they may therefore not be fully motivated for NTSC things, or may not have the needed source samples for proper testing. It's hard to say. But the source code for MPEG2Source contains the solution and it is open source so in my opinion there's no excuse for having a buggy pulldown implementation. The original implementation was by jackei of DVD2AVI fame, by the way, and DG enhanced it for robust random access. So there's no need to succumb to DG derangement syndrome, at least in this case. ;)

[poisondeathray]

Any objections? Audio sync is perfect, and I could not detect any jerkiness due to duplicated or dropped frames.




It's better and the other ffms2 versions and in sync

But there is significantly more aliasing and combing, even though you didn't set pp=0. (For testing purposes, I always use pp=0 , so it doesn't deinterlace and it's easier to see the problems). Check the first few minutes again, about 1/3-1/4 of the frames in the first few minutes are combed. It's easy to see with the spaceship coming in. In the other sections, significantly more combing introduced in other scenes than the directshow methods. Some of the problems might be partially hidden by your resizing and encoding. Preview the script without resizing and pp=0 to see it clearly. When using TIVTC, you should set the field order explicitly if not using DG source filters (DG stuff passes the info, others usually do not. For example, it's TFF here, but avisynth assumes BFF otherwise - but it didn't help with the problems in this case) . There is a duplicate in the same "cockpit up" section (along with combing), there might be more duplicates but I didn't check closely

So this ffms2_r1140+101-avs+vsp is significantly worse than directshowsource with ffdshow . That latter option would be second best to the DG options, with only minor aliasing/combing.

Cheers

[manolito]

Don't you think you should make available your source, the title00 (1).mkv?

pdr sent me the link via PM, see here:
https://forum.doom9.org/showthread.php?p=1850845#post1850845


@ poisondeathray
When using TIVTC, you should set the field order explicitly if not using DG source filters (DG stuff passes the info, others usually do not. For example, it's TFF here, but avisynth assumes BFF otherwise - but it didn't help with the problems in this case)

I did have "AssumeTFF()" right before the TFM call in my script. TFM defaults to "order=-1", which means it takes the AviSynth field order. So my "AssumeTFF()" should have taken care of it.


@ videoh
The original implementation was by jackei of DVD2AVI fame, by the way, and DG enhanced it for robust random access. So there's no need to succumb to DG derangement syndrome, at least in this case.

jackei was a little bit before my time at doom9. And I don't think that I showed any symptoms of the "DG derangement syndrome" here, when I said that obviously noone except you was able to write a working source filter for such sources, then I was merely expressing my appreciation for your work.
Anything else would certainly have minimized our chances for this:
Maybe it's time to add MKV support to DGIndex/MPEG2Source.


Cheers
manolito

[videoh]

jackei was a little bit before my time at doom9. And I don't think that I showed any symptoms of the "DG derangement syndrome" here, when I said that obviously noone except you was able to write a working source filter for such sources, then I was merely expressing my appreciation for your work. Thank you for your reply. The remark wasn't directed at you but rather a little dig at the cabal (they know who they are -- Fiona's ex flying monkeys) that delight in trolling and provoking me. If I was as bad a coder as they would have you believe, probably I wouldn't have gotten the pulldown working correctly in DGIndex, and I wouldn't have 10,000 users of DGDecNV, etc.

Just gonna finish a few things with my CUDASynth experiment, then I'll get started on MKV for DGMPGDec. Stay well.

[manolito]

:thanks:

[videoh]

You're welcome. Starting on MKV tomorrow...

[FranceBB]

I wouldn't have 10,000 users of DGDecNV

Including one of my colleagues that uses it at work.

Starting on MKV tomorrow...

About that...
This is really appreciated, but since you are willing to develop DGIndex (and DGDecNV) even further, it would be really useful for professional usage to have a proper support for XDCAM-HD422 files in the future (Container: .mxf Video: MPEG-2 50 Mbit/s 4:2:2 planar (yv16) 8bit, closed GOP M=3 N=12 Audio: CH.1-2 DolbyE 5.1 CH.3-4 DolbyE 5.1 CH.5-6 PCM Stereo Downmix CH.7-8 PCM Stereo Downmix + Ancillary data like timecode).

[manolito]

Sorry it might look a little bit like I must always have the last word, but I cannot have this post
https://forum.doom9.org/showthread.php?p=1851171#post1851171
stand unchallenged.

Here is the converted clip using the "gold standard"
https://we.tl/t-o57QJNLXz2

Using this AVS script:
Import("C:\Program Files (x86)\AVStoDVD\Lib\A2DSource.avsi")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\DGIndex\DGDecode.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Video = MPEG2Source("D:\title00 (1)_1.d2v")
Audio = A2DAudioSource("C:\Download\title00 (1)_2_EN.ac3", CacheFolder="D:")

Video = Video.ConvertToYV12(interlaced=true)
Video = Video.TFM(d2v="D:\title00 (1)_1.d2v").TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)
Prefetch(4)


And this is the result using ffms2 with the "rffmode=1" parameter:
https://we.tl/t-beNb9tWTPx

Using the following script:
LoadCPlugin("C:\Program Files (x86)\AVStoDVD\Lib\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Audio = FFAudioSource("C:\Download\title00 (1).mkv", track=-1)
Video = FFVideoSource("C:\Download\title00 (1).mkv", track=-1, rffmode=1, seekmode=0)

Video = Video.ConvertToYV12()
Video = Video.AssumeTFF()
Video = Video.TFM().TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)


Now please compare the results and tell me if you can detect any differences in quality...

Do not test by stepping through frames, play the clip in a player software instead. PDR insisted that the ffms2 output showed more aliasing and combed frames, I could not confirm this. I prefer using real world scenarios, and here field matching is never 100% accurate, so there is a reason that TDecimate uses postprocessing by default.


I showed the two clips to a friend who works for a German public broadcaster in a senior position. He is not involved in the technical aspects, he manges he content side. But he certainly has a very trained eye when it comes to video content. I showed both clips to him, he could not see any quality differences, and he would have approved both clips for broadcasting right away.


Cheers
manoito

[hello_hello]

I haven't read the whole thread thoroughly, or looked at manolito's samples yet because they're downloading at dialup speed, but I have a question.....

Has anyone been using the ffms2 frame rate conversion (fpsnum/fpsden) and the rffmode=1 option interchangeably? It's just that I recall a mention of combing problems.

rffmode=1 enables CFR mode itself. If there's no repeat field flags it'll output an error when rffmode=1, otherwise it should output 29.970fps. No need for frame rate conversion too. In fact the way I remember it, frame rate conversion doesn't work when rffmode=1, but that could be wrong.

And when it's a job for rffmode=1, you can't use frame rate conversion instead, because I'm pretty sure ffms2 outputs the average frame rate by default (when there's both soft and hard telecined sections), so it can mess with the audio sync and I doubt frame rate conversion will help. Plus if you used frame rate conversion for a source that had some hard telecined sections and set it to 23.976fps, I assume ffms2 will drop frames in the 29.970fps sections as required without giving a thought to field matching. Although what probably happens is.... ffms2 starts off at the average frame rate and drops frames from the entire video to get the frame rate down to 23.976fps, so without rffmode=1 it'd be dropping frames from the soft telecined sections too..... I think.

Anyone know why rffmode=1 isn't the default for mpeg2, at least? I've sometimes wondered if it should be.

[manolito]

From my (limited) experience "rffmode=1" is pretty much the same as "Honor Pulldown Flags" in DGIndex. As opposed to DGIndex it will throw an error if the source has no pulldown flags, and it cannot do tricks like detect and fix field order transitions. To IVTC the output you can use the normal "TFM().TDecimate()", but there is no way to teach TFM about the pulldown flags like "TFM(d2v="my d2v file"). So it is not as reliable as DGIndex / DGDecode, but for sources which are less than 100% film it works pretty well for me.

Cheers
manolito

[hello_hello]

Something in the help file wording doesn't seem quite right to me.

rffmode

- **0**: Ignore all flags (the default mode).
- **1**: Honor all pulldown flags.
- **2**: Equivalent to DVD2AVI's "force film" mode.

Also note that "force film" is mostly useless and only here for completeness' sake, since if your source really is safe to force film on, using mode 0 will have the exact same effect while being considerably more efficient.

I wonder what the definition of "really is safe to force film" is, because if a source is mostly soft-telecined, with just a small section or two hard telecined, won't modes 0 and 2 output different frame rates? The average frame rate for mode=0 and, I assume 23.976fps for mode 2.

The same would happen for soft telecined sources with small sections of interlaced video. I understand "force film" to mean "the studio promo at the beginning is interlaced and I don't care if it's converted to 23.976fps by dropping frames", or something like that, but wouldn't mode=0 output the wrong (average) frame rate in that situation?

Although while I was typing it occurred to me it's possibly better to output a timecodes file instead and give it to x264 to use for VFR encoding. And now I've thought about it, ffms2 no doubt writes the timecodes file while it's indexing, whereas I've been indexing with DGIndex, running a first pass with TIVTC to create the timecodes file, then a second pass for encoding. Now I'm wondering why I'd use DGIndex in preference to ffms2. :)

Sorry manolito. I realise you do a lot of encoding for DVD and VFR is no good for you..... I was thinking as I was typing..... but....

Is there a tool for applying pulldown to existing video? Well... I'm aware of DGPulldown although I've never used it, but it appears to only convert from one constant frame rate to another. What if it could accept a timecodes file to determine which fields to repeat, so then you could use ffms2 mode=0, output a timecodes file, apply the pulldown after encoding and bypass having to convert the frame rate and applying IVTC etc.

It sounds like a good idea in my head....

[poisondeathray]

And this is the result using ffms2 with the "rffmode=1" parameter:
https://we.tl/t-beNb9tWTPx

Using the following script:

LoadCPlugin("C:\Program Files (x86)\AVStoDVD\Lib\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AVStoDVD\Lib\TIVTC.dll")
Import("C:\Program Files (x86)\AVStoDVD\Lib\Downmix.avsi")

Audio = FFAudioSource("C:\Download\title00 (1).mkv", track=-1)
Video = FFVideoSource("C:\Download\title00 (1).mkv", track=-1, rffmode=1, seekmode=0)

Video = Video.ConvertToYV12()
Video = Video.AssumeTFF()
Video = Video.TFM().TDecimate()
Video = Video.Spline36Resize(720,576)
#PALSpeedUp: using AssumeFPS() to upsize FPS

Audio = Audio.Dmix6Stereo()

AudioDub(Video, Audio)
AssumeFPS("pal_film", sync_audio=true).SSRC(48000)



How is that any different than the last one in post #60 ? The script is the same. Was the .dll different ? Do I really have to download it?

The poor results were already confirmed and I reproduced the issues locally with the same .dll. Some sections had ~ 30% combing, and that' s with post processing enabled (PP wasn't set to 0 in that script) - so those passed through in your encode (TFM didn't detect, didn't apply post processing on them).


Do not test by stepping through frames, play the clip in a player software instead. PDR insisted that the ffms2 output showed more aliasing and combed frames, I could not confirm this. I prefer using real world scenarios, and here field matching is never 100% accurate, so there is a reason that TDecimate uses postprocessing by default.

I showed the two clips to a friend who works for a German public broadcaster in a senior position. He is not involved in the technical aspects, he manges he content side. But he certainly has a very trained eye when it comes to video content. I showed both clips to him, he could not see any quality differences, and he would have approved both clips for broadcasting right away.



Don't tell me how to test it. Step through it and see the combing issues. Some people have terrible eyesight or cannot see these issues. If you blink, you might miss some sections.

Maybe your method of viewing is flawed ? Maybe your player is deinterlacing ?

Just because you or your so called expert can't see it , does not mean it's not there. Some sections had ~ 30% combing! It's really obvious. Usually content managers don't do this low level QC stuff. But people in senior positions are ultimately responsible. If that' s how he does QC and he misses stuff this obvious - he really should be fired .

We are testing the effectiveness of a source filter +/- field matching ... not post processing. We just want the original progressive frames back. You don't add x,y,z variables to convolute your results. You don't add lossy encoding on top of it. That's why you look at PNG images not JPG. Why not resize it to UHD? Why not use pointresize or bicubicresize ? Why don't you just add QTGMC or some deinterlacer or denoiser afterwards ? Maybe vinverse? Answer to the rhetorical questions : Because other variables can pollute your results and it's not appropriate for what you set out to test . When you are looking at this , why are you even converting to PAL and resizing ? I know you're doing it for a specific scenario, but it just adds other variables . You want to strip out all the other extraneous variables to test exactly what you set out to test - It's called the "scientific method."

I seriously can't tell if you're pulling my leg. The combing and aliasing are so obvious, even in a media player. Some sections are fairly clean with only a few frames combed, but that spaceship coming in section has ~30% bad frames. Even if you had just forced film without field match , or used directshow , you only have <0.5% combing total! Recall the film% in the log

[Groucho2004]

Maybe your method of viewing is flawed ? Maybe your player is deinterlacing ?Or the decoder could be de-interlacing. LAV filters have de-interlace options and also ffdshow if I recall correctly.

[poisondeathray]

Or the decoder could be de-interlacing. LAV filters have de-interlace options and also ffdshow if I recall correctly.

I asked about that earlier, he said it was disabled for LAV when doing the directshow testing .

Presumably he didn't enable it for playback if he was using a directshow based media player

[StainlessS]

In ExBlend thread, one user was wanting to deblend a clip whose blending originated in DirectShow (doing some kind of de-interlacing),
easiest way was to not use DirectShow, blending gone.

[Groucho2004]

I asked about that earlier, he said it was disabled for LAV when doing the directshow testing .Ah, OK. I did not read the entire thread.

[hello_hello]

manolito, I've only just started downloading the "gold standard" sample so I haven't compared the two yet, but the ffms2_rffmode_1 sample definitely has lots of combing. Frame #14631.

https://s33.postimg.cc/bn435p1cv/ffms2_rffmode_1.mpg_snapshot.jpg

[hello_hello]

I don't understand the "why" but I think I discovered the "what". I tested with poisondeathray's sample (he PM'd me a link) after remuxing the MKV as a TS file.

After opening the source with both DGIndex and ffms2, Info() shows TFF for DGIndex and assumed TFF for FFMS2.

I tried the following and it was obvious DGIndex and FFMS2 were repeating different fields.

FFVideoSource("E:\title00 (1).mkv", threads=1, rffmode=1)
SeparateFields()
A=last
B=A.SelectOdd().Subtitle("ffms2 odd")
C=A.SelectEven().Subtitle("ffms2 even")
mpeg2source("D:\title00 (1).d2v")
SeparateFields()
D=last
E=D.SelectOdd().Subtitle("DGIndex odd")
F=D.SelectEven().Subtitle("DGIndex even")
G=StackVertical(B,E)
H=StackVertical(C,F)
StackHorizontal(G,H)

AssumeBFF() for FFMS2 makes it repeat the same fields as DGIndex
Or instead of AssumeBFF(), you can tell TFM to Assume BFF and the result is the same after IVTC.

Best as I can tell using the ShowDiff function I borrowed from jagabo at VideoHelp, aside from what might be some decoding differences (which I've decided to call deblocking for the moment) the following outputs the same video.

FFVideoSource("E:\title00 (1).mkv", threads=1, rffmode=1)
AssumeBFF()
TFM().TDecimate()
A=last
mpeg2source("D:\title00 (1).d2v")
TFM().TDecimate()
B=last
ShowDiff(A,B)

As does this:

FFVideoSource("E:\title00 (1).mkv", threads=1, rffmode=1)
TFM(order=0).TDecimate()
A=last
mpeg2source("D:\title00 (1).d2v")
TFM().TDecimate()
B=last
ShowDiff(A,B)

Frame #14654 courtesy of ShowDiff() without and with AssumeBFF.

https://s33.postimg.cc/nw9saj0xb/title00_1_a.jpg

https://s33.postimg.cc/bueegdjz3/title00_1_b.jpg

function ShowDiff(clip Clip1, clip Clip2, bool "Amp", bool "Show", bool "Comp")
{
Amp = default(Amp, false)
Show = default(Show, false)
Comp = default(Comp, false)

N1 = Clip1.levels(96, 1.0, 160, 96, 160).greyscale()
N2 = Clip1.subtract(Clip2)
N3 = (Amp) ? N2.levels(124, 1.0, 131, 0, 255) : N2
N4 = (Show) ? N3.merge(N1) : N3
N5 = (Comp) ? Clip1.compare(Clip2, show_graph=true) : N4

return N5
}

[poisondeathray]

Nice detective work hello_hello . I can confirm it works. And the field order really is BFF with that ffms2 CPlugin version if you check with separatefields(). In hindsight we should have double checked and gone back to basics

Not sure about the minor decoding differences but those are negligible . I would call that an alternative working solution, at least on this disc .

Now why don't the other ffms2 versions work ?

I guess l33tmeatwad's version is technically the "newest" , although it's not on the official github page. But it works .

ffms2_r1140+101-avs+vsp => ok
FFMS2_2.32.1_MSVC l33tmeatwad => ok
ffms2-2.23.1-msvc => wrong framecount/sync issue
ffms2-2.23-clang => wrong framecount/sync issue
ffms2000-test8 => ok

[manolito]

Wow, this is getting exciting... :D

I can confirm that using a working ffms2 version and specifying BFF results in a perfect output stream (with rffmode=1). Nice...

I did stick with the MKV container as the source. When I extracted the video stream from the MKV and loaded it into DGIndex the info was that this stream was TFF.

What does this mean? When using ffms2 with rffmode=1 do I have to reverse the field order which is reported by DGIndex, or does ffms2 always expect BFF? I would like to get a universal solution which works with each and every MKV source created with MakeMKV. Unfortunately I do not own any NTSC DVDs with progressive 23.976 soft pulldowned content to test this...


Cheers
manolito

[hello_hello]

Now why don't the other ffms2 versions work ?

I guess l33tmeatwad's version is technically the "newest" , although it's not on the official github page. But it works .

ffms2_r1140+101-avs+vsp => ok
FFMS2_2.32.1_MSVC l33tmeatwad => ok
ffms2-2.23.1-msvc => wrong framecount/sync issue
ffms2-2.23-clang => wrong framecount/sync issue
ffms2000-test8 => ok

I wasn't clever enough to think about ffms2 versions and now I've looked I'm not 100% sure which one I was using.

MeGUI loads ffms2 from it's tools folder, and it's some flavour of version 2.23.1, but all I know for sure is the dll is dated 2016/12/29, the copying text file still says version 2.22 and the link within is this one. https://github./FFMS/ffms2

The Avisynth auto-loading folder has a dll dated 2017/05/24. I normally make sure any dlls MeGUI might load and the versions in the auto-loading folder are the same, but I think I put ffms2000 in the auto-loading folder for testing and forgot about it.

Fortunately, a quick check shows they both output the same number of frames for the sample when rffmode=1 (98912). The index files they create aren't the same size.

I'm pretty sure I left MeGUI's load plugin line in the script so I was probably testing with ffms 2.23.1. I'm just not sure which flavour it is.

I did stick with the MKV container as the source.

I did the same for ffms2. I only remuxed as a TS file for DGIndex.

[hello_hello]

What does this mean? When using ffms2 with rffmode=1 do I have to reverse the field order which is reported by DGIndex, or does ffms2 always expect BFF? I would like to get a universal solution which works with each and every MKV source created with MakeMKV. Unfortunately I do not own any NTSC DVDs with progressive 23.976 soft pulldowned content to test this...

I had one thought, although it's a long shot.....
MKV has an element for specifying field order. Unfortunately I've deleted poisondeathray's source file, but if someone still has it, it might be worth checking to see if the element it present and if it's correct. You can check with the MKVToolNix Header Editor.

None of the MKVs I checked contained the element, and after remuxing a vob file with MKVToolNix it wasn't automatically created, but I'm not sure which program was used to create poisondeathray's MKV.

Chances are FFMS2 doesn't look for that element even if it exists, but it's easy enough to eliminate it as the cause.

[manolito]

Good idea, but unfortunately no luck...

The MKV was created with MakeMKV, and the latest MKVToolNix says that there is no field order element in the header.

Neither MediaInfo (in debug mode) nor MKVInfo give any information about the field order. (I did not try ffprobe so far)

When you extract the video track from the MKV and feed it to DGIndex then you will get the info "TFF" when you play or preview the track. But when you create the D2V file and the log then the field order row is empty. To be absolutely certain you probably have to use the old AssumeXFF().SeparateFields() method and step through the fields.

What I found out in the meantime is that ffms2 with the rffmode=1 parameter outputs the opposite field order compared to MPEG2Source (with Honor Pulldown Flags). I tested this by slowing down a progressive PAL clip to 23.976 and then manually adding pulldown flags with DGPulldown. Here you can specify the field order, and when I selected TFF then ffms2 (with rffmode=1) delivered BFF and vice versa. Weird... I wonder what happens with a source which has field order transitions.


Cheers
manolito