Source filter behavior for progressive NTSC @23.976 with pulldown flags

[manolito]

Please forgive me if this question is stupid or has been discussed before, but I live in PAL land... :o

My source is a progressive NTSC clip @23.976 fps with standard 3:2 pulldown flags. I want to convert this clip to PAL. My arsenal of AviSynth source filters consists of DGIndex/DGDecode, DSS2Mod and DirectShowSource (LSmash does not run on my old computer).

Now I found that all of these source filters treat this source differently:
1: Mpeg2Source has "Honor Pulldown Flags" set by default. It correctly decodes to (pseudo-) interlaced 29.97 output. If I want the original 23.976 progressive output then I have to change the field operation to "Force Film".

2: FFMpegSource does not honor pulldown flags by default (There is an RFFMode option which can change this). It decodes to the original progressive 23.976 output.

I have no problem with the behavior of these two source filters, but DirectShowSource and also DSS2Mod do it differently.

3: On the first look DSS2Mod (DirectShowSource has the same behavior) seems to honor the pulldown flags, it decodes to 29.97. But unlike Mpeg2Source the output shows no combing. It looks progressive, but with tons of duplicate frames. So in reality the pulldown flags are not honored at all. This is not what anybody would want. To get the original 23.976 output without dupes I need to add "fps=23.976" to the source filter call.

Is there a way to bring the DSS2Mod behavior in line with the other source filters? Any (hidden) option to either honor pulldown flags or to force film?


Cheers
manolito

[manono]

Just use MPEG2Source after having created the D2V using DGIndex. Not what you asked, but it's the best way to handle MPG/VOB sources.

"My source is a progressive NTSC clip @23.976 fps with standard 3:2 pulldown flags."

If it's already 100% film (check the bottom of the D2V file), just make the D2V using the 'Forced Film' option and your video is already 23.976fps. An easy edit of the D2V will turn it into a Forced Film D2V. If it's less than 100% film but still not pure video, make the D2V using the default "Honor Pulldown Flags" and use TIVTC like so:

TFM(D2V="Movie.d2v")
TDecimate()

The effect will be (pretty much) to treat the soft pulldown parts as if they had been Forced Film, and to IVTC the rest.

Again, not what you asked but why try and force an inferior source filter?

[manolito]

Thanks manono,

this is how I already do it most of the time. Mpeg2Source has the additional benefit that it can automatically detect and repair field order transitions.

My question was mainly for AVStoDVD. Here users can choose between DSS2Mod, FFmpegSource and Mpeg2Source, and it would be very nice if it was possible to treat all these source filters identically without ruining the result. And most AVStoDVD users are not experienced, they cannot be expected to be able to change DGIndex options depending on the source.

MrC did change the procedure recently (after having to listen to my nagging way too long), and the current behavior is like this:

MediaInfo reports 23.976 progressive and 3:2 pulldown for such sources. AVStoDVD now evaluates the pulldown report and sets the source frame rate to 29.97 whenever pulldown is reported. Additionally it adds IVTC to the AVS script (TFM().TDecimate() ) for such sources.

This is correct for Mpeg2Source (Honor Pulldown Flags is the default). But for FFmpegSource and DSS2Mod the decoded result has no repeated fields, it is progressive with lots of dupes. But still applying IVTC seems to work nicely. TFM won't find any fields to match, but TDecimate reliably removes all the dupes so the end result is the original 23.976 progressive clip. Not very elegant, but the same procedure works for all the possible source filters.


Cheers
manolito

[manono]

I didn't understand much of that and have never used AvsToDVD. But if you're looking for a one-size-fits-all approach to MPG/DVD sources, then the script I showed earlier works well. Just make all D2Vs using Honor Pulldown Flags followed by:

TFM(D2V="Movie.d2v")
TDecimate()

You won't have the mistakes sometimes made by the field matching when IVTCing everything and, if it's all soft pulldown, it's nearly (not quite) as fast as using Forced Film to make the D2V to begin with.

If you're looking for a one-size-fits-all approach for all sources, then I don't know what to tell you and maybe someone else can answer your questions.

[FranceBB]

My source is a progressive NTSC clip @23.976 fps with standard 3:2 pulldown flags. I want to convert this clip to PAL.

I have a sample which is 23.976 with 3:2 pulldown in my network drive to play with.
This is what I would do to convert it to PAL:

#Index video and audio 29.970fps progressive with dups
video=DGDecode_MPEG2Source("I:\Production\RAW\test.d2v")
audio=FFAudioSource("I:\Production\RAW\audio T80 2_0ch 224Kbps DELAY 0ms.ac3")
AudioDub(video, audio)

#Decimate to original 23.976 progressive
tfm(mode=1,pp=5,slow=2,micmatching=2,clip2=tdeint(mode=2,type=3))
Tdecimate(mode=2, rate=23.976)

#Resize from 720x480 to 720x576 using NNEDI
nnedi3_rpow2(cshift="Spline64ResizeMT", rfactor=2, fwidth=720, fheight=576, nsize=4, nns=4, qual=1, etype=0, pscrn=2, threads=0, csresize=true, mpeg2=true, threads_rs=0, logicalCores_rs=true, MaxPhysCore_rs=true, SetAffinity_rs=false, opt=3)

#Blending to 50fps and interlacing to 25i
ConvertFPS(50)
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()

Some people just don't like blending.
I don't think it's actually that bad, especially in motion and I do it all the time at work, but if you prefer a different approach, you can just do a speed up with pitch adjustment and encode the progressive 25fps file as interlaced:

#Speed up 4% with pitch adjustment
ResampleAudio(48000)
AssumeFPS(25, 1, true)
SSRC(48000)

As to the DirectShowSource behaviour, I rarely use it. I mainly use FFMpegSource2 for general purpose contents and DGIndex for old MPEG-2 files.

[Cary Knoop]

Some people just don't like blending.
I don't think it's actually that bad, especially in motion and I do it all the time at work, but if you prefer a different approach, you can just do a speed up with pitch adjustment and encode the progressive 25fps file as interlaced:

I am one of them (who does not like blending).

Also, it is uncommon in a professional setting, 24p (or 23.976p) is always speed converted to 25p.
If it isn't I personally would call it a sloppy job.

[FranceBB]

23.976p is always speed converted to 25p.
If it isn't I personally would call it a sloppy job.

If the content is a tv series or a movie that needs to be aired from start to end on a linear channel, then yes, absolutely, I do speed up with pitch adjustment, but whenever I have to use a few scenes of a series/movie to cover something in the news, I just blend (due to people saying "chop chop", journalists pushing and me being lazy).

[manolito]

TFM(D2V="Movie.d2v")
TDecimate()


Thanks manono for bringing up the d2v parameter for TFM. I will definitely lobby MrC to add it to AVStoDVD.

Otherwise AVStoDVD tries to be as universal as possible, and DGIndex / DGDecode cannot always be used for sources with pulldown flags. An example is a NTSC DVD title. The recommended procedure is to rip the title to MKV using MakeMKV, because this way all the chapters and subtitles are retained. When this MKV is fed to AVStoDVD only DSS2Mod or FFmpegSource can be used as source filters.


@ FranceBB
Thanks for the suggestions. I believe though that you really should insert "TFM(d2v="{your D2V file}") before the TDecimate call, because your source will have repeated fields (with the default "Honor Pulldown Flags").


A little OT, but just the other day I discovered an old thread at VideoHelp with an alternative method to convert progressive 23.976 to progressive 25. This comes from gavino, it gives perfect results, no speedup required, the slight blending is almost invisible. It looks like this:
f1=ChangeFPS(24)
f2=Trim(0,-1).AssumeFPS(24) + Trim(1,0).ChangeFPS(24)
Film=Merge(f1,f2)
p1=ChangeFPS(Film,25)
p2=Trim(Film,0,-1).AssumeFPS(25) + Trim(Film,1,0).ChangeFPS(25)
Pal=Merge(p1,p2)
Return Pal

The original thread is here:
https://forum.videohelp.com/threads/322430-Frame-rate-conversions-How-to-blend-select-frames-in-AVISynth

I prefer this method over using 23.976 -> 25 pulldown because the output stays progressive, audio does not need to be touched, and I do not get motion judder. The problem with PAL Speedup is that pitch correction only sounds good when using highest quality commercial tools (like iZotope RX). The AviSynth TimeStretch plugin can sound awful for music.


Cheers
manolito

[Cary Knoop]

The problem with PAL Speedup is that pitch correction only sounds good when using highest quality commercial tools (like iZotope RX).
I think that is nonsense.

Use SoX, which is free.

sox <in> <out> tempo 1.0427

Does the job just fine.

http://sox.sourceforge.net/

[manolito]

Have you tried this with music content? With string layers? Didn't you notice the flanging effect?
Why do you think that the pros spend big bucks for pitch changing tools?

[wonkey_monkey]

Also, it is uncommon in a professional setting, 24p (or 23.976p) is always speed converted to 25p.
If it isn't I personally would call it a sloppy job.

Torchwood: Miracle Day as shown on the BBC is the only example I can think of. I can only imagine it's because they either didn't want to change the running time (by a whole 2.5 minutes) or didn't want to pitch shift the voices, which everyone would be too familiar with as characters from a previously 25p production.

[Cary Knoop]

Have you tried this with music content? With string layers? Didn't you notice the flanging effect?
Why do you think that the pros spend big bucks for pitch changing tools?

Here is a sound clip and a time stretched Sample-A and Sample-B.

https://www.dropbox.com/sh/2hofgu1nnff3cgv/AAC7XG2AZbiU02l0GPIj1ay0a?dl=0

Can you hear which one is the Rx6 and which one is the SoX? And do you think any of them is inferior?

[Cary Knoop]

Torchwood: Miracle Day as shown on the BBC is the only example I can think of. I can only imagine it's because they either didn't want to change the running time (by a whole 2.5 minutes) or didn't want to pitch shift the voices, which everyone would be too familiar with as characters from a previously 25p production.
You can avoid shifting the pitch.

[wonkey_monkey]

You can avoid shifting the pitch.

You can, but the BBC still typically don't.

[Cary Knoop]

You can, but they still typically don't.
They typically do!

Pitch shifting is far more noticeable than time stretching.

[Cary Knoop]

Torchwood: Miracle Day as shown on the BBC is the only example I can think of. I can only imagine it's because they either didn't want to change the running time (by a whole 2.5 minutes) or didn't want to pitch shift the voices, which everyone would be too familiar with as characters from a previously 25p production.
According to IMDB Torchwood was recorded in 25p.
What is it you think they have not done?

https://www.imdb.com/title/tt0485301/technical?ref_=tt_dt_spec

[poisondeathray]

(And you can probably avoid doing anything ; 99.999% of PAL DVD players play NTSC discs just fine)

DSS2 /mod or any directshow derivative will be at risk of whatever the system has installed and whatever settings are set. If the directshow filter is set to deinterlace it will deinterlace, degrading the image . Maybe that's why there is no combing. Can you be sure it' s not actually doing other stuff ? Not really, unless you personally setup and configured your filters directly. If some general user doesn't know what's going on or anything about this stuff - I would make the smarter choices for them if I was making a GUI - and make choosing the bad options more difficult or impossible . DSS2mod tends to drop a frame at the end too . Really inconsistent . Avoid.

[wonkey_monkey]

According to IMDB Torchwood was recorded in 25p.
What is it you think they have not done?

https://www.imdb.com/title/tt0485301/technical?ref_=tt_dt_spec

Series 1 and 2, and the Children of Earth miniseries were all shot 25p in the UK as a UK production.

Miracle Day (series 4) was largely a US production shot at 23.976fps. When the BBC broadcast it, frames/fields were blended to make it 25p instead of doing the usual PAL speed-up. Audio was presumably untouched.

The BBC don't show many US imports these days (apart from films), but when they do they are usually PAL sped-up with audio at a higher pitch.

Channel 4, on the other hand, have in the past shown both Stargate and Enterprise with (not very good) pitch correction but I don't think they've used any correction lately with, for example, Agents of Shield.

[Cary Knoop]

Miracle Day (series 4) was largely a US production shot at 23.976fps. When the BBC broadcast it, frames/fields were blended to make it 25p instead of doing the usual PAL speed-up.

While it appears they blend frames they prefer others not to do it:

"Speed change is the preferred method of converting from 24fps (including 23.976fps) to 25fps. Due attention must be given to the audio. "

Source: http://dpp-assets.s3.amazonaws.com/wp-content/uploads/specs/bbc/TechnicalDeliveryStandardsBBCFile.pdf (Page 9).

The BBC don't show many US imports these days (apart from films), but when they do they are usually PAL sped-up with audio at a higher pitch.

I assume you know this for a fact (that the pitch is higher).
Then shame on the BBC, when you speed up the audio I think you should really compensate for the pitch shift.

[wonkey_monkey]

But it does cause artefacts, no matter how good the resampling. And decent resampling hasn't been around all that long, so plain speed-up has been the de facto standard for years, and no-one really notices anyway. For someone like the BBC it's probably far preferable to stick with a simple, guaranteed clean method that everyone can be told to stick to than mess around and get it wrong like Channel 4 has been known to do - both Stargate and Enterprise never sounded great, and they once played out a film with a horribly mangled soundtrack because something had gone wrong with pitch-shifting somewhere along the line.

[Cary Knoop]

But it does cause artefacts, no matter how good the resampling. And decent resampling hasn't been around all that long, so plain speed-up has been the de facto standard for years, and no-one really notices anyway. For someone like the BBC it's probably far preferable to stick with a simple, guaranteed clean method that everyone can be told to stick to than mess around and get it wrong like Channel 4 has been known to do - both Stargate and Enterprise never sounded great, and they once played out a film with a horribly mangled soundtrack because something had gone wrong with pitch-shifting somewhere along the line.
A 4% pitch shift is definitely noticeable while those suggested artifacts, well do you hear any artifacts in the comparison below?
https://forum.doom9.org/showthread.php?p=1850449#post1850449

[manolito]

You really cheated on this one. Your source file is not demanding at all, just about any pitch correction software handles this one.

What about this source?
It is the intro from the Yellowjackets Greenhouse track. Ripped from the original CD (no intermediate MP3). One of the most excellent analog recordings ever by Jan Erik Kongshaug.
Download here:
https://www.zeta-uploader.com/1143320492

I do not own iZotope RX, the two conversions are by SoX and the integrated TimeStretch plugin from AviSynth 2.61 Alpha.

For the first 30 seconds they are tolerable but after that listening becomes really painful for both pitch corrected versions. Absolutely unusable.


Cheers
manolito

[Cary Knoop]

You really cheated on this one.
If that is the way you think you can have a respectful discussion you have it wrong.

[manolito]

(
DSS2 /mod or any directshow derivative will be at risk of whatever the system has installed and whatever settings are set. If the directshow filter is set to deinterlace it will deinterlace, degrading the image . Maybe that's why there is no combing. Can you be sure it' s not actually doing other stuff ? Not really, unless you personally setup and configured your filters directly. If some general user doesn't know what's going on or anything about this stuff - I would make the smarter choices for them if I was making a GUI - and make choosing the bad options more difficult or impossible . DSS2mod tends to drop a frame at the end too . Really inconsistent . Avoid.

This may be your personal experience - mine is totally different... :devil:

I use DSS2Mod together with LAV Filters. I set it up using all the default settings, so there is no deinterlacing whatsoever. And still the 23.976 clip with pulldown flags gets decoded to 29.97, but not with the repeated fields, instead it did ignore the RFF flags and duplicated frames to reach the 29.97 frame rate. I have no idea if this would be different when using ffdshow instead of LAV Filters, but ffdshow is not an option for me.

And your advice to avoid DSS2Mod because it might skip a frame at the end or maybe at the beginning is totally misguided. Nobody cares about a skipped frame at the end as long as there are no decoding artifacts and no audio sync problems. And from my experience DSS2Mod (with a reasonable Prefetch value) is much more reliable than ffms2.


Cheers
manolito

[manolito]

If that is the way you think you can have a respectful discussion you have it wrong.

You started being disrespectful by telling me that my experience with free pitch correction tools was nonsense.
https://forum.doom9.org/showthread.php?p=1850424#post1850424

[FranceBB]

believe though that you really should insert TFM before the TDecimate call, because your source will have repeated fields (with the default "Honor Pulldown Flags").

True. Fixed:

tfm(mode=1,pp=5,slow=2,micmatching=2,clip2=tdeint(mode=2,type=3))

I should stop replying at 2am in the morning xD

just the other day I discovered an old thread at VideoHelp with an alternative method to convert progressive 23.976 to progressive 25. This comes from gavino, it gives perfect results, no speedup required, the slight blending is almost invisible.

If I understand correctly, that's because it actually blends 1 frame from 23.976 (24) to 25, making a 25fps progressive.
That's a different approach, but it can be done, it's not a "big deal".
Ideally, whenever I have to blend, I always blend to 50fps and then divide in fields to get a truly interlaced 25i.
Blending just 1 frame every second actually looks... kinda... weird to me.
I don't know how to explain that, but it's like... you get a single frame with something that "doesn't feel right" and my mind seems to be focussed on that.
When I blend to 50 and divide in fields, though, it's like having a truly interlaced 25i content and since there are "many" blended frames, my mind doesn't notice.
I don't know how to explain this, but anyway the method you found is a way to do it and if it looks good to you, go for it.

As to the speed-up with pitch adjustment, there are many ways to do it instead of using Avisynth; some are actually better for some contents, some are better for others, but it's still about compromises.
What I can say, however, it's that even with professional tools like the Sony DP600 I have at work, a speed up with pitch adjustment it's still gonna be noticeable if you compare it with the original version, but hey, there's nothing we can do about it.
The point is that it's always gonna be different, no matter what you use.
Sure, you can fine-tune it, but it's still gonna be noticeable.
Anyway, for many people, it's gonna be fine.
Honestly, in my whole career (it's not long, though xD) I've never received a complaint by a user at home that didn't like a speed-up.

So, in the end, it's all about compromises.
You still have to convert from a frame-rate to another and there's always going to be a catch:

- do you wanna use blending? It's gonna look a bit odd in some scene-changes.
- do you wanna use the speed-up + pitch adjustment? It's gonna be noticeable by someone and it's gonna sound slightly different.
- do you wanna use motion interpolation? It may introduce artifacts.
- do you wanna duplicate frames? It's gonna stutter.

In other words, it doesn't matter how much time you spend trying to figure out a perfect method, it's still gonna have some "issues", the point is trying to figure out which issue looks better for you. ;)

[poisondeathray]

This may be your personal experience - mine is totally different... :devil:


Says the guy with limited experience, living in PAL land asking for help with NTSC discs :D :devil:

It's not just my personal experience - those are the facts . Directshow is inconsistent. It's dependent on how the user has it configured. User A might have it configured differently that user B.


I use DSS2Mod together with LAV Filters. I set it up using all the default settings, so there is no deinterlacing whatsoever. And still the 23.976 clip with pulldown flags gets decoded to 29.97, but not with the repeated fields, instead it did ignore the RFF flags and duplicated frames to reach the 29.97 frame rate. I have no idea if this would be different when using ffdshow instead of LAV Filters, but ffdshow is not an option for me.

And did you check the final output closely with tdecimate or decimate ? :D because I bet you are in for a surprise. Check the motion

Sure you might have some knowlege of setting up directshow filters , so you might be predisposed to only some of the problems - but what about Joe Public? This is in the context that you brought up - ie. some general user using a GUI like avs2dvd. What if he had some filters activated? Things like deinterlacing, denoising, saturation, color correction? There are other directshow decoding filters too. In your small world you were only thinking of LAV and ffdshow, but there are others.

In LAV alone, you might have MPEG2 decoding set to HW CUVID, or quicksync, or DXVA, on CPU . They don't necessarily give the same results. The inconsistency is a deal breaker for many people . The only consistent thing about Directshow is it's consistently inconsistent :)



And your advice to avoid DSS2Mod because it might skip a frame at the end or maybe at the beginning is totally misguided.

But that was only one of the many reasons :) . That issue affects basically everything, even outside of the NTSC MPEG2 DVD soft telecine scenario - it frequently occurs in many other scenarios and video types


Nobody cares about a skipped frame at the end as long as there are no decoding artifacts and no audio sync problems.


That's about as ignorant as saying something like: people don't care about audio pitch shift issues as long as it's in sync. Sounds close enough right ?? :D :devil:

Maybe you don't care, but there are other people DO care about these problems. And I' m not just talking dropped frames, occasional green frames (rare, but usually only at the beginning or end), but the other potential issues as well.

How about jerky playback for one ? Frequently there jumps in motion when using dss2mod or any directshow derivative with tdecimate when used in this NTSC MPEG2/DVD soft telecine scenario. It just chooses the wrong frame so you get the incorrect duplicated frames sometimes, and dropped wrong frame others. This is not even talking about non linear seeking (regardless of preroll). This is complete frame by frame linear seeking, linear encode, no other filters. The encodes are jerky and botched. It's easy to reproduce, regardless of CPU/GPU HW directshow setting. Let me know if you want any samples to chew on :D But I bet the one you have now already shows the problems if you look closely




And from my experience DSS2Mod (with a reasonable Prefetch value) is much more reliable than ffms2.

Actually, neither are particularly reliable for soft telecine NTSC MPEG2 from DVD . ffms2 tends to have fewer jerky spots, but still there are sections that are jerky. I really suggest you look closer. MKV container actually makes it worse for some issues. MPEG2 in MKV is known a bad combo prone to issues

You have a proven, reliable and consistent method with DGIndex. Instead you suggest using buggy, unreliable methods? I don't see the point. It's one thing to make flawed stuff for yourself, but to suggest others to do so too is just wrong.

Cheers

[wonkey_monkey]

A 4% pitch shift is definitely noticeable while those suggested artifacts, well do you hear any artifacts in the comparison below?

A 4% pitch shift is not noticeable if you're not doing an A/B test - okay, if you're very familiar with the speaker, it might be, but usually not. Just as the 4% change in video speed isn't noticeable.

And it's guaranteed to work perfectly every time, with every input, which are extremely valuable attributes to have.

[SeeMoreDigital]

For anyone interested...

I've used both TSmuxer GUI and UsEac3to to remove 3:2 pull-down from all of my old 'movie' NTSC DVD sources and some Blu-ray disc sources.

UsEac3to is also pretty good at changing the speed of audio streams too ;)

[Sharc]

A 4% pitch shift is not noticeable if you're not doing an A/B test ….
Hmm …. how about people who have "absolute pitch"?
https://en.wikipedia.org/wiki/Absolute_pitch

[Katie Boundary]

If it's less than 100% film but still not pure video, make the D2V using the default "Honor Pulldown Flags" and use TIVTC like so:

TFM(D2V="Movie.d2v")
TDecimate()

If the content was originally 100.00% film, and the small amounts of non-film content are the result of an incorrectly applied pulldown pattern, then I'd specify this:

TFM(mode=0,pp=0,d2v="movie.d2v",micmatching=0).TDecimate()

It will improve accuracy (negligibly) and save a few CPU cycles.

blah blah speedup, blah blah pitch shifting, blah blah blending

Or you could rescale to 720x576, separate fields, duplicate one field from every 24th frame, rearrange the fields into a sensible order using Selectevery, and weave them back together. Example:

assumefps(24).spline144resize(720,576).separatefields()

selectevery(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,22,25,24,27,26,29,28,31,30,33,32,35,34,37,36,39,38,41,40,43,42,45,44,47,46,47)

Weave()

[manolito]

Thanks everybody, but my goal is not to apply manual tweaks to achieve optimal results. Instead I need a method which works under a One-click-Software like AVStoDVD for all the available source filters (MPEG2Source, DSS2Mod, ffms2 and DirectShowSource).

I think I know quite well how to do this for sane sources which have a regular pulldown pattern.

From the DGIndex manual for the "Ignore Pulldown" option:
However, because repeated fields intended for display are ignored and not displayed, the resulting frame rate may differ from the source frame rate. It may even vary throughout the clip, due to irregular patterns of pulldown flags. If the pulldown is irregular, use of this option will cause the audio-video sync to change at different parts of the clip, and most likely sync will not be acceptable

What do I do with sources like these? I do not have any such source files with irregular pulldown patterns to test. So if anyone could upload such a source I would be really grateful...


Cheers
manolito

[mp3dom]

You need to use TFM specifying the d2v parameter. TFM will take the rff indication inside the d2v file and will field-match the other parts to still match the frame rate.
Irregular pulldown is quite common if the ivtc is made by the encoder, because it's conservative, so in case of any doubt, it will simply encode the frame as is without using rff flags.

[manolito]

Yes, this is the preferred and safest method, but more than often you don't have a d2v file because the source is in a different container. For converting DVD titles the easiest way is to use MakeMKV which removes copy protection, repacks the VOBs into an MKV container while retaining all chapters and subs.

I know that poisondeathray thinks that "MPEG2 in MKV is known a bad combo prone to issues", but in my experience he is pretty much the only person who says this. MakeMKV has been around for a while now, and the current versions do a perfect job.

Again my request: Could anyone upload a short clip which has this "irregular pulldown", so I can experiment with it using other source filters than DGIndex / DGDecode?


Cheers
manolito

[videoh]

http://rationalqm.us/misc/IrregularPulldown.ts

It's a mix of hard and soft pulldown, so the repeat flags are irregular. Is that what you are looking for?

[poisondeathray]

I know that poisondeathray thinks that "MPEG2 in MKV is known a bad combo prone to issues", but in my experience he is pretty much the only person who says this. MakeMKV has been around for a while now, and the current versions do a perfect job.



MKV is great if you just keep it like that to watch or archive. To be clear, the problems occur when you use it with avisynth as DVD/MPEG2 in MKV with source filters.

Problem(s) go away when you demux it and use standard dgindex method. Many of the old timers know about this. I know manono knows about it. When people upload DVD samples in MKV for examination in various threads, people know to demux it and use dgindex because of these known issues. I'm surprised you don't know about it, because you're old timer too :)

I would say it occurs more than "occasionally", but less than "very frequent" . Frequent enough to never use directshow or ffms2 in this scenario. This DVD/MPEG2 in MKV combo causes a reproducible flaky behaviour with avisynth source filters. Problems such as mixed up frames, jerky decimation/duplication, combing on progressive sources that you otherwise wouldn't get (even with pp=0 in tfm with using mpeg2source/tivtc) .

[videoh]

I would say it occurs more than "occasionally", but less than "very frequent" . Frequent enough to never use directshow or ffms2 in this scenario. This DVD/MPEG2 in MKV combo causes a reproducible flaky behaviour with avisynth source filters. Problems such as mixed up frames, jerky decimation/duplication, combing on progressive sources that you otherwise wouldn't get (even with pp=0 in tfm with using mpeg2source/tivtc) . Just curious, poisondeathray. Do these issues also arise when DGDecNV is used, the point being that DGDecNV opens MKV directly without demuxing? If so, it could be something to be investigated.

[wonkey_monkey]

I thought MPEG2 caused those problems with DSS and FFMS2 in any container.

[manono]

I know that poisondeathray thinks that "MPEG2 in MKV is known a bad combo prone to issues", but in my experience he is pretty much the only person who says this.

Nope, I agree with him 100%. When someone makes available a sample from a DVD in an MKV container, all it does is make me mad because I have to first extract the M2V before then running it through DGIndex so I can use MPEG2Source on it. Making an MKV out of it screws it up royally. You just can't work with it much of the time.

When using a DVD as a source, one should use a proper decrypter.

[poisondeathray]

Just curious, poisondeathray. Do these issues also arise when DGDecNV is used, the point being that DGDecNV opens MKV directly without demuxing? If so, it could be something to be investigated.

Not sure, it occurred so frequently in the past that I know many people automatically demux the mkv by knee jerk reflex . It might be worth investigating


I thought MPEG2 caused those problems with DSS and FFMS2 in any container.
Yes, but not necessarily "generic" MPEG2 .

MPEG2 from other sources (not DVD) in other containers do not necessarily have these problems . For example, MPEG2 cameras (XDCAM in MP4 or MXF work fine with ffms2)

But in the case of the DVD/MPEG2 MKV - You can demux it and it will still have at least some of the problems when using ffms2/directshow. (Ignoring the additional problems caused by DSS for now, we are just taking about field matching or frame order)




Another thing is there might have been improvements to the mkv container spec and/or makemkv , so it might be worth revisiting. manolito said "current versions" of makemkv. Maybe some special something was applied in the last few years. I doubt it. I can rerip some discs done a few years ago but I don't think it will make a difference. I know mkv's made by mkvtoolnix v26 (today) still exhibit this issue when used with ffms2 or directshow (regardless of the workarounds like threads=1 for ffms2, or dss2mod with prefetch any value)

[wonkey_monkey]

Nope, I agree with him 100%. When someone makes available a sample from a DVD in an MKV container, all it does is make me mad because I have to first extract the M2V before then running it through DGIndex so I can use MPEG2Source on it. Making an MKV out of it screws it up royally. You just can't work with it much of the time.


I see I get picture errors when I try. It carries on for a bit, then halts. Initial loading and scrubbing seems fine, though.

DGIndexNV, however, has no such issues.

[manolito]

http://rationalqm.us/misc/IrregularPulldown.ts

It's a mix of hard and soft pulldown, so the repeat flags are irregular. Is that what you are looking for?

Thanks for uploading this clip...

But no, this is not what I am looking for, I want a clip where MediaInfo reports 23.976 progressive AND 3:2 pulldown. Like the clips I made myself from progressive PAL sources which I slowed down to 23.976 and then applied pulldown (either by HCenc or DGPullown). These self-made clips in an MKV container show no (or almost no) problems using ffms2 or DSS2Mod as the source filter. But some folks here are adamant that this method is evil and only MPEG2Source can handle such clips. And I want proof of this...

The clip you uploaded is interesting, though, and I did play with it a little bit. All the following is from the perspective of a AVStoDVD user who will not have any expertise on such sources.

Loading the TS into AVStoDVD prompts the user to index it with DGIndex. Did this, it then shows up in the input window as 29.97 progressive. Of course this is MediaInfo's fault, but in AVStoDVD MediaInfo is all we got. I did not even try to start a conversion with these settings, but the average user would, and he would get a very ugly result. Adding IVTC manually to the script solved it of course, but again an average AVStoDVD user would never be able to do this.

Next thing I tried was to repack the TS to MKV. To my surprise the current version of mkvmerge failed to do this. It could not detect a video stream in this TS file. Weird...
FFmpeg was able to do this, and feeding the resulting MKV to MediaInfo now showed that it was 29.97 Interlaced BFF. Loading it into AVStoDVd was no problem, the default was DSS2Mod as the source filter, interlaced encoding set by default. Which is a good choice for DVD output. Of course there was also the option to apply deinterlacing.

So AVStoDVD users would be able to get a decent conversion result from this clip after repacking the clip into an MKV container. Only the second best choice, but very watchable.


Cheers
manolito

[manolito]

Nope, I agree with him 100%. When someone makes available a sample from a DVD in an MKV container, all it does is make me mad because I have to first extract the M2V before then running it through DGIndex so I can use MPEG2Source on it. Making an MKV out of it screws it up royally. You just can't work with it much of the time.

When using a DVD as a source, one should use a proper decrypter.

All this is because you guys are so damned preoccupied with using MPEG2Source (and nothing else) when it comes to MPEG2 sources. And are you really saying that MakeMKV is not a proper decrypter? Think again...

Can you please try for a minute to put yourself into the shoes of an average user who knows nothing about all these inner workings of the different source formats. Of course all you guys know (and I know, too, I have been around for a while) how to manually demux a DVD title, then extract the chapters using a different tool, and then extract the subs with yet another tool. But there are other users who will use one-click tools like AVStoDVD, and they expect a decent conversion result even for difficult sources.

Now back to the original issue:
In AVStoDVD using MakeMKV to convert DVD titles is by far the most convenient method. This means that MPE2Source cannot be used, but I think that this is no big problem, even with soft telecined NTSC sources. During the last couple of days I did make many such conversions using DSS2Mod and ffms2, and they all came out pretty good...

For such sources MediaInfo reports Progressive 23.976 AND 3:2 Pulldown. DSS2MOD and ffms2 both ignore the pulldown flags (can be changed in ffms2 by using the RFFMode=1 parameter). For DSS2Mod I use DSS2("my_source", fps=23.976, preroll=15). The fps parameter forces the frame rate, if the source has issues then you may get some duplicated or dropped frames. But this is only noticeable if you step through the frames. And this makes sure that there will be no audio sync problems.

I believe that this method does what most users want. It may not be perfect, but IMO it is "Good Enough".

And if any of you guys can provide a source clip where this method fails miserably, go ahead and upload such a clip.


Cheers
manolito

[poisondeathray]

I want a clip where MediaInfo reports 23.976 progressive AND 3:2 pulldown. Like the clips I made myself from progressive PAL sources which I slowed down to 23.976 and then applied pulldown (either by HCenc or DGPullown). These self-made clips in an MKV container show no (or almost no) problems using ffms2 or DSS2Mod as the source filter. But some folks here are adamant that this method is evil and only MPEG2Source can handle such clips. And I want proof of this...



Here you go, fresh re-rip using newest makemkv (in case it did something new and magical the last few years), newest mkvtoolnix to strip everything else out and cut a 5min30sec sample. (from the beginning in case there were issues caused by cutting the in point)
http://www.mediafire.com/file/ke3pjoubowoxlod/title00+%281%29.mkv


Original frame rate : 23.976 (24000/1001) FPS
Standard : Component
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Scan order : 2:3 Pulldown


I made sure to use single treaded, no prefetch. DSS2Mod/Directshowsource/any DS derivative fails miserably (jerky motion). Completely unusable with lav splitter, lav decoder (cpu, hw or otherwise), regardless of settings, regardless of DSS2 preroll, tested on multiple configurations and computers.

DSS2("title00 (1).mkv", preroll=X)
AssumeTFF().TDecimate()
#AssumeTFF().TFM(pp=0).TDecimate() #Doesn't matter if you include TFM, still jerky
#AssumeTFF().Decimate() #Doesn't matter if you use Decimate instead of TDecimate, still jerky


FFMS2 does better here than directshow derivatives, but notice combing in a few frames ~ 2:54-2:55 as the ship goes up. It does not matter if you demux to ES, or leave it in MKV. Tried multiple ffms2 versions.

FFVideoSource("title00 (1).mkv", threads=1)


If you try FFMS2 with rffmode=1 (so it honor flag), with TIVTC, it becomes jerky in some parts and lots of combing (remember we set PP=0 to disable post processing to check)


The "gold standard" . Demuxed mkv, then DGIndex using Honor flags. Notice I put PP=0 to disable post processing and comb detection deinterlacing. Notice no combing where ffms2 had problems. (AssumeTFF() is redundant because MPEG2Source passes that info, but I left it in to be complete to compare to DSS)

MPEG2Source("title00 (1)_track1_eng.d2v", cpu=0)
AssumeTFF().TFM(pp=0).TDecimate()





It does not matter if other valid decrypting methods are used (e.g. anydvd, dvd decrypter, dvd shrink, etc...this is not a decryption issue). There are many examples over the years in various threads. If I re-rip a random DVD , you will see some problems , quite frequently when using DSS or FFMS2. (But not "always" - or maybe I missed some errors that were there; it occurs so frequently , so repeatedly, that many people just don't waste time using FFMS2 and especially DSS, for this DVD/MPEG2 scenario). Granted, FFMS2 wasn't as badly affected in this specific example, but it can be, where frames are actually misplaced or mixed up in order. Why is FFMS2 prone too? It uses an index... My understanding is that the difference is because indexing is different ; DGIndex does something with bytes, but the ffms2 index isn't as robust. I'm sure videoh or myrsloik can clarify the exact details if you're interested.

https://forum.doom9.org/showthread.php?p=1835524#post1835524
Myrsloik:

Basically FFMS2 makes a lot of assumptions like timecodes are correct and that the libavformat demuxers can always seek accurately. Other source filters like the d2v using ones don't and instead more or less index which specific bytes are needed by each frame (or something thereabout).


Eitherway, the DGIndex method is the most consistent for DVD/MPEG2. By far. Directshow is prone to a few issues (this is well known), and ffms2 might be slightly better but not immune to problems. I haven't even gone into other issues using filters. There are sometimes compounding errors when you have filters that require non linear access. Things such as temporal filters. ffms2 can be slightly improved in that regard in general (more robust seeking) if you use seekmode=0 (slower). Also some of the other ffms2 issues can be from MKV container (that's why I always test ES too), because it makes some assumptions about timecodes. Well sometimes the MKV timecodes are just buggy and maybe that's why, not sure, but this has been shown to improve some of the problems with samples in other threads over the years; but not all the problems everytime.






Can you please try for a minute to put yourself into the shoes of an average user who knows nothing about all these inner workings of the different source formats. Of course all you guys know (and I know, too, I have been around for a while) how to manually demux a DVD title, then extract the chapters using a different tool, and then extract the subs with yet another tool. But there are other users who will use one-click tools like AVStoDVD, and they expect a decent conversion result even for difficult sources.

Now back to the original issue:
In AVStoDVD using MakeMKV to convert DVD titles is by far the most convenient method. This means that MPE2Source cannot be used, but I think that this is no big problem, even with soft telecined NTSC sources. During the last couple of days I did make many such conversions using DSS2Mod and ffms2, and they all came out pretty good...

For such sources MediaInfo reports Progressive 23.976 AND 3:2 Pulldown. DSS2MOD and ffms2 both ignore the pulldown flags (can be changed in ffms2 by using the RFFMode=1 parameter). For DSS2Mod I use DSS2("my_source", fps=23.976, preroll=15). The fps parameter forces the frame rate, if the source has issues then you may get some duplicated or dropped frames. But this is only noticeable if you step through the frames. And this makes sure that there will be no audio sync problems.

I believe that this method does what most users want. It may not be perfect, but IMO it is "Good Enough".

And if any of you guys can provide a source clip where this method fails miserably, go ahead and upload such a clip.


But why can't AVS2DVD be coded to demux the MKV and use DGIndex? Other GUIs can if I'm not mistaken. I think megui can

[poisondeathray]

For DSS2Mod I use DSS2("my_source", fps=23.976, preroll=15). The fps parameter forces the frame rate, if the source has issues then you may get some duplicated or dropped frames. But this is only noticeable if you step through the frames. And this makes sure that there will be no audio sync problems.


aha ! fps=23.976 fixes the jerky issues . Still combing in the cockpit up scene, but that' s minor compared to jerky

EDIT: spoke too soon... it completely messes up other sections in the larger movie, where every 2nd frame is dropped, and duplicated. I'll try to see if I can isolate an example


I cannot reproduce on a cut sample for some reason, only the whole movie. Even a 10 min sample is not enough to produce the issue. When loading the whole movie with dss2("full.mkv" , fps=23.976, preroll=15), there are sections that have ~ every 2nd frame dropped and replaced with a duplicate frames. It's like the errors just get pushed later into the movie for several scenes. But then other sections recover perfectly, then later on others have it again. It affects both x86 avisynth classic or mt (in single thread mode) , or avs+ x64 (in single thread mode) . Affects different lav settings (CPU, HW etc..). Very flaky behaviour.

ffms2 has 64 extra frames. The running time is ~2.6 seconds longer and out of sync if you use FFAudioSource/FFVideoSource with audiodub. Progressive sync issue, simple shift won't fix. It's not immediately clear where the extra frames are inserted, some here some there. (not an easy fix) . Some jerky frames (fwd/back) too (even when going way back past a few GOP's then "sneaking" up on the section). It might be those jerky frames are the inserted frames.

Both are unusable on the whole movie

But DirectShowSource("full.mkv", fps=23.976) doesn't have those dropped/duplicated frames sections. A grey frame at the end. But that might be semi- usable with only minor issues if you did a straight linear encode, no filters.

[manolito]

aha ! fps=23.976 fixes the jerky issues . Still combing in the cockpit up scene, but that' s minor compared to jerky

Maybe you should have read some of my earlier posts... :devil:

From the opening post of this thread:
3: On the first look DSS2Mod (DirectShowSource has the same behavior) seems to honor the pulldown flags, it decodes to 29.97. But unlike Mpeg2Source the output shows no combing. It looks progressive, but with tons of duplicate frames. So in reality the pulldown flags are not honored at all. This is not what anybody would want. To get the original 23.976 output without dupes I need to add "fps=23.976" to the source filter call.

DSS2Mod ignores all pulldown flags, using "fps=23.976" forces the output frame rate (CFR) by duplicating or dropping frames. But this does not explain the combed frames you are getting occasionally. DSS2Mod certainly did not add them, and in my tests I cannot reproduce this.

I converted your uploaded clip using both DSS2Mod and ffms2, and both conversions came out nicely. Download here:
https://www.zeta-uploader.com/1333956887

I could not detect any combing, and because I resized vertically this would have looked especially ugly. Neither did I see any dupes or dropped frames, for me these conversions look perfect.

EDIT: spoke too soon... it completely messes up other sections in the larger movie, where every 2nd frame is dropped, and duplicated. I'll try to see if I can isolate an example

This looks fishy, especially if you say that this does not happen with DirectShowSource. I use DSS2Mod almost daily with captured HEVC sources in an MKV container, and these are full 2 hour movies. Never saw any issues like these. I suspect that something with your DSS2Mod installation is problematic.

Are you using the original forclip avss_26.dll? Do not use avss.dll under AVS 2.60 or AVS+. Also there is an unofficial 64-bit version floating around, don't...


Some of my ffms2 experiences:
The versions do matter, newer versions seem to be less reliable. The ones I prefer are 2.23.1 from Github and the C-Plugin version by qyot27 ffms2_r1140+101-avs+vsp.7z. Newer versions of the C-Plugin seem to have issues.

Basically ffms2 with the default parameters also ignores the pulldown flags, but the output for such soft telecined sources will be at 23.976 fps. Still it is recommended to force CFR 23.976 fps because you never know which issues the source might have. For ffms2 this is done by adding the fpsnum and fpsden parameters to the call. But this can also introduce problems. I won't go into the details here, you can find this in the ffms2-C-Plugin thread. What fixed it for me is to generally not use fpsnum and fpsden, but instead use "ChangeFPS(23.976)" right after the source filter call. The ffms2 conversion in my uploaded file was done this way.

My conclusion:
DSS2Mod (maybe even DirectShowSource) and FFmpegSource can handle 23.976 progressive NTSC sources with 3:2 pulldown just fine. You just need to have an understanding how these source filters work. Dealing with hybrid sources with a mixture of soft and hard telecined sources, possibly adding real 29.97 interlaced video is a different story. No source filter will handle these cases automatically.


Cheers
manolito

[poisondeathray]

I could not detect any combing, and because I resized vertically this would have looked especially ugly.

Look more closely at the scene where the cockit goes up. Both your encodes have it in a few frames



Neither did I see any dupes or dropped frames, for me these conversions look perfect.

There is on the full movie, and ffms2 is out of sync.

I'll try to reproduce it on a larger cut file, but I suspect it's going to have to be very large


This looks fishy, especially if you say that this does not happen with DirectShowSource.

I agree, super fishy. I would have expected the same behaviour on linear seeking. I'll try to debug it tonight


I use DSS2Mod almost daily with captured HEVC sources in an MKV container, and these are full 2 hour movies. Never saw any issues like these. I suspect that something with your DSS2Mod installation is problematic.


But did you try a 2hr soft pulldown DVD/MPEG2 source ripped with makemkv ?



Are you using the original forclip avss_26.dll? Do not use avss.dll under AVS 2.60 or AVS+. Also there is an unofficial 64-bit version floating around, don't...

Yes, avss_26.dll, the original forclip . The version is 2.0.0.13

Can you post a link to the exact one if that is not the correct one


Some of my ffms2 experiences:
The versions do matter, newer versions seem to be less reliable. The ones I prefer are 2.23.1 from Github and the C-Plugin version by qyot27 ffms2_r1140+101-avs+vsp.7z. Newer versions of the C-Plugin seem to have issues.

I tried 2.23.1 from Github, and a few others, but not the recent c-plugins. I deleted the index each time. Progressively out of sync, too many frames



Generally ffms2 with the default parameters also ignores the pulldown flags, but the output for such soft telecined sources will be at 23.976 fps.


Because rffmode=0 by default. If set rffmode=1 it's supposed to honor flags (But it's buggy) .


Still it is recommended to force CFR 23.976 fps because you never know which issues the source might have. Generally for ffms2 this is done by adding the fpsnum and fpsden parameters to the call. But this can also introduce problems. I won't go into the details here, you can find this in the ffms2-C-Plugin thread. What fixed it for me is to generally not use fpsnum and fpsden, but instead use "ChangeFPS(23.976)" right after the source filter call. The ffms2 conversion in my uploaded file was done this way.


Maybe for other types buggy source files in general, but if it's ripped correctly, DVD should be CFR.

If the source read at 23.976 CFR already, that is a no-op. The versions of ffms2 I used read it at 23.976 CFR

You can check with info() for what frame rates are internally, and ffinfo() for the CFR/VFR times


My conclusion:
DSS2Mod (maybe even DirectShowSource) and FFmpegSource can handle 23.976 progressive NTSC sources with 3:2 pulldown just fine.

But are you basing that "conclusion" on a single small test clip without audio ? A bit premature maybe ? So how was the sync? :D

eg. In your encodes, you have 7914 frames in your ffms2 version 7910 frames in your dss2mod version. Do you think that's ok? Well that's what I'm seeing too on the small test too. Is it not plausible that on longer video, there are more frames ? Because that's what I saw on the full movie with 64 extra frames, therefore progressively out of sync. Not an easy fix where all 64 frames are located nicely at the end or beginning.

My conclusion is there are plenty of problems with DSS2Mod and FFMS2 with DVD/MPEG2 sources. Many problems in the past. Many problems now. DirectShow had fewer problems here (go figure!) but still some issues like combing and grey frame at the end. It might be my dss2mod configuration is messed up, I'll look into it

I might have to upload a very long clip for you to see the problems.

[videoh]

Maybe it's time to add MKV support to DGIndex/MPEG2Source.

[manolito]

Your DSS2Mod version looks right, maybe compare it with my version:
https://www.sendspace.com/file/if6r6r

In one previous post you mentioned your source filter call:
DSS2("title00 (1).mkv", preroll=X)

I hope you did not really set "preroll=X". Use 15, and do specify the fps value...

And yes, a longer clip which shows the issues would be nice.

Cheers
manolito

[manolito]

Maybe it's time to add MKV support to DGIndex/MPEG2Source.

Hey, this would be really great... :D