PAL Clip with interlacing in every frame conversion to NTSC

[nfv]

https://www.mediafire.com/file/tl88deayaeq61n8/clip_TFF.avi/file

The following clip has interlacing in every frame. What script would you recommend to bring it back to 23.976p?

Thanks,
J

[Sharc]

It's phase-shifted. The content is progressive.
You can restore the 25fps progressive sequence by field matching with TFM(), or 24fps film using TFM().tdecimate(cycle=25).

[StainlessS]

Here an explanation of Phase Shifted (AKA Perverse) Telecine [click a 2nd time for magnified image].
https://i.postimg.cc/14MJpkkm/Perverse-TC.jpg (https://postimg.cc/14MJpkkm)

Taken from Scharfis_Brain Exotic Interlacing [originally in German],
see DATA Folder @ MediaFire in my sig below this post for PDF's in English or German.
[There is also a zip containing Web page]

EDIT: The Website is no longer online, but LigH posted a link to his saved copy somewhere.

EDIT: Original web page on WayBack Machine:- https://web.archive.org/web/20140411222900/http://home.arcor.de/scharfis_brain/ExotischesInterlacing/

And a thread I started on it (forgot about it) some time ago:- https://forum.doom9.org/showthread.php?t=176104&highlight=Exotic+Interlacing

EDIT: LigH hosted original web page in German:- https://www.ligh.de/scharfis_brain/ExotischesInterlacing/

[Sharc]

Yeah, nice to remember Sharfis, the interlace Guru.... :)

The OP's clip can also be restored to 24 fps progressive with
AssumeTFF().separatefields().trim(1,0).weave().tdecimate(cycle=25).
This leaves however few interlaced frames which TFM(pp=6) deinterlaces.
Still some fields of the source are blended. So I don't know how this source has been butchered overall.

[hello_hello]

I find myself needing to argue.... :)

There only seems to be repeated frames after field matching on scene changes due to orphan fields. If you do this you'll see frame number 31 is combed.

AssumeTFF().separatefields().trim(1,0).weave()

This fixes the combing but frame 32 will be a duplicate of frame 31.
TFM(pp=1)

If you want to, you can move the duplicate frame to the other side of the scene change.
AssumeTFF().TFM(pp=1)

There's no duplicate frames in between scene changes though, so I'm certain tdecimate(cycle=25) will remove good frames.
There's chroma ghosting as though it's been encoded as interlaced and decoded as progressive, or the other way around, but this seems to work well to me and avoids some of the duplicate frames on scene changes.

TDeint(order=1)
Or to slow it down (you'd have to slow the audio to match)
TDeint(order=1).AssumeFPS(24000,1001)

It's probably not as sharp as TFM(Order=1, pp=1) but there's not a lot in it and I'd tend to sharpen it a tad with QTGMC anyway.
If TDeint does better on scene changes I'd go with it instead.

TDeint(order=1).AssumeFPS(24000,1001)
QTGMC(InputType=1, Preset="Medium", EzDenoise=1)

[Sharc]

Agree that Tdeint helps with this clip, but it leaves jaggies on horizontal lines which one needs to clean up with QTGMC, for example, as you suggested.
I am however hesitant to throw a deinterlacer at phase-shifted footage, but prefer to restore the original progressive frames by field matching instead. As you noted, TFM() leaves few duplicated frames (I missed that in my testing), so we need to tune the scthresh parameter, like
TFM(mode=1,pp=6,scthresh=25)
which outputs clean 25fps 'original' progressive frames.
Agree to slowdown rather than decimate for 24fps, if speed change is absolutely needed.

[manono]

I agree. Don't deinterlace when you can field match. If it doesn't work with every single frame, then something like Vinverse can take care of the stray frames that remain interlaced.

And anyway, TDeint isn't a very good deinterlacer to begin with.

[hello_hello]

Edit: f%$ck me! I swear I hadn't read your post before I started experimenting and writng this one Sharc. In fact I don't think you'd edited your post to add the scthresh part when I opened this thread again. Or I missed it. And here was me thinking I'd come up with something clever. f%$ck!

TDeint can actually work well for phase shifted sources because it can do basic field matching. For some reason though, I had it in my head it did so by default, however looking at the help file it appears that needs to be enabled.

TDeint(tryWeave=true)

Unfortunately though, the end result is duplicate frames on scene changes as happens when using TFM.

Anyway.... because you made me feel a bit sorry for TDeint, and it seemed like I might learn something while looking for an excuse to argue, I read some help files and experimented a little and.....

A = last.Greyscale()
AssumeTFF()
TFM(PP=1, mode=0, scthresh=100)
TDeint(hints=true, clip2=A)

The greyscale clip as the deinterlaced clip was just to make it easy to see which frames will be deinterlaced. I think scthreshold=100 is effectively a shortcut to playing with options in order to beg TFM not to look for alternative matches around scene changes.
TDeint only deinterlaces when TFM tells it to, and the end result is the the phase shifted fields are fixed and there's no duplicate frames on scene changes.

While I think that method works quite well, I did discover it's possible to achieve the same thing with TFM alone, so that left me disappointed and without a good reason to defend TDeint too much, but at least I think I found a slightly better way to process the OP's sample.

A = last.Greyscale()
AssumeTFF()
TFM(PP=2, mode=0, scthresh=100, clip2=A)

Are TDeint and TFM() on a par for de-interlacing? They both have a clip2 argument and appear to have similar de-interlacing methods.

[johnmeyer]

Are TDeint and TFM() on a par for de-interlacing? They both have a clip2 argument and appear to have similar de-interlacing methods.Just to be clear, TFM is not a deinterlacer. More importantly, field matching using TFM, and inverse telecine (IVTC) using TFM and TDecimate, are not deinterlacing.

They are totally different operations.

Thus TFM and TDeint are orthogonal and cannot be compared to each other.

Deinterlacing requires the construction of fields that never existed, and therefore involves degradation of the video.

By contrast, field matching simply rearranges existing fields in a different order and, for IVTC, also decimates duplicate fields that were added to make slower fps material play on original CRT displays which were hard-wired for one specific frame rate. There is zero degradation with the IVTC operation other than the degradation that happens when re-encoding the video after the fields are put in the correct order, and duplicate fields removed.

Finally, while not everyone will agree with the following statement, for my work I always do IVTC in order to recover the original progressive content (this is required when doing any sort of restoration). By contrast I never deinterlace, even when doing re-sizing or other operations which "require" deinterlacing, because there are other ways to do that without introducing deinterlacing artifacts.

[Sharc]

Yes, this is clear. For the phase-shifted source under discussion the deinterlacing had only to be applied on the few orphaned fields at scene changes; all the rest is perfectly restored by field matching.
pp=6 in TFM takes care of deinterlacing the orphaned fields for which no match is found, the rest is correctly field matched. Scthresh is a threshold to remove duplicates. display=true in TFM shows how it goes.
For true interlaced material I also prefer - with exceptions - to keep it interlaced for the reasons you mentioned.
I would assume that the deinterlacer which kicks in with TFM for deinterlacing the orphaned fields (residual combed frames) is using much the same algo as tdeint because tritical is the author of both. Pure speculation though.

[hello_hello]

Just to be clear, TFM is not a deinterlacer. More importantly, field matching using TFM, and inverse telecine (IVTC) using TFM and TDecimate, are not deinterlacing.

They are totally different operations.

Yeah but NTSC can be a mixture of film and video etc, so something must be de-interlacing the video sections. TFM actually has more de-interlacing choices than TDeint

0 - nothing (don't even look for combed frames)
1 - find/hint combed frames but don't deinterlace
2 - dumb blend deinterlacing
3 - dumb cubic interpolation deinterlacing
4 - dumb modified-ela deinterlacing
5 - motion-adaptive blend deinterlacing
6 - motion-adaptive cubic interpolation deinterlacing
7 - motion-adaptive modified-ela deinterlacing

It's just a pity TFM can't bob de-interlace when you want a VFR.

Thus TFM and TDeint are orthogonal and cannot be compared to each other.

tritical?
TDeint 1.1 is dated Jan 2007 and TIVTC 1.05 is a year younger, so maybe that's why.


Deinterlacing requires the construction of fields that never existed, and therefore involves degradation of the video.

When it's bob de-interlacing, I'm pretty sure yadif doesn't change the existing scanlines, making it's de-interlacing reversable. I'd have to check, but maybe TDeint is the same. QTGMC has several variations of a lossless mode, but I'm pretty sure there is at least one that's 100% lossless.

By contrast, field matching simply rearranges existing fields in a different order and, for IVTC, also decimates duplicate fields that were added to make slower fps material play on original CRT displays which were hard-wired for one specific frame rate. There is zero degradation with the IVTC operation other than the degradation that happens when re-encoding the video after the fields are put in the correct order, and duplicate fields removed.

Finally, while not everyone will agree with the following statement, for my work I always do IVTC in order to recover the original progressive content (this is required when doing any sort of restoration). By contrast I never deinterlace, even when doing re-sizing or other operations which "require" deinterlacing, because there are other ways to do that without introducing deinterlacing artifacts.

The main point for me is QTGMC is very good. If you have a player/TV that de-interlaces at even close to QTGMC quality I'd like to see it, and unless you can display interlaced content on a progressive display, you'll be de-interlacing on playback anyway. My video card's de-interlacing, as well as my TV's own de-interlacing, would be roughly on a par with Yadif.

Plus after we're dead and the grand-kids throw most of our stuff away because it's old and crappy, I'd prefer any video they keep to be progressive. Unlike the huge stack of 78 records my grandmother owned and my idiot sister threw away even though they'd probably be worth something now, 50 years after we're dead nobody will be nostalgic over interlaced video. :) And then I can be dead without the grand-kids mocking my encoding skills because I didn't de-interlace, assuming they even realise what's causing the horrible they're seeing. :)

QTGMC isn't perfect. It tends to enhance ghosting a bit (Edit: I meant halos, not ghosting), and occasionally you can see an interpolation oddity (similar to what can happen when interpolating new frames to change the frame rate) although it's not common or obvious. I'm yet to de-interlace a source with QTGMC where the output didn't look better than the input, which is why I run it in progressive mode for almost every progressive SD source I encode. Something like:

QTGMC(InputType=1, Preset="Medium", EzDenoise=1)

I use it to denoise HD quite a bit too as it's noise removal to blurring ratio is better than most scripts, but often I only need to denoise HD and not also stabilise the picture, which is generally beneficial for SD, so sometimes I use different denoising for HD. I'm pretty sure even in progressive mode though, QTGMC throws half the scanlines away, but the output looks better than the source close to 100% of the time, so I stopped caring.

1080p.
https://i.postimg.cc/qNB9CtZv/1080p.jpg (https://postimg.cc/qNB9CtZv)

Downscaled to 720p for encoding. Denoised with QTGMC. Probably followed by a small dose of LSFMod and finishing with GradFun3(). The screenshot is of the 720p encode after it was resized to full screen by MPC-HC.
https://i.postimg.cc/SJdPqSNF/720p.jpg (https://postimg.cc/SJdPqSNF)


I have a somewhat related question....
I asked in the LSmash thread by nobody replied. How does a decoder tell if a source is encoded as interlaced or progressive, at least for mpeg2? Is that info always stored in the video stream? I ask because lsmash has been given another update, and now Repeat=true is he default.
The help file says when repeat=true, LSmash will treat all frames as interlaced. Would that effect the upsampling of the chroma?

Although are the progressive sections of an NTSC DVD encoded as progressive or interlaced? I have no idea if it's a one or the other choice for an entire DVD, or if the progressive sections can be encoded as progressive (film with soft pulldown etc).

Given PAL DVDs seem to be encoded as interlaced regardless of the content (most of the time), would it be the same for NTSC?

[videoh]

How does a decoder tell if a source is encoded as interlaced or progressive, at least for mpeg2? MPEG2 has the progressive_frame flag, but it may not always match the content:

https://forum.doom9.org/showthread.php?t=101700

The help file says when repeat=true, LSmash will treat all frames as interlaced. That doesn't seem like a good idea to me. They should use the progressive_frame flag.

[hello_hello]

That doesn't seem like a good idea to me. They should use the progressive_frame flag.

Maybe you could mention it in the LSmash thread, next time you're there. I don't think HolyWu likes me because I don't index video via AvsPmod after creating a script.

[StainlessS]

MPEG2 has the progressive_frame flag, but it may not always match the content
Some time ago (years) I had some MPEG2 clips [transport stream origin] where they constantly [In DGIndex Play] changed from Interlaced to Progressive and back, multiple times per second,
presumably due to this Progressive_frame flag. [sometimes periods where every frame toggled], weird.
Also PAL DVD, Twin Peaks Season 1, think it was mainly Progressive, with very occasional very short periods [~1 sec] of mid-scene interlacing, for no apparent reason.

Sometimes I think also happens where Black_frames Intro precede or Ending postcede, a DVD movie, where may switch from Interlaced to Progressive [or vice versa], and also with a
possible Illegal transition at the join. [The Black frames Intro/Exit join is in my experience the position where Illegal transitions are flagged in DGIndex]

Is it this Progressive_frame flag that is responsible for the Progressive/Interlaced transition in DGIndex status box, during Play ?

I don't think HolyWu likes me because I don't index video via AvsPmod after creating a script.
I dont think that HolyWu is so narrow minded as to limit himself to disliking you for only that reason, but it could well be a contributing factor. https://www.cosgan.de/images/smilie/froehlich/a065.gif
[Most of us dont 'give-a-monkies' how, where, why, or if, you index a video :) ]

[Sharc]

… Also PAL DVD, Twin Peaks Season 1, think it was mainly Progressive, with very occasional very short periods [~1 sec] of mid-scene interlacing, for no apparent reason.

Agent Cooper may have the answer :D

[videoh]

Is it this progressive_frame flag that is responsible for the Progressive/Interlaced transition in DGIndex status box, during Play ? Correct.

[StainlessS]

Thanks V.

[SaurusX]

Some time ago (years) I had some MPEG2 clips [transport stream origin] where they constantly [In DGIndex Play] changed from Interlaced to Progressive and back, multiple times per second,
presumably due to this Progressive_frame flag. [sometimes periods where every frame toggled], weird.


Check the later season Transformers DVD's from Rhino for an example of this.

[hello_hello]

I dont think that HolyWu is so narrow minded as to limit himself to disliking you for only that reason, but it could well be a contributing factor. https://www.cosgan.de/images/smilie/froehlich/a065.gif

On the other hand, when someone's first post on a subject announces their opinion and declares it "end of story", it's a fair indication disagreeing will be reason enough. :)

[nfv]

Thanks for all the input and suggestions.

How about this one?

http://www.mediafire.com/file/e7ft807hvwdeclr/Title_1TFF.avi/file

[Sharc]

Same problem as before: Phase-shifted rather than interlaced.
You can fix it with the same tools as before.

converttoYV12(interlaced=true)
AssumeTFF()
TFM(pp=1) #field matching without postprocessing
vinverse() #removes residual combing

[scharfis_brain]

Yeah, nice to remember Sharfis, the interlace Guru.... :)

It's always nice being remembered by the people, despite my long time absence ;-)

[manono]

Some people are never forgotten. They're immortal. Nice to see you again, scharfis_brain.

[videoh]

I miss the scharfis_brain avatar. It was too cool for words.

[bruno321]

I'm in a similar situation so I'm posting here, hope that's fine. Here's an m2v snippet from a PAL DVD:

https://1fichier.com/?c63nesx3ix06ovaaoe3a

QTGMC deinterlaces that fine, but I wonder if there's something better to do to make it progressive, for example, field matching. The reason is that I would like to avoid the artifacts of deinterlacing, like messing up striped patterns.

I've tried things like tfm (with various parameters) + vinverse but I always get some jagged teeth here and there, like on the car edges in that sample.

[Sharc]

I'm in a similar situation so I'm posting here, hope that's fine. Here's an m2v snippet from a PAL DVD:

https://1fichier.com/?c63nesx3ix06ovaaoe3a

QTGMC deinterlaces that fine, but I wonder if there's something better to do to make it progressive, for example, field matching. The reason is that I would like to avoid the artifacts of deinterlacing, like messing up striped patterns.

I've tried things like tfm (with various parameters) + vinverse but I always get some jagged teeth here and there, like on the car edges in that sample.
It's phase shifted. So field matching is the correct procedure.
You can then try to fix the residual jaggies with QTGMC for example
TFM()
QTGMC(InputType=2) #or 3

[bruno321]

That's great, thanks Sharc. One of these days I'll read the QTGMC manual in depth :)

[Sharc]

You could 'eliminate' the jaggies (aliases) further by adding to the script
santiag(1,2)
It will however slightly affect (reduce) the vertical resolution / sharpness.