Proper PAL Video (not Film) to NTSC conversion

[nfv]

This clip is from a PAL DVD and appears to be shot on video.

Here's link to clip:

https://www.mediafire.com/file/m4nrdu0mz891lkq/invisble25tff.avi/file

If I'm correct in assuming it was shot on video at 25fps interlaced what's the correct way to convert this to NTSC?

Thanks,
J

[FranceBB]

It indeed looks Interlaced Top Field first, so if you wanna convert it to NTSC I would do the following:

- Bob everything to 50p
- Crop inactive lines
- Downscale from anamorphic PAL to anamorphic NTSC (720x576 -> 720x480)
- Blend to 60fps progressive
- Re-interlace in NTSC

This should do it:

video=LWLibavVideoSource("invisble25tff.avi")
audio=LWLibavAudioSource("invisble25tff.avi")
AudioDub(video, audio)
AssumeTFF()
Bob()
Crop(18, 4, -18, -2)
Spline64Resize(720, 480)
ConvertFPS(60)
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()

Interlaced PAL:

https://i.imgur.com/SuJxMr3.png

Progressive PAL 50p:

https://i.imgur.com/Aihz2Px.png

Interlaced NTSC 60p:

https://i.imgur.com/rCWuLUz.png


I re-encoded your sample in IMX50 NTSC yv12 interlaced TFF: https://we.tl/t-o6agvm9Diu


General
Complete name : C:\Users\bucciantini\Desktop\IMX50 NTSC yv12 interlaced TFF.mxf
Format : MXF
Format version : 1.3
Format profile : OP-1a
Format settings : Closed / Complete
File size : 133 MiB
Duration : 21 s 488 ms
Overall bit rate : 52.0 Mb/s
Encoded date : 0-00-00 00:00:00.000

Video
ID : 2
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings : BVOP
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=12
Format settings, picture structure : Frame
Format settings, wrapping mode : Frame
Codec ID : 0D01030102046001-0401020201011100
Duration : 21 s 488 ms
Bit rate mode : Constant
Bit rate : 50.0 Mb/s
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 29.970 (30000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 4.827
Time code of first frame : 00:00:00:00
Time code source : Group of pictures header
GOP, Open/Closed : Open
GOP, Open/Closed of first frame : Closed
Stream size : 128 MiB (96%)

Audio
ID : 3
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (AES)
Codec ID : 0D01030102060300
Duration : 21 s 488 ms
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 29.970 FPS (1601.6 SPF)
Bit depth : 24 bits
Stream size : 5.90 MiB (4%)
Locked : Yes

Other #1
ID : 1-Material
Type : Time code
Format : MXF TC
Time code of first frame : 00:00:00:00
Time code settings : Material Package
Time code, striped : Yes

Other #2
ID : 1-Source
Type : Time code
Format : MXF TC
Time code of first frame : 00:00:00:00
Time code settings : Source Package
Time code, striped : Yes

Other #3
Type : Time code
Format : SMPTE TC
Muxing mode : SDTI
Time code of first frame : 00:00:00:00




It looks perfectly smooth to me as nobody will care about the blended 10fps, I've done it countless times for PAL to NTSC contents.

My line of reasoning for PAL to NTSC generally is:

Case1:
if it's PAL truly interlaced (50fps) then blend to 60p and divide in fields.

Case2:
if it's PAL progressive flagged as interlaced (25p) I apply a slow-down + pitch adjustment by 4% to bring it to 23.976fps progressive and then I telecine it in NTSC.

In your case, the source it's PAL truly interlaced (i.e 50fps), so blending it to 60p and then divide it in fields is my de-facto choice.

As a side note:
please note that your content is anamorphic and since we're working with anamorphic in both standard, you MUST flag it correctly when you re-encode it so that players will know how to resize; in your case it's 4:3.


Cheers,
Frank.

[wonkey_monkey]

Bob()

Eek, you should use something better than bob(). Yadifmod, at least.

nobody will care about the blended 10fps

That's a bold claim, too. Although interpolating is going to give you a headache when it causes artefacts, it looks much nicer than blended fields.

Crop(18, 4, -18, -2)

Pretty sure you shouldn't be cropping off all the NAB, too.

[wonkey_monkey]

The difference between Bob() and QTGMC() is:

https://i.imgur.com/f9ci1aI.png

(obviously QTGMC() also has some cleaning effects)

And the difference between convertfps and something better like mflowfps instead is:

https://i.imgur.com/KwezStr.png

Now not every frame is going to be quite that bad, and this is before resizing and reinterlacing, but the better quality you have before reinterlacing, the better quality you'll have afterward.

[johnmeyer]

I haven't done a lot of PAL to NTSC, but someone I know who does it all the time swears by the following script he created which uses QTGMC. I've used it a few times and thought the results were pretty good.

The color conversion might not be needed, depending on the nature of your source.

#Script to convert PAL to NTSC
#Output is for PAL DVD

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE2.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\SSE3Tools.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll")

Import("E:\Documents\My Videos\AVISynth Scripts\QTGMC-3.31a.avsi")

SetMemoryMax(1000)
AviSource("e:\fs.avi").AssumeTFF()

ConvertToYUY2(interlaced=true, matrix="PC.601")

Spline36Resize(720,height)

QTGMC( Preset="Slower", SubPel=2, EdiThreads=4,ThSCD1=180, ThSCD2=98, Blocksize=32,Overlap=4) #Nick's latest settings default scene detection is thSCD1=304, thSCD2=100
#QTGMC( Preset="Slower", SubPel=2, EdiThreads=4,ThSCD1=180, ThSCD2=98 ) #Nick's latest settings default scene detection is thSCD1=304, thSCD2=100
#QTGMC("faster", sharpness=0, SubPel=2) #SubPel must match pel in MSuper below
super = MSuper(levels=1, pel=2)
MFlowFps(super, QTGMC_bVec1, QTGMC_fVec1, num=60000, den=1001,ThSCD1=180, ThSCD2=98) #Change scene detection to match QTGMC
Spline36Resize(width,480)

#Spline36Resize(720,480)

blur(0.0,1.0)
sharpen(0.0,0.75) #Adjust 2nd value between 0.5 and 1.0 to taste

separatefields()
SelectEvery(4,0,3)
#Selectevery(4,1,2) #For reversed fields
Weave()



Here's a link to the moribund forum where Nick Hope first described this script:

Perfectionists' PAL<=>NTSC Conversion (https://www.vegascreativesoftware.info/us/forum/perfectionists-pal-ntsc-conversion--84824/)

[nfv]

Thanks for the suggestions.

I just need the proper frame rate conversion, no need for cropping or anything else. The final result will be a NTSC DVD.

Thanks again!

[Cary Knoop]

It indeed looks Interlaced Top Field first, so if you wanna convert it to NTSC I would do the following:

- Bob everything to 50p
- Crop inactive lines
- Downscale from anamorphic PAL to anamorphic NTSC (720x576 -> 720x480)
- Blend to 60fps progressive
- Re-interlace in NTSC

This should do it:

I would make two changes:

1. Use qtgmc (without the default noise reduction) instead of bob()
2. Use optical flow instead of blending. Blending is just awful!

[Boulder]

Thanks for the suggestions.

I just need the proper frame rate conversion, no need for cropping or anything else. The final result will be a NTSC DVD.

Thanks again!

No need to do anything special as you can use DGPulldown to do the 25 -> 29.97 fps conversion without any playback issues. I'd use QTGMC with FPSDivisor=2 to deinterlace the source before applying pulldown.

http://rationalqm.us/dgpulldown/dgpulldown.html

[Sharc]

You could also slowdown and hard telecine, something like

AssumeTFF()
qtgmc()
spline36resize(720,480)
selecteven()
separatefields()
SelectEvery (8, 0,1, 2,3,2, 5,4, 7,6,7) #3:2 pulldown
#SelectEvery (10, 0,1, 2,3,2, 5,4, 7,6,7,8,9) #2:2:3:2:3 pulldown alternative, slightly more judder than 3:2 pulldown but very little speed change / little audio pitch shift
weave()
assumefps(30000,1001,true)

No blends, but loosing temporal resolution.

[manolito]

Would you care to explain why you prefer hard telecine over soft telecine? I only see disadvantages, the most important one is that it will result in lower quality when using the same bitrate, since more frames have to be encoded. Plus the encoder needs to use Interlaced settings (alternate scan, interlaced motion estimation), all making the encoder less efficient.

[Sharc]

Would you care to explain why you prefer hard telecine over soft telecine? I only see disadvantages, the most important one is that it will result in lower quality when using the same bitrate, since more frames have to be encoded. Plus the encoder needs to use Interlaced settings (alternate scan, interlaced motion estimation), all making the encoder less efficient.
I don't prefer it. I showed it as an alternative solution only.
A reason for hard telecining could be when you have a mix of true interlaced video and "pulldown" (originally progressive) material in a stream. I found this often in bonus material of discs. I am from PAL region, but I think to remember that not all players / TVs handled soft pulldown properly.
As far as I know DGPulldown works for progressive mpeg2 material only. So the HW telecine approach is perhaps more general.

[FranceBB]

I would make two changes:

1. Use qtgmc (without the default noise reduction) instead of bob()
2. Use optical flow instead of blending. Blending is just awful!

First change, approved. QTGMC is indeed better than plain Bob(); alternatively if they want something faster than QTGMC and not use Bob(), they can use tdeint(mode=1, order=-1, field=-1, mthreshL=6, mthreshC=6, map=0, type=2, debug=false, mtnmode=1, sharp=true, cthresh=6, blockx=16, blocky=16, chroma=true, MI=64, tryWeave=true, link=1, denoise=true, slow=2) as well.

As to the second one... well... blending 10fps isn't that bad. I would totally blend over something like linear interpolation to be 100% sure to avoid artifacts. I've blended many contents and nobody ever complained. I would never blend something like 23.976fps to 50fps and divide it in fields in PAL regions of course as I would apply a speed-up + pitch adjustment and have 25p and encode it as progressive flagged as interlaced, however when it comes to blending as little as 10 frames (from 50p to 60p), it's not that bad.

[Cary Knoop]

...however when it comes to blending as little as 10 frames (from 50p to 60p), it's not that bad.
But totally unnecessary with present-day technology!

[wonkey_monkey]

FranceBB is right though that "present-day technology" isn't 100% perfect. Fast-motion scenes will cause artefacts, and there is no 100% robust solution for detecting and eliminating those, either. For a BBC drama filmed in the 80s, though, it's unlikely to cause many visible problems.

What should really be unnecessary with "present-day technology" is having to convert stuff in the first place, but for some reason most American-sold TVs are still not multi-standard.

[Cary Knoop]

FranceBB is right though that "present-day technology" isn't 100% perfect. Fast-motion scenes will cause artefacts, and there is no 100% robust solution for detecting and eliminating those, either.
Optical flow is not perfect but far better than blending.

That that even has to be argued here is rather surprising.

[wonkey_monkey]

It's only far better than blending most of the time. Blending will give a more consistent result, albeit a consistently not-as-good one.

[FranceBB]

FranceBB is right though that "present-day technology" isn't 100% perfect. Fast-motion scenes will cause artefacts, and there is no 100% robust solution for detecting and eliminating those, either.

Exactly.

Optical flow is not perfect but far better than blending.

It's better than blending when it works.

It's only far better than blending most of the time. Blending will give a more consistent result, albeit a consistently not-as-good one.

Indeed.


Cary, let's put it this way: once I encode a file, it's gonna be sent to the QC; we have three types of QC: Gold, Silver and Spot-Check. Gold QC is basically a 1:1 playback in which they're gonna see the file I encoded entirely, Silver is not a full 1:1 playback as it's about 3 minutes every 15 minutes and Spot-Check is literally Spot-Check. Let's suppose that I did some sort of linear interpolation, let's say with MVTools to have it super smooth and it fails in some scenes by creating some artifacts.

- Case 1: QC Gold - they notice it, they report each and every artifact at each and every timecode and they report it to me and my boss. After that, I'm gonna try to modify my script and take care of those artifacts manually and send them another file which is gonna be checked entirely. If it's gonna have other artifacts, the process goes on again and again and again 'till there are none left and it's gonna be a nightmare for me and for the person who has to QC the file several times.

- Case 2: QC Silver and Spot-Check - let's suppose that the artifact falls outside the range checked by them and that Cerify (an automatic QC Tool by Tektronix) doesn't detect it and it goes on air. If the playout detects it while it's on air and reports it, it's gonna be a "disservice" with my name on it. If a viewer watches it on air, notices the artifacts, reports them either via a telephone call or via an email or even just by posting those things on social media and it turns out that it wasn't his satellite dish but the artifacts were actually there on the file I encoded, then it's gonna be a big fat disservice with my name on it.

In case of blending, it's immediately a QC Pass. So, for me the choice is between risking to have a disservice with my name on it or having a guaranteed seal of approval by the QC guys...

[manolito]

What about replacing ConvertFPS in FranceBB's first answer with ChangeFPS?

Like this (by Xesdeeni):
[AVISYNTH_PAL2NTSC (Interlaced)]
Your_Source_Filter()
AssumeTFF() # Specify source field order
ConvertToYV12(interlaced=true)
Your_DoubleRateDeinterlacer()
Spline36Resize(720,480)
ChangeFPS(59.94)
AssumeTFF() # Specify target field order
SeparateFields()
SelectEvery(4, 0, 3)
Weave()

FranceBB has a point when he says that probably no viewer will ever notice the few blended fields in the resulting conversion. But of course there are the purists who insist that blending is a mortal sin.

If you duiplicate some fields instead will viewers notice? I am from PAL land, so I don't know. But I have read many times that folks who grew up in NTSC land are so used to 3:2 pulldown that they never even notice the resulting motion judder.

[Boulder]

As far as I know DGPulldown works for progressive mpeg2 material only. So the HW telecine approach is perhaps more general.

If the final destination is a DVD, a hard telecine could possibly lose quite a bit of quality because the amount of frames to encode is much bigger (29.97 / 25 = ~120%). Besides, you already need to mangle the source through QTGMC to do hard telecining so the better approach is to use DGPulldown.

I think that most NTSC people are still used to the pulldown jitter, and any modern high end TV will get rid of it nicely anyway.

Nevertheless, it would be interesting to hear what the final conclusion of the OP is after testing all the various suggested methods.

[Sharc]

….Besides, you already need to mangle the source through QTGMC to do hard telecining so the better approach is to use DGPulldown.

AFAIK there used to be (high end) DVD players which did not read flags but relied on the telecine pattern cadence detection for doing IVTC. These players may not work with soft telecine. I think these players are a minority though. Someone from NTSC land may have more on this.

[johnmeyer]

AFAIK there used to be (high end) DVD players which did not read flags but relied on the telecine pattern cadence detection for doing IVTC. These players may not work with soft telecine. I think these players are a minority though. Someone from NTSC land may have more on this.I've always been confused as to whether any DVD players actually do IVTC. Instead, I think what most people are referring to is that many DVD players are smart enough to not add telecine to 24 fps material. Hollywood movies have always been encoded as 24 fps. To play these on traditional NTSC TV sets, the DVD players initially sold, back in 1997 when DVDs were introduced in this country, all had the ability to add pulldown in real time (a trivial operation, especially compared to the rather complex operation of IVTC).

When we finally had TVs that could play any fps, we started getting disc players that could ignore the pulldown flag and output the native 24 fps.

So, as I said above, I wonder if any players really do true IVTC. It would be tough to do and get right all the time.

[Sharc]

So, as I said above, I wonder if any players really do true IVTC. It would be tough to do and get right all the time.
Perhaps this was the reason why cadence detection was abandoned and substituted by flag reading players.
Out of curiosity: How did the traditional NTSC players/TVs process streams with mixed hard telecined film and true interlaced video sequences, as sometimes found in bonus material of commercial discs (interviews mixed with film sequences)? Did they just bob deinterlace / decimate everything? Or was hard telecined film hardly ever used for NTSC DVDs?

[manono]

AFAIK there used to be (high end) DVD players which did not read flags but relied on the telecine pattern cadence detection for doing IVTC.

Oppos, high end Denons, a few others.

These players may not work with soft telecine.

They work well with soft telecine. Why wouldn't they? If they can IVTC hard telecine they can easily IVTC soft telcine which, after all, also outputs interlaced 29.97fps.

To play these on traditional NTSC TV sets, the DVD players initially sold, back in 1997 when DVDs were introduced in this country, all had the ability to add pulldown in real time...
They were just following the RFF and TFF flags to output interlaced 29.97fps (really 59.94 fields per second). That's what flag readers do.


Out of curiosity: How did the traditional NTSC players/TVs process streams with mixed hard telecined film and true interlaced video sequences, as sometimes found in bonus material of commercial discs (interviews mixed with film sequences)? Did they just bob deinterlace / decimate everything? Or was hard telecined film hardly ever used for NTSC DVDs?

The old square tube televisions? They only played fields so they didn't have to do anything much. No bobbing, no deinterlacing.

[kuchikirukia]

No need to do anything special as you can use DGPulldown to do the 25 -> 29.97 fps conversion without any playback issues. I'd use QTGMC with FPSDivisor=2 to deinterlace the source before applying pulldown.

http://rationalqm.us/dgpulldown/dgpulldown.html

Why would you destroy 50fps video like that?

[Boulder]

Why would you destroy 50fps video like that?

I find a blended result much more irritating than a deinterlaced one.. matter of taste I guess. QTGMC does have the option of applying motion blur which makes single rate output look quite good compared to the double rate one. I've sometimes used it for less important stuff to save a lot of bits.

[Sharc]

Oppos, high end Denons, a few others.


They work well with soft telecine. Why wouldn't they? If they can IVTC hard telecine they can easily IVTC soft telcine which, after all, also outputs interlaced 29.97fps.

Oh I see. These players detect the cadence based on the flags. I was wrongly assuming that they ignore the flags and make the decision for IVTC or deinterlace based on the direct analysis of the frames only; somehow similar to what we do when stepping through the fields.

[Boulder]

I think they do use some sort of analysis as well. My Oppo was able to "fix" the playback of field shifted PAL sources (ones which you simply fix by applying Telecide() or VFM() on). Or do they have some special flags in the stream?

[Sharc]

I think they do use some sort of analysis as well. My Oppo was able to "fix" the playback of field shifted PAL sources (ones which you simply fix by applying Telecide() or VFM() on). Or do they have some special flags in the stream?
You are right. There exist all kind of strategies, depending on the chipset. I found this an interesting reading:
https://hometheaterhifi.com/technical/technical-reviews/dvd-benchmark-part-5-progressive-scan-dvd/
Scroll down and see for example the section 'A Look At Some Common Chipsets' and the section 'What Can Go Wrong. Flag Reading vs. Cadence Reading'.

In the end the OP may try all the proposed options in this thread, and see how it looks on his DVD player/TV model.
Awaiting his conclusions …. :)

[manono]

These players detect the cadence based on the flags.
No, that's what flag readers do. So, if your DVD player is a flag reader (the vast majority) and it comes across hard telecine (encoded as interlaced) it deinterlaces.

Cadence detecting players note progressive and interlaced frames and can IVTC. They reassemble the progressive frames just as TFM does, remove the duplicate frames and then send them off to a progressive television in that 3-2-3-2-3-2 pattern to output 59.94 frames per second. That's for 60Hz televisions. They can also, in most cases, determine what to do with interlaced content encoded as progressive and progressive content encoded as interlaced.

I was wrongly assuming that they ignore the flags and make the decision for IVTC or deinterlace based on the direct analysis of the frames only; somehow similar to what we do when stepping through the fields.
No, you were correct in that assumption. I've read they can also handle 3-2-3-2-2 pulldown, such as you might get with PAL to NTSC DVDs. But I don't know how to confirm that. And, as Boulder mentioned they don't have to deinterlace phase-shifted videos, something a flag-reader will always do.

The article to which you linked is a very good one. The Home Theater site also used to have reviews of dozens of players but those were all removed when the authors wanted to make their writing commercial.

[wonkey_monkey]

Cadence detecting players note progressive and interlaced frames and can IVTC. They reassemble the progressive frames just as TFM does, remove the duplicate frames and then send them off to a progressive television in that 3-2-3-2-3-2 pattern to output 59.94 frames per second.

Can you give an example? Like what kind of input pattern might they IVTC, remove duplicates, and output as 3-2-3-2-3-2?

[Sharc]

So, if your DVD player is a flag reader (the vast majority) and it comes across hard telecine (encoded as interlaced) it deinterlaces.
Thanks for clarification. So flag readers are inferior for hard telecined footage, but they may have less transition artefacts for mixed true interlaced/hard telecined sequences.
I've read they can also handle 3-2-3-2-2 pulldown, such as you might get with PAL to NTSC DVDs. But I don't know how to confirm that.
Apparently it seems to work. Years ago I made a hard telecined 3-2-3-2-2 DVD for friends in NTSC region. According to their feedback it played smoothly and the picture was sharp. I'd assume they had a cadence detecting player (OPPO as far as I remember).

[manono]

Can you give an example? Like what kind of input pattern might they IVTC, remove duplicates, and output as 3-2-3-2-3-2?DVDs were developed for CRT televisions that play fields. All DVD players output 59.94 fields per second. Then those with progressive scan (both flag readers and cadence readers) intercept that stream of fields and do whatever they can do. Flag readers can reassemble the fields into frames, take soft telecine and send out the original progressive frames. They can only deinterlace hard telecine, sending out bobbed fields to your progressive television set.

Cadence readers can successfully IVTC both hard and soft telecine and pass on the original progressive frames. The article to which Sharc linked earlier goes into some detail about that. The article is fairly old now, though, and deinterlacers are pretty good these days.

So flag readers are inferior for hard telecined footage, but they may have less transition artefacts for mixed true interlaced/hard telecined sequences.

Yes, when the film switches from soft to hard telecine there might be several frames of interlacing before the player locks onto the change of cadence. One of the best DVD producing companies - Criterion - has (or had) a glaring error in this regard. They set their chapter points during the encoding without regard to cadence. What this means is that there might be 2-3 frames of hard telecine on either side of a chapter point, in a DVD that might otherwise be soft telecine. And these few interlaced frames can be obvious and annoying. Ordinarily, chapter points are set on GOP boundaries after the encode (like at scene changes) and you don't get that.

I don't know whether or not you get the same sort of thing when transitioning from pure interlace to hard telecine. If the article said so, believe it. Those guys know their stuff.

[real.finder]

I wonder if it also need to convert color primaries

z_ConvertFormat(colorspace_op="470bg:601:470bg:limited=>470bg:601:170m:limited")

[DTL]

I wonder if it also need to convert color primaries


Yes it is.

[Balling]

I wonder if it also need to convert color primaries

z_ConvertFormat(colorspace_op="470bg:601:470bg:limited=>470bg:601:170m:limited")

Yes, of course. Just like BT.709 primaries to BT.2020 primaries, on linear light. It is said so on wikipedia.

https://en.wikipedia.org/wiki/Rec._709#Converting_standard_definition

"In addition, the NTSC color primaries of red, green, and blue are different than those of BT.709. The red and blue primaries for PAL and SECAM are the same as BT.709, with only a minor change in the green primary. Converting NTSC properly means using a LUT (lookup table) to convert the colors to the new colorspace."