Encoding progressive hard-telecined material as interlaced? Or progressive?

[zoinbergs]

Okay, so I have been researching this for days now, and I cannot seem to find an answer I can understand!

Hopefully somebody here can help me?

I have to hard-pulldown a clip that has flags in it created from a recent 25-->29.97 DGPulldown of it. The source PAL is encoded as progressive, as well as the "hybrid" clip (720x480 @ 25fps).

My question is how I should encode the final clip with the flags being honored... Progressive, or Interlaced?


Here's my steps (from beginning to end):

1) Frameserve the original PAL video

2) Resize it to 720x480 using avisynth

3) Encode it using TMPGEnc @ 25fps, Progressive, 720x480

4) Pulldown it from 25-->29.97

5) Frameserve the flagged NTSC video with "Honor Pulldown Flags"

6) Run a simple aviysynth script on it:

LoadPlugin(DGDecode.dll)
Mpeg2Source("...\video.d2v")

(to account for the frames with telecining in them)

7) Encode it using TMPGEnc @ 29.97, INTERLACED, 720x480


And here is what I get as the final result:

http://www.zoinbergz.com/Interlaced.bmp

Looks kinda ugly doesn't it? Especially the blue diagonal across the front of the stage! Almost no chroma quality left!

My best guess is that encoding a progressive clip as interlaced is doing this (see step #7).

So, I encoded the same avisynth script created in step #6 using TMPGEnc @ 29.97, PROGRESSIVE, 720x480... and voila! I get this instead:

http://www.zoinbergz.com/Progressive.bmp

Much better chroma resolution! (For both this frame, and the interlaced ones created from the pulldown process).

Then again, I'm viewing this clip on a progressive display, using a software DVD player, and not on an interlaced display, using a hardware DVD player.

My question is if it's okay to encode a telecined (from a progressive source) clip as PROGRESSIVE, as opposed to Interlaced. Even though the pulldown clearly has created interlaced frames.

When my video is decoded using a DVD player, and it decodes it based on the Progressive Frame Flag (if I encode progressively), will it not decode the interlaced frames correctly? Or should I stick with encoding interlaced, to protect the interlaced frames when it comes time to decode those (all at the expense of chroma quality loss for the progressives frames)?

What do DVD authoring studios do? I read somewhere that they sometimes encode interlaced video as progressive (which has been deemed innapropriate to do), but that they sometimes encode progressive video as interlaced (equally not good to do).

What about the all popular progressive clips that are telecined? Is there someway to truly convert a progressive clip to an interlaced one (with a correct UV map -- that's decoded correctly)?

My oh my I need to shut up! I need to learn how to ask fewer questions, don't I! Thank for reading them all though. Hopefully there's a person or two out there who know a little about chroma and telecining and what not!

Thank you so much to anyone who can shed some light on this matter.

[zoinbergs]

Also, I have the option to encode the hard-telecined clip as Progressive (to retain chroma quality), but restream the m2v with the "Framtype Progressive" unchecked.

This would tell DVD players to decode it based on

C1/C3
C2/C4

and not

C1/C2
C3/C4

right?

But wouldn't that incorrectly decode the progressive frames, and correctly decode the interlaced ones?

Or actually, I did read somewhere that the interlaced_frame_flag doesn't influence how a progressive frame is decoded: progressive frames are all decoded the same. Right?

Man this gets confusing!

[zoinbergs]

Also, I have 23.976 clip that I am telecining using avisynth to 29.970, with this script:


LoadPlugin("DGDecode.dll")

Mpeg2Source("...\blah.d2v")

SeparateFields()
SelectEvery(8, 0,1, 2,3,2, 5,4, 7,6,7)
Weave()


Does anyone know if this should be encoded as progressive, or interlaced? Encoding it as progressive yields the same chroma resolution as the original progressive clip, and encoding it as interlaced seems to show the same chroma quality loss displayed in my first post.

[zoinbergs]

Alright, now HERE'S a question!

How come when I encode interlaced material in progressive mode, the interlacing artifacts stay perfectly intact? Isn't a UV plane that's encoded progressively store chroma for lines 1+2 in line 1, lines 3+4 in line 3, etc?

I believe a picture encoded as interlaced relies on the chroma for line 1 to show colors for line 1 and 3 of the luma, right?

Wouldn't an interlaced clip encoded in progressive destroy the chroma somehow, because one can't store two different chroma values for line 1 and 2 in line 1 of the luma? Theoretically it should create a blend since the separate values would have to be averaged or something like that.

[zoinbergs]

Okay, woah! I just found something out that could change the entire course of this thread as we know it!

I re-took a snapshot of the frame encoded as INTERLACED, using VDubMod's Ctrl-1 feature, and to my utter surprise, it looks absolutely nothing like the first of the two pictures I posted above!

It looked flawless in fact! Like the second picture of the two!

Before I was taking snapshots with Media Player Classic, as it is what I've always used... but perhaps the program somehow was weaving and/or displaying my videos wrong! I wonder if it displays videos based on the progressive/interlaced flags?

To be honest, I almost wasn't suprised when I found this out.. MPC doesn't honor pulldown flags in an MPEG2 stream either (which can be quite useful though, sometimes).

I guess now my question is: does there exist a versatile player out there, with the features of MPC, that honors pulldown flags AND shows a video's original frames, untouched? My dear that would be a valuable player!

I should write to the producers of MPC to see if they can fix these bugs...

But for all intents and purposes, this thread is closed. =)

Well, actually, I AM still wondering if it's okay to encode interlaced material as progressive. But I'm leaning toward no, as it doesn't seem feasable. Something's bound to go wrong somewhere if I do so, right? Interlaced means interlaced, progressive means progressive.

Maybe the other way around works though: encoding progressive material as interlaced? Although I don't quite know what the advantages would be to do such a thing!

If only DVD was in 4:2:2 =)

[Wilbert]

My question is if it's okay to encode a telecined (from a progressive source) clip as PROGRESSIVE, as opposed to Interlaced. Even though the pulldown clearly has created interlaced frames.
There's not an ideal solution for this. If you decode as progressive, the progressive frames will come out with better quality (like your second screenshot), but the combed frames will have chroma blending (chroma from different time instances). If you decode as interlaced, the combed frames will be fine, but the progressive frames will show the "artefacts" (like your first screenshot).

Best is to avoid hard-telecined stuff :) But at the end, it's only a display problem, so it's not that your video itself is mangled.

Wouldn't an interlaced clip encoded in progressive destroy the chroma somehow
Yes, but do not confuse:
1) decoding interlaced stuff as progressive (or feeding encoder with interlaced YV12 and add a progressive flag).
2) feeding the encoder with interlaced YUY2 and encode as progressive.

In the first case it's only a display problem and in the second case your encoding is mangled.

[zoinbergs]

Best is to avoid hard-telecined stuff :)

Oh if only I could do that! Unfortunately though, I can't. :(

I am using TMPGEnc DVD Author v2.1.9.90 (as I cannot use v1.6 for many reasons), and it doesn't accept the irregular flagging pattern created from a pulldown of 25-->29.97.

I guess it's okay though.. since my DVD will be played in other countries, and I'm shooting for as close to 100% compatibility as I can.

I hope someday everything goes progressive... boy that would be easier to implement and use! But I have a hunch that it won't be happening, seeing as 1080i is the new standard.

Oh well. Can't have everything perfect!

Thanks for your explanation though, Wilbert. As long as I don't actually encode interlaced stuff as progressive, I shouldn't go wrong!

[manono]

...and it doesn't accept the irregular flagging pattern created from a pulldown of 25-->29.97

It won't author perfectly compliant video? Time for a different authoring app, I think. Why use it at all? The original DVD doesn't come with its own menu? If it does, use Muxman for the job and then stick the Muxman authored DVD back into the original DVD (perhaps after also converting the menu to NTSC). And why encode it twice, first to 25fps followed by DGPulldown, and then secondly using the reencoded video as a source for another encode? All to hard telecine it so you can use a defective authoring app? Heck, if you really have to hard telecine it (which I don't believe you do), you could do that using the original PAL DVD as a source. There's something very screwy about your work flow.

[zoinbergs]

Oh believe me, I have thought this through many a times, and have spent dozens of hours perfecting my work flow. If I could have it any other way, I would, but I think I kinda like how I do it now anyways! My goal is to make standard MPEG2 streams all the way through. Personal preference I guess, to have each clip be output with the same properties.. but that way I minimize risk of corruption/non-compatibility. =) This project has got to be as close to PERFECT and seamless as I humanly possibly can get it.

To give you a quick insight:

What I'm doing is collecting music videos from different sources and compiling them into one collection. About a third of the videos are from PAL sources, the others NTSC. About 3/4ths of the videos need complete remastering anyways, so I figured I'd just remaster (and standardize) them all.

For every single video I picked as my standards:

Format: MPEG-2 Video

Resolution: 720x480
Aspect Ratio: 4:3
Frame Rate: 29.97fps

Rate Control: VBR = 2000 Min / 7000 Avg / 8000 Max
VBR Buffer Size: 224kb

Video System: NTSC
Intra DC Prec: 10 bit
Video Mode: Interlaced
Field Order: TFF

Flags Used: NONE

Motion Search: Highest (w/ Error Correction)

GOP Structure: IBBPBBPBBPBBPBB

Picture Struct: Frame-Based
Scan Mode: Alternate
Quant Scale: Non-Linear

None of the videos I have originally match these properties, and mixing videos of different properties into one track just makes me itch all over! I know, it's totally OCD of me to do what I do, but the end result will be pretty much studio quality. =)

Each video takes less that 10 minutes to encode before DGPulldown is used on it, so I'm not worried about lost time. And quality never suffers, because each pre-pulldown encode I just run at VBR CQ100 @ 15000kbps.

Some authoring capabilities of TMPGEnc DVD Author v2.1.9.90 I consider to be absolutely priceless, and it's a pretty easy/reliable program to use. It does exactly what I want it to do. I don't expect anything authoring program to be dependable when it comes to flags either, even though it is an MPEG standard. It just seems fishy ground to me, that's all.

I do thank you for your input though manano!

I also have one question for you: Looking up at my encoding specifications, could you perhaps enlighten me into what you think of them? Are they good specs to use? What would you change about them? I know you'd probably say no flag use is ridiculous! Which I agree with you on. But other than that, does it seem okay? I'm shooting for ultimate DVD player compatibility, worldwide. I think I've got it down, but I seem to learn new stuff everyday in this forum!

Thank you so much for any insight you may be able to provide too.

My apologies for getting so off topic though! I wouldn't know what topic to put this post under actually! Oh well...


-------------------------------------------


BTW, I totally figured out what I was doing wrong that lead me to start this new thread:

I wasn't using ConvertToRGB(interlaced=true) before encoding my material w/ TMPGEnc Xpress 3.

I know TMPGEnc can read YV12 streams, but I found out that it can only read them correctly from direct M2V/MPG/VOB sources. If an avisynth script is imported using YV12, it doesn't know whether the script's chroma plane is progressive or interlaced, and so it always considered it progressive -- thus screwing up the chroma so much. Through some experiments, I found out that TMPGEnc internally converts an avisynth script to RGB (probably so its filters could be used), and then converts it back to YV12 right before encoding. The problem though is that when it converts to RGB, it does so progressively, and not interlaced.

I believe one has to convert one's avisynth scripts to YUY2 before encoding w/ CCE, correct? Well the same goes with TMPGEnc except that it requires RGB. For people actually reading this far into this post, always use ConvertToRGB(interlaced=true) in your avisynth BEFORE importing it into TMPGEnc, if you are indeed encoding truly interlaced footage. Otherwise a ConvertToRGB(interlaced=false) can be used, which is technically what TMPGEnc does internally anyways!

Whew. I hope that all made sense! I hope somebody else out there can use this little piece of information to their advantage someday! Because it sure solved all my problems with chroma!

Gotta get back to work, so I will shut up now. =)

[manono]

Hi-

I know you'd probably say no flag use is ridiculous! Which I agree with you on.

Yep, If I can encode as progressive, I always do.

Other than that it looks OK to me. Since these are music videos, often with a lot of action, I'd set my max bitrate as close to the limit as possible, in order to minimize artifacts such as mosquito noise, color smearing, and macroblocking. As I don't use TMPGEnc, I don't know whether or not it honors the max bitrate you set, but I'd go to at least 9500 max. Others might disagree. This also depends on the audio bitrate. If it's PCM audio, 8000 is about right. If it's DD 2.0@192, I'd jack up the max bitrate. You'll probably write another book telling me why you disagree. :)

...I found out that TMPGEnc internally converts an avisynth script to RGB...

RGB24, I believe.

[zoinbergs]

I'll try to keep this response short. =)

I decided to cap it off at 8000 because I've read in so many places that some players cannot handle video feeds over 8000kbps. They start to studder, I presume?

I'm upmixing all my audio to DD5.1 @ 448kbps, so my max could be around 9000 I guess.

I have to ask your expert opinion again, if that's okay with you: Am I overeacting to the compatibility settings I have chosen for bitrate? I REALLY want to go to 9000, but I guess you could say I'm a little scared. I don't want my video to studder!

I'm starting to think that if a DVD player can handle a DVD-R disc in the first place, chances are that it can handle a max bitrate of 9000 as well! I mean, we live in the 21st century, right?

Also, if I go to 9000, would there be any noticable difference in quality between an average bitrate of 7000, or 8000? I know it's not very wise of me to ask that, as I think there's some type of forum policy rule against asking what's "best" to use -- as there is no true answer! But perhaps you could make an exception in my case? You are a very wise person, and I could really use your guidance!

I'd like to use 2K/7K/9K if possible, but if 2K/8K/9K yields better results, then I'd love to use that.

Oh! Here's a good way to put my question actually: Would I be passing the point of diminishing returns (when it comes to quality) if I go from 7K average to 8K? Or would having the max bitrate set at 9K take care of any quality issues regardless of average bitrate?

Hmmm.....

I think I'll stop asking questions now! I could go on forever! =/



...I found out that TMPGEnc internally converts an avisynth script to RGB...

RGB24, I believe.

Really? I thought it was RGB32! My bad.

So does converting RGB32 back to RGB24 (as TMPGEnc has been doing for me) lose any color quality, other than the alpha channel (which I presume I don't use)? Because if it does, I'd like to save an extra color conversion along the way.

YV12 --> RGB32 --> RGB24 --> YV12

seems like a lot of color conversions to me!


-------------------------------

Well I'm certainly learning a lot these days! I *think* I'm slowly asking less simple questions that you experts in this forum are prolly tired of hearing!

Hopefully someday I'll be at the wisdom level of the rest of you, and be able to help the newbies out in here, kinda like what you all are doing for me right now! :)

[foxyshadis]

Mug Funky, a professional video engineer here, uses 8500 to guarantee compatibility across as many players as possible. There are a lot of extremely cheap fly-by-night chipsets in noname players (and even some big-name ones!), especially older ones, that choke on the entirely in-spec material. Well, if it's not going to be for broad distribution there's probably a fairly low chance of you hitting one of those.

[manono]

Hi-

I decided to cap it off at 8000 because I've read in so many places that some players cannot handle video feeds over 8000kbps. They start to studder, I presume?

All DVD players are required to handle the max video bitrate of 9800 kbps and 10080 kbps for everything together (audio, video, subs, and overhead). Whether or not they all do is another question. If you have one that can't (stutters, freezes), I'd toss it into the nearest trashcan. When I have DD 5.1@448 AC3 audio, I set the max video bitrate for 9600. Like I said, I don't know if TMPGEnc always honors the max set.

Also, if I go to 9000, would there be any noticable difference in quality between an average bitrate of 7000, or 8000?

Since I work with DVDs, I haven't had much experience using the very high average bitrates you're using. I'd say there could very well be a difference in the quality between when the average is 7000 and the max is 8000, and the average is 7000 and the max is, say, 9500. Make a couple of test encodes with those 2 sets of parameters and then compare the max Q-Levels in BitRate Viewer. The higher max bitrate could mean the difference between blocking up during complex scenes and not blocking up. This also depends on the quality of the source. If the source is all blocked up because of a low max bitrate, or the action is just too intense, then there's not much you can do.

Oh! Here's a good way to put my question actually: Would I be passing the point of diminishing returns (when it comes to quality) if I go from 7K average to 8K? Or would having the max bitrate set at 9K take care of any quality issues regardless of average bitrate?

Like I said, I don't often have the luxury of working with such high bitrates. My own opinion is, if the source is excellent, I wouldn't worry much about whether or not my average is 7 or 8000, but what my max is, and even more, which quantisation matrix I can safely use. If you're getting quality music videos from DVDs, you might start checking out the matrices they're using. If you cap them, or get them from other sources, I wouldn't worry about it I guess, as they probably don't look like much anyway. I've seen a lot of crummy looking music videos in my time.

So does converting RGB32 back to RGB24 (as TMPGEnc has been doing for me) lose any color quality,...

Not my speciality, I'm glad to say. The whole subject gives me a headache.

[zoinbergs]

I don't know if TMPGEnc always honors the max set.

I forgot to comment on this last time! Using BV I've never seen TMPGEnc (Xpress 3 at least) actually encode using more bits than what I set as the max. It's a pretty reliable program, IMO.


As far as the answers to my question goes, I thank you both for being so helpful! I have a pretty good idea of what I'm gonna do now: 2K/7K/9K. Most of the videos I've encoded so far show maxing out at around 8500, to support what foxy was saying about MugFunky, and the full 9000 is there if the scene really needs it!

The source for almost all these videos is indeed DVD, so I think I might have to take a look at what matrices they're using! Good idea manano. =)

I have also seen MANY crummy looking music videos in my time.. it is a shame what some other encoders out there are doing! It makes me all sad inside to see how butchered some videos have become. =( Kinda why I'm doing what I'm doing I guess! Trying to save as many videos (in the quality they deserve) as I can!

Thank you again for all your input manano (and foxy!). Another lesson in MPEG2 authoring down. 965 (or something) to go!


Not my speciality, I'm glad to say. The whole subject gives me a headache.

Tell me about it! I cannot even recall how many hours I have spent so far learning about RGB, YUY2, and YV12. What a mess I say! What a mess...