HCenc 023 released

[Koutsoubos]

I've been observing this in a short test:

Source: 25fps PAL true progressive.
Aim: to make a 25fps PAL DVD preserving the progressive frames

1st Attempt (Progressive Frame=On; TFF=on; Progressive Sequence=off)
This is what is advised as CCE-like output. Plays fine on interlaced PAL displays. On two progressive-scan PAL players (different brands), with progressive PAL displays (different brands), very obvious combing artefacts. Perhaps players are wrongly assuming from TFF flag that frames are interlaced?

2nd Attempt (Progressive Frame=On; TFF=on; Progressive Sequence=on)
Same problem. It seems that the TFF throws the tested progressive scan models off.

3rd Attempt (Progressive Frame=On; TFF=off; Progressive Sequence=on)
Now displays are just "normal". Note that this is the old HCEnc and QuEnc behaviour with "progressive" encodes. It seems that Procoder also behaves this way: http://forum.doom9.org/showthread.ph...53#post1061653

But ... it is said that Progressive_Sequence=On is out of spec for DVD??? http://forum.videohelp.com/threads/2...=1#post1378458 [Note: I cannot find such a remark in the current version of the DVD FAQ]

In another discussion about menus, users are advised to set Prog_frame and Prog_seq both to ON even though these are "theoretical spec violations"?? http://www.dvdafteredit.com/node/1433


Oh, what is happening and how does one make a proper 25fps progressive DVD?

[Koutsoubos]

Actually, is it actually known as a fact that progressive_sequence=ON is out of spec in DVDs? (Of course those who really know are under NDA)

See here, where a Progressive_sequence problem is flagged for the reason: "progressive_sequence shall be 1 when the vertical_size equals 240": http://www.ip.philips.com/download_a.../4119/4119.pdf

[Guest]

The 1st attempt should work. Maybe you have a field shift in your progressive PAL as inputted to HCEnc.

[Koutsoubos]

Regarding 1st attempt: I think there is no field shift because the source is absolutely progressive 25fps PAL. Nothing is interlaced.

[Guest]

Post a link to an unprocessed source sample.

[Koutsoubos]

The source is big. But anyway, it is absolutely 25fps progressive AVI. No mistake there.

The inquiry should be about the flags now, as to what is permissible under DVD

[Guest]

You can direct stream copy a small sample.

Also, give a sample of the encoded M2V.

Anyway, no samples, no help from me.

[hank315]

@Koutsoubos
This table can be found on the web, the "Pulldown Table of Truth"

nr  prog    prog      pic       TFF    RFF                                                                                                                                                              
    seq     frame     struct

 1  0       0         Field     0      0     First coded field displayed first (TB or BT)
 2  0       0         Field     0      1     Illegal combination
 3  0       0         Field     1      0     Illegal combination
 4  0       0         Field     1      1     Illegal combination
 5  0       0         Frame     0      0     Bottom first, 2 fields displayed (BT)
 6  0       0         Frame     0      1     Illegal combination
 7  0       0         Frame     1      0     Top first, 2 fields displayed (TB)
 8  0       0         Frame     1      1     Illegal combination
 9  0       1         Field     0      0     Illegal combination
10  0       1         Field     0      1     Illegal combination
11  0       1         Field     1      0     Illegal combination
12  0       1         Field     1      1     Illegal combination
13  0       1         Frame     0      0     Bottom first, 2 fields displayed (BT)
14  0       1         Frame     0      1     Bottom first, 3 fields displayed (BTB)
15  0       1         Frame     1      0     Top first, 2 fields displayed (TB)
16  0       1         Frame     1      1     Top first, 3 fields displayed (TBT)
17  1       0         Field     0      0     Illegal combination
18  1       0         Field     0      1     Illegal combination
19  1       0         Field     1      0     Illegal combination
20  1       0         Field     1      1     Illegal combination
21  1       0         Frame     0      0     Illegal combination
22  1       0         Frame     0      1     Illegal combination
23  1       0         Frame     1      0     Illegal combination
24  1       0         Frame     1      1     Illegal combination
25  1       1         Field     0      0     Illegal combination
26  1       1         Field     0      1     Illegal combination
27  1       1         Field     1      0     Illegal combination
28  1       1         Field     1      1     Illegal combination
29  1       1         Frame     0      0     1 prog frame displayed
30  1       1         Frame     0      1     2 progressive frames displayed (illegal in MP@ML)
31  1       1         Frame     1      0     Illegal combination
32  1       1         Frame     1      1     3 progressive frames displayed (illegal in MP@ML)

This would mean your 1st attempt should be OK (nr 15).
Your 2nd attempt is illegal (nr 31).
The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

[qyot27]

Using the 04-04-2010 beta, is the TFF flag being set by default on progressive encodes? ReStream seems to think so, at any rate. If it is, how can I turn it off? My .ini does not include the *TFF option - I actually used the *PROG_SEQ option but the resulting encode didn't have that flag set, again according to ReStream. This is why I'm thinking it's flagging for TFF and then ignoring the PROG_SEQ parameter.

I ask, because I was encoding 25fps material and then using DGPulldown for 25->29.97. On playback in MPC, the files had line jumping issues that went away if I opened the .m2v in ReStream, set the Progressive Sequence header, and then remuxed.

[hank315]

No, the 04-04-2010 beta doesn't set TFF by default.
But if the stream has been processed by DGPulldown, some frames will have the TFF set, it's just the way pulldown works.
I don't know how Restream handles pulldowned streams.

In HCenc you can always disable TFF by using the *BFF command, this will clear the TFF flag.

[Koutsoubos]

Originally Posted by hank315

This would mean your 1st attempt should be OK (nr 15).
Your 2nd attempt is illegal (nr 31).
The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

Thanks for replying. I don't know much about the "Pulldown Table of Truth". This is the first time I have heard of it.

The 1st attempt is HCenc's current behaviour (granted that the TFF has to be set manually). This gives Progressive_Frame=ON; TFF=ON; Progressive_Sequence=OFF

This "legal" attempt is what's giving artifacts on my progressive setups.


The 3rd attempt was HCenc's default behaviour prior to the 04-04-10 version. Previous versions would give Progressive_Frame=ON; TFF=OFF; Progressive_Sequence=ON.

The 3rd attempt, albeit "illegal", is displaying fine on my progressive and interlaced setups !?

Prior to 040410, the 3rd attempt would also be the output if I clicked on the "ensure DVD compliance" option.

[Bear in mind that my source is absolutely kosher 25fps PAL progressive. I don't know about NTSC or pulldowns, which may involve other conventions or rule-sets.]


1) The $64000 question is, what set of flags should I adopt? A lot will have to depend on whether Progressive_sequence=ON is allowed in DVD MPEG2 spec. We have only very obscure and indirect references. Frankly I have my doubts and would really appreciate something more direct and authoritative.

2) This comment from DVD FAQ #3.4:

In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays. In other words, the player doesn't "know" what the encoded rate is, it simply follows the MPEG-2 encoder's instructions to produce the predetermined display rate of 25 fps or 29.97 fps.

The $128,000 question. Can I feed the 25fps source into HCEnce as 24fps, so that HCEnc adds the 2:2: pulldown flags for PAL encoding? What would the flags be, or maybe I should ask what flags I should activate then?

[qyot27]

Yeah, my previous inquiry as to the TFF flags and PROG_SEQ option were wrong. I re-did some of the tests and found out that what I was seeing was related to the fact that pulldown sets TFF flags normally - in specific, it's that DGPulldown removes the PROG_SEQ header when performing pulldown. I didn't think of that possibility before. So I'm thinking that it's just my software MPEG-2 decoder that has a problem with that sort of pulldown, although manually respecifying the Progressive sequence flag doesn't cause a problem on my DVD player.

Originally Posted by Koutsoubos

2) This comment from DVD FAQ #3.4:

In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays. In other words, the player doesn't "know" what the encoded rate is, it simply follows the MPEG-2 encoder's instructions to produce the predetermined display rate of 25 fps or 29.97 fps.

The $128,000 question. Can I feed the 25fps source into HCEnce as 24fps, so that HCEnc adds the 2:2: pulldown flags for PAL encoding? What would the flags be, or maybe I should ask what flags I should activate then?

2:2 pulldown isn't used on 24fps footage. It's used on 25fps footage. As it said in the FAQ, "or 2-2 pulldown (with resulting 4% speedup)" - the speedup has to be performed prior to the pulldown, meaning you just speed up 24fps to 25fps and then the pulldown happens. And if my interpretation of Wikipedia's Telecine article is correct, 2:2 pulldown contains no flagging because it's simply a regular conversion of a progressive stored frame into two interlaced fields. The exact same thing happens for progressive 29.97fps footage in NTSC regions.

[Koutsoubos]

Originally Posted by qyot27

Yeah, my previous inquiry as to the TFF flags and PROG_SEQ option were wrong. I re-did some of the tests and found out that what I was seeing was related to the fact that pulldown sets TFF flags normally - in specific, it's that DGPulldown removes the PROG_SEQ header when performing pulldown. I didn't think of that possibility before. So I'm thinking that it's just my software MPEG-2 decoder that has a problem with that sort of pulldown, although manually respecifying the Progressive sequence flag doesn't cause a problem on my DVD player.


2:2 pulldown isn't used on 24fps footage. It's used on 25fps footage. As it said in the FAQ, "or 2-2 pulldown (with resulting 4% speedup)" - the speedup has to be performed prior to the pulldown, meaning you just speed up 24fps to 25fps and then the pulldown happens. And if my interpretation of Wikipedia's Telecine article is correct, 2:2 pulldown contains no flagging because it's simply a regular conversion of a progressive stored frame into two interlaced fields. The exact same thing happens for progressive 29.97fps footage in NTSC regions.

Your setup is different. It seems you're doing a PAL --> NTSC conversion. Mine is PAL all the way, even to DVD. So I'm wondering about the flags and all, what is permissible, because it seems Prog_Seq=ON solves the problems I am observing. It could well be that what applies to your workflow does not apply to mine.

2:2 pulldown etc. This is "black magic" to me. The passage from DVD-FAQ sounded as if Repeat_field flags are added to individual frames or something. Are you saying that no flags are added, and that the decoder should be able to repeat frames on its own?

[qyot27]

Originally Posted by Koutsoubos

Your setup is different. It seems you're doing a PAL --> NTSC conversion. Mine is PAL all the way, even to DVD. So I'm wondering about the flags and all, what is permissible, because it seems Prog_Seq=ON solves the problems I am observing. It could well be that what applies to your workflow does not apply to mine.

Well, that part was just an update on my previous post, I wasn't directing it in my quote.

2:2 pulldown etc. This is "black magic" to me. The passage from DVD-FAQ sounded as if Repeat_field flags are added to individual frames or something. Are you saying that no flags are added, and that the decoder should be able to repeat frames on its own?

No frames would be repeated at all, no flags would be set (again, from how I understand it; I don't claim to be any kind of expert on pulldown flagging). The player would manually interlace the progressive frames on the fly, turning each frame into two fields and giving them to the display interlaced. The point being, if it was 24 (or rather, 23.976) fps, then it would undergo 3:2 pulldown, which involves the spreading of fields delayed/repeated over time so as to make it compatible with 29.97 fps (or 25 fps, in the case of Euro pulldown which is 2:2:2:2:2:2:2:2:2:2:2:3). In order to make 23.976fps Film content compatible with 25fps for PAL, the traditional method involves speeding up the video by 4% and adjusting the audio pitch so that things aren't out of sync. Then the pulldown is applied just like it is to content produced natively for 25fps. You can see what 2:2 pulldown actually does in diagram form on the Wikipedia page for Telecine. The difference is that - again, my understanding - 2:2 pulldown isn't actually flagged, it's assumed by the player. 3:2, Euro, 25->29.97 pulldown, these are all flagged because the input and output framerates don't match. In 2:2, the input/output does match, meaning flags aren't necessary.

Interlacing all progressive frames on playback is probably the automatic behavior of the player, so it might be able to be adjusted either A) in the DVD player's settings by forcibly setting for progressive scan if it hasn't been already or B) the progressive_sequence header being ON tells the player not to perform the interlacing, resulting in the player giving the display normal progressive frames, not interlaced ones. Even if it's not spec-compliant, it would be counted as a necessary evil.

[Koutsoubos]

Well, despite some potentially misleading uses of the term "2:2 pulldown" from web sources, I am now convinced that it is a nullity as stated so expertly by neuron2 here: http://forum.doom9.org/showthread.ph...68#post1187268


Now, I'm hoping that someone can shed more light on the progressive_sequence flag? For a definite fact, is it or is it not a violation of spec?

If it is a violation of spec, what is a good way for me to encode and/or flag a progressive 25fps PAL source into a DVD-compliant MPEG2 stream?

[Guest]

You can encode progressive with progressive_sequence false. I don't understand your problem. However, I only read the last post of the thread.

[mp3dom]

2:2 pulldown on PAL land is simply the term that mean: on a 25p film (24->25 with 4% speedup) display 2 fields for every frame a.k.a. display a progressive frame in an interlaced way without field shift. Basically if you're encoding your video as interlaced, you've already made a '2:2 pulldown'. The term pulldown is erroneous here, compared to what 'pulldown' means in NTSC land.
Anyway, regarding the progressive_sequence flag... if you encode your video with progressive_sequence flag on, top field off, progressive_frame on, picture structure frame (table #29) you're making a in-spec stream for PAL. If you're encoding a 24p in that way and THEN apply the pulldown (for 24p->50i with field repetition) your stream will be like table #15/#16 which is, again, in spec.

[Koutsoubos]

Originally Posted by hank315

The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

Originally Posted by mp3dom

Anyway, regarding the progressive_sequence flag... if you encode your video with progressive_sequence flag on, top field off, progressive_frame on, picture structure frame (table #29) you're making a in-spec stream for PAL. If you're encoding a 24p in that way and THEN apply the pulldown (for 24p->50i with field repetition) your stream will be like table #15/#16 which is, again, in spec.

Alright, here's the point-blank question. In respect of DVD specs (not just generic MPEG2), if I encode my 25fps progressive PAL source into a stream with:

1. TFF flag off
2. Progressive frame ON
3. Progressive sequence ON


Is this within the DVD spec?
Does the "Pulldown Table of Truth" #29 mean conclusively that this is allowed within DVD specifications? I mean, this is PAL and not NTSC (note the 3:2 in the title "3:2 Pulldown Table of Truth")

[mp3dom]

Yes, it's within the dvd specs as long as the flags/encode are made in the proper way. Note that also picture structure must be fixed to 'frame' and not change over time. I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process).

[Koutsoubos]

Originally Posted by mp3dom

Yes, it's within the dvd specs as long as the flags/encode are made in the proper way. Note that also picture structure must be fixed to 'frame' and not change over time. I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process).

Thanks for that clarification. Yes, all the frames are encoded as frames. No fields.

So, DVD FAQ #3.4: http://www.dvddemystified.com/dvdfaq.html#3.4
where it says: "MPEG-2 progressive_sequence is not allowed"

Can we debunk this FOR CERTAIN?


Also:

"In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays"

Can we lay this to rest, at least for PAL?