HCenc 023 released

[Koutsoubos]

I've been observing this in a short test:

Source: 25fps PAL true progressive.
Aim: to make a 25fps PAL DVD preserving the progressive frames

1st Attempt (Progressive Frame=On; TFF=on; Progressive Sequence=off)
This is what is advised as CCE-like output. Plays fine on interlaced PAL displays. On two progressive-scan PAL players (different brands), with progressive PAL displays (different brands), very obvious combing artefacts. Perhaps players are wrongly assuming from TFF flag that frames are interlaced?

2nd Attempt (Progressive Frame=On; TFF=on; Progressive Sequence=on)
Same problem. It seems that the TFF throws the tested progressive scan models off.

3rd Attempt (Progressive Frame=On; TFF=off; Progressive Sequence=on)
Now displays are just "normal". Note that this is the old HCEnc and QuEnc behaviour with "progressive" encodes. It seems that Procoder also behaves this way: http://forum.doom9.org/showthread.php?p=1061653#post1061653

But ... it is said that Progressive_Sequence=On is out of spec for DVD??? http://forum.videohelp.com/threads/234894-How-to-get-progressive-ouput-using-cce-sp?p=1378458&viewfull=1#post1378458 [Note: I cannot find such a remark in the current version of the DVD FAQ]

In another discussion about menus, users are advised to set Prog_frame and Prog_seq both to ON even though these are "theoretical spec violations"?? http://www.dvdafteredit.com/node/1433


Oh, what is happening and how does one make a proper 25fps progressive DVD?

[Koutsoubos]

Actually, is it actually known as a fact that progressive_sequence=ON is out of spec in DVDs? (Of course those who really know are under NDA)

See here, where a Progressive_sequence problem is flagged for the reason: "progressive_sequence shall be 1 when the vertical_size equals 240": www.ip.philips.com/download_attachment/4119/4119.pdf

[Guest]

The 1st attempt should work. Maybe you have a field shift in your progressive PAL as inputted to HCEnc.

[Koutsoubos]

Regarding 1st attempt: I think there is no field shift because the source is absolutely progressive 25fps PAL. Nothing is interlaced.

[Guest]

Post a link to an unprocessed source sample.

[Koutsoubos]

The source is big. But anyway, it is absolutely 25fps progressive AVI. No mistake there.

The inquiry should be about the flags now, as to what is permissible under DVD

[Guest]

You can direct stream copy a small sample.

Also, give a sample of the encoded M2V.

Anyway, no samples, no help from me.

[hank315]

@Koutsoubos
This table can be found on the web, the "Pulldown Table of Truth"

nr prog prog pic TFF RFF
seq frame struct

1 0 0 Field 0 0 First coded field displayed first (TB or BT)
2 0 0 Field 0 1 Illegal combination
3 0 0 Field 1 0 Illegal combination
4 0 0 Field 1 1 Illegal combination
5 0 0 Frame 0 0 Bottom first, 2 fields displayed (BT)
6 0 0 Frame 0 1 Illegal combination
7 0 0 Frame 1 0 Top first, 2 fields displayed (TB)
8 0 0 Frame 1 1 Illegal combination
9 0 1 Field 0 0 Illegal combination
10 0 1 Field 0 1 Illegal combination
11 0 1 Field 1 0 Illegal combination
12 0 1 Field 1 1 Illegal combination
13 0 1 Frame 0 0 Bottom first, 2 fields displayed (BT)
14 0 1 Frame 0 1 Bottom first, 3 fields displayed (BTB)
15 0 1 Frame 1 0 Top first, 2 fields displayed (TB)
16 0 1 Frame 1 1 Top first, 3 fields displayed (TBT)
17 1 0 Field 0 0 Illegal combination
18 1 0 Field 0 1 Illegal combination
19 1 0 Field 1 0 Illegal combination
20 1 0 Field 1 1 Illegal combination
21 1 0 Frame 0 0 Illegal combination
22 1 0 Frame 0 1 Illegal combination
23 1 0 Frame 1 0 Illegal combination
24 1 0 Frame 1 1 Illegal combination
25 1 1 Field 0 0 Illegal combination
26 1 1 Field 0 1 Illegal combination
27 1 1 Field 1 0 Illegal combination
28 1 1 Field 1 1 Illegal combination
29 1 1 Frame 0 0 1 prog frame displayed
30 1 1 Frame 0 1 2 progressive frames displayed (illegal in MP@ML)
31 1 1 Frame 1 0 Illegal combination
32 1 1 Frame 1 1 3 progressive frames displayed (illegal in MP@ML)

This would mean your 1st attempt should be OK (nr 15).
Your 2nd attempt is illegal (nr 31).
The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

[qyot27]

Using the 04-04-2010 beta, is the TFF flag being set by default on progressive encodes? ReStream seems to think so, at any rate. If it is, how can I turn it off? My .ini does not include the *TFF option - I actually used the *PROG_SEQ option but the resulting encode didn't have that flag set, again according to ReStream. This is why I'm thinking it's flagging for TFF and then ignoring the PROG_SEQ parameter.

I ask, because I was encoding 25fps material and then using DGPulldown for 25->29.97. On playback in MPC, the files had line jumping issues that went away if I opened the .m2v in ReStream, set the Progressive Sequence header, and then remuxed.

[hank315]

No, the 04-04-2010 beta doesn't set TFF by default.
But if the stream has been processed by DGPulldown, some frames will have the TFF set, it's just the way pulldown works.
I don't know how Restream handles pulldowned streams.

In HCenc you can always disable TFF by using the *BFF command, this will clear the TFF flag.

[Koutsoubos]

This would mean your 1st attempt should be OK (nr 15).
Your 2nd attempt is illegal (nr 31).
The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.


Thanks for replying. I don't know much about the "Pulldown Table of Truth". This is the first time I have heard of it.

The 1st attempt is HCenc's current behaviour (granted that the TFF has to be set manually). This gives Progressive_Frame=ON; TFF=ON; Progressive_Sequence=OFF

This "legal" attempt is what's giving artifacts on my progressive setups.


The 3rd attempt was HCenc's default behaviour prior to the 04-04-10 version. Previous versions would give Progressive_Frame=ON; TFF=OFF; Progressive_Sequence=ON.

The 3rd attempt, albeit "illegal", is displaying fine on my progressive and interlaced setups !?

Prior to 040410, the 3rd attempt would also be the output if I clicked on the "ensure DVD compliance" option.

[Bear in mind that my source is absolutely kosher 25fps PAL progressive. I don't know about NTSC or pulldowns, which may involve other conventions or rule-sets.]


1) The $64000 question is, what set of flags should I adopt? A lot will have to depend on whether Progressive_sequence=ON is allowed in DVD MPEG2 spec. We have only very obscure and indirect references. Frankly I have my doubts and would really appreciate something more direct and authoritative.

2) This comment from DVD FAQ #3.4:
In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays. In other words, the player doesn't "know" what the encoded rate is, it simply follows the MPEG-2 encoder's instructions to produce the predetermined display rate of 25 fps or 29.97 fps.

The $128,000 question. Can I feed the 25fps source into HCEnce as 24fps, so that HCEnc adds the 2:2: pulldown flags for PAL encoding? What would the flags be, or maybe I should ask what flags I should activate then?

[qyot27]

Yeah, my previous inquiry as to the TFF flags and PROG_SEQ option were wrong. I re-did some of the tests and found out that what I was seeing was related to the fact that pulldown sets TFF flags normally - in specific, it's that DGPulldown removes the PROG_SEQ header when performing pulldown. I didn't think of that possibility before. So I'm thinking that it's just my software MPEG-2 decoder that has a problem with that sort of pulldown, although manually respecifying the Progressive sequence flag doesn't cause a problem on my DVD player.

2) This comment from DVD FAQ #3.4:
In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays. In other words, the player doesn't "know" what the encoded rate is, it simply follows the MPEG-2 encoder's instructions to produce the predetermined display rate of 25 fps or 29.97 fps.

The $128,000 question. Can I feed the 25fps source into HCEnce as 24fps, so that HCEnc adds the 2:2: pulldown flags for PAL encoding? What would the flags be, or maybe I should ask what flags I should activate then?
2:2 pulldown isn't used on 24fps footage. It's used on 25fps footage. As it said in the FAQ, "or 2-2 pulldown (with resulting 4% speedup)" - the speedup has to be performed prior to the pulldown, meaning you just speed up 24fps to 25fps and then the pulldown happens. And if my interpretation of Wikipedia's Telecine article is correct, 2:2 pulldown contains no flagging because it's simply a regular conversion of a progressive stored frame into two interlaced fields. The exact same thing happens for progressive 29.97fps footage in NTSC regions.

[Koutsoubos]

Yeah, my previous inquiry as to the TFF flags and PROG_SEQ option were wrong. I re-did some of the tests and found out that what I was seeing was related to the fact that pulldown sets TFF flags normally - in specific, it's that DGPulldown removes the PROG_SEQ header when performing pulldown. I didn't think of that possibility before. So I'm thinking that it's just my software MPEG-2 decoder that has a problem with that sort of pulldown, although manually respecifying the Progressive sequence flag doesn't cause a problem on my DVD player.


2:2 pulldown isn't used on 24fps footage. It's used on 25fps footage. As it said in the FAQ, "or 2-2 pulldown (with resulting 4% speedup)" - the speedup has to be performed prior to the pulldown, meaning you just speed up 24fps to 25fps and then the pulldown happens. And if my interpretation of Wikipedia's Telecine article is correct, 2:2 pulldown contains no flagging because it's simply a regular conversion of a progressive stored frame into two interlaced fields. The exact same thing happens for progressive 29.97fps footage in NTSC regions.


Your setup is different. It seems you're doing a PAL --> NTSC conversion. Mine is PAL all the way, even to DVD. So I'm wondering about the flags and all, what is permissible, because it seems Prog_Seq=ON solves the problems I am observing. It could well be that what applies to your workflow does not apply to mine.

2:2 pulldown etc. This is "black magic" to me. The passage from DVD-FAQ sounded as if Repeat_field flags are added to individual frames or something. Are you saying that no flags are added, and that the decoder should be able to repeat frames on its own?

[qyot27]

Your setup is different. It seems you're doing a PAL --> NTSC conversion. Mine is PAL all the way, even to DVD. So I'm wondering about the flags and all, what is permissible, because it seems Prog_Seq=ON solves the problems I am observing. It could well be that what applies to your workflow does not apply to mine.
Well, that part was just an update on my previous post, I wasn't directing it in my quote.

2:2 pulldown etc. This is "black magic" to me. The passage from DVD-FAQ sounded as if Repeat_field flags are added to individual frames or something. Are you saying that no flags are added, and that the decoder should be able to repeat frames on its own?
No frames would be repeated at all, no flags would be set (again, from how I understand it; I don't claim to be any kind of expert on pulldown flagging). The player would manually interlace the progressive frames on the fly, turning each frame into two fields and giving them to the display interlaced. The point being, if it was 24 (or rather, 23.976) fps, then it would undergo 3:2 pulldown, which involves the spreading of fields delayed/repeated over time so as to make it compatible with 29.97 fps (or 25 fps, in the case of Euro pulldown which is 2:2:2:2:2:2:2:2:2:2:2:3). In order to make 23.976fps Film content compatible with 25fps for PAL, the traditional method involves speeding up the video by 4% and adjusting the audio pitch so that things aren't out of sync. Then the pulldown is applied just like it is to content produced natively for 25fps. You can see what 2:2 pulldown actually does in diagram form on the Wikipedia page for Telecine (http://en.wikipedia.org/wiki/Telecine#2:2_pulldown). The difference is that - again, my understanding - 2:2 pulldown isn't actually flagged, it's assumed by the player. 3:2, Euro, 25->29.97 pulldown, these are all flagged because the input and output framerates don't match. In 2:2, the input/output does match, meaning flags aren't necessary.

Interlacing all progressive frames on playback is probably the automatic behavior of the player, so it might be able to be adjusted either A) in the DVD player's settings by forcibly setting for progressive scan if it hasn't been already or B) the progressive_sequence header being ON tells the player not to perform the interlacing, resulting in the player giving the display normal progressive frames, not interlaced ones. Even if it's not spec-compliant, it would be counted as a necessary evil.

[Koutsoubos]

Well, despite some potentially misleading uses of the term "2:2 pulldown" from web sources, I am now convinced that it is a nullity as stated so expertly by neuron2 here: http://forum.doom9.org/showthread.php?p=1187268#post1187268


Now, I'm hoping that someone can shed more light on the progressive_sequence flag? For a definite fact, is it or is it not a violation of spec?

If it is a violation of spec, what is a good way for me to encode and/or flag a progressive 25fps PAL source into a DVD-compliant MPEG2 stream?

[Guest]

You can encode progressive with progressive_sequence false. I don't understand your problem. However, I only read the last post of the thread. :)

[mp3dom]

2:2 pulldown on PAL land is simply the term that mean: on a 25p film (24->25 with 4% speedup) display 2 fields for every frame a.k.a. display a progressive frame in an interlaced way without field shift. Basically if you're encoding your video as interlaced, you've already made a '2:2 pulldown'. The term pulldown is erroneous here, compared to what 'pulldown' means in NTSC land.
Anyway, regarding the progressive_sequence flag... if you encode your video with progressive_sequence flag on, top field off, progressive_frame on, picture structure frame (table #29) you're making a in-spec stream for PAL. If you're encoding a 24p in that way and THEN apply the pulldown (for 24p->50i with field repetition) your stream will be like table #15/#16 which is, again, in spec.

[Koutsoubos]

The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.



Anyway, regarding the progressive_sequence flag... if you encode your video with progressive_sequence flag on, top field off, progressive_frame on, picture structure frame (table #29) you're making a in-spec stream for PAL. If you're encoding a 24p in that way and THEN apply the pulldown (for 24p->50i with field repetition) your stream will be like table #15/#16 which is, again, in spec.



Alright, here's the point-blank question. In respect of DVD specs (not just generic MPEG2), if I encode my 25fps progressive PAL source into a stream with:

1. TFF flag off
2. Progressive frame ON
3. Progressive sequence ON


Is this within the DVD spec?
Does the "Pulldown Table of Truth" #29 mean conclusively that this is allowed within DVD specifications? I mean, this is PAL and not NTSC (note the 3:2 in the title "3:2 Pulldown Table of Truth")

[mp3dom]

Yes, it's within the dvd specs as long as the flags/encode are made in the proper way. Note that also picture structure must be fixed to 'frame' and not change over time. I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process). :)

[Koutsoubos]

Yes, it's within the dvd specs as long as the flags/encode are made in the proper way. Note that also picture structure must be fixed to 'frame' and not change over time. I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process). :)

Thanks for that clarification. Yes, all the frames are encoded as frames. No fields.

So, DVD FAQ #3.4: http://www.dvddemystified.com/dvdfaq.html#3.4
where it says: "MPEG-2 progressive_sequence is not allowed"

Can we debunk this FOR CERTAIN?


Also:

"In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays"

Can we lay this to rest, at least for PAL?

[mp3dom]

Can we debunk this FOR CERTAIN?
To say 'for certain' you must have the dvd-video specification from the DVDForum. What I can say is that I've made a lot of these dvd and no one complain about strange incompatibility with standalone players. Also, EclipseSuite (which is very reliable) validate the verification.


"In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays"
Can we lay this to rest, at least for PAL?

If your source is 24p and you want to convert to PAL there are only 3 ways:
- Encode as 25p with 4% speedup
- Encode as 50i with 4% speedup (this is the 2:2 pulldown)
- Encode as 24p with 2^12:3 euro-pulldown

As for the third option, there's no progressive-scan players able to reconstruct the original 24p and display at 25p since the cadence is quite uncommon. You'll see it always as interlaced with 2 stutters every second. This is different with software dvd decoders that will output true 24p. I don't know if this happends even for dvd players with HDMI output. If the player can output 24p via HDMI it could be that the hdtv will display the original 24p... but I've never tried it.

[Koutsoubos]

Point One is quite settled in my opinion, and to draw it back to the topic of this thread, the Prog_frame=ON TFF=off Prog_sequence=ON was the default behaviour in HCEnc when doing progressive encodes.

I think it'd be good to have that default behaviour back? But anyway the choices are now clear.


As to the second point of 2:2 pulldown again! Apparently this is a nullity. Are there any real world applications of 50i, where 2:2 pulldown happens, when encoding from progressive sources (film or not, it has been sped up to 25fps frame-wise)?

Is this another possibility for HCEnc? 50i?

[qyot27]

Are there any real world applications of 50i, where 2:2 pulldown happens, when encoding from progressive sources (film or not, it has been sped up to 25fps frame-wise)?

Is this another possibility for HCEnc? 50i?
50i = 25fps interlaced. The number differs only because 50i measures the total fieldrate per second, rather than relating it in terms of how many complete frames per second - interlaced or not - are being displayed.

So yes, there are plenty - anything that broadcasts or displays interlaced material uses it.

[Koutsoubos]

I was responding to mp3coder's observation that with a progressive source we could choose to "Encode as 50i with 4% speedup (this is the 2:2 pulldown)".

It may seem like splitting hairs, but what I was driving at is whether that would entail interlacing the progressive source and causing the MPEG stream to be hard-encoded at 50i (i.e. picture structure = field)

If, when fed a truly progressive source, HCEnc or any other EN-coder splits the frame into two fields, it would be hard 50i encoding and then we'd probably have a case where "2:2 pulldown" is an apt description.

Of course, interlacing a progressive source is at the EN-coder level is programmmatically simple. But should we do it? Is it done this way? Perhaps. I don't know.

Right now it seems that HCenc (or any other encoder) does not do that to a truly progressive source. Instead the frames are encoded (25fps nominally) and it is the job of the DE-coder to interlace the output auto-magically. There is no "pulldown", in the strict sense of the term, by the EN-coder.

And even if such an option were available, I still think it would be cleaner to encode as progressive stream.

Any thoughts?

[Koutsoubos]

I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process). :)


You know, I was pretty convinced. But just a few minutes ago I came across this statement:
"Although Eclipse Image Analysis is crucial, it does not check the VIDEO_TS folder for DVD Spec compliance (IFO files, VOB data structure, navigation commands, etc)."
http://www.dvdverification.com/public/119.cfm


I honestly wish that this basic information became public instead of staying subject to some hefty licence fee and NDA.

[mp3dom]

It may seem like splitting hairs, but what I was driving at is whether that would entail interlacing the progressive source and causing the MPEG stream to be hard-encoded at 50i (i.e. picture structure = field)

Picture structure field is not the only choice for encoding interlaced. Very often, for interlaced encoding the picture structure is still 'frame'. Generally, the field structure is optimized for high-motion interlaced scene, whereas the frame structure work best for progressive source or moderate-motion interlaced scene. Frame structure is also more compatible. You can also mix the structure on a per-frame basis (so use frame structure for normal motion and switch to field when the encoder detects high-motion). The latter method is quite incompatible since it's a situation not covered by the specs (the specs says that both field and frame structure must be supported by mpeg decoder but doesn't says anything regarding the fact that the structure could be switched over time on the same segment. So some player support it, and other doesn't).
Returning in topic, if you encode your progressive source as interlaced (top field, frame or field structure, alternate scanning order etc etc), you're making a '2:2 pulldown encode'.


If, when fed a truly progressive source, HCEnc or any other EN-coder splits the frame into two fields, it would be hard 50i encoding and then we'd probably have a case where "2:2 pulldown" is an apt description.

Exactly, but this depends by your settings and your needs.


Of course, interlacing a progressive source is at the EN-coder level is programmmatically simple. But should we do it? Is it done this way? Perhaps. I don't know.

A lot of dvd were encoded in that way (with interlaced flags). This also prevent some dvd players to output in progressive-scan (if they rely completely to the flags embedded in the mpeg2 stream)


But just a few minutes ago I came across this statement:
I don't remember which part of the suite were used, but I've received a report from the replicator (made with the EclipseSuite) with a warning regarding the dts audio: "Since dts is not mandatory, some dvd players could be not able to decode it correctly". So I think that at least one tool of the suite have indeed read the IFO/VOB.

[foxyshadis]

That might be a lawyer-worded way of saying that while they'll check for their idea of conformance, they do not guarantee that their idea is a perfect complete test of the spec. Otherwise, lawsuits and yada yada.

[manolito]

First of all a small bug report for HC 0.24 beta 04-04-2010:
In 1-pass VBR mode the HC log file always reports an average quantizer which is too small by a factor of 10 (0.400 instead of 4.00). Haven't tested it in 2-pass VBR. Peanuts...:)

@Koutsoubos
Interesting discussion, but I still think you should follow neuron's advice and post a sample. If your source is purely progressive, but your display chain still shows combing artifacts, a phase shift in your source is pretty much the only explanation.

Just assume for a moment that I (and neuron) are right and your source is phase shifted by 1 field:

Play it through a standard DVD player (no progressive scan) on an interlaced display (CRT), no combing would be visible as long as the field order is right.

Play it on a progressive scan player fed to a progressive display device, and say that either the player or the display has turned on their built-in deinterlacer: No combing artifacts, but certainly a loss of quality. And according to your first post not setting the TFF flag must have triggered the deinterlacer which is not good...

Play it in a progressive chain where neither the player nor the display have turned on their deinterlacer (which would be correct) then you would see combing artifacts.


So please provide a sample...

Cheers
manolito

[Koutsoubos]

Hey, it's not that I do not want to heed the advice of the likes of neuron2 or you. I see that both of you have a very rich bank of experience behind you.

Just that the source is very big (even a 4 second Lagarith-compressed snippet is 80 MB) and copyrighted. I am not authorized to release the material.

And it cannot be a field shift. For starters, there are no fields. The source is progressive.

In my "case 1 encoding" (HCEnc's new behaviour), in two progressive test setups (progressive player and display), when I actually force the DVD standalones to output progressive, things are fine. When I leave them in "standard" interlaced mode, perceptible combing appears, shall we say they do not seem smart enough to produce a kosher 50fps interlaced image from the 25p encoding. Commercial interlaced DVDs play fine.

In a plain-vanilla non-progressive scan DVD player and interlaced CRT, the decoder-interlaced image appears just fine.


So... that led to the long discussion about flags and all and whether "case 3" is the way to go ...


I would like to pose a question to all. Given a nominally 25 fps (i.e. forget about speed-up or not, it's now 25fps) progressive source bound for PAL, would you encode it as 25p or go ahead and encode it as 50i? Why or why not? How would you do it?

[Koutsoubos]

I don't remember which part of the suite were used, but I've received a report from the replicator (made with the EclipseSuite) with a warning regarding the dts audio: "Since dts is not mandatory, some dvd players could be not able to decode it correctly". So I think that at least one tool of the suite have indeed read the IFO/VOB.


This from an individual known as Trai, expressed in the Creative Cow forums (http://forums.creativecow.net/thread/155/869917):

The EclipseSuite of tools is crucial, no doubt; I've been licensed with them since September 2002, and use them everyday. But these tools make sure that what's presented to you as a replicator, i.e., the DVD Image, and that it can be mastered. Only a very few checks of the contents of the build folder (DVD-Video file structure in the VIDEO-TS folder) are included that pertain to mastering the Image; things like proper placement and flagging of layer break cell (which I and my partner working on DVDAfterEdit got Eclipse to implement during 2003/2004), making sure the DSI (data search information) has the correct number of sectors listed as are actually on the disc (more to it than that), and just a very few others...

And yes, it's true I'm afraid, Eclipse, as good as it is, doesn't check for DVD for spec compliance. The MEI DVD-Video Verifier does that. Eclipse (worth it's weight for the crucial tasks it performs) verifies if the Image presented is configured properly for your mastering equipment, and that the Image follows certain file system specifications - but again (and again) does not check the DVD for DVD-Video spec compliance.

[mp3dom]

Given a nominally 25 fps (i.e. forget about speed-up or not, it's now 25fps) progressive source bound for PAL, would you encode it as 25p or go ahead and encode it as 50i? Why or why not? How would you do it?

Personally I would encode it as 25p. Just to be sure, have you tried to encode your footage with same/similar settings on other encoders? Does it changes something?


This from an individual known as Trai, expressed in the Creative Cow forums

From that thread, the answer is not that clear. Other users says that the ImageMapper tool is able to check dvd compliancy. Maybe it depends by the version too. Anyway, if you're so worry, stay on the safe side by using conservative settings.
Regarding your problem about 'combing' artifacts on progressive image, it could be that some flags are switching over time. Even the scanning order (from Alternate to Zig-Zag and viceversa) could led to some strange artefacts on some standalone players.

[manolito]

Maybe to avoid all possible issues with different hardware player setups you could do what virtually all commercial authoring houses do:

Flag progressive PAL DVDs as interlaced TFF

Out of curiosity I just pulled 20 commercial progressive PAL DVDs from my collection and checked their flagging with ReStream.

Result:
None of them had progressive flags whatsoever, all were flagged as interlaced TFF. Nine of them showed zig-zag scan order, eleven had alternate scan order.

This leads to a feature request for the next HC version:
I would like to have an additional setting called *NO_PROGRESSIVE_FLAGS (or similar)
This way I could correctly encode progressive content with zig-zag scan order, but there would still be no progressive flag in the stream. This is currently not possible with HC (CCE SP can do this...) In AutoInterlaced mode this would also avoid the constantly alternating progressive flags which has been reported to cause trouble in some hardware configurations.


Cheers
manolito

[Koutsoubos]

Just to be sure, have you tried to encode your footage with same/similar settings on other encoders?
I have only tried HCEnc.


Other users says that the ImageMapper tool is able to check dvd compliancy. Maybe it depends by the version too.

ImageMapper seems to be a software used to mount DDP images as if they were DVDs: www.eclipsedata.com/PDFs/ImageMapper.pdf



Regarding your problem about 'combing' artifacts on progressive image, it could be that some flags are switching over time. Even the scanning order (from Alternate to Zig-Zag and viceversa) could led to some strange artefacts on some standalone players.

I don't know about switching flags. Would HCEnc do that sort of thing? What could I use to check this? Restream?

[Koutsoubos]

Out of curiosity I just pulled 20 commercial progressive PAL DVDs from my collection and checked their flagging with ReStream.


Out of curiosity, how would you know that the DVDs are "progressive PAL"? I have never seen a PAL DVD labelled as "progressive" or otherwise, in fact I have a few that seem to be encoded interlaced. Did you inspect the stream and the picture structure?



Result:
None of them had progressive flags whatsoever, all were flagged as interlaced TFF. Nine of them showed zig-zag scan order, eleven had alternate scan order.

Do I correctly understand that the streams are flagged as: Progressive_frame=OFF; TFF=ON; Progressive_sequence=OFF ?



This is currently not possible with HC (CCE SP can do this...)

How is this achieved in CCE SP?

[mp3dom]

I have only tried HCEnc.
It could be a good thing to test other encoder if you can.


I don't know about switching flags. Would HCEnc do that sort of thing? What could I use to check this? Restream?
Maybe, other encoders can do that too if you tell to. With ReStream you can check only the first header, not the whole file. If the scanning order switch over time, you'll see only the starting scanning order.


I have never seen a PAL DVD labelled as "progressive" or otherwise, in fact I have a few that seem to be encoded interlaced.
I've seen quite of them but in general they don't have the progressive_sequence set to ON (because in general the encoder used for dvd is CCE which doesn't support the progressive_sequence). They set progressive frame = ON and Top Field First set to ON, with constant ZigZag scanning order. This is probably the most compatible settings since it allows anyway to output progressive in a progressive-scan player.


How is this achieved in CCE SP?

Set ZigZag scanning order, leave unchecked 'progressive frame', set offset line to 0 and check Top Field First

[manolito]

Out of curiosity, how would you know that the DVDs are "progressive PAL"? I have never seen a PAL DVD labelled as "progressive" or otherwise, in fact I have a few that seem to be encoded interlaced. Did you inspect the stream and the picture structure?
To determine if a movie is progressive or interlaced you have to step through single frames (in a scene with horizontal motion) and look for combing. DGIndex or VirtualDub-MPEG2 are perfect for this, do not use a player which automatically deinterlaces. If there is no combing then the film is progressive no matter if it was encoded interlaced.
The probable reason why most commercial progressive PAL DVDs are encoded (or just flagged) as interlaced is that the authoring folks are too lazy or just too ignorant to flip a couple of switches on their hardware encoders.

Do I correctly understand that the streams are flagged as: Progressive_frame=OFF; TFF=ON; Progressive_sequence=OFF ?
Yes!

How is this achieved in CCE SP?
mp3dom already answered this.

They set progressive frame = ON and Top Field First set to ON, with constant ZigZag scanning order.
Unfortunately not. None of the 20 DVDs I tested had progressive frame = ON, and not even half of them had been encoded with ZigZag scanning order.


Cheers
manolito

[Koutsoubos]

Set ZigZag scanning order, leave unchecked 'progressive frame', set offset line to 0 and check Top Field First
In CCE, assuming a 25fps progressive source, is the "Rate Conv" or "Rate Conversion" option selected in the GUI?




The probable reason why most commercial progressive PAL DVDs are encoded (or just flagged) as interlaced is that the authoring folks are too lazy or just too ignorant to flip a couple of switches on their hardware encoders.
How would YOU encode a 25fps progressive source PAL DVD, Manolito? What settings would you use (assuming spec compliance I guess).



I really want to thank you all, guys. You were a revelation.

I'm off to do more tests. I hope this discussion has been fruitful for the HCEnc developers too. It's a great program.

[Lyris]

The probable reason why most commercial progressive PAL DVDs are encoded (or just flagged) as interlaced is that the authoring folks are too lazy or just too ignorant to flip a couple of switches on their hardware encoders.


I also thought that the reason for encoding interlaced was simply to avoid any cases where small Interlaced segments were hiding in progressive content. (Or what you said, worded less strongly :) )

The Cinema Craft line of encoders (at the very least the high-end versions) are designed for DVD compressionists, and the manual tells you to set PAL Progressive as Progressive if the source is such. So I would be surprised if Cinema Craft a) even allowed you to produce non-compliant output when its "DVD" mode is enabled and b) would recommend you to do so in their user manual!

FWIW, this is the UK PAL version of "Howl's Moving Castle" which is one of the only PAL discs I've seen that's flagged Progressive. I believe the same version is on sale in Australia:

http://img96.imageshack.us/img96/2411/howls.jpg

I've only done one PAL DVD (sold in the UK) and encoded it Progressive where applicable. I would hate to encode it Interlaced due to the decline in compression efficiency/quality. Has anyone ever seen any issues with doing this?

[Emulgator]

Some posts back:
http://forum.doom9.org/showthread.php?p=1367663#post1367663

I encode all my PAL-Speedup25p-from-film-24p DVDs as 25p with HC since then
and am happy with the outcome on different setups.

To flag Progressive Sequence as: no
seems to be the vital part to be DVD compliant.

So, DVD FAQ #3.4: http://www.dvddemystified.com/dvdfaq.html#3.4
where it says: "MPEG-2 progressive_sequence is not allowed"

Yes, I would follow this.
Given the fact that PAL and NTSC are field-based
and the main video connection 1995 was the yellow Cinch connector
carrying the composite video signal in interlaced sequence
there was no need to allow an encoding parameter to be set to an untransmittable value.

Otherwise, DVD players should simply skip this value if set to Progressive sequence.
But it made a visible difference on my setup...
All pulldowns, 3:2 (23.976p Film on 29.97i NTSC),
Euro (2:2:2:2:2:2:2:2:2:2:2:2:3 24p Film on 25i PAL) come out a bit borked on this particular combination.

To encode and flag the progressive frame as progressive frame came out beautiful,
player will have to perform 2:2 pulldown to output as fields via Composite anyway.
A progressive-capable player may probably output such encodes as progressive if this is allowed via HDMI.
I can't tell exactly, but reflagging tests showed an advantage in certain player/TFT-combinations.

To encode and flag progressive material as interlaced (no matter if tff=1 or 0)
came out less optimal encodingwise.
On playback player will follow tff/rff flags anyway.

[hank315]

Encoding settings suggestion:
- progressive sequence: off
- progressive frame: off
- TFF: set
- alternate scan: off
- MacroBlock dct_type: always frame

This way the encode is pure progressive and the player will see it as TFF interlaced (2:2 pulldown).

But AFAIK there's no encoder that can do this.

[manolito]

But AFAIK there's no encoder that can do this.
Could you implement it into a test version of HC? :devil:

Cheers
manolito

[Koutsoubos]

The Cinema Craft line of encoders (at the very least the high-end versions) are designed for DVD compressionists, and the manual tells you to set PAL Progressive as Progressive if the source is such.
Where is exactly stated in the manual? I don't remember reading anything like that. (Could be a case of senility.)

What program did you use to check out "Howl's Flying Castle"? What are the flags? If Prog_Frame=ON; TFF=off; Prog_Seq=ON that would be good news!

Any artifacting on playback?
(Nevermind this question. I came across this: "What's more annoying still, is that only ONE of the PAL discs I tried - the UK release of "Howl's Moving Castle" - was free of combing artefacts on movement (particularly noticeable on camera pans)." http://www.lyris-lite.net/dv490v_review3.html)



Euro (2:2:2:2:2:2:2:2:2:2:2:2:3 24p Film on 25i PAL) come out a bit borked on this particular combination.
I think there's one key difference! In the scenario I am postulating the source is PURELY 25FPS PROGRESSIVE (either sped up or not, film source or not, it's 25fps nominally) and so there is no need for the "Euro pulldown".

Given that situation, would that change your opinion? Would your setups show any artifacting? Or "borking"? ;)

In a nutshell, I am seeing artifacting wih progressive setups (without forcing progressive output from progressive scan DVD) when TFF=on; prog_seq=off, and no artifacting when TFF=off; prog_seq=ON.


Encoding settings suggestion:
- progressive sequence: off
- progressive frame: off
- TFF: set
- alternate scan: off
- MacroBlock dct_type: always frame

But AFAIK there's no encoder that can do this.
Umm ... any difference between this suggestion and the CCE settings proposed by mp3dom (http://85.230.118.163/showthread.php?p=1409231#post1409231) If even CCE cannot do this, then ... we have a scoop with HCEnc?

[mp3dom]

CCE can do this. In SP/SP2 you can't decide the macroblock dct type (which is fixed to "always frame") and progressive sequence (fixed to "off") but all the other settings could be changed. So if you uncheck 'Progressive Frame' (progressive frame off), check "Top Field First" (TFF set), Offset to 0 (keep the TFF) and select ZigZag Scanning Order (Alternate scan off) you end with the same settings suggested by hank315.

Where is exactly stated in the manual? I don't remember reading anything like that. (Could be a case of senility.)
He's speaking (I think) of CCE-SP3, or CC-X, not SP2 or SP. SP3 and X are the highest level/quality of CCE and allows more quality and also a lot of more other options too. The SP3 manual says: "If your PAL source is 2:2 pulldowned (conversion from film to PAL video), Progressive will be appropriate." However from the manual it isn't clear if that option will set the progressive sequence to ON or (as SP/SP2 already does) set only the progressive frame.

[Lyris]

Where is exactly stated in the manual? I don't remember reading anything like that. (Could be a case of senility.)

What program did you use to check out "Howl's Flying Castle"? What are the flags? If Prog_Frame=ON; TFF=off; Prog_Seq=ON that would be good news!

Any artifacting on playback?
(Nevermind this question. I came across this: "What's more annoying still, is that only ONE of the PAL discs I tried - the UK release of "Howl's Moving Castle" - was free of combing artefacts on movement (particularly noticeable on camera pans)." http://www.lyris-lite.net/dv490v_review3.html)
Yes, that's my site - an old review I might add :D
I used DGIndex. What other program should I check with?

Regarding CCE manual - this is from SP3. It recommends setting Progressive instead of Interlaced in the "Picture" menu, which codes Frames instead of Fields.

If your PAL source is 2:2 pulldowned (conversion from film to
PAL video), Progressive will be appropriate.

And from SP2's manual:
6.3.11 Progressive frame
Select when the footage is progressive. This setting works on MPEG-
2 output. If you apply Inverse 3:2 pulldown, this setting does not
work.

If your source is 2:2 pulldowned (film PAL), it may bring a
better result.

If you want me to make some test files with SP2 and/or SP3 for analysis, let me know.

[hank315]

CCE can do this. In SP/SP2 you can't decide the macroblock dct type (which is fixed to "always frame") and progressive sequence (fixed to "off") but all the other settings could be changed. So if you uncheck 'Progressive Frame' (progressive frame off), check "Top Field First" (TFF set), Offset to 0 (keep the TFF) and select ZigZag Scanning Order (Alternate scan off) you end with the same settings suggested by hank315.
The MB dct_type has a large impact on the encoding efficiency, on a progressive source it should be set to "always frame", on an interlaced source the encoder has to decide which method to use (frame/field) based on the interlaced motion of the MB.
Do you really mean in CCE dct_type is "always frame" even when progressive frame is unchecked ?

Some very nice (free) MPEG apps which can analyse a stream onto the MacroBlock level can be found here: http://tsviatko.jongov.com/

[Koutsoubos]

FWIW, this is the UK PAL version of "Howl's Moving Castle" which is one of the only PAL discs I've seen that's flagged Progressive.

What are the flags of the MPEG2 stream? Is it:
1. Progressive_frame=ON
2. TFF=OFF
3. Progressive_sequence=ON ?


If that's good enough for Optimum Releasing, that's good enough for me!


What were the flags for the UK DVD that you encoded?

[Boulder]

Is it possible to have AutoGOP for shorter GOPs than 12 frames? I have to use a maximum of 10 frames per GOP for my Super-8 material which I'll pulldown from 18 to 25fps. Without AutoGOP, most scene changes are missed.

[Lyris]

What are the flags of the MPEG2 stream? Is it:
1. Progressive_frame=ON
2. TFF=OFF
3. Progressive_sequence=ON ?


If that's good enough for Optimum Releasing, that's good enough for me!


What were the flags for the UK DVD that you encoded?

The UK DVD I encoded was done with CCE SP2. I checked "Progressive" in the picture window for the progressive parts. One of the music videos switched between I and P so I manually told it to encode the Progressive parts as Progressive frames for better compression.

Howl's Moving Castle shows the following in Restream:

Frametype Progressive: CHECKED
Top field first: *not* checked
Progressive Sequence: CHECKED

Interestingly, the "User data" section shows this for "Char string":
"| HC 0.16A beta - (c)2004/2005 |"

Really? It was encoded with HCenc? In case you're wondering, yes, this is a real store-bought copy, not a "backup".

The disc I did shows the following:

Frametype Progressive: CHECKED
Top field first: CHECKED (I left this unchanged from the CCE default)
Progressive Sequence: NOT checked, because CCE does not write Sequence Display Extension. Too bad, if it did, even cheap Progressive PAL DVD players would be able to play back without jaggies.

[Mug Funky]

@Lyris:

it's actually a mix of different encoders... bits that one encoder handled better than another were used where appropriate. good old segment based re-encoding. the feature was a patchwork of re-encoded segments by the end of it.

good to see it got a good review...

[Lyris]

I'm impressed! Were you involved with that title? How was the "stitching" done - combining the separate .m2v files in the authoring program?