What Is the Most Rancid PGS Stream Out There?

[wswartzendruber]

Greetings everyone,

I have a project I have been working on and it is quickly reaching its conclusion. So now come the sea trials.

What is the most rancid PGS stream out there? I mean something commonly available that is expected to decode and render correctly.

IIRC, Avatar 3D has particularly difficult subtitles.

Any input is appreciated. :)

[cubicibo]

These PGS threads are just personalised bait.

These two will fail on FFmpeg-based decoders except HandBrake 1.9.0+:
• All features (decoupled): File (m2ts) (https://streams.videolan.org/ffmpeg/incoming/00019.m2ts) - Explanation (https://streams.videolan.org/ffmpeg/incoming/00019.m2ts.txt) (Courtesy of Masstock)
• Graphic plane test: File (SUP) (https://streams.videolan.org/ffmpeg/incoming/graphicplane_test.sup) - Explanation (https://streams.videolan.org/ffmpeg/incoming/graphicplane_test.sup.txt)
Cropping can only be tested with handcrafted samples. The feature works on all Blu-ray players regardless (thanks to the numerous testers).

You can also create your own problems with SUPer.

[wswartzendruber]

These PGS threads are just personalised bait.
I'm not sure if you're accusing me of something, but I thank you for your help regardless.

These two will fail on FFmpeg-based decoders except HandBrake 1.9.0+:
• All features (decoupled): File (m2ts) (https://streams.videolan.org/ffmpeg/incoming/00019.m2ts) - Explanation (https://streams.videolan.org/ffmpeg/incoming/00019.m2ts.txt) (Courtesy of Masstock)
• Graphic plane test: File (SUP) (https://streams.videolan.org/ffmpeg/incoming/graphicplane_test.sup) - Explanation (https://streams.videolan.org/ffmpeg/incoming/graphicplane_test.sup.txt)
I'll run these through PGS4NET's caption compositor and see what it gives me...

Cropping can only be tested with handcrafted samples. The feature works on all Blu-ray players regardless (thanks to the numerous testers).
Yeah, all of the crop testing my project has are from entirely synthetic vectors. I don't have anything I've ripped from my collection that uses this feature.

EDIT: Oh weird. For the graphic plane test, PGS4NET will see a missing WDS and then rip the drawn image down.

EDIT: Fixing this in my implementation is going to suck...

[wswartzendruber]

These PGS threads are just personalised bait.
Oh, sorry! I wasn't understanding this earlier this morning before drinking coffee...

Yes, these threads are meant to get your attention, but not exclusively. I welcome anyone who knows about PGS.

With that said, the behavior described by "graphics plane test" almost seems really odd.

What's the URL for the VLC Trac issue? I'm not finding it.

[cubicibo]

It was more of a joke than anything.

With that said, the behavior described by "graphics plane test" almost seems really odd.
I should make an addendum to this answer (https://forum.doom9.org/showthread.php?p=1998877#post1998877):
"The graphics plane is [...] where the uncompressed ODS are copied, as instructed by the composition objects whenever a WDS is provided."
Any change to the composition objects can only be reflected to the viewer via a WDS, as I had discovered in that thread (https://forum.doom9.org/showthread.php?p=1984822#post1984822). The content of the graphic plane shall be left untouched from the previous composition if no WDS is provided, and the PCS is not starting a new epoch.

[wswartzendruber]

Does the absence of a previously-defined composition object mark the end of an object's time on the screen?

[cubicibo]

All windows are redrawn, per the composition objects instructions, when you provide a WDS. If an object is no longer referenced in that display set, it will not be reproduced onto the graphics plane.

In all subsequent Normal Case DSs, and as long the buffer slot is not overwritten, the object remains in the decoded object buffer and may be re-used later if a composition object references it again.

[Otyli]

Another contender is Disney's 3D Blu-ray releases, such as the Jungle Book or Beauty and the Beast, which can have complex subtitle placements and depth issues that can test your decoding and rendering capabilities.

[wswartzendruber]

All windows are redrawn, per the composition objects instructions, when you provide a WDS. If an object is no longer referenced in that display set, it will not be reproduced onto the graphics plane.

In all subsequent Normal Case DSs, and as long the buffer slot is not overwritten, the object remains in the decoded object buffer and may be re-used later if a composition object references it again.

I'm noticing that the third display set in the sequence has palette_update_flag = 0x80. Is it fair to say that this is what triggers a graphics plane update when no windows are present? It seems like this flag causes an entirely separate rendering process to follow.

Another contender is Disney's 3D Blu-ray releases, such as the Jungle Book or Beauty and the Beast, which can have complex subtitle placements and depth issues that can test your decoding and rendering capabilities.
Why thank you. I've found a rather cheap copy of Jungle Book on eBay.

[cubicibo]

palette_update_flag instructs the decoder to set the palette indicated by the PCS palette_id in the CLUT at the PCS PTS. The palette_update_flag skips the graphic plane drawing process triggered by a Window Definition Segment. Again, the decoder model applies.

[wswartzendruber]

palette_update_flag instructs the decoder to set the palette indicated by the PCS palette_id in the CLUT at the PCS PTS. The palette_update_flag skips the graphic plane drawing process triggered by a Window Definition Segment. Again, the decoder model applies.
Is that the decoder model from the patent? I've learned to be skeptical of what it says.

[cubicibo]

Yes. Your implementation requires a buffer for the graphic plane anyway, like it or not.

[wswartzendruber]

And I need to rewrite my graphics plane. Do any PGS streams in the wild use moving objects or wipe effect?

EDIT: If I'm understanding the decoding model correctly, the entire screen can only have one active palette (in the CLUT) at any given time.

EDIT: And what is the present consensus on windows changing mid-epoch?

[cubicibo]

1. Maybe. Assume that there is at least one.
2. Yes, as we had discussed.
3. Strictly forbidden.

[wswartzendruber]

What do you think happens if PaletteUpdateOnly is passed without composition objects? I find it curious that composition objects are present in the PCS but window definitions aren't, in this case.

[cubicibo]

This is implementation dependant. Quite a few Blu-ray players (= reference decoders) ignore the composition objects if palette_update_flag is set. Garbage or none, the player just applies the palette update to the current composition. Authoring tools become confused if the list of compositions changes, so the specs likely claim the compositions should remain identical.

[wswartzendruber]

This is implementation dependant. Quite a few Blu-ray players (= reference decoders) ignore the composition objects if palette_update_flag is set. Garbage or none, the player just applies the palette update to the current composition. Authoring tools become confused if the list of compositions changes, so the specs likely claim the compositions should remain identical.
Whoever came up with this scheme is also whoever is in charge of the Ministry of Silly Walks.

[cubicibo]

I think you really miss the point of the entire format. Maybe if you work with SMPTE-TT and all of these other css+xml insanities you will get to appreciate these clever tricks that makes PGS the most powerful subtitle format the industry (pros, not fansubs) ever came up with to this day. No kidding.

[wswartzendruber]

EDIT: Nevermind.

[wswartzendruber]

I stepped away for a bit, but now I'm back at this.

Is the seemingly nonsensical idea of objects being placed outside their window as nonsense as it would seem?

[cubicibo]

The composition controller can only draw the area defined in a window. If a composition does not intersect with its window, it will not be rendered. However, it is undefined behaviour for an object to partially intersect its referenced window. The composition controller would have to initiate a copy from the object buffer with an offset that is not conveyed in the composition (cropping) parameters. It would also have to terminate the copy of the said object to an unexplicited width and height that may differ from the object dimensions.

I would fiercely forbid it.

[wswartzendruber]

The composition controller can only draw the area defined in a window. If a composition does not intersect with its window, it will not be rendered. However, it is undefined behaviour for an object to partially intersect its referenced window. The composition controller would have to initiate a copy from the object buffer with an offset that is not conveyed in the composition (cropping) parameters. It would also have to terminate the copy of the said object to an unexplicited width and height that may differ from the object dimensions.

I would fiercely forbid it.
Do you think it would be prudent to still render any object data still in the window area (decode) while forcibly prohibiting such streams from ever being generated (encode)?

Where I'm trying to go with this is that the library should parse malformed streams as best it can while always creating valid ones.

[cubicibo]

I would not render the composition if any coordinate after cropping lies outside of the referenced window. IMO a decoder should never go out of its way to support non-compliant files.

[wswartzendruber]

Upon further consideration, my original idea of making the decoder as fault-tolerant as possible could serve to encourage mediocrity within the PGS ecosystem.

"That one C# library that one dude made can be used to clean up my output."

So it seems beneficial to the ecosystem if the PGS4NET decoder is also retentively up-tight about correctness. I assume all of the following are nonsensical:

1. Segments with inconsistent PTS and DTS stamps within a single display set.
2. A subsequent display set having a time stamp before the preceding one.
3. A crop area being outside the bounds of an associated object.
4. An object being outside the bounds of a window.
5. Two objects overlapping within a window.
6. A window being outside the bounds of the screen (DisplaySet.Width and DisplaySet.Height).
7. Overlapping windows.

EDIT: Nevermind. Your is_compliant function is answering much of this.

[wswartzendruber]

Actually, I do have a question:

# The Coded Object Buffer can hold up to 1 MiB of raw PES data

Does this mean all ODS segment chains for all objects must not exceed one mebibyte on legacy (2006-era) Blu-ray Discs?

[cubicibo]

The 1 MiB buffer is the usual elementary stream buffer found at the input of any decoder. Its purpose is to collect and store temporarily incoming packets. Packets are removed by the decoder on their DTS. The buffer fullness depends on the muxer packet scheduling decision.

That's correct. A given object cannot exceed 1 MiB in the elementary stream. That also means a muxer should not buffer excessively segments ahead of their decoding time, as that may also overflow the buffer. In any case, this is an encoder and muxer concerns, not one for a decoder. The is_compliant function in SUPer checks encoding constraint to ensure consistent presentation across all vendors implementation. A decoder should only implement basic sanity checks like maximum object dimensions of 4096 pixels, object resides in window that are on screen, object exists in buffer, etc.

Anyway don't overly restrict your decoder. FFmpeg one blends two overlapping object, it does not implement the concept of a Window either. HandBrake patches it and make it closer to a proper decoder without adding excessive constraints.

[wswartzendruber]

Does palette entry 255 implicitly mean "no pixel"?

[cubicibo]

A transparent pixel isn't void.

[wswartzendruber]

What does it mean when PTS stamps vary within a display set?

[cubicibo]

Read the patent. Each segment affects the decoder differently and take effect at its own PTS. In a Display Set, data segments must be fully decoded to be used: at the given PTS they are guaranteed to be available.

[wswartzendruber]

Is there a concise way to convey which parts of the patent are reliable and which parts aren't?

[wswartzendruber]

Read the patent. Each segment affects the decoder differently and take effect at its own PTS. In a Display Set, data segments must be fully decoded to be used: at the given PTS they are guaranteed to be available.
For the graphic plane test that you attached, the second epoch's PTS stamps are strange to me:

PCS: 00:00:10.010
WDS: 00:00:09.971
PDS: 00:00:09.893
ODS: 00:00:09.971
ODS: 00:00:09.971
ES: 00:00:09.971

All of these ought to coalesce into a single-frame update...

The patent seems to indicated that DTS (decode time stamp?) is when we start decoding the object and PTS (presentation time stamp?) is when it goes on-screen. The difference between the two is decode duration.

It looks like multiple frame updates (at different times) can happen because of a single display set?

[cubicibo]

The various segments of a Dispaly Set should have different PTS. SUP files where all PTS within a DS are equal are incorrect and generally mkv extracts.

The process is straightforward:
- The PCS and subsequent segments enters the decoder at their specified DTS.
- Segments are decoded starting from their DTS (equal or larger than the PCS DTS) and become available (= usable) by the decoder at their own PTS.
- The on-screen display update is performed at the PCS PTS, once any subtask initiated by a segment in the DS has been completed.
- Only the PCS PTS is observable by the viewer. All other PTSes are internal to the decoder.

[wswartzendruber]

Okay so the patent is correct in saying that DTS + DECODE_DURATION = PTS.

Can I trust the patent for how to calculate DECODE_DURATION?

[cubicibo]

Why care about the decode duration if you're writing a software decoder? Only potato decoders running on embedded devices and SoCs need the DTS, and it is specified in the bitstream anyway. Only an encoder shall compute the decode_duration. And yes, albeit the patent makes it more convoluted than necessary (https://github.com/cubicibo/SUPer/blob/f97fd23106c64f7f7ec335715d41ffe1c618d68a/SUPer/render2.py#L1491) and, again, it's not helping a SW decoder.

[wswartzendruber]

I am writing a software encoder as well and am trying very hard to do this correctly. This is why I am so particular.