SUPer - BDNXML to BD SUP subtitle converter that supports animations

[cubicibo]

SUPer is a Python tool to convert BDNXML to Blu-ray SUP ("HDMV PGS") subtitles. SUPer is designed to make full use of the PGS capabilities and to produce compliant bytestreams that drive correctly the Blu-ray PGS decoder. SUPer notably has the capability to encode animations within a bitmap to use efficiently palette updates to make animations like colour effects, karaoke, fades and wipes. Finally, unlike Scenarist or avs2bdnxml, SUPer is not subject to buffer overflows as it drives the hardware PG decoder as intended by the designers.

Features
Users that only want to convert BDNXML can simply use the stand-alone app. The app lets you convert PNG+BDNXML assets to a SUP file or PES+MUI assets for Scenarist BD. An origin SUP file may also be provided optionally to perform merging to a new file.

For developers, SUPer also acts as a Python library to parse SUP files or generate them from code. It offers many module to perform image analysis, processing and PG bytestream handling.

Example
This example showcase the capabilities of SUPer. This file function on Blu-Ray players without any flickering or adverse effect. You can enjoy the full experience by playing it back with Nichijō second opening.

Pyton package with GUI app
https://github.com/cubicibo/SUPer

Download stand-alone executable (macOS aarch64, win x64)
https://github.com/cubicibo/SUPer/releases

For users that uses the Python3 package (Python3.9+), you need to install it to your virtual environment before running supergui.py or import the package.

Create a Python3 virtual environment, cd in the SUPer base folder then do:

python3 -m pip install .

Then, to execute the GUI to convert BDNXML to SUP:

python3 supergui.py

and everything will work nicely

[cubicibo]

SUPer version 0.0.1a
Changelog

I fixed some bugs in the code, you can now export very large objects. I had made an example that defines a total of 10 ODS segments in the first two displayset of the epoch (for two full screen 1920x1080 overlays).

[cubicibo]

SUPer 0.1.0 is out!
There is now a user-friendly interface (a Python GUI) akin to PunkGraphicStream.

Changelog

SUPer version 0.1.0.
Front-end
• Graphical interface to generate a .SUP file from a BDNXML with the possibility to inject data in an existing SUP.
• Changed some of the API for rendering and optimising
• Simplified the example file.

SUPer back-end
• SUPer now supports multi-object rendering!
• Added a graphics grouping engine, that associate objects and minimizes windows area footprint before rendering.
• Added a pre-rendering module to avoid image shape mismatch issues (a pain in 0.0.1)
• Bugfixed rendering engine (bad color merging, now disabled).
• Filestreams had a vast overhaul to properly parse BDNXML.
• Revisited the segments module.
• Use precise FPS values
• Added a PGS stream compliancy validator using all possible bitrate and buffer limits described in the original patent.

[cubicibo]

SUPer 0.1.1 is out and it is all about bugfixes.

Changelog

SUPer version 0.1.1 (back-end changes only)
• Attempt to split single composition object in two to improve animation quality when two animations partially overlap in time.
• Fixed palette bounding when composition object #0 was appearing on screen (displayed bad colors for a single displayset)
• Fixed value overflows for ODS.
• Fixed bad palette ID/version number values.
• Fixed memory leak.
• Fixed an offset of one pixel (?)
• Added one more sanity check in SUP compliancy function.

Next step is to replace the default image quantizer which is rather limited Pillow by something else and multiprocessing for those who can't wait.

[cubicibo]

SUPer 0.1.2

GUI front-end
• Anchor the buttons in the GUI.
• Add a check box for content that requires fine grouping (must be used for SD content).
• Add a check box to change the grouping decision when just two objects are overlapping (can fix or break some animations).
• Add a check box to correct drop-frame time drift (divide timestamps by 1.001).
• Add a check box for subsampled BDNXML (e.g 30 fps BDNXML for 60 fps SUP).
• Add a check box to adaptively switch between PIL and K-Means image quantize methods.

Back-end
• Add graphic plane bandwith/refresh compliancy test to detect flickering subtitles.
• Extend DisplaySet and Epoch classes, fix a bug in TimeConv.
• Fix a crash with single composition objects.
• Fix a temporal object-window matching issue (Hopefully? Can be fixed by ticking grouping decision checkbox)
• Fix composition object instantiation with cropping flag.

Notes:
• K-Means quantization is useful to work around a bug in PIL library. PIL is unable to quantize images with high transparency and can change the background to be full black. This breaks some fades.
• K-Means quantization is noisy, unlike PIL. If you have karaoke aside of it, it can (but not necessarily) damage the non-fade animations.

_________________
Some documentations for the few curious readers:

What's the max refresh rate?
- When the screen is re-drawn within an epoch (acquisition or a normal case), objects are copied to the hidden graphic plane.
- On the presentation timestamp, the playback device swaps the two graphic planes (visible<->hidden).
- If the update rate is too high, the graphic planes are not hidden sufficiently long to copy the data.
- This leads to flickering.

The allocated area for a frequently refreshed graphic must follow this rule:

window_area < (k/fps)*16*(1024^2)

With:
- k: the update rate (1 if updated every frame, 2 if every other, and so on)
- fps: the video framerate
- window_area: area of the window on the graphic plane (object area is always equal or smaller to the window area).

For Full HD content (1920x1080), the area in which we can update the rendering once per frame must not be more than 25% of the screen at 29.97 fps, or 31% at 23.976. For high framerate content (50i, 59.94i), the usable area decreases considerably. If the objects are additionnaly refreshed, the decoding time adds up to the rest.
To work around this, SUPer lets you flag subsampled BDNXML (BDNXML generated at @25-29.97 fps, 1/2 of the true framerate). SUPer will flag the output .SUP as twice the framerate.

________
Example file, generated with the latest version of SUPer and tested on a commercial Blu-Ray playback device. Karaoke courtesy of kara.moe.

[tormento]

When installing, I get the following error:

      copying skimage\io\_plugins\_histograms.pyx -> build\lib.win-amd64-3.11\skimage\io\_plugins
      running build_ext
      INFO: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
      INFO: customize MSVCCompiler
      INFO: customize MSVCCompiler using ConditionalOpenMP
      INFO: CCompilerOpt.cc_test_flags[1073] : testing flags (/O2)
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      INFO: CCompilerOpt.cache_flush[857] : write cache to path -> C:\Users\AlbertoGennari\AppData\Local\Temp\pip-install-1vmru19u\scikit-image_965be48bfc2f42c0badcd913f2431977\build\temp.win-amd64-3.11\Release\ccompiler_opt_cache_ext.py
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for scikit-image
Failed to build scikit-image
ERROR: Could not build wheels for scikit-image, which is required to install pyproject.toml-based projects

I won't install VisualC++ for sure. Any workaround?

[cubicibo]

If you're on Windows, just get WSL. It will endlessly simplify your life.

For what you're trying to do, you can remove the scikit/sklearn entries in requirements.txt. As long as you then don't use the render module, you will be fine.

[tormento]

Unfortunately I need the render module.

[cubicibo]

The script I wrote for you (updated) to split the PGS in two does not use the "render.py" module at all. I know the name may sound confusing but render.py is used to render a chain of BDN.XML events to a single SUP epoch and that is not what the script does. It only decodes/re-encodes RLE bitmaps and generates new PGS segments. None of this is "rendering".

If you need to use SUPer for other purpose, like exporting a BDNXML to SUP, you will need to install VC++ or WSL. I am not dropping skimage module as it provided an essential backbone to the grouping algorithm.

[tormento]

Originally Posted by cubicibo

For what you're trying to do, you can remove the scikit/sklearn entries in requirements.txt.

I usually grab packages by PIP. How to tell the installation process to use a requirements.txt of mine?

[cubicibo]

Download the package from the release page (or clone the repository itself). Unzip it, edit requirements.txt to your need and execute:

python3 -m pip install C:/path/to/SUPer

Where SUPer is the folder that contains setup.py.

If the command does not work for some reason, change directory to the SUPer folder then run:

python3 -m pip install -e .

And that will install the package in dev mode, it always works.

[cubicibo]

SUPer 0.1.3

User interface
• Add colorspace dropdown menu (select BT.601 or BT.709).
• Updated the log message format.

Back-end
• Fix some grouping issues related to temporal overlapping with single objects within an epoch.
• Add rendering "interface" class for scripting.
• Add ODS visualisation utility to analyze an encoded animation within the bitmap.
• Compute adaptively the maximum delay required for a new epoch with respect to BDN dimensions (for coarse epoch definition)

[cubicibo]

SUPer v0.1.5

User interface
• None, it is all under the hood.

Back-end
• Implement a brand new grouping engine to define windows.
• Implement a brand new conversion pipeline using some image processing to define objects and find refreshs.
• Optimise the code and algorithms, add numba support. Up to 10x faster on some costly functions.
• Add code to perform palette updates with two objects.
• Update TimeConv internals to have accurate SMPTE timings.
• Drop some dead code.
• Fix many small issues

Here's an example SUP file made with the new engine and tested on a BD play-back device.

[cubicibo]

SUPer v0.1.6

SUPer is now distributed as stand-alone binaries too! Only an Apple M1 (aarch64) build is available presently, Windows may follow later.

Front-end
• Add option to use K-Means quantization on all frames (guarantees quality but slow)
• Add option to fix a possible time drift with NTSC framerates (23.976, 29.97)
• Add compatibility mode option for software players that do not decode palette update display sets correctly.
• Add a progress bar while generating an epoch bytestream.

Back-end
• Revisit again the grouping engine to make it faster and more robust.
• Fix a memory leak in the conversion pipeline.
• Some clean-ups.

[cubicibo]

SUPer v0.1.7

Windows-x64 and macos-aarch64 stand-alone apps are provided.

Front-end
• Add Scenarist BD output format PES+MUI (Output dialog box -> select .PES format at the bottom)
• Add option to enforce DTS timestamps in stream. This should NOT be used unless your muxing software is terrible.
• Add more tooltips on the checkboxes

Back-end
• Fix timecode conversions and rounding errors, SUPer should now be frame accurate.
• Use adaptive structual similarity threshold to account for possible large transparent areas.
• Fix a crash due to blinking bitmaps.
• Filter out empty bitmaps generated for blinking subtitles - saves bandwidth.
• Avoid acquisitions to undisplay one object when two are on screen - saves bandwidth.

Known issue
• A runtime warning is sometime displayed by numpy about empty slice. This is not a bug and has no effect whatsoever on the output.
• Windows binary may show a pyopencl warning when the app or optimiser process start. This can be ignored.

Here's an example with .ASS animations generated with v0.1.7 and tested on a BD player.

[cubicibo]

SUPer v0.1.8

Gigantic update. SUPer now generates compliant PG stream by default, including exact DTS values to ensure compatibility with old PG decoders (e.g. PS3).
The update notably adds double buffering, ahead of time displayset decoding and normal case redefinition for smooth animations, even with complex and large objects.

Credits to Masstock for the many tips, testing and feedback that made this update as complete as it is!

Front-end
• Add a command line client (guizero can be removed from requirements.txt if you prefer to use the cli over the gui)
• Add an option to further tune acquisition rate (for compression when animations are used).
• Update existing tooltips and add more.
• Compatibility mode tickbox now disables cropping.
• DTS checkbox is automatically ticked and cannot be unticked for PES+MUI output.
• Auto-add SUP extension on GUI if none provided.
• Fix a rare bug where the GUI kept using the first selected BDNXML file.

Back-end:
Rendering:
• Optimise composition objects together to better distribute the palette entries.
• Use palette update to undisplay graphics in the middle of a palette animation.
• Implement double buffering to decouple decoding and rendering to let hardware decoders perform multitasking! (*)
• Implement NORMAL CASE object redefinition, used if worthwile.
• Implement palette updates buffering to perform screen updates while long object decoding is in progress.
• Implement events dropping to ensure necessary object definitions are feasible and stream is strictly compliant. (**)
• Reduce memory usage and fix numerous small issues.
PG bytestream:
• Re-implement PTS and DTS computation to be identical to Scenarist BD.
• Fix a bug that caused the penultimate display set of an epoch to refresh the object.
• Prevent the usage of the palette entry ID 255, apparently it should not be used.
• Cycle through palettes ID when palette version overflows to increase compatibility.
• Remove the work-around for palette updates with two objects.
• Add auto-cropping of objects to minimize transfer time on hardware decoders.
Parsing:
• Escape BDNXML files to avoid a crash on badly encoded data.
• Filter out zero duration events in BDNXML (events where InTC == OutTC).

Notes:
(*) Hardware decoders are designed to perform decoding and rendering at the same time. It is possible to continuously decode incoming display sets as long as the decoder does not write to the memory region that the renderer is reading! This is alleviated by performing double buffering and using the numerous palettes.
(**) the dropped event is generally one or two frames long at 23.976-29.97 fps and should go unnoticed.

Windows x86_64 and macOS aarch64 binaries are available on the release page. Reminder that SUPer works best with ass2bdnxml.

[cubicibo]

SUPer v0.1.9

This version addresses the issues faced by files generated by SUPer imported in Scenarist BD. Many thanks to Masstock further testing.

Front-end
• Add a checkbox to force SUPer to always write the entire palette rather than just the difference. Enforced for PES output.
• Add a stricter mode to further constrain the PG stream and ensure it passes all Scenarist BD checks without warnings or undesirable modifications. Enforced for PES output.
• Add a checkbox to output PES and SUP together. Handy for validation of the output on a video player of the authored file.

Back-end
• Fix a grouping engine regression. It could get stuck in a loop.
• Fix timestamps, PTS and DTS should be always identical to Scenarist BD now.
• Filter out empty palette updates.
• Fix a missing palette assignment on screen wipes preceeding an acquisition point.

Dependencies
• Updated to ScenariStream v0.0.4 to have accurate negative PTS and DTS in PES+MUI.

Here are few samples PES+MUI and SUP generated with SUPer so you can play around with software players and Scenarist BD. And here's one of such SUP imported in Scenarist BD. Pay attention to buffer usage and number of display sets (one epoch in this file has more than 1400)

[cubicibo]

SUPer v0.2.1

I exhausted all compliancy rules and checks known by us mere mortals. I now expect the output of SUPer to pass the BDA verification tests.

Front-end
• Add conservative PTS/DTS strategy checkbox/cli flag.
• Parse config.ini in command line client.
• Remove the event overlap checkbox ("additional Scenarist BD checks") as it is always enabled internally.

Back-end
• Rewrite and optimise RLE encoders and decoders.
• Rewrite events filter. Watch for the minimum delta between two PTS when filtering.
• Revisit PTS and DTS calculations, fix wrong WDS PTS, fix wrong DS decode duration with no ODS.
• Fix normal case object redefinition, it was prone to crashing.
• Estimate the number of colours necessary with PIL when encoding an animation via PILIQ.
• Prevent usage of colour 0x00 to avoid illegal RLE line lengths above bitmap width + 16.
• Add PTS/DTS test at the end to catch errors.
• Terminate all epochs with a palette update to clear the screen rather than a drawing operation.
• Additonally wipe the screen if there's enough time between two epochs.

[Emulgator]

Wow, big thanks cubicibo !

[cubicibo]

SUPer v0.2.2

Numerous small changes that improves greatly the conversion process and the quality of the output. There should be fewer dropped events starting v0.2.2.

Front-end
• Add option to perform acquisitions after N palette updates. This ensures the quality of stills that directly follow palette animations.
• Add a target bitrate test at stream output to help authorers anticipate excessive bitrate before a mux. (v0.2.1b)
• Add optional logging to file. (v0.2.1b)

Back-end
• Optimise buffer slots dimensions to the maximum area shown within each window. Fairly isolated moves will now be much more consistent.
• Replace windows definition algorithm with a new one that finds more optimised layouts.
• Perform BDN XML preprocessing to remove consecutive events with identical PNGs, rewrite event timeline on the fly accordingly.
• Allocate all transparent pixels to palette ID 0xFF, one palette ID saved in the general case, two in normal case object redefinitions.
• Append an acquisition after N palette updates to wipe any potential artifact due to animation encoding (option enabled by user)
• Avoid encoding dropped events to improve quality of preceding events in palette effect chains.
• Fix screen wipes in the middle of epochs that could have incorrect PTS/DTS.
• Add spatial consistency checks for windows and compositions.