sub3d: plugin for rendering subtitles on 3D video with correct depth

[slavanap]

Render your 2D subtitles on 3D video with correct depth.

>>> CURRENT RELEASE AVAILABLE HERE <<< (https://github.com/slavanap/ssifSource/releases)

CASES THAT PLUGIN SUPPORTS

1. You have subtitles in xml format.
I know that BD3D2MK3D (http://forum.doom9.org/showthread.php?t=170828) tool somehow generates this format.
The output should be a folder with .xml file (descriptor) + .png images (subtitles).
.xml file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<BDN Version="0.93" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="BD-03-006-0093b BDN File Format.xsd">
<Description>
<Name Title="FRA_2D" Content=""/>
<Language Code="eng"/>
<Format VideoFormat="1080p" FrameRate="23.976" DropFrame="False"/>
<Events Type="Graphic" FirstEventInTC="00:00:02:17" LastEventOutTC="00:04:33:18" NumberofEvents="47"/>
</Description>
<Events>
<Event InTC="00:00:02:17" OutTC="00:00:05:07" Forced="False">
<Graphic Width="840" Height="50" X="540" Y="875" Depth="24">FRA_2D_0001.png</Graphic>
</Event>
...


2. You have .srt, .ass or .ssa subtitle file.
3. You have a other software that can render subtitles over 2D video (AviSynth plugin is preferable, because rendering result must be lossless).


PLUGIN USAGE

It is AviSynth plugin, plugin API described in sub3d.avs. I'll just use function names here.
I assume, you are familiar with AviSynth a little bit.

3D subtitle rendering over 3D video consists of 2 steps:
1) subtitle depth computation, 2) rendering.
These steps are separate. That means, that after depth computation is performed, you will be able to tinker computed depths, and then render subtitles.

Case #1. If you have subtitles with .xml format.
Use CalcXMLDepths function to calculate depths: .xml file will be updated: to every Graphic tag Depth attribute with computed value will be added.
(This AviSynth function outputs nothing, just do it's job. Function input must be full-size horizontal stack of the views. Left view is on the left.)
Use RenderXML function to render subtitles over 3D video. This function input is horizontal stack video over that you want render .xml subtitles. Output is same horizontal stack, but with rendered subtitles.

Case #2. If you have .srt, .ass or .ssa subtitles.
First of all, you need vsfilter.klite.dll AviSynth plugin that provides a MaskSub function that renders subtitles with alpha channel. sub3d.dll uses this function internally. See script example sub3d-2.avs for reference. vsfilter.klite.dll is an .dll that comes with K-Lite Codec Pack (https://ru.wikipedia.org/wiki/K-Lite_Codec_Pack). There might be older version in my package.

Use CalcSRTDepths to calculate depths. This function returns nothing (i.e. no rendering), accepts 3D video (horizontal stack) as input + .srt filename. It will produce *.depths file that is needed by RenderSRT function. The *.depths file format is straight forward: every line contains:
<subtitle_number> <#frame_when_subtitle_starts> <subtitle_length_in_frames>
You may tinker around this file before using RenderSRT.

Use RenderSRT function to render *.depths file over 3D video.

Case #3. If you have some other software (AviSynth plugin) that can render subtitles over 2D video.
sub3d-3.avs script is demonstration of SupTitle.dll plugin usage that provides SupTitle function that can render .sup file over simple 2D video.

The pipeline here is more complex. First of all we need to pass rendered subtitles with alpha channel to CalcSRTDepths function, but SupTitle function renders only subtitles over video. That's where RestoreAlpha function comes to help. It accepts another AviSynth function name that can render subtitles over video in RGB32 format and a videoclip, from what it will take clip duration and video size. RestoreAlpha function outputs rendered subtitles with alpha channel.
Now we can pass this output to CalcSRTDepths function (it's 3rd parameter is a function name that provides subtitles with alpha channel) and finally calculate the depths. Subtitle rendering is performed (as in case #2) with RenderSRT function.


NOTICES

1. sub3d works with RGB32 colorspace. It output is in RGB32 too. Sometimes a color conversion is required.
2. Every Calc*Depths function uses sub3d.lua file for raw subtitle depth values processing. As shown in sub3d.lua, it may write .ofs planes file easily, if needed. For LUA Apis look at SetDepthComputationAlg description in sub3d.avs.


ACKNOWLEDGEMENTS

Thanks to r0lZ (http://forum.doom9.org/member.php?u=37912), who spend decent amount of time testing the plugin.

-

[slavanap]

HOW TO USE PLUGIN WITH BD3D2MK3D (http://forum.doom9.org/showthread.php?t=170828)

I know that r0lZ (http://forum.doom9.org/member.php?u=37912) probably will include plugin into BD3D2MK3D.
This manual below shows how to do it right now by manually modifying generated .avs script.

Below is modified script __ENCODE_3D_MOVIE.avs.
Changes are highlighted with bold and red text.
The script represents demonstration of Case #3 described in top post.
Note, RenderSRT function hardcoding subtitles. If you need to get .ofs file as a result, please read sub3d.lua script.

# Avisynth script generated Sat May 28 12:36:42 AST 2016 by BD3D2MK3D v0.89
# to convert "F:\BDMV\PLAYLIST\00001.mpls"
# (referencing the M2TS/SSIF files 00001, 00010, 00011, 00012, 00013, 00014)
# to 3D Side-by-Side.
# Movie title: test
#
# Source MPLS information: Blu-ray 3D.
# MPLS file: 00001.mpls
# Video : Und AVC, 1080p, 23.976 fps, left-eye
# Video : Und MVC, 1080p, 23.976 fps, right-eye (32 3D-Planes)
# Audio : Eng DTS-HD-Master, Multi-channel, 48 Khz
# Audio : Ces AC3, Multi-channel, 48 Khz
# Audio : Kaz AC3, Multi-channel, 48 Khz
# Audio : Rus AC3, Multi-channel, 48 Khz
# Audio : Ukr AC3, Multi-channel, 48 Khz
# Audio : Eng AC3, Stereo, 48 Khz
# Subtitle: Eng PGS, 3D-plane: 1
# Subtitle: Ara PGS, 3D-plane: 2
# Subtitle: Bul PGS, 3D-plane: 3
# Subtitle: Hrv PGS, 3D-plane: 4
# Subtitle: Ces PGS, 3D-plane: 5
# Subtitle: Est PGS, 3D-plane: 6
# Subtitle: Heb PGS, 3D-plane: 7
# Subtitle: Lav PGS, 3D-plane: 8
# Subtitle: Lit PGS, 3D-plane: 9
# Subtitle: Ron PGS, 3D-plane: 10
# Subtitle: Rus PGS, 3D-plane: 11
# Subtitle: Srp PGS, 3D-plane: 12
# Subtitle: Slk PGS, 3D-plane: 13
# Subtitle: Slv PGS, 3D-plane: 14
# Subtitle: Eng PGS, 3D-plane: 15
# Subtitle: Ces PGS, 3D-plane: 16
# Subtitle: Rus PGS, 3D-plane: 17

LoadPlugin("C:\Users\Vyacheslav\Downloads\BD3D2MK3D\toolset\LoadHelper.dll")
LoadPlugin("DGMVCDecode.dll")
#LoadPlugin("FRIMSource.dll")

##LoadPlugin("VSFilter.dll")
#LoadPlugin("SupTitle.dll")

# add sub3d.dll plugin loading before opening video stream
# there's a full path to installed plugin, the folder with Sub3D.dll must contain Sub3D.lua file as well.
LoadPlugin("c:\Program Files (x86)\ssifSource\bin\Sub3D.dll")

# Load the two video streams (203670 frames per stream)
# wrap file opening with SequentialToSeekable function
interleaved = SequentialToSeekable("""DGMVCSource("00001.track_4113.264", "00001.track_4114.mvc", view = 0, frames = 203670, hw = 0)""")
#interleaved = FRIMSource("mvc", "00001.track_4113.264", "00001.track_4114.mvc", num_frames = 203670, cache = 2, platform = "")
# Current base view: left eye.
# The views are in the common order: AVC stream = left view, MVC stream = right view.
left = SelectEven(interleaved)
right = SelectOdd(interleaved)

## --- CUT HERE -------------------------------------------------------------------------
# here's what you need to add to render subtitles with computed depth
# note, specify path to your .sup file at line below
global SupFileName = "00001.track_4608.Eng.sup"
LoadPlugin("SupTitle.dll")
function SuptitleExtract(clip param) {
return param.ConvertToYV12.SupTitle(SupFileName).ConvertToRgb32
}
global gleft = left.ConvertToRgb32
function SupMaskSub(string srtfile, int width, int height, float fps, int numframes) {
Assert(width == gleft.width && height == gleft.height \
&& fps == gleft.framerate && numframes == gleft.framecount, \
"Unexpected error")
return Sub3D_RestoreAlpha("SuptitleExtract", gleft)
}
FullSbsStack = StackHorizontal(left, right).ConvertToRgb32
# calculate depths only when .depths file is missing
Exist(SupFileName + ".depths") ? NOP : Sub3D_SetLogger(console = true)
Exist(SupFileName + ".depths") ? NOP : FullSbsStack.Sub3D_CalcSRTDepths(SupFileName, masksub = "SupMaskSub", progress = true)
FullSbsStack = FullSbsStack.Sub3D_RenderSRT(SupFileName, masksub = "SupMaskSub").ConvertToYV12
left = FullSbsStack.crop(0, 0, FullSbsStack.width/2, FullSbsStack.height)
right = FullSbsStack.crop(FullSbsStack.width/2, 0, FullSbsStack.width/2, FullSbsStack.height)
## --- CUT HERE -------------------------------------------------------------------------

# Build Side-by-Side stream
StackHorizontal(HorizontalReduceBy2(Left), HorizontalReduceBy2(Right))
AssumeFPS("ntsc_film")

# Hardcode subtitles
##VobSub(".sub")
#SupTitle(".sup")

# Return the 3D clip.
Return(last)#.Info()

[amayra]

is this open source plugin ?

[r0lZ]

Sorry slavanap, I haven't noticed this thread before today! :o I'm glad you have released your plugin. I will try to find some time to include it in BD3D2MK3D. Thanks anyway for your work!

[slavanap]

amayra,
No, it is not opensource, but freeware.
Any software that uses a plugin must include in About section a link to this thread.

But, you can tinker around the algorithm of raw depth values post-processing. See sub3d.lua for more examples.

[slavanap]

Recent changes

added manual (http://forum.doom9.org/showthread.php?p=1758283#post1758283) about how to use plugin with BD3D2MK3D
version update to 1.009 (progress parameter added to CalcSRTDepths, sub3d.lua script simplified).

[robl45]

I'd like to do this, but this is a bit over my head, is there a simple tutorial? Right now I'm using http://forum.doom9.org/showthread.php?t=154957 BDtoAVCHD to mvc encode my 3d movies to smaller size. That puts in the subtitles and flags them forced, but as I just noticed, they are flat. After it is encoded and smaller, can I add the subtitles in?

[slavanap]

robl45,
That's partially possible with few exceptions: you'll need to:
1. break/interrupt second step encoding with BDtoAVCHD by
- temporarily renaming "C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll" to other name. The second step will fail because this plugin is required.
- killing/terminating BDtoAVCHD.exe process with Windows task manager when second step started,
2. modify .avs file in BDtoAVCHD temporary folder (by default it is set to "C:\Users\Vyacheslav\AppData\Local\Temp\BDtoAVCHD", you can change it in Settings),
3. manually perform encoding with x264.exe. That's the most difficult part because BDtoAVCHD hides x264 commands under the hood and does not support "resuming" of the interrupted encoding. Currently I can't advice you any simple solutions for this step except to switch from BDtoAVCHD to BD3D2MK3D (http://forum.doom9.org/showthread.php?t=170828) or learn how to run encoding with x264.exe by yourself. I also posted a question to BDtoAVCHD topic (http://forum.doom9.org/showthread.php?p=1774988#post1774988) about this "resuming" issue.

P.S. I was also surprised that BDtoAVCHD does not work on clean system (SupTitle.dll requires Visual Studio redistributable package to be installed -- these Microsoft's .dll-s that lies in its folder weren't picked up by default).

Here's the original "'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avs"
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

Interleaved = MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", \
"Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)

Right = SelectOdd(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
forcedOnly=false, swapCbCr=false, relocate=false)
Left = SelectEven(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
forcedOnly=false, swapCbCr=false, relocate=false)

return StackHorizontal(HorizontalReduceBy2(Left), HorizontalReduceBy2(Right))

Modified version:
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

LoadPlugin("C:\Program Files (x86)\ssifSource\bin\LoadHelper.dll")
LoadPlugin("Sub3D.dll")
Import("C:\Program Files (x86)\ssifSource\bin\Tools3d.avsi")

Interleaved = SequentialToSeekable("""MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", """ + \
""" "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)""")
global Left = SelectEven(Interleaved).ConvertToRGB32
FullSBS = Interleaved.Convert3D(FORMAT_INTERLEAVED, FORMAT_FULLSBS).ConvertToRGB32
global subfile = "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup"

function SuptitleExtract(clip param) {
return param \
.ConvertToYV12 \
.SupTitle(subfile, forcedOnly=false, swapCbCr=false, relocate=false) \
.ConvertToRgb32
}

function MyMaskSub(string srtfile, int width, int height, float fps, int numframes) {
Assert(width == left.width && height == left.height && fps == left.framerate && numframes == left.framecount, \
"Something bad happened")
return RestoreAlpha("SuptitleExtract", left)
}

Exist(subfile + ".depths") ? NOP : Sub3D_SetLogger(console = true)
Exist(subfile + ".depths") ? NOP : FullSBS.CalcSRTDepths(subfile, masksub = "MyMaskSub", progress = true)
FullSBS = FullSBS.RenderSRT(subfile, masksub = "MyMaskSub")

return FullSBS.Convert3D(FORMAT_FULLSBS, FORMAT_HALFSBS)

[robl45]

Thanks, but that won't work because I need it encoded with the MVC encoder as I want framepacked 3d. Almost nothing plays full sbs.

robl45,
That's partially possible with few exceptions: you'll need to:
1. break/interrupt second step encoding with BDtoAVCHD by
- temporarily renaming "C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll" to other name. The second step will fail because this plugin is required.
- killing/terminating BDtoAVCHD.exe process with Windows task manager when second step started,
2. modify .avs file in BDtoAVCHD temporary folder (by default it is set to "C:\Users\Vyacheslav\AppData\Local\Temp\BDtoAVCHD", you can change it in Settings),
3. manually perform encoding with x264.exe. That's the most difficult part because BDtoAVCHD hides x264 commands under the hood and does not support "resuming" of the interrupted encoding. Currently I can't advice you any simple solutions for this step except to switch from BDtoAVCHD to BD3D2MK3D (http://forum.doom9.org/showthread.php?t=170828) or learn how to run encoding with x264.exe by yourself. I also posted a question to BDtoAVCHD topic (http://forum.doom9.org/showthread.php?p=1774988#post1774988) about this "resuming" issue.

P.S. I was also surprised that BDtoAVCHD does not work on clean system (SupTitle.dll requires Visual Studio redistributable package to be installed -- these Microsoft's .dll-s that lies in its folder weren't picked up by default).

Here's the original "'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avs"
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

Interleaved = MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", \
"Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)

Right = SelectOdd(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
forcedOnly=false, swapCbCr=false, relocate=false)
Left = SelectEven(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
forcedOnly=false, swapCbCr=false, relocate=false)

return StackHorizontal(HorizontalReduceBy2(Left), HorizontalReduceBy2(Right))

Modified version:
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

LoadPlugin("C:\Program Files (x86)\ssifSource\bin\LoadHelper.dll")
LoadPlugin("Sub3D.dll")
Import("C:\Program Files (x86)\ssifSource\bin\Tools3d.avsi")

Interleaved = SequentialToSeekable("""MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", """ + \
""" "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)""")
global Left = SelectEven(Interleaved).ConvertToRGB32
FullSBS = Interleaved.Convert3D(FORMAT_INTERLEAVED, FORMAT_FULLSBS).ConvertToRGB32
global subfile = "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup"

function SuptitleExtract(clip param) {
return param \
.ConvertToYV12 \
.SupTitle(subfile, forcedOnly=false, swapCbCr=false, relocate=false) \
.ConvertToRgb32
}

function MyMaskSub(string srtfile, int width, int height, float fps, int numframes) {
Assert(width == left.width && height == left.height && fps == left.framerate && numframes == left.framecount, \
"Something bad happened")
return RestoreAlpha("SuptitleExtract", left)
}

Exist(subfile + ".depths") ? NOP : Sub3D_SetLogger(console = true)
Exist(subfile + ".depths") ? NOP : FullSBS.CalcSRTDepths(subfile, masksub = "MyMaskSub", progress = true)
FullSBS = FullSBS.RenderSRT(subfile, masksub = "MyMaskSub")

return FullSBS.Convert3D(FORMAT_FULLSBS, FORMAT_HALFSBS)

[slavanap]

Thanks, but that won't work because I need it encoded with the MVC encoder as I want framepacked 3d. Almost nothing plays full sbs.

Well, then what subtitle format you need to get? Or what video output format?
Sub3D supports .ofs file generation and
Tools3D.avsi may convert your video to FORMAT_INTERLEAVED (the one produced by MVCSource).

FULLSBS is only needed by CalcSRTDepths and RenderSRT (Sub3D plugin's) functions.

[robl45]

I found solution for me, both mpc-be and KODI MVC build can do stereoscopic subtitles.