Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > (HD) DVD & Blu-ray authoring

Reply
 
Thread Tools Search this Thread Display Modes
Old 30th January 2016, 00:19   #1  |  Link
slavanap
Registered User
 
slavanap's Avatar
 
Join Date: May 2011
Location: Moscow, Russia
Posts: 124
sub3d: plugin for rendering subtitles on 3D video with correct depth

Render your 2D subtitles on 3D video with correct depth.

>>> CURRENT RELEASE AVAILABLE HERE <<<

CASES THAT PLUGIN SUPPORTS

1. You have subtitles in xml format.
I know that BD3D2MK3D tool somehow generates this format.
The output should be a folder with .xml file (descriptor) + .png images (subtitles).
.xml file looks like this:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<BDN Version="0.93" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="BD-03-006-0093b BDN File Format.xsd">
    <Description>
        <Name Title="FRA_2D" Content=""/>
        <Language Code="eng"/>
        <Format VideoFormat="1080p" FrameRate="23.976" DropFrame="False"/>
        <Events Type="Graphic" FirstEventInTC="00:00:02:17" LastEventOutTC="00:04:33:18" NumberofEvents="47"/>
    </Description>
    <Events>
        <Event InTC="00:00:02:17" OutTC="00:00:05:07" Forced="False">
            <Graphic Width="840" Height="50" X="540" Y="875" Depth="24">FRA_2D_0001.png</Graphic>
        </Event>
...
2. You have .srt, .ass or .ssa subtitle file.
3. You have a other software that can render subtitles over 2D video (AviSynth plugin is preferable, because rendering result must be lossless).


PLUGIN USAGE

It is AviSynth plugin, plugin API described in sub3d.avs. I'll just use function names here.
I assume, you are familiar with AviSynth a little bit.

3D subtitle rendering over 3D video consists of 2 steps:
1) subtitle depth computation, 2) rendering.
These steps are separate. That means, that after depth computation is performed, you will be able to tinker computed depths, and then render subtitles.

Case #1. If you have subtitles with .xml format.
Use CalcXMLDepths function to calculate depths: .xml file will be updated: to every Graphic tag Depth attribute with computed value will be added.
(This AviSynth function outputs nothing, just do it's job. Function input must be full-size horizontal stack of the views. Left view is on the left.)
Use RenderXML function to render subtitles over 3D video. This function input is horizontal stack video over that you want render .xml subtitles. Output is same horizontal stack, but with rendered subtitles.

Case #2. If you have .srt, .ass or .ssa subtitles.
First of all, you need vsfilter.klite.dll AviSynth plugin that provides a MaskSub function that renders subtitles with alpha channel. sub3d.dll uses this function internally. See script example sub3d-2.avs for reference. vsfilter.klite.dll is an .dll that comes with K-Lite Codec Pack. There might be older version in my package.

Use CalcSRTDepths to calculate depths. This function returns nothing (i.e. no rendering), accepts 3D video (horizontal stack) as input + .srt filename. It will produce *.depths file that is needed by RenderSRT function. The *.depths file format is straight forward: every line contains:
<subtitle_number> <#frame_when_subtitle_starts> <subtitle_length_in_frames>
You may tinker around this file before using RenderSRT.

Use RenderSRT function to render *.depths file over 3D video.

Case #3. If you have some other software (AviSynth plugin) that can render subtitles over 2D video.
sub3d-3.avs script is demonstration of SupTitle.dll plugin usage that provides SupTitle function that can render .sup file over simple 2D video.

The pipeline here is more complex. First of all we need to pass rendered subtitles with alpha channel to CalcSRTDepths function, but SupTitle function renders only subtitles over video. That's where RestoreAlpha function comes to help. It accepts another AviSynth function name that can render subtitles over video in RGB32 format and a videoclip, from what it will take clip duration and video size. RestoreAlpha function outputs rendered subtitles with alpha channel.
Now we can pass this output to CalcSRTDepths function (it's 3rd parameter is a function name that provides subtitles with alpha channel) and finally calculate the depths. Subtitle rendering is performed (as in case #2) with RenderSRT function.


NOTICES

1. sub3d works with RGB32 colorspace. It output is in RGB32 too. Sometimes a color conversion is required.
2. Every Calc*Depths function uses sub3d.lua file for raw subtitle depth values processing. As shown in sub3d.lua, it may write .ofs planes file easily, if needed. For LUA Apis look at SetDepthComputationAlg description in sub3d.avs.


ACKNOWLEDGEMENTS

Thanks to r0lZ, who spend decent amount of time testing the plugin.

-

Last edited by slavanap; 20th July 2016 at 14:02. Reason: plugin version update
slavanap is offline   Reply With Quote
Old 23rd February 2016, 14:26   #2  |  Link
slavanap
Registered User
 
slavanap's Avatar
 
Join Date: May 2011
Location: Moscow, Russia
Posts: 124
HOW TO USE PLUGIN WITH BD3D2MK3D

I know that r0lZ probably will include plugin into BD3D2MK3D.
This manual below shows how to do it right now by manually modifying generated .avs script.

Below is modified script __ENCODE_3D_MOVIE.avs.
Changes are highlighted with bold and red text.
The script represents demonstration of Case #3 described in top post.
Note, RenderSRT function hardcoding subtitles. If you need to get .ofs file as a result, please read sub3d.lua script.

Code:
# Avisynth script generated Sat May 28 12:36:42 AST 2016 by BD3D2MK3D v0.89
# to convert "F:\BDMV\PLAYLIST\00001.mpls"
# (referencing the M2TS/SSIF files 00001, 00010, 00011, 00012, 00013, 00014)
# to 3D Side-by-Side.
# Movie title: test
#
# Source MPLS information:  Blu-ray 3D.
# MPLS file: 00001.mpls
# Video   :  Und  AVC, 1080p, 23.976 fps, left-eye
# Video   :  Und  MVC, 1080p, 23.976 fps, right-eye (32 3D-Planes)
# Audio   :  Eng  DTS-HD-Master, Multi-channel, 48 Khz
# Audio   :  Ces  AC3, Multi-channel, 48 Khz
# Audio   :  Kaz  AC3, Multi-channel, 48 Khz
# Audio   :  Rus  AC3, Multi-channel, 48 Khz
# Audio   :  Ukr  AC3, Multi-channel, 48 Khz
# Audio   :  Eng  AC3, Stereo, 48 Khz
# Subtitle:  Eng  PGS, 3D-plane: 1
# Subtitle:  Ara  PGS, 3D-plane: 2
# Subtitle:  Bul  PGS, 3D-plane: 3
# Subtitle:  Hrv  PGS, 3D-plane: 4
# Subtitle:  Ces  PGS, 3D-plane: 5
# Subtitle:  Est  PGS, 3D-plane: 6
# Subtitle:  Heb  PGS, 3D-plane: 7
# Subtitle:  Lav  PGS, 3D-plane: 8
# Subtitle:  Lit  PGS, 3D-plane: 9
# Subtitle:  Ron  PGS, 3D-plane: 10
# Subtitle:  Rus  PGS, 3D-plane: 11
# Subtitle:  Srp  PGS, 3D-plane: 12
# Subtitle:  Slk  PGS, 3D-plane: 13
# Subtitle:  Slv  PGS, 3D-plane: 14
# Subtitle:  Eng  PGS, 3D-plane: 15
# Subtitle:  Ces  PGS, 3D-plane: 16
# Subtitle:  Rus  PGS, 3D-plane: 17

LoadPlugin("C:\Users\Vyacheslav\Downloads\BD3D2MK3D\toolset\LoadHelper.dll")
LoadPlugin("DGMVCDecode.dll")
#LoadPlugin("FRIMSource.dll")

##LoadPlugin("VSFilter.dll")
#LoadPlugin("SupTitle.dll")

# add sub3d.dll plugin loading before opening video stream
# there's a full path to installed plugin, the folder with Sub3D.dll must contain Sub3D.lua file as well.
LoadPlugin("c:\Program Files (x86)\ssifSource\bin\Sub3D.dll")

# Load the two video streams (203670 frames per stream)
# wrap file opening with SequentialToSeekable function
interleaved = SequentialToSeekable("""DGMVCSource("00001.track_4113.264", "00001.track_4114.mvc", view = 0, frames = 203670, hw = 0)""")
#interleaved = FRIMSource("mvc", "00001.track_4113.264", "00001.track_4114.mvc", num_frames = 203670, cache = 2, platform = "")
# Current base view: left eye.
# The views are in the common order: AVC stream = left view, MVC stream = right view.
left  = SelectEven(interleaved)
right = SelectOdd(interleaved)

## --- CUT HERE -------------------------------------------------------------------------
# here's what you need to add to render subtitles with computed depth
# note, specify path to your .sup file at line below
global SupFileName = "00001.track_4608.Eng.sup"
LoadPlugin("SupTitle.dll")
function SuptitleExtract(clip param) {
    return param.ConvertToYV12.SupTitle(SupFileName).ConvertToRgb32
}
global gleft = left.ConvertToRgb32
function SupMaskSub(string srtfile, int width, int height, float fps, int numframes) {
    Assert(width == gleft.width && height == gleft.height \
    	&& fps == gleft.framerate && numframes == gleft.framecount, \
    	"Unexpected error")
    return Sub3D_RestoreAlpha("SuptitleExtract", gleft)
}
FullSbsStack = StackHorizontal(left, right).ConvertToRgb32
# calculate depths only when .depths file is missing
Exist(SupFileName + ".depths") ? NOP : Sub3D_SetLogger(console = true)
Exist(SupFileName + ".depths") ? NOP : FullSbsStack.Sub3D_CalcSRTDepths(SupFileName, masksub = "SupMaskSub", progress = true)
FullSbsStack = FullSbsStack.Sub3D_RenderSRT(SupFileName, masksub = "SupMaskSub").ConvertToYV12
left = FullSbsStack.crop(0, 0, FullSbsStack.width/2, FullSbsStack.height)
right = FullSbsStack.crop(FullSbsStack.width/2, 0, FullSbsStack.width/2, FullSbsStack.height)
## --- CUT HERE -------------------------------------------------------------------------

# Build Side-by-Side stream
StackHorizontal(HorizontalReduceBy2(Left), HorizontalReduceBy2(Right))
AssumeFPS("ntsc_film")

# Hardcode subtitles
##VobSub(".sub")
#SupTitle(".sup")

# Return the 3D clip.
Return(last)#.Info()

Last edited by slavanap; 20th July 2016 at 14:27.
slavanap is offline   Reply With Quote
Old 31st March 2016, 18:34   #3  |  Link
amayra
Quality Checker
 
amayra's Avatar
 
Join Date: Aug 2013
Posts: 275
is this open source plugin ?
__________________
I love Doom9
amayra is offline   Reply With Quote
Old 4th April 2016, 08:40   #4  |  Link
r0lZ
PgcEdit daemon
 
r0lZ's Avatar
 
Join Date: Jul 2003
Posts: 7,404
Sorry slavanap, I haven't noticed this thread before today! I'm glad you have released your plugin. I will try to find some time to include it in BD3D2MK3D. Thanks anyway for your work!
__________________
r0lZ
PgcEdit homepage (hosted by VideoHelp)
BD3D2MK3D A tool to convert 3D blu-rays to SBS, T&B or FS MKV
r0lZ is offline   Reply With Quote
Old 26th May 2016, 11:32   #5  |  Link
slavanap
Registered User
 
slavanap's Avatar
 
Join Date: May 2011
Location: Moscow, Russia
Posts: 124
amayra,
No, it is not opensource, but freeware.
Any software that uses a plugin must include in About section a link to this thread.

But, you can tinker around the algorithm of raw depth values post-processing. See sub3d.lua for more examples.

Last edited by slavanap; 28th May 2016 at 12:11.
slavanap is offline   Reply With Quote
Old 28th May 2016, 14:02   #6  |  Link
slavanap
Registered User
 
slavanap's Avatar
 
Join Date: May 2011
Location: Moscow, Russia
Posts: 124
Recent changes
  • added manual about how to use plugin with BD3D2MK3D
  • version update to 1.009 (progress parameter added to CalcSRTDepths, sub3d.lua script simplified).
slavanap is offline   Reply With Quote
Old 21st July 2016, 23:23   #7  |  Link
robl45
Registered User
 
Join Date: Dec 2012
Posts: 162
I'd like to do this, but this is a bit over my head, is there a simple tutorial? Right now I'm using http://forum.doom9.org/showthread.php?t=154957 BDtoAVCHD to mvc encode my 3d movies to smaller size. That puts in the subtitles and flags them forced, but as I just noticed, they are flat. After it is encoded and smaller, can I add the subtitles in?
robl45 is offline   Reply With Quote
Old 22nd July 2016, 18:25   #8  |  Link
slavanap
Registered User
 
slavanap's Avatar
 
Join Date: May 2011
Location: Moscow, Russia
Posts: 124
robl45,
That's partially possible with few exceptions: you'll need to:
1. break/interrupt second step encoding with BDtoAVCHD by
- temporarily renaming "C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll" to other name. The second step will fail because this plugin is required.
- killing/terminating BDtoAVCHD.exe process with Windows task manager when second step started,
2. modify .avs file in BDtoAVCHD temporary folder (by default it is set to "C:\Users\Vyacheslav\AppData\Local\Temp\BDtoAVCHD", you can change it in Settings),
3. manually perform encoding with x264.exe. That's the most difficult part because BDtoAVCHD hides x264 commands under the hood and does not support "resuming" of the interrupted encoding. Currently I can't advice you any simple solutions for this step except to switch from BDtoAVCHD to BD3D2MK3D or learn how to run encoding with x264.exe by yourself. I also posted a question to BDtoAVCHD topic about this "resuming" issue.

P.S. I was also surprised that BDtoAVCHD does not work on clean system (SupTitle.dll requires Visual Studio redistributable package to be installed -- these Microsoft's .dll-s that lies in its folder weren't picked up by default).

Here's the original "'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avs"
Code:
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

Interleaved = MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", \
    "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)

Right = SelectOdd(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
    forcedOnly=false, swapCbCr=false, relocate=false)
Left = SelectEven(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
    forcedOnly=false, swapCbCr=false, relocate=false)

return StackHorizontal(HorizontalReduceBy2(Left), HorizontalReduceBy2(Right))
Modified version:
Code:
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

LoadPlugin("C:\Program Files (x86)\ssifSource\bin\LoadHelper.dll")
LoadPlugin("Sub3D.dll")
Import("C:\Program Files (x86)\ssifSource\bin\Tools3d.avsi")

Interleaved = SequentialToSeekable("""MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", """ + \
    """ "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)""")
global Left = SelectEven(Interleaved).ConvertToRGB32
FullSBS = Interleaved.Convert3D(FORMAT_INTERLEAVED, FORMAT_FULLSBS).ConvertToRGB32
global subfile = "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup"

function SuptitleExtract(clip param) {
    return param \
        .ConvertToYV12 \
        .SupTitle(subfile, forcedOnly=false, swapCbCr=false, relocate=false) \
        .ConvertToRgb32
}

function MyMaskSub(string srtfile, int width, int height, float fps, int numframes) {
    Assert(width == left.width && height == left.height && fps == left.framerate && numframes == left.framecount, \
        "Something bad happened")
    return RestoreAlpha("SuptitleExtract", left)
}

Exist(subfile + ".depths") ? NOP : Sub3D_SetLogger(console = true)
Exist(subfile + ".depths") ? NOP : FullSBS.CalcSRTDepths(subfile, masksub = "MyMaskSub", progress = true)
FullSBS = FullSBS.RenderSRT(subfile, masksub = "MyMaskSub")

return FullSBS.Convert3D(FORMAT_FULLSBS, FORMAT_HALFSBS)

Last edited by slavanap; 22nd July 2016 at 19:40.
slavanap is offline   Reply With Quote
Old 25th July 2016, 12:09   #9  |  Link
robl45
Registered User
 
Join Date: Dec 2012
Posts: 162
Thanks, but that won't work because I need it encoded with the MVC encoder as I want framepacked 3d. Almost nothing plays full sbs.

Quote:
Originally Posted by slavanap View Post
robl45,
That's partially possible with few exceptions: you'll need to:
1. break/interrupt second step encoding with BDtoAVCHD by
- temporarily renaming "C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll" to other name. The second step will fail because this plugin is required.
- killing/terminating BDtoAVCHD.exe process with Windows task manager when second step started,
2. modify .avs file in BDtoAVCHD temporary folder (by default it is set to "C:\Users\Vyacheslav\AppData\Local\Temp\BDtoAVCHD", you can change it in Settings),
3. manually perform encoding with x264.exe. That's the most difficult part because BDtoAVCHD hides x264 commands under the hood and does not support "resuming" of the interrupted encoding. Currently I can't advice you any simple solutions for this step except to switch from BDtoAVCHD to BD3D2MK3D or learn how to run encoding with x264.exe by yourself. I also posted a question to BDtoAVCHD topic about this "resuming" issue.

P.S. I was also surprised that BDtoAVCHD does not work on clean system (SupTitle.dll requires Visual Studio redistributable package to be installed -- these Microsoft's .dll-s that lies in its folder weren't picked up by default).

Here's the original "'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avs"
Code:
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

Interleaved = MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", \
    "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)

Right = SelectOdd(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
    forcedOnly=false, swapCbCr=false, relocate=false)
Left = SelectEven(Interleaved).SupTitle("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup", \
    forcedOnly=false, swapCbCr=false, relocate=false)

return StackHorizontal(HorizontalReduceBy2(Left), HorizontalReduceBy2(Right))
Modified version:
Code:
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\MVCsource.dll")
LoadPlugin("C:\Program Files (x86)\BDtoAVCHD\SupTitle\SupTitle.dll")

LoadPlugin("C:\Program Files (x86)\ssifSource\bin\LoadHelper.dll")
LoadPlugin("Sub3D.dll")
Import("C:\Program Files (x86)\ssifSource\bin\Tools3d.avsi")

Interleaved = SequentialToSeekable("""MVCsource("Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.avc.h264", """ + \
    """ "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.mvc.h264", 158664, 2)""")
global Left = SelectEven(Interleaved).ConvertToRGB32
FullSBS = Interleaved.Convert3D(FORMAT_INTERLEAVED, FORMAT_FULLSBS).ConvertToRGB32
global subfile = "Y:\TEMP\'PROJECTOR' THE THREE MUSKETEERS 3D (2011) 3D-HSBS.job_0.sup"

function SuptitleExtract(clip param) {
    return param \
        .ConvertToYV12 \
        .SupTitle(subfile, forcedOnly=false, swapCbCr=false, relocate=false) \
        .ConvertToRgb32
}

function MyMaskSub(string srtfile, int width, int height, float fps, int numframes) {
    Assert(width == left.width && height == left.height && fps == left.framerate && numframes == left.framecount, \
        "Something bad happened")
    return RestoreAlpha("SuptitleExtract", left)
}

Exist(subfile + ".depths") ? NOP : Sub3D_SetLogger(console = true)
Exist(subfile + ".depths") ? NOP : FullSBS.CalcSRTDepths(subfile, masksub = "MyMaskSub", progress = true)
FullSBS = FullSBS.RenderSRT(subfile, masksub = "MyMaskSub")

return FullSBS.Convert3D(FORMAT_FULLSBS, FORMAT_HALFSBS)
robl45 is offline   Reply With Quote
Old 25th July 2016, 12:54   #10  |  Link
slavanap
Registered User
 
slavanap's Avatar
 
Join Date: May 2011
Location: Moscow, Russia
Posts: 124
Quote:
Originally Posted by robl45 View Post
Thanks, but that won't work because I need it encoded with the MVC encoder as I want framepacked 3d. Almost nothing plays full sbs.
Well, then what subtitle format you need to get? Or what video output format?
Sub3D supports .ofs file generation and
Tools3D.avsi may convert your video to FORMAT_INTERLEAVED (the one produced by MVCSource).

FULLSBS is only needed by CalcSRTDepths and RenderSRT (Sub3D plugin's) functions.
slavanap is offline   Reply With Quote
Old 28th July 2016, 22:20   #11  |  Link
robl45
Registered User
 
Join Date: Dec 2012
Posts: 162
I found solution for me, both mpc-be and KODI MVC build can do stereoscopic subtitles.
robl45 is offline   Reply With Quote
Reply

Tags
bd3d, depth estimation, rendering, subtitle

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:07.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.