Help needed : MCF subtitles specs, Vobsub specs ....

[ChristianHJW]

Hi folks,

we are about to rethink the subtitle specs for MCF, so i was asking for help at the Doom9 subtitles section here http://forum.doom9.org/showthread.php?s=&threadid=24484 , unfortunately without any reply.

Tronic was posting to the last mailinglist it would be nice to get a link with the official Vobsub specs ( mwillberg reading ? ) because we are looking at this issue again, but the link to the vobsub homepage i got ( http://members.tripod.com/vobsub ) dont work for me ( server down ? ). I did a search of the complete board for a few keywords like 'vobsub', 'specs' , 'specifications' and 'link' in different combinations, but all i could find was this link to the SSA specs here :

SSA specs : http://www.eswat.demon.co.uk/substation.html

and of course Doom9's subtitles links :


Radioactive pages (http://www.man.poznan.pl/%7Eradeks/english/subcreator.html) - home of SubCreator
Ripitall (http://www.ripitall.be.tf/)- homepage of Subadjust and synchronizer
S3Converter homepage (http://www.studioann.com/shared/gifs/bkp/mustafah/tutorials/s3tut.htm)
SubRip (http://www.subrip.fr.st/) - great subtitle ripper
Subripper (http://borsuk.am.lublin.pl/subripper) - another good subtitle ripper
SubSynch (http://users.pandora.be/vlad/subsync/) - homepage
VirtualDub subtitler plugin homepage (http://www.virtualdub.org/)
VobSub Plugin (http://vobsub.edensrising.com/) - add subtitles to your avis directly from DVD

When browsing through all the links ( well, most of them :D ) i found this page here : http://www.subviewer.com/ and contacted the author of this program about help.

@gabest : can we expect some input from you here ?

[gabest]

Originally posted by ChristianHJW
Tronic was posting to the last mailinglist it would be nice to get a link with the official Vobsub specs ( mwillberg reading ? )There are no specs, the idx file contains comments about nearly everything inside, and the .sub is just a copy of the subtitle packets of the vob files (mpucoder has already made a good site for vob specs including the subtitle part). And not to mention, vobsub is opensource.because we are looking at this issue again,Issue? Again? I'm getting curious :pbut the link to the vobsub homepage i got (http://members.tripod.com/vobsub) dont work for me (server down?).
...
and of course Doom9's subtitles links :
...
VobSub Plugin (http://vobsub.edensrising.com/) - add subtitles to your avis directly from DVD
Did you ever try to click that "VobSub Plugin" link? :D@gabest : can we expect some input from you here ? I'm here. Ask anything and I'll try to reply.

[gabest]

I've just checked that other thread, and I have two comments that I didn't make last time. First, why UTF-8? Are you going to send subtitles over email or just some self mortification? :) My other notice is about SSA. It is a format which isn't very streamable, it has style definitions at the beginning, the dialogues may not be sorted at all, and they can overlap too. So, the program that handles the drawing (dshow filter or the main app owning the video-overlaid window) must know the information of ssa globally.

[ChristianHJW]

Originally posted by gabest I'm here. Ask anything and I'll try to reply. .... sending Tronic here asap, plus PM to mwillberg ... Thanks !

[kxy]

I am assuming you are talking about the MCF in regard to ogm. Since this is what you mentioned in the subtitle thread.

Well, here are my complains. :(
(If you don't want hear my rant, basically I think UTF-8, is way too little, so if you want to go png route that is fine. Please make it as little human interaction as possible)

Okay, now why I think ogm should have external subtitle support. Because I am just hoping for a easy way of doing subtitles. I am used to the ease of using vobsub to rip the idx and sub, then rar the sub files and it is done. You can have different languages with a VERY little human interaction and ripping time.

Currently with ogm, it doesn't provide that ease. You have to used subrip or use other programs and covert it to srt or ssa(look like that is going to be implemented). Then you got to go back to the srt file and manually check it to see if the spacing are correct, and manually fixing ten or hundred of typos, or whatever the OCR algorithm fails to pick up. If there are special symbols you gotta got to the ascii table and look them up and manually putting them in. This resulted hours and hours of work just for one subtitle. Now what if you want to rip it in another language? You got to go back and repeat that all over again.

If there is a way of just pointing the ogm to the idx files which leads to the subs then it would save all the work! And you can have all the languages with min amount work.

[ChristianHJW]

Originally posted by kxy
[B]I am assuming you are talking about the MCF in regard to ogm. Since this is what you mentioned in the subtitle thread.

MCF has nothing to do with OGM. OGMs subtitle specs have been implemented by Tobias, the MCF team would like its subs specs to fully meet the purposes of the potential users .. this is why we start conversation about it.

BTW : i admit i am no subtitles expert, but for sure it is not necessary to use OCR for subs, you can add them as compressed pictures also .... we are looking at .png for this purpose as i understand it. The downside is, it will take much more file size than UTF-8 subs ...

[gabest]

I think you misunderstood me about UTF-8. I wasn't complaining about the above mentioned problems, I just wanted to say that the standard c api and everything built on it supports 2 byte unicode natively (including Windows), and would be a little harder to handle text in 8-bit unicode.

[Tronic]

Issue? Again? I'm getting curious

Well, one option is using PNGs; one for each subtitle. The problem is that it's not very space-efficient.

VobSub format seems to offer surprisingly good compression, so we are somewhat interested on it. However, it might not be very good for high color images or other stuff we also want to support.

The third option is developing a new lossless format. Todays CPUs are quite fast, so we could use very good compression for those images.

For the beginning, we can pack 40 3-color pixels into 64 bits, and there will still be some space to be used for something else. For compression, we are currently going to use two modes - packed raw, and packed & optimized RLE. I believe that this gives very good compression with subtitles and probably with other data types too.

[ChristianHJW]

Copied and pasted from IRC, #mcf :

<robUUUU> UTF-8 is easy to use because it's close to ASCII
<robUUUU> so compatible with a LOT of existing softwares
<Ingo> There are UTF-16/32/8 converters in the library already, if nobody deleted them

[DeXT]

Chris, here you have some net resources regarding the RLE compression format:

http://www.wotsit.org (search for RLE)
http://www.rasip.fer.hr/research/compress/algorithms/fund/rl/
http://www.simtel.net/pub/pd/42636.html
http://www.simtel.net/pub/pd/42635.html
http://inkvine.fluff.org/~forrey/compression/packbits.html
http://astronomy.swin.edu.au/~pbourke/dataformats/rle/
http://www.geocities.com/hmaxf_urlcr/rle.htm
http://www.cs.tut.fi/~albert/Dev/pucrunch/ (RLE/LZ77 hybrid)
http://www.rasip.fer.hr/research/compress/algorithms/adv/bmp/ (BMP with RLE)

And below are some pages with lots of informations and working implementations of several compression formats:

http://datacompression.info/index.shtml
http://www.neutralzone.org/home/faqsys/cates/compress.html
http://www.gamedev.net/reference/list.asp?categoryid=25
http://www.rkeene.org/devel/dact/

For subtitles a 4-bit implementation of RLE bitmap format should be more than enough I think, look at wotsit for information about that. I think that since all these compression algorithms have already been implemented by experienced people there is no need to create a new one.

Oh I forgot, the comp.compression FAQ (http://www.faqs.org/faqs/compression-faq/) is a must-read for anyone interested in data compression. It also talks about the patents covering current compression algorithms.

[raistlin2k]

Why do you dislike subrip 0.97b?

Have 120 movies ripped with it,it's very fast you just have to use Smartipper in streamprocessing mode to write the subtitles into a separate file, this way OCR gets boosted a lot, just 90 seconds per language (first one is slower due to user interaction)

Raist

[dblmask]

http://forum.doom9.org/showthread.php?s=&postid=129130#post129130

[kxy]

Have you guys give vobsub a try? Get out of your comfort zone, try it first. What I am saying is that it require very little human interaction, and you can spent you time elsewhere. But I thought this thread isn't about existing subtitles, but about new MCF spec. Maybe we should do this else where?




Originally posted by raistlin2k
Why do you dislike subrip 0.97b?
Have 120 movies ripped with it,it's very fast you just have to use Smartipper in streamprocessing mode to write the subtitles into a separate file, this way OCR gets boosted a lot, just 90 seconds per language (first one is slower due to user interaction)
Raist

Please re-read my post. I explained my reasons there. I didn't say I dislike it, I said time could be spent elsewhere. If you don't want to re-read, I will sum it up for you. It is fast after the first one, but that just the ripping part. Did you go through each of your subtitles after you are done, I am sure there are some typos or spacing error in it, that you have to go through the entire subtitle to fix it.


I have been on the subject for quite some time and found that the best subtitles choices has to have:

0. Text over Graphics. We want to save space, it's a valuable thing. Don't get LAZZY. OCR ripping is boring, hard and takes time. But is the BEST. don't listen to those who complains: "don't wanna make my homework". Since the people become lazy for programming we got Gb progs that runs super-slow.


Does few kilobytes of space REALLY matter that much to you? Have a nice subtitle with all the symbol and all the languages only end up to be around 500-600 kikobytes after it has being rared. Yet you have to spent couple hours just to do one subtitle, just for a few kikobytes. I don't think the trade off is worth it.

[mwillberg]

@kxy:

Yep, I agree with all your arguments, and that's the reason I've been trying to convince Tronic to include VobSub in some form in MCF.

@Tronic:

If you have any specific specifications (phew, that's a word) you would need help researching or something, please let me know. I would love to help out with anything I can.

[ChristianHJW]

About the use of OCR : did you guys think about whats someone supposed to do who wants to include subs in a language he doesnt speak ? How can you edit the subs if you dont know what its saying ;) ? Ever tried editing a chinese subtitle :D ?

Of course MCF will have a possibility to include text subs, and this is one thing we have to discuss about. Another thing is how to put picture based subs in, and how to do this most intelligent with very best compression and lowest CPU performance possible ....

[gabest]

How about about vectorized subtitle text as a third option? It wouldn't need the specific fonts to be installed on the target machine and would take up less space than the images.

[Rasqual]

Originally posted by gabest
How about about vectorized subtitle text as a third option? It wouldn't need the specific fonts to be installed on the target machine and would take up less space than the images.
I had already proposed to use graphic outlines in SVG (png?) in another post (http://forum.doom9.org/showthread.php?s=&threadid=24437) and I pointed out the fact there might be copyright problems if these outlines were ripped off an official font. I also wonder how much space it will require if many symbols are to be used, like in a Chinese movie for example (if anyone cares ;)). Isn't it the same problem as in PDF/PostScript files? Maybe only specific and uncommon fonts should be embedded and more common fonts like Arial, etc. would be set as a requirement to play the file.

Concerning the subtitle file, I think that if it should placed around the beginning of the file for the the contents to remain streamable.
[off-topic]I also say that because it seems that when I mux using SubMux, the subfile chunk gets pushed a bit far from the beginning of the AVI file. Is it because the sub stream is only the third pin?[/off-topic]
BTW, about Unicode, what's the thing about big and little endian? Does the programmer have to worry about it?

Originally posted by gabest
Originally posted by ChristianHJW
but the link to the vobsub homepage i got (http://members.tripod.com/vobsub) dont work for me (server down?).
...
and of course Doom9's subtitles links :
...
VobSub Plugin - add subtitles to your avis directly from DVD

Did you ever try to click that "VobSub Plugin" link?
lol!! :D

[gabest]

Originally posted by Rasqual
[off-topic]I also say that because it seems that when I mux using SubMux, the subfile chunk gets pushed a bit far from the beginning of the AVI file. Is it because the sub stream is only the third pin?[/off-topic]The file position doesn't matter, the parser filter will send it to the connected filter according to its timestamp, and that's 0.BTW, about Unicode, what's the thing about big and little endian? Does the programmer have to worry about it?Actually, that's the only advantage of UTF-8. It is a packed form of the 16 bit version where the first 128 symbol matches the ascii table, but the rest of them are stored using variable byte length.