SubExtractor - New Sub Ocr App

[Tappen]

I've released an app to extract subs from (non-encrypted, on hard drive) DVDs and convert to Advanced Substation Alpha or SRT format. It can also convert sup (PGS) and sub/idx formats to same. I wrote this because I hate the blocky, too-high-on-the-screen look of regular DVD subtitles and wanted to re-encode my DVD collection in h264/aac/assa with mkv containment.

http://subextractor.codeplex.com/

It's a wizard-style app, allowing you to pick program chains, angles, audio and subtitle tracks from a DVD folder and create mpg, d2v and bin (my own data format similar to sub/idx combined) files for each. DGIndex is used to help line up the subs to the video since DVD programs often have discontinuities that mess up sync. The mpg and d2v files created is great for further re-encoding of DVDs to h264 using a tool like MeGui.

The OCR is pretty basic, just exact pattern matching of the characters. The starting OCR database is good though so most DVDs should require manual matching of just a few characters. Some characters like i, l, I, '.', and o must be manually matched for every DVD since they have a lot of false positives. Some Bluray sup files can be tedious to OCR since the Bluray authors used scaled-up fonts, which means there ends up being 5 or more bit pattern matches for each character. Persistence pays off though if you get one of those files, just keep matching.

The line and word layout functions are pretty sophisticated and should give good results unless the characters are very unusual (vertical or upside-down text is bad).

[nibus]

Very nice, I'll give this a shot. Can it export to .srt?

[Tappen]

Yes it can also export to srt, though of course that's a much more limited format (no colors, positioning, etc).

Also, the first 3 steps of the wizard are kind of like a easier to use version of ifoedit: they produce an mpg (mpeg-2 program stream) file of just the angles and tracks from the dvd you want to re-encode.

[nautilus7]

Do italic letters work correctly with .srt output? I tested one .sup file but I didn't have any success. I'll test more tommorrow.

[nibus]

I ran Ice Age 3 through it and I must say, it was painless. Worked extremely quick and I can't find any OCR errors. This is definitely my favorite subtitle OCR utility! Well done!

A few ideas -

1) Being able to type the text instead of clicking it would be nice, but not a huge deal as the recognition is excellent.

2) My default "save" directory was in the "My Videos" folder. It would probably be easier if it defaulted to the current working directory.

3) The other issue is on some subtitles the alignment is a little off. Not a huge deal - but it would be nice if there was a feature that allowed you to "align" text blocks to the same left-side position.

Here's an example:

http://dl.dropbox.com/u/5637223/ia3.jpg

edit: also the ability to set the OCR bin file to the program directory for portable use.

[nautilus7]

Do italic letters work correctly with .srt output? I tested one .sup file but I didn't have any success. I'll test more tommorrow.
Tested one more file. Both .srt files created with your application don't contain italic formatting. The <i> and </i> tags are omitted.

Also both .ass files can't be loaded in aegisub. I get "error processing line: style: blah blah blah".

Samples: http://www.mediafire.com/?eh78xxcdoc9siw0

Finally: What about subtitles in other than English languages? How do I insert foreign letters?

[Tappen]

nautilus7: I'll look into your issues, thanks for the source files. srt output is a feature I didn't work on much so I'm not surprised I missed some things. Should be easy fixes though

nibus: good suggestions.

1. I've thought about adding a "enter matching text manually" textbox myself. Hopefully I can do it without messing up the flow of the ocr

2. I worry that the files will be installed in a directory where the user doesn't have write access without Windows bringing up a UAC dialog so I went with My Videos. Maybe I should check if the current directory is writable and make that the default if so.

3. I have an option to "Exactly Position every Line" when creating ass files which will turn off the processing that allows text which is centered and in the lower 3rd of the screen to use the default position of ass renderers. But that doesn't solve the left alignment problem. I could add a "left-align" checkbox but then all text would be left aligned and probably (since the source and dest will have different widths) make things look bad in a different way.

[Tappen]

nautilus7: your 2 issues should be fixed with 1007 release

[nautilus7]

Working like charm. thanks!

What about non-English languages?

[Tappen]

nibus:

I added the ability to enter the OCR character match manually in 1008. See how you like the UI.

I couldn't figure out how to deal with UAC in Windows Vista and 7 reliably to change the output and OcrMap directories to the current app directory when it's sensible to do so. If you install (copy) into a "Program Files" sub-directory for those operating systems Windows secretly moves files created by the program elsewhere - very hard for the user to find. So I haven't changed the default directories yet.

[Tappen]

nautilus7: certainly localization is a feature I want to add. I know how to do it, not too hard in .Net, but just haven't as yet. We'll see what the response looks like in a month or so.

[nautilus7]

Ok, I see. But let me say this: Your program is currently the only in development ocr program that can read blu-ray subs (the other is suprip but is dead) and from the 2nd public release it can output perfect english subs, at least, something that suprip is not able to do till now... So i think response will go high! :P

[mastrboy]

Ok, I see. But let me say this: Your program is currently the only in development ocr program that can read blu-ray subs (the other is suprip but is dead) and from the 2nd public release it can output perfect english subs, at least, something that suprip is not able to do till now... So i think response will go high! :P

http://www.nikse.dk/SubtitleEdit can also read SUP files, and is very much alive and still being developed on...

[nautilus7]

Nice! Wasn't aware of this. I'll have a look.

@Tappen a few suggestions:

Almost every time SubExtractor finds ." or ," letter combination in italic writing, it puts a space between them. Maybe some optimization can be done there so the user don't have to fix the space with the "advanced word spacing" feature.
Also some times an unwated space is placed after 1 (also in italic).

[Tappen]

The accuracy of the space detection depends on the font kerning, and so is different for every font the disc subtitle authors use. I haven't found . , or 1 characters in italics to have a lot of problems with the samples I've OCR'd, but it's fair to say that it's very rare for there to be a space in front of . or , and very common to have a space after the same, so maybe I'll tilt the base adjustments by 1 pixel in that direction.

How much are you having to "advanced word spacing" the left and right adjustments around those characters to fix the problem?

[nautilus7]

Hi, the samples I sent the other day demonstrate this problem. They both use arial font. The problems were fixed by moving 2 units (pixels?) IIRC.

[Tappen]

nautilus7 I don't see the problems when I run your .sup files. I don't see any extra spaces in front of periods or commas, or after 1s. Did you accidentally un-check "1080p Adjustments" on the Create Subtitles page? I notice that your *.ass files have the DVD (480p) default margins and font sizes instead of Bluray (doubled) values.

[nautilus7]

In eng.sup file i sent you, you can see the following:

http://img855.imageshack.us/img855/5789/63740798.th.png (http://img855.imageshack.us/i/63740798.png/)
http://img16.imageshack.us/img16/4419/20009265.th.png (http://img16.imageshack.us/i/20009265.png/)

Space between . and " in italic writing.
Space after 1 in italic writing.

In watchmen.dc.eng.sup file i sent you, you can see:

Space between . and " in italic writing.
Space between , and " in italic writing.

[Tappen]

I see. I think the problem is with " rather than . or , for the first issue. Not much I can do as many subtitle fonts (whatever the Bluray or DVD authors used to generate the bitmaps that I'm OCRing, not the font we're using in the output files) have tighter spacing around " and 1 italic characters than we're seeing here. Fixing your problem would probably break a bunch of other sup files. I'm just going to have to admit that I can't do perfect word spacing. Personally I usually run the subs I produce through the Aegisub spell checker to catch and fix any repeated errors. It would be great to be perfect but I don't think it's going to happen.

One thing I'm considering is that I've seen quite a few errors with numbers. I might auto-adjust the spacing rules so that 2 numbers next to each other can't have a space in between. It's a really visually jarring error that may be worth some extra work to avoid.

I've also considered a rule where I automatically add a space before the 1st, 3rd, etc. double-quotes, and remove any space after them, and do the reverse for the 2nd, 4th etc. double-quotes. But sometimes quotes don't work exactly like that - they're continued from the previous subtitle and the order is reversed. I'd hate to deliberately mess up those cases.

[Thunderbolt8]

would it be possible to change the order in which the different characters gets asked to be orc'ed sticks to horizontal lines?
e.g. when a subtitle consists of two or more lines, then all characters from the words of the first line are asked to be recognized first and only then characters from the next line.
atm, the program keeps going on a vertical axis and this is quite irritating.

also, the programm seems to halt when I choose a character from the windows character map which is not listed in your programm among those few characters presented on screen (it does not crash, but I cannot seems to proceed unless I undo that choice and choose one of those characters you suggest with your list; in this special case its the 'em dash')

[Tappen]

Thunderbolt8:

Thanks for the feedback. I'll put up a new version 1011 with a fix for the "non-standard character entered manually" problem shortly.

I agree that the order of the OCR is annoying. I've gone through over 1000 DVDs and maybe 100 Bluray Subs and gotten used to it but I still wish I could fix the issue. I just haven't come up with a simple solution programming-wise. It's strangely difficult to put characters into lines until you know what the characters are and I don't want to spend a lot of CPU and coding time on the issue.

[Thunderbolt8]

do the characters you have already OCRed actually get transfered over to be used for other subs as well? have only done 100 lines of one subtitle file so far which already took more than 1 hour. its really good that this program is so accurate, but if the results already achieved with some characters cannot be used again in case of other subtitle files, then it would be pretty useless, considering that it takes so much time. then it would be faster to use SupRead which is a little less accurate, but in the end you'd still be faster with using spell correction afterwards. so I hope that characters get stored in a kind of growing database.

[Tappen]

Yes the database is stored in the location shown at the bottom of the first tab of Options. It should be re-used even if you run a different version from a different location. Except for a few characters which commonly cause mistakes (Il1oO0.,'\/|-_) the matches you make are carried over between movies, based on the name of the sup file. The install also comes with the database I've built up from my own collection of DVDs and Blurays and initializes yours from it.

If it's taken an hour to do 100 lines you must have a really bad sup file. I've only seen this once myself with the subtitle tracks from "Master and Commander". Most movies will require 1 or 2 matches for each character and runs through the rest of the file quickly. But sometimes the Bluray authors have made the PGS subtitles by roughly scaling up bitmaps from a different resolution. This means there can be a dozen or more matches for each character. My really simplistic OCR is very poor at handling this and you end up clicking forever. Honestly I'd switch to SupRead or SubtitleEdit and try the file with them if you find you've got say 6 or more different "e" matches and no sign of the end.

I'd like to improve the OCR for Sup files to better handle these situations, matching the core of the character and ignoring minor changes in the edge pixels, but haven't found the time to do so yet.

[Thunderbolt8]

I guess Im just unlucky then with my subtitle file. sometimes, got even more than 6 different matches for a letter ;) but anyway, can only get better I guess.

since you said you've already done so many blu-rays and DVDs, I guess it could be very helpful if you could upload your character database for us. that would be huge help for everyone and we'd have even more reasons to use your tool ;)

p.s. it would be nice if you could add the 'em dash' character to the list of clickable characters so that you dont have to open the windows character map first. not sure if you want to start a new line, though. but thats definately one of the most common special characters, so it would make sense to throw something else out for it (like those brackets {} maybe, have never seen them being used anywhere)

[Thunderbolt8]

another thing, I hate all that SHD & hearing impaired stuff and would like to get rid of it. but automatic processing is rather complicated here, because often, you need to edit some (part of) lines manually to make adjustments afterwards.

but it would be nice if there was an option to delete lines which are completely put into brackets, either () or [], optionally followed by ":", at the stage of saving the file. those are safe to delete with automatic processing. e.g.

(laughing)
- blaaabla?
- blaa!

or

[man]:
blaaaaaaaa

since the complete top line is set into brackets, it would be rather easy to have it deleted without compromising the other lines. that would already save some more minutes in subtitleworkshop afterwards.

[Tappen]

My database is part of every install (the OcrMap.bin file). If I see you don't have a database at program startup I already copy mine so you never have to start from scratch. But if it's for one of those nasty scaled bitmap sup files the characters are going to be unique, sorry.

Excellent feature suggestion to optionally remove SHD and hearing-impaired text (identified by lines fully surrounded by brackets and optionally ending in : as you say). I'll add a checkbox to the "Create Subtitle File" step shortly.

[Tappen]

I didn't even realize there was such a thing as the 'em dash' character. I've seen it in my subtitles but I always just OCR'd it as a normal hythen. I'll see if I can add it to the list (and the auto-line-break list, meaning if it starts a line in the original subtitle I will put a hard line-break in the text before it so the renderer won't combine it with the previous line).

[nautilus7]

But if it's for one of those nasty scaled bitmap sup files the characters are going to be unique, sorry.Came across 2 subs where I had to manually input each character several time (more than 10). It seems that these come from very old bluray discs, some of the first that were out. I hadn't such issues with newest titles that i tried.

Excellent feature suggestion to optionally remove SHD and hearing-impaired text (identified by lines fully surrounded by brackets and optionally ending in : as you say). I'll add a checkbox to the "Create Subtitle File" step shortly.When a sub entry only contains SDH stuff, will it be removed completely with the timings and all?

A few suggestions related file naming and saving location:
Instead of having a predefined location for saving every sub, I would like to have an option to save the OCRed sub in the same location where the original bitmap sub is.
Also, when using .sup input, the output file name is "xxxxx T1 English Wide", which doesn't make much sense. It would better to use the same name as the input file.

[Tappen]

nautilus7:

Should removing SDH stuff remove it completely? What's your opinion? I could put the text in comments for ass files, but I don't think srt files like blank lines.

The naming issues you bring up are related to DVD subtitles. Your suggestions are good ones for sup files, which are treated as a special case of DVD subs in many parts of the code. I'll try to think how to add those options soon.

[Thunderbolt8]

Came across 2 subs where I had to manually input each character several time (more than 10). It seems that these come from very old bluray discs, some of the first that were out. I hadn't such issues with newest titles that i tried.the one I had problems with is a new BD though (nice mixture of italicized and normal characters)

Should removing SDH stuff remove it completely?if the complete line or all indivudual lines of a subtitle 'line' consist only of SHD stuff, then imho it should be completely removed and not only left as an empty line.

again another thing, the kind of subtitle I hate most are those which consist of 3 lines, SHD and different positions on screen. I generally try to change all my subs to 2 lines, centered and no SHD crap.
one very annoying thing of that subtitle type is when there are lines wich are being spoken by more than 1 person and therefore are adjusted to different locations at the screen. if those lines dont begin with a "-", then it will be quite irritating to find out that those two lines are actually spoken by different speakers, once both lines are centered and placed above each other, like a normal subtitle line consisting of two individual lines.
in such cases, I'd have to check each line of the original subtitle file manually and then add "-" characters manually to indicate the difference between speakers.

maybe it would be possible to have this prog mark those kind of line for me after ORCing, so that I would know which lines are affected by this and dont have to check all XXXX lines of the orignal .sup manually. I could go directly to these lines and see what I have to change. but I guess that would be hard to establish, because of the detection method. the tool would need to check the difference of space in the way those lines are placed to each other. in case of one line being at the left and the other at the right side of the screen, this seems rather easy to realize. but sometimes, those lines are placed almost directly above each other with no difference of space of their first character to the left side of the screen. the vertical distance between those lines is only minimal smaller, if not the same as that of how two normal lines following each other, belonging to the same speech part are placed (although in such a case, one of those lines would most likely be italicized, if their distance is that small, to give the viewer the same idea).

[nautilus7]

Tappen, I agree with Thunderbolt8, that you should remove the whole line when it contains only SDH subs.

Thunderbolt8, What you say about missing "-" in SDH is correct. In addition, a dialog in SDH subs can be seen like in the following example:

51
00:05:29,371 --> 00:05:31,540
[Man1]: I don't know any.
[Man2]: You don't?

In this case SubExtractor should be careful when removing the SDH parts and maybe putting the "-" automatically.

[Thunderbolt8]

that why I said in such a case better dont remove anything. such stuff can also be between two sentences in a middle of a line. and then adding a "-" would be wrong

e.g.

blaaaaa. (coughs) blaaaaaaa.

there are quite a few possibility in which SHD stuff occurs so Im only in favour of removing the idiot safe stuff when it fills a complete line

[Jaja1]

Awesome tool, works fast and accurate. Better than any tool I used so far. Thanks Tappen. Next phase: improved OCR of BD PGS subtitles ;)

SDH don't always have brackets, there are lots of them like this:
JOHN:
Blabla

Recognizable by the name being in capitalized characters followed by a colon.

There are however subtitle editors that remove SDH subs, so why should they be added to the OCR phase? Don't you spell- and OCR check the resulting srt or ass during which SDH will be removed as well? So, is there an advantage to do this during OCR?

[Thunderbolt8]

Im not in favour of this last change, because that can mess up stuff as well. e.g.

JOHN: bla
MAN: bla

would result in

bla
bla

while it should be

- bla
- bla

to indicate the difference in speakers. sometimes, even though this is rare, there could be a word and not a name capitalized & followed by ':' and it gets even more tricky if speakers names are not capitalized apart from the initial letter.

there are just too many exceptions. in the end, you'd have to check the original .sup file again manually, if something got removed which shouldnt get removed, because its either important or creates confusion.
therefore, I'd suggest to keep the automatic removal process as basic and safe as it can get. or at least Im for different steps in the automatic removal process, so that I could still maintain my basic removal setting, while others might want to risk to remove more.

[Jaja1]

and it gets even more tricky if speakers names are not capitalized apart from the initial letter.
I've seen confusing examples of this. Like "McDOWELL:Bla"

[Thunderbolt8]

it would be nice if there was a list of ALL characters ocred so far, not only those of the subtitle file you are currently working on. sometimes, mistakes might get transfered over and this would then be a possibility to notice and correct them.

[nautilus7]

Tappen, are you going to compile latest source so i can test new features?

[Tappen]

Thunderbolt8: It's a reasonable idea, but the list is so huge it's unmanageable to look at. One thing to note is that if you remove an OCR from the database for 1 movie, it's removed for all movies. So if you notice the problem once you'll fix it for good.

nautilus7: I've compiled a new release which includes several new features and fixes, including SDH removal (for SRT files only at this time).

My tests for what is SDH are:
Any text between () or [] inclusive and any spaces or dashes before or after
Any text before a : on a line and any spaces after same unless the characters immediately before and after the : are numbers (usually time values like 11:00) or there are both at least 1 space character and at least one lower-case character in the text to be removed (usually a true, non-SDH, colon in the subs).

[Thunderbolt8]

Thunderbolt8: It's a reasonable idea, but the list is so huge it's unmanageable to look at. One thing to note is that if you remove an OCR from the database for 1 movie, it's removed for all movies. So if you notice the problem once you'll fix it for good.but what if you only notice this mistake later on, when doing another movie? then you cant remove it any more and this could be really annoying, especially in case its an "I" and "l" mix up. takes really long to correct mistakes like 'Ionely', 'suddenIy' etc. in some files.

[nautilus7]

Thanks Tappen. I 've posted one small issue on the tracker.

In addition, there's a problem with subtitles in different languages. When a character is the same in two or more languages like "o", which language is selected? For example when i rip subs in greek "o"s are typed in english. The problem is visually nowhere, but when i later scan the subs in word for speelling errors, i get a whole bunch of them. Any way to deal with this? What i can think of, is a way (maybe ask for every sub) to tell subextractor what is the language of the sub. Knowing this would result in choosing (or automatically replacing) common characters with the ones that match the sub language. What do you think?

[Thunderbolt8]

I'd like to see the SDH option to be more refined. e.g. I atm I see myself being fine with the first one with () and [], but not with the 2nd one. a line like

man 1: blaa (or MAN 1: or namex etc.)
man 2: bleep

gets changed to

blaa
bleep

while it should then be

- blaa
- bleep

to indicate 2 different speakers. therefore, I dont like to have the 2nd SHD type recognition being applied automatically as well with the first one, because then I'd have to look through the whole file again manually to find all such lines and the whole process becomes useless for me. I'd rather go through the file with seach for ":" and delete & change all such lines myself (if needed).

so it would be nice if you could differentiate between those SDH recognition features.

[Tappen]

Thunderbolt8: There's an l and I tab in Options to remove words that you've mistakenly mixed up. Remove the word in Options, rerun the sub, and choose the word you want during spell check.

[Tappen]

Thunderbolt8: if 2 or more lines on a single page of subtitles have had their starting characters removed during SDH removal, all lines with starting characters removed (or the line following if the entire line was SDH) will have a '-' prefix added. This is already in the code.

[Tappen]

nautilus7:

I don't think the issue you posted is actually a bug. Check out my comment and see if you agree.

o is a character that is only matched per subtitle. You should be able to specify new o matches for each subtitle. But when I say "per subtitle" this is based on the name of the sup file (with all numbers, dashes, spaces and symbols removed). So if you name all your sup files "sub1.sup", "sub2.sup", etc. then they'll share all OCR matches including l, o, -, etc.

I see that the name of your subtitle file is machine-generated. This will unfortunately result in all sup files of the same language being treated as a single file for OCR purposes. This rule exists so that multiple tracks, angles, programs from a single DVD or Bluray will share OCR, but I guess it causes problems as well.

Personally I put the name of the movie in my sup filenames, which is why I haven't seen the problem.

[nautilus7]

I see your point, but what about other characters that are common in many languages but subextractor doesn't asked for them? There are a lot of these like "k", "m", "n", "a", etc and of course the CAPITAL ones.


Personally I put the name of the movie in my sup filenames, which is why I haven't seen the problem. What about checking file size as well? There's almost zero possibility two subtitle files have the exact same size.

[Tappen]

But I want files of different sizes to use the l, I, o, etc. matches in common. As I said it's so different tracks or programs on the same disk, or a series of disks like a TV series, will share the full OCR.

I'm not clear on exactly what the problem is. Is it that you want to distinguish between k (Latin Western European alphabet) and k (Kappa)(Greek alphabet) and you want to manually type in the Kappa character because it isn't in the character selection box? Because that's a tricky problem if so.

[nautilus7]

But I want files of different sizes to use the l, I, o, etc. matches in common. As I said it's so different tracks or programs on the same disk, or a series of disks like a TV series, will share the full OCR.Al right then.

I'm not clear on exactly what the problem is. Is it that you want to distinguish between k (Latin Western European alphabet) and k (Kappa)(Greek alphabet) and you want to manually type in the Kappa character because it isn't in the character selection box? Because that's a tricky problem if so.Yes, I want to distinguish between two identical characters (that's oxymoron :p ) because i need sleeping check to work afterwards.

Not "k" in particular because they are a little bit different anyway (k vs κ). But often "v" (latin) is confused with "ν" (greek n) and similar examples. Also capital character like E, T, Y, A, H, K, M, N, B, X and Z are exactly the same in both alphabets.

If I don't distinguish between these characters, I get a lot of false errors in spelling check, which makes real error correction impossible.

[Thunderbolt8]

Thunderbolt8: if 2 or more lines on a single page of subtitles have had their starting characters removed during SDH removal, all lines with starting characters removed (or the line following if the entire line was SDH) will have a '-' prefix added. This is already in the code.
that doesnt seem to work in all cases though. Ive got a line here consisting of:

HAROLD: Doc, he's hit, too.
JUNIOR: Taylor's hit!

and with SDH removal ticked it gets changed to:

Doc, he's hit, too.
Taylor's hit!

missing out the "-"s at the beginning of the lines

[Tappen]

Thunderbolt8: I sent you a private message

nautilus7: That feature would be tough. You can swap out OcrMap.bin files yourself if you want to have English and Greek databases but I'll admit that would be cumbersome. It's hard to think of a user interface that wouldn't annoy 99% of users with unused options. I've been thinking that I'd support different character sets like Cyrillic, Greek and Arabic by swapping out the entire character selector list. It might make sense to separate the databases if the user chose a different character set. That's a pretty big feature for the future, though.

[nautilus7]

I see, that's why i thought an option to select the sub language would be easier to deal with. Thanks for your support anyway. ;)