SubExtractor - New Sub Ocr App

[Tappen]

I've released an app to extract subs from (non-encrypted, on hard drive) DVDs and convert to Advanced Substation Alpha or SRT format. It can also convert sup (PGS) and sub/idx formats to same. I wrote this because I hate the blocky, too-high-on-the-screen look of regular DVD subtitles and wanted to re-encode my DVD collection in h264/aac/assa with mkv containment.

http://subextractor.codeplex.com/

It's a wizard-style app, allowing you to pick program chains, angles, audio and subtitle tracks from a DVD folder and create mpg, d2v and bin (my own data format similar to sub/idx combined) files for each. DGIndex is used to help line up the subs to the video since DVD programs often have discontinuities that mess up sync. The mpg and d2v files created is great for further re-encoding of DVDs to h264 using a tool like MeGui.

The OCR is pretty basic, just exact pattern matching of the characters. The starting OCR database is good though so most DVDs should require manual matching of just a few characters. Some characters like i, l, I, '.', and o must be manually matched for every DVD since they have a lot of false positives. Some Bluray sup files can be tedious to OCR since the Bluray authors used scaled-up fonts, which means there ends up being 5 or more bit pattern matches for each character. Persistence pays off though if you get one of those files, just keep matching.

The line and word layout functions are pretty sophisticated and should give good results unless the characters are very unusual (vertical or upside-down text is bad).

[nibus]

Very nice, I'll give this a shot. Can it export to .srt?

[Tappen]

Yes it can also export to srt, though of course that's a much more limited format (no colors, positioning, etc).

Also, the first 3 steps of the wizard are kind of like a easier to use version of ifoedit: they produce an mpg (mpeg-2 program stream) file of just the angles and tracks from the dvd you want to re-encode.

[nautilus7]

Do italic letters work correctly with .srt output? I tested one .sup file but I didn't have any success. I'll test more tommorrow.

[nibus]

I ran Ice Age 3 through it and I must say, it was painless. Worked extremely quick and I can't find any OCR errors. This is definitely my favorite subtitle OCR utility! Well done!

A few ideas -

1) Being able to type the text instead of clicking it would be nice, but not a huge deal as the recognition is excellent.

2) My default "save" directory was in the "My Videos" folder. It would probably be easier if it defaulted to the current working directory.

3) The other issue is on some subtitles the alignment is a little off. Not a huge deal - but it would be nice if there was a feature that allowed you to "align" text blocks to the same left-side position.

Here's an example:

http://dl.dropbox.com/u/5637223/ia3.jpg

edit: also the ability to set the OCR bin file to the program directory for portable use.

[nautilus7]

Do italic letters work correctly with .srt output? I tested one .sup file but I didn't have any success. I'll test more tommorrow.
Tested one more file. Both .srt files created with your application don't contain italic formatting. The <i> and </i> tags are omitted.

Also both .ass files can't be loaded in aegisub. I get "error processing line: style: blah blah blah".

Samples: http://www.mediafire.com/?eh78xxcdoc9siw0

Finally: What about subtitles in other than English languages? How do I insert foreign letters?

[Tappen]

nautilus7: I'll look into your issues, thanks for the source files. srt output is a feature I didn't work on much so I'm not surprised I missed some things. Should be easy fixes though

nibus: good suggestions.

1. I've thought about adding a "enter matching text manually" textbox myself. Hopefully I can do it without messing up the flow of the ocr

2. I worry that the files will be installed in a directory where the user doesn't have write access without Windows bringing up a UAC dialog so I went with My Videos. Maybe I should check if the current directory is writable and make that the default if so.

3. I have an option to "Exactly Position every Line" when creating ass files which will turn off the processing that allows text which is centered and in the lower 3rd of the screen to use the default position of ass renderers. But that doesn't solve the left alignment problem. I could add a "left-align" checkbox but then all text would be left aligned and probably (since the source and dest will have different widths) make things look bad in a different way.

[Tappen]

nautilus7: your 2 issues should be fixed with 1007 release

[nautilus7]

Working like charm. thanks!

What about non-English languages?

[Tappen]

nibus:

I added the ability to enter the OCR character match manually in 1008. See how you like the UI.

I couldn't figure out how to deal with UAC in Windows Vista and 7 reliably to change the output and OcrMap directories to the current app directory when it's sensible to do so. If you install (copy) into a "Program Files" sub-directory for those operating systems Windows secretly moves files created by the program elsewhere - very hard for the user to find. So I haven't changed the default directories yet.

[Tappen]

nautilus7: certainly localization is a feature I want to add. I know how to do it, not too hard in .Net, but just haven't as yet. We'll see what the response looks like in a month or so.

[nautilus7]

Ok, I see. But let me say this: Your program is currently the only in development ocr program that can read blu-ray subs (the other is suprip but is dead) and from the 2nd public release it can output perfect english subs, at least, something that suprip is not able to do till now... So i think response will go high! :P

[mastrboy]

Ok, I see. But let me say this: Your program is currently the only in development ocr program that can read blu-ray subs (the other is suprip but is dead) and from the 2nd public release it can output perfect english subs, at least, something that suprip is not able to do till now... So i think response will go high! :P

http://www.nikse.dk/SubtitleEdit can also read SUP files, and is very much alive and still being developed on...

[nautilus7]

Nice! Wasn't aware of this. I'll have a look.

@Tappen a few suggestions:

Almost every time SubExtractor finds ." or ," letter combination in italic writing, it puts a space between them. Maybe some optimization can be done there so the user don't have to fix the space with the "advanced word spacing" feature.
Also some times an unwated space is placed after 1 (also in italic).

[Tappen]

The accuracy of the space detection depends on the font kerning, and so is different for every font the disc subtitle authors use. I haven't found . , or 1 characters in italics to have a lot of problems with the samples I've OCR'd, but it's fair to say that it's very rare for there to be a space in front of . or , and very common to have a space after the same, so maybe I'll tilt the base adjustments by 1 pixel in that direction.

How much are you having to "advanced word spacing" the left and right adjustments around those characters to fix the problem?

[nautilus7]

Hi, the samples I sent the other day demonstrate this problem. They both use arial font. The problems were fixed by moving 2 units (pixels?) IIRC.

[Tappen]

nautilus7 I don't see the problems when I run your .sup files. I don't see any extra spaces in front of periods or commas, or after 1s. Did you accidentally un-check "1080p Adjustments" on the Create Subtitles page? I notice that your *.ass files have the DVD (480p) default margins and font sizes instead of Bluray (doubled) values.

[nautilus7]

In eng.sup file i sent you, you can see the following:

http://img855.imageshack.us/img855/5789/63740798.th.png (http://img855.imageshack.us/i/63740798.png/)
http://img16.imageshack.us/img16/4419/20009265.th.png (http://img16.imageshack.us/i/20009265.png/)

Space between . and " in italic writing.
Space after 1 in italic writing.

In watchmen.dc.eng.sup file i sent you, you can see:

Space between . and " in italic writing.
Space between , and " in italic writing.

[Tappen]

I see. I think the problem is with " rather than . or , for the first issue. Not much I can do as many subtitle fonts (whatever the Bluray or DVD authors used to generate the bitmaps that I'm OCRing, not the font we're using in the output files) have tighter spacing around " and 1 italic characters than we're seeing here. Fixing your problem would probably break a bunch of other sup files. I'm just going to have to admit that I can't do perfect word spacing. Personally I usually run the subs I produce through the Aegisub spell checker to catch and fix any repeated errors. It would be great to be perfect but I don't think it's going to happen.

One thing I'm considering is that I've seen quite a few errors with numbers. I might auto-adjust the spacing rules so that 2 numbers next to each other can't have a space in between. It's a really visually jarring error that may be worth some extra work to avoid.

I've also considered a rule where I automatically add a space before the 1st, 3rd, etc. double-quotes, and remove any space after them, and do the reverse for the 2nd, 4th etc. double-quotes. But sometimes quotes don't work exactly like that - they're continued from the previous subtitle and the order is reversed. I'd hate to deliberately mess up those cases.

[Thunderbolt8]

would it be possible to change the order in which the different characters gets asked to be orc'ed sticks to horizontal lines?
e.g. when a subtitle consists of two or more lines, then all characters from the words of the first line are asked to be recognized first and only then characters from the next line.
atm, the program keeps going on a vertical axis and this is quite irritating.

also, the programm seems to halt when I choose a character from the windows character map which is not listed in your programm among those few characters presented on screen (it does not crash, but I cannot seems to proceed unless I undo that choice and choose one of those characters you suggest with your list; in this special case its the 'em dash')

[Tappen]

Thunderbolt8:

Thanks for the feedback. I'll put up a new version 1011 with a fix for the "non-standard character entered manually" problem shortly.

I agree that the order of the OCR is annoying. I've gone through over 1000 DVDs and maybe 100 Bluray Subs and gotten used to it but I still wish I could fix the issue. I just haven't come up with a simple solution programming-wise. It's strangely difficult to put characters into lines until you know what the characters are and I don't want to spend a lot of CPU and coding time on the issue.

[Thunderbolt8]

do the characters you have already OCRed actually get transfered over to be used for other subs as well? have only done 100 lines of one subtitle file so far which already took more than 1 hour. its really good that this program is so accurate, but if the results already achieved with some characters cannot be used again in case of other subtitle files, then it would be pretty useless, considering that it takes so much time. then it would be faster to use SupRead which is a little less accurate, but in the end you'd still be faster with using spell correction afterwards. so I hope that characters get stored in a kind of growing database.

[Tappen]

Yes the database is stored in the location shown at the bottom of the first tab of Options. It should be re-used even if you run a different version from a different location. Except for a few characters which commonly cause mistakes (Il1oO0.,'\/|-_) the matches you make are carried over between movies, based on the name of the sup file. The install also comes with the database I've built up from my own collection of DVDs and Blurays and initializes yours from it.

If it's taken an hour to do 100 lines you must have a really bad sup file. I've only seen this once myself with the subtitle tracks from "Master and Commander". Most movies will require 1 or 2 matches for each character and runs through the rest of the file quickly. But sometimes the Bluray authors have made the PGS subtitles by roughly scaling up bitmaps from a different resolution. This means there can be a dozen or more matches for each character. My really simplistic OCR is very poor at handling this and you end up clicking forever. Honestly I'd switch to SupRead or SubtitleEdit and try the file with them if you find you've got say 6 or more different "e" matches and no sign of the end.

I'd like to improve the OCR for Sup files to better handle these situations, matching the core of the character and ignoring minor changes in the edge pixels, but haven't found the time to do so yet.

[Thunderbolt8]

I guess Im just unlucky then with my subtitle file. sometimes, got even more than 6 different matches for a letter ;) but anyway, can only get better I guess.

since you said you've already done so many blu-rays and DVDs, I guess it could be very helpful if you could upload your character database for us. that would be huge help for everyone and we'd have even more reasons to use your tool ;)

p.s. it would be nice if you could add the 'em dash' character to the list of clickable characters so that you dont have to open the windows character map first. not sure if you want to start a new line, though. but thats definately one of the most common special characters, so it would make sense to throw something else out for it (like those brackets {} maybe, have never seen them being used anywhere)

[Thunderbolt8]

another thing, I hate all that SHD & hearing impaired stuff and would like to get rid of it. but automatic processing is rather complicated here, because often, you need to edit some (part of) lines manually to make adjustments afterwards.

but it would be nice if there was an option to delete lines which are completely put into brackets, either () or [], optionally followed by ":", at the stage of saving the file. those are safe to delete with automatic processing. e.g.

(laughing)
- blaaabla?
- blaa!

or

[man]:
blaaaaaaaa

since the complete top line is set into brackets, it would be rather easy to have it deleted without compromising the other lines. that would already save some more minutes in subtitleworkshop afterwards.

[Tappen]

My database is part of every install (the OcrMap.bin file). If I see you don't have a database at program startup I already copy mine so you never have to start from scratch. But if it's for one of those nasty scaled bitmap sup files the characters are going to be unique, sorry.

Excellent feature suggestion to optionally remove SHD and hearing-impaired text (identified by lines fully surrounded by brackets and optionally ending in : as you say). I'll add a checkbox to the "Create Subtitle File" step shortly.

[Tappen]

I didn't even realize there was such a thing as the 'em dash' character. I've seen it in my subtitles but I always just OCR'd it as a normal hythen. I'll see if I can add it to the list (and the auto-line-break list, meaning if it starts a line in the original subtitle I will put a hard line-break in the text before it so the renderer won't combine it with the previous line).

[nautilus7]

But if it's for one of those nasty scaled bitmap sup files the characters are going to be unique, sorry.Came across 2 subs where I had to manually input each character several time (more than 10). It seems that these come from very old bluray discs, some of the first that were out. I hadn't such issues with newest titles that i tried.

Excellent feature suggestion to optionally remove SHD and hearing-impaired text (identified by lines fully surrounded by brackets and optionally ending in : as you say). I'll add a checkbox to the "Create Subtitle File" step shortly.When a sub entry only contains SDH stuff, will it be removed completely with the timings and all?

A few suggestions related file naming and saving location:
Instead of having a predefined location for saving every sub, I would like to have an option to save the OCRed sub in the same location where the original bitmap sub is.
Also, when using .sup input, the output file name is "xxxxx T1 English Wide", which doesn't make much sense. It would better to use the same name as the input file.

[Tappen]

nautilus7:

Should removing SDH stuff remove it completely? What's your opinion? I could put the text in comments for ass files, but I don't think srt files like blank lines.

The naming issues you bring up are related to DVD subtitles. Your suggestions are good ones for sup files, which are treated as a special case of DVD subs in many parts of the code. I'll try to think how to add those options soon.

[Thunderbolt8]

Came across 2 subs where I had to manually input each character several time (more than 10). It seems that these come from very old bluray discs, some of the first that were out. I hadn't such issues with newest titles that i tried.the one I had problems with is a new BD though (nice mixture of italicized and normal characters)

Should removing SDH stuff remove it completely?if the complete line or all indivudual lines of a subtitle 'line' consist only of SHD stuff, then imho it should be completely removed and not only left as an empty line.

again another thing, the kind of subtitle I hate most are those which consist of 3 lines, SHD and different positions on screen. I generally try to change all my subs to 2 lines, centered and no SHD crap.
one very annoying thing of that subtitle type is when there are lines wich are being spoken by more than 1 person and therefore are adjusted to different locations at the screen. if those lines dont begin with a "-", then it will be quite irritating to find out that those two lines are actually spoken by different speakers, once both lines are centered and placed above each other, like a normal subtitle line consisting of two individual lines.
in such cases, I'd have to check each line of the original subtitle file manually and then add "-" characters manually to indicate the difference between speakers.

maybe it would be possible to have this prog mark those kind of line for me after ORCing, so that I would know which lines are affected by this and dont have to check all XXXX lines of the orignal .sup manually. I could go directly to these lines and see what I have to change. but I guess that would be hard to establish, because of the detection method. the tool would need to check the difference of space in the way those lines are placed to each other. in case of one line being at the left and the other at the right side of the screen, this seems rather easy to realize. but sometimes, those lines are placed almost directly above each other with no difference of space of their first character to the left side of the screen. the vertical distance between those lines is only minimal smaller, if not the same as that of how two normal lines following each other, belonging to the same speech part are placed (although in such a case, one of those lines would most likely be italicized, if their distance is that small, to give the viewer the same idea).

[nautilus7]

Tappen, I agree with Thunderbolt8, that you should remove the whole line when it contains only SDH subs.

Thunderbolt8, What you say about missing "-" in SDH is correct. In addition, a dialog in SDH subs can be seen like in the following example:

51
00:05:29,371 --> 00:05:31,540
[Man1]: I don't know any.
[Man2]: You don't?

In this case SubExtractor should be careful when removing the SDH parts and maybe putting the "-" automatically.

[Thunderbolt8]

that why I said in such a case better dont remove anything. such stuff can also be between two sentences in a middle of a line. and then adding a "-" would be wrong

e.g.

blaaaaa. (coughs) blaaaaaaa.

there are quite a few possibility in which SHD stuff occurs so Im only in favour of removing the idiot safe stuff when it fills a complete line

[Jaja1]

Awesome tool, works fast and accurate. Better than any tool I used so far. Thanks Tappen. Next phase: improved OCR of BD PGS subtitles ;)

SDH don't always have brackets, there are lots of them like this:
JOHN:
Blabla

Recognizable by the name being in capitalized characters followed by a colon.

There are however subtitle editors that remove SDH subs, so why should they be added to the OCR phase? Don't you spell- and OCR check the resulting srt or ass during which SDH will be removed as well? So, is there an advantage to do this during OCR?

[Thunderbolt8]

Im not in favour of this last change, because that can mess up stuff as well. e.g.

JOHN: bla
MAN: bla

would result in

bla
bla

while it should be

- bla
- bla

to indicate the difference in speakers. sometimes, even though this is rare, there could be a word and not a name capitalized & followed by ':' and it gets even more tricky if speakers names are not capitalized apart from the initial letter.

there are just too many exceptions. in the end, you'd have to check the original .sup file again manually, if something got removed which shouldnt get removed, because its either important or creates confusion.
therefore, I'd suggest to keep the automatic removal process as basic and safe as it can get. or at least Im for different steps in the automatic removal process, so that I could still maintain my basic removal setting, while others might want to risk to remove more.

[Jaja1]

and it gets even more tricky if speakers names are not capitalized apart from the initial letter.
I've seen confusing examples of this. Like "McDOWELL:Bla"

[Thunderbolt8]

it would be nice if there was a list of ALL characters ocred so far, not only those of the subtitle file you are currently working on. sometimes, mistakes might get transfered over and this would then be a possibility to notice and correct them.

[nautilus7]

Tappen, are you going to compile latest source so i can test new features?

[Tappen]

Thunderbolt8: It's a reasonable idea, but the list is so huge it's unmanageable to look at. One thing to note is that if you remove an OCR from the database for 1 movie, it's removed for all movies. So if you notice the problem once you'll fix it for good.

nautilus7: I've compiled a new release which includes several new features and fixes, including SDH removal (for SRT files only at this time).

My tests for what is SDH are:
Any text between () or [] inclusive and any spaces or dashes before or after
Any text before a : on a line and any spaces after same unless the characters immediately before and after the : are numbers (usually time values like 11:00) or there are both at least 1 space character and at least one lower-case character in the text to be removed (usually a true, non-SDH, colon in the subs).

[Thunderbolt8]

Thunderbolt8: It's a reasonable idea, but the list is so huge it's unmanageable to look at. One thing to note is that if you remove an OCR from the database for 1 movie, it's removed for all movies. So if you notice the problem once you'll fix it for good.but what if you only notice this mistake later on, when doing another movie? then you cant remove it any more and this could be really annoying, especially in case its an "I" and "l" mix up. takes really long to correct mistakes like 'Ionely', 'suddenIy' etc. in some files.

[nautilus7]

Thanks Tappen. I 've posted one small issue on the tracker.

In addition, there's a problem with subtitles in different languages. When a character is the same in two or more languages like "o", which language is selected? For example when i rip subs in greek "o"s are typed in english. The problem is visually nowhere, but when i later scan the subs in word for speelling errors, i get a whole bunch of them. Any way to deal with this? What i can think of, is a way (maybe ask for every sub) to tell subextractor what is the language of the sub. Knowing this would result in choosing (or automatically replacing) common characters with the ones that match the sub language. What do you think?

[Thunderbolt8]

I'd like to see the SDH option to be more refined. e.g. I atm I see myself being fine with the first one with () and [], but not with the 2nd one. a line like

man 1: blaa (or MAN 1: or namex etc.)
man 2: bleep

gets changed to

blaa
bleep

while it should then be

- blaa
- bleep

to indicate 2 different speakers. therefore, I dont like to have the 2nd SHD type recognition being applied automatically as well with the first one, because then I'd have to look through the whole file again manually to find all such lines and the whole process becomes useless for me. I'd rather go through the file with seach for ":" and delete & change all such lines myself (if needed).

so it would be nice if you could differentiate between those SDH recognition features.

[Tappen]

Thunderbolt8: There's an l and I tab in Options to remove words that you've mistakenly mixed up. Remove the word in Options, rerun the sub, and choose the word you want during spell check.

[Tappen]

Thunderbolt8: if 2 or more lines on a single page of subtitles have had their starting characters removed during SDH removal, all lines with starting characters removed (or the line following if the entire line was SDH) will have a '-' prefix added. This is already in the code.

[Tappen]

nautilus7:

I don't think the issue you posted is actually a bug. Check out my comment and see if you agree.

o is a character that is only matched per subtitle. You should be able to specify new o matches for each subtitle. But when I say "per subtitle" this is based on the name of the sup file (with all numbers, dashes, spaces and symbols removed). So if you name all your sup files "sub1.sup", "sub2.sup", etc. then they'll share all OCR matches including l, o, -, etc.

I see that the name of your subtitle file is machine-generated. This will unfortunately result in all sup files of the same language being treated as a single file for OCR purposes. This rule exists so that multiple tracks, angles, programs from a single DVD or Bluray will share OCR, but I guess it causes problems as well.

Personally I put the name of the movie in my sup filenames, which is why I haven't seen the problem.

[nautilus7]

I see your point, but what about other characters that are common in many languages but subextractor doesn't asked for them? There are a lot of these like "k", "m", "n", "a", etc and of course the CAPITAL ones.


Personally I put the name of the movie in my sup filenames, which is why I haven't seen the problem. What about checking file size as well? There's almost zero possibility two subtitle files have the exact same size.

[Tappen]

But I want files of different sizes to use the l, I, o, etc. matches in common. As I said it's so different tracks or programs on the same disk, or a series of disks like a TV series, will share the full OCR.

I'm not clear on exactly what the problem is. Is it that you want to distinguish between k (Latin Western European alphabet) and k (Kappa)(Greek alphabet) and you want to manually type in the Kappa character because it isn't in the character selection box? Because that's a tricky problem if so.

[nautilus7]

But I want files of different sizes to use the l, I, o, etc. matches in common. As I said it's so different tracks or programs on the same disk, or a series of disks like a TV series, will share the full OCR.Al right then.

I'm not clear on exactly what the problem is. Is it that you want to distinguish between k (Latin Western European alphabet) and k (Kappa)(Greek alphabet) and you want to manually type in the Kappa character because it isn't in the character selection box? Because that's a tricky problem if so.Yes, I want to distinguish between two identical characters (that's oxymoron :p ) because i need sleeping check to work afterwards.

Not "k" in particular because they are a little bit different anyway (k vs κ). But often "v" (latin) is confused with "ν" (greek n) and similar examples. Also capital character like E, T, Y, A, H, K, M, N, B, X and Z are exactly the same in both alphabets.

If I don't distinguish between these characters, I get a lot of false errors in spelling check, which makes real error correction impossible.

[Thunderbolt8]

Thunderbolt8: if 2 or more lines on a single page of subtitles have had their starting characters removed during SDH removal, all lines with starting characters removed (or the line following if the entire line was SDH) will have a '-' prefix added. This is already in the code.
that doesnt seem to work in all cases though. Ive got a line here consisting of:

HAROLD: Doc, he's hit, too.
JUNIOR: Taylor's hit!

and with SDH removal ticked it gets changed to:

Doc, he's hit, too.
Taylor's hit!

missing out the "-"s at the beginning of the lines

[Tappen]

Thunderbolt8: I sent you a private message

nautilus7: That feature would be tough. You can swap out OcrMap.bin files yourself if you want to have English and Greek databases but I'll admit that would be cumbersome. It's hard to think of a user interface that wouldn't annoy 99% of users with unused options. I've been thinking that I'd support different character sets like Cyrillic, Greek and Arabic by swapping out the entire character selector list. It might make sense to separate the databases if the user chose a different character set. That's a pretty big feature for the future, though.

[nautilus7]

I see, that's why i thought an option to select the sub language would be easier to deal with. Thanks for your support anyway. ;)

[Tappen]

Thunderbolt8: ignore my message. I found and fixed the problem you saw. Download 1014 and try it out.

[Thunderbolt8]

seems to work so far, at least with that same file I tested. thanks

[Thunderbolt8]

is it possible to adjust positions in case SHD stuff gets removed, but only for those subs which are 3 lines & have position all over the screen in .ass format? positioning looks a bit strange here after SHD removal as it seems

http://thumbnails37.imagebam.com/15334/0ce862153330427.jpg (http://www.imagebam.com/image/0ce862153330427)

Original:

OFFICER 3: Where'd you get
the beauty scar,
tough guy? Eating pussy?

so not only the first line would need to be fixed in positioning, also the 2nd line is a little bit off for some reason.

but as said, that only only apply to these kind of subs for which I dont have another choice than to keep them this way instead of converting to 2 centred lines .srt/.ass (unless I want to spent hours on that). not sure if thats possible

[Tappen]

I've been thinking that I need to do a little better with left-aligned blocks of text. Currently all positioned text in ASS format is centered (at the same point as the original is centered) so depending on what your font looks like compared to the font originally used to generate the SUP bitmaps the left edge will get ragged. Removing some SDH text from a line can make this a lot worse of course since there's less text centered on the same spot.

So a fix would be to identify left aligned blocks of text and create the ASS tags accordingly. Should be possible to implement pretty soon.

[Thunderbolt8]

it looks the same with 1080p adjustment unticked btw.

[aMvEL]

Very nice OCR-tool, Tappen ... It produces the best results of all the applications I've tested (For my use anyway..)

However, are there any way to batch-process idx/sub-ripping? As it is with my collection of TV-series, I use to rip all subtitles in vobsub-format for all episodes when I rip the episodes.
So when I want to convert them to .srt, it becomes a time-consuming process, especially when I need to open all single idx/sub-files and re-identify obscure characters, like I,l o and punctuations. I assume that is because when I choose a new idx-file it gets processed like a new movie.

It would be nice if I could open all idx/sub-files for a season of a series, and process them in the same ocr-process.

[Thunderbolt8]

^^that would be a good idea in case of series or different parts of movies which all use the same subs. for all 6 star wars parts, I had to identify more than 200 characters for the letter 'o' -.-

[Tappen]

aMvEL and Thunderbolt8:

If the input file name is similar it is treated as the same file and the problem characters (o, l, I, etc.) are not OCR'd again.
By similar I mean take the filename without extension, remove all symbols, numbers, hyphens and underscores, then make what's left lower-case.
For example "TV show Season 1 disc 2 epi 3" will match "TV Show Season 3 Disc 5 Epi 17"
So if you are reasonably consistent in file naming you won't have to re-OCR the problem characters.
You can also temporarily rename the files before inputting them (Starwars 1, Starwars 2, etc.) if you know it's going to be a pain and they use the same subtitle authoring fonts. Definitely easier than clicking 'o' 200 times.

I'll look into multi-selection of Sup/IDX files so you get a list. There's already a button on the "Create Subtitle File" page to "OCR Next Encoded Title" but it's only hooked up for multiple tracks on a single DVD that you've run through my parser to create my custom "bin" file format. Should be a pretty easy and useful feature to add.

The "1080p Adjustments" just doubles the size, border, shadow and margins of the font chosen for ASS file creation. Nothing else.

[Tappen]

aMvEL:

I support IDX/SUB format but I never use it myself for DVDs. When I do I get the Subs out of sync with video about 25% of the time.

The problem is the way DVDs are written. Often there are discontinuous sections (chapters, cells) of video and the gaps have to be estimated by whatever program is extracting the subtitles to try to match whatever a program like Handbrake does to combine the chapters or cells into a single stream.

On the other hand, if you run SubExtractor on a DVD from the beginning you'll get mpg/d2v/ac3 files which are ready for re-encoding in a program like MeGUI or Handbrake, but also an srt/ass file which syncs to the mpg file exactly because the same program is doing the tricky appending of cells for both video and subtitles. Getting this perfect was my main motivation for creating this program: I wanted to re-encode my 1000+ DVD collection and not have to hand-adjust every subtitle track.

[Tappen]

Version 1015 now left-aligns instead of center-aligning positioned text in ASS files, and allows multi-select on the Choose Subtitles step to allow batch processing.

Give it a try

[aMvEL]

The multi-selection of idx-files is working good, however I still have to re-ocr l/I/o and punctuation... which would imply that there still is a need for similar filenames?
I should think that when you choose "OCR next encoded title" you would automatically re-use all ocr from this subtitle/subtitle-selection?

Either way It's an improvement :)

[Tappen]

I'm worried that many people would use multi-selection just to load up a directory full of unrelated movies they want to OCR. The mess that happens when those problem characters are confused is almost impossible to sort out manually and you end up clearing the OCR database of a large number of matches for a bunch of movies. I could put up a dialog box asking if you want to treat all the files as sharing the same subtitle style and OCR dataset, but that's a question most people wouldn't understand.

Anyway, I think the reliability of the OCR is the best feature of SubExtractor, so I don't want to compromise it. I'll try to think of another way to solve this: one idea is to leave things as they stand but if I find, during OCR, that someone has matched 3 problem characters (alphabetic, not punctuation) in common with another movie I automatically pre-populate the OCR database with the rest of the matches from the other movie. That should cut down on the number of clicks needed per file by more than half.

[aMvEL]

Yes, I see how that could become a problem ....

You could add it as an option selectable for advanced users though, or something to that effect... But as you say, you shouldn't compromise the reliability of the OCR-process.

EDIT:
I seem to have mistaken a upper-case 'Z' for a lower-case 'z' somewhere during an OCR. How can I remove it from the dictionary, without deleting everything?

[Tappen]

Open one of the files that resulted in the problem and let the OCR run to completion normally. Then hit "Review and Correct OCR Matches" button in bottom right. Open the "OCR Training" drop-down list and select the Z that's the problem and hit "Remove a Training". If you're not sure which one it is delete all the Zs in the list one at a time. When you hit "Done" the OCR will restart and you can choose the correct matches this time around. This will fix it for all future movies (or ones you re-run) as well.

[Thunderbolt8]

in case of subtitles in which SHD stuff goes over two lines, those lines remain. this is not a problem, but when trying to search for such lines with the brackets symbols () and "exact position every line" is also ticked, then it will find those symbols in every line, because its part of the positioning.

so is there maybe another way to look for those () symbols in this situation? because having to look through 4000 lines manually could take quite a bit of time.

[Thunderbolt8]

sometimes, theres a strange mixup with I and l right in the middle of a sub. I and l are recognized fine, but up from a sudden point on, I is mistaken for l in many words (and also the other way round). deleting and rechecking all I and l orc'ed chars, theres no mistake to be noticed. Im wondering why this is. do I and l share the same character from what point on? why not before? or does is the same character being used for I and l? and why then the decision to do for I and not l?

[Tappen]

Thunderbolt8 Question 1: I haven't seen SDH stuff that goes over 2 lines. It might be possible to remove this text in SubExtractor by looking for an unmatched '(' or '[' on 1 line and an unmatched ')' or ']' on the next line but I'd need a sample to test with. I can't change the ASS tag syntax to help you with this though, \pos(x,y) is how it has to be.

Thunderbolt8 Question 2: I just assume I and l are the same character during OCR (because the problem is so common) and sort them out in the spell-checking step if there's any doubt. By doubt I mean unless it's in the middle or end of a word and there are other reliably lower-case characters on either side, in which case I can safely assume it's an 'l' and not an 'I'. If you have a bunch of words with l and I mixed up in the final output I'd guess there are some incorrect words in the "l and I Spelling" word list (3rd tab in Options). Can you find these words, remove them in Options, and re-run the OCR for the messed up subs? If there's still a problem please put a sample on a file sharing site and send me a link so I can find and fix the bug.

[Thunderbolt8]

found out that problem 1 is actually easy to solve, simply by searching for }( instead of just (

I have never used the spell checking option in this tool, does it work on a case to case basis or also like the ORCing process, that some words can be saved & wont turn up again next time?

[Tappen]

The spell checking is only there to fix the l vs. I problem. It's not at all a full spell-check with a real dictionary, so it tends to go very fast. You should be using it as part of every OCR, then do a real spell check with another program afterwards if you want. Basically all it does is ask which is the right spelling for words that have l or I in them where the choice isn't obvious. Typically that's just a word or 2 per movie since I've already entered over 1000 words in the list that the database starts with.

[Thunderbolt8]

but those spell checking changes I make at this stage for I & l get added to the database, yes? otherwise, I could do it in aegisub just as well.

[Tappen]

Yes, your choices get added to the database so it won't ask about the same word twice. Unlike Aegisub the l & I stage will auto-correct the words without stopping if they're in the database.

Also the "l & I" tab in Options is there if you make a mistake and need to correct it.

I also use Aegisub to spell-check afterwards. That's the reason I put the "Edit Subtitle File" button on the Create Subtitle page: if I find there's a consistent spacing error around one or more the characters I can close the Aegisub window, go to the "Advanced Word Spacing" page to tweak things, hit "Previous", re-Create the sub file and open Aegisub again for another try in just a couple of clicks.

I should mention there's 1 exception: the words Al and AI are both quite common so the choice you pick only applies for a single movie and doesn't go into the database. Just hope you don't get a movie about artificial intelligence that also has a guy named "Al" in it, haha.

[Thunderbolt8]

some examples doesnt seem to get picked up by the spellchecker though e.g. 'l'm --> I'm with ' at the beginning as signal of speech etc. and mispelled with l instead of I. Or 'lllogical or 'l've



got a subtitle example here which is

-C'mon!
- (Loudspeakers) 'Martinez.'

which gets changed to

- C'mon!
'Martinez.'

with SHD removal, missing out the '-' at the beginning of the 2nd line.


and a funny one:

- We're just gonna wheel right by 'em (!)
- We gonna try brother.

with SHD removal, the (!) get removed :D

[Tappen]

Spellchecker issues:

I should really drop the ' and spell-check the rest of the word normally. Fixed in 1016.

SDH issues:

The first two are already fixed - I found them in my own testing. Fixed in 1016.

The 3rd issue is what I'd expect. Actually I think in this case the (!) really is SDH text. It's signalling an emotional tone of voice that someone deaf or hard-of-hearing wouldn't catch. Maybe I should replace a (!) or (?) with a . (period) if there are other lines that end with a period?

[Thunderbolt8]

guess that would be ok

btw. what happens if a character is present twice, in your built in character map and also in that one stored in the user dir? because your file gets updated as well with every new version, so there is no internal comparison of those databases in this situation. which one is then being used in case a character is present in both databases?

[Tappen]

I only use my database if you don't have one already. There is no merging, just a file copy if I find you have no starting database at all, so you only get my database on a first install or if you deleted your database by hand.

1 thing - I messed up the initial 1016 release - forgot something. So re-download if you got it between 5 and 15 minutes before the time of this message. The real 1016 is attached to changeset 10586.

[loekverhees]

This is by far the best subtitle OCR program I've ever used! Thanks a lot Tappen! Though I found one thing that was quite annoying: I used the 'Manually Enter Character' feature because typing is much faster than clicking the right character. But every time I typed the correct character, I had to grab my mouse and click the 'Normal' button. This is very annoying. It would be much, much easier and faster if one was able to simply hit the Return key instead of clicking the 'Normal' button all the time. Normally, there are far less italic styled subtitles, so this shouldn't be a big problem (if one wants to enter italic characters, one just needs to use the mouse again to click the 'Italic' button). Maybe even better would be a check box that one can select if the characters are italic (because most of the time, there are multiple italic characters in a row and if the characters get normal again, one simply needs to uncheck that check box again).

[Thunderbolt8]

clicking the button or return key should be the same. if you have your right hand on your mouse which is hovering above the normal button, you only need to click

left hand for the corresponding letter, right hand does only do the clicks. same as pressing enter.

[Tappen]

loekverhees: When the keyboard focus is in the "Manual Enter" textbox I should be able to make whichever button was last chosen the "Windows Default" button, either Normal or Italic. That means it will have a thicker black border around it and be pressed automatically by Windows whenever the Enter key is hit without changing the focus. It's a good suggestion and I will look into it.

Thunderbolt8 is also right though: I can type characters with my left hand and keep my right hand on the mouse pretty fast. Personally, since I've used it so much, I have the character selector layout memorized and can play "whack-a-letter" very fast now, much faster than manual entry. But regardless, I'll look into adding that feature.

[Thunderbolt8]

what is the scale image option actually good for?

[Tappen]

All the subs in DVDs and Blurays are bitmaps in various forms, and they have different sizes. Sometimes a subtitle bitmap fills the whole video frame, with most of the pixels transparent, and sometimes it's just a small rectangle in one part of the video. Depends on the authoring tool the disc creator used.

With "Scale ..." unchecked, you see just the bitmap, scaled up or down to fill the orange window that takes up most of that step. The subs can look really distorted if their bounding rectangle is very small or has a strange aspect ratio.

When you check "Scale..." I show the subtitle as it would appear on the final video, at the correct x and y coordinates, and scaled as if the orange window was the full video size.

I prefer it checked. If there's something I think went wrong I like to compare what I see on the video player with what I see in SubExtractor in an apples to apples comparison. But it's up to you. The checked or unchecked state is remembered like an option, but doesn't change the OCR or final output steps in the slightest.

[Tappen]

loekverhees: I changed manual entry to work better with the Enter key in 1017. Give it a try and let me know what you think.

[loekverhees]

I have tried the 1.0.1.8 version and it works perfectly! Now I can OCR the subtitles really quick (as I'm used to type with 10 fingers). Thank you!

[sl1pkn07]

is possible make version for linux?

greetings

[Tappen]

I could try compiling it in Mono (the Linux C# development environment). Or someone who has experience in Mono would probably be able to do it pretty quickly. Anyone interested in helping send me a note.

Since Windows 7 came out I've sort of lost interest in Linux on the desktop and don't have a machine or even a virtual image of Ubuntu to build and test with.

[Thunderbolt8]

another small thing related to SHD removal:

(GUNSHOT)
LESTER: Lotte, no!

gets changed to

-Lotte, no!

while it should be just

Lotte, no!

because the first SHD line contains only SHD as sound indication, but no actual person speaking. so would it be able to add this kind of recognition as well or would this then break something with the other kind of SHD removal and '-' adding if someone indeed speaks, indicated by (), [] or : ?


and another thing:

Oh!
LOTTE: Oh, God.

gets changed to

Oh!
Oh, God.

while it should be

Oh!
- Oh, God.

I can see why the '-' is not added, because theres no such indication for the first line being another speaker. so the question is whether it would be able to add this case as well without breaking anything or if that is the case, maybe it could be considered just to add another '-' to the first line as well?

sample: http://www.mediafire.com/?zeda517adw3uz2d

[nautilus7]

while it should be

Oh!
- Oh, God.


Is this correct? I believe it should be:

- Oh!
- Oh, God.

to indicate 2 different persons are speaking.

[Thunderbolt8]

the above is indeed correct (at least according to the SUP file), because the first person spoke some more lines before this change in dialogue. so by watching that scene you'd be able to tell that the first line was another speaker, but maybe not from seeing that picture only. some subs do indeed have this kind of presentation. but if its not possible to differentiate between this and other cases of SHD removal and '-' adding (as said above, I edited my post), then maybe changing it automatically to the other case could be a solution.

[nautilus7]

Yes, I've seen this case your refer to, but I still think that the correct way to present 2 different speakers is to put a "-" for each one anyway.

[Tappen]

Certainly if a line is entirely removed it shouldn't be counted in considering whether there's more than 1 speaker. I'll fix that.

The 2nd case is more interesting. Does a line without any SDH text count as a different speaker is there's another line anywhere on the subtitle which does have some SDH and wasn't completely removed? I suspect it does.

I'll change the code and run some of the test cases and see what the results are before I check in a change.

[Thunderbolt8]

Yes, I've seen this case your refer to, but I still think that the correct way to present 2 different speakers is to put a "-" for each one anyway.
in general I wouldnt mind, but I dont agree on all cases

e.g. when the first line is spoken by an outside narrator and maybe also presented in italics then imho the look of it wouldnt be as fitting as without (cant seem to center the 2nd line here though)

...and the princess ran home as fast as she could
- Mother!

imho this fits better than

- ...and the princess ran home as fast as she could
- Mother!

because in this situation the narrator in not a person inside the story and the '-' expresses a more immediate presence to me.

[nautilus7]

Maybe you're right. I won't insist. It's a minor issue anyway.

[Thunderbolt8]

I think the most important thing is to keep things working. so if it can be implemented like this, fine. but if not, then I also wont mind to have it changed as suggested.

[Thunderbolt8]

moar: http://www.mediafire.com/?ugbgtacinxlx3b8


<i>- ♪ Is mighty chilly♪</i>
- [ Whimpering Continues ]

gets changed to

<i>- ♪ Is mighty chilly♪</i>

while it should be

<i>♪ Is mighty chilly♪</i>


;)

edit: might be the same case as in #85 though -.-

[Tappen]

<i>- ♪ Is mighty chilly♪</i>
- [ Whimpering Continues ]

gets changed to

<i>- ♪ Is mighty chilly♪</i>

while it should be

<i>♪ Is mighty chilly♪</i>


This just seems like the subtitle creators are trying to make my life hard. If there's a - in the original text I don't think I can reliably remove it without causing more problems than I solve.

[Thunderbolt8]

have you stored all those recent samples I uploaded?

when you have a sample for each different case we had so far and note down the line in which the typical feature of each sample occurs, then it should be rather easy to test all your samples after each change and you can see if anything breaks.

if there is anything you cannot implement without breaking other stuff, maybe its a good idea to collect all those cases with examples in a seperate post so that a user always knows what he has to look out for by himself if he reliably wants to get rid of SHD.

[Tappen]

I do have all the test cases. It's sort of amazing how you guys keep finding and documenting some pretty rare bugs in the SDH removal. When I'm having a particularly clear-headed day this week I'll try to improve the multiple speaker code. I also need to consider how much of the code is common between SRT and ASS creation, since positioning on the screen can indicate multiple speakers in ASS format.

Honestly I'm probably more interested in adding 2 new features at this point: 1. multiple character sets (Greek, Cyrillic, User Custom 1 and 2 is my first thought) along with multiple OCR databases and 2. localization (easy UI translation to other languages) to the program.

[nautilus7]

Looking forward for these. You can count me in for Greek translation.

[aMvEL]

I'm trying to OCR a vobsub, but I get an error: "No Subtitle Found".
It is working when using SubRip.

Example: http://www.mediafire.com/?bl3brwq58umv5be

[Tappen]

aMvEL: your idx file had an extra piece of data in every subtitle timestamp line that I wasn't parsing correctly (because I'd never seen it before). 1019 fixes the problem.

[Chetwood]

Yes, I've seen this case your refer to, but I still think that the correct way to present 2 different speakers is to put a "-" for each one anyway.
Apparently the professional sub studios see it differently. I've seen various official DVD subs lately that do make this distinction only on line 2 when the new person starts to talk. There however seems to be no consensus about whether a blank should follow the dash or not. Some subs have them, some don't.

[nautilus7]

I 've also seen professional studio subs that use 2 dashes to distinct each speaker in that case.

[Tappen]

I've seen hyphens, dash ems, and double-hyphens all used to separate speakers, and also sometimes, but it's not common, the first speaker doesn't have an indicator only the 2nd and any others down the screen.

It's also common for speakers to be separated by positioning of the subtitles on the screen, so this is much more of an SRT conversion issue than ASS conversion where positioning can be preserved.

Interesting problems. I think I'm going to try to preserve the choices of the original subtitle authors where possible, but use a single hyphen the rest of the time to indicate multiple speakers after SDH removal. No one will be perfectly happy but things should work out pretty well.

[Thunderbolt8]

but use a single hyphen the rest of the time to indicate multiple speakers after SDH removal. No one will be perfectly happy but things should work out pretty well.give an example please of how you mean to look it, not sure what you mean. imho the one hypen thing should only be used when its really one hyphen in the original subtitle layout, if possible. otherwise, the two hyphen thing for two lines we have so far is better imho. I find that doing your own one hyphen layout can sometimes look a bit strange compared to what the studios would do. they seem to have give it more thought when it seems logical to put the one hyphen or two or none when needed. that wouldnt be possible here.

[Tappen]

The current "Change of speaker prefix" I use is a single "-". All I'm saying is that I'm going to keep this the same. If there's a different prefix in the subs already I leave it alone.

[nautilus7]

version 1019 has some issues with SDH removal...

143
00:15:34,893 --> 00:15:37,062
[ALARM CONTINUES SCREECHING
IN DISTANCE]


isn't remove at all, probably because the brackets are not in the same line.

[Thunderbolt8]

that is the same for all versions so far. tappen said he hasnt thought of a way to remove SHD stuff which goes over two lines (without being sure to break anything)

[nautilus7]

Ah, ok, missed that.

[mindbomb]

neat.

[Thunderbolt8]

would it be possible to block that the '-' hyphens get added in the course of SHD removal when .ass and exactly position every line output both are ticked?

because then the different positions of lines of different speakers on screen already indicates that there is more than 1 person speaking at the moment. the additional hyphens are then superfluous and look strange with that kind of subtitles (the situation I am referring to are those subtitles containing of up to 3 lines which are positioned like everywhere on screen)

if others have a different opinion on this, then at least having the option for this would be nice

[Thunderbolt8]

http://www.mediafire.com/?yujdlgn86d157uk

imho it would be useful to implement that lines which begin with '--' dont get the additional hyphen added in case of SHD removal. currently

MAN 1 [ON RADIO]:
<i>--supported Senator Eagleman.</i>

gets changed to

<i>---supported Senator Eagleman.</i>

while that 2nd line would look just fine the way it was:

<i>--supported Senator Eagleman.</i>



now I am not sure, are there cases in which another speaker is indicated first, like

man 1: blabla
<i>--supported Senator Eagleman.</i>

which normally would get changed to (? just a guess, havent seen such a case yet)

- blabla
<i>- --supported Senator Eagleman.</i>

but that looks somehow strange. after the proposed change, it would look like

- blabla
<i>--supported Senator Eagleman.</i>

would look strange a bit as well. but maybe that situation doesnt really occur? at least as far as I can remember, usually a -- at the beginning of a line is only used in combination of other lines if they dont contain a hyphen at the beginning. I might be wrong though.

[Thunderbolt8]

http://www.mediafire.com/?zz19kis7hcfcxe7

some inconsistency:

(HORN HONKING)
GIRL: Hi, John.

gets changed to

-Hi, John. (hyphen superfluous, but we know that problem already)

or

GIRL: <i>Can I wizz</i>
<i>on you, Wolfman?</i>
(SOFT ROMANTIC SONG PLAYING)

gets changed to

<i>-Can I wizz</i>
<i>on you, Wolfman?</i> (SRT, 1 hyphen)

{\an4\pos(377,737)}{\i1}-Can I wizz{\i0}
{\an4\pos(377,817)}{\i1}-on you, Wolfman?{\i0} (ASS, 2 hyphens; seems to depend on whether you tick exact position... or not, then the SHD line () gets inserted above that dialogue instead below)



while

Hi, John.
JOHN: Not too good, huh?

gets changed to

Hi, John.
Not too good, huh?

instead of

Hi, John.
- Not too good, huh?

here the hyphen is actually missing (which wouldnt be bad for .ass in combination with exact position of every line as proposed some posts ago, but bad for .srt and/or when not using exact position of every line)

[Chetwood]

Why change that at all and not simple convert what's in there?

Also some more issues/feature requests:

I'm having trouble OCRing the word "figures" in italics. I can only split the word in 2 instead of 3 parts and I'm not automatically asked for a second split.

http://img97.imageshack.us/img97/1424/splitbj.th.png (http://imageshack.us/photo/my-images/97/splitbj.png/)


I can't save SRTs ripped from Vobsubs elsewhere but to c:\Users\Chetwood\videos. "Store Sup File Outputs in Source Directory" only applies to SUP but not Vobsub?


I can't change the "OCR data file location"


please add an option to save to ANSI instead of UTF (a lot of standalones have problems with the latter)


Bein able to drag and drop a sub file onto the program window would be cool


Thanks!

[Thunderbolt8]

Why change that at all and not simple convert what's in there?because these problems occur in combination with SHD removal.

[Chetwood]

Right. Apparently overread this. I'm still annoyed though, that the authoring people just don't add seperate stream for this. Should be piece of cake for them.

[Tappen]

Chetwood:

Just split multiple times if you have to. So in the case you shared highlight the "dot" of the i and complete split, start split again and get the bottom of the i. (Sorry I'm making people think like a programmer rather than a normal human in this case but my long estimate of the time it'd take to code it to work the proper way makes it a low priority issue)

I'll make the option to save in same directory apply to sup and idx/sub next release.

Sorry I don't yet allow moving the OcrMap.bin file location. I show the location so people can back up or move it to new machines manually. I'm thinking about how to allow this to be changed safely. Different versions of Windows have really different default program data locations and security around writing files. I don't want to spend a lot of time on error handling on such a minor feature.

Good idea to allow option to save to ANSI SRT as well as UTF. I'll try to add it soon. Whatever ANSI codepage the Windows UI Culture is currently running should be ok.

Drag'n'drop files. Yeah maybe.

[Tappen]

Thunderbolt8: I'll look into the SDH errors soon. One thing I'm definitely going to change is to remove the added hyphens when saving to ASS if the 2 lines aren't part of the same block (in terms of position on the screen).

[Thunderbolt8]

how do you plan to find out whether the 2 lines are part of the same block? by distance of letters and lines?

usually, even when 2 people are standing next to each other, theres always enough space to indicate 2 different speaker. but sometimes, when for example one person is speaking from the off or maybe standing behind another speaker, it can happen that those 2 or 3 lines of speech on screen are quite close to each other that its easy to mistake all those lines belonging to a single speaker. but in such cases the lines of one speaker are often differentiated from the other speaker by being italicized. so maybe italics can also be a criteria to distinguish in these situations when determining whether lines belong to the same block and narrowing the criteria of distance down too far wouldnt be of help.

[Tappen]

I already break characters into rectangular blocks and OCR them separately in the code. The rule is something like "within 4 normal character's width left or right or 2 normal character's height up or down means it's in the same block".

If there's an error, you'll see an extra couple of hyphens occasionally. Add too many rules and it'll just make the code unfixable AND unreliable. So we'll go with what I've already got for blocks.

[Chetwood]

Just split multiple times if you have to. So in the case you shared highlight the "dot" of the i and complete split, start split again and get the bottom of the i. (Sorry I'm making people think like a programmer rather than a normal human in this case but my long estimate of the time it'd take to code it to work the proper way makes it a low priority issue)
MMh, gonna retry this next time it occurs. IIRC splitting it once made the item not appear again so another split was impossible. Same goes for three letters "erj" recognized as one.

I get it that you want to minimize any potential troubleshooting for users but I'd really appreciate being able to select the OcrMap.bin file location myself. I don't trust 'c:\users' or 'My documents' so I put all important files into a folder that I backup regularly. Maybe you could pop up a short message ("all changes at your own risk!") when someone tries to deviate from the default location and be done with it.

BTW, I had another char not recognized, it had low double quotes Germans often use and looked like this: ,,e''. I had to manually fix it cause Subextractor would not accept it. Thanks again.

[Tappen]

There's an automated attempt to split every unknown character: if there's a perfect split SubExtractor won't stop and ask, it'll just do it. So sometimes even 3 sections joined together require only 1 split.

I can add the low double quotes to the character selection box if it's a common occurrence in German. I think there's an empty spot right now. Let me see if I can find the unicode character point. I'll have to make it work like double quotes I guess.

[Tappen]

1.0.2.0 is out with some fixes for Chetwood and Thunderbolt8

[Thunderbolt8]

thanks

[Thunderbolt8]

is there anything you can do to improve character recognition with this file here? http://www.mediafire.com/?75hlrz4g7dmzsz9

its the wort one I've ever encountered (and also the first one using this program), so far Im barely half through with it and already have got about 150 different characters for 'o' :/

[Tappen]

I've seen a few of those myself, and just gave up and used another program. I have some ideas on how to improve the OCR (it's issue #1 on codeplex) to allow for slight variations on the letters but haven't implemented it yet. It'll probably take a few months (of my spare time) to do.

[Thunderbolt8]

also resorted to another program to do them, but then I thought maybe it was nice to have those letters in my character library in case a similar BD turns up and I hopefully might be able to use some chars from that.

[nautilus7]

Thanks for the new version.

Tappen, would you consider adding basic subtitle syncing capabilities like framerate change and time delay to both bitmap and text based formats?

[Tappen]

Well, there already is a "Subtitle Offset (ms)" field on the Create Subtitle step which is the time delay you're looking for.

I have a 25->24fps conversion already in the code but only visible in Debug builds since I thought only I would use it. Are you looking for a generic Numerator/Denominator type framerate change option?

[nautilus7]

I've seen the offset you refer to, but i was thinking for a more flexible functionality like synchronizing already ocr'ed subtitles. Or just synchronize pgs subs without orc'ing them.

If by "generic Numerator/Denominator type framerate change option" you mean the user will be able to type whatever frame rate he wants, something like this would be great:

http://i40.tinypic.com/9t0h91.png

[Tappen]

I'm going to try to improve the OCR as my first priority. I might be able to put in a framerate conversion quickly before that though. Synchronizing functions will have to wait till after the new OCR and alternate character sets are in.

[Chetwood]

1.0.2.0 is out with some fixes for Chetwood and Thunderbolt8
As my usual way of sayin thanks I'm gonna add some more feature requests ;)

Please make your tool remember folders! When I rip subs with VSrip, I keep English and German in one Vobsub. When I open it in Subextractor with some files I get the message "Subtitle file out of date or corrupted" (BTW, is there a way to copy the whole text of such an error message?). Subextractor seems to be picky cause playback of these Vobsubs is just fine and it also opens in Subtitle Creator, Vobsubstrip, etc.

Anyway, I split the sub to de/en and now I can open it in Subextrator. However, when one language is finished I have to click all the way down to the folder to open the next file. Having Subextractor remember the folder would be nice.

Also, I'm not quite sure about where do you save the OCR results when I correct or enter any chars? Like I said, I'm ripping two langauges from the same TV show, both of which have the same font and colours. So if I enter some chars not automatically recognized like , o l O how come I have to enter them again when doing the second language? In case you need some demo files just let me know.

[Thunderbolt8]

afaik unless the names of the subtitle file are similar and you open both during the same session youd have to enter those letters again.

[Tappen]

Chetwood: I don't support multiple streams in 1 Vobsub file because I've never seen what the file format looks like. I'll add support for it. There's just a lot I still don't know about subtitles.

Also, I set the starting directory to the default output directory for the open file dialog. I should probably only do that the first time - after that I'll let the system remember the last location.

[Chetwood]

Nice. In case you wanna create some test file just reauthor some DVD in DVD Shrink. Please remember that you do need to select all subtitle streams, cause otherwise the stream order will be messed up in tools like VSrip.

[Thunderbolt8]

got another thingie: (tree.sup)

unmodified .ass output (actually before and after looks the same here for .srt output, only that the 2nd line is already split into two separate lines there)


{\i1}- [Patrons Chattering]{\i0}\N- Hope it's not gonna take as long as last time.


gets changed in combination with SHD removal and exactly positioning of every line to:


{\i1}-{\i0}
- Hope it's not gonna take
-as long as last time.


while it should be ideally:


Hope it's not gonna take
as long as last time.


or maybe at least something which involves only 1 hyphen instead of that that one line gets split up as if there were different speakers. actually, one hyphen or 3 doesnt matter, they have to be removed manually afterwards anyway. so maybe the ideal solution is possible without breaking stuff - or not :P

[Tappen]

I'll look at that Thunderbolt8, but I'd rather get to work on other features right now than another round of SDH fixes.

[Thunderbolt8]

when clicking the start over with this movie button, it seems that all the characters OCRed during the same session get deleted. I did one movie and wanted to make the other subtitle track afterwards (only difference in filename was the tracknumber), but then stopped in between and changed my mind and clicked that button in case there were some mistakes, because I didnt feel to review the characters for that track I didnt want to do any more. when I checked back on the first track, I noticed that I had to enter every character again I already did during the first run.

so if this is really the case, it would be nice if it can be distinguished between recognized characters from different subtitle track during one session. otherwise, you either would waste time spending to check on the other track you might not feel doing any longer or discard the characters ocred of other tracks during the same session.

[Tappen]

The Start Over button does really do that. It should probably be called something different. If you want to change your mind in the middle of an OCR just click the Previous menu item to get out. The Start Over button is there in case you need to clear all the OCR matches for the movie for some reason, like your cat walked over the keyboard of something.

[Thunderbolt8]

that is what I mean, but it seems not only to delete all the matches for the current movie, but all the movies already recognized before during the whole session.

when I press the previous menu button before I finish ocring a movie, will the characters recognized up to then get deleted?

[Tappen]

Previous saves everything done so far on the movie, so that's what you want.

Restart clears all the OCR matches for the current movie, even if they were manually made in another movie and just were found again and used in the current movie by the OCR engine. I really should put a warning message when people press it. I should maybe even remove the button from the Release version of the product since it's hard to see where it'd be useful for anyone but a developer.

[Thunderbolt8]

so if I understand correctly that restart process could remove more characters for a movie than you actually ocred during a session, right? so if a movie i.e. consists of 20 characters done just now and another 20 already taken from your database, then all 40 chars will get deleted and not only those 20 you did this session?

[Tappen]

Correct.

[Thunderbolt8]

eh then I suggest to remove this button completely and have the program point to a better solution instead please :p

[Thunderbolt8]

got another one for your SHD list (actually similar to the last one) :D (.srt) http://www.mediafire.com/?ugnugrbvxue6uds


- There's close to $10,000. Where?
<i>- [Siren Wailing]</i>

get changed to:

- There's close to $10,000. Where?
<i>-</i>

when its supposed to be:

There's close to $10,000. Where?



and another strange one:

- Well, what can we do, Mother?
- I thought if you went and talked to him —
you know, another man.

here actually a wrong hyphen get added:

- Well, what can we do, Mother?
- I thought if you went and talked to him —
-you know, another man.

[Thunderbolt8]

another strange thing (.srt): http://www.mediafire.com/?9se3uk3cej12d0j


BOSUN: All hands, check equipment.
MAN: Let's go.

gets changed to

All hands, check equipment.
- Let's go.

even though it should be

- All hands, check equipment.
- Let's go.


did something break? in 1019 its still fine. maybe its only restricted to this single one file?
this one is a rather crucial for me, if this is the same for all those situations with other subtitles, I guess Ill revert to 1019 then for the time being.

[Tappen]

Things were just changed around quite a bit. Some things fixed and others broken.

- There's close to $10,000. Where?
<i>- [Siren Wailing]</i>
What kind of subtitle author puts a hyphen in front of a sound effect? Seems stupid, but I guess I have to take that possibility into account.

BOSUN: All hands, check equipment.
MAN: Let's go.
This last one is by design: I don't start adding hyphens at the start of lines until I'm sure there's a reason to going top to bottom (so no hyphen on the first line of a group). This avoids some problems though maybe looks a little funny to some people. But it's one of the standard ways to indicate different speakers.

[Thunderbolt8]

if possible, imho it should rather be with adding a hyphen. there are some subtitles which only or mostly work with a bottom hyphen, but as already said somewhere in this thread, this seems to work only well in context with spoken lines before, e.g. if there is a change of speaker or not. imho it could turn out a bit confusing when trying to do a 1-hyphen subtitle track by oneself.

I'd vote in favour of staying with the original choice of the subtitle track creators. meaning if there are people named with : at the beginning of each line, then there should be 2 hyphens. otherwise, they could have only made the SHD indication for one line.

BOSUN: All hands, check equipment.
MAN: Let's go.

should imho stay

- All hands, check equipment.
- Let's go.

while if the orignal SHD choice were

All hands, check equipment.
MAN: Let's go.

then it would be converted with one hyphen to

All hands, check equipment.
- Let's go.


btw. could you give an example what would break when doing the 2 hyphens again? maybe its only comparably minor stuff.

in general, I think its good to have a list which always tells and gives examples of which kind of lines create which problem at SHD removal. might be easier to have an overview what to look out for durinf SHD removal checking and also could help with discussion which situation causing problems can be regarded as comparably unimportant to others.

[Thunderbolt8]

another one, rather special as it seems: http://www.mediafire.com/?ic08cto87qi5gco

- Sal-adin.
<i>- (belches) </i>Gibberish.

-->

- Sal-adin.
<i>-</i>Gibberish.

seems to be right according to the SHD removal rules, but it would be nice if the hyphen could get moved out of removal and italicized field directly in front of the spoken part, because only the hyphen being italicised but not the rest of the line is rather weird.

so --->

- Sal-adin.
- Gibberish.

[Slasher]

Hi Tappen, I just wanted to say "thank you" for the wonderful app!

Also, I would like to see some minor changes/features in the upcoming versions:
* the ability to specify a path for the OCR data file (for example I want it to be the program path)
* the ability to "save as" the subtitles or to be able to change their name when saving
* the app should remember the path of the last opened file, it's better when having multiple subtitles to process
* resizeable window, it can make it easier for the user when doing the OCR
* mouseover tooltip help for the options not described in the vertical right help bar (or add this information to the help bar)

[nautilus7]

I agree with all the requests. Remembering the last file position would be a very nice addition and i was thinking to ask for this also.

[Chetwood]

Me too. That's why I already requested some of them. Reading previous posts of a thread can be helpful.

[Tappen]

All good ideas Slasher. I'll see how many I can implement next release.

As to when that will happen: had a forced 2 week break from development recently but should be back at it this weekend.

[Thunderbolt8]

would it be possible to adjust the spacing of the sentence final quotation mark " in combination with italics? I realise correct spacing is hard to achieve and there will always be some problems with some words (especially in case of italics), but at least in case with the sentence final quotation mark in combination with italics, very often if not almost always theres a space between the fullstop or the exclamation mark at the end of the sentence and the quotation mark, like:

blaa. "

since this happens so frequently, maybe its possible to fix this independently from making other changes? maybe something like a rule to bind a sentence final " always directly next to the last character of the line?

[Thunderbolt8]

heres another interesting one: http://www.mediafire.com/?ytequ63d67dvm3q


- Oh, Mr. Blume, this is
my chapel partner, Dirk Calloway.
- [ Guggenheim Whistles ]

-->

- Oh, Mr. Blume, this is
my chapel partner, Dirk Calloway.
-


not sure why both hyphens remain, whether its due to this being a 3 line centred subtitle type or because theres always a space between the SHD brackets [] and the first and last letter in between for this subtitle file.

[Tappen]

I just need to remove the entire line if all that's left is hyphens and whitespace. Same problem as you showed in your post on 12th November 2011, 12:36

[Tappen]

Slasher's list of features are in release 1021
Thunderbolt8's fixes are in as well, except the case where hyphen is differently italic from the text which follows

[nautilus7]

Thanks!

[aMvEL]

Is there anything to be done about spacing in italic? I've stumbled upon some subtitles where most of the sentences in italic are without spaces.
I'd like to be able to adjust the spacing only in italic-sentences during or after OCR, or any other solution really...

Except that minor issue, this app has really improved my OCR and workflow, nice job Tappen :)

[Tappen]

aMvEL: on the last page (Create Subtitles) there's a button that says "Adjust Word Spacing" which allows you to adjust the adjustments made around characters. If you select the Italics button you'll only be changing the italics spacing. Once you select italics, then the character, you can add (if you're missing spaces) or subtract from the default adjustment to the left and right of various characters.

However, if you're missing all spaces, the problem isn't with the individual character adjustments. The program creates a histogram of the size of the spaces between characters (2 histograms, one normal and one italic) in pixels and tries to find 2 peaks in the graph. The first peak is when the characters are next to each other, the 2nd peak is when they are separated by a space. Then it runs through the OCR'd lines and inserts spaces where the separation is closer to the 2nd peak than the first. I've tweaked this algorithm a number of times, but when the sample size is small (italics, typically) there can still be problems accurately finding the 2nd peak. Can you put up a sample file which shows the problem somewhere for me to look at?

[aMvEL]

Actually I somehow hadn't noticed that button on the program, so I'm a little embarrassed right now...:P
I've gotten it tweaked now to fix almost all my space-related errors... I run it through a spell-check afterwards to fix the remaining errors.

[Thunderbolt8]

thanks.

any chance to implement the fix for the sentence final " in case of italics thing in the future? because that one not being wrong is rather the exception than the rule.

[Tappen]

I'm thinking about implementing a rule, something like: If there are an even number of quotation marks in the lines of a block of text, remove any spaces after the odd numbered ones and before the even numbered ones. Seems safe enough. I think it would fix many more mistakes than it creates.

Thunderbolt8: could you try the new release I've just put out - named "Manual Entry Improvement Test". It also has a slight change in italic double quote spacing, so see if it fixes your problem. It's at http://subextractor.codeplex.com/releases/view/77807

[Thunderbolt8]

looks good so far. with a track in which every single final italicised " was wrong, each them is correct now.

will report back in case I should encounter problems

[Tappen]

If anyone else reading this wants to try out the new Manual Entry user interface in the test release (http://subextractor.codeplex.com/releases/view/77807) go ahead and let me know how it feels.

Basically you can select the OCR match just by typing the character - you don't have to hit the Enter key any longer - and can switch to Italic and back just by a quick tap (press and release) of either Alt key. I find it's much faster than clicking on the glyph in the Character Selector box with the mouse.

[Thunderbolt8]

one thing I noticed: the letters newly added with this version are bigger in the ORC training review table than all the others. this is quite irritating, especially because before you had the possibility to scroll fast through the whole list and spot any mistake, because then that one different identified character stick outs notably, even when scrolling very fast. that is not possible now, because all these new additions with a different size create an obstacle for the eyes now. spoting mistakes would take way longer that way.

is there any way to revert this back to normal or to have all characters of the same letter with the same size again?

[Thunderbolt8]

that one is fixed:

- Rushmore.
- [ Whispers ]
- Shh.

-->

- Rushmore.
- Shh.


but that one still remains:


- Oh, Mr. Blume, this is
my chapel partner, Dirk Calloway.
- [ Guggenheim Whistles ]

-->

- Oh, Mr. Blume, this is
my chapel partner, Dirk Calloway.
-


similar/same as this one:


- But the fools first.
<i>- [Seagull Squawking]</i>

-->

- But the fools first.


just saying, because you said only that one hyphen italic thing remained unfixed. not sure if you covered this one somewhere, am losing track with so much stuff :p and its definately getting less :D

[Chetwood]

go ahead and let me know how it feels.
I love it! What I would like though, is an additional notifier that italics is ON in the window where the current item is listed. Maybe atop the window or better yet, have the usual black frame around the window turn blue when italics is on. Cause now that I don't have to use the mouse anymore, I can focus on that window alone. Having to look down below all chars just to see whether the italic checkbox is checked or not, takes time.

http://img528.imageshack.us/img528/2104/italicson.th.png (http://imageshack.us/photo/my-images/528/italicson.png/)

[Tappen]

Thunderbolt8: I'm not getting that error. In the sup file you uploaded for srt with SDH removed I see

47
00:04:05,622 --> 00:04:07,081
- Thank you.
- Hello.

48
00:04:07,165 --> 00:04:10,668
- Oh, Mr. Blume, this is
my chapel partner, Dirk Calloway.

49
00:04:10,752 --> 00:04:13,087
Nice to meet you, Dirk.

And for ASS I get:

Dialogue: 0,0:04:05.62,0:04:07.08,Dialogue1,Unknown,0000,0000,0000,,- Thank you.\N- Hello.
Dialogue: 0,0:04:07.16,0:04:10.66,Dialogue1,Unknown,0000,0000,0000,,- Oh, Mr. Blume, this is my chapel partner, Dirk Calloway.
Dialogue: 0,0:04:10.75,0:04:13.08,Dialogue1,Unknown,0000,0000,0000,,Nice to meet you, Dirk.

Neither of which has the extra line with the lonely hyphen that you list. What am I doing wrong?

[Thunderbolt8]

sorry, I copy pasted that wrong,

- Oh, Mr. Blume, this is
my chapel partner, Dirk Calloway.
-

the last hyphen is actually not there any more. but anyway, what I meant by posting this is that the other hyphen is still there :p not sure if its safe to fix, though, since both lines belong to the same speaker.

[Tappen]

Ah I see the issue: I never remove hyphens, only add them. I don't think I'll change that behavior for now.

[Tappen]

one thing I noticed: the letters newly added with this version are bigger in the ORC training review table than all the others. this is quite irritating, especially because before you had the possibility to scroll fast through the whole list and spot any mistake, because then that one different identified character stick outs notably, even when scrolling very fast. that is not possible now, because all these new additions with a different size create an obstacle for the eyes now. spoting mistakes would take way longer that way.

is there any way to revert this back to normal or to have all characters of the same letter with the same size again?

I don't see this. I added some new OCR trainings from an sup file I was part-way through and they all look about the same size in the OCR review table. I checked the code and there's been no change in that dialog except for adding the "Remove all trainings for this character" button since July.

[Tappen]

I love it! What I would like though, is an additional notifier that italics is ON in the window where the current item is listed. Maybe atop the window or better yet, have the usual black frame around the window turn blue when italics is on. Cause now that I don't have to use the mouse anymore, I can focus on that window alone. Having to look down below all chars just to see whether the italic checkbox is checked or not, takes time.


A 2nd test build of the new Manual UI: http://subextractor.codeplex.com/releases/view/77913 with an indicator to the right of the Ocr box. Also now the Backspace key works as Undo.

[Thunderbolt8]

I don't see this. I added some new OCR trainings from an sup file I was part-way through and they all look about the same size in the OCR review table. I checked the code and there's been no change in that dialog except for adding the "Remove all trainings for this character" button since July.

http://thumbnails38.imagebam.com/16231/3e905c162301999.jpg (http://www.imagebam.com/image/3e905c162301999)

http://thumbnails46.imagebam.com/16231/14a769162302012.jpg (http://www.imagebam.com/image/14a769162302012)

here, in pic 1 one B is bigger than the others and in pic 2, two Cs. thats quite a bit irritation when scrolling fast through the list spotting for mistakes.

(havent checked small case letters or italics)

[Tappen]

I think it's just that your subtitle file uses two different sized fonts. Often signs are drawn differently from dialogue. The characters on the left side of the list aren't scaled: they're exactly as I found them in the Sup or Sub bitmaps. I just draw the items in the drop-down list at whatever size I find them so you can see the exact pixel pattern.

With DVDs the difference can be huge - some text is twice or 3x as big as others - but if you've only been working with Sup files you might have gotten the impression that the subtitle authors only use one font per movie since that's pretty common on Bluray.

On another note - I had some good success yesterday with a new "fuzzy logic" OCR method. It seems to match 80-95% of the characters that are now treated as different because they've got a few different pixels on the edges. Hopefully by Monday I can finally solve the problem of Sup files where you have to make 1000s of OCR matches because the characters have been stretched or shrunk during Bluray authoring.

[Chetwood]

Thanks for implementing this, Tappen but I'm still in favour of a differently coloured bar around the window or something. Cause the way it is right now, I can only see the notifier in the corner of my eye and |||| is quite similar to |||| (<- should be italics).

[Tappen]

Well this is why I didn't release this feature immediately. Thanks for the feedback Chetwood, I'll think about it some more.

[Thunderbolt8]

has that sencence final " in case of italics change been (accidentally) reverted again in 1021c? all the " at the end of italicised sentences have a space between them and the last letter of the sentence again.

[Chetwood]

Some more observations on 1.0.2.1c:

To enter [ and ] on a German keyboard you gotta press ALTGr (which is right of the space bar) plus 8, 9 respectively. This of course interferes with ALT for toggling italics. How about using space bar as the toggle instead?


When reviewing and correcting OCR Matches I see all chars listed, but not [ or ], despite me having entered them in this particular session.


I got an 'Character Matched but Off-Baseline when trying to enter a question mark in italics where only the upper half was recognized. Clicking on the dot and then typing ? did not work. I had to ignore the char. (Weird, after deleting OcrMapOrig.bin and doing the whole sub from scratch, the ? was recognized properly)


IIRC ASS subs allow for vertical positioning with the MarginV style tag. Would be nice if SubExtractor would translate the positions from VobSubs so credits do not overlap with subs.


How do I delete all trainings for a sub, wasn't it possible pre 1.0.21? When checking the finished sub, I realized several errors so I openend the original VobSub again (selecting 'previous step' was impossible since I'd already closed SubExtractor). So I reopened it to start the training on the whole sub but since all chars were already trained, it was zapping through all items without me being able to interrupt this. And when it finished I also could not select to redo the training for the complete sub. Please put this feature back in. Thx.

In case you need someone to do the German translation of the GUI, count me in.

[Thunderbolt8]

maybe only using the left alt key could already do it

[Tappen]

Some more observations on 1.0.2.1c:

To enter [ and ] on a German keyboard you gotta press ALTGr (which is right of the space bar) plus 8, 9 respectively. This of course interferes with ALT for toggling italics. How about using space bar as the toggle instead?


When reviewing and correcting OCR Matches I see all chars listed, but not [ or ], despite me having entered them in this particular session.


I got an 'Character Matched but Off-Baseline when trying to enter a question mark in italics where only the upper half was recognized. Clicking on the dot and then typing ? did not work. I had to ignore the char. (Weird, after deleting OcrMapOrig.bin and doing the whole sub from scratch, the ? was recognized properly)


IIRC ASS subs allow for vertical positioning with the MarginV style tag. Would be nice if SubExtractor would translate the positions from VobSubs so credits do not overlap with subs.


How do I delete all trainings for a sub, wasn't it possible pre 1.0.21? When checking the finished sub, I realized several errors so I openend the original VobSub again (selecting 'previous step' was impossible since I'd already closed SubExtractor). So I reopened it to start the training on the whole sub but since all chars were already trained, it was zapping through all items without me being able to interrupt this. And when it finished I also could not select to redo the training for the complete sub. Please put this feature back in. Thx.

In case you need someone to do the German translation of the GUI, count me in.


I'll switch the Italics toggle to the Space bar. Much safer.

I don't know why [ and ] aren't showing. I've never seen that and have no idea why but I'll look into it.

There's currently a problem with lines that only contain matched characters that aren't trusted for baseline identification (- is most common) showing the "Ignore Baseline" message when it really should just move on to identifying the rest of the characters on the line. Should be fixed in next release. Just hit "Ignore Baseline" for now - it won't affect the final result (the character isn't ignored, only its contribution to defining the lines).

So you want the program to notice when positioned subs are low enough on the screen that they would conflict with the default dialogue subs and move the dialogue up for those cases? Good idea for a future feature.

You can open the Review OCR button and and just spam the "Remove all Trainings for this Character" button till the list is empty. Putting the "Restart OCR for this Movie" button on the main window was too dangerous. I can add it back into the Review OCR dialogue if you think we need it.

[Tappen]

has that sencence final " in case of italics change been (accidentally) reverted again in 1021c? all the " at the end of italicised sentences have a space between them and the last letter of the sentence again.

I don't think I adjusted it back. Check the "Adjust Word Spacing" page you can get to from the "Create Subtitle File" page. Select Italic, then select the " character. I changed the default left adjustment from "-1" to "-2". Your subtitle might need it to be "-3" or "-4" if the tails on the quotes are really long and/or stretch far to the left.

[Chetwood]

I don't know why [ and ] aren't showing. I've never seen that and have no idea why but I'll look into it.
Do you have an FTP or something? Whenever I stumble on something like this, I could upload the sub and a text file explaining the issue. Using Megaupload is too much of a hassle for small files like these.

So you want the program to notice when positioned subs are low enough on the screen that they would conflict with the default dialogue subs and move the dialogue up for those cases?
Actually I'd like it to recognize Vobsub item position. Usually you have two lines at the bottom of the screen. Often however, a few items at the beginning of a TV show are displayed at the top of the screen so they do not overlap the show's credits displayed at the bottom. The info on vertical position must be encoded either in the idx or the sub (BDSup2sub displays this info). But even if it weren't, since it's an OCR tool you should be able to determine vertical position and copy over this info to the ASS file.

Putting the "Restart OCR for this Movie" button on the main window was too dangerous.
Was it? How about adding 'are you sure?' and a checkbox to never ask this question again? In any case, I do think it's necessary so please at least add it back into the Review OCR dialogue. Thx.

[Tappen]

A new test build is up, with the new OCR and better Manual Entry features implemented (changed to Space bar for Italics, with blue indicators when on).
Also the OCR Review dialog has "Remove All ..." buttons to allow a full restart on a movie. I hate "Are you sure?" buttons.

http://subextractor.codeplex.com/releases/view/78152

[Thunderbolt8]

I don't think I adjusted it back. Check the "Adjust Word Spacing" page you can get to from the "Create Subtitle File" page. Select Italic, then select the " character. I changed the default left adjustment from "-1" to "-2". Your subtitle might need it to be "-3" or "-4" if the tails on the quotes are really long and/or stretch far to the left.the night of the hunter subs need -9 which seems to be quite a lot.
I guess as good would be to look for ". "" (or ! ? instead of the fullstop )and replace it with "."" in your subtitle editor.

[Tappen]

the night of the hunter subs need -9 which seems to be quite a lot.
I guess as good would be to look for ". "" (or ! ? instead of the fullstop )and replace it with "."" in your subtitle editor.

OK, I'll look into making a special rule to do this. I think the problem is that there are left double-quotes and right double-quotes in the subs but I only allow you to choose the non-left-or-right version during OCR. The right double-quotes in italics have a REALLY long tail which messes up the spacing.

It's not as easy as you'd think because different languages have different rules for quotes.

[Thunderbolt8]

thanks for trying ;)

[Tappen]

Thunderbolt8: I added a simple "don't put a space after .? or ! and before italic double-quotes" rule in the next release

Chetwood: I'll have to look into the issue with vertical position on IDX/Sub files. Must be some part of the file format I am missing.

[Chetwood]

Have you seen this (http://sam.zoy.org/writings/dvd/subtitles/)? VobSub layout should be similar. I'm not exactly sure how it's implemented. If the item is as large as the screen (720x576 on PAL) and all but the text is transparent, you'd need to OCR to determine position. However, if they use bitmaps only as large as two lines of text, they would have to store vertical position information somewhere. Probably in the header of each bitmap in the sub file?

[Tappen]

Have you seen this (http://sam.zoy.org/writings/dvd/subtitles/)? VobSub layout should be similar. I'm not exactly sure how it's implemented. If the item is as large as the screen (720x576 on PAL) and all but the text is transparent, you'd need to OCR to determine position. However, if they use bitmaps only as large as two lines of text, they would have to store vertical position information somewhere. Probably in the header of each bitmap in the sub file?

Yes that's the normal DVD subtitle format. If the bitmap is smaller than the video size and has an x,y origin other than (0,0) I definitely use it during OCR and in ASS file creation.

On the Choose Subtitle page there's a checkbox "Scale Image" that toggles between showing just the subtitle rectangle bitmap (unchecked) and placing the bitmap in its correct origin and size in the full video window (checked) (the checkbox doesn't change the final output, it's for user convenience only). I've tried a few IDX files and when the rectangle is not at the bottom middle of the screen I definitely pick it up and use the {\an4\pos(x,y)} tag to position the lines in ASS output files. So I'm not sure what the problem is you're seeing.

[Tappen]

I've put up what I think is the final test before release of the new Manual Entry and Fuzzy logic OCR features.

http://subextractor.codeplex.com/releases/view/78306

Currently you can toggle between Italics and Normal for Manual Entry using the Space bar. When using the mouse and the character selection box getting italics still requires holding the Ctrl key down. I'm thinking I should combine these and use just the Space bar for both entry methods to avoid confusion. Opinions?

[Chetwood]

Yes, please use the Space bar for both entry methods.
So I'm not sure what the problem is you're seeing.
Well, I didn't get them to display properly in VLC. Now, after some more tests with your new version, vertical positioning works just fine, even without me having checked 'Exactly Position Every Line' (what does it do?). Sorry.

BTW, marking the characters window with blue stripes as an italics indicator is an improvement over the old |||-method but looks a little tame to me. Why not go all out and make the whole window framed 2 pixels wide? Also, how about adding a color selection dialogue (not necessarily as complex as in Subtitle Creator) for the ASS format? Green subs look kinda tacky ;)

Do you plan on adding (customizable) shortcuts and batch functionality for future releases? Ripping entire TV shows every unecessary click saved would speed up the process considerably.

[Tappen]

I'm thinking of making the Space bar toggle both entry methods, but still allow the Ctrl key to temporarily reverse the character selection box. That way anyone used to using the mouse won't have to re-learn their workflow.

Normally I look for text that's centered in the bottom third of the video window and don't use positioning tags on it in ASS. That means it's displayed in the standard position for dialogue text by the ASS renderer in your video player rather than where the DVD author put it. 'Exactly Position Every Line' puts a positioning tag on every entry so it exactly matches the DVD. Usually this puts dialogue text too high on the screen because DVDs have to compensate for TV overscan. This option is normally useful if the subtitles are SDH (for the deaf and hard of hearing) where the dialogue is positioned all over the screen based on who is speaking. It can be very distracting if some of that text is moved to the lower, default position because the speaker happens to be in the middle of the screen.

I'll add blue horizontal bars to the top and bottom edges of the window when you're in italics as well. Should be clear enough.

The color of the subs is by default whatever the DVD used. Typically white or yellow but occasionally yes, a horrible green. On the "Subtitles" page in Options you can over-ride that. Choose "Use Custom Color" to set the color of all text, or "DVD Colors Except Default Text" will over-ride just the color of the text that is in the lower center part of the screen (any text that has a positioning tag will still use the DVD colors). This third option can give you a nice, consistent look for your dialogue but allow translated signs and text at the top of the screen to use the DVD colors. The only issue with this is that sometimes the DVD uses 2 different colors to indicate 2 different speakers. I add hyphens to the dialogue to indicate the speaker change if you've overridden the default colors.

[Tappen]

Do you plan on adding (customizable) shortcuts and batch functionality for future releases? Ripping entire TV shows every unecessary click saved would speed up the process considerably.

It is possible to select multiple idx or sup files from a directory. The button on the last page labelled "OCR Next Encoded Title" will then be available to quickly go through the list. Also, if you name the idx files consistently for a batch ("TV Show Season 1 Disc 1.idx", "TV Show Season 2 Disc 3.idx") the program won't ask for iolI,.' characters to be re-matched.

[Tappen]

For the next stage in the project I'm thinking of doing a documentation pass. Some better tooltips over various options and controls, also a couple tutorials: 1 for DVDs and 1 for Sup/Idx files. And a proper Installer rather than just a zip file.

I still want to allow multiple character sets/databases and support full localization/internationalization of the program but I feel those should be done after the more obviously missing English pieces are complete.

[Chetwood]

I really appreciate they way you're going with this. Adding these features will make SubExtractor more useful and like I said before, I'd be happy to do the German localization of the GUI. However, I think first and foremost the OCR capabilities should take priority. There are still some things that need ironing out:


Since you don't support mutitple streams in a VobSub yet, I'm stripping them down to one with VobSubstrip before opening them in SubExtractor. Still, they do stem from the same sub and thus have size, colors, font, positioning in common. So how come that when I enter all chars in the German stream, some of those have to be entered again when OCRing the English stream? It's particularly strange that these are usually simple non-italic chars like , - O . ' o l


The leading dash that marks two people speaking in one item sometimes translates to dash with a space behind (or a dash that has a space behind translates to dash only):

- Du brauchst Geld für diesen Monat?
- Ja.

is displayed in BDSUP2SUB exactly like this (dash + space) but is OCR'ed as

- Du brauchst Geld für diesen Monat?
-Ja.

It mostly happens with German subs but I can't tell from looking at the bitmap sub what causes this (as usual I can upload demo files, if you like).



The current beta does not seem to remember the last dir opened.


Please add an option to match the SRT/ASS filename to the original, dropping additional info like 'T1 Deutsch (German) Wide'.


What exactly is the meaning of '1080p Font Adjustments (2x Options Values)'?


Also, if you name the idx files consistently for a batch ("TV Show Season 1 Disc 1.idx", "TV Show Season 2 Disc 3.idx") the program won't ask for iolI,.' characters to be re-matched.
Are you referring to the OCR match process or the spellchecking as the last step before saving? Cause this dialogue pops up even if the files are named consistently.

BTW, is it possible to use another font in this spellchecking window? Cause with the current monospaced font the capital I and the capital l look almost identical and are hard to differentiate between with words like INITIATIVE which get 16 option listed (please check attached image). Thx.


http://img819.imageshack.us/img819/6742/spellcheck.th.png (http://imageshack.us/photo/my-images/819/spellcheck.png/)

[Thunderbolt8]

missing out the space after a dash happens quite often during OCR, but can be corrected via subtitleworkshop easily (though downside is that sometimes a space is added then in between a sentence with a dash, e.g. I-I din't know. -> I- I didn't know.)

[Tappen]

Chetwood:

I support multiple language streams in a VobSub idx. I just tried it and it works fine. If I take off the extra text (e.g. T1 English Wide) in the filename then each stream will over-write the previous if you try to do more than 1. You can just use "Save As" now and change the filename yourself.

Spacing before/after hyphens is more error-prone than other characters. I don't do anything special with it currently but perhaps I should. The easiest would be to make an option that lets you choose whether all hyphens at the start of a line have 0 or 1 spaces after them (it might be wrong but at least it's consistent). UPDATE: see my next forum post

The current release (1.0.2.3) remembers both the "Choose Subtitles" directory and "Save As" directory correctly.

1080p font adjustments means I double the font size, vertical and horizontal margins specified in Options when outputting an ASS file.
For example:
Style: Dialogue1,Tahoma,32,&H0000FFDB,&H000000FF,&H1F000000,&HC7000000,0,0,0,0,100,100,0,0,1,1.4,1.7,2,80,80,15,1
becomes:
Style: Dialogue1,Tahoma,64,&H0000FFDB,&H000000FF,&H1F000000,&HC7000000,0,0,0,0,100,100,0,0,1,1.4,1.7,2,160,160,30,1

If you name files similarly (i.e. same except for numbers and symbols) the error-prone OCR characters (iIl1oO etc.) won't stop the OCR process as they do for differently named files. I saw many cases of these characters being mistaken during OCR before I put this in. Multiple subtitle streams in a single bin or idx file never ask for you to repeat the error-prone characters. Also the spell-check step is completely unrelated to this and remembers its word database for all movies (except for choosing between AI and Al)

If you can find a font that distinguishes between l and I better than the one I use in the "l & I Spellcheck" step let me know. I chose it because it was clearer than the Arial, Tahoma or Microsoft Sans Serif I use elsewhere.

[Tappen]

I could force all lines that begin with - (hyphen) to never have a space after the hyphen. It seems the safest way to achieve consistency. Em dash (long hyphen) would be treated similarly. I would italicize the hyphen to match the next character.

I could also force a space after a hyphen at the start of a line if that's more to people's liking. Probably best to force no space after an Em dash even so since it's such a wide character.

I'd prefer not to add another hard-to-explain option to the program and just make a decision (always no space or always 1 space) that will make the output consistent. Personally I'd prefer forcing 1 space even if the OCR doesn't find one because I think that looks better.

Opinions?

[Chetwood]

missing out the space after a dash happens quite often during OCR
Weird. Given the nice and simple shape of the dash, I'd never expected it to cause any trouble. I use Ultraedit with regular expressions to replace it only at the beginning of a sentence.

I'd prefer not to add another hard-to-explain option to the program and just make a decision (always no space or always 1 space) that will make the output consistent.
Well, I'm more of an 'lot of options' guy. Granted, I do not have to code this stuff. Still, adding an option in the settings menu (not on the ripping page so you need to make a conscious decision to fiddle with it) with a reasonable default setting would be cool:

When OCRing lines beginning with hyphens:
(°) try to recognize spaces behind and add them accordingly
( ) always add space behind hyphen
( ) always strip space behind hyphen

I support multiple language streams in a VobSub idx. I just tried it and it works fine.
I just tried it with 1.0.2.3 and several subs I ripped with VSrip, all of which result in an error message: 'subtitle file out of date or corrupted'. It happens to any VobSub > 1 stream.

If I take off the extra text (e.g. T1 English Wide) in the filename then each stream will over-write the previous if you try to do more than 1. You can just use "Save As" now and change the filename yourself.
Well, it's your tool but I still think it would be best for you to adhere to best practices where the default is the other way round: you save under the same name and if you want to name it differently, you use "Save As". Also, when a file is to be overwritten, you get a window asking you about it where the default is 'no' rather than having the save dialogue default to overwrite.

You know, I'm always considering batch ripping and I don't know how much effort it would be to change functionality later on. I'd also like to be able to write to both ASS and SRT cause some TV's media players only render SRTs.

1080p font adjustments means I double the font size, vertical and horizontal margins specified in Options when outputting an ASS file.
That was my guess though I'm not sure it might be better to leave any adjustmenst to the player. Standalones these days often come with options to alter size and color for playback. I'm not familiar with the ASS format but when (un)checking this setting I got:

unchecked
Style: Dialogue1,Tahoma,32,&H00E6FFE6,&H000000FF,&H1F000000,&HC7000000,0,0,0,0,100,100,0,0,1,1.4,1.7,2,80,80,15,1
Style: Dialogue2,Tahoma,62,&H00E6FFE6,&H000000FF,&H1F000000,&HC7000000,0,0,0,0,100,100,0,0,1,1.4,1.7,2,80,80,15,1

checked
Style: Dialogue1,Tahoma,64,&H00E6FFE6,&H000000FF,&H1F000000,&HC7000000,0,0,0,0,100,100,0,0,1,2.8,3.4,2,160,160,30,1
Style: Dialogue2,Tahoma,62,&H00E6FFE6,&H000000FF,&H1F000000,&HC7000000,0,0,0,0,100,100,0,0,1,2.8,3.4,2,160,160,30,1

Does this setting apply only to Dialogue1?

If you name files similarly (i.e. same except for numbers and symbols) the error-prone OCR characters (iIl1oO etc.) won't stop the OCR process as they do for differently named files. I saw many cases of these characters being mistaken during OCR before I put this in.
Mmh, so does this result in a higher chance of missing error-prone characters? Or are they assumed correct as they were verified on the first sub of the batch?

If you can find a font that distinguishes between l and I better than the one I use in the "l & I Spellcheck" step let me know.
Probably some font with serifes? Gonna try to find one.

[Thunderbolt8]

I could force all lines that begin with - (hyphen) to never have a space after the hyphen. It seems the safest way to achieve consistency. Em dash (long hyphen) would be treated similarly. I would italicize the hyphen to match the next character.you'd have to distinguish this from lines beginning with 2 hypens -- though, as occurs after a scene switch and someone is talking or the subs constantly switching back and forth between a conversation/one person talking and background speech of e.g. a TV:

00:16:34: When did he say he wants to come over?

00:16:36: --wheather will be hot and cloudly
with some sunny spells in the morning and afternoon

(^^maybe also be italicised)

(dunno if that still happens, but theres also the case in which a line ends up with a double hyphen -- after SHD removal. at least this used to happen in former versions).

[Tappen]

Thunderbolt: yes, I'd have to watch for this case and apply the rule after 2 hyphens the same as after 1.

Some of the problem isn't in the OCR, since there are cases where I add hyphens but don't currently check the spacing on other lines in the sub to try to match it.

In any case, do you have a preference? There's no reason for us to keep the subtitle author's choice in this case; it's purely a style decision, not content.

[Thunderbolt8]

you mean the rule has to be the same in both cases?

dunno, you'd have to manually correct one of these cases anyway then. just by numbers, that case in which the one hyphen is wrong happens way more often that a double hyphen occurs in a movie. but in subtitleworkshop, fixing the one hyphen is just one click, no matter how many of them.
not sure whats with the double hyphen, afaik it would be accidentally "fixed" automatically in the same way. so you'd have to "re"correct that one manually afterwards. not sure what happens though when check for "-" in subtitles with one line is ticked for automatic correction as well, if one of the two hyphens is removed, then you most likely wont that line again after that. so you'd have to look out carefully for what settings etc.etc.

anyway, when saving directly to .ass with keeping exact position settings, not sure if subtitleworkshop can be used then without breaking some of stuff like position or fonts properties. in that case, having the one hyphen with space fix should ensure that less time for manual fixing would be needed than in case of not doing the space automatically, because that two hyphens after another case happens less frequently.

so I'd say that vote goes for automatically adding the space between hyphen and the beginning of the rest of the line.

in case I misunderstood and the question was whether I generally think that a space belongs between the one hyphen and the rest of the line or not: yes, it does.

[Tappen]

you mean the rule has to be the same in both cases?

dunno, you'd have to manually correct one of these cases anyway then. just by numbers, that case in which the one hyphen is wrong happens way more often that a double hyphen occurs in a movie. but in subtitleworkshop, fixing the one hyphen is just one click, no matter how many of them.
not sure whats with the double hyphen, afaik it would be accidentally "fixed" automatically in the same way. so you'd have to "re"correct that one manually afterwards. not sure what happens though when check for "-" in subtitles with one line is ticked for automatic correction as well, if one of the two hyphens is removed, then you most likely wont that line again after that. so you'd have to look out carefully for what settings etc.etc.

anyway, when saving directly to .ass with keeping exact position settings, not sure if subtitleworkshop can be used then without breaking some of stuff like position or fonts properties. in that case, having the one hyphen with space fix should ensure that less time for manual fixing would be needed than in case of not doing the space automatically, because that two hyphens after another case happens less frequently.

so I'd say that vote goes for automatically adding the space between hyphen and the beginning of the rest of the line.

in case I misunderstood and the question was whether I generally think that a space belongs between the one hyphen and the rest of the line or not: yes, it does.

Your last paragraph is right: I was asking what you generally prefer as a style. I'm in agreement, so next release I'm going to force a space after any hyphen that starts a line anywhere on the screen, and make sure the hyphen is italicized the same as the next (non-space) character in the line. I don't think it's worth an option; this way just looks better.

What about double-hyphens and em dash characters that start a line? Force no space after them? Or maybe 1 space after double hyphens but no space after em dash (wide hyphen).

[Thunderbolt8]

em dash and double hyphen should be treated the same way, because sometimes one of them is being used in case of the other (depending on the specific subtitle file), but bot serve the same function in the end.

which means at the end of a line there can be both, a space between the last word/letter and the em dash/double hyphen or not. this is due to the decision how the subtitle track is created and imho this can be kept the way it is.

for the beginning, theres usually only the case in which theres no space, as it indicates an abrupt change of speaker or being able to listen what one speaker has to say only in the middle of his sentence.

[Chetwood]

Concerning hyphens/dashes, I'd think having an option to keep as close to the original would be nice. Most subs I've seen do have a dash followed by space but some don't and they are still readable cause they usually have a decent font that allows for this. Concerning folders: my output folder is set to c:\test and "Store sup/idx file outputs in source directory" is checked. However, when I open a Vobsub in e:\movies it does save back to e:\movies but next time I open Subextractor it still wants to open from c:\test instead of e:\movies.

[Tappen]

Chetwood:

Since we're changing fonts when we do the OCR I think adding spaces after line-beginning hyphens is a good idea. Certainly if people keep the Tahoma font I have as the default for SubExtractor a space looks much better than no space.

I see the problem with directories. I don't remember the directory when you stop and re-start SubExtractor, just while it's running. Wouldn't be hard to remember between restarts as well I suppose.

[Tappen]

Various fixes are in the test release: http://subextractor.codeplex.com/releases/view/78727

[Thunderbolt8]

regarding the latest version and this "program will always add a space after a hyphen or em dash that begins a line", the em dash is basically like a -- (double hyphen, some tracks use the em dash, some use a double hyphen), so if possible, there shouldnt be a space at the beginning of a line if an em dash is used, because there wouldnt be a space for a double hyphen either.

[Tappen]

OK I'll take your word for it, Thunderbolt8. I'll change it so there's always a space after a single hyphen but not after double hyphen or em dash.

[Thunderbolt8]

thanks

[Tappen]

http://subextractor.codeplex.com/releases/view/78808

has the next test release code.

Confucio's Post-OCR bug fix is in (issue 757 on codeplex). No forced space added when text changes from italic to normal or vice versa.

Leading double hyphens and em dashes now will have no spaces after them, while single hyphens will always have a space.

[Confucio]

Thanks Tappen. Release 1024 works perfectly regarding the space on italics.

[JoseFina54]

:thanks:

[Tappen]

1025 is out: A minor fix for italic/non-italic spacing, a new option for Create Subtitle that keep the original line breaks in all cases, more tooltip help text.

Next up is more documentation: I'm thinking 2 tutorials for DVDs using either Handbrake or MeGui, and 1 tutorial for Blurays with Eac3to and MeGui

Also I'd like a proper install program. Some users are asking for at least an automatically created desktop icon.

[Thunderbolt8]

I dont have much time atm to spent on blu-rays/subs, but will continue to report errors in case I should meet some.

thanks for the new update

[Chetwood]

Also I'd like a proper install program. Some users are asking for at least an automatically created desktop icon.
Personally I hate automatically installed icons. You should give Inno Setup (http://www.jrsoftware.org/isinfo.php) a try which I used for the German version of DVD Shrink. It's free and you can set a lot of options that people can choose from during installation like desktop icon, quick launch icon, etc. Cool thing is, you can check them as default so they get installed automatically but people can also simply uncheck them if they don't need em.

And what about tutorials on Handbrake and MeGUI? There are a lot of tutorials already covering how to import subs ripped/converted with an external program already or am I missing something? Please include info on how to batch rip several subs (not necessarily from the same TV show). I'm still not entirely sure why some chars already OCR'ed for the German stream of a VobSub have to be re-OCRed for the English stream of the very same sub despite having the same font/color/size.

[ben_franklin]

Just used it. Awesome job Tappen!!! Thank you very much for this app!

[wilfried]

Thanks for a great program!

However, in 1025 (and 1024) there is a bug.
If you work on a dvd which doesn't have a language code set on the sub track, you get an exception on the last page and you can't save the ass file.

Exception thrown at
Illegal characters in path.
mscorlib
at System.IO.Path.CheckInvalidPathChars(String path)
at System.IO.Path.Combine(String path1, String path2)
at DvdSubExtractor.CreateSubtitleFileStep.subtitleStyle_SelectedIndexChanged(Object sender, EventArgs e)
at DvdSubExtractor.CreateSubtitleFileStep.Initialize(ExtractData data)
at DvdSubExtractor.SubWizard.LoadCurrentStep()
at DvdSubExtractor.SubWizard.nextButton_Click(Object sender, EventArgs e)

[Tappen]

OK wilfried, an easy fix. I'll release a new version soon.

[ben_franklin]

Apparently I spoke too soon. After doing several subs almost effortlessly I tried to do the subs for "black mask" bluray. They don't seem to be any different that other subs, yet I have to do identification on letters in every single sentence..... :(

[Chetwood]

I've seen something like this mentioned on a German forum too, can't remember the movie though. Does changing the palette make a difference?

[Tappen]

Apparently I spoke too soon. After doing several subs almost effortlessly I tried to do the subs for "black mask" bluray. They don't seem to be any different that other subs, yet I have to do identification on letters in every single sentence..... :(

My OCR fu is still pretty basic I'm afraid. If whatever authoring tool the bluray disc creators used for subtitles produces inconsistent characters (due to scaling most likely) then my OCR step will require a lot of manual work. This is why I spent so much time getting the manual entry user interface to be easy and efficient.

Basically I look for an exact match of the pixels between the characters in the database and the test character on the screen. If it's a bluray sup file I also shrink both the database and the test character by a factor of 3 in the x and y dimensions to try to find an approximate match. This results in 9 smaller patterns for each match and 9 for each test character (the shrinking can be done done 9 different ways with different starting positions) for a total of 81 chances at a match. But there's nothing more complicated than that going on - I don't in any way understand the shape of the characters.

[nibus]

I've had a weird issue where the program detects all "i" characters with "¡" (the upside down !). I can't figure out how to fix this. Is there a way to manually set a character assignment?

[Tappen]

When doing OCR there's a button in the bottom right labelled "Review and Correct OCR Matches". Press that.

Open the OCR Training drop-down list and find the bad match between i and the upside-down ! and press "Remove a Training".

Pressing "Remove all Trainings for this Character" would also work. Assuming you don't OCR Spanish much it would probably prevent future problems since it would eliminate all matches for the upside-down ! character currently in the database.

[nibus]

I actually tried that, but strangely there is no listing for the letter i. I also tried starting from a brand new OcrMap.bin file.

[Tappen]

If you delete your OcrMap.bin file the program will re-initialize it with the OcrMapOrig.bin file so that might explain why the 2nd fix didn't work.

The bad match would be for the upside down "!" character, not for "i". That character is probably at the beginning or end of the list - outside of the alphabet.

[nibus]

Strangely, the upside down "!" is matched correctly - I think it's the regular letter i that is being seen as the upside down "!". But like I said there is no letter i (lower case) listed, except in italics. I guess I could always just do a search and replace.

http://dl.dropbox.com/u/5637223/Clipboard02.jpg
http://dl.dropbox.com/u/5637223/Clipboard03.jpg

[Tappen]

Can't you just remove the training error (hilited in the 2nd picture) and your problem is fixed? I don't understand what the problem is.

[nibus]

The training in that second screenshot isn't an error - it's correctly identified the upside down ! mark. So removing it has no effect on the incorrect training of matching a lowercase letter i with the same character. I would remove the training of the letter i with the upside down ! but it is not listed, as shown in the first screenshot.

[Tappen]

Ah, I finally see the problem. This is the same as the issue with l and I having the same bit pattern in many subtitle fonts making accurate matching impossible. I added the entire spellcheck step just to solve that issue.

I'll have to make an option in the spellcheck step to discriminate between i and ¡ to fix this. I suppose the rule is that if it's not at the beginning of a word, or just after a ¿ at the beginning of a word, I can assume it's an i (eye) and not an inverted exclamation point. Otherwise I'll have to ask and build up a dictionary of words that really begin with i. Quite a bit of work, but I'll see what I can do.

For now, I'd remove the training and when you next run the OCR choose i (eye) and not the inverted exclamation because there are likely more of the former than the latter making cleanup easier.

[aMvEL]

Any chance for a possibility to select more than one language at a time when ripping, or at least being able to filter out or prioritize languages from the Subtitle track selection list?

It would speed up the ripping for me, since I usually rip English and Norwegian subtitles. The problem is usually that the Norwegian subtitle track is near the bottom of the list which makes it tiresome when ripping several seasons of tv-series, seeing as how I need to scroll down the subtitle track list every time.

[Tappen]

aMvEL: you guys and your batch processing: always surprises me what my customers want to do. But this is a reasonable request, and shouldn't be too difficult, so I'll see what I can do.

I suppose some sort of option that lets you choose 2 languages to put at the top of the sorted subtitle track list would be a start. Also the list currently only shows 5 items. Perhaps I can change the layout of the dialog to make the list taller and allow more to be visible without scrolling.

[aMvEL]

:) It wouldn't have been as much of a problem if I only rip movies, but since I rip mostly tv-series with a lot of episodes, I'd like to make it as easy as possible.

Both of your suggested solutions seems like excellent to me, as it would simplify things alot. :)

[Chetwood]

Same here. There could be some tweaking done to streamline the process but I'd thought to hold back with suggestions till more pressing issues (like proper vertical positioning when OCRing to ASS or changing palettes to counter blurred outlines) are solved.

[Slasher]

Hi Tappen,

Thanks again for all your work.

I want to point out Chetwood's request about proper vertical positioning when exporting to ass. Other than this issue the app worked fine for me.

[Tappen]

Could you guys explain again what exactly you want to change with vertical positioning that isn't done when the "Keep Source Lines and Positions" option is set on the "Create Subtitle File" page? Do you just want a "left-align" or "center-align" option for the ASS tags?

I have to say the reason I wrote this app in the first place was because I didn't like the vertical positioning of DVD subtitles (too high on the screen, with too many line-breaks on 16:9 film) so for me allowing the ASS rendering software to place and line-break the subtitles wherever it wants is the main reason I use my own program. This is why it's hard for me to understand other points of view on the subject and you have to spell it out repeatedly and in simple language.

[Chetwood]

It's like I wrote in my last mails to you: I want word-wrap and horizontal/vertical positioning to be identical to the Vobsub's:

http://www.dvdshrink.info/chetwood/stuff/tgw-vobsub.jpg

But when outputting to ASS and having 'Keep Source Line Breaks' selected it looks like this:

http://www.dvdshrink.info/chetwood/stuff/tgw-ass.jpg

The SRT has the proper horizontal placement but vertical is off and thus blocking credits:

http://www.dvdshrink.info/chetwood/stuff/tgw-srt.jpg

[Tappen]

You have to understand that ASS won't use identical fonts to the DVD subtitles so the width of a line of text won't be the same.

This means you have to choose between left-aligned and center-aligned for all text. The pictures above are left-aligned so their centers don't match. I can put in an option to center-align but then any text that is left-aligned will look strange. Early versions of SubExtractor worked that way and people reported it as a bug.

I'm sorry to say you'll never be completely happy with any solution that a computer can produce.

[Slasher]

I have to say the reason I wrote this app in the first place was because I didn't like the vertical positioning of DVD subtitles (too high on the screen, with too many line-breaks on 16:9 film).

Let me better explain my issue. When ocring a bluray subtitle and outputting to ass with the "Keep Source Lines and Positions" option enabled, the resulting ass subtitle is too high on the screen compared to the original bluray subtitle position.

I think this is linked to the fact that when calculating the positions the program assumes 1080p video. But I want these positions adapted to 720p. Could you do that? Maybe offer an option like scaling positions to a preset set of resolutions or maybe custom resolutions? I already tried the "resample resolution" option in Aegisub but with no effect, the subtitle stays the same, even though it changes the values for positions.

[Chetwood]

I'm sorry to say you'll never be completely happy with any solution that a computer can produce.
I get that. Since the Vobsub does not provide info on what font is used (and even if it were), it's not certain that an ASS set to that font would display identically cause there's no way of knowing how the standalone/software player will render the font.

Still, AFAIK many Vobsubs do not use the whole screen and position it at coordinates 0,0 but have a bitmap as small as the rendered text and position this accordingly. Could Subextractor translate this position info to the ASS? If not, it should default to horizontally centered items since the overwhelming majority of the subs I've seen so far are centered.

I've seen very few DVDs that place color-coded items off-center close to the person speaking and only one TV show that does this so far (and this only on the US DVD, the English sub of the German DVD has centered subs only):

http://www.dvdshrink.info/chetwood/stuff/m-vobsub.jpg

which looks like this in ASS:

http://www.dvdshrink.info/chetwood/stuff/m-ass.jpg

and like this in SRT:

http://www.dvdshrink.info/chetwood/stuff/m-srt.jpg

I agree with Slasher that scaling options would be awesome. When saving to ASS, the save dialogue would offer to pop up a preview window where the longest item is displayed and people could change size, color and position. If that is too much work, please add the option to center-align the subs (a mouseover would explain the implications so people would not report this as a bug again). Thanks!

[Tappen]

SDH subtitles (for the deaf and hard of hearing) position the text near the speaker and left-align. A lot of people use this type of subtitle for various reasons and they were the ones complaining about the center-aligned text. I have a lot of sympathy for their viewpoint so I'll have to add centered as an option and leave left as the default.

In the DVD screenshots above the font used to create the DVD subtitle bitmaps was clearly unusually tall and narrow. If you changed the font used by SubExtractor to something like Arial Narrow instead of Tahoma it would probably line up better. Just a tip.

[Thunderbolt8]

Could you guys explain again what exactly you want to change with vertical positioning that isn't done when the "Keep Source Lines and Positions" option is set on the "Create Subtitle File" page? Do you just want a "left-align" or "center-align" option for the ASS tags?

I have to say the reason I wrote this app in the first place was because I didn't like the vertical positioning of DVD subtitles (too high on the screen, with too many line-breaks on 16:9 film) so for me allowing the ASS rendering software to place and line-break the subtitles wherever it wants is the main reason I use my own program. This is why it's hard for me to understand other points of view on the subject and you have to spell it out repeatedly and in simple language.Not completely sure whether I understand it correctly, but I also agree with too high on screen positioning and too many line breaks with DVD subtitles. There shouldnt be no more than 2 lines on the screen, if avoidable.

as for the left or center align option, standard is center align position, right? Thats fine with me. Dont like left align, unless its necessary (well, it looks better then) to use in case of SDH titles spread over the screen for different speakers.

[Tappen]

The standard is currently to use the left-aligned tag in ASS. This is because there tends not to be many positioned text entries, and even less that is 2 or more lines tall where the choice of left or center alignment is noticeable. The exception is SDH subtitle tracks where left-alignment is essential so they got priority.

I'm thinking that rather than trying to add another option I'd just find text that is centered and near the top of the screen and use the ASS center-align tag for those entries and continue with left-aligned tags for the rest. I already have working code to find text that is centered and near the bottom of the screen after all.

[Chetwood]

SDH subtitles (for the deaf and hard of hearing) position the text near the speaker and left-align.
Not on German DVDs.

In the DVD screenshots above the font used to create the DVD subtitle bitmaps was clearly unusually tall and narrow. If you changed the font used by SubExtractor to something like Arial Narrow instead of Tahoma it would probably line up better.
Right. But since fiddling in ASS is inconvenient it would be nice if we could change these settings from within Subextractor like this (http://www.dvdshrink.info/chetwood/stuff/subcextractorgui.png) (this mockup shows a maximum of settings, basics like font and position would be fine).

I'm thinking that rather than trying to add another option I'd just find text that is centered and near the top of the screen and use the ASS center-align tag for those entries and continue with left-aligned tags for the rest.
Can it be both? Cause as you wrote ealier, depending on font and font size even left-aligned subs may look terrible. I mean, you already have this coded as well so adding another option (again see linked screenshot) might not be too much work?

[Tappen]

This is going too far into Dvd-specific (non-Bluray) options for my taste. I just want my subs to look good compared to the ugly, blocky Vobsub bitmaps on an HD monitor with a minimum of effort. Perfect layout and accuracy has never been the goal for me, sorry, and I'm not interested in doing to work needed to get there.

Some sort of centered vs. left-aligned option is coming in the next release. I'm just trying to determine if there's an intermediate state where an intelligent decision can be made instead of forcing all text to be either left-aligned or centered.

[Betsy25]

After stuggling and extremely bad sub recognition experiences with SubtitleEdit, this tool simply ROCKS !
it took me 10 minutes to teach it some few things, and now it works wonders ! :)

1) Just a tiny feature request : Is it possible for the last step (Save As....) to make it default to the actual DVD location instead of the default "library" location please ?

2) Is it normal that ignored characters end up being trained as a blank space ?
If I press "ignore" for a line of characters at the start of the movie, they end up in the "Training" list from the "Review and correct OCR Matches" window, trained as blank space ?


Thanks for the brilliantly clever tool.

[Tappen]

Where would the actual DVD location be? I suppose the VIDEO_TS folder is the most reasonable, or maybe the directory 1 up from that? It's not a hard option to code just hard to design.

Yes, ignored characters are trained as mapping to no character. This allows you to specify patterns which will have no output. However, like o, l, I and a few others these patterns only apply to the current DVD so don't worry about it messing up the OCR of other DVDs. If instead you just don't want to try to split a big lump of stuff for OCR - happens sometimes - I'd recommend using a Greek character so you can find it in the output file easily and insert what you actually want to appear in the subs.

[Betsy25]

That for the clear reply Tappen, I'm just a little lost about which training rules are global and which are source specific.

[Tappen]

I move training rules from global to source specific when I find they cause errors in my own OCR'ing. It's pretty unscientific but that's what has happened. Characters l, I, period, commas, apostrophes were obvious, but I moved o and O into the source-specific category only when I found a few DVDs where mistakes were being made because of trainings from other DVDs.

I could try to indicate which characters are in which category but I don't think it would help people much: they'd have to follow the same process anyway. It's only folks like you who are curious and interested enough to want to know how things work who are affected. The current list (from the source code) is:

public static readonly char[] MovieSpecificChars = new char[] { '1', 'l', 'I', '.', '\'', ',', '-', '—', '_', '\\', '/', '|', 'o', 'O', '°', OcrCharacter.UnmatchedValue };

(OcrCharacter.UnmatchedValue is what you get with the "Ignore" button)

[aMvEL]

I have a slightly bad vobsub that when using the default pallette it doesnt show the dot above the letter 'å', so that it recognizes the letter as 'a'.
If I change to pallette to 1,2 it detects it just fine, however it always changes back to pallette 1 (default). Is there anything I could do like force a certain pallette for the entire subtitle?


I've added a sample of the first episode. The nordic languages nor, dan, sve, are the ones with trouble.
http://dl.dropbox.com/u/2914045/FAMILY_GUY_S1_D1%20Track%201.zip

[Tappen]

aMvEL, you need to hit "Start Over for the Whole Movie" and change the palette to 1,2 from the first subtitle. The program will stay with 1,2 from then on (with 1 exception in the middle where you have to manually choose 1,2 again in my test).

The problem is that the characters on this subtitle are unusually fat and the program thinks palette 1 is more likely to be correct because of the average number of pixels per character. Once a few of the characters are in the OCR database it no longer depends on the size of the characters to choose the palette but bases the decision on how many of the characters can be immediately identified.

This is a hard problem to fix in general and I'd likely break more subtitles than I fix if I changed the algorithm for palette-choosing now so I'm not sure whether I'll change anything for the next release.

[aMvEL]

Working semi-perfectly now ... alot better than SubRip in any case, which I could never get to acknowledge that there was a dot over the a making it a 'å'.

Just an extra question, is the "alt gr"-key supposed to trigger italics along with ctrl (and space)?

[Mug Funky]

i wonder whether it'd spoil the layout formatting logic if top titles could behave in a similar way to bottom titles?

regular subs are detected as centre-aligned and have the "pos" info ditched, and this works wonderfully. however, a lot of DVDs have subtitles in the top of frame, either for intro song translation, or if action is at the extreme bottom of screen and the subs move to the top to not hide the action.

would it be possible to extend the centre-align behaviour to the top part of the screen?

am i making sense? i'm afraid English is my first language, and i'm still bloody awful at it.

thanks!

[Tappen]

Mug Funky. I believe that I understand your question, and can tell you that this change will be in the next release.

It was requested by other users as well that I notice when subtitles are centered on the screen and use the /an5 tag (middle-center) instead of the /an4 tag to position those lines of text.

[Tappen]

aMvEL, "alt gr" isn't supposed to trigger italics. I'll have to grab a Euro keyboard from somewhere and figure out how to turn that off. Ctrl is supposed to temporarily turn on italics, and space toggles the base state, but alt-gr isn't supposed to do anything. I'm guessing that I'm getting a false positive on my check if Ctrl is down in this case.

[Mug Funky]

excellent! i believe you've just made the closest thing to a perfect sub OCR utility that the world has ever seen :)

[edit]

one thing that might be of use is the ability to run "no good character" matches through the spell check so we can manually enter what they're supposed to be without influencing the character database. i've noticed in some cases square brackets can be spotted as Is and Ls, and wasn't able to make it behave. perhaps a checkbox to not store the entered character as a match? not sure how it should be done, but there's a few approaches one can take.

[Tappen]

Mug Funky: If I understand your question correctly, I believe you might need to use the "Split in 2" function to deal with characters which are connected instead of just marking them as "no good character".

[loekverhees]

I have subtitles with little chunks around it, see the image below. Every time I have to press 'Different Palette', for every subtitle. Can the sensitivity be changed manually, so it ignores these little pieces around the letters? I think it is some kind of shadow effect.

http://puu.sh/vndM.png

[Tappen]

loekverhees: I thought I had taken care of all the situations like this but I guess not. The more common problem is with shading that wraps entirely around each letter, which I do handle pretty well now. This is a weird one. The program sees 2 palettes as containing the same number of recognizable characters, but one has some extra pixels which it thinks are unknown characters that need to be identified. aMvEL's problem a few posts up is an example of how sometimes characters really are drawn in 2 colors and the program is already too insensitive for some subtitles so I can't reduce the sensitivity even more.

I'm going to try to add a learning feature to manual palette choices made by the user so that the program won't require you make the same adjustments again and again.

[loekverhees]

Thanks for your reply. Or just an option to force the program to use a certain palette for the entire subtitle, as aMvEL mentioned. In my case, palette 1 is the right one. However, it just automatically defaults again to palette 1,2 all the time (even if I choose palette 1 for the first subtitle).

[Tappen]

Palettes are a little tricky so I'd worry about adding such an option. The colors can change in the middle of a subtitle track: 1,2 might be the letters and shading in one sub, then the color of index 2 changes and it's used for some independent text on the next sub while 1,3 are used for the main letters and shading. If you had forced it to be just palette 1 you'd miss the index 2 colored text (and possibly never realize it was gone from the output file).

I suppose this is rare enough that adding an option to force the palette wouldn't hurt. If I change the "Next Palette" button to NOT automatically begin OCR'ing and add 2 more buttons: "Apply" and "Apply for Movie" underneath it this should provide the function you need. I've always worried a little about OCR automatically starting after the "Next Palette" button anyway.

[loekverhees]

Okay, sounds great ;-). Thanks!

[cheer]

OK, no questions or problems, just a note to say...this is a FANTASTIC app. I'd been using SubRip on my DVDs for years, and since it was good enough, I never bothered looking for another. Then today I stumbled in here and...wow. You've just made my sub-ripping life so much easier.

Thank you so much.

[Betsy25]

OK, no questions or problems, just a note to say...this is a FANTASTIC app. I'd been using SubRip on my DVDs for years, and since it was good enough, I never bothered looking for another. Then today I stumbled in here and...wow. You've just made my sub-ripping life so much easier.

Thank you so much.

That's exactly what I've been thinking. So cleverly programmed, minimalistic yet so powerful. I've since never looked back at SubRip.:)

[Tappen]

1.0.2.7 is out with a few fixes and features:

Fix: out-of-memory exception when reading DVDs with very large (over 1GB) cells
Fix: AltGr key toggling Italics during OCR
Feature: use centered alignment SSA tag for centered text in the upper part of the frame
Feature: increased number of subtitle tracks visible in Choose Subtitles step listbox
Feature: allow change of palette for entire movie

The memory use issue is one that I hope nobody has a problem with: I recently bought some unusual DVDs that I wanted to encode with 3GB+ cells in them and was running out of memory space in 32-bit Windows and out of physical memory in 64-bit during the SubExtractor encode step. (Cells are the smallest kind of "chunk" of audio/video defined in a DVD IFO file and I'd never seen one over 1GB before so it was quite a surprise.) Anyway, now any cell over 500MB uses a temp file to hold the extra data while loading and deletes the file when saving is complete. This change might have the added benefit of not slowing down your computer quite as much during this step.

[sneaker_ger]

http://www.abload.de/img/split41bjc.png
How to split these?

[MokrySedeS]

Hit "Split in 2" (obviously) --> paint "f" with your cursor --> Save split --> program will ask you to recognize the remaining part, which you left black --> select the dot --> hit "i"

[sneaker_ger]

Thank you, it did indeed work. It didn't ask me to manually enter f nor i, so I assumed something went wrong. I had manually typed in a special char for this combination and just searched and replaced all occurrences with a text editor by hand later. But now that I just followed your advice, the resulting file came out bit-identical to my manually created one, so Subtitle Extractor did its work just fine after all.
Maybe it could be changed to first click on the i-dot and then click split in 2, kinda feels more intuitive. If it were two i characters (or something like "i?") you wouldn't really be able to correctly split them.

[MokrySedeS]

Maybe it could be changed to first click on the i-dot and then click split in 2, kinda feels more intuitive.

+1

It's even more confusing when 2 characters from first and second line are picked together, like this:

http://i49.tinypic.com/2lmqeya.png

I wasn't sure if it's gonna get recognized as comma or something... but it actually worked as it should :D

[Tappen]

I know splitting can be non-intuitive sometimes but I can't think of how to improve it without adding a lot of clicks.

SubExtractor does know how to auto-split many 2-character combinations but it requires manual intervention for ones that could easily be errors: for example it used to sometimes split m into r and n if the m had a pattern that hadn't been trained. Now it requires you to manually enter any splits containing the letter r (as well as i, l, I, most punctuation and all accents).

Don't forget that painting the character green can be made easier using the left-mouse click which acts as a "paint-bucket" like you see in drawing programs. Just mouse paint along the joint between the 2 sections then left-click to fill the rest. (Right-click resets all painting)

[MokrySedeS]

it used to sometimes split m into r and n if the m had a pattern that hadn't been trained.

Still does. Happened to me couple of days ago with this (http://www.sendspace.com/file/rmonf1) file.
Luckily for me this particular BD had 4 subtitle tracks in my language and this issue didn't occur in this one (http://www.sendspace.com/file/ac9mbq).

[Tappen]

I tried both those subs and didn't see the problem using version 1.0.2.7. It might be that you've trained more Polish specific characters (ń in particular) that led to the problem. I'll add something to 1.0.2.8 that hopefully will catch this.

Yes that was the problem. ń needed to be tested for as well as n & ñ. Fixed in 1028

[mood]

This is the best piece of software to extract sub and Ocr application.

Thanks for this fantastic software ;)

[loekverhees]

Feature: allow change of palette for entire movie

Great! I tested the new version and it works perfectly! Thanks ;-).

[pandv2]

Two suggestions, for your consideration:

- When only one part of a two parts char is recognized (as in i, ñ or á) extend the selection with the keyboard (maybe up arrow if the lower part is the recognized part, and down arrow in the inverse case). Maybe, it can be also automàtic, if the key pressed is a two parts char, add the part not selected.

- When I remove a training don't return to the begin of the list, use the next item to the deleted one. I am checking a lot of training (hundreds) and i need to search for the point each time i see an error and delete it.

Thanks.

[Tappen]

pandv2: Both good suggestions.

1. I thought of the first one already and am in the process of implementing it: Up arrow will add the nearest fragment whose center is above the top of the currently selected fragment, Down arrow the nearest fragment below.

2. I'll change the training removal dialog to use large Listboxes instead of Comboboxes so this isn't a problem. I've been bothered by this behavior as well.

[Tappen]

1.0.2.8 is out with a fix and some features:

Fix: handle padding bytes in IDX files
Feature: added "last character matched" feedback after making OCR matches
Feature: added support for Thai language accents and punctuation
Feature: added left/right/up/down key functions during OCR matching to allow selection of nearby fragments (dots, accents) without using the mouse
Feature: changed the OCR Match Review dialog to have less annoying list scrolling and selection behavior

[masster64]

I've trained SubExtractor v1.028 for 113 lines out of 1601, and it still asks to identify almost all letters of the alphabet.
This is getting very boring...

[pandv2]

Hello,

2 questions or suggestions.

Is the new version pre-trained?. Because if I delete all the trainings for this movie, the app doesn't ask for all the chars. It confuses í with i (accented i with letter i), and ¡ with i (open exclamation with letter i).

Somethimes a sub file contains a lot of variants for each letter (differing only in a few pixel on the border). I thinked about how to resolve this, and maybe reducing the letters to a pixel wide edge previously to compare, can works. Because the differences are mainly in the external border.

Actually I am resolving the confusions passing the result for a ortographic corrector.

A little more test results: If I delete all the trainings for this movie, press ok, and after, return to the training editions, the list is not empty.


Thanks.

[Tappen]

masster64: Sorry, there are some subtitles that just aren't consistent enough to be trained. The disc authors create the subtitle bitmaps in all sorts of different ways and some just don't OCR well (at least how SubExtractor does OCR).

pandv2: If a character was trained on another movie and matched on the current movie and then you delete all trainings the match still exists in the database (linked to the other movie). When you hit OK the match will be found again if it's near the start of the subtitles (before SubExtractor pauses for the first unmatched character) and not one of the i, o, I, l, O, etc. characters that don't share trainings between movies.

If you really want to start from scratch you need to delete both your OcrMap.bin file and the OcrMapOrig.bin file that comes in the zip package.

There is a problem with Blu-ray subtitles and errors on small characters or fragments of characters like accents. This started when I added the fuzzy logic OCR feature. I've seen it confuse periods (.) and commas (,) myself. I'm working on a fix, though it will inevitably mean there will be fewer automatic matches and more typing.

[Chetwood]

This is getting very boring...
Complain to the morons who were to stupid to properly author their subs. SubExtactor has the best OCR routines ever.

I'm working on a fix, though it will inevitably mean there will be fewer automatic matches and more typing.
Maybe you could add an option so people could switch between routines per subtitle? Kinda like switching between palettes.

[Thunderbolt8]

I've trained SubExtractor v1.028 for 113 lines out of 1601, and it still asks to identify almost all letters of the alphabet.
This is getting very boring...there are some discs which are simply a PITA e.g. the blu-ray of the tree of life gave me nightmares.

[masster64]

masster64: Sorry, there are some subtitles that just aren't consistent enough to be trained. The disc authors create the subtitle bitmaps in all sorts of different ways and some just don't OCR well (at least how SubExtractor does OCR).
But there is a solution to that. A more relaxed 'recognition success' algorithm. Let's say 100% is a pixel per pixel recognition consistency. Give us a slider to allow using lower rates of success and all will be easier.

[pandv2]

About the fuzzy logic OCR feature.

Maybe you can allow to disable it (in the beginning of the movie training) for all the chars or for a list of selected chars (user selected). So when it's trained in the diferences between (for example) i , í, ì, ¡ the user can reactivate it After this, the fuzzy logic needs to be applied after the strict logic, or a metric defined to find the nearest match.

I deleted OcrMap.bin and OcrMapOrig.bin, and retried the failing subs. It's a lot better now. The only error is now the confusion between i and ¡ (voyel i and opened exclamation), but now, it not happens all the time. Maybe a feature to search for a concrete word bad ocr'd (as: iHola!) and retrain it, can be useful.

[Tappen]

pandv2: The fuzzy logic is only applied to Bluray sups so if your problem is with a DVD sub then there's something different causing the problem (let me know here right away if the problem is with DVDs).

Assuming it's with a Bluray subtitle, first let me explain that the fuzzy logic is just downscaling: each character or part of a character is shrunk by 3x in each dimension (there's actually 9 ways to do this so I end up with 9 mini characters). If any one of these 9 is an exact match of a mini character from the database then the OCR match is completed just as if the full-sized characters exactly matched.

During testing this caused too many bad matches like what you're seeing on small fragments, so I set the downscaling to 2x if the initial character size is 9 or less pixels on each side (so then there's 4 mini-match possibilities = much less chance of a bad match). Unfortunately this isn't sensitive enough and I should have set it at something like 10 pixels on 1 side and 16 pixels on the other to make the matches fully reliable even though it would be less "fuzzy" and find fewer matches. You can't change this on the fly because it requires re-downscaling all the characters in the database (would take a minute or more on the fastest machine given the 1000s of characters in the typical database). This is why a slider isn't possible.

I'll try making this change and putting a test version up on Codeplex for you. If it fixes your problem I'll do a real release with the change.

In case other people are nervous about the change, the fuzzy logic only helps about 1/3 of the bluray subtitles in my experience - the other 2/3 are created with authoring tools where the characters match perfectly. So this change will mean more typing on just the 1/3 that are dodgy to begin with, and those are the subtitles that are likely to see mismatched character problems, so the trade-off seems reasonable.

[pandv2]

Thanks, and yes there are sup files extracted straight from a bluray.

But, now i know the reason to the problems with a sub+idx converted from a bluray. I think in this case the fuzzy logic doesn't get applied.

[masster64]

@Tappen
any reply to my suggestion?

[Tappen]

masster64: sorry but if you changed how much "fuzzy logic" is being used you'd have to re-compute the database. So it would take a minute or more for every click of the slider. I don't think anyone wants that. The only simple change is an on/off switch.

pandv2: There is a "size:" definition near the top of most idx files. If the horizontal resolution (the first number) is greater than 1400 then the fuzzy logic code is used.

[Tappen]

I've created a release - Tighter Fuzzy Logic 1 - on Codeplex. Please try it out and see if it fixes the recognition problems on High Def subtitles.

http://subextractor.codeplex.com/releases/view/92608

[mscsmyrpln]

Very impressive. I've wanted a tool like this for a long time. Thank you!

[rbauer]

@Tappen
If possible, could you make It portable?

Now It writes to (Win7x64-standard account):

HKEY_USERS\S-1-5-21-3668210609-2088346886-2201473197-1001\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.idx\
HKEY_USERS\S-1-5-21-3668210609-2088346886-2201473197-1001\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.idx\OpenWithList\

c:\Users\<USER>\AppData\Local\DvdSubExtractor\
c:\Users\<USER>\AppData\Local\DvdSubExtractor\DvdSubExtractor.exe_Url_anvjrhxtsierb2tr4i2lphhngd2nuvhe\
c:\Users\<USER>\AppData\Local\DvdSubExtractor\DvdSubExtractor.exe_Url_anvjrhxtsierb2tr4i2lphhngd2nuvhe\1.0.1.3\user.config

c:\Documents and Settings\<USER>\AppData\Local\DvdSubExtractor\
c:\Documents and Settings\<USER>\AppData\Local\DvdSubExtractor\DvdSubExtractor.exe_Url_anvjrhxtsierb2tr4i2lphhngd2nuvhe\
c:\Documents and Settings\<USER>\AppData\Local\DvdSubExtractor\DvdSubExtractor.exe_Url_anvjrhxtsierb2tr4i2lphhngd2nuvhe\1.0.1.3\user.config


Many thanks

[Tappen]

rbauer: I think you just need to go into Options and check "Use Program Exe Location" to make the OCR database portable. Most other options you're likely to change depend on the machine directories so I'm not sure it can be made fully portable. In any event I just use the default .Net Settings system and I'm not sure how to customize that.

I don't register the .idx extension - you must have done that yourself.

[rbauer]

rbauer: I think you just need to go into Options and check "Use Program Exe Location" to make the OCR database portable.
Unfortunately that option ("Use Program Exe Location") is for OCR Data File ("OcrMap.bin") location only.

User settings (user.config) are still write to
%appdata%\DvdSubExtractor\DvdSubExtractor.exe_Url_anvjrhxtsierb2tr4i2lphhngd2nuvhe\1.0.1.3\user.config

Thanks

[Tappen]

rbauer: sorry, further portability isn't going to happen. It's too much trouble to support all the possible Windows versions which have different file system access rights if I tried to write my own settings storage.

I'll stick with the standard .Net settings for now. On the bright side if you are on a Active Directory Domain these settings should persist across machines.

[rbauer]

rbauer: sorry, further portability isn't going to happen. It's too much trouble to support all the possible Windows versions which have different file system access rights if I tried to write my own settings storage.
Ok, Thank you anyway :thanks:

[pandv2]

I've got time to test your Tighter Fuzzy Logic 1. I uncompressed it in another directory and executed from it.

The results are similar to the results obtained deleting the OcrMap.bin and OcrMapOrig.bin, but I needed to enter less chars in general (but more times the , char).

It continues only confusing i and ¡ (voyel and oppening exclamation). The rest it's ok.

In the mean time I detected another things:

- In the spelling phase, its not possible to select the correct option for some roman numerals (as III).

- The idx file from the bluray is from a rip to a 720p resolution. It has 1280 horizontal resolution, so the fuzzy logic doesn't get activated in this case.

Thanks.

[Tappen]

pandv2: I think the only way to discriminate between i and ¡ is to make these characters movie-specific. If I allow all the database matches to be searched it's just too likely that an i from some other movie matches the ¡ in the current movie. I don't think the slightly lower baseline position of the ¡ character is reliable enough to allow SubExtractor to tell the difference. This means you'll have to re-enter i and ¡ matches every movie like you currently have to for o, O, l, I, etc.

I can add this as an option for people working with languages that contain the ¡ character (and turn it on automatically if I know the subtitle is Spanish). I'll put a build up on Codeplex with this option if you want to test it, or just let me know here what you think of the idea.

[pandv2]

I can add this as an option for people working with languages that contain the ¡ character (and turn it on automatically if I know the subtitle is Spanish). I'll put a build up on Codeplex with this option if you want to test it, or just let me know here what you think of the idea.

For me it's ok.

Another solution, is to add to the I-L discrimination phase. The exclamation opened char, only can appears at the beginning, never in the middle of a word. And normally precedes a uppercase letter.

[Tappen]

Adding to I-l is a good idea, but the reason this feature works is that you eventually build up a database of words where the choice of letter is questionable (starts with l, mostly) which isn't too big. A similar database for i and ¡ discrimination would have to contain all the words which could start a sentence. Not as useful since it would have to be huge and there'd be a lot of false positives (words which are also a word when prefixed by the letter i).

I could replace all words that start with i followed by an upper-case letter with ¡ automatically during the Spellcheck step. Hard to think of any legitimate cases of this pattern. On the whole I think per-movie i and ¡ ocr is the best solution.

[Chetwood]

Agreed.

[Mug Funky]

yes, iPods, iPads, iTunes and all other iTrademarks can take one for the team.

[Tappen]

I wouldn't want to be sued for patent infringement because I typed those words into the program. I'm amazed you're willing to take the risk by writing them here, Mug Funky.

[MokrySedeS]

Hi Tappen. I've encountered an error with this (http://www.sendspace.com/file/rsjml0) file. In subtitle #1524 the text "Listen!!!" is recognized as "Listen"..!"
Also e.g. in #84 "tt" is recognized as "lt".

[Betsy25]

I don't know which timings are the correct ones, for all movies I processed by both DvdSubExtractor & SubRip, they always return different timings for the content. Sometimes difference up to half a second. I don't know which one would actually produce the "correct" timings ? :helpful:

[Tappen]

MokrySedeS: I was able to reproduce the problem with tt being recognized as lt. I'm not sure it's fixable, though. The text in this sup file is strangely small for a Blu-ray and the cross on the t is just small enough to be fuzzy logic'ed away by the code that automatically looks to split unrecognized characters in 2.

I wasn't able to reproduce the Listen!!! being recognized as Listen"..! problem. I think you have a " character in your database that I don't have. Again I can see how this would happen but can't think of a way to fix it that wouldn't create other recognition problems. After all !! could really be an exact match to a double quote above 2 periods in some other subtitle's font.

This are both errors but I can't think how to fix them. Thank you for bringing them to my attention and I'll try to come up with ideas in the future.

[Tappen]

Betsy25: One of the reasons I wrote SubExtractor is because the timings were wrong on a bunch of Subrip IDX files I had made. There is no perfect timing for many DVDs, which are made up of discreet pieces (cells, programs, program chains) of video+audio+subtitles designed to be concatenated together at the whim of a hardware DVD player. Within a cell and most program chains (made up of 1 or more cell) there are reliable timestamps, but once you have more than 1 program chain making up a movie - or the program chain has timing discontinuities - exactly how to keep the time continuous is an art more than a science. Watch how various video player software deals with MPEG PS files (.mpg usually) and you'll see how timing is pretty random.

Short answer: I spent a lot of time trying to match the timestamps SubExtractor creates to what you get when you just simply stick all the video and audio pieces together for the main program(s) of a DVD. I think it's more reliable than SubRip. If you find where that's not true let me know the DVD and I'll see if there's a problem I can fix.

[Betsy25]

Thanks Tappen. Guess that's just one more reason to stick with SubExtractor.:)

[MokrySedeS]

Tappen, could you add Polish characters to "Advanced Word Spacing Adjustment"?
"ą" in particular is causing me problems and I can't fix it.
Also very annoying thing - Subtitle Extractor is stealing focus when it finds unrecognized character. I've screwed up my ocr just now writing this post and had to hunt down the error. Option to disable focus stealing would be nice.

[Tappen]

MokrySedeS: OK I'll look into both of these issues.

[MokrySedeS]

Thanks in advance :bow:

[deco20]

Tappen, is it possible to create editor of OcrMap.bin? Because once you make a mistake and don't notice that just after OCR process, you must delete this file and start collectioning chars again.

[Tappen]

deco20: you can run the same OCR again and then delete the mistakes when it gets to the end of the file using the "Review and Correct OCR Matches" button. SubExtractor will recognize that the file name is the same and correctly edit the part of the database that was created when you ran the OCR the first time on the file.

I don't think I could write an editor big enough to let you find mistakes in the full database: there are just too many characters. Maybe if I let you choose from the list of movie names and brought up the "Review and Correct OCR Matches" dialog without making you go through the OCR again. I'll think about adding it as a feature.

[Tappen]

Version 1029 is out on http://subextractor.codeplex.com/ with the following fixes - mostly requests from this thread:

Feature: Added option to make i and ¡ characters movie-specific for improved OCR on Spanish subs (Special Characters tab in Options)
Feature: Allow switch to Word Spacing dialog directly from Spell Check dialog
Fix: Added more default word spacings for accented characters
Fix: Changed Word Spacing dialog to show all OCR'd characters in current sub
Fix: Removed application focus grab during OCR
Fix: Tightened HD subs fuzzy logic to reduce false matches in small characters
Fix: Improved Arrow key selection during OCR

[errantkkn]

Tappen, could you add a feature in SubExtract like, press buttons, then Enter, then ORC. 'Cause in my languague, I've to press two or three buttons for each letter (Ex: "e" + "6" + "3" = "ể"). Thank you.

[Tappen]

errantkkn: Would a button that controlled whether an "Enter" key is required to finish a character fix your problem? By default the first character would be used like it is today, but if this new button was in the toggled state SubExtractor would wait for the Enter key before reading the character in the text box.

We could make a hotkey like F2 toggle the state as well so it would be easy to switch modes if you have a section of Latin characters in the subtitle.

[errantkkn]

errantkkn: Would a button that controlled whether an "Enter" key is required to finish a character fix your problem? By default the first character would be used like it is today, but if this new button was in the toggled state SubExtractor would wait for the Enter key before reading the character in the text box.

We could make a hotkey like F2 toggle the state as well so it would be easy to switch modes if you have a section of Latin characters in the subtitle.

That's it. I tried to tell that but I couldn't :D. I'll wait for your new pro. Thank you. :thanks:

[GrofLuigi]

Hi Tappen,

First time user here. The program hangs if confronted with DVD (unprotected) ripped with DVD Decrypter with option "File Splitting: None". I create all my rips in such a way to have only 1 Vob and 1 Ifo file per DVD; most of media players treat it as DVD if I open the Ifo file, so I know it works. I can also rip subs from them with other sub rippers.

GL

[Tappen]

GrofLuigi: this is just something that I never considered. It's not allowed on real disks of course so I had no reason to think someone would extend the IFO file format this way. Maybe I can look into it. DVD Decryptor stopped development 8 years ago and doesn't work on many disks these days so it's hard to get too interested in supporting it. Also, SubExtractor does support idx/sub files.

[GrofLuigi]

Tappen, this was not a negative comment, just a FYI. I wanted to try your program and maybe give some comments/suggestions/praises, but I couldn't even start. I will try with DVD disks later.

Most other tools parse correctly this kind of ifo files (and yes, few also choke/act buggy, but don't crash), so I keep doing it this way since forever. :p

Examples of parsing and OCR-ing properly: as old as SubRip and as new as Subtitle Edit. Playing properly: VideoLan. Playing with bugs (not always able to select/display subtitles): Mpc-HC.

*Edit: It started working with 1Gb vobs. The produced file is excellent, I see no errors in it! This is a very good program! :eek:

GL

[Chetwood]

most of media players treat it as DVD if I open the Ifo file
Usually it's only needed for standalones cause software players handle regular vobs just fine.

[errantkkn]

Hi Tappen,
I have a question, that is could I backup the ORC library, so I can bring it to another computer without typing again? And how?

Thanks.

[GrofLuigi]

Another FIY, for anyone interested: It worked with .sub/idx extracted with VsRip from those non-split VOBs, but didn't accept the .sup extracted with PgcDemux.

The result of OCR was again outstanding.

GL

[Tappen]

errantkkn: On the bottom of the first page of options there's a checkbox labeled "Use Program Exe Location". Check this and hit OK. This will move the OCR database to the directory where you unzipped the install file. You can then copy this OcrMap.bin file to other machines. Either rename it to OcrMapOrig.bin and substitute it into the zip file before you install, or copy it on top of the OcrMap.bin file on the other machine AFTER unzipping, running the program for the first time, and checking "Use Program Exe Location" on the new machine.

[speedoflight]

Hi, im new here and i just discovered this great ocr tool.

I just have 1 "tiny" problem. My languaje is spanish (well, not exactly, it is Castillan, that is not the same as universal spanish but anyways it has the same alphabet) and in some subtitles (yup, i dunno the reason but it happened in 2 / 4 subtitles i tried), i have the problem of the "i" and "¡" characters, even if im using the last version of the program. So far, thats maybe the only problem i encountered, but when it happens, i need to use another ocr program because sometimes it is almost impossible to correct all the "¡" for "i" or viceversa. Btw, im using subextractor to ocr blu-ray subtitles.

I dunno if there are still problems with that fix, or it is not yet 100% fixed.

O , another thing, it will be great to be allowed to make the preview screen bigger (or to be allowed to maximize the window or something like that), i work on a big resolution pc screen and it is soooo small that i cant see anything at all

Anyways, thx for this OCR tool, the best i tried so far.

[Tappen]

errantkkn: I've put a test release up on CodePlex (http://subextractor.codeplex.com/releases/view/97076) with the option to wait for an Enter key before taking OCR matches. It is toggled with the check box or F2. Please try it out.

[Tappen]

speedoflight: Did you check the "i and ¡ per movie" checkbox in Options? The fix for this issue isn't always on as it would slow down other users so I made it an option.

[speedoflight]

speedoflight: Did you check the "i and ¡ per movie" checkbox in Options? The fix for this issue isn't always on as it would slow down other users so I made it an option.

Not exactly, i had another version of the program before i downloaded the last one posted in this post, that had the option implemented, and it was checked, but still didnt work.

So, i discovered this thread and i donwloaded the last version posted in the link of the first post, thinking that maybe i had the wrong one. It happens the last version i downloaded from here doesnt have the option, like the other one. I assumed the fix was just cored in the program and there was no option anymore.

There must be an option anyways??

Thx in advance (and btw sorry for my cute english ^^).

[errantkkn]

Perfect. Thanks Tappen.

[Tappen]

speedoflight: The only release that has this fix is "Release 1029", which is currently what you get from the big download button from the main page of the project (http://subextractor.codeplex.com/). Once you unzip this download, run the exe, and in Options, on the "Special Characters" page, check the option "Check for i and ¡ per movie (Spanish Specific Fix)" before you OCR.

Also, if you'd send me the sup files that are giving a problem I can try to work on this myself. I'll send you a private message.

[speedoflight]

Yes, thats the one i tried, and im still having issues with "i" and "¡". Well, since right now i dont have any sub working on, when i had one making trouble i will send it to you =), but one thing for sure, the version of the program im using is 1.0.29, the one donwloaded from the codeplex page.

Thank u.

[fantasmanegro]

first try, just to say, i really like the app, and the results, anybody knows how to parse just idx/sub file?.

[Tappen]

fantasmanegro: On the menu "Jump to" -> "Load Subtitle File" I think it's named (will be different next release so I can't look it up right now). Click Browse and open the IDX file (can multi-select if you want). OCR, Spelling and Save steps as usual.

[fantasmanegro]

thank you very much!

[Tappen]

Thanks for the great comments, TheRancher. I agree with you about the Cons and am working on it.

"Program window is kind of big": This is a UI style that I prefer (not burying features in menus) but it gives things a cluttered look. People have compared my form designs to 747 cockpits. I'd like to simplify but I'm worried that any feature not on the main window will never be found.

"Lack of localization": On the todo list some day soon.

"You have to extract all the content from the DVD": Not really true. On the "Re-Encode Tracks" dialog by default the "Create Movie File(s)", "Create Subtitle Data File(s)" and "Create DgIndex Files(s)" options are all checked. If you just want to OCR the subtitles and are not going to re-encode the video and audio just choose "Create Subtitle Data File(s)" and de-select the other 2.

"Unknown application name.": Yep this is a source of confusion. After I added BluRay support leaving DVD in the name seemed misleading. I'm trying to move to just SubExtractor but it takes a while.

[Chetwood]

Is there an option to skip an entire subtitle when batch OCR'ing?

[Tappen]

The "Preview and Correct OCR Matches" looks at the entire database but only lists the matches that were used in OCRing the current movie. Deleting a match (or split) will apply to the current and all future movies. If you find a mistake in a sub after exiting SubExtractor you can run the OCR again on that movie's bin/idx/sup file and then use this feature to fix the mistake in the database so it doesn't appear in future movies.

Hard-subbed video files are a whole different issue. Separating the subtitles from the background would be at least as difficult as all the work done so far on this project. So sorry there's no plans to start work on that.

[Tappen]

Chetwood: there currently isn't a way to skip a file when doing batch OCR. Maybe in the "Jump To" menu I could put an item to "End OCR and Skip to Next" if there is a next file to go to (part of a batch).

[Tappen]

That means if I lose a DVD and .bin file or I don't know on which DVD I made the mistakes, I won't be able to fix them? I know I am exaggerating, but it would be nice if you could preview the whole database and eventually edit some wrong characters, apart from showing the subtitles in the current movie. A mistake can easily be made without even knowing it.


Most letters have over 1000 matches, some over 2000 just in the default database. If I showed them all you'd probably never find an error. It just seems like a pointless feature without applying a filter such as per-movie to narrow down the list.

[speedoflight]

Hi, about the spanish problem with ¡ and i characters, i need to say that i tried with other ocr tool (the one that comes withing subtitle edit) and even if the tool itself is awful, i changed the color palette and the tool perfect recognizes ¡ and i. The only problem is that the ocr tool is maybe one of the slowest i tried (i spent about 5 hours and i didnt finish half of the movie...).

I tried that in dvdsubextractor, but the change of the palette didnt work. So im saying this to see if there is something we can do about it. Since its very weird, but this is the only ocr tool i tried that works fine in most cases. Subrip doesnt work with the subs im trying, and other ocr tools cant just open idx files or the results are awful epic.

Cheers.

[Tappen]

speedoflight: I've been looking at the problem. I haven't given up. Someone sent me a Bluray subtitle where I reproduced it, and if you could send me a sub/idx with the same problem it could help me come up with an answer.

[speedoflight]

I already sent u a pm days ago with the subtitle, and u said u will look into it, but that it will be hard to fix. It was on sub/idx format...

[deco20]

1) In step "Choose Tracks" there's a listbox "Subtitle Tracks". Scrollbar doesn't work, because control is disabled. If there's more subtitle tracks, I can't see if there's a track I want to OCR.

2) In step "Spelling and Spacing" after OCR when spellchecking is complete, there's no way to undo if we made a mistake in last word.

3) I don't use options "Create movie file" and "Create DgIndex file", so maybe there's chance to add to the settings setting in which user will be able to choose what kind of files program should create, so I won't uncheck it every time.

[Tappen]

deco20:
1) I always save all the subtitle tracks so there's no reason to select them. It's working as designed
2) I'll look into it. I suspect you're right though.
3) Good point. I should make those remembered checkboxes from movie to movie.

[deco20]

deco20:
1) I always save all the subtitle tracks so there's no reason to select them. It's working as designed.

I wasn't writing about selecting, but simple scrolling the listbox. Now it's disabled. Sometimes I don't know if there's my language and I want to be sure it exists.

[pball]

Firstly, I love your program. I've done a bunch of encoding and subs have always been a pain and your program makes it much better.

There are just a few things I've noticed while ocring this sup file.

Would it possible to let you edit matches you have entered. I've hit the wrong key a few times and it be nice if it was possible to change the matched character instead of just deleting it.

Also after making changes to the manual matches could there be an option to not rescan from the beginning. I screwed up more than a few times and it'd be nice to wait and rescan after doing a complete first run.

I did a spell check of the subs I just ripped and other than having f in place of some g's (which was probably my bad) it did quite a good job. Very few I and L mistakes.

One last thing, would it be possible to have user adjustable sensitivity of the ocr? I had to enter around 10 different matches for just about every letter and the sub character can't of had more than a few pixels difference some times.

[speedoflight]

I agree with the first pball point, but due to the fast ocr engine of this program, usually u dont need to wait that much until u encounter the required match again.

Agree with the second point as well.

About the last point, i think the sensivity of the program is just amazing, the problem is, vobsub subtitles , for example are a pain no matter wat ocr program u use. I am in the same case scenario, sometimes, i cant use the program because it doesnt recognize the "¡" and "i" characters of my languaje alphabet, and maybe with more sensitivity , as u say this can be solved, but i dont think is that easy. And its very normal the need to enter the same character over and over again, specially on vobsub subtitles. Happened to me a lot of times. But still, this program is incredibly fast. With other ones, u will need even 1 hour to make a ocr recognition (not vobsub, with vobsub subtitles u will need even more lol), with this program u only need 10 minutes or not much more... amazing. It will be perfect if the problem im having wasnt there =)

[pball]

I just want to add the first thing I used this to ocr was a bluray sub file and as I said before I had to enter in many duplicates. But just now I did an older dvd with LOADS of signs and I was amazed at how well it did. Not only did I only have to enter 20-30 or characters but something I didn't realize at first, it locates signs. The last time I attempted to ocr these subs with subrip I had to practically type the whole thing out and the end format was horrible with all the signs and such.

[speedoflight]

I noticed something that seems to be a little bug. When i save the subtitles in .ASS format, the program always produce some duplicate lines adding the position the subtitles should have on the screen, instead of just adding the position counter on the proper lines.

When they are only 4 or 5 lines, its easy to fix, but when it comes out 50 lines, its almost a frustrating task. In those cases i need to save in on .SRT cuz i waste too much time on em. But of course i prefer the ASS format with the correct positions on the screen.

Need to say, it doesnt matter if i deactivate the program default subtitle positon correcion or the other options, the result is always the same.

Im a doing something wrong?

[Tappen]

I don't understand the problem, speedoflight. Could you copy a line or 2 of an ass sub with the problem highlighted for me?

[errantkkn]

Tappen, could you make the software ability to extract subtitles from video files, like *.avi

[Chetwood]

Technically it's possible but I've never ever seen an AVI contain subtitles.

[errantkkn]

It's like a hardsub muxed video, not a video contain sub stream

[sneaker_ger]

@errantkkn:
Hard-subbed video files are a whole different issue. Separating the subtitles from the background would be at least as difficult as all the work done so far on this project. So sorry there's no plans to start work on that.

[speedoflight]

I don't understand the problem, speedoflight. Could you copy a line or 2 of an ass sub with the problem highlighted for me?

I didnt explain too well anyways, here it is a screenshot, and this is wat the program does ->

http://imageshack.us/a/img341/5729/clipboard01rk.jpg

As u see, its very weird.I tried opening the subtitles with subtitle editor and aegis subs, both with the same result.

[speedoflight]

Now, im having a lot of trouble with "I" and "L", the program doesnt ocr right these characters on vobsub subtitles. Im getting more bugs every subtitle i try...

[An IP BreAKDoWN]

I'm trying to OCR some subs that have underline in them, it would be nice if SubExtractor had an underline option. If you need an example here is the sup (http://www.filefactory.com/file/5q4hotorvknl/).

[Tappen]

speedoflight: ASS Subtitles can have overlapping, non-sequential time-stamps. This is normal and expected.
Only SRT subs must be non-overlapping. So what you highlighted isn't a bug.

If you go through the spell-check step of the program it should disambiguate I and lower L characters for you. They are often the same pattern so can only be told apart after the OCR.

An IP Breakdown: Underlining is on the list of issues I'd like to fix (http://subextractor.codeplex.com/workitem/625). Thanks for the subtitle example. For now you can deal with it by Splitting the underline from the characters and Ignoring the underline pieces but I agree it isn't ideal.

[radigast]

Your program is by far the best out there. However, with this (http://www.sendspace.com/file/csq9ra) BluRay subtitle set, ! is recognized as either '. or l. I reset all characters and ensured that, when prompted, each letter was reprogrammed correctly. I never received a prompt to OCR !. I'm not sure if this is something on my end or whether it is a bug that can be fixed in the image database within the program. Suprip can recognize the ! with no problems, so proper recognition is possible. Again, love the program. I'm hoping this can be sorted out!

Thanks again!

[deco20]

Strange thing: I always have to type chars like "l", "o", "O" and special characters like ",", ".", "-" although I typed them before in ocr of the same font.

[speedoflight]

speedoflight: ASS Subtitles can have overlapping, non-sequential time-stamps. This is normal and expected.
Only SRT subs must be non-overlapping. So what you highlighted isn't a bug.

If you go through the spell-check step of the program it should disambiguate I and lower L characters for you. They are often the same pattern so can only be told apart after the OCR.



I know about ass subtitles. But the thing is , those lines should not be overlapping each other. They should be regular lines, one after another. If i check the original subtitle watching the blu ray, there is no overlapping in em. So i dont understand y the program makes that. And wat the hell, the screen i posted is the text section, not the coding of the ass. U cant see anything related to those overlapped lines in the coding of the ass. Its just the program that writes that for some reason.

Besides, i tried the ocr with, for example the ocr tool of subtitle edit, and it gives me a good ass subtitle without those overlapped lines, well not really , it gives me a subtitle that i can save it as .ASS after ive done with the ocr.

I already tried the spell-check a lot of times about "I" and "L" without result. I deleted all the subtitle ocr images and i started from the beginning, with the same result.

I dont need to say, its a similar problem with "¡" and "i", it doesnt matter if i delete and i start from the beginning, even with a different palette, the program will not recognize those characters.

[nautilus7]

Hi, I have some problems with greek letters. SubExtractor splits some letters no matter if i tell it not to and this makes ocr impossible. See the screenshots below.

http://t.imgbox.com/abwXHoct.jpg (http://imgbox.com/abwXHoct) http://t.imgbox.com/abuySDWh.jpg (http://imgbox.com/abuySDWh) http://t.imgbox.com/adr3DfTv.jpg (http://imgbox.com/adr3DfTv)

The letters are "Γ" (greek capital "gamma"), "Η" (greek capital "heta") and "Π" (greek capital "pi"), either normal or italic. Why even if i remove these specific splits, they are introduced again? I have deleted the map files (the one that comes with the program and the other in the /users folder) with no change.

This happens in several pgs files i have, one is here: http://www.sendspace.com/file/kgd5po

[Tappen]

nautilus7: There is an automatic split function which always runs on unknown characters. If both the left and right of the split make letters that aren't in the list of known bad auto-split characters or make a pair of characters known to be bad (rn = m for example) we don't stop and ask, we just do the split. I'll need to add an option to allow the user to add characters to this list to fix this problem based on their own language's characters.

[nautilus7]

Thanks for the lightning fast respond. I understand the logic and i will welcome such an option. I have to say though that this "problem" didn't occur in the past at least with Greek characters. I've used subExtractor to ocr a lot of subtitles with 100% success over the previous few years. If I recall correctly automated splits didn't take place in Greek characters at that time. Is that a recent change in the split logic?

[nautilus7]

Also I've noticed that some letters (greek and english) are split in a bizarre way. Anything you can improve here?

http://t.imgbox.com/abfqrxlE.jpg (http://imgbox.com/abfqrxlE)

[Tappen]

Nautilus7: Removing Γ Ι ι Ξ Π from the auto-split lists will probably solve your problems. Actually just Ι and ι will solve all the problems you've shown me so far I think. Splits using Iota gives the same bad results as the latin I and i characters to no one's great surprise. I'll make the change in the next release (soon).

[nautilus7]

Hi, there are a few letters more.

It's greek "π" (lower case pi) which is detected as double "τ" (lower case tau), because the horizontal line in "π" can exceed the 2 vertical lines and then there's big similarity.

Also, it's "Θ" (capital theta) which is detected as "O" with a "-" after it.

"Ξ" didn't gave me any problems whatsoever, but i guess removing it wont harm.

[nautilus7]

Tappen, I got this exception log yesterday, but i can't remember what i did to cause it.

Exception thrown at 20/1/2013 5:12 πμ
Parameter is not valid.
System.Drawing
at System.Drawing.Bitmap..ctor(Int32 width, Int32 height, PixelFormat format)
at DvdSubOcr.BlockViewer.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WmWindowPosChanged(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
Exception thrown at 20/1/2013 5:12 πμ
Parameter is not valid.
System.Drawing
at System.Drawing.Graphics.FromImage(Image image)
at DvdSubOcr.BlockViewer.RedrawBlockPicture()
at DvdSubOcr.BlockViewer.UpdateBlockPicture(BlockEncode block, IEnumerable`1 otherSelectedBlocks, IEnumerable`1 allEncodes)
at DvdSubExtractor.OcrBlocksStep.LoadCurrentSubtitleData()
at DvdSubExtractor.OcrBlocksStep.FindNextOcr()
at DvdSubExtractor.OcrBlocksStep.reviewButton_Click(Object sender, EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[Tappen]

OK will look into it.

[Overdrive80]

Hi, thanks for you magic app. When I select "Split by cell" and I want go back to "Split by chapter", button is desactived. I dont know if its a bug, but I report. Thanks

[nautilus7]

Tappen, I saw you made some changes to the source repo. Are those you mention above? Will you compile a binary, because it seems that my installation of vs 2012 lucks some components and can not compile the source code.

[Tappen]

This weekend Nautilus7 (Jan 26-27)

[Tappen]

Overdrive80: A chapter is made up of 1 or more whole cells. So once you split by cells there are no more chapters.

You can go back to the first page where you Browse for the DVD folder and hit the "Reload" button if you make a mistake splitting a track up.

technical note: There are really only cells on a DVD. What I call a new chapter is when a cell starts with a different, non-continuous, time-stamp (internal numbering of video/audio/subtitle/system packets) from the previous cell. Many but not all DVDs follow this convention.

[nautilus7]

Looking forward to it! Thank you.

[nautilus7]

Hi tappen thanks for the updated verion. It's an improvement to the previous version regarding wrongly split greek characters, but there are still some issues.

Greek lower case p (π) is still split in 2 lower case t's (τ). Maybe you should put "τ" in the "cheap split symbols" or "very cheap split characters" lists as well? Or it wont do anything?
There also a case where greek capital P ("Π") is split in "Γ" and "Ι" (gamma and giota respectively), though mentioned in the above mentioned lists (I read you code - lines 52, 53 in subconstants.cs file).

52 public const string CheapSplitSymbols = "\'\".,:-—_|„!ΓΠπΘГҐПӨг";
53 public const string VeryCheapSplitCharacters = "lIÌÍÎÏiìíîïj][ΙιΞІЇӀ";

Here is a screen with examples:

http://t.imgbox.com/adx7blWU.jpg (http://imgbox.com/adx7blWU)

[nautilus7]

There are also some cases where greek capital theta ("Θ") is still detected as "O-".

[rizu]

edit->Nevermind found the italic checking :)

[Betsy25]

@Tappen,

just have a little question. I ran a Blu-ray .sup file through the extractor and made the necessary inputs to make it through. However after I had the file saved as a *.srt, I noticed SubExtractor made quite a lot of "mistakes" by not seeing the <space> before a lot of words starting with a "j" (when on an italic line). I tried redoing the file but now it doesn't prompt anymore for any unrecognized items and runs through the 100% at once.

So, in short, I would like to delete the "database" meaning some file so it will once again prompt for corrections, however I do not wish to delete my "learned words" database, which is already fairly complete for the dutch language.

which file to delete please ?:helpful:

[Tappen]

nautilus7: I'm sorry I made a mistake with this fix for you. This is what comes of taking a break from the code for a couple of months. Of course I should have added the characters into the CheapSplitSymbols list that are the RESULT of the split, not the source. If the code knew what the character was before it was split then it wouldn't do the split in the first place. I'll fix this with a new version today.

Betsy25: The database is the OcrMap.bin file. You can see it's location on the first tab of Options. However, the database isn't your problem: you need to go to "Advanced Word Spacing" which is accessible from either the Spell-Check page or the File Save page (the last). Select lower case "j", and increase the "Normal Left" and/or "Italic" left values until the spaces to the left of the "j" characters in the samples look good. These adjustment values will last till you shut down SubExtractor.

J and j, as well as y and a couple of other characters are pretty common spacing errors. I've played with the defaults to improve the results in the majority of cases but many subtitles still draw these characters with either too long or too short tails and throw off the calculations of word spacing.

[nautilus7]

Tappen, thanks! Looking forward for the fix to this great ocr application.

[Betsy25]

Betsy25: The database is the OcrMap.bin file. You can see it's location on the first tab of Options. However, the database isn't your problem: you need to go to "Advanced Word Spacing" which is accessible from either the Spell-Check page or the File Save page (the last). Select lower case "j", and increase the "Normal Left" and/or "Italic" left values until the spaces to the left of the "j" characters in the samples look good. These adjustment values will last till you shut down SubExtractor.

J and j, as well as y and a couple of other characters are pretty common spacing errors. I've played with the defaults to improve the results in the majority of cases but many subtitles still draw these characters with either too long or too short tails and throw off the calculations of word spacing.

Thanks for the help Tappen, I've found and increated the "default before" space for the italic j, and now everything appears fine, HOWEVER, I can not go to the next step, the save as window, it's greyed out ?:confused:

EDIT: Sorry, I found out i had to click the "Previous Step". Evenything is working fine in this fantastic converter.

[speedoflight]

Tappen, did u look into the program .ASS problem of my post??

Well, it doesnt have too much trouble anyways, i just save it on SRT instead and then i convert it to ASS. But the problem with "I" and "L" and "i" and "¡" continues. Impossible to fix. Suprip makes perfect converted subtitles, with no problem on these characters. Thats the ocr tool im using now, since (sadly) subextractor seems not be able to handle it.

[Betsy25]

Perhaps just a little feature request ?

Could it be possible to have a DEL hotkey in the Options/Special Characters screen, so we don't have to constantly do -Select Item -> press "Remove from list" ?

I for example, only have dutch subtitles, from which a lot will clash with the default English words, cleaning it up now takes a extereme lot of clicking and selecting, while with a "del" hotkey, this can be sped up quite a bit. Ideally, it would be great when able to navigate the list using the arrow keys & have the DEL key for deleting items.

[73ChargerFan]

Nice app, thanks. I was trying to review a SUP track from the BD director's cut of Dark City, which crashed every other program I tried. With SubExtractor I could see that the images were animated pictures (cue cards with text, but expanding) that I could ignore.

Thank You!

[Tappen]

Betsy25: any reason you wouldn't want just a "Remove All" button? You could then train the program in just the Dutch spelling words.

I found the initial words during the OCR of the English subtitles in my personal DVD/Bluray library. I didn't use a dictionary so there's nothing special about this list.

[Tappen]

speedoflight: I think I have an idea what's happening in your case with "i" and "¡": the best fit character isn't being chosen sometimes when OCRing Bluray subtitles due to an error in the algorithm. It's pretty complicated to fix and I've not found the time to sit down and work on it yet.

I still haven't reproduced the 'I' and 'L' problem. My guess is that it's somehow related to being a Spanish subtitle because others aren't seeing the problem.

[Betsy25]

Betsy25: any reason you wouldn't want just a "Remove All" button? You could then train the program in just the Dutch spelling words.

I found the initial words during the OCR of the English subtitles in my personal DVD/Bluray library. I didn't use a dictionary so there's nothing special about this list.

That would be really handy for non-English people like myself !:)

[Tappen]

nautilus7: Please try out release 31b and let me know which Greek characters are still being split incorrectly and into which sub-characters. Thanks.

[nautilus7]

Hi, I am already testing for 1 hour... :p

It's seems that most problems are gone now. Only "Θ" (capital theta) is being split in "O" and "-" in some caes. Actually this not detected as split. It never did. Maybe because the horizontal inside bar is not attached to the outer circle of the letter, but has some space. That's causing theta to be ocred in "O-" without any indication of being split up.

If I come accross any other issue I'll let you know.

May I ask for a feature as well? In the "Correct OCR matches" window, where you can overview all matches for the current subtitle stream, it would be great if a window pops up and asks for confirmation before saving any changes. I accidentally hit the "remove all ocr matches for this movie" button and had to start over.

[Tappen]

I hate confirmation dialogs in general but in this case you have a point.

[nautilus7]

You can also put a check box in the options page to choose whether to ask for confirmation or not. This way the pop up won't be shown to users they don't want to.

[Tappen]

The "Θ" problem will involve some special code to detect. Not difficult to do, though, so I should be able to write it fairly quickly. I'm glad you told me it wasn't an auto-split problem because that was making my head hurt.

[nautilus7]

Yeah, although, when I first reported this specific character problem (in the previous page) I wrote "it's been detected as.." and not "been split up to...", I thought I should be more clear now. :p

Anyway, glad to hear that it is easy to fix.

[Thunderbolt8]

got a bit of trouble with this file here: http://www.sendspace.com/file/42kue9

apparently some "l" (small letter L) are misrecognized as 1 or I. but I cant seem to delete those letters from the training list of that movie, because no small letter "l" is listed there (after k comes m there). I also cant change this during the spell check stage, because e.g. if the word villain is recognized as "vi1Iain" then spellcheck will only give me suggestions for the first 3 letters vi1, but not the complete word. and in case of words with only one wrong letter "l", those are not even suggested to me for spellchecking.

I already did delete all trainings for 1, I, L, i, l and also its italic variants from that movies trainings list. but it didnt change anything.

[Tappen]

Thunderbolt8: It doesn't matter if a letter is trained as l (lower L) or I (upper i). The program treats them as the same internally and sorts out which is which in the spell-check stage because they're so often identical.

There is a problem when 1 (one) has the same pattern as l or I, however. I've seen this myself. Currently there's no way to fix this problem and the spell-check step will ignore the 1 characters. I think in the few cases where this came up for me I restarted the OCR and trained the 1 character as I then edited the final file by hand because 1s are pretty rare.

[Betsy25]

Tappen, Is adding a "Remove All" button in the Options/Special Characters window on the to-do list please ?

[Tappen]

Betsy25: Yes I already implemented it. I just put up a minor version build 1031c for you.

[Overdrive80]

@Tappen Would be possible hardsubs rip with your soft??

[Tappen]

Overdrive80: It would take a huge amount of work, sorry. The OCR algorithm I'm using doesn't work at all well on anything but soft subs so I'd have to start over.

[nautilus7]

Hi, Tappen, any news regarding the special code for Greek upper case Theta ("Θ") detection?

[Betsy25]

Betsy25: Yes I already implemented it. I just put up a minor version build 1031c for you.

Perfect ! Thank you so much for implementing this.

[Thunderbolt8]

Thunderbolt8: It doesn't matter if a letter is trained as l (lower L) or I (upper i). The program treats them as the same internally and sorts out which is which in the spell-check stage because they're so often identical.

There is a problem when 1 (one) has the same pattern as l or I, however. I've seen this myself. Currently there's no way to fix this problem and the spell-check step will ignore the 1 characters. I think in the few cases where this came up for me I restarted the OCR and trained the 1 character as I then edited the final file by hand because 1s are pretty rare.I think in case of .srt sub style it wouldnt be much of a problem. but in this case its a file for which I need to retain the subtitle positions at their original position via .ass and therefore I have lots of 1 in the position information of every line and also in case of italic markers -.-

[Betsy25]

I have problems with the "Monty Python - The Meaning of Life" DVD.

Suddenly the subtitle times become negative for the rest of the movie :
345
00:20:54,422 --> 00:20:56,698
<i>kwamen er overal kinderen.</i>

346
00:-11:-56,-386 --> 00:-11:-51,-994
DE ZIN VAN 'T LEVEN
DEEL II - GROEI EN ONDERWIJS

347
00:-11:-51,-106 --> 00:-11:-46,-953
"En zagen zij andermaal de kamelen
voor het derde uur.

348
00:-11:-45,-666 --> 00:-11:-42,-195
"En zo gingen de Midianieten
op weg naar Ram Gilead...

349
00:-11:-41,-746 --> 00:-11:-39,-948
"in Kadesh Bilgemeth...

etc...

When I'm in the "Select subtitle data file" page, I can enter 346 in the Index field, and there it shows the correct timing, but when the file gets saved after recognition, the timings are negative from sub 346 onwards...

Please, what could be wrong ?:confused:

[Tappen]

nautilus7: sorry no news yet - busy at my real work
Betsy25: you'll have to send me the source sup or idx/sub file to look at - I'll send you a private message

[nautilus7]

No problem. Real life is priority.

[Sm3n2]

Hi,
Your program is very good but something is missing. Are you thinking to add .SON support ? Because I extract subtitle from TS to SON file (with .bmp images) and I OCR using Subtitle Edit. If your program was able to do this, it will be great.
Cheerz.

[Thunderbolt8]

got a subtitle file in which the letter "P" is automatically split into "F>". I can remove this split in the review and correct OCR matches dialog, but it automatically gets recognized as such again. so theres no way for me to actually change this within the program.

would it be possible in the future to add the option to define any character at any line of a subtitle file? then these problems shouldnt really matter much any more, because then you could simply tell "P" to be recognized as character "P".

[Tappen]

Thunderbolt8: Try getting rid of the letter > (untrain all instances of it) to fix this particular problem. I'm not sure what you means by "define any character at any line of a subtitle file". One thing I can do is allow you to turn off auto-splitting temporarily, but you'd have to notice that there's a problem for that to work.

[PowerGamer]

Can you add an option to produce

00:01:02,981 --> 00:01:05,108
RALPH: <i>My passion bubbles
very near the surface,</i>

instead of

00:01:02,981 --> 00:01:05,108
RALPH: <i>My passion bubbles</i>
<i>very near the surface,</i>

when outputting in srt format?

[Tappen]

PowerGamer: I think the first should probably be the default. I'll look into it. I hate adding options like this that are hard to understand.

[nautilus7]

What if the 1st line is written in italics and the 2nd in regular letters?

[Tappen]

nautilus7: I just meant that if the last character on 1 line and the first character on the next line are both italic, I shouldn't close then open the italic tag. It shouldn't make a difference but there are lots of funny renderers in the world.

[Thunderbolt8]

Thunderbolt8: Try getting rid of the letter > (untrain all instances of it) to fix this particular problem. I'm not sure what you means by "define any character at any line of a subtitle file".what I mean is basically to take any line of the first screen when you still can look through all lines manually before you start the OCR process with the next step and then take such a line to the OCR table of the next process, be able to click on any character of the original .sup file and assign a character from the table to it manually.

or something like that ;)

[Tappen]

nautilus7: please try the test release I put up on Codeplex for your Theta issue

[Tappen]

Thunderbolt8: I just got rid of <> and () from the auto-splittable character list. It's safer and I don't think it will hurt people much if at all.

[nautilus7]

nautilus7: please try the test release I put up on Codeplex for your Theta issue

I will later today. Thanks.

[nautilus7]

Tappen, it works fine now with the subs i tested. If i come across any other issue, i will report. Thanks.

[Tappen]

speedoflight: I've made progress on the accuracy of the HD OCR algorithm. i and ¡ characters are recognized properly in the cases I've tested. The release is 1032b on Codeplex.

[nautilus7]

I guess this will be helpful for the greek giota ( ί ) as well. Right?

[Tappen]

nautilus7: Probably will help. I'll add it in 1 extra place to be sure. Also I'll change the option text to read "Spanish or Greek". The latest is 1032c

[nautilus7]

Thanks. As always, I 'll let you know when I come across any issues.

[Tappen]

Thunderbolt8: You might want to try 1032c for the various subs you've found with errors (I'm thinking of the 1 recognition problem in particular). It may be fixed.

I didn't get the problem file from Sendspace in time to test it myself

[Chetwood]

Just OCRed a new batch with 1.0.3.1 and I cannot undo the last item that was spellchecked?

[Tappen]

Chetwood, That's a bug that's been around forever and I've never bothered to fix. Maybe now that someone other than me has found it...

[Johnny_B_E-Work]

Hello, could you please integrate functionality to manually enter string of letters? Sometimes no matter what I try SubExtractor will not split correctly / select the correct segments of particular letters. Ignoring such symbol cannot be used because if the same segment is part of any other letter, it is ignored too. If there is any other way how to solve it, I could not find out how...

I hoped it would work this way: I hit F2 (Enter mode), select with keyboard arrows the section of text (for example the whole word), manually type in the letters, hit Enter = unfortunately this leads to training of the first letter typed only...

[Thunderbolt8]

got a file here in which the hyphen "-" still remains after removing SHD stuff in case when [] is used for brackets instead of ().
http://www.sendspace.com/file/lbj4ez

e.g.

- blaa. blabala
- [shd stuff]

-->

- blaa. blabla


should be easy to fix though I guess.

[Tappen]

Johnny_B_E-Work - I'm pretty sure it's possible to use the current split function to solve your problems, but how you use it is not intuitive at all. (Adding multi-character matching is just too hard with the code as it is).

A couple of hints for this (admittedly too difficult to use) feature:

1. If you're getting a lot of characters that need to be split you should try a different Palette. Sometimes the shadows or highlights around characters are selected by mistake resulting in characters merging.

2. Select the largest part of the character you need to split before hitting the Split button. The Split feature will work on the biggest chunk selected, not the 1st.

3. Tricky characters running together will sometimes require you to split multiple small pieces off a big chunk one at a time, hitting the Split button again and again to do so, then connecting pieces back together to do the match. If this is happening be sure to double-check if reason 1. is the basic problem.

[Tappen]

Thunderbolt8: I'm not sure I ever remove a hyphen if it is in the original subs and part of a line that I'm keeping. I'll check what's happening though.

[Johnny_B_E-Work]

I figured how to use split function already and in most scenarios it works fine. The problem I sometimes have is with particular special letters of my native language (Czech) like " ď " and " ť " and believe me that no matter what I try I am unable to achieve satisfactory results (I can provide some examples if you are interested).
It's a shame that multi-letter enter function cannot be implemented (like old SubRip has) - sometimes it is really much easier just to type in the letters manually and move on. If this could be solved somehow then your program would be absolutely perfect (already MUCH faster than SubRip, more user friendly and all in all great to work with)

[Chetwood]

Chetwood, That's a bug that's been around forever and I've never bothered to fix.
I usually don't run into this but this time was the end of a long batch of subs and I clicked on the wrong word so it'd be nice to be able to correct it too.

[Tappen]

Johnny_B_E-Work: Please send me a bin or sup file with the problems so I can see for myself using the Private Message system here or on Codeplex. I might be able to see a way to solve things that is easier to implement.

[Ghitulescu]

According to your web page, you plan an installer. I believe people that need to work with subtitles have enough experience to unpack a ZIP ... haven't they? :)
Is the Italic thing solved?
What about musical characters? They must be Unicode, as none of them are in ANSI (some are in the extended ANSI code pages) ...

[Thunderbolt8]

got a case here in which

Bangkok:
good-time city,

is recognized as SHD and therefore "Bangkok:" is removed if SHD removal is checked.

maybe it would be useful only to consider these cases as SHD if the trigger "Xxxxx:" is not the only thing a line consists of? Or are there examples in which such cases (e.g. "Name:" and the blablalbla in the next line) are really SHD subs?

[Tappen]

There are definitely cases where the speaker name: is on one line and the text is on another I'm afraid.

[SJX]

My first post... I'm in the process of converting my dvd's to H.264 compressed files. I wanted soft subtitles so I started using Subextractor instead of encoding hard sub's with Freemake. BUT, I'm unable to produce any good subtitle files and I cannot figure out what is wrong. I have tested 5 dvd's now. The most common problem is that the sub timestamp keeps resetting during the movie. For instance this DVD "Umur"

155
00:02:55,560 --> 00:02:58,155
En mie tarvitse. Sie saat sen.

156
00:00:00,800 --> 00:00:03,679
<i>Umur kävi luonani.</i>
time stamp is resetted after 3 minutes
and again after 5mins
218
00:04:42,280 --> 00:04:45,398
Minä rakastan sitä miestä.

219
00:00:03,200 --> 00:00:06,398
<i>Lähetin Umurille kaktuksen,</i>
<i>en ruusua.</i>

this results that I have a bunch of overlapping subtitles from various parts of the movie. Usually this resetting seems to happen at 3-5min interval, but one title had 10 minute intervals.

[Chetwood]

What tool do you use to encode to which container? Usually there's no need to extract subs.

[SJX]

What tool do you use to encode to which container? Usually there's no need to extract subs.

I don't see how the encoding is related to subtitle extraction, but:
-for encoded (hard) sub's, I use Freemake all the way
-For soft subtitle i.e. subtitle files that could be turned on/off I use DVDFab to rip, SubExtractor to OCR sub's, Freemake to convert.
For final format I usually use MP4 (H.264/AAC) since my Sony tv and PS3 are able to play those directly.

[Tappen]

Are you using DVDFab to convert to an unencrypted VIDEO_TS folder or to go to a single main movie file? If the 2nd choice that explains the problem. For accurate subs SubExtractor should be used to convert from the VOB files to an mpg file (which can be used in Handbrake or whatever) and a bin file (SubExtractor sub format) which can be OCR'd.

SUB/IDX files created from DVDFab main movie rips will have bad timing regardless of which OCR problem you use. DVDs have chapter breaks which reset the time to 0 inside them and DVDFab does not clean this up when it rips to a single file unfortunately.

[Chetwood]

MMh, you're sure about that? Cause so far I had no timing problems when ripping main movie only. In fact, Fengtao stated on the forums that the only way to be sure all protections are gone (if not they may cause out of sync subs) is to rip the main movie only rather than the complete disc.

[SJX]

Are you using DVDFab to convert to an unencrypted VIDEO_TS folder or to go to a single main movie file?
I extracted only the main movie to harddrive, not the extras. But of course it resulted an unenctrypted VIDEO_TS folder.

But I think I found the reason. I was careless about encoding the mpg and others - really didn't understand the point until you explained. I had not necessarily deleted the old mpg's and others or even skipped mpg encoding. Now I managed to get sub's without time stamp issue.

It would be nice to have an button in Subextractor "delete aux. files" after successful extraction since those are no longer needed.

[Johnny_B_E-Work]

I extracted subtitles from THX 1138 DVD and I get weird timings in the output SRT file which makes it unsuable:


278
00:06:19,027 --> 00:06:21,861
...and speed it up by four times...

279
00:06:22,027 --> 00:06:27,421
...you'll hear "Stabat Mater"
by Pergolesi.

280
00:-11:-12,-022 --> 00:-11:-09,-666
I was so afraid.

281
00:-11:-05,-462 --> 00:-11:-03,-584
So alone.

282
00:-10:-55,-782 --> 00:-10:-53,-313
I wanted to touch you...


...and so on until the very end:


1280
-07:-07:-57,-139 --> -07:-07:-53,-589
We only want to help you.

1281
-07:-07:-52,-979 --> -07:-07:-49,-781
This is your last chance.

[Tappen]

Johnny_B_E-Work: Are you OCRing from a SUB/IDX file combo? If so can you open the IDX file in a text editor like Notepad and see if the problem is there? Otherwise let us know - maybe there's some other suggestions.

[Johnny_B_E-Work]

Are you OCRing from a SUB/IDX file combo?

No, directly from DVD e.g. *.bin file was created by SubExtractor.

[Johnny_B_E-Work]

I think I figured it out. I originally extracted (the longest) track 1 which - as I just discovered - is not the feature film only but feature PLUS some behind the scenes which are played during the movie e.g. they are inserted on thy fly hence the first 6 minutes were just fine, after that first extra scene was inserted and the timing got screwed.
Now I extracted track 2 (shorter, feature only) and there it seems to work just fine.

[Thunderbolt8]

is it actually possible to adjust the spacing for italic characters? the list apparently only shows non-italic characters and all line examples at the left side are also non-italic ones.

[Weirdo]

Great application, thanks! Doing a quick test now, and it seems to have problems with the medium DPI setting (125%), several windows/dialogs are cut off.

[Tappen]

Thunderbolt8: There's 2 radio buttons to select this: one labelled Normal and the other Italic - roughly in the middle top of the Word Spacing page.

Weirdo: It looks like the Options dialog has most problems, I should probably make it bigger. I didn't see any issues with the main pages.

[rhaz]

Hi. Very nice tool indeed. It worked fine with one subtitles from one DVD, but now I am having problems with other subtitles from other DVD. It gives incomplete characters.

I.e. characters 'š', after OCR it gives me only 's' on some words, but not all. Why? I am sure I selected it correctly when OCR'ing (also I cannot find how to delete the saved characters from tool's memory so I could try to reapply it for the character again and see if this gets fixed).

Also there's misspelled word 'hilite' in couple areas in the tool. I think it should be 'highlight'.

[rhaz]

Nevermind. Fixed it in 'Review and Correct OCR Matches'.

[Tappen]

rhaz: I use the spelling hilite because I'm a programmer and that's the way it's spelled in most software APIs. I't s usually done that way because it's shorter and therefore requires less valuable screen space. Just a personal quirk, sorry.

Also - if you didn't notice you can use the arrow keys to select the multiple pieces making up a character like š. I was surprised when I added that feature how faster it made even English OCR (ij!: characters).

[rhaz]

Ok other problem.

http://i.imgur.com/lKDgs10.gif

It is letter 'ą' and ',' so it's 'ą,', but as you can see split doesn't work because it doesn't touch together or something. And I tried many ways splitting it, but as you can see it gives me just 'a', not 'ą' when splitting nor 'ą,'.

So if I select it as 'a', the word will be misspelled and would probably affect the rest of the words having same characters.

[rhaz]

Another problem. I get error from one DVD subs. I tried extracting subs with various tools like MeGUI tool, VSRip and with your tool, but your tool doesn't even let me to extract subs, buttons are grayed out.

http://i.imgur.com/Ar4Vtl8.jpg

Subs plays fine on DVD though. Don't get it what's the problem. It works fine on SubtitleEdit, but I prefer your tool because it is waaay more accurate and less of work and saves tons of time compared to SubtitleEdit.

There's (http://www72.zippyshare.com/v/75761929/file.html) subs attached

[Thunderbolt8]

tappen, could you please add something to show the subtitle line number of each line listed when using advanced word spacing adjustment? when trying to find a certain problematic line in combination with a letter which is used a lot (e.g. non-italic "y") it can take quite some time to find the correct line in that list. if theres some indication of the subtitle line say in the top right corner of the presented subtitle picture, then it would be much faster to find your way through the list till you find the line you are looking for.

[Tappen]

rhaz: in the first problem, I think you can just match a, then split the accent+, character, then when the accent lights up for matching select it and the a and type in the real character. This shouldn't cause other problems.

in the 2nd problem - I'll look into it.

Thunderbolt8: good idea. I'll see if the line number can be added

[Taskforce]

Hi, thanks for the terrific tool. I was wondering if it would be possible to change things so that when you are at the saving dialog, it defaults to the directory of the open file, instead of the last directory used (at least when using save as).

It does this when you first open the tool for the first OCR job. However, if you're doing a series, and are doing multiple episodes separately, it defaults to the last directory used instead of the one you've loaded the next ep from. I know this really isn't anything major, but would help speed up processing episodic series a little bit by saving a few mouse clicks.

[Weirdo]

Isn't it possible for italics, to simultaneously press CTRL+(key) or is it my Logitech keyboard that won't accept it (I get the Windows error sound)? I'm forced to press Ctrl and use the mouse for italics, which is a bit of a time-waster for the otherwise ultra-smooth SubExtractor.

edit: Just saw the space bar option, nice workaround.

[speedoflight]

Is the actual app working with older subtitles, even if they are still .sup?? im trying to ocr a pair of subtitles and the results are just incredible horrible.. just like an alien languaje.. xD. I make the ocr,i totally waste about 1 hour , and when i go to the subtitle file, not even a word is correctly ocred. I dunno wat i am doing wrong.

Here is the sub, download it and investigate, i dunno wats going on.. (It is spanish languaje)

https://www.dropbox.com/s/mtcat2dz9sah44j/Extra%203%20-%20spanish.sup

EDIT: Well, since it looks like nobody is answering, i just discarded this program. Im using subtitle edit instead, that , surprisely, it produces far better results (almost in my case). In fact, this program started really well, and i used it in a couple of subtitles, but i realized that is not really a good ocr program, too many bugs, too many errors and it looks like the author is not supporting the main languajes after all.

[rhaz]

Yeah I agree this thread is very dead. It's two weeks I'm still waiting reply for my subs issue I mentioned in #453 and attached subs below. I can't even OCR those on Subtitle Edit, gives too many errors (at least it opens though).

[Thunderbolt8]

Thunderbolt8: good idea. I'll see if the line number can be addeddid you already find some time for this? would be really helpful to me now :thanks:

btw. could please also change in the spacing adjustment view that the point of focus jumps back to the first line each time you are doing any change of spacing? if you want to find out how much change/clicks are needed for a character in order to have a visible change in the text and you are dealing with a characters which features many examples in the subtitle, then you have to find and click yourself down to that instance you are inspecting each time again to see if anything has changed. thats just annoying. it would be nice if the point of focus just stays where it is after committing a change.

[Thunderbolt8]

got a movie in which the exclamation mark "!" is recognized as " '. " (apaostrophe+fullstop). deleting both the characters for fullstops and apostrophes & commas doesnt help, deleting all characters of the movie doesnt do anything either. any idea what I could do here?

[Tappen]

Thunderbolt8: I'm very busy with other things at the moment. It'll probably be a month or more before I can tackle the list of issues.
I also can't help with your apostrophe+fullstop problem, sorry. I can see how this would happen but the software just isn't set up to deal with the case where the top part of an exclamation mark is identical to an apostrophe. Normally the apostrophe is a much smaller character.

[Thunderbolt8]

its not that big of a problem, I can just copy & replace all the wrong ones for good. just want to be sure that doesnt translate over to other movies when saving characters. so I deleted all characters for this movie after OCRing.

about the rest, better late than never :D

[Thunderbolt8]

are difference made in spacing of letter also saved in the OcrMap.bin?

[Tappen]

are difference made in spacing of letter also saved in the OcrMap.bin?

No they are not.

[Thunderbolt8]

so which files Id have to save in case of a reinstalling windows to keep spacing changes I made?

[Tappen]

They're saved in the user.config file. It's in the directory that's somewhere like:

C:\Users\AccountName\AppData\Local\DvdSubExtractor\DvdSubExtractor.exe_Url_vxbiiw1ruyu1vjmfw1fhh1ifophvjesa\1.0.1.3

[johnsonlam]

I got some idx and sub file, that's why I start using Subtitle Extractor, but soon I found it may be still ANSI? Chinese input method simply can't active in the program, however I can copy a Chinese character (or Japanese, in Unicode) into the "Manually Enter Character" and it works fine. Since it's not optimize for this kind of block character, always have duplicate, suggest a fuzzy logic in percentage (+-2%) or tolerance can be added.

And I still trying to figure out how to save the "trained data", since a series of anime simply have same subtitle.

Great program, thanks!

[Tappen]

johnsonlam: Could you try downloading version 1032d, the latest beta, and checking the box "Wait for Enter Key"? This might solve your problem for entering Chinese characters.
There is a fuzzy logic component but it is only active on HD (720p or higher) subtitles.

[johnsonlam]

johnsonlam: Could you try downloading version 1032d, the latest beta, and checking the box "Wait for Enter Key"? This might solve your problem for entering Chinese characters.
There is a fuzzy logic component but it is only active on HD (720p or higher) subtitles.

Thanks for your advice. I'm using that function already.

A bit more research, it's the program eat up the special-key combination, so Windows OS can't switch to that input method, using alternate way to enable the language switch seems solved the problem.

Too bad the great fuzzy logic not enabled in lower resolution.

[Chetwood]

Chetwood, That's a bug that's been around forever and I've never bothered to fix. Maybe now that someone other than me has found it...
Will it be fixed in the next release? The more subs I rip in a row the more likely it becomes I mistype. Just happenede to me again and there's still no Undo for the last item. Thx.

[rhaz]

Hi. I have a question. I was using this tool for two months now and I have collected big collection of various characters, versions and etc.

So now for some reason when I start OCR'ing all my collected characters are gone from OCR Matches and I have to start from scratch for no reason. Why's that?

[Tappen]

The OCR Matches dialog only shows the ones that have been used in the current file (either added or re-used).

I can't think of a way to show all the matches (100s for each character just in the starting database) in a way that would be useful. So it's heavily trimmed down.

[Wizzu]

Very glad to have discovered this nifty app.

Really helps.

Congrats Tappen! And thanks! :cool:

[Wizzu]

After having processed about 25 movies subtitles files with this app, I really want to congrat the developer again.

Everything is so well thought-out, ergonomy is top-notch (I really love the [ctrl-arrow] shortcuts to select characters parts)

Thanks for making my life easier! Where's the "donate" button?

[Thunderbolt8]

tappen, just asking: are you still busy? or do you think you might have a bit of time soon to work on those little improvements in the advanced word spacing tab I suggested?

[Thunderbolt8]

is the fullstop character "." for some reason excluded from the adjusting spacing effect? got a line in which a pistol clip e.g. " .22" is set right next the to the word preceeding it e.g. "the.22" and even when I increase left spacing of "." to 20 and right spacing of "e" to 20, nothing happens (apart from all other "e" characters getting set apart from their following letter)

[Tappen]

Sorry Thunderbolt8 I'm still busy at work. I think you're right and there's a rule about "." (and other punctuation) spacing that over-rules the word spacing options on the left side. It saves so many mistakes and causes so few I'd hesitate about removing such a rule.

[Thunderbolt8]

well problem in this case usually only occur in that cases if weapon calibre " .xx". its rather easy to find such cases by searching for digits 0-9 (even though its annoying having to do this potentially for each file). so if such a chance really led to more problems, then Id say its better to keep it as it is.

[CoolRaoul]

Hello everybody
Just discovered this application which I thought could help me to convert DVB Subtitle streams from my records made with USB Stick TNT Recorder

I demux streams using "TS Doctor", then with Subtitle Exxtractor use file->open to open the .sub file.

Unfortunately after a few seconds a popups appears with "no subtitles found" error message.

What I'm doing bad?

**edit**
I may upload the .sub file somewhere if it can help someone to diagnose.

[Tappen]

The problem is that this tool doesn't support .sub files, sorry. Only .sup and idx/sub pairs from dvds

[CoolRaoul]

The problem is that this tool doesn't support .sub files, sorry. Only .sup and idx/sub pairs from dvds

Oh typing error: TS Doctor generate ".sup" files not ".sub"

Would you like a sample one?

NB: answering to random questions when posting in this forum is definitively not easy for newbies like myself!

[CoolRaoul]

Maybe this thread is not the "official" place to discuss about SubExtractor issues.

In that case could someone give me the correct link?

[Thunderbolt8]

it is but the creator is busy atm with other things as he said.

[CoolRaoul]

it is but the creator is busy atm with other things as he said.

Ah ok,
I'll wait then..

(And since I'm not receiving forum email notifications I will have to check this thread periodically)

Note: answering to random questions to validate post here is an horrible thing for non specialist like me!!!

[Thunderbolt8]

would it be possible to add "er..." and "erm..." to the SHD removal as well?

for " er..." " erm..." the space and the the er(m) need to go

for ",er..." ",erm..." the comma and the er(m) need to be removed

for "Er..." "Erm..." at the beginning of a sentence or line, it should be removed and the first letter of the next word italicised.

I hope that wont break anything.

edit: well not to sure about the beginning of a sentence or line thing, because it could potentially lead to slight loss of sync for that specific sentence/line, depending on how much of a break is there during the speech. thats a boundary we perhaps shouldnt cross.

[Betsy25]

Regarding pure OCR, this is by far the best tool out there, it would be a shame if this project would just die out.:(

[Chetwood]

Word. Apparently Tappen is busy with RL so I guess, we'll just have to wait.

[Thunderbolt8]

tappen, do you know have time for little improvements?

[CoolRaoul]

Regarding pure OCR, this is by far the best tool out there, it would be a shame if this project would just die out.:(

Unfortunately I've been unable to make it work yet and did'nt find any alternative neither.

It's a pity if the projet would be be discontinued as it seems.

[rhaz]

Hi. Using this great tool for over a year now. Still using 1.0.3.2, why no updates? Anyway, I have a question. No matter what ♪ I use for that music symbol (ALT 13) when I save it to .srt, all ♪ symbols becomes ? marks. Really pain in the ass then to replace each ? with ♪ manually. So why's that? Maybe it would work if it saved to UTF8 format.

[deco20]

Hi. Using this great tool for over a year now. Still using 1.0.3.2, why no updates? Anyway, I have a question. No matter what ♪ I use for that music symbol (ALT 13) when I save it to .srt, all ♪ symbols becomes ? marks. Really pain in the ass then to replace each ? with ♪ manually. So why's that? Maybe it would work if it saved to UTF8 format.
Definitely, you have to save it with UTF-8 encoding.

[rhaz]

How do you do that? There's no option to choose UTF8 when clicking Save as.

[deco20]

How do you do that? There's no option to choose UTF8 when clicking Save as.
Go to Options and uncheck "Store Srt files as ANSI (instead of UTF-8) Codepage".

[rhaz]

Edited. Nevermind, solved. Used better ripper to extract subs first.

[Thunderbolt8]

why cant some HD DVD .sups actually not be opened with this prog?

[Thunderbolt8]

is SubExtractor actually open source? if so, would anyone like to take over and work on this project? there hasnt been any development for over two years now and there a few bits and pieces which still could need improvement. I still like to use this tool because its really fast with OCRing and has good removal of hearing impaired stuff for certain type of subtitles which need to be .ass in order to retain their original screen line position.

[Thunderbolt8]

any updates here? still my most reliable & fastest goto program to OCR subs.

[locotus]

any updates here? still my most reliable & fastest goto program to OCR subs.

Plus 1, hope Tappen is still on line.