Mask out hardsubs of a video based on clean source?

[nateify]

I have two video sources for an obscure anime which has no softsubbed release. The two sources are very close in PQ (different encodes) and have the same resolution.

I cannot use traditional OCR sub rip methods as the subtitles are low quality and both the fill and stroke differ in color greatly from scene to scene. I haven't figured out a way to mask and extract the hardsubs for me to process externally.

Take for example this frame:
without sub:
https://thumbs2.imgbox.com/b4/52/Y4vgswy7_t.png (http://imgbox.com/Y4vgswy7)
with sub:
https://thumbs2.imgbox.com/c2/d8/64zZx0O1_t.png (http://imgbox.com/64zZx0O1)

And with this script:

raw=FFMS2("raw.mkv",fpsnum=24000,fpsden=1001)
hs=FFMS2("hardsub.mkv",fpsnum=24000,fpsden=1001)
Overlay(hs,raw,mode="difference")


The resulting subtitle is still barely legible and I would not be able to use it in OCR:
https://thumbs2.imgbox.com/68/15/g5GQWte0_t.png (http://imgbox.com/g5GQWte0)

For this scene, the subtitle fill and stroke are close to grey:
without sub:
https://thumbs2.imgbox.com/f9/df/jA8yueZw_t.png (http://imgbox.com/jA8yueZw)
with sub:
https://thumbs2.imgbox.com/ba/40/gTch9Qu5_t.png (http://imgbox.com/gTch9Qu5)


And the overlay produces a nearly illegible result:
https://thumbs2.imgbox.com/61/5f/nKOrsnNh_t.png (http://imgbox.com/nKOrsnNh)

Changing the Overlay mode to from difference to subtract yields worse results:
https://thumbs2.imgbox.com/0d/11/VwTpODOh_t.png (http://imgbox.com/VwTpODOh)

My goal is to find a way to mask and extract just the subtitles and perhaps I can have an easier time preprocessing it with something like imagemagick to end up with more legible images that can then be OCR'd. Any help would be appreciated.

[buddha9]

I have two video sources for an obscure anime which has no softsubbed release. The two sources are very close in PQ (different encodes) and have the same resolution.

...

My goal is to find a way to mask and extract just the subtitles and perhaps I can have an easier time preprocessing it with something like imagemagick to end up with more legible images that can then be OCR'd. Any help would be appreciated.

Hi, maybe this InpaintDelogo - advanced logo removal & hardcoded subtitles extraction (https://forum.doom9.org/showthread.php?t=176860&highlight=hardsub) ?

[VoodooFX]

This thread was posted almost 5 years ago...

[LunaRabbit]

At one point I am pretty sure I had this script (or at least one without timing) but it seems lost to the sands of time. How many episodes do you need to OCR? Is it only two? Because I do have scripts for everything but the first two episodes.

If you'd like I'd be willing and could probably find the time to manually type these back up. For me it'd be quicker than fooling around with trying to OCR it. If you'd like some help feel free to send me a PM.

I would personally just OCR what you can and manually fix the errors. But it would be nice to know a better way if anyone has any ideas. The post above seems like a good start.

[LunaRabbit]

This thread was posted almost 5 years ago...

Indeed it was! Here I was trying to be helpful. Oh well maybe he might come back around.

[VoodooFX]

I would personally just OCR what you can and manually fix the errors. But it would be nice to know a better way if anyone has any ideas.

You can use InpaintDelogo to extract images with subs and OCR with SE or AI.

[buddha9]

You can use InpaintDelogo to extract images with subs and OCR with SE or AI.

Hi @VoodooFX,
what is the right workflow to do this?

# Can be used to remove hardcoded subtitles

Thanks

[LunaRabbit]

You can use InpaintDelogo to extract images with subs and OCR with SE or AI.

In this case I was just going to re-translate for the OP. Since I'm familiar with the source material.

I know these AI tools have come a long way and most people probably don't care. But 3 or 4 months ago someone was using some kind of AI to push out scripts about a day before our team translated them. I was not very impressed with the quality of the work I saw. Especially considering a human translator was doing a much better job in a manner of just a couple of hours. Our day or two delay wasn't from waiting on translation to be finished. It was from everything else involved.

I don't think it's there yet. At least not for some languages. It doesn't understand nuance and it simply fails on less commonly used terms (e.g. slang).

Hi @VoodooFX,
what is the right workflow to do this?

# Can be used to remove hardcoded subtitles

Thanks

I have not used that particular filter. But the idea is instead of throwing away the detected area you export it to a text file. From there you'll want to import it into whatever subtitling application you use and clean up after it. As no OCR is perfect ime.

[VoodooFX]

Hi @VoodooFX,
what is the right workflow to do this?

1) Created a dynamic mask of the subtitles.
2) Then using that mask: Replace HC subtitles from a clean lower resolution video or inpaint.


In this case I was just going to re-translate for the OP.

I can "re-translate" for you: OCR is not a translation task.


But the idea is instead of throwing away the detected area you export it to a text file. From there you'll want to import it into whatever subtitling application you use and clean up after it. As no OCR is perfect ime.

This doesn't make any sense.