Filter to make Clearer Subs

[SacredCultivator]

Dear Readers,

I hope this is the right section to ask this...

I was just wondering, if I'm encoding Subtitles (.ass) into a file (H264 in .mp4), the preview window comes out 'clear' but the completed product shows a lot of pixelation in the subs.

My script is as follows:

MPEG2Source("__vid__")
Tomsmocomp(-1,1,1)
#tfm(pp=5,cthresh=3,mode=0).tdecimate()
Crop(left,top,right,bottom)
Spline36Resize()
Removegrain(2)
gradfun2db()

And I've tried disabling both the Remove + grad and no luck.
^Also, for the "resize" filter, I'm not too sure if the different ones make any difference, so if there is one that is suggested to replace the one I use, please share.

So I was wondering if I'm missing a filter that can enhance it to be like the preview.

Thanks.

[Comatose]

I'm confused. The TextSub line is missing, and left,top,right,bottom aren't defined, so I'm getting the feeling you aren't giving us the entire script :P

[SacredCultivator]

Ooops that's my template ><

So yes all the information is fille din for the video

MPEG2Source("__vid__")
Tomsmocomp(-1,1,1)
#tfm(pp=5,cthresh=3,mode=0).tdecimate()
Crop(4,0,-2,-0, align=true)
Spline36Resize(720,400)
Removegrain(2)
gradfun2db()

textsub("Path.ass")

Well taht should give the general idea.

Not sure if you can really tell from the pictures:
meGUI Preview:
http://img367.imageshack.us/img367/9620/previewip0.th.png (http://img367.imageshack.us/my.php?image=previewip0.png)

Encoded Video:
http://img370.imageshack.us/img370/4770/encoddeou8.th.png (http://img370.imageshack.us/my.php?image=encoddeou8.png)

---
If encoding method matters at all.. well I'm using Constant Quality at 19 for the video.
^Attempted a Constant Quantizer at 21 and the subs came out smoother, but I'm not too familiar with that method, as the fileout is MUCH larger than Constant Quality, but again the Subs are near perfect on that.

Attached my Preset up so you can check the H264 settings if needed.

[Comatose]

How did you take that screenshot? Use DGAVC + Avisynth and make sure you get the exact same frame (the entire frame please).
AvsP may make life a bit easier than using a media player to take the screenshot, if you don't have it already :P

[kemuri-_9]

subtitles as a principle introduce a large sharpness/contrast compared to the video it's going onto,
which often will introduce blocking/artifacts due to that.

if your end result is for viewing on the PC, you can just mux the video and subs together into a .mkv
only will need to hardcode the subs on for an end result that will be acceptable on standalone players.
(i'm not an expert on SA players, but afaik, none support .ASS)

[thetoof]

into a file (H264 in .mp4),
afaik, DGAVC does not support mp4 yet...

and yeah, softsubs are great to have a great video quality and unaltered subs. For hardsubs, adding a semi-transparent greyish border sometimes helps reducing artifacts because the transition is not too brutal.

[[P]ako]

Change of font, some fonts (stylish ones) introduce more artifacting than others, and add more bitrate.

[SacredCultivator]

Indeed I want to continue using the .mp4 container and not .mkv, although I do know that for .mkv for hardsubs it is much clearer.

And I've tried a few more test encodes using Constant Quality, and when set at 1, it's perfect, but of course the size is ridiculous and now trying to see how high of a number I have to use to get it to be near perfection, although this doesn't matter much as the filesize is quite large in comparison to when I used 19.

So apart from changing the container / encoding method (As Constant Quantizer works), there aren't any filters that might help improve the quality of the subs to make them look more 'integrated' into the video?

[roozhou]

That's x264 who blurs your subtitle. Try using negative deblock settings, e.g. -3:-3.

[SacredCultivator]

Hmmm sorry, which filter would that be? As I'm not too sure if I have it or not =X

[wyti]

it's not a filter, but a setting of x264 (--deblock -3:-3 in the command line)
I think that a filter who improve the quality of the subtitles doesn't exist, cause the only way i know to descrase bitrate, is to blur yourself the subtitle...
the subtitle are in a perfect quality when they are hardsubbed into the video (before encoding), and any filter that you may apply to them would give a worse result before encoding.

[Zarxrax]

Maybe its just me, but I think the encoded image looks sharper than the preview image, or at least just as good.

[SacredCultivator]

@wyti: Thanks will give that a try right now.

@Zarxrax: Yes it is 'sharper' but in doing that, it makes the pixelations stand out more.

(And yeah wasn't toos ure how to capture it, as when taking those screenshots, it isn't as noticeable as watching it (FullScreen).)

---
EDIT:
Okay tried with the deblocker and it didn't really so much in respect to improving the subs.

And the two filters I use (Remove / grad) don't really affect the Subs do they as again I didn't notice much when I disabled the two.

[DigitalDeviant]

The subtitlles in the preview image appears stretched vertically so there is some resize going on somewhere after the subtitles. This might account for some blurring. Also as [P]ako said, the fancier the font the higher the chance that it won't look good after encoding.

[SacredCultivator]

Hmm not too sure on the stretching given that I did set the "textsub" at the end of the script so not too sure where that resize would come from.

[wyti]

Maybe during encoding ? did tha output have the same resolution than your source ?

[SacredCultivator]

Yes same resolution as what I resized it to.

[DigitalDeviant]

OK, nvm it was the resizing done by the image host. Can you post a screenshot at the native resolution since your pics are 761 and 911 pixels horizontal and it's hard to try to compare pics of such different resolutions?

[SacredCultivator]

Okay here you are:

Preview in meGUI:
http://img136.imageshack.us/img136/9146/23247155da1.th.png (http://img136.imageshack.us/my.php?image=23247155da1.png)

Encoded File:
http://img396.imageshack.us/img396/741/snapshot20081215113109oo3.th.jpg (http://img396.imageshack.us/my.php?image=snapshot20081215113109oo3.jpg)

Yes the video is grainy as well, but more so just concentration on the Subtitles.

And again only 'filters' I use thusfar are
Removegrain(2)
gradfun2db

Settings wise, I don't have deblocking enabled, and AQ is set at 1.0

---
Sharktooth suggested I turn on deblocking at -2,-1, and disable AQ and this did help although the filesize was increased.

[wyti]

Sharktooth suggested I turn on deblocking at -2,-1, and disable AQ and this did help although the filesize was increased.
he's right, aq have difficulties to keep sharp edges, but may help with the overall quality of the clip.

you can disable psy-rdo / psy-trellis too if you use them,

otherwise, the only things you can do is increasing bitrate / use slower settings / more filtering the source to make it more compressible (stronger denoising...)

[SacredCultivator]

Hmmm well trellis is at 0, and the rdo is at 1.

Will try disabling RDO and try, and for the AQ

1.0 usually is okay? As I'm not quite sure what to use..

As for denoiser gonna try using UnFilter(-5,-5), as I don't want to incrase encoding time too much.

[wyti]

for AQ is what you want, 1.0 is ok in most scenario with real time content, but tend to make edge ugly, so if you care much about your subtitles than your movie, keep it at 0.0 but if you care much to details in the movie, use it until 1.0 (you can use smaller value if you want, but higher than 1.0 isn't really recommanded for your source)
PSY-RDO is different of RDO, it's the PSY-RO Strength, and the PSY-Trellis value who should be both 0.0 (it's like AQ, you will get more detailled movie with them activated, but worse subtitle)
and stupid question, but have you an up to date Megui / X264 version ? (this can improve both speed and quality, so you should update if you don't have the last version)

[SacredCultivator]

Okay, runninga f ew more encodes right now.

And for the RDO i meant the PSY, so will set those 0.

And for the AQ, I guess I'll try different numbers till 1 for it.

I'm using the most up to date version, although for the x264, still using 1051, as there are no updates for it yet so haven't updated that, which is 1055 I think.

[[P]ako]

Okay, this is a karaoke. Try using the vsfilter.dll that comes with Aegisubs, it renders text better than Gabest's old versions.

[SacredCultivator]

I'm pretty sure I did, as our Typesetter told me to update my vsfilter, so I updated that and used it, now if you say I have to add a line into the script then that I'm not sure..