Fuzzyness around Subtitles with Xvid

[Zerox20]

I am using Koepi's latest Build for Xvid and whenever i do encodes (2 pass with 158 MB set for 2nd pass int) The picture looks pretty good (except for some noise) but the subtitles have a lot of fuzzyness around them like colored blocks/noise I've tried using 2d cleaner with vdub and I am running avisynth filters:
asharp(1.5,2,1,false)
UnDot()
TemporalCleaner(2,10)
Convolution3D(preset="animeHQ")
ConvertToYUY2()
unfilter(-5,-5)
Limiter()

Yet I still get it, is it because of the lower filesize I am setting my Xvid too?

[N_F]

I assume you're using permanent subtitles a la Vobsub? When in the script are you applying them?

[Zerox20]

I am using Textsub .ssa Files. and textsub in Virtual Dub to apply the subs. I run them first, then I run warpsharp, then I run 2dcleaner in vdub.

[WyldeF1r3]

mosquito noise around hardsubs is a common problem. Is the 158 filesize what you're using for this one file or for all files as a standard size to always hit? Is 158 the optimal size (according to gknot bitrate calculation) or are you compressing too much? Mosquito noise around subs, which did not exist in the original source is mainly due to the contrast between sub and image and a lack of bitrate (though any good pixel averaging filter should be able to minimize or remove these blocks around the subs).

[N_F]

WyldeF1r3 have already said most that needs to be said. Permanent subtitles are rarely a very pretty sight

I don't know exactly what you're using from warpsharp, but if it's a sharpener it's possible that makes it a bit worse.

A way (I'm not saying it's necesarily a good or the best way) to avoid this is to use "temporary" subtitles. You can either save them in a container like ogm or you can have them as separate files, letting the player lay them over the picture. This usually looks great.

[mf]

Use AVISynth's TextSub. You can stay in YV12 and speed up the encode and improve quality.

[Zerox20]

I am actually doing that now and vdub keeps illegal operating!
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\LoadPluginEx.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\asharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\UnDot\UnDot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Convolution3D\Convolution3DYV12.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\UnFilter\UnFilter_Avisynth_2.5_Alpha\UnFilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\temporalcleaner\temporalcleaner.dll")
LoadPlugin("C:\Program Files\Gabest\VobSub\textsub.vdf")
AviSource("clip1.avi")
TextSub("script1.ssa")
asharp(1.5,2,1,false)
UnDot()
TemporalCleaner(2,10)
unfilter(-5,-5)
Limiter()
Convolution3D(preset="animeHQ")
ConvertToYUY2()

When I do the pluginEX.dll it illegal operatins if I don't include it vdub says its not a valid avisynth 2.5 Filter.

[Kyo]

The loadpluginext.dll isn't a plugin is a helper to load *old* version plugins into avisynth 2.5X, you only need the last VSfilter.dll from the subtitles forum and nothing more, it has the textsub on it ans is YV12 like mf says.

[Zerox20]

well good news I got it working but I still got all the crap around subtitles, Only way I can fix it is to set the 2nd pass - int to like 400 MB which is just not an option. Any other suggestions?

[Asmodian]

I would do as others have suggested and use non-"burnt-in" subtitles. Just put the text subtitle file in the same directory and called the same name as the video (eg. clip1.avi and clip1.srt) and dvobsub will overlay it on the video while playing with a direct show media player, which looks very nice. You are seeing compression artifacts from the sharp edge of the burnt-in subtitles. If you really want burnt-in subtitles I would suggest reordering filters in your script to:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\asharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\UnDot\UnDot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Convolution3D\Convolution3DYV12.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\UnFilter\UnFilter_Avisynth_2.5_Alpha\UnFilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\temporalcleaner\temporalcleaner.dll")
LoadPlugin("C:\Program Files\Gabest\VobSub\textsub.vdf")
AviSource("clip1.avi")
UnDot()
TemporalCleaner(2,10)
unfilter(-5,-5)
TextSub("script1.ssa")
Convolution3D(preset="animeHQ")
Limiter()
ConvertToYUY2()

Why are you using two sharpening filters? I would also suggest just one (unfilter) maybe on stronger settings. If you use virtualdubmod you could also eliminate the ConvertToYUY2() and gain speed and color detail (because of the lack of color interpolation during a color space conversion).

[ssjkakaroto]

zerox20 does this happen if you put
TextSub("script1.ssa")
in the last line of your script?

btw how were you able to do such great encodes for a-kraze without using avisynth?

[Zerox20]

Well just messing with settings you can obtain awsome results this is first time I have ever come across the subtitle problem, and I can't use the overlay method They are .ssa .ass I tried using directvob with it earlier and it was not working at all for me period. It made myv ideo like 3 minutes long and a black screen heh. So they pretty much have to be burnt in.

[N_F]

I would also suggest that you move the textsub() down in your script. In the script you provided above you are actually sharpening the subtitles edges, something that isn't a very good idea and may likely give just the problems you have.

[ssjkakaroto]

that's what i suggested, cuz i never got any problems with subtiltes when they were at the end of my scripts

[Zerox20]

I have tried moving the subtitles to the end of the script, middle of the script, They used to be in Virtual Dub Originally, so Vdub is executed after avisynth filters right? So Since They have been moved all over and still get results. I mean one way to fix it is to change the 158000 second pass to 570000 :P but we all don't want that now do we lol.

[WyldeF1r3]

well, a hardly desirable, but very effective, method of fixing this is to load smartsmoother into virtualdub (dunno if it'll work if you do it with avisynth), even at lower-end settings (3,10) or (3,15) will wipe most of it out (as well as some of the detail you want to preserve). 2dcleaner might work as well.

I see you've had c3d loaded after textsub (which can possibly clean up some of this noise [despite the fact that this is compression noise, not filter, created noise]). Since convolution3d YV12 version doesn't have a temporal cleaning aspect, you might want to use loadpluginex to load the YUY2 version, take the speed hit, swap your colorspaces, and see if the temporal cleaning helps out.

What colors are these subtitles (yes that is a factor), is it all subs, or just some? Have you encoded this to huffyavi before compressing (I hope so since I assume you've been working from a YV12 compressed file already)? If not, you're recompressing a compressed source, so you might want to consider it.

Also, when compressing to xvid are you using any of the extra options, b-frames, etc...?

[Zerox20]

Sadly I am new to Xvid but to answer your questions..

I am using a Compressed DIVX 5.02 QB 100 RAW, that I compress into XVID, I have never messed with b-frames because I Was told not to get into them heh, as for the subs I am already running 2d cleaner and it doesn't seem to wipe it out, I imagine it is not wiping it out because it runs the filters then compresses right? so since its a compression error, the filters run --> Xvid compresses Noise appears. (I could be wrong I am not the most experienced encoder) as for Sub colors, They are for the most part white with black background.

[ssjkakaroto]

don't know if this will help but you can try enabling chroma optimizer on xvid, it was supposed to work better on sharp edges

[Asmodian]

Are you sure overlay subtitles don't work, how hard have you tried to get them to work? :) I have never had problems with it, are you using any unusual direct show filters, what OS, player, and video card? Do you know your average quants?, you can get an idea of the quants used by turning on the OSD in ffdshow (if you don't have the debugview output from xvid) but you have to decode xvid with ffdshow for it to be able to report quants. I don't think it is possible to get totally clean sharp edges in xvid without post-processing. (though avg. quant around 3 or less looks very nice, it will still have some ringing around sharp edges with post-processing off)

I believe chroma optimizer is supposed to reduce (eliminate?) color stair-stepping artifacts by doing chroma interpolation so it shouldn't help in this instance (but is generally a good idea with anime).

[Asmodian]

I though you might want some Xvid settings to try, If not just ignore this.

I like:

H263
VHQ 1
Chroma motion
q-pel
B-frams OFF (-1)
Chroma optimizer
with the quants not capped (2-31) and altCC (on defaults) if the quants are high. Dropping the resolution also helps really a lot if the quants are too high.

[sungey]

@zerox20
i think your are compressing the video too much ... there are times where 158mb isnt sufficient for an episode. You need to lower the first pass size somehow ... or increase the target size.

[ssjkakaroto]

zerox what's the first pass size?