compression noise around subtitles driving me nuts..

[por_que2k]

I'm using XviD 04102002-1, and every time I try to encode any anime episode, there is severe compression noise around the subs. I'm using the XviD setting

[por_que2k]

I'm using XviD 04102002-1, and every time I try to encode any anime episode, there is severe compression noise around the subs. I've tried using the XviD settings from doom9's guide and from on this forum, but nothing has helped.

I'm telling the file to be 200 megs for 24 minutes, and it's encoding off of a perfect looking hufyuv. I'm using VDub 1.4.13.

My exact XviD settings:

Global TAB:

motion search: 6
Quantizer: H.263
Max I-frame dist: 300
Min I-frame dist: 10
no lumimasking

Quantization TAB:

Min I-frame quant: 2
Min I-frame quant: 8
Min P-frame quant: 2
Min P-frame quant: 8

Two Pass TAB:

I-frame boost %: 20
Below I-frame dist: 6
I-frame bitrate reduction %: 30
High bitrate scenes %: 25
Low bitrate scenes %: 25
Bitrate payback delay: 240
Payback w/ bias
No Hinted ME

Alt Curve TAB:

Agression: Medium
High distance from avg %: 200
Low distance from avg %: 90
Strength %: 50
Automatic bonus bias.

Max bitrate: 10000
Max overflow improvement %: 60
Min overflow improvement %: 60


Am I doing something greviously wrong or what?

[NuclearFusi0n]

try DirectVobSub (check the doom9 guides)
good luck :)

[por_que2k]

I dunno how some of it got posted once and the rest in another post...

But anyway Nuke.. I don't want to use soft subs. That's why I am concerned about these nasty subs.

[NuclearFusi0n]

Originally posted by por_que2k
I dunno how some of it got posted once and the rest in another post...

But anyway Nuke.. I don't want to use soft subs. That's why I am concerned about these nasty subs.
try using the latest developer build, see if it improves quality with b frames and qpel and chroma motion
also, if you are encoding anime, are you filtering it? try convolution3d to make it more compressible

[por_que2k]

But the noise is still there, just ot as bad. Thanx for the tip on convolution3d though, it quite noticibly improves compression and smoothness of the image, although at quite a bit of time cost (but i have more than enough of that :)

I'm still stumped on these subs. How come no other xvid anime I have seen, thats the same filesize/length ratio, has this nasty compression noise around the subs? :(

[Koepi]

Try switching to h263 quant matrix instead of mpeg quantization, this helps the ringing quite noticably.

EDIT: ok, now after merging your thread :P I found you already use that. Try different quant matrices like the animeCG matrix (use the search for finding those matrices, terms "HVS matrices" or "hvs matrix").

You also may want to lift your quantizer restrictions (though they shouldn't be reached anyways).
IFrame boost should be 0.
Below I-frame dist: 10
I-frame bitrate reduction %: 20
High bitrate scenes %: 0
Low bitrate scenes %: 0

Disable altCC.

I don't know how bad lumi masking is for anime, but you could try using it as well (I fear that anime content suffers from/with lumi masking...)

Regards
Koepi

[Bulletproof]

Try using Bilinear filtering on the video, or on the subtitles themselves.