quality problems

[Egladil]

I'm encoding HDTV material, and I'm having problems with the quality xvid produces.

Here's the source frame:
http://img321.imageshack.us/img321/7224/srcee1.png

I tried encoding to a DVD5 size, got undersized encode, every I and P frame with quantizer 2 (MPEG quantization matrix). Despite that the quality sucks. Here's the same frame from this encode.
http://img214.imageshack.us/img214/5771/enczt6.png
Vertical lines everywhere, the edges are full of jpeg artifacts, blocking everywhere - looks horrible.

Second encode, with 6of9 matrix and AQ enabled, the target bitrate was hit, but there are still those vertical lines everywhere, and the edges are broken again:
http://img456.imageshack.us/img456/271/newencmm0.png

Settings used: ultra high motion search precision, VHQ mode = 2, qpel and gmc were OFF, chroma motion on, trellis, and I also enabled AQ for the 2nd encode.

Is there any way to get a high quality xvid encodes? Maybe my settings are flawed?

cheers
-egladil

[Sharktooth]

It seems you applied a (or some) sharpening filter(s)... post your .avs.
Also are you sure you encoded with "unrestricted" profile?

[Egladil]

I did use a sharpening filter:
---------
Mpeg2source("video.d2v")
tfm().tdecimate()
DeGrainMedian(limitY=2,limitUV=4, mode=4)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,Smode=3,strength=90,overshoot=0)
Lanczos4Resize(1280, 720, 0, 0, 1920, 1080)
------------
I used Megui, and I just checked I didn't find any xvid profile selection (besides the usual profiles). Now I'm trying to encode with xvid @q1 to see what it looks like.

[Teegedeck]

I'm viewing this on a shitty display ATM but it really seems that sharpening has emphasized the macroblock-borders of the source - unless the source frame you've uploaded is filtered already.

[Egladil]

the source frame is filtered.

[Egladil]

ok even with q1 there are these lines and the edges are broken:
http://img79.imageshack.us/img79/269/q1rh3.png

the commandline used:
xvid_encraw -i test.avs -single -cq 1 -smoother 0 -nopacked -vhqmode 4 -qtype 1 -closed_gop -bvhq -threads 1 -mkv xvid-q1.mkv

is it a bug? I can't really imagine that xvid is unable to properly compress that video. x264 has no problems at all.

[Sharktooth]

weird... really weird...
Try, removing the sharpening and using EQM V3LR at Q2.
However the source image and the encoded image do not match (the encoded is much sharper). So are you sure yuo posted the "filtered" source?

[Egladil]

yes I am dead sure.

I'll try encoding without sharpening and see what it looks like

[Egladil]

Ok encoded with V3LR @q2.

The avs is just plain resize:
Mpeg2source("video.d2v")
tfm().tdecimate()
Lanczos4Resize(1280, 720, 0, 0, 1920, 1080)
Crop(...)


the source frame:
http://img502.imageshack.us/img502/6458/nofilterssrcpj8.png

encoded:
http://img290.imageshack.us/img290/1364/nofiltersenchx0.png

still looks like crap. and still the encoded image looks somewhat "sharper", no idea why.

[Sharktooth]

Ah! What decoder are you using (i suppose FFDShow due to the OSD)? Is it possible you have some post processing filter enabled (like warpsharp or sharpen)?

[Egladil]

thanks sharktooth.

I am using ffdshow. postprocessing filters were disabled, but then i realized that i haven't updated ffdshow since a while. after installing the current ffdshow version everything is ok. looks good :)

[Sharktooth]

np, glad you fixed it.

[henryho_hk]

the current ffdshow version everything is ok. looks good :)

Yup, I love clsid's icl91 compiles. http://sourceforge.net/project/showfiles.php?group_id=173941