bad quality with x264 what am i doing wrong?

[MaxPlanck]

I've been using xvid for a few years to do all my DVDs and was always happy with quality. In the last few months I've been playing with x264, but after I encoded the movie gladiator I got disappointing results. I've seen x264 look much better so I wonder what I'm doing wrong...

here's my synthscript

DGDecode_mpeg2source("C:\Users\XXXXXXX\Desktop\gladiator project\gladiator project.d2v")
edeintted = AssumeTFF().SeparateFields().SelectEven().EEDI2(field=-1)
tdeintted = TDeint(edeint=edeintted,order=1)
tfm(order=1,clip2=tdeintted).tdecimate(hybrid=1)
crop( 0, 54, -4, -60)

BilinearResize(640,272) # Bilinear (Soft)
Undot() # Minimal Noise


and here's a screenshot...I don't have the right terminology to describe what I'm seeing. Edges look like they are "aliased" and detail seems more smudged than it should be. I used the HQ-Slowest Profile to do this with 2100 MB file size. Any suggestions would be appreciated http://www.mediafire.com/?10nzynhwjdm

I know I posted a jpeg, but it doesn't look far from my encode at all.

[check]

two things firstly:
* use a real imagehost! what's wrong with imageshack or the one in my signature :P
* don't capture at a resolution that isn't 100%. Try media palyer classic's 'save image' feature, or ffdshow's grab functionality.

Anyway, the problem you're seeing can probably be fixed with a litle adaptive quantizer. In megui, add --aq-strength 0.3 to the custom commandline section.

[MaxPlanck]

I will try that, thank you!

[MaxPlanck]

Thanks for the suggestions, but I just noticed something. I believe this has something to do with how my video is decoding in Windows Vista. One of my previous movies I did was Batman Begins, I use HQ-Insane and I'm blown away by the quality when I play it on Windows XP. On Vista, the quality looks atrocious. I'm using FFDSHOW to decode. I'm not sure what info I could provide.

[check]

I need more info. What media player, what video renderer, what ffdshow version, etc etc.

[MaxPlanck]

I'm using Media Player Classic 6.4.9.0 and FFDSHOW rev. 896 (20070208)
VLC Media Player and WMP produce the same poor quality

[check]

the the low coour quality due to the blacks & whites being washed out?

[Blue_MiSfit]

Make sure postprocessing is disabled in ffdshow.

Inloop deblocker + ffdshow deblocker = smeary mess of putrification :D

~MiSfit

[MaxPlanck]

LOL, the "smeary mess of putrification" is already turned off, just because I'm crazy I decided to reinstall Windows to rule out anything weird...same problem.

[MaxPlanck]

the the low coour quality due to the blacks & whites being washed out?

they do look washed out...not only that, but edges appear slightly jagged, similar to the way a video game looks with no anti-alliasing.

[check]

i bet you are using the VMR9 renderer with bad nvidia drivers. Try Overlay mixer instead, although make sure your video card config isn't set to 114% saturation or something similarly broken.

[MaxPlanck]

I have an ATI card, both the "stock" vista drivers and the official ATI drivers look the same. I swithced to overlay, but there isn't any noticeable difference.

[MaxPlanck]

weird, my XVID movies look fine, what gives??

[check]

chances are your h264 decoder is borked. Are you using coreavc with 'fix VMR9 levels' checked?

[MaxPlanck]

chances are your h264 decoder is borked. Are you using coreavc with 'fix VMR9 levels' checked?

definitely not...all I did was install media player classic, then ffdshow.

[DarkZell666]

The screenshot you posted shows a smoky-low-contrast-scene: just what x264 typically doesn't like :D (though I would be satisfied with what I'm seeing there if I had done the encoding).

AQ should indeed do something about it, it won't be perfect but better than nothing (use sharktooth's builds for that).
Next stop is at the CQM & deadzone stand imho :)

[MaxPlanck]

The screenshot you posted shows a smoky-low-contrast-scene: just what x264 typically doesn't like :D (though I would be satisfied with what I'm seeing there if I had done the encoding).

AQ should indeed do something about it, it won't be perfect but better than nothing (use sharktooth's builds for that).
Next stop is at the CQM & deadzone stand imho :)

well the thing that is bothering me the most, is that the same movie plays beautifully in Windows XP. The colors in vista appear too bright and washed out. Changing the renderer in MPC didn't seem to have any effect. I'm starting to worry that the new playback DRM crap in vista is having an effect on my movies. My understanding of that however, is that it only decreases quality for protected content. Am I wrong?

[nm]

The problem is not in DRM but in the graphics card drivers. Complain to AMD.

[MaxPlanck]

I'm sure my puny little complaint wouldn't accomplish anything, but at least I'll head over to their forums to see if anyone else is experiencing this. Thanks for your help guys.

[DarkZell666]

The problem is not in DRM but in the graphics card drivers. Complain to AMD.Did I miss something or did you mean ATI ? lol

well the thing that is bothering me the most, is that the same movie plays beautifully in Windows XP.Gotcha, one more reason for me not upgrade to vista yet (not that my hardware allows me to anyway but ... :p)

[foxyshadis]

You might notice the funny green logo and the new link when you head over to ati.com...

[quake74]

well the thing that is bothering me the most, is that the same movie plays beautifully in Windows XP. The colors in vista appear too bright and washed out. Changing the renderer in MPC didn't seem to have any effect. I'm starting to worry that the new playback DRM crap in vista is having an effect on my movies. My understanding of that however, is that it only decreases quality for protected content. Am I wrong?

I installed vista and the video pipeline has been messed up with. I usually use VLC, and the Direct3D video out doesn't work, the DirectX works, but disables aero, and the WindowsGDI one works, doesn't disable aero, but looks like crap (the same jagged edges you described before). I don't think it makes a difference, but I'm on ATI as well (Radeon Mobility X600SE).

[DarkZell666]

You might notice the funny green logo and the new link when you head over to ati.com...

Oh my when did that happen ? It could be a good thing though if they collaborate properly :)

Anyhow, it seems it isn't an MPEG4-AVC problem anymore isn't it ^^

[MaxPlanck]

no it doesn't appear to be an mpeg4 avc problem anymore...but I would like to know if it's a problem exclusively with ATI (amd) cards and drivers. Anybody running Vista with an Nvidia card see the same problems?

[MaxPlanck]

confirmed on a different machine with an Nvidia card...same exact problem. So, I don't think it's a driver issue (at least one that is specific to one manufacturer). Does anyone know why this is happening?

[DarkZell666]

Googling around, I found 2 other people (on french forums) that report teh same problem, (and unfortunately for them, the only advice they were given was "upgrade your GC's drivers"), to which they answered "I already done this, is my GC performant enough?" ... lol (especially when they also reported having crystal-clear video under XP ... ;)).


Would you mind trying mplayer, just for completeness ?

I would suspect Vista has some "supadupacoool-color-enhancer", that can only be turned off via the registry (does vista still have one ? XD). I believe that's just the sort of silly thing they would do :)

[MaxPlanck]

I just tried Mplayer...same exact problem on both machines.

[SpAwN_gUy]

I've noticed (a week ago..) that my x264 encodes look Extremeley Crappy (i can provide shots) when decoded with ffdshow(rev.906) ...
And play nice and smooth and sexy, when decoded with CoreAVC 1.2 (i have)..
and some more people noticed that also and agree with me..

[foxyshadis]

Did you turn deblocking off, or leave some kind of filter on?

[DarkZell666]

I've noticed (a week ago..) that my x264 encodes look Extremeley Crappy (i can provide shots) when decoded with ffdshow(rev.906) ...
And play nice and smooth and sexy, when decoded with CoreAVC 1.2 (i have)..
and some more people noticed that also and agree with me..

Are you sure ffdshow was configured properly (RGB32 output and no post-processing) ? Also, are you on vista or xp ? If you're not on vista you're unfortunately off-topic since the problem MaxPlanck has is obviously related to vista (or the current "vista-enabled" driver versions from ati AND nvidia).

If you are on vista, would you mind explaining what you did to get things working properly ?

[MaxPlanck]

This confuses me even more. I see the same quality issues when playing x264 on my Linux distro. On XP, it looks fine. On Vista it looks like crap. I'm going insane...

[DarkZell666]

This confuses me even more. I see the same quality issues when playing x264 on my Linux distro. On XP, it looks fine. On Vista it looks like crap. I'm going insane...

Oh my how surprising is that ? o_O

Actually, did you try setting the output to RGB32 in ffdshow ? I should have suggested that first (though if the defaults are fine on XP, they should be fine on vista too ¬_¬).

[nm]

How about playing it on Linux with mplayer -vo x11 -zoom ?
If that displays the video correctly (although it may be slow), the problem is in hardware overlay or the driver (which is unlikely since you have tried so many different setups).

Another possibility is that you have adjusted the display gamma or some similar setting on the XP setup. That could make the compression artifacts less visible while other systems that run on default settings show the problems clearly.