H.264 vs. Xvid at high bitrates please advise

[aLu]

I was greatly intrigued by x264, but am now for the most part returning to Xvid. Why? Because x264 can look great at any bitrate, but tends to smooth away fine detail while it enhances/darkens borders, which can look great if you have strong shadows. In direct comparison, I simply found Xvid to preserve more detail (especially with a custom matrix). I've tried x264 at it's slowest, optimum quality setting with high as well as with low bitrates.
This is a purely visual judgement. I'm sure H264 supposedly yields more quality, but as always it is statistics vs. eyes/taste. If you use Xvid with smooth h263 quant, they will be indistinguishable.

[foxyshadis]

There's a more recent thread that discusses the effect and how to avoid the smoothing. To recap: Lower deblocking, custom quant matrices, subme 5 + no trellis, and/or deadzone.

[Dark Shikari]

There's a more recent thread that discusses the effect and how to avoid the smoothing. To recap: Lower deblocking, custom quant matrices, subme 5 + no trellis, and/or deadzone.I find that turning off RDO doesn't really help avoid smoothing (it just wastes bits); the most important thing is to use obnoxiously low deadzone settings.

[plane]

I find that turning off RDO doesn't really help avoid smoothing (it just wastes bits); the most important thing is to use obnoxiously low deadzone settings.

Do you have any recommended low deadzone settings?

[Dark Shikari]

Do you have any recommended low deadzone settings?Just keep dropping them lower until it gets how you like it, IMO.

Note that very low deadzones effectively "sharpen" the image.

[Sharktooth]

subme 1 is infinitely more sharp than subme 6...

[Dark Shikari]

subme 1 is infinitely more sharp than subme 6...That's ridiculous; subme 1 = more intra blocks = lower sharpness because intra blocks are worse psy-wise when there is a valid MV. I have never seen a case in all my work with x264 where lower subme values help; rather I've found they result in terrible-looking "stepping" where slow-moving backgrounds no longer move smoothly, and other such things.

The only way that can be true is if you're at a constant quantizer, and as a result subme1 has a 30-40% higher bitrate than subme6--no crap its going to be sharper.

Edit: actually, there is one other case I would think lower motion search precision would increase sharpness. If you have low quantizers (so inter blocks are preferred to intra in any case a good MV can be found), a lower-precision MV could result in "fake sharpness" due to the high frequencies from the previous block being carried over to the new block but not matching up with the high frequencies in the source. This would really only apply in the case of grain, I would think.

[Didée]

Well, let's say that "sharpness" is a term that can be interpreted in different ways ... just because it sounds good, it hasn't necessarily to be.