Sharper x264 output?

[bkman]

It seem to me that even with deblocking disabled x264 gives a softer output and keeps less detail than the Ateme codec at equivalent bitrates. Particularly x264 seemingly has no setting even close to Nero's "automatic sharp" mode. Is there something I am missing?

[Sharktooth]

uhm... x264 has no psy... so try using a custom quant matrix

[bkman]

Can you suggest one? I have tried your AVC-HR but it doesn't help for my source and bitrate (1240).

[Sharktooth]

there are only few custom matrices for x264. try using soulhunter's one or just sharpen the source with an avisynth filter.

[bkman]

I don't really want to sharpen the source as I think that it will cost me in overall encoding quality. Can you elaborate as to why x264 is not as sharp as Ateme's?

[Sharktooth]

ateme has psychovisual enanchments which help sharpness, while x264 does not.
however the difference shouldnt be so "hard". what settings you used for x264 and for ateme?

[bkman]

x264 settings:
C:\Program Files\x264\x264.exe --pass 2 --bitrate 1240 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -3,-3 --subme 6 --trellis 1 --analyse all --8x8dct --sar 74:75 --progress --no-psnr

Ateme settings are basically Max Definition - AVC with 5 refs.

The differences are maybe not that noticeable to others, but to me things like skin and various fabrics lose much of their texture with x264. I've tried with adaptive quantisation (which I understand is similar to phychovisual enchancement) but it doesn't help much.

[bond]

x264's current, not in svn, AQ is not meant to enhance details

in my own experiments i also found atemes AQ to be pretty good (eg in treating the walls)

[Sharktooth]

x264 settings:
C:\Program Files\x264\x264.exe --pass 2 --bitrate 1240 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -3,-3 --subme 6 --trellis 1 --analyse all --8x8dct --sar 74:75 --progress --no-psnr

Ateme settings are basically Max Definition - AVC with 5 refs.

The differences are maybe not that noticeable to others, but to me things like skin and various fabrics lose much of their texture with x264. I've tried with adaptive quantisation (which I understand is similar to phychovisual enchancement) but it doesn't help much.
AQ won't help, it may reduce blocking in dark and blue areas but it wont improve facial or foreground details.
your settings seem to be good...
the only thing i feel to suggest you is improving sharpness with an high quality filter (like limitedsharpen) or by trying the few different avc custom matrices you can find with the search function.
if you have some time to spend you can even try making your own.

EDIT: you can maybe trying disabling trellis or using trellis 2 and lowering the inloop to -5,-5 or even -6,-6.

[bkman]

Trellis 2 actually helps quite a lot! But its quite a bit slower also.

I think the best solution for now is simply to use Nero Recode for sources where I want to keep maximum detail. I hope that the x264 devs can improve x264 this regard.

[Sharktooth]

weird, nero recode should provide less details than x264 coz it havent 8x8 transform... it's the old ateme encoder...

[bkman]

Not in terms of surface texture detail (which is what I was referring to). It has good detail in other aspects, like the creases on a leather glove in my source are more pronounced with x264.

Btw, you seem to suggest that there is a newer ateme codec I can use?

[Sharktooth]

this one: http://forum.doom9.org/showthread.php?t=96203 . it's not public though.
it supports high profile, film grain modeling and other cool stuff.

[bond]

i have done tests some time ago between nero and x264 and nero was clearly sharper at that time
i havent redone the test so i cant say if thats still true (maybe it is?)

all i know is that there are lots of psnr tests which make people believe that x264 is superior to ateme. but my opinion about the validity of psnr results is known ;)

[Sharktooth]

you know we share the same thoughts on psnr...
however i find difficult to believe nero recode can retain more details than x264 right now.
maybe im wrong but...

[fight2win]

mpeg-4 is usually softer than mpeg-2 source, whatever may the encoding be! if sharpness is desired, keep it as mpeg-2!

[ChronoCross]

mpeg-4 is usually softer than mpeg-2 source, whatever may the encoding be! if sharpness is desired, keep it as mpeg-2!

or just sharpen the source. rather than wasting space on mpeg2. sharpen the source then let h264 hack away at the sharpeness until it returns to what you would consider normal.

[*.mp4 guy]

In my experience details that are sharp are always kept (almost perfectly) by X264 and Ateme, but lower contrast details often get murdered by X264, Ateme also has some trouble with them, but not to the extent that X264 does. So are you loosing smaller details with X264, or are sharp details getting blurred?

[fight2win]

has anyone tried sony vegas 6's mainconcept h.264 encoder?

[bond]

has anyone tried sony vegas 6's mainconcept h.264 encoder?dont hijack threads with different topics!

[Archimedes]

The PSNR metric doesn’t work well for comparing different encoders, may be it is usefull for comparing different settings with the same encoder. The SSIM metric makes a much better job. For a while i’ve tested also the x264 and Nero Recode 2 encoder (version 2.2.16.17), but with dv sources (PAL). Interlaced and not interlaced. I know already, x264 doesn’t support interlaced stuff. The mathematical results (SSIM) says, that x264 makes a better job with the mentionend sources.

However, i was surprised about most of the i frames from Nero, which looked much better than the i frames from x264. Regarding this i frames, my very first impression was not so good for x264. In the following post you will see, that Nero is much closer to the original frame than x264. As far as i know, in this test, most of the i frames from Nero looked better than the x264 i frames.

http://forum.doom9.org/showpost.php?p=774345&postcount=10

But, after viewing the following frames (after the i frames), i realized, that Nero also smoothes some areas. After analyzing a lot of reference clips, the SSIM values was allways better for the x264 encodings.

To say something about the compare between Mpeg 2 and H.264/AVC, i did also some tests in this field. I did some tests with different Mpeg 2 encoders and with Nero Recode 2 (as i worked with interlaced stuff). Regarding the SSIM vaules, you need at least 5400 kbps in Nero Recode 2 (it depends on the source) to be better than Mpeg 2 encodings with its typicall 6500 kbps (tested with Procoder Express, VBR, 2 pass, 3500/6500/8500). I verified this by viewing and analyzing of some frames. After that i tested progressive dv stuff (PAL) with nearly the same results.

[Sharktooth]

the test is flawed coz you used interlaced material that may screw the ME algos thus influencing the final frames quality.

[Archimedes]

Because of this and regarding the images above, i did this test also with progressive source. The results where similar. Here are the progressive frames.

X264, „HQ-Slow“ profile, 6500 kbps
http://img503.imageshack.us/img503/6685/videox2646500deinterlaced0360x.jpg

Nero Recode 2, 6,52 Mbps
http://img141.imageshack.us/img141/4242/videonero6500deinterlaced0360x.jpg

[Sharktooth]

seems both nero and x264 retained more details but still nero has an advantage.
IMHO it's the psy...
x264 picture seems darker too...