x264 vs Recode2 quality comparison & news

[JohnV]

http://www.liquidcreativity.co.uk/comparison/

Didn't see discussion yet about this comparison. For some reason seems it didn't get much publicity at Doom9.

On the news side:
The Nero update in September will have updated Nero Digital/Ateme AVC high profile codec (the above comparison uses the old codec build). I have Recode2 version with HP 8x8 transform support installed in my HD, so it is a sure thing finally. :)

Also, the Nero Digital ASP codec will get a quality boost which will give DivX and XviD a run for their money: Ateme is providing an updated ASP codec, though I'm not sure if it will make it already to the September update, but if not, in any case will be included soon after.

In addition Recode2 will get a nice audio quality update, since the new Nero Digital audio codec (which was also released as free stand-alone commandline software earlier) will be used in the upcoming September Recode2 for encoding the mp4 audio.

[bourtzovlakas]

For some reason seems it didn't get much publicity at Doom9.

Do you imply that this was because the results were not in favour of x.264?
It seems to be a rather limited test compared to Doom9 's codec comparison 2006...

[JohnV]

Sure it's limited. I was just saying it wasn't mentioned yet even on the forum. I was referring to the forum rather than to the front page.

[bourtzovlakas]

When was it published?
Do you have any info on the codec versions and codec settings that were used?

[*.mp4 guy]

The inloop filter was set to high in the X264 encode, there wasn't a single block visible anywhere in it, it would have benefitted a lot from lower deblocking settings. However I agree, between the two encodes presented there The Ateme one looked better.

[JohnV]

When was it published?
Do you have any info on the codec versions and codec settings that were used?
It was created quite recently afaik. I saw one of the authors (I believe) discussing about this at #x264.

[JohnV]

The inloop filter was set to high in the X264 encode, there wasn't a single block visible anywhere in it, it would have benefitted a lot from lower deblocking settings. However I agree, between the two encodes presented there The Ateme one looked better.
Yes, at least at #x264 there has been discussion, that with ideal settings latest x264 may actually be sometimes just a tiny bit sharper than Ateme. But x264 has also irritating flicker on static textures, like walls, which is visible on LCD screens (not on CRTs), meaning that Ateme has more stable image.

This would give an advantage to Ateme, since you could add easily a little bit of sharpness with post-processing, but x264 texture flickering can't be fixed in anyway with pp.

[bratao]

JohnV,
Please add the Output to Avi, at least to the ASP !
Remember that many people use Divx Player, and not all have a Mp4 player

[akupenguin]

Raptor4128 (i.e. Diablo-D3, but under his original nick he was banned from freenode for being annoying) listed the following options (he used mencoder, but I translated them to x264cli):
--pass=3 --bitrate=1721 --ref=16 --bframes=16 --b-pyramid --qpmin=1
--weightb --analyse=all --8x8dct --me=umh --merange=64 --mixed-refs --b-rdo
--bime --trellis=2 --no-fast-pskip --subme=7 --direct=auto --no-dct-decimate

[bkman]

Typical max settings, but with default deblock it will always look soft.

[acidsex]

http://www.liquidcreativity.co.uk/comparison/

On the news side:
The Nero update in September will have updated Nero Digital/Ateme AVC high profile codec (the above comparison uses the old codec build). I have Recode2 version with HP 8x8 transform support installed in my HD, so it is a sure thing finally. :)



Definitely the best news of all. Much thanks.

[*.mp4 guy]

Yes, at least at #x264 there has been discussion, that with ideal settings latest x264 may actually be sometimes just a tiny bit sharper than Ateme. But x264 has also irritating flicker on static textures, like walls, which is visible on LCD screens (not on CRTs), meaning that Ateme has more stable image.

This would give an advantage to Ateme, since you could add easily a little bit of sharpness with post-processing, but x264 texture flickering can't be fixed in anyway with pp.
If you set B predict mode to none, or turn off bframes (obviously turning off bframes is a bad idea) the flickering goes away. The flickering can be visible on crts aswell, its just harder to spot.

[CruNcher]

If you set B predict mode to none, or turn off bframes (obviously turning off bframes is a bad idea) the flickering goes away. The flickering can be visible on crts aswell, its just harder to spot.


the problem we talk about is not a b-frame problem it wont help turning off b-frames, only a smarter prediction (skiping) can avoid this flickering.

[Sagittaire]

1) Certainely comparison with low quantizer. Useless for me because very hard to make real comparison. X264 and Recode produce certainely very close result. MPEG4 ASP will be certainely very good too here ...

2) Screen capture are always useless for me because frame can be bframe for first codec and pframe for second codec. Local and general Rate Control can be differnent (ateme use lower compression quantizer than the default x264 setting with certainely different ratio/offset for I/P frame). If you want make subjectif comparison then you must use only video with exactly the same size ...

3) Nero recode can't use the best possible setting for Ateme's H264 implementation (not best RDO+ME+trellis mode, not 8*8 DCT mode, not best psy mode). Actually Recode use very old and incomplete Ateme's h264 implementation. IMO at this time x264 is really better for quality and speed than Recode. I think that Manao, Bobololo and Babayaga can say that too because it's simply true ...

[GodofaGap]

A screenshot comparison without even the originals and apparently different scripts used for both encodes (they have different AR and cropping) tells us what exactly?

They don't give settings, they don't give scripts and one wonders why this doesn't get any attention... I'll tell you, because the test is useless like this.

Also, the Nero Digital ASP codec will get a quality boost which will give DivX and XviD a run for their money
Please stop being a Nero troll and instead provide us with proper test results.

[JohnV]

Please stop being a Nero troll and instead provide us with proper test results.
What is actually trolling here? Recode is getting an improved quality ASP codec soon, that is a fact.
And anyway the test should be independent, otherwise it will be stamped as biased don't you think?

[JohnV]

A screenshot comparison without even the originals and apparently different scripts used for both encodes (they have different AR and cropping) tells us what exactly?

They don't give settings, they don't give scripts and one wonders why this doesn't get any attention... I'll tell you, because the test is useless like this.
.
I think the idea of the test was that 2 people encode a video with 2 high quality encoders using what they think are the best settings and compare results.
At least it shows how easy it is to not get the ideal results if you just max out all the x264 settings, like Joe-average would do..
The Recode version used in that test is old like Sagittaire in point #3 said. Imo what this test shows, is that Joe-average doesn't necessarely get the quality he would expect by maxing out the settings of x264, and infact goes below year old Recode.

[GodofaGap]

I think the idea of the test was that 2 people encode a video with 2 high quality encoders using what they think are the best settings and compare results.
So actually what you are saying is that Diablo-D3 doesn't know what he is doing (which is denied in the conclusions :p). Well I can live with that.

The problem is I have no idea what either one of these guys did, they don't tell me. And this not just about the codec settings, it's about the whole road from DVD to end-encode. They certainly haven't used the same script, or someone really messed up with taking screenshots.

The question is what use is it to me to know that Diablo-D3 doesn't know what he is doing? It's not a comparison between x264 and Ateme but between Diablo-D3 and ZoFreX. :)

Imo what this test shows, is that Joe-average doesn't necessarely get the quality he would expect by maxing out the settings of x264, and infact goes below year old Recode.
I think it's known by a while now that x264 can be misconfigured. I don't think it would be needed to test it against Ateme to find that out.

[foxyshadis]

Haha, obviously it only wasn't discussed because none of the regulars had heard of it. :p Maybe I'll try an encode of that chapter, and I can guarantee it wouldn't take hours just to set up the encode. Silly, using 3-pass and default inloop for something with medium-high bitrate. (x264 could use "-sharp" and "-soft" switches for those who don't know how to set inloop, or perhaps just in mencoder in this case.)

I'm extremely interested in the new ASP/AVC codecs, having seen what last winter's shootout prodced, especially if they're even faster. =D

[Dark Eiri]

Hey hey. Obviously the AVS scripts for the encodes are different. You can see color and cropping differences between both, clearly.

This guy spent HOURS in x264 configuration? Isn't easier to use MeGUI Insane profile, that is already maxed out, and just customize the CLI, if you need? o.o
Even if you don't, it's pretty easy to understand the CLIs and stuff, I'm sure you won't spend more then 30min trying to configure it, if you're a newbie to CLI.
Well, I'm sure he knows what he was doing, because the final result got pretty good :D

I do believe Ateme is an excellent encoder, I REALLY do. But I was not convinced by this test. Maybe someone should test it with proper results (same person, same scripts, it's about codecs, not about "people thinking what's better", you know).

Ah, the last question: what was the EXACT version of x264 used? I use 1200kbps Insane Profile for 720 x 432 encodes, and I always get crystal-clear (for cartoon, 268kbps AE-Max is pretty good). Maybe we should test x264 and Ateme with more limited bitrate. Around 1000. And a High Bitrate test, like, 5000. Then, the differences will show, and the winner will be worshiped forever and ever ;)

[CruNcher]

The question is what use is it to me to know that Diablo-D3 doesn't know what he is doing? It's not a comparison between x264 and Ateme but between Diablo-D3 and ZoFreX.

exactly, btw look @ the latest X264 changes if you know what they mean you will be soon suprised whats coming *g*

[Sharktooth]

with "latest" you mean r537?

[*.mp4 guy]

the problem we talk about is not a b-frame problem it wont help turning off b-frames, only a smarter prediction (skiping) can avoid this flickering.
Have you tried what I suggested? If you haven't you should atleast try it before giving summary judgment on the matter. In point of fact bframes use prediction, and perhaps changing the way it works or disbabling them would remove the flickering textures you experience. I used to have problems with "jumping" tectures, they would get smeared, move a bit, then jump back into place and it would happen every couple of frames (so its not anything to do with Iframes) doing what I suggested completely fixed the problem.

[CruNcher]

@Sharktooth
jep

@ *.mp4 guy
i did nothing else the last weeks than analyzing this problem (i found it) it happens mostly in mv areas of 0,0 (the skiping there is to inconsistent so to much shape changes occur) what you seem to expirience is flicker, because of to low bitrate like it was the case in ASP (i tried to lower that in my EDP build via RC tweaks) but that shouldn't happen this hard for AVC i tested this time way upper in the bitrate scale with high quality HD source.

[Sharktooth]

the changes are all concerning rc_method.
now it can be switched easily (maybe during encoding?) but i fail to foresee the practical reasons for those changes (maybe im too drunk...).

[CruNcher]

@ Sharktooth
Lookahead maybe :D

[foxyshadis]

This guy spent HOURS in x264 configuration? Isn't easier to use MeGUI Insane profile, that is already maxed out, and just customize the CLI, if you need? o.o
Even if you don't, it's pretty easy to understand the CLIs and stuff, I'm sure you won't spend more then 30min trying to configure it, if you're a newbie to CLI.
Presumably that includes the time spent ripping an putting together the Ultimate Avisynth Script(tm).

ps, sharktooth, is tcplx gone for good or something? I miss it. ;_;

[Sharktooth]

@ Sharktooth
Lookahead maybe :D
i knew i was missing something obvious...
Presumably that includes the time spent ripping an putting together the Ultimate Avisynth Script(tm).

ps, sharktooth, is tcplx gone for good or something? I miss it. ;_;
gone... it was completely screwing lossless encoding.

[akupenguin]

Lookahead maybe
That's... more correct that you might think, but also completely wrong. It's not the Lookahead™ quantization that has been hyped in #x264, rather it's lookahead ratecontrol (see the discussion with Tuukka on the mailinglist).

[CruNcher]

RDRC yeahhh, throw away your 3 passes :D

[Adub]

I looked "Lookahead" up on google.
It said, that a paper was published about this claiming that it meant better performance.

Here is the actual quote:The performance of the proposed scalable scheme is compared with that of the simulcast technique and the single layer coding. Remarkable performance improvement over the simulcast coding is achieved. While spatial scalability involves multi-layer coding, the new scalable scheme also achieves comparable or better performance that the single layer coding.

So, what kind of performance gain are we talking? ;)

[akupenguin]

I looked "Lookahead" up on google.
It said, that a paper was published about this claiming that it meant better performance. Here is the actual quote:
Wrong paper. I invented the term "Lookahead", since the papers it's based on don't give it a concise name. Thus, you won't find them by googling. Note also that the algorithms I'm implementing are not those described in the papers, that's just where my inspiration comes from.

ratecontrol:
Sermadevi_DCC2005 (http://foulard.ee.cornell.edu/publications/sermadevi_dcc2005.pdf)
1995_ICIP_Lin (http://viola.usc.edu/newextra/Publication/PDF/ICIP/1995_ICIP_Lin.pdf)

quantization:
Schumitsch_SPIE_05 (http://www.stanford.edu/~bschumit/Schumitsch_SPIE_05.pdf)

I won't post any numbers here, because it's not adequately tested and I don't want to give false hopes. I'll just say that it looks both promising and really slow.

[*.mp4 guy]

@ *.mp4 guy
i did nothing else the last weeks than analyzing this problem (i found it) it happens mostly in mv areas of 0,0 (the skiping there is to inconsistent so to much shape changes occur) what you seem to expirience is flicker, because of to low bitrate like it was the case in ASP (i tried to lower that in my EDP build via RC tweaks) but that shouldn't happen this hard for AVC i tested this time way upper in the bitrate scale with high quality HD source.
Thanks for the info, its frustrating when people disagree without giving enough explanation.

[IgorC]

More clear selection between ASP and AVC may improve the usability of Recode Gui. There are too much people who confuse the selection between ASP and AVC profiles. There are only 3 letters (AVC) while ASP has no "ASP" signature.

[Dark Eiri]

Plus, Recode doesn't let your tweak EVERY SINGLE option on H.264, like x264 with MeGUI does, right? =P

[Chainmax]

If you set B predict mode to none, or turn off bframes (obviously turning off bframes is a bad idea) the flickering goes away. The flickering can be visible on crts aswell, its just harder to spot.

Most of the highest quality profiles have this on "Auto" (you are referring to the "B-Frame mode" switch, right?), how would setting it to "None" affect quality aside from removing the flicker?

[williedigital]

the high profile thing is nice, but i care more about bringing recode into the dual-core world. using x264 is twice as fast since it uses both cores in my core duo and recode uses only one. is that going to change in the sept update?

[*.mp4 guy]

Most of the highest quality profiles have this on "Auto" (you are referring to the "B-Frame mode" switch, right?), how would setting it to "None" affect quality aside from removing the flicker?
Changing Bframe predict mode to none would make Bframes less compressable, so essentially you would be sacrificing a small overall quality drop to obtain a stable picture. You shouldn't turn of Bframe prediction for cartoons, ever. Its really source dependant, I've had some sources where stability was horrible unless I changed the bframe settings, and I've had others where it made essentially no difference.

[JohnV]

the high profile thing is nice, but i care more about bringing recode into the dual-core world. using x264 is twice as fast since it uses both cores in my core duo and recode uses only one. is that going to change in the sept update?
Yes..

[Kostarum Rex Persia]

JohnV, that's really good news. Finally, Ateme codec will support dual-core processing.

Two questions for you, JohnV:

1) Do future Ateme H.264 High profile codec supports SSE2 and SSE3 optimizations?

2) About SMP machines: If someone have a quad-core server machine in SMP mode, can Ateme codec use all four processors in encoding?

[JohnV]

SSE2 and SSE3 support is there, but codec update is probably needed for the point 2.

[Kostarum Rex Persia]

Ok,thank you for answering me.

But, can you, at least, do the job and include support for Intel quadcore Kentsfield. Quadcore support will be very very nice.

[slavickas]

Ok,thank you for answering me.

But, can you, at least, do the job and include support for Intel quadcore Kentsfield. Quadcore support will be very very nice.
thats nice that you got kentsfield

[JohnV]

Ok,thank you for answering me.

But, can you, at least, do the job and include support for Intel quadcore Kentsfield. Quadcore support will be very very nice.
Would be cool, yes. This is Ateme's field, so lets hope they implement this soon... ;)

[Kostarum Rex Persia]

I havn't got Kentsfield, of course, but I am planing to buy him as soon as quadcore hits the market, in December 2006.

[Sharktooth]

...still asking for something and still you havent the requisites...
will K8L and 4x4 be supported in the new version? 8-cores are just crazy and i want them to be supported!

[odditory]

@JohnV:

It's great to hear the encoder will finally step into the multi-threaded world. I do have one huge question: will BATCH ENCODING support ever happen? This is the single greatest frustration as far as missing feature in the app. Most every other commercial encoder has this (Mainconcept, Procoder, etc.) but sadly i'm in love with nero digital's AVC output so I wait...

..God forbid grid processing (aka encoding farm) ever came along - DVD2one's rudimentary implementation blows even with gigabit interconnects between PC's but you gotta give 'em credit for trying. :) Yes I know DVD2one isn't an H.264 encoder.