H.264 vs. Xvid at high bitrates please advise

[markrb]

I am about to start converting many of my DVD's into mkv's and I am still debating over which I should use.

I am not asking what is best here, but more to the point is there enough of a quality difference at higher bitrates for me to use H.264 and the extra encoding time?

I am looking to create files around the 2gb range for a typical movie.

Since these will remain on the HD hitting a filesize is not important as long as it isn't 400mbs too big. I am currently leaning towards a one pass XVid encode for the time savings. Several hours at least for me over H.264.

Is there a compelling reason I should change to H.264 instead?

I have done a couple of personal tests, but the time involved is too much to do a true comparison.
From the few I have done I really can't see much of a difference, but maybe I am missing something or maybe there is a movie type that I haven't tried that will prove to be worth the extra time.

I am looking at close to 100 conversions so time is a factor.

Thanks,
Mark

[Adub]

For High bitrates, no there really is no reason to use x264. Xvid is faster, and with the increased bitrate there will not be a huge difference in quality, almost none infact. I would go with Xvid on this one, although 2 passes, not 1. 2 passes is still fast, but also greatly increases quality.

[Sagittaire]

Xvid is faster

it's not true. With same bitrate and and same speed x264 done always very better quality XviD VHQ1 (default setting) for example ...

it's not a obligation to use 16 ref, RDO, high profil ... etc etc etc with H264. H264 can be very fast codec too ... ;-)

[siddharthagandhi]

Merlin is entirely right. At bitrates of over 1500 kbps, you will get a very minimal, if at all, an advantage for H.264. Stick with the ASP codecs, like Xvid, as Merlin said.

And you're a retired Doom9 person how come you're asking questions arent you supposed to know everything? :P

[markrb]

Ahhh, but now you know our secret. We pretend to know everything, but really we are all just noobs playing with ya.

I was a Mod in the DVD2SVCD forum. Ask me about CCE, DVD2SVCD, some avisynth and maybe I can help, but this stuff is new to me.

Mark

[Adub]

@Sagittaire,
What options are you using? I never get the same speed with x264 as I do with xvid. and if I do reduce some of the standard options I get a noticeable quality drop.

[Sagittaire]

@Sagittaire,
What options are you using? I never get the same speed with x264 as I do with xvid. and if I do reduce some of the standard options I get a noticeable quality drop.

http://forum.doom9.org/showthread.php?t=105763

certainely but even with quality drop x264 is always by far better than XviD

[siddharthagandhi]

Yes I also have a colossal speed increase with xvid.

[foxyshadis]

The only way I've ever been able to get x264 up to xvid's speed (more than 100 fps with VHQ1) is to use turbo options - me dia, subme 1, analyse none. At low bitrates x264 is only slightly worse, but at high bitrates it looks much worse. I know some people've said x264's fastest is better than xvid's slowest, but I've never seen anything remotely like that, even though it contradicts sagittaire's results. Maybe it's different on other processors, or different movies, or whatever.

I'd say once you get a standalone that supports AVC, if ever, that would be a compelling reason to switch. The one thing I feel xvid really lacks is an equivalent to --crf, for constant quality (not constant quant), since you can get increased quality across most of the movie for the same size.

[DeathTheSheep]

I've found x264 to be to superior to XviD in quality even at high bitrates, and the speed, if not exactly equal, was comparable. At such high bitrates, one may not notice the quality difference as easily, however.

What's "best" depends on the viewer, unfortunately.... ;)

[*.mp4 guy]

http://forum.doom9.org/showthread.php?t=105763

certainely but even with quality drop x264 is always by far better than XviD
You should learn to avoid making overly broad sweeping statements based on metric tests. For one your test was at 1400kbps a bitrate which X264 usually still has an advantage at, your test also relied on metrics which tend to favor Mpeg4-avc more then people do. Furthermore you tested on a clip from king kong which was very bitrate hungry (Xvid only hit an OPSNR of 43, which is much lower then it should be for a typical movie encode at 1400 kbps) which means the effective compressability of that clip was most likely in the 1cd-sub 1cd range, an area where it is widely recognized that Mpeg4-AVC has a clear advantage. Your results are not at all aplicable to this situation.

for a 1hour 30minute movie the effective bitrate for a 2GB encode would be 2911Kbps (with one 192Kbps audio track) which is a bitrate Way out of Mpeg4-AVC's usefull range. There is absolutely no reason to use X264 for encoding under these conditions, you could even end up with an encode with noticibly worse grain retention if you used X264. Ignoring the longer encoding time and the more demanding playback requirements there is still no reason to use X264 in this instance.

[IgorC]

Unfortunetly not each day and not many people provide some results. But Sagittarie is doing it well. People have no enough time or no skils ..... . In fact it was only one pretty usefull comparison for last 2-3 months.

[*.mp4 guy]

Unfortunetly not each day and not many people provide some results. But Sagittarie is doing it well. People have no enough time or no skils ..... . In fact it was only one pretty usefull comparison for last 2-3 months.
I'm not saying his tests can't be usefull, I was just pointing out that that test was completely out of context and wasn't valid in this context.

[IgorC]

I'm not saying that you're saying it. ;)

[Audionut]

All my tests show that x264 is better than xvid reguardless of bitrate.

it's not a obligation to use 16 ref, RDO, high profil ... etc etc etc with H264. H264 can be very fast codec too ... ;-)

[markrb]

It seems then as long as I keep the bitrate above 1500 then any added benifit would be minimal if I used h.264?

I am doing a test now where the bitrate is above 1700. I like to keep the original audio AC3 since I can send it digitally to my reciever and have it decode it internally. This adds a little size, but I can live with that.

The current test movie is a little over 2hrs with AC3 audio and I set the target size to 2gb.

Has anyone done any tests with the higher bitrates I am looking at?

Igorc/*.mp4 guy I think there is an understanding error. Seems like a language issue.


If I want to test x264 with high quality and good speed can you recomend some settings to try then?
I am using Megui as the interface.
I just can't deal with 7hr long encodes for each video I have to do.

Thanks,
Mark

[siddharthagandhi]

Yes I have. Funny thing is, despite the technical fact that there should be no difference at the high bitrates, I do get a difference. But maybe its just codec discrepancies.

[Audionut]

If I want to test x264 with high quality and good speed can you recomend some settings to try then?
I am using Megui as the interface.
I just can't deal with 7hr long encodes for each video I have to do.

Thanks,
Mark


--pass 2 --bitrate 1700 --stats ".stats" --ref 5 --mixed-refs --bframes 3 --b-pyramid --bime --weightb --filter -1,-1 --analyse all --8x8dct --direct auto --progress --no-psnr --output "C:\test.mkv" "C:\test.avs"

With a fast first pass.
Speed should be comparable to xvid with all options enabled.

Things you can try.
--filter 0,0 <-- deblocking in megui
--subme 6 or 7 <-- subpixel refinement in megui, with --b-rdo <-- RDO for b-frames.
--trellis 2 (always)

Of course your best bet is to make your own little test to verify.
Encode a sample of a movie with xvid, and the same sample with x264(with the settings posted above).
Judge both encodes with your eyes.
For my eyes, I very much prefer the look of x264 encodes.

Then run this scipt.
loadplugin(c:\x\ssim.dll")

# --> Load source with same options as avs file <--

source=Mpeg2Source("C:\test.d2v")
source=Crop(source,6,68,-14,-74)
source=LanczosResize(source,704,304)
source=selectrangeevery(source,10000,100)


# --> Load the video <---


video=directshowsource("C:\test.mkv")


# --> SSIM analysis <--

SSIM(source,video,"results.csv","c:\x264.log",lumimask=2)


P.s. Give xvid a little help with extra bitrate. It will need it.:D

[markrb]

The only thing I can see different from my current setup is the trellis setting is at 1.

I am assuming with this code I only see what is on. What do you have shut off in Megui?

I am using the CE Highprofile setting.

I am also using Bicubic nuetral, but that is from my SVCD days of testing as I didn't like the others.
I will do a test with Lanczos(sharp).

Mark

[Audionut]

The differences between my settings and CE Highprofile are,

subpixel refinement 5 instead of 6.
Trellis none instead of 1
reference frames 5 instead of 3.
b-frame mode auto instead of spatial.

or download the config file here.
http://rapidshare.de/files/18705185/Balanced.rar.html

[kotrtim]

I would recommend to have no fast pskip always on, change trellis to 1 for faster speed, reduce the reference frame to 2 + mixreference or even 1 for even more speed, normally according to the statistics after encoding, x264 rarely use the 3rd and above frames for reference, only around 1% or less.

[Sagittaire]

You should learn to avoid making overly broad sweeping statements based on metric tests. For one your test was at 1400kbps a bitrate which X264 usually still has an advantage at, your test also relied on metrics which tend to favor Mpeg4-avc more then people do. Furthermore you tested on a clip from king kong which was very bitrate hungry (Xvid only hit an OPSNR of 43, which is much lower then it should be for a typical movie encode at 1400 kbps) which means the effective compressability of that clip was most likely in the 1cd-sub 1cd range, an area where it is widely recognized that Mpeg4-AVC has a clear advantage. Your results are not at all aplicable to this situation.

for a 1hour 30minute movie the effective bitrate for a 2GB encode would be 2911Kbps (with one 192Kbps audio track) which is a bitrate Way out of Mpeg4-AVC's usefull range. There is absolutely no reason to use X264 for encoding under these conditions, you could even end up with an encode with noticibly worse grain retention if you used X264. Ignoring the longer encoding time and the more demanding playback requirements there is still no reason to use X264 in this instance.

Bitrate doesn't mean anything ... lol

As always bitrate in my test doesn't mean anything. I use XviD q4 for reference in this test. With this source q4 XviD produce 1400 Kbps and ~q4 for XviD is typicaly the quant used for 100-120 min 720*400 movie for 1CDR. This test is really representative for real test conditions and AVC is simply always better than ASP.

@ Merlin7777

1) use anamorphic encoding 720*576 without crop and resize for your source (perhaps mod16 resize and recrop could be good for perfect borders quality)

2) Use LC-AAC 5.1 encoding to replace AC3 with mp4 container : hardware compatibility will be very better

[*.mp4 guy]

Bitrate doesn't mean anything ... lol

As always bitrate in my test doesn't mean anything. I use XviD q4 for reference in this test. With this source q4 XviD produce 1400 Kbps and ~q4 for XviD is typicaly the quant used for 100-120 min 720*400 movie for 1CDR.
Your right bitrate doesn't mean anything, what is important is compressability and Xvid H.263@Q4 is a compressabilty range that AVC excels at (I've done tests).

This test is really representative for real test conditions. Yes, this test is representative of real conditions when using Xvid at fixed Q4 with H.263, but it is not representative of anything else.


AVC is simply always better than ASP AVC is always better then ASP for metrics, that is true, but at high bitrates things like RDO optimisation deblocking and motion estimation all become less usefull. You can see the effect by comparing the difference by computing the SSIM difference between fast settings and slow settings at a low compressability, then doing the same at a high compressability, the differences will be much smaller at a high compressability (you can do the same thing with OPSNR, but its not a very good metric to test how people will percieve things). AVC also becomes less usefull due to the same restrictions.

This is an overall trend in lossy data compression, high complexity compression algorithms work best at high compression ratios. For instance take a look at Mpeg2 audio, mp3, and LC AAC/HE AAC, Mp2 audio compression works just as well as mp3 at low compression ratios (Musepack which is based off of Mp2 is actually significantly better, while still being less complex then mp3), but quicky becomes less efficient as compression ratios increas. Mp3 works just as well as LC AAC at 160kbps, LC AAC only starts to have a definate edge around ~100 kbps. HE AAC (the highest complexity audio compressor) only becomes useful at about ~64 kbps and actually hurts quality above this point, it is most usefull for the ~32kbps range. High gains at high compression ratios says nothing of what will happen at low compression ratios, if it did then we would all be using HE AAC all the time...

[siddharthagandhi]

MP4 Guy you're kind of outnumbered on this one.

I agree with Saggitaire

"AVC is simply always better than ASP"

Regardless of technical details such as metrics and all that, the overall visual quality is better, and surpisingly, I see a difference at all bitrates, though AVC is designed to be a low bitrate format. By overall visual quality I mean when you're looking at the video what it looks like, disregarding these metric details and deblocking etc. etc.

And this occurs in all AVC codecs I have used, including Nero Digital, x264, and VSU H.264 codec. And I have encoded a lot of different samples and have observed this pattern.

[TonyMo]

Just to add my rarely heard voice to the x264 chorus.

I've done extensive personal testing of both xvid and x264 all with the goal of providing transparent backups for DVD. On my rig (and I'm including my eyes here), x264 is vastly superior to XVID at higher bitrates (or any comparable bitrate)... This is using the latest Chronos/Sharkhunter builds and matrix...

The only problem is the dark scene/bluesky issues, and I still need to do some more experimenting with the new AQ options to try and get past those, I'm keeping my fingers crossed. Given that neither codec really provides absolute transparency while maintaining some form of compression, x264 looks the best overall to my eyes and there are many a time I forget I am not looking at the original DVD master.

To reap the maximum benefits though it does help to use some of the slower encoding settings.

XVID is great though at crunching though encoding time and is not to be sniffed at at any means.

[siddharthagandhi]

"Sharkhunter"-- do you mean sharktooth?

[Sagittaire]

AVC is always better then ASP for metrics, that is true, but at high bitrates things like RDO optimisation deblocking and motion estimation all become less usefull. You can see the effect by comparing the difference by computing the SSIM difference between fast settings and slow settings at a low compressability, then doing the same at a high compressability, the differences will be much smaller at a high compressability (you can do the same thing with OPSNR, but its not a very good metric to test how people will percieve things). AVC also becomes less usefull due to the same restrictions.

Well be carefull PSNR and SSIM are logaritmic function. For high quality little metric improuvement is high size gain.

Here quality metric equivalence for King Kong Trailer 720*400

SSIM size equivalence
~q2 ASP encoding at 2800 kbps done same quality than 2100 Kbps for AVC -> 25% for size gain
~q4 ASP encoding at 1400 kbps done same quality than 1020 Kbps for AVC -> 28% for size gain
700 Kbps encoding for AVC done same quality than 1050 Kbps for ASP -> 33% for size gain

In fact you can't compare AVC and ASP like you can't compare MPEG2 and MPEG4 ASP ... XviD can't fight with x264 in all situation

[Sharktooth]

AVC is clearly better than ASP at low medium and mid-high bitrates.
on higher bitrates though there's actually no differece coz both codecs can reach transparency.
also AVC may have problems keeping ultra fine details (such as noise and smaller grain) even at higher bitrates. That's the only case where ASP can be slightly superior than AVC.

[Revgen]

X264 is also more flexible than Xvid. You can choose between Main Profile, High Profile, Deblocking, b-frame options, SubMe, and others to find a good balance between speed and quality.

Awhile back I decided to try and adjust my X264 settings to match the speed of Xvid at the same bitrate, and X264 still came out looking better. I would advise you to play around with the settings a little and see what you can find.

IMHO using Xvid for speed purposes is an excuse for not trying out a new codec that you may not understand so well. Granted Xvid is a lot easier to use and understand then X264.;)

[*.mp4 guy]

Well be carefull PSNR and SSIM are logaritmic function. For high quality little metric improuvement is high size gain.

Here quality metric equivalence for King Kong Trailer 720*400

SSIM size equivalence
~q2 ASP encoding at 2800 kbps done same quality than 2100 Kbps for AVC -> 25% for size gain
~q4 ASP encoding at 1400 kbps done same quality than 1020 Kbps for AVC -> 28% for size gain
700 Kbps encoding for AVC done same quality than 1050 Kbps for ASP -> 33% for size gain

In fact you can't compare AVC and ASP like you can't compare MPEG2 and MPEG4 ASP ... XviD can't fight with x264 in all situation
You completely missunderstood what I was saying.

[edit]actually I missunderstood you, you are right about the logarithmic nature of SSIM and PSNR, but the effect I described is still easily noticible.

[Lil' Jer]

IMHO using Xvid for speed purposes is an excuse for not trying out a new codec that you may not understand so well. Granted Xvid is a lot easier to use and understand then X264.;)

Well, to be frank, your opinion is quite wrong.

[Sagittaire]

You completely missunderstood what I was saying.

[edit]actually I missunderstood you, you are right about the logarithmic nature of SSIM and PSNR, but the effect I described is still easily noticible.

well in fact it's more like a(1-e^(-ax)) for SSIM ... ;-)

[Kopernikus]

well in fact it's more like a(1-e^(-ax)) for SSIM ... ;-)

Huh, is this the SSIM I think it is? Can you explain how you get this formula?

[markrb]

I may end up being forced to use x264 anyway.
For some reason Megui and Mkvmagic both are crashing during the encode with Xvid. While x264 makes it through without issue.
I don't have the time to do much debugging. The AVS plays fine in media player and it encodes for a good time, but dies everytime at different points.
I even have Megui create and check everything.

BTW what would you consder to actually be low, mid and high bitrates anyway?
I hear the words, but no numbers.

Thanks,
Mark

[siddharthagandhi]

Low= 500 kbps to 1 mbps
Medium- 2 mbps
High- 5-8 mbps
Ultra High HD Bitrate- 16-20 mbps
Insane High- 50 mbps
WTF is this?- 200 mbps and above

[*.mp4 guy]

BTW what would you consder to actually be low, mid and high bitrates anyway?
I hear the words, but no numbers.

Thanks,
Mark
Low= 1cd (650-950 kbps)
Medium= 2cd (1000-1650 kbps)
High= 1/2 DVD (1700-3200 kbps)
Transparent= ~3-9mbps for standard definition ~6-18mbps for 720P ~12-24mbps for 1080P
(all assuming a framerate of 23.976 or 25, and that all footage is progressive or telecined)

[GodofaGap]

I'm not sure if it makes sense to talk about bitrates as high or low without knowing anything about the source. Perhaps defining it as a quant(-range) would be more useful?

So for XviD: high ~q2, medium ~q4 and low ~q8? Which makes x264: high ~q18, medium ~q24 and low ~q30.

[siddharthagandhi]

I did some testing, and I figured out that AVC vs. ASP depends on resolution as much as bitrate. With a 400 by 300 file at 2 mbps, AVC is better than ASP. With a 512 by 384 file at 2 mbps, asp beats AVC, not by a lot, but still beats AVC. The two codecs I tested were Nero Digital AVC and Xvid ASP.

[nm]

I very much doubt you would arrive to the same conclusion if you tested 10 different sources at both resolutions. Your observation is most likely just a fluctuation caused by the sources you tested.

[Audionut]

And the fact your using nero based avc.

[siddharthagandhi]

What's wrong with Nero based AVC? I get much better results with Nero AVC than x264.

And it is a fluctuation. Now that I look at it closer, its probably AVC thats better. Its just that there are certain scenes that look horribly worse.

But btw for some reason I get better quality using Xvid than x264 what could be responsible for this? I use default settings, is there anything I should change?

[Blue_MiSfit]

I think Sharktooth has a very good point about ASP preserving minute details like film grain better at high bitrates.

I have found that if you do an unfiltered encode (only crop and resize, or just mpeg2source) with a movie with lots of film grain (I recently did Three Kings), XviD tends to preserve the grain better. x264 seemed to turn the nice grainy blue sky into a soup of blue and red chroma soup - this was at full NTSC with 1/2 DVD target filesize with the original 448kbps 5.1 AC3.

Of course, some gentle denoising (tweaked fft3dgpu IIRC) really improved this, while preserving most of the grain. However, I still found XviD with the EQM V3HR matrix to be superior to AVC. That same chroma noise soup was much less apparent with this when compared to x264.

In conclusion, with cleaner sources, and/or tighter compression (TV shows etc) I always use x264. But with very film-like movies (lots of grain, high contrast), XviD just seems to preserve minute details and sharpness a tiny bit better, but only when I provide sufficient bitrate, and take the time to make a good script.

However, nobody but a video encoder with trained eyes can tell the difference, honestly. Most of my friends still think my fuzzy old half resolution divx rips from 4 years ago look good :)

~MiSfit

[siddharthagandhi]

Yea I agree on the last part. I sent my friend a video 92 kbps Windows Media 9 and he said it looked excellent. LMFAO!

[Audionut]

What's wrong with Nero based AVC? I get much better results with Nero AVC than x264.


But btw for some reason I get better quality using Xvid than x264 what could be responsible for this? I use default settings, is there anything I should change?


http://forum.doom9.org/showthread.php?t=110427

[Momotte]

If I had to encode 720p or 1080i HD, would you recommend I use AVC or ASP ? If I understood well, high bitrates should not mather that much so since HD is high bitrate... I am confused...

[ChronoCross]

If I had to encode 720p or 1080i HD, would you recommend I use AVC or ASP ? If I understood well, high bitrates should not mather that much so since HD is high bitrate... I am confused...

use AVC.

[Blue_MiSfit]

aggreed. At larger frame sizes especially AVC exhibits a strong advantage IMHO

[KafesneBikaina]

aggreed. At larger frame sizes especially AVC exhibits a strong advantage IMHO
Could you explain the reason for that?
I'm testing Nero-AVC vs XviD with HD source now at high bitrates.

[akupenguin]

well in fact it's more like a(1-e^(-ax)) for SSIMHuh, is this the SSIM I think it is? Can you explain how you get this formula?
The "internal" representation of SSIM, as presented by the original paper (http://www.cns.nyu.edu/~zwang/files/research/ssim/), produced scores in the range [0,1] with 0 being highest quality. Whoever it was that wrote the avisynth filter implementing SSIM decided to remap the scores to a different range, [0,100] with 100 being highest quality. The remapping formula was arbitrarily picked as: 100*(1-internalssim)^8.
I don't know offhand whether the convergence is e^(-ax) or some polynomial, but this is where the form of the formula comes from.

The same way the internal representation of PSNR is SSD, and psnr=-10*log10(ssd/max_ssd).

[Momotte]

aggreed. At larger frame sizes especially AVC exhibits a strong advantage IMHO


Strong ? How strong... is it worth the huge time difference to encode in AVC ?
With Cyberlink power encoder, the process is relatively fast but I cannot manage to get AAC / AC3 audio with the tool.

I tried to encode using x264 + MeGUI, but it is so slow, it is not viable (< 1fps most of the time)

My system: Athlon X2 3800+ OC
Original content: .ts file / 1920x1080i / AC3 5.1
Destination content: .mp4 / 1920x1080 / x264 2 pass / AAC 5.1

I am willing to try AVC as long as the encoding does not take forever, maybe its me who does not understand how to speed up the process...

thanks for your time,

[Sagittaire]

Strong ? How strong... is it worth the huge time difference to encode in AVC ?
With Cyberlink power encoder, the process is relatively fast but I cannot manage to get AAC / AC3 audio with the tool.

I tried to encode using x264 + MeGUI, but it is so slow, it is not viable (< 1fps most of the time)

My system: Athlon X2 3800+ OC
Original content: .ts file / 1920x1080i / AC3 5.1
Destination content: .mp4 / 1920x1080 / x264 2 pass / AAC 5.1

I am willing to try AVC as long as the encoding does not take forever, maybe its me who does not understand how to speed up the process...

thanks for your time,

1 fps with your X2 ... lol
Well with esa me, 16 ref, trelli, and rdo yoau can make 0.1 fps I think ... why not ... ???

I can make MPEG4 ASP encoding at 0.1 fps too with libavcodec and Insame setting and my conclusion will be "ASP is slow".

Use the slowest setting for x264 is not a obligation and it's certainely very bad idea for 1080p encoding ... lol

One more time at the same speed x264 done always very better quality than XviD (quality is objective test here).

[Momotte]

It was not a bad critical statement, you can view this also as a question... maybe someone can point me the proper settings for 1080p since obviously I am doing something wrong to have such a slow encoding speed. I used the HQ slow template, which one is suggested for 1080i HD content ?

thanks, sorry if my comment was not appropriate.

[emmel]

I tried to encode using x264 + MeGUI, but it is so slow, it is not viable (< 1fps most of the time)

My system: Athlon X2 3800+ OC
Original content: .ts file / 1920x1080i / AC3 5.1
Destination content: .mp4 / 1920x1080 / x264 2 pass / AAC 5.1
I have a similar setup, and also I'm encoding 1080i mpeg2 ts-content into AVC.

My target is basically to reduce the file size with 50% while achieving an almost transparent quality (let say PSNR 45...50dB). This is not necessarily possible with ASP (or at least I have not found out a way of doing it right), but iboth goals can be achieved with x264 --crf quite easily. And if you choose the parameters correctly, you can get about 4fps. Good enoug for me.

[Lefungus]

Whoever it was that wrote the avisynth filter implementing SSIM decided to remap the scores to a different range, [0,100] with 100 being highest quality. The remapping formula was arbitrarily picked as: 100*(1-internalssim)^8.

This is why I also added the 'scaled' parameter to disable this arbitrary scaling and keep range between 0-1, like in the original paper.
However, due to the absence of documentation, this feature may be unknown to most people :)

[Isochroma]

Some things to think about:

1. Crop. x264 needs mod16. Is your source (after cropping) mod16? Do you want to use a resize filter to make it such if it's not?

2. Encoding speed. XviD is faster
3. Decoding speed. XviD is faster
4. Encoding flexibility. Many more apps support vfw xvid.

Since I only do q-1 encodes I use DivX 6.2 for SD. I never use avc because of its dimensional inflexibility, high CPU usage on decode, encoding speed limitation, etc.

[GodofaGap]

1. x264 doesn't need mod16 anymore for a long time.
2. Depends, quality/speed wise x264 is probably better. If you compare encoding speed, you must compare encoding quality as well.
3. MPEG4 ASP with pp can be just as slow as MPEG4 AVC, I think. Depends on decoders used. Would make for an interesting test though. :)
4. There is a x264vfw too, it has worked in any app I tried so far.

But yea... with q1, thank god for DL-DVD's :)

[Isochroma]

1. It can accept non-mod16, but uses an internal workaround to expand the actual encoded frame. AVC only supports mod16.

2. I was only talking about speed.

3. I've tested decoding speed via ffdshow for asp vs. coreavc for avc. ASP decode is faster at same bitrate and also at same quality.

4. VFW mode can be used for x264, but it is not recommended. See other threads on the forum for more information. MKVmerge will throw a warning and refuse to mux vfw-mode avc-in-avi files.

[GodofaGap]

1. This is true for any MPEG codec. Macroblocks are 16x16. Period. XviD uses a workaround too.
2. You win. Although I never tried x264 without inloop and cabac. :)
3. I will try and test myself too. :)
4. Some people say it is not recommended for XviD as well. Same b-frame problems. And there is always "--engage allow_avc_in_vfw" :)

[Isochroma]

Do note that some AVC decoders (such as Nero) will pad the output frame with an ugly green bar if either dimension is not mod16. However, the divx codec will do x mod4 y mod2, ffdshow vfw encoder will do both mod2, and no asp decoder that i've ever tested padded the frame for mod2 sizes.

[foxyshadis]

So does ASP, dude. All of MPEG is based on the 16x16 DCT, which is where that requirement comes from. Xvid uses the same exact workaround. (x264 copied it verbatim.) However, the real and visible frame sizes do not have to match. Nero is just stupid in that regard.

Speedwise, AVC can be as fast as fast ASP, but at that point it practically is ASP with minor changes, and decoding won't suffer much either (because you'd use CAVLC and no 8x8. b-pyramid and an extra ref or two would be the major gains).

[Sagittaire]

Since I only do q-1 encodes I use DivX 6.2 for SD. I never use avc because of its dimensional inflexibility, high CPU usage on decode, encoding speed limitation, etc

MPEG4 ASP at q1 ... it's certainely not an optimal use for this codec. Well if you want speed (encoding and decoding) and flexibility MPEG2 will be really better for your quality level.

[Zorrander]

I usually encode movies in the 1500-2000kbps range and can see very noticable differences between Xvid and H264. Xvid files, even at high bitrates, tend to have dark color blocks and other noticable artifacts. At least that's how they end up on my machine.

[Blue_MiSfit]

Seriously. Q1 with ASP is just a bad idea IMHO. If you need to preserve that much information stick with MPEG-2.

Regarding encoding speed... I can encode 1080p in x264 with my usual settings - fft3dgpu, lvl1 rdo, lvl1 trellis, 5 refs, 8x8dct etc at roughly 2fps for secondpass (about 5 fps for first). This is slow, but doable. Encoding speed is of little importance when compared to output quality IMHO. Provided of course you actually have the time to do the encode. <1fps is unacceptable though.

And regarding my statement about AVC being superior in HD resolutoins, I will admit I dont have any quantitative proof to show that. However, it does make sense, considering that with HD there's more information per frame, and with more information, things like multiple references and 8x8dct would be more useful. Perhaps :)

But I have nothing to back it up with I suppose, so I will retract that statement for now, since there seemed to be some questioning of its validity.

On another note, I have been looking for some really high quality MPEG-2 HD clips. The only ones I have are from satellite, and are starved for bitrate, not to mention hybrid progressive/interlaced. *YUCK*... And all the Apple stuff is in AVC already... Anyone know where to get some really high quality MPEG-2 HD to play with encoding?

Thanks

~MiSfit

[Manao]

So does ASP, dude. All of MPEG is based on the 16x16 DCT, which is where that requirement comes from. Xvid uses the same exact workaround. (x264 copied it verbatim.) However, the real and visible frame sizes do not have to match. Nero is just stupid in that regard.Erm. Firstly, MPEG1/2/4 use the same 8x8 DCT ( it has never been 16x16 ). Secondly, MPEG1/2/4 and AVC use the same base unit : the macroblock, which is 16x16 ( for AVC in MBaff, it could be argued that the unit is the macroblock pair, 16x32 ). Thirdly, it's not a workaround. The standards perfectly accepts mod 2 resolutions ( and mod 1 for 4:4:4 ). In such a case, the encoder must fill the missing border pixels. He can use any algorithm to do so, because they won't be shown anyway. x264, XviD - and actually most of the encoders I think - use the same technic which consists of copying the border pixels. Filling with black would have create an edge, which is something hard to encode, so it wouldn't have been wanted.Do note that some AVC decoders (such as Nero) will pad the output frame with an ugly green bar if either dimension is not mod16And that is a bug. Those pixels shouldn't be seen, and actually aren't green when decoded ( so it's most probably pixels that haven't been initialized at all ).

ASP decoding will always be faster then AVC decoding, even if you encode with CAVLC, no deblocking, no partition... AVC is just intrinsicly more complicated. In the same way, ASP's fastest encoding speed will be better than AVC's.If I had to encode 720p or 1080i HD, would you recommend I use AVC or ASP ? If I understood well, high bitrates should not mather that much so since HD is high bitrate... I am confused...High bitrate must be considered relatively to the complexity of the encoded video. 640x272 will require a quite lower bitrate than 1280x720 to look as nice, because there's 5.3x less pixels to encode.I think Sharktooth has a very good point about ASP preserving minute details like film grain better at high bitrates.It only preserves the grain. Everything else will be sharper and more detailed with AVC. And imho, the grain is something that should be handled by FGM anyway.What's wrong with Nero based AVC? I get much better results with Nero AVC than x264.Perhaps you're misconfiguring x264. Because, x264 is a bit better than Ateme's december 2005 codec, which itself is better ( by a fair margin ) than Nero's codec ( Ateme's september 2004 ).

[temporance]

It only preserves the grain. Everything else will be sharper and more detailed with AVC. And imho, the grain is something that should be handled by FGM anyway.
Interesting thread. IMHO, it is not just grain that is better preserved by ASP, it is all fine texture. AVC, for me, becomes too bland, flat and cartoon-like at low bitrates. ASP, on the other hand, preserves textures like facial stubble and wrinkles at the cost of some blockiness. Parametised texture (e.g. FGM) can not possibly do this.

Edit: Oh, and the reason why AVC has an advantage at higher resolutions is that it's low-pass filtering effect is similar to resizing and encoding at a smaller resolution. So, if you find yourself encoding HD in AVC and notice texture loss, try downsizing first and you'll possibly get a lower quantizer encode with better detail preservation, even though you're working at a lower res.

[Manao]

Indeed, it can't. But I think that at low bitrates, artifacts brought by ASP are more disturbing anyway. AVC comes out clean ( too clean, according to you ), and remarkably sharper than ASP ( imho ). And since ASP at low bitrates is a block fest in high motion, it makes deblocking mandatory, which in turn wash out textures anyway. So...

[*.mp4 guy]

Indeed, it can't. But I think that at low bitrates, artifacts brought by ASP are more disturbing anyway. AVC comes out clean ( too clean, according to you ), and remarkably sharper than ASP ( imho ).

The lack of texture details combined with the very sharp lines can give some very odd looking low bitrate encodes, the problem used to be much worse, but it is still a rather odd effect.

[Sharktooth]

AVC can preserve fine textures but that requires good psy algos, and of course, no loop deblocking (or a low qp...).

[siddharthagandhi]

"Perhaps you're misconfiguring x264. Because, x264 is a bit better than Ateme's december 2005 codec, which itself is better ( by a fair margin ) than Nero's codec ( Ateme's september 2004 )."

You can't just say that based on one codec comparison like Doom9's. MSU's says the opposite.

[Manao]

MSU's tests are not good tests ( read the following answers in the respective threads ). And anyway, I do not say that based only on Doom9 comparison, but from experience. The fact is that you're the only one to find Nero AVC better than x264.

And since, on another thread, you said :Nero isn't silly. Wait till I release my codec comparison. The PSNR's show that MP is better than x264, and HP is a lot better than x264.I can bet you that you misconfigured x264. Because, metric-wise, x264 is better than Nero ( except if the video is exclusively made of fades ). And how can you speak of Nero HP while it's still only vaporware ?

[IgorC]

Notice that Manao is Ateme developer. It's more than enough.

[IgorC]

Because, metric-wise, x264 is better than Nero ( except if the video is exclusively made of fades ). And how can you speak of Nero HP while it's still only vaporware ?

It is true if video consists only from fades as it says the message.

x264 had not problem with fades. x264 has weight prediction for B frames (not for P?). Maybe x264 adds some extra bitrate on fades but as fades aren't enough often so it shouldn't be noticeble waste of bitrate.

[siddharthagandhi]

I got a Nero HP encoder from someone. He said he got it from the Doom9 forum.

"Notice that Manao is Ateme developer"

Is he really? Then why is he diminishing his own codec?

[IgorC]

First of all it's illegal.
Second it's not HP cli version but old MP licked beta.
Check the version number.

Bye.

[xyloy]

I can bet you that you misconfigured x264.
I bet this too!

@siddharthagandhi:
To see x264's true quality( :P ), update to the last rev. build (http://x264.nl/) and try theses settings(wich maximise the quality without killing the encoding speed) for a 2-pass encode(3pass mode is only when the filesize is wrong):
x264.exe --pass 1 --bitrate xxx --stats "x264.stats" --ref 8 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --analyse all --8x8dct --me umh --merange 16 --cqm "jvt" --progress --no-psnr --output NUL "x:\x264.mp4"
(bitrate and output folder are your choice)
Or a turbo 1 pass(very little side effects on finale quality AFAIK):
x264.exe --pass 1 --bitrate xxx --stats "x264.stats" --bframes 3 --b-pyramid --direct auto --subme 1 --analyse none --me dia --merange 16 --cqm "jvt" --progress --no-psnr --output NUL "F:\x264.mp4"

2nd pass:
x264.exe --pass 2 --bitrate xxx --stats "x264.stats" --ref 8 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --analyse all --8x8dct --me umh --merange 16 --cqm "jvt" --progress --no-psnr --output "x:\x264.mp4"

Last note, I'm currently using --merange 24, but that's because time is unimportant to me, and I like to play with x264. ;)

[Manao]

IgorC : Extract a fade from a video, and encode it with weighted prediction in Nero and with x264, both at the same bitrate. The difference should be obvious. But of course, fades mostly happen in trailers, and there must be less than 0.1% of a movie that is made of fades. So it doesn't tip the balance towards Nero.

Is he really? Then why is he diminishing his own codec?I don't think I am. Saying that our codec is on par with x264 isn't diminishing it in any way, since x264 is the best ( there's no best, but you know what I mean ) codec out there at the moment. Saying otherwise would be a misstatement.

[Sagittaire]

I got a Nero HP encoder from someone. He said he got it from the Doom9 forum.


it's not me, it's not me ... lol
Manao is certainely happy to know that ... :eek:

"Notice that Manao is Ateme developer"
Is he really? Then why is he diminishing his own codec?

Simply because it's true. x264 is little better than Ateme/Elecard for metric and IMO visually all these top H264 encoder are really close now.


Extract a fade from a video, and encode it with weighted prediction in Nero and with x264, both at the same bitrate

perhaps because x264 use only bwpred ... ???

[foxyshadis]

Ateme devs don't need to worry about absolute quality being just a tiny bit less, given how much faster theirs is... They were scorching in the 2005 comparison.

And yeah, aku said he hasn't done full b-wpred yet:
http://forum.doom9.org/showthread.php?p=817250#post817250

[Blue_MiSfit]

It only preserves the grain. Everything else will be sharper and more detailed with AVC. And imho, the grain is something that should be handled by FGM anyway.

What is this fgm? I dont recall reading about it.

I think grain preservation is extremely important when dealing with film sources. Noise removal is always important, but keeeping grain ads so much to the image. Removing grain with AVS filtering usually works, but at the cost of too much detail / sharpness IMHO. I totally agree with what temporance said regarding the wrinkles and stubble. That's something I have ALWAYS noticed.

However, what Manao responded with is also quite valid. Low bitrate, non PP'd (obviously), slightly oversmooth AVC looks a lot better than low bitrate, PP'd blocky ASP. I was thinking more along the lines of midrange bitrate.. .8-1.2 mbit

Maybe I'm not configuring things properly or using the right prefiltering.. or maybe this FGM thing is the holy grail of grain preservation (:D)

~MiSfit

[IgorC]

Ateme devs don't need to worry about absolute quality being just a tiny bit less, given how much faster theirs is...They were scorching in the 2005 comparison.

Yes, it was 2005 year. (revision aprox. 380)

Now it looks more like that http://forum.doom9.org/showthread.php?t=105763 . The last revisions are even better in trading of quality/speed.

[Sagittaire]

Yes, it was 2005 year. (revision aprox. 380)

Now it looks more like that http://forum.doom9.org/showthread.php?t=105763 . The last revisions are even better in trading of quality/speed.

well I think that x264 is the best for quality (aka metric) and now for speed too ... but I don't test the last Elecard and Ateme encoder. If x264 can progress certainely that Elecard and Ateme can progress too.

Now I think that big improvement for quality will be really difficult (no impossible but difficult and slow). The last improvement for x264 are now majoritary speed improvement.

[*.mp4 guy]

Now I think that big improvement for quality will be really difficult (no impossible but difficult and slow). The last improvement for x264 are now majoritary speed improvement.

It may seem like that now, but look at the amazing amount of progress LAME has made, in recent double blind tests (at 128kbps vbr) it was statistically tied with Itunes AAC, AoTuv vorbis and WMA Profesional. Source (http://www.maresweb.de/listening-tests/mf-128-1/resultsz.png)

[foxyshadis]

He never specified whether it was the encoder that was current as of his test, or the old test codec that was released in late spring/early summer 2005 iirc. If it was current, great, but if it was the old beta release, then its results can't really be trusted to apply to the current codec a year later (even though we can't use it yet). I'd be interested in seeing current results, but I'm sure Ateme's devs have better things to do than that. :p

[edit] Looks like the last beta.

I think the big leaps in quality will end up being when currently unrealized portions of the spec are implemented, and entirely new tunings and psychovisual models developed to put bits in more important areas.

[Sharktooth]

x264 still lacks psy enhancements (psy adaptive quant, psy motion search, psy RD), weighted prediction in Ps, edge detection based on intra search, macroblock partitions (B8x4,B4x8,B4x4), adaptive refs, and some other possible tunings and enhancements i dont actually remember.
despite all that, x264 is one of the best avc encoders out there and it's free...

[akupenguin]

edge detection based on intra search
edge detection (http://students.washington.edu/lorenm/src/x264/x264_edge_intra.0.diff) is a speed/quality tradeoff, and not a good one.

[Sharktooth]

thanks for the clarification.

[Dayvon]

Just to throw a quick 2 cents in the ring here...

I would use x264 for anything under 2000kbps. Of course, it all depends on the frame size too, but for a normal frame (900x500 and down) Xvid is just too weak under 2000kbps. For a true high quality rip. You can see this by actually watching the "sharper" details of the Xvid encode... They exhibit this pixelated look where you can see the "bits".

Now, I do find that at 2500kbps+ I would use xvid over x264 because of the details. At this high of bitrate, the problem I mentioned before is lessend dramatically and x264's difficulties with "fog" type motions becomes comparably worse. Xvid is better with rain at high bitrates as well because of its detail retention.

For HD, I wouldn't be able to say, but I would have the hardware you use as a major consideration. Xvid obviously is quicker on the decode (and encode) which could make all the difference for your PC/hardware. 1080p@60fps is not something that you will want in AVC. But if you have good decoding, then conserve some space and use AVC for size/quality improvements.

I know I write in general terms and not specifics (like the PSNR guys), but that is because I consider experience and my eye to be a great judge. Because I am so dang critical ;).

BTW, is there a xvid version that works with MeGUI yet? I switched to the MP4 container awhile ago (and MeGUI) and I'd rather keep my encodes non-VFW and MP4 compliant. I know of Xvid_encraw but have had no luck with it yet. I find it's also hard to find the most recent build.

[foxyshadis]

The best place to get it is always ftp://squid80.no-ip.com/xvid_encraw.zip and you can check the encraw thread for other mirrors (not updated as often) as well. The new versions work better with megui than they used to.

[Dayvon]

The best place to get it is always ftp://squid80.no-ip.com/xvid_encraw.zip and you can check the encraw thread for other mirrors (not updated as often) as well. The new versions work better with megui than they used to.
Thank you sooooooo much.
Xvid_encraw needs a good mirror. I wish I had a ftp. :o

[Morte66]

I have a similar setup, and also I'm encoding 1080i mpeg2 ts-content into AVC. [...] And if you choose the parameters correctly, you can get about 4fps. Good enoug for me.

So what settings, specifically, do you use?

[Morte66]

1 fps with your X2 ... lol[...]
Use the slowest setting for x264 is not a obligation and it's certainely very bad idea for 1080p encoding ... lol

So what do you suggest for 1920x1088/24p at about 4000 to 6000kbps starting from very imperfect MPEG2 HDTV TS?

Perhaps you could post a pair of command lines that don't include "lol".

[Sagittaire]

x264.exe --bframe 2 --weightb --ref 2 --direct auto --filter -1:-1 --crf 26 --qcomp 0.75 --ipratio 1.25 --pbratio 1.33 --analyse "all" --8x8dct --me "hex" --subme 5 --progress -o 1080p.mp4 1080p.avs

don't forget to use deblocking for your HDTV source

[foxyshadis]

Thank you sooooooo much.
Xvid_encraw needs a good mirror. I wish I had a ftp. :o
Well, like I mentioned, there are a few in the thread since Squid's home server is so flaky. :p

http://elicit.vn.ua/files/hosted/
http://brother-john.net/

I'm sure there's others I missed.

Morte:
Try 1P-Intermediate profile in MeGUI with a CRF of your choice, 2 threads, b-pyramid, and an extra reference or two. That should be fast enough. (Don't bother with ABR, just switch to auto 2pass if you need a size.) Depending on the speed of your computer you may or may not need to switch off CABAC. Hopefully not, it'll drop quality and/or increase bitrate.

That profile would make a good starting point for an HD 1080p profile (hint sharktooth).

[Edit]
Yes, approximately what sagittaire suggested.

[Morte66]

I am about to start converting many of my DVD's into mkv's and I am still debating over which I should use.

I am not asking what is best here, but more to the point is there enough of a quality difference at higher bitrates for me to use H.264 and the extra encoding time?

I am looking to create files around the 2gb range for a typical movie.

I've been mucking about with some higher-quality encodings today on a 113 second HD clip made from 8.24GB of 1080p uncompressed YUV samples. I used Xvid, x264 and QuEnc. Nothing rigorous here, but it might interest you.

I used x264 with a variety of high profile settings similar to Sharktooth's HQ-Slow profile for MeGUI, at around 4500kbps (thinking HDTV movie to DVD5 backups) and around 8000kbps (wondering what HDDVD might look like). They all looked the same as the source and each other, apart from a sky shot that I'll come to. The only settings that really changed the character of the video were the deblocking options: -2,-2 looked a hell of a lot better than -1,-1 to my tastes.

Xvid single pass q2 looked pretty much level with the best x264 encodes, but the bitrate came out at 26103kbps. [These settings tend to give about half original size for a DVD backup, with a big variation.]

QuEnc on guestimated Blu-Ray MPEG2 settings (16000kbps VBR 2 pass) looked poor compared to the MP4, but wonderful compared to MPEG2 HDTV transport streams. If you like PS3 games and want to watch a few HD movies but not many...


As for that sky shot (3 seconds out of 2 minutes with some smoothly graduated haze):

The haze came out subtly posterised/banded in all the h264 encodes. This was the only shot where I noticed a difference between crf18 and crf22 without actively looking for it, for about half a second I saw a block on crf22.

The haze was smooth in xvid, no sign of banding, but the scene was 271 frames long and at frame 250 where the I-frame came in Xvid suffered an obvious jump (like a blob of colour hitting the screen).

QuEnc showed gross blocking on this section.

Just sitting back and watching it like a movie, I didn't notice a difference between the encoders except that x264 with stonger deblocking was a bit soft and QuEnc was very noisy/blocky. Xvid vs x264 was pretty even, but x264 needed less than a third of the bitrate to be "transparent for my purposes" and Xvid encoded five times as fast.

[Morte66]

Sagittaire/Foxyshadis: thanks, I'll give those a spin.

{edit} I wasn't so happy with that at --crf 26 --filter -1,-1; but it looks good at --crf 22 filter -2,-2. I'm pretty impressed that it gave twice the speed (2.32fps on A64 X2 3800+) for 20% more output size, compared to using my usual DVD backup settings at 1920x1080. Unfortunately it comes out at more than 8000kbps, which would mean DVD9 rather than DVD5. Well, I'll have to do some more tests.

[My DVD backup settings, developed by trial and error on a wet Sunday, are --crf 22 --ref 5 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct temporal --filter -2,-2 --subme 6 --trellis 1 --analyse all --8x8dct --threads 2 --thread-input --progress --no-psnr --output "" "". I reaslise they're on the slow side of "best bang for buck", but I encode overnight so 4 hours vs 8 hours doesn't really matter and it saves me time because it's less discs to burn. Then along camed HDTV, and specific target sizes needing 2-pass encodes, and everything changed. If I didn't like DGIndex deblocking and ColorMatrix() so much I'd just burn the transport streams over three discs.]

[audioman]

Hi all forumers

Sagittaire , I sent you a PM and no answers :)

As far as I am concerned , I Made my choice on X264 codec
Just make a test on the " NEMO MOVIE " which causes the xvid and divx codecs to fail , showing some big artefacts and some image degradation even at High bitrates

Make the same test using X264 with HQ matrix slow, Lanczos and Bitrate 900 , it is quite astonishing
The compressibility test gives me a 52 % value
Neither artefacts nor image degradations and the final image file is 650 megs

My question is :

Is it better to use Lanczos instead of Neutral ?
Do you often use compressibility test ?
Is it worth it using higher bitrates when encoding Non HD movies

Thanks

[smok3]

what would be x264 single pass transparent settings for 720x576 progressive (dv) material, where speed is relatively important?

[ToS_Maverick]

what would be x264 single pass transparent settings for 720x576 progressive (dv) material, where speed is relatively important?

you can check this (http://forum.doom9.org/showthread.php?t=121304&page=9) thread, where i tried to get the last out of x264, and compared it with xvid.

here is the commandline i use if i need good quality and speed (HDTV coding for example):

--crf 20.0 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --bime --weightb --filter -2,-2 --subme 1 --analyse p8x8,b8x8,i8x8 --8x8dct --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output

[smok3]

ToS_Maverick, tnx will try, is --threads auto for dual-core cpus or should i force threads 2? (quick test didnt reveal any significant speed changes...)

[ToS_Maverick]

threads auto will pick the optimal thread-setting. no need to change it. i should work, as i get 50 fps @ dvd-resolution :D

[aLu]

I was greatly intrigued by x264, but am now for the most part returning to Xvid. Why? Because x264 can look great at any bitrate, but tends to smooth away fine detail while it enhances/darkens borders, which can look great if you have strong shadows. In direct comparison, I simply found Xvid to preserve more detail (especially with a custom matrix). I've tried x264 at it's slowest, optimum quality setting with high as well as with low bitrates.
This is a purely visual judgement. I'm sure H264 supposedly yields more quality, but as always it is statistics vs. eyes/taste. If you use Xvid with smooth h263 quant, they will be indistinguishable.

[foxyshadis]

There's a more recent thread that discusses the effect and how to avoid the smoothing. To recap: Lower deblocking, custom quant matrices, subme 5 + no trellis, and/or deadzone.

[Dark Shikari]

There's a more recent thread that discusses the effect and how to avoid the smoothing. To recap: Lower deblocking, custom quant matrices, subme 5 + no trellis, and/or deadzone.I find that turning off RDO doesn't really help avoid smoothing (it just wastes bits); the most important thing is to use obnoxiously low deadzone settings.

[plane]

I find that turning off RDO doesn't really help avoid smoothing (it just wastes bits); the most important thing is to use obnoxiously low deadzone settings.

Do you have any recommended low deadzone settings?

[Dark Shikari]

Do you have any recommended low deadzone settings?Just keep dropping them lower until it gets how you like it, IMO.

Note that very low deadzones effectively "sharpen" the image.

[Sharktooth]

subme 1 is infinitely more sharp than subme 6...

[Dark Shikari]

subme 1 is infinitely more sharp than subme 6...That's ridiculous; subme 1 = more intra blocks = lower sharpness because intra blocks are worse psy-wise when there is a valid MV. I have never seen a case in all my work with x264 where lower subme values help; rather I've found they result in terrible-looking "stepping" where slow-moving backgrounds no longer move smoothly, and other such things.

The only way that can be true is if you're at a constant quantizer, and as a result subme1 has a 30-40% higher bitrate than subme6--no crap its going to be sharper.

Edit: actually, there is one other case I would think lower motion search precision would increase sharpness. If you have low quantizers (so inter blocks are preferred to intra in any case a good MV can be found), a lower-precision MV could result in "fake sharpness" due to the high frequencies from the previous block being carried over to the new block but not matching up with the high frequencies in the source. This would really only apply in the case of grain, I would think.

[Didée]

Well, let's say that "sharpness" is a term that can be interpreted in different ways ... just because it sounds good, it hasn't necessarily to be.