ABR vs CRF - quality versus size (simple) question

[lchiu7]

I have started to rip and encode all my HD-DVD to h.264 retaining the original audio (usually DD+ and sometimes THD) and then authoring and burning to DVD (DL if need) in AVCHD format to watch on my Oppo.

Up till now I have been fairly simplistic in the x264 settings - just set ABR to about 6500 making sure the resultant output file plus the audio can fit on a DL disc. I am not sharing these with anybody so space isn't a real concern.

But encountered some forum discussion on CRF as being the best one pass encoding technique so out of interest encoded a 1H45 minute using these settings

--profile main --level 4.1 --preset fast --crf 22 --thread-input --keyint 24 --min-keyint 2 --no-scenecut --ref 3 --ipratio 1.1 --pbratio 1.1 --rc-lookahead 40 --me dia --subme 2 --partitions p8x8,b8x8,i4x4,p4x4 --no-8x8dct --trellis 0 --weightp 0

(I have to use no weighted p frames since the Oppo currently can't handle weighted p frames).

The output video file is about 2.5G (I usually end up with files around 5-6G) and looking at the result on my 100" 1080P projector I can't see much difference between the original and the encode.

Is CRF that much better than ABR or do I have some other compromises going on that I am not aware of?

Thanks for any guidance on this.

An an aside encoding speed is about 8 hours on a dual core AMD64 X2 3800 under Windows 7

[Soichiro]

ABR, from what I know, is not very efficient at distributing the bitrate. It has to do some weird stuff to maintain the average bitrate or something. I don't know all the details since I'm not a dev (nor do I ever use ABR), but CRF will produce much better quality. If you absolutely must have a constant filesize, it is best to use 2-pass VBR encoding. It will take longer, but the quality will be much higher. If you aren't overly concerned about quality/compression efficiency and need faster encoding (and specific filesize), ABR is probably your best alternative though.

I can run a test encode to compare ABR and VBR at the same bitrate and get back to you, if you would like.

Edit: Tests done. Actually not as big of an improvement as I was expecting, but VBR still beats ABR in quality (and isn't much slower with fast first pass on). You can also see that ABR is very bad at hitting the target bitrate (off by 5.8% in this example).

ABR Encode:
x264 --preset medium --tune psnr --bitrate 2500 --psnr --ssim --threads auto --thread-input
avs [info]: 1280x720p 0:0 @ 24000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile High, level 3.1
x264 [info]: frame I:40 Avg QP:18.45 size: 69571 PSNR Mean Y:48.40 U:53.83 V:53.57 Avg:49.22 Global:48.61
x264 [info]: frame P:1327 Avg QP:21.61 size: 16536 PSNR Mean Y:46.58 U:52.21 V:52.11 Avg:47.46 Global:46.35
x264 [info]: frame B:790 Avg QP:23.06 size: 2221 PSNR Mean Y:47.24 U:50.01 V:50.37 Avg:47.96 Global:47.48
x264 [info]: consecutive B-frames: 42.1% 21.0% 10.1% 26.8%
x264 [info]: mb I I16..4: 39.2% 38.2% 22.6%
x264 [info]: mb P I16..4: 13.8% 11.8% 4.0% P16..4: 16.1% 6.5% 2.8% 0.0% 0.0% skip:45.1%
x264 [info]: mb B I16..4: 0.6% 0.3% 0.1% B16..8: 15.2% 1.9% 0.4% direct: 0.6% skip:80.9% L0:42.9% L1:49.0% BI: 8.1%
x264 [info]: final ratefactor: 17.83
x264 [info]: 8x8 transform intra:39.5% inter:68.7%
x264 [info]: coded y,uvDC,uvAC intra: 37.3% 36.9% 19.0% inter: 9.7% 8.0% 1.0%
x264 [info]: i16 v,h,dc,p: 39% 29% 13% 19%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 17% 27% 6% 6% 7% 7% 7% 7%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 22% 17% 19% 6% 8% 9% 7% 6% 4%
x264 [info]: i8c dc,h,v,p: 63% 17% 15% 5%
x264 [info]: Weighted P-Frames: Y:6.0%
x264 [info]: ref P L0: 63.6% 15.3% 14.2% 6.7% 0.3%
x264 [info]: ref B L0: 86.7% 11.7% 1.7%
x264 [info]: ref B L1: 96.2% 3.8%
x264 [info]: SSIM Mean Y:0.9865517
x264 [info]: PSNR Mean Y:46.852 U:51.433 V:51.500 Avg:47.674 Global:46.765 kb/s:2354.74

encoded 2157 frames, 13.98 fps, 2354.80 kb/s

VBR Encode:
x264 --preset medium --tune psnr --pass 1 --bitrate 2500 --psnr --ssim --threads auto --thread-input
avs [info]: 1280x720p 0:0 @ 24000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT

encoded 2157 frames, 34.21 fps, 2368.21 kb/s

x264 --preset medium --tune psnr --pass 2 --bitrate 2500 --psnr --ssim --threads auto --thread-input
avs [info]: 1280x720p 0:0 @ 24000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT

x264 [info]: profile High, level 3.1
x264 [info]: frame I:43 Avg QP:18.77 size: 64493 PSNR Mean Y:48.39 U:53.62 V:53.37 Avg:49.23 Global:49.00
x264 [info]: frame P:1334 Avg QP:21.20 size: 17532 PSNR Mean Y:46.97 U:52.23 V:52.13 Avg:47.85 Global:47.49
x264 [info]: frame B:780 Avg QP:23.09 size: 2288 PSNR Mean Y:47.35 U:50.08 V:50.45 Avg:48.07 Global:47.54
x264 [info]: consecutive B-frames: 42.8% 21.2% 8.8% 27.2%
x264 [info]: mb I I16..4: 37.6% 41.2% 21.2%
x264 [info]: mb P I16..4: 13.3% 13.4% 4.4% P16..4: 15.2% 6.4% 2.6% 0.0% 0.0% skip:44.7%
x264 [info]: mb B I16..4: 0.5% 0.3% 0.1% B16..8: 15.0% 2.0% 0.4% direct: 0.6% skip:81.1% L0:44.2% L1:47.0% BI: 8.8%
x264 [info]: 8x8 transform intra:42.7% inter:69.4%
x264 [info]: coded y,uvDC,uvAC intra: 41.5% 35.6% 18.5% inter: 9.6% 7.7% 0.9%
x264 [info]: i16 v,h,dc,p: 37% 29% 14% 20%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 17% 25% 6% 7% 7% 7% 7% 7%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 22% 17% 19% 7% 8% 9% 7% 7% 4%
x264 [info]: i8c dc,h,v,p: 64% 16% 15% 5%
x264 [info]: Weighted P-Frames: Y:5.9%
x264 [info]: ref P L0: 62.3% 15.9% 14.6% 6.9% 0.3%
x264 [info]: ref B L0: 86.7% 11.7% 1.6%
x264 [info]: ref B L1: 96.7% 3.3%
x264 [info]: SSIM Mean Y:0.9884162
x264 [info]: PSNR Mean Y:47.136 U:51.479 V:51.549 Avg:47.957 Global:47.532 kb/s:2485.02

encoded 2157 frames, 14.90 fps, 2485.08 kb/s

[lchiu7]

I was aware that ABR isn't that efficient but at the moment I am doing an encode from a HD-DVD with a crf of 21 using the same command line as above and the target size is 2G. Of course it could change since x264 can't really predict a final size using CRF but I doubt if it's going to double in size.

[lchiu7]

I don't need to hit a certain filesize - just make sure I can burn the output to a DL disc as a max.

Your encode speeds are pretty fast. From a VC1 source using the settings I posted, I am getting about 5fps max. I guess I need more cores?

[Soichiro]

I don't think there's a good way to limit the filesize of the whole video in CRF mode. I suppose you could use VBV to limit the max bitrate, but this isn't the best option as it could potentially hurt quality in high-bitrate sections of the movie while leaving the total filesize far below the maximum. So I guess it's mostly about guessing and making sure you don't set the crf too low.

And yeah, I used to have the same CPU you have, and recently upgraded to a Phenom II X4. The speed difference is ridiculous, lol. It's not even necessarily the number of cores, but also just the speed of the processor plus the new instruction sets that weren't on the Athlon 64 generation of chips. A lot of parts of x264 have optimizations specifically for C2D and newer processors, so older CPUs won't get the same kind of encoding speed as a newer CPU even at the same processing speed.

[lchiu7]

Interesting - my goal is good quality, one pass and < 7.5G so I can mux the audio and still get on a DL disc. WIth the CRF settings I have I am seeing video < 3G which I guess is good but only viewing will tell.

I should upgrade. I also have a an AM2 X2 3500 but it doesn't seem much faster than the older one. Still 8 hours is okay since I can run them overnight.

[Blue_MiSfit]

If you absolutely must hit a specific filesize, then you should use 2 pass VBR (with VBV, since you're targeting AVCHD compatibility). If you can't afford the time, then either use faster settings, or live with 1 pass ABR :)

CRF is wonderful, but not such a good idea when targeting fixed size media.

Finally, why no 8x8dct?!?!? It's absolutely HUGELY beneficial!

Derek

[lchiu7]

Actually I don't care about a specific filesize since as I mentioned, so long as it's < about 8G I am okay. So 4-8G is in the range for me. If CRF one pass does it with good quality then I am okay.

Interestingly the encodes which are looking smaller originate as VC1 since that seemed to be quite common on HD-DVD whereas I just did one which originated as H.264 (BD title) and with CRF=21 and all other settings the same, the target size is about 4.6G for the same length movie. Of course scene changes etc. will change the final size but 2G seems like a lot of difference.

[Blue_MiSfit]

Cool, as long as you're okay with doing a re-encode if you exceed your ~8G soft limit.

Do use VBV if you're targeting AVCHD though. I'm not sure what the suggested VBV settings were for AVCHD / BD9... There seems to be a general consneus here: http://forum.doom9.org/showthread.php?t=145917&page=2 that ~24000 should be good values for maxrate and bufsize for BD9.

Also, looking at your original command line:

--profile main --level 4.1 --preset fast --crf 22 --thread-input --keyint 24 --min-keyint 2 --no-scenecut --ref 3 --ipratio 1.1 --pbratio 1.1 --rc-lookahead 40 --me dia --subme 2 --partitions p8x8,b8x8,i4x4,p4x4 --no-8x8dct --trellis 0 --weightp 0


There are some big :confused: bits here.

As I said in my previous post, why no 8x8dct? Your player can surely handle high profile! Also, why disable scenecut detection? AFAIK, this turns off adaptive I-Frame placement, which is a big no-no. If --preset fast is too slow still, then use --preset faster.

I understand your need for no weightp, that's fine.

Here's my suggested CLI:


--level 40 --preset faster --crf 21 --vbv-maxrate 24000 --vbv-bufsize 24000 --nal-hrd vbr --keyint 24 --weightp 0


Derek

[lchiu7]

Thanks for the suggestions - will check them out. As a point of note, I can't do 24000 since I think that will exceed the transfer rate off a DVD and cause stuttering to occur. I have no idea what 8x8dct means but I can add it.

[Dark Shikari]

Thanks for the suggestions - will check them out. As a point of note, I can't do 24000 since I think that will exceed the transfer rate off a DVD and cause stuttering to occur. I have no idea what 8x8dct means but I can add it.You have no idea what it is, yet you went out of your way to disable it?

If you don't know what it is, don't touch it!

[lchiu7]

You have no idea what it is, yet you went out of your way to disable it?

If you don't know what it is, don't touch it!

No I didn't go out of my way to disable it. I had just got one of the profiles in megui which indicated fast, changed to ABR and didn't really play with any other parameters apart from turning off weighted p frames since the Oppo can't handle them.

[Dark Shikari]

meguiThere's your problem. Use a GUI that isn't terrible.

[lchiu7]

There's your problem. Use a GUI that isn't terrible.

And that would be?

[Dark Shikari]

And that would be?Basically anything else? (http://forum.doom9.org/forumdisplay.php?f=78) Handbrake, Ripbot264, staxrip...

[Blue_MiSfit]

I think the GUI itself is okay, as long as you're using a recent build. However, the presets are sometimes terrible :)

I personally like Lord_Mulder's x264 gui, as it's extremely simple. Handbrake and ripbot264 are both great tools as well.

Derek

[aegisofrime]

MeGUI is good to me. It does have one thing that Lord Mulder's app doesn't have, and that's queuing. Like DS said you will be fine as long as you don't touch anything that you don't know, so just use x264's built in presets. That's what I do.

[lchiu7]

Basically anything else? (http://forum.doom9.org/forumdisplay.php?f=78) Handbrake, Ripbot264, staxrip...

You could be right. I found out why noDecimate was turned on and corrected it in the XML file. But (and I have the latest build) it still picked it up no matter what I did. Since I have a sort or production process where I am taking similar content to encode, I just grabbed the most appropriate command line and just now use the CLI and a small bat file to fire off x264.

I didn't like Ripbot264 at all and Handbrake encodes to MKV which I find a nuisance since that is not my target container (mine is m2ts)

[me7]

If you are using the CLI then use x264's internal presets. They are always up-to-date and they were made by people who really know what they do.
If you want an "advanced CLI", try Lord_Mulder's launcher (http://forum.doom9.org/showthread.php?t=144140). It's the simplest and best x264 GUI yet IMO.

[kypec]

LM's launcher is my favourite too. I do audio processing separately and also preparing/editing AVS scripts in AvsP before encoding.