x264 cpu intensive options: In-Loop Vs. Cabac. Which Is More Efficient?

[Terranigma]

From a quality standpoint, which would be more efficient and why? In-Loop Deblocking or Cabac (Trellis 2 + all macroblock options)? Last night I did a crf encode at 22.0 with the cabac options first, then later did an encode with CAVLC with in-loop, and i've noticed the bitrate with cabac was smaller by roughly 33% but there was visible macroblocks (since i opted not to use in-loop for cpu purposes). Laters I did an encode at the same bitrate using cavlc & in-loop, and noticed it lacked as much detail as with cabac.

My question is, if you could choose only one, which would you choose? Cabac + Trellis 2 + all macroblock options or CAVLC + In-Loop + All macroblock options except Adaptive DCT (which should only be used with CABAC) & I8x8?

Also, is it just me, or does in-loop requires more cpu usage than cabac?

[Terranigma]

Why shouldn't adaptive dct be used with cavlc?

Because it distorts the image. See pic
http://img170.imageshack.us/img170/9864/cavlcandadaptivedctfr0.png

You can download the settings i've used here (http://www.savefile.com/files/663764)

Edit:
Seems you've deleted the post.

Anyways, which would you prefer? The Cabac settings or In-Loop settings?

[LoRd_MuldeR]

Note: CABAC is a lossless compression. It makes file ~15% smaller without any loss in quality.

IMO you should use both, CABAC and In-Loop Deblocking, always!
Why the heck do you want to disable one of them ???

[akupenguin]

Why shouldn't adaptive dct be used with cavlc? Sure, it's not quite to as efficient to code a 8x8 dct using the 4x4 cavlc function as it would be if the standard had bothered including a cavlc function specific to 8x8, but that doesn't seem much worse than any other approximation made in cavlc.

Based on you other question, I assume the reason you want to disable something is that your cpu isn't fast enough to play back a stream with both features enabled. My standard response is: cabac and loopfilter are some of the most basic and important features of h264. If you would have to disable them for any reason, then just don't bother with h264 and go back to mpeg4asp.

Because it distorts the image.
That's a bug, which has simply gone unnoticed because no one uses cavlc+8x8dct. The only common reason for cavlc is baseline compatibility, and then 8x8dct isn't allowed either.

[Terranigma]

IMO you should use both, CABAC and In-Loop Deblocking, always!
Why the heck do you want to disable one of them ???
I agree, but this concerns certain software that is cpu sensitive. Disabling one or the other could remedy this. =P
What made me bring this question up was the discussion that was brewing in this (http://forum.doom9.org/showthread.php?p=992043#post992043) thread.

[LoRd_MuldeR]

What made me bring this question up was the discussion that was brewing in this (http://forum.doom9.org/showthread.php?p=992043#post992043) thread.

Well, this discussion is about encoding speed. But I still think it's kind of silly to first use a H.264 encoder and then disable the most important features of H.264. Why not simply use a MPEG-4 ASP encoder (e.g. Xvid) instead? Also there are other options in x264 you can tweak in order to get higher encoding speed (e.g. Trellis or ME Method).

[Terranigma]

Well, this discussion is about encoding speed. But I still think it's kind of silly to first use a H.264 encoder and then disable the most important features of H.264. Why not simply use a MPEG-4 ASP encoder (e.g. Xvid) instead? Also there are other options in x264 you can tweak in order to get higher encoding speed (e.g. Trellis or ME Method).

Well that discussion is about encoding speed, yes, but where this one differentiates at from that one, is not with the decoding/encoding speed by disabling certain settings (e.g. CABAC, Trellis, or In-Loop), but at cpu usage.
Could you perhaps list all the parameters that affects the cpu usage besides Cabac, Trellis, and In-loop (Only if it's not of too much trouble)? :cool:

[akupenguin]

huh? speed is cpu usage.

All options that affect decoding speed, however minor:
keyint (though only 1 vs >1. if it's a big number, 100 vs 200 doesn't matter.)
bframes
b-pyramid
cabac
ref
deblock
interlace
bitrate (and all ratecontrol options insofar as they affect bitrate)
partitions
direct
weightb
8x8dct
fast-pskip (indirectly. it just changes the relative numbers of macroblock types.)

[Terranigma]

Thanks akupenguin, i'll see what kind of profiles I can generate using that information :)

[CruNcher]

only the first used B-frame is heavy @ decoding additional won't take much more decoding time anymore (1 ref)

-b 0 --ref 1 = -00 fps SSIM Mean Y:0.9844503 Global:45.731 final ratefactor: 22.14 3023.12 kb/s 9.17 fps
-b 1 --ref 1 = -10 fps SSIM Mean Y:0.9857154 Global:46.107 final ratefactor: 21.57 3018.34 kb/s 9.08 fps
-b 2 --ref 1 = -13 fps SSIM Mean Y:0.9857187 Global:46.070 final ratefactor: 20.85 3026.20 kb/s 9.00 fps
-b 3 --ref 1 = -14 fps SSIM Mean Y:0.9853885 Global:45.932 final ratefactor: 20.52 3023.57 kb/s 8.89 fps

i have pretty good results with 720p @ 24 fps 7 Mbits @ 800 Mhz (x86) so far 60 fps with CoreAVC :)
optimizing this any further we end @ Apples settings im sure the Engineers @ Apple knew why they defined them like those for the trailer
The better your source is the less complex it needs to be @ decoding and they have pretty good Studio source to work with ;)

[bond]

altough its not really new anymore, but still, for checking how specific encoding options affect decoding speeds in different decoders (yeah, different decoders are affected differently) look at the decoder comparison sticky

[CruNcher]

Well, this discussion is about encoding speed. But I still think it's kind of silly to first use a H.264 encoder and then disable the most important features of H.264. Why not simply use a MPEG-4 ASP encoder (e.g. Xvid) instead? Also there are other options in x264 you can tweak in order to get higher encoding speed (e.g. Trellis or ME Method).

This is true LoRd_MuldeR but one thing you never will be able to fix is ASPs Qpel (or why do you think DivX Inc introduced their PP Sharpener) and AVCs Qpel is much better allready and don't forget it's allready active with ME 1.

[burfadel]

I would say have the normal features enabled (ie CABAC, in loop filter etc etc, you can have the settings quite good), then use a modern decoder. Note that as someone pointed out to me the deblocking occurs at both encoding and decoding, but if speed is a problem it can be disabled at the decoding stage, whilst still being on in the encoding stage without effecting the playback speed. This is the best way of doing it for quality as it seems the encoding deblocking is much more important (and more effective at deblocking then just at the decoder stage). The reason why you had blocking with CABAC and no filter is that the filter wasn't there to remove the blocks!

The decoders seem to have improved significantly in the past year, ffdshow for example. Also with ffdshow you can set it to skip h.264 deblocking if there is a problem with playback speed. I use quite high settings (in terms of decoding performance):

- Cabac (of course)
- 16 b-frames (gives the encoder freedom. If 16 is never used then there's no loss, but at least if there's a situation where 10 is beneficial it can use 10)!
- 6 reference frames (with mixed references).

Stuff encoded with these setting can play back quite well even on older computers using ffdshow which automatically skips the deblocking as I said.

If a trial encode doesn't work with those settings then try 3 b-frames and 3 reference frames. Don't change anything else. Now if you do disable CABAC to get the decoding performance like someone said you might as well use Xvid or something, because thats less CPU intensive to decode and using CAVLC etc negates any quality benefits that h.264 has over XVID file size wise in most cases.

The latest builds of ffdshow are available from:
http://ffdshow.info

There are still A LOT of old builds of ffdshow out there, and they can be slow for h.264!

[foxyshadis]

Note that as someone pointed out to me the deblocking occurs at both encoding and decoding, but if speed is a problem it can be disabled at the decoding stage, whilst still being on in the encoding stage without effecting the playback speed. This is the best way of doing it for quality as it seems the encoding deblocking is much more important (and more effective at deblocking then just at the decoder stage).

You're probably using a high enough bitrate that the artifacts aren't obvious. If you really push the codec and try to turn off inloop, it gets very messy, looks like a broken stream. When you have high overall quality, it just looks like an ASP stream without deblocking.

I don't know, maybe some people do like having h.264 as a ASP with little tweaks, like more partitions, b-ref, and slices. All it'd do is use a little more memory for the extra references. I certainly don't.

[CruNcher]

- 16 b-frames (gives the encoder freedom. If 16 is never used then there's no loss, but at least if there's a situation where 10 is beneficial it can use 10)!

Sorry it sounds like you think that the Quality of this decission has to be 100% right even if it is mathematicaly SSIM/PSNR, that doesn't mean it is HVS wise and to say it like this is like to say "Throw away your Human Mind/Brain" :P
Real (lossy) codec improvements can't be done by following numbers blindly you have todo alot of watching encoding watching encoding it is a awfull lot of work belive me the numbers can give you a hint if something might go wrong but they never ever gonna replace our Brain,Eyes and Visual Cortex.
XviD for example has a chance to get near H.264 and even be better with HVS improvements visualy (qpel detail preservation can be enhanced with a custom matrix so that the speed lose becomes a little more bearable compared to the unmirrored h.264 interpolation see my EDP build) and with more HVS features it could compete with H.264.

You're probably using a high enough bitrate that the artifacts aren't obvious. If you really push the codec and try to turn off inloop, it gets very messy, looks like a broken stream. When you have high overall quality, it just looks like an ASP stream without deblocking.

I don't know, maybe some people do like having h.264 as a ASP with little tweaks, like more partitions, b-ref, and slices. All it'd do is use a little more memory for the extra references. I certainly don't.

Please dont forget the better Qpel :D and i have to say yes i like that where it's usefull and enough for the given source.
I would prefer it instead of haveing a uber complex bitstream where i can't see in terms of HVS a difference to each other.
But block based aproaches will never be sucessfull at this we have to see the the whole thing Wavelett Codecs are the future :)

[LoRd_MuldeR]

I would say have the normal features enabled (ie CABAC, in loop filter etc etc, you can have the settings quite good), then use a modern decoder. Note that as someone pointed out to me the deblocking occurs at both encoding and decoding, but if speed is a problem it can be disabled at the decoding stage, whilst still being on in the encoding stage without effecting the playback speed. This is the best way of doing it for quality as it seems the encoding deblocking is much more important (and more effective at deblocking then just at the decoder stage). The reason why you had blocking with CABAC and no filter is that the filter wasn't there to remove the blocks!

In fact it works this way: In H.264 videos the P- and B-Frames refer to the deblocked Frames, so the encoder has to take the deblocking into account for obvious reasons. That is a major difference to MPEG-4 ASP, where deblocking is not part of the specifications and thus is just an optional post-processing feature. Every H.264 decoder must do the deblocking as defined in the specifications. Otherwise it's not a proper H.264 decoder! If the decoder does skip the deblocking (some decoders can be forced to do so) it will produce distorted output! That's because the reference Frames are "wrong" (not deblocked) and so the P- and B-Frames cannot be decoded properly. So skipping the deblocking on the decoder site is a very bad idea! Of course still better than not being able to see the video at all...

[burfadel]

Ah ok!

I use a constant quality of 25 and AAC sound (at q 0.36) with those settings mentioned previously. I also enable Trellis 2, which I know isn't ideal with constant quality but the file size does end up a little smaller and the quality in some areas seems to be a little better! Thats just what I've found anyway.

I choose 16 b-frames, 6 reference frames, motion detection of 6, RDO on b-frames etc etc because I find its the best balance for me between quality, end file size, and encoding speed. I found that most tv shows encoded online is in mp3 and xvid, in 350MB file sizes. With the settings i use tv shows are usually between 130mb and around 190mb and seem to be better quality than those download (at the same resolution, say 640x352 (widescreen). Each person has a difference reason for encoding, I just find they're the best settings for me. Also it means 700gb of video isn't 1400Gb+. The benefit of having a max of 16 b-frames and 6 ref's frames probably isn't much higher than say 4 b-frames and 3 reference frames, but it all ties in with my preferred file size/quality/encoding speed ratio! :)

For animation like Family Guy which I've use in a previous example I use 16 reference frames. It sounds excessive but for animation it does seem to benefit. Thats probably because of the limited changes and no colour gradients in the frames.

[akupenguin]

I also enable Trellis 2, which I know isn't ideal with constant qualityWhat isn't ideal? You should always use trellis unless the encoding speed hit is too much for you.

The reason lots of refs help in animated content is that animators are lazy and sometime re-use frames or pieces of frames. If they're re-used while the first copy is still within the reference buffer, then the second copy can be encoded for free.

[burfadel]

What isn't ideal? You should always use trellis unless the encoding speed hit is too much for you.

The reason lots of refs help in animated content is that animators are lazy and sometime re-use frames or pieces of frames. If they're re-used while the first copy is still within the reference buffer, then the second copy can be encoded for free.

Ah ok! that make sense :) My trellis comment was from old VFW guides... this is not where I read it but it says it:
http://www.avidemux.org/admWiki/index.php?title=H264

On a side issue, it also says there that the 2x2 partitions aren't available in x264. Is that a feature of h264, and if so is there any benefit with it and will it ever be included in x264?

Also, there are many x264 compiles out there, how come the speed varies so much between them when encoding? The difference between x264 from www.x264.nl and the build cef did on here on my core 2 duo was excessive, I mean, the build cef did was at least 30 percent faster... others had good experiences also. Enabling trellis to 2 doesn't effect the encode rate anywhere near that much!!

[akupenguin]

That list is of features that are in x264, but not exposed in avidemux. Nonetheless, it's wrong: H.264 doesn't have 2x2 partitions.

[Manao]

akupenguin : the "trellis bad at constant quantizer" comes from XviD, where some people had noticed that trelllis changed the rate distorsion tradeoff, sometimes lowering the PSNR. Since those people wanted a defined quality, they declared trellis to be bad. Since, the (completely groundless) reputation has sticked to trellis.

[akupenguin]

That's a bug, which has simply gone unnoticed because no one uses cavlc+8x8dct.
fixed.

[Terranigma]

Thanks akupenguin :)