Sharp background but low-contrast actors, need help with X264 parameters

[gamebox]

Which strategy do you recommend for encoding a (professional) HD video shot with complete disrespect for general rules of filming - it has painfully sharp and colorful background details (furniture, carpets, bed and chair covers, pop-art graphics on the wall, etc.) while the people themselves (and their skin) are visually very "flat" and "unpronounced"? I mostly get blocking and blurring exactly where I don't need it - on people of course.

Setting aq strength from 1.0 either to 1.5 or 0.5 didn't create obvious-enough difference in visual quality (1.5 was worse than 1.0, 0.5 somewhat better than 1.0). AQ mode selected was 1.

Trellis mode 2 with strength of 0.15 helped also only a bit over Trellis mode 1 with default strength of 0.00 (does that mean it was off?). Also, with Trellis, I'm not exactly sure what it does and how it treats the picture (adaptive quantization, I assume, gives slight bitrate advantage to sharper details on account of less pronounced ones). I'd be grateful if you point me to some tutorial, forum subject of results-comparison thread on Trellis.

The rest of the parameters I use are typical very-slow setup (bitrate 3000, me umh, subme 10, merange 24, 8 reference frames, 3 B-frames, B-pyramid and B-frame RDO optimizations, deblock -1 -1 (to try keeping a bit more detail), all macroblock partitions selected, etc.).

Thanks in advance.

[poisondeathray]

Did you consider "fixing" the input first ? (address the underlying problem)

Adjusting encoding settings probably isn't a good approach to your issue

[gamebox]

Interesting idea poisondeathray, but what do you suggest then? How to selectively unsharp/reduce contrast on "background" parts of the frame without affecting the already critical "important" area with low details and low contrast? How can I "select" or "define" the area of the picture whose position, size and layout are constantly changing?

I thought I just needed to reduce the codec's general bias to sharp details, or actually reverse it to get the proper treatment here.

[poisondeathray]

If you can't reshoot the video, it's usually done with masks and post production work . You might be able to do some of it with avisynth

If you can upload a sample I suggest what some of your options are

[poisondeathray]

Originally Posted by gamebox

I thought I just needed to reduce the codec's general bias to sharp details, or actually reverse it to get the proper treatment here.

yeah...well that' s not going to help much. You can test all you want with various encoding settings or different encoders and I guarentee that you'll get pretty much nowhere.

If your description was accurate - No encoder will give you enough control over what is a "detail" and what is a background object.

The primary job of an encoder is to reproduce what is given to it. If you give it something messed up, you'll likely get something messed up. Garbage in = Garbage out.

[gamebox]

I can't reshoot it poisondeathray. It's not quite...presentable either. It's mostly a part of my low bitrate-high resolution transcoding "exercises", where I try to reprocess 5-8 Mbps originals (intended for online (not streaming) distribution) into 3 Mbps "lite" versions of comparable quality, through use of all the advanced coding tools x264 can offer (though not breaking the standalone-decoder compatibility). Most videos process fine with an end-quantizer usually around 24-26 (which still looks far better than what you get through on-demand streaming sites, probably because of two-pass encoding, high-complexity parameters and RDO optimizations), however this video is a special case that prompts me to reach out for some more knowledge.

The original looks pretty fair, though it was encoded with just-about-enough bitrate and without using any B-frames or relying on RDO optimizations, and those make it harder to keep things under control of course.

[poisondeathray]

hahahaha ... ok it's THAT type of video ....

What the hell was the camera operator focusing on the background for ? Maybe he couldn't concentrate on the actual camera work

But seriously, if your description is accurate, the only good fix is to do some post production work . This doesn't fall under the realm of "encoding"

I'm making some assumptions but production quality usually is pretty low on these (unless you have high budget), so I doubt many people will notice how bad it is - the people that watch these won't be looking at the background

[gamebox]

Yeah, THAT type.

Mmm, it is professional-quality shot, but the director favourizes extremely kitchy details, colors and textures in a scenography, making it one hell of a mess to see. Even if the actors are well-focused themselves, the background is still high-contrast and (too) well-lit even if not sharp, and that makes it hard to get a "desired" treatment from an encoder.

[rwill]

Hello gamebox,

First, I have very limited experience with encoding your type of content but my guess is that its somewhat close to the weather_cif.yuv sequence. I know a bit or two how the visual quality tools of x264 work though.

AQ Mode = 1 should have a positive impact on your type of content because it lowers the quality of macroblocks with high variance and raises the quality of macroblocks with low variance. So high contrast patterns will get less bits and low contrast patterns will get more bits.

x264 psy optimizations only work well when "enough" bits are available. As it sounds like you do not have enough bits I suggest to turn psy completely off ( --psy-rd 0:0 or --no-psy ).

You should still set Trellis to 2 as this is a coding efficiency optimization which will save some bits and improve quality at the cost of encoding speed. It also enables the encoder to "see" what a macroblock will look like earlier instead of delaying this to the final encode of the macroblock.

You can try to set the deblocking parameters back to 0,0 or even try 1,1 so the background gets blurred more. While this goes against the artistic intend of your scene description it might help to further improve overall quality.

Then there is MbTree. MbTree shifts bits from areas which can not be predicted well to areas which can be predicted well. Prediction here refers to temporal prediction. Further, macroblocks with higher complexity will get improved more than macroblocks with lower complexity. So a difficult pattern which does not change ( your background ? ) gets improvement and an almost flat area which has medium to high motion ( your actors ? ) gets almost improved none. You can try if disabling MbTree helps visual quality in your case, --no-mbtree.

If you "encode once, decode many" you should always throw as much processing power at the encode as you can. Some hours or even days of CPU time should not matter if the total time of decodes goes into months or years. So use these veryslow/placebo presets.

You should be careful with enabling all partitions ( p4x4 ) because a Level of 3.0 or higher imposes certain restrictions on the number of motion vectors in two consecutive macroblocks, which I think x264 will violate.

Then again there are sequences where only more bits help...

[gamebox]

Thanks a lot, qwill

Originally Posted by rwill

AQ Mode = 1 should have a positive impact on your type of content because it lowers the quality of macroblocks with high variance and raises the quality of macroblocks with low variance. So high contrast patterns will get less bits and low contrast patterns will get more bits.

Yes, it did help, and I use AQ 1 most of the time doing encodes. However, in the case of this video value of 0.5 for AQ strength gave better results than the default 1.0.

Originally Posted by rwill

You should still set Trellis to 2 as this is a coding efficiency optimization which will save some bits and improve quality at the cost of encoding speed. It also enables the encoder to "see" what a macroblock will look like earlier instead of delaying this to the final encode of the macroblock.

Trellis 2 and value of 0.15 brought significant change here. The end quantizer was improved to 26-27 instead of 27-28, at about 15-20% speed penalty. It seems to me though some textures (eg. faces) got a bit "simpler" and "smoother" and lost some details, but blocking is reduced on "medium" complexity textures and increased only in "lowest" complexity ones, so the end result is less blocking overall (though this might actually be quantizer related).

Originally Posted by rwill

You can try to set the deblocking parameters back to 0,0 or even try 1,1 so the background gets blurred more. While this goes against the artistic intend of your scene description it might help to further improve overall quality.

I lowered deblocking to -1,-1 before as default 0,0 made H264 videos noticeably "softer" to me than what I got used to see with XVID. I had that impression of softness even with quantizers like 24-25, especially in areas of high motion where I still prefer (slight) blocking over complete smoothness. I don't know, however, whether lower deblocking decreases compression efficacy (increases quantizers) and by how much.

Originally Posted by rwill

You should be careful with enabling all partitions ( p4x4 ) because a Level of 3.0 or higher imposes certain restrictions on the number of motion vectors in two consecutive macroblocks, which I think x264 will violate.

I encode targeting level 4.1 (more like 4.0 since I use low bitrate), as the videos are mostly 720p. I consider playing the video on my Radeon series 4 GPU a "standalone-compatibility test" of sort, as UVD decoder built into it should be pretty much "standard" in its capabilities and features. It refused decoding some clips which were clearly breaking the spec (too many reference frames, etc).

Originally Posted by rwill

You can try if disabling MbTree helps visual quality in your case, --no-mbtree.

I never thought about changing mbtree parameter, as I thought it was always beneficial as some sort of RDO optimization for B-frames. I should give this a try for this particular movie (I will continue using it on the ones with normal visual "lineup" - sharp actors, dark and blurred background).

Originally Posted by rwill

If you "encode once, decode many" you should always throw as much processing power at the encode as you can. Some hours or even days of CPU time should not matter if the total time of decodes goes into months or years. So use these veryslow/placebo presets.

Yes, that's what I do. I process movies for storage primarily, so I aim for best possible coding and PSY efficiency. Typical 720p movies of about 90 minutes get processed in about 30-40 hours on my single core Athlon64.

And regarding Trellis - what exactly does it do? It seems to me it is some sort of quantization pre-step which "simplifies" macroblocks before coding, making them compress somewhat smaller. That is somewhat similar to quantization matrices used in XVID before, right? I disliked Trellis in XVID for analog captures back in the days as I was getting clearly less details I cared for, but in X264, for digitally sourced material, it seems to bother me somewhat less (though I still alternatively see frames which look better with Trellis and the ones which don't).

[rwill]

Trellis is rate distortion optimizing quantization ( RDOQ ). Like - for a macroblock, after spatial ( Intra ) or temporal ( Inter ) prediction, you end up with residual which corrects the error of the prediction. For a perfectly encoded residual you would end up with a perfect reconstruction of the original when decoding. As this would cost a lot of bits there is quantization etc.. In H.264, after the DCT like transform of the residual, comes quantization and after that the quantized residual is written to the bitstream. When writing the residual, coefficients that are written first have certain effects on how the following coefficients are written. With CABAC this is somewhat simple while for CAVLC it can become complicated. When the quantization function of the encoder is simply applying quantization with a certain rounding deadzone to the residual it does not consider neither bits ( rate ) nor the distortion of the outcome. The distortion is limited by the QP but the bits are completely unknown. So to improve the coding efficiency most modern encoders have some sort of rate/distortion aware quantization function. These tend to look at each single coefficient and consider optimal distortion in relation to the bits the different distortion options would cost.

Now the distortion in x264's Trellis is either the squared difference of the original coefficient and the dequantized coefficient ( think --tune psnr) or that plus some bias which takes the value of the same coefficient of the transformed original source and the reconstructed value of the coefficient into account. Like if there is some frequency present in the block of the original it biases to it also being present in the reconstructed block. I guess the 0.15 you mention are the strength of this frequency bias ( psy ).

If you set trellis to 1 the RDOQ is only done for the final encode. Now before the final encode comes the mode decision which compares modes ( think Inter/Intra decision, partitioning etc ) and decides which mode to use for the final encode. If you set trellis to 2, RDOQ is also used for the modes tested by the mode decision instead of using the faster but less efficient "dumb" quantization. Using RDOQ in the mode decision instead of using it only for the final encode gives the mode decision a better idea which mode is the best because the result of a tested mode in the mode decision is closer to the result the mode will produce when its used for the final encode ( "almost identical" instead of "slightly different" ).

RDOQ has only a slight impact when tuning for PSNR or SSIM but when its properly tuned to the psy distortion metric the mode decision uses and the distortion metric of the mode decision picks modes that are favored by humans, it can give quite the subjective quality boost.


Unless deblocking parameters are insane ( < -3 or > 3 ) they tend to have little effect on coding efficiency. As deblocking is applied after encoding a picture the only thing that changes are reference frames. It depends on the source material really.


Level 4.1 limits the number of motion vectors in two macroblocks coming after another to 16. If small partitions are enabled with "p4x4" a macroblock can have, in extreme cases, 16 motion vectors, leaving no motion vector for the next macroblock and producing an invalid bitstream if the macroblock before it was Inter coded. I have not looked at the source for some times but I remember it to be a problem of x264 a couple of years ago. It might have been fixed.

A single core Athlon... you know they sell mainboards for ~70$ now that have a quad core soldered onto them ..

[foxyshadis]

That quad core running full tilt will use a lot less power, too, easily paying for itself in a few months between that and encoding in a tenth the time.

[gamebox]

@rwill: When I use default settings in x264, Trellis is set to 1 (Final macroblock only) and psy-trellis to 0,00. I've read somewhere that it means Trellis is actually off (because of zero value for bias), but I think that statement was actually wrong - I think it is on, with somewhat reduced strength.
Mbtree option's influence needs to be tested on these videos with sharp background/low contrast actors, but I will probably process the current one without changing it (as encoding takes a lot of time) and try what turning mbtree off does on next one (I'll test it on a few slices first). Movies with "usual" visual layout (unsharp background) don't need that much tweaking to look ok (Trellis 1 at bias 0.0 and AQ on default level is usually good enough). Therefore I can process them somewhat faster and without added visual distortion stronger Trellis brings. A tutorial for MPEG-4 encoding in VirtualDub I read before said Trellis leads to longer encoding times, considerable bitrate reductions and a "slight quality loss along the way". My experiences with Trellis in x264 are similar (some textures do get simplified, especially with Trellis=2 and psy=0.15).

Originally Posted by rwill

A single core Athlon... you know they sell mainboards for ~70$ now that have a quad core soldered onto them ..

Originally Posted by foxyshadis

That quad core running full tilt will use a lot less power, too, easily paying for itself in a few months between that and encoding in a tenth the time.

I agree to both of you that quad core processors aren't expensive, would process the video faster by far, and would be more energy-efficient. I plan buying one with about the same TDP as my Athlon64 (approximately 65W), together with some lower-consumption memory (I currently use 2.5V DDR1), but I would use that machine for HEVC encoding instead of H264 and I want it to support HEVC in hardware too, so an upgrade is postponed until such processors arrive (AMD announced them for 2015 in GPUs and APUs). Regarding cost-efficiency of my setup I'll say that (since I don't process many videos) I can afford to postpone encoding to off-peak electricity only, and that reduces cost to eurocent-per-day range.

[gamebox]

My encoding just got an appreciable quality boost now that I used some time to evaluate different combinations of options rwill mentioned. I finally chose to use Trellis 2 for all encodes I do. Increased blocking in certain object's edges I saw in my previous encodes was traced down to PSY-Trellis' bias of 0.15. Without bias (set to 0.00) Trellis 2 proved to be a higher quality option than Trellis 1 in about 80-85 percent of the frames I analyzed. As a result of finer Trellis, quantizers got better by about 2, so a movie previously encoded with quants of 23/24/25 now has enough bitrate to use 22-23 (occasionally even 21 or 20). Curiously enough, Trellis 2, despite bitrate reductions, also preserved substantially more background details. Savings in bitrate and a quantizer boost led me to drop mbtree too to adapt encoding to this "special" sort of material where focus is exactly on the area of changing textures, although on evaluation dropped mbtree didn't seem to create any detectable visual difference. The encoding speed, however, suffered a permanent 15-20 percent hit, though it seems bearable for now.