Banding even in CRF19 - slower

[BeStBeSt]

Hello,

I'm quite new in x264 encoding. But I read many messages in this forum, and I realised that simple settings give the best results: CRF, preset medium or slow and tune film. When needed, I add yadif, and sometimes hqdn3d=2:1:2 (I use mencoder only). Usually I compress some SD sources (DVD mainly) at CRF 21 and the result is quite fine.

But on a quite clean source, I have some problems especially with banding. Below you can see screenshots (in PNG) from the source and the compressed version (the source and compressed clip here (http://www.2shared.com/file/S5New-xL/data.html)). Here is the command-line
mencoder [...] -vf yadif,crop=720:368:0:56,harddup -ovc x264 -x264encopts crf=19:preset=veryslow:tune=film -nosound

I found the result to contain a lot of banding (color gradient on the end of the fingers), at such a CRF level (19) and such preset (veryslow).
Source:
http://img35.imageshack.us/img35/672/sourceframe155.png
CRF 19 - veryslow:
http://img803.imageshack.us/img803/145/crf19frame155.png


How can I remove the banding of this 15-second clip:

Without infinite size for the whole 2h-video (no crf 16), and without infinite CPU time (no preset=placebo)
No complicated setting (not 10 custom settings that only you can understand)
No filter during play (no gradfun during playback, although it gives good results)

[BeStBeSt]

Another screenshots to the clip I've uploaded (link in first post):

Look at the hand (thank you Captain Obvious :sly:)
Source:
http://img707.imageshack.us/img707/7913/sourceh.png

CRF 19 - preset=slower:
http://img688.imageshack.us/img688/2828/testcrf19vslow.png

[Atak_Snajpera]

use x264 10bit version or enable deband filter in ffdshow. your choice.

[BeStBeSt]

What a quick answer !! Thanks !

How to activate 10-bit x264 ? Will the result of the compression be still be compatible with usual players ?
I do not want to impose any filter at playback (one of my constraints), so you mean deband (=gradfun) during encoding ?

[Atak_Snajpera]

10 bit video will work on pc only!

[nm]

How to activate 10-bit x264 ?

At configuration phase when building it.

Will the result of the compression be still be compatible with usual players ?

It's compatible with many libavcodec-based players and even Flash Player, but not with every sofware decoder out there. Current consumer-level hardware decoders don't support it at all.

I do not want to impose any filter at playback (one of my constraints), so you mean deband (=gradfun) during encoding ?

He meant during decoding. It won't help much during encoding in this case.

With this kind of source you could try using higher qcomp and/or higher AQ strength too. Try adding qcomp=0.8:aq-strength=1.5 to your x264encopts for a pronounced effect.

[BeStBeSt]

With this kind of source you could try using higher qcomp and/or higher AQ strength too. Try adding qcomp=0.8:aq-strength=1.5 to your x264encopts for a pronounced effect.
No specific filter to turn on during decoding, and no mencoder rebuilding: well that's the kind of suggestion I was looking for ;)
I tried both qcomp and aq-strength... and the result is really nicer, at really acceptable oversize ! :thanks:

After googling the meaning of these 2 settings, my conclusion was this. Am I right ?

qcomp = 0.6 on default. Put it higher (~0.8) if you want to give more bits to the motion scenes, but filesize increases a bit (depends on the source motion)
aq-strength = 1.0 on default. Put it higher (~1.1) if you want less banding, but you'll have ringing (what it is ?)

[nm]

qcomp = 0.6 on default. Put it higher (~0.8) if you want to give more bits to the motion scenes, but filesize increases a bit (depends on the source motion)

Yep, although in your clip motion isn't that fast, but I guess because of blurring CRF tends to see it as such and use higher QPs.

aq-strength = 1.0 on default. Put it higher (~1.1) if you want less banding, but you'll have ringing (what it is ?)
Ringing means artifacting at sharp edges.

[hello_hello]

There was a similar thread regarding banding when encoding not so long ago. I'm too lazy to look for it at the moment but I was one of the posters experimenting with someone else's "problem" sample. The source contained banding but it was less obvious than after encoding it, even when using a fairly low CRF value.
What I did discover was if I converted the same sample to uncompressed RGB, it still made the banding worse. I don't understand why, although I'd be curious to know if converting your video to an uncompressed/lossless format produces the same result as the source, or whether it's the encoding which is the problem.

[nm]

What I did discover was if I converted the same sample to uncompressed RGB, it still made the banding worse.

Maybe you had different YUV to RGB conversion during encoding and during playback of the original video.

I'd be curious to know if converting your video to an uncompressed/lossless format produces the same result as the source, or whether it's the encoding which is the problem.

Well, if you keep the original colorspace, lossless H.264 will be identical to it. There may be playback issues with levels and color matrices, but those aren't the main problem here. x264 just lowers quality too much at CRF 19 on this blurry source.

[smegolas]

Is your monitor calibrated? Incorrect darkness and gamma settings on your monitor can exacerbate artifacts such as banding.

[kolak]

x264 is not very good with preserving gradients, but it looks like there is more going on- something with decoding/color spaces.

[nm]

x264 is not very good with preserving gradients, but it looks like there is more going on- something with decoding/color spaces.

I looked at the video samples and didn't have problems with decoding or color matrices and levels. The pictures posted here don't have colorspace problems either.

It's just that the video got encoded at too high quantizers (low bitrate) because of the content. When moving towards CQP encoding by raising qcomp, quality gets much better.

[Atak_Snajpera]

x264 is not very good with preserving gradients,
If you are using 8bit version only. In 10 bits it is different story :)

http://forum.videohelp.com/threads/344136-How-to-encode-anime-to-10bit-x264?p=2146361&viewfull=1#post2146361

[BeStBeSt]

With this kind of source you could try using higher qcomp and/or higher AQ strength too. Try adding qcomp=0.8:aq-strength=1.5 to your x264encopts for a pronounced effect.
Actually, you were right. qcomp = 0.7 works better (at identical bitrate on a 5-minute clip, at about crf=20).


But I also found that aq-mode=2 (or qcomp=0.7:aq-mode=2) works quite well too at identical bitrate !! Here are some screenshots (encoded clips would be better), but the video do look better to my eyes.

crf=22:qcomp=0.7:aq-mode=2: preset=slow
http://img685.imageshack.us/img685/2306/00000183.png

http://img12.imageshack.us/img12/1494/00000336t.png


Although qcomp=0.7 may be specific to this sample, aq-mode=2 (Auto-Variance AQ) looks very convincing to me. I will now google it to understand why it is still experimental, and what are the bad points of this mode.

[Dark Shikari]

It probably just gave the frame a lot more bits for whatever reason.

[Dion]

If you are using 8bit version only. In 10 bits it is different story :)

http://forum.videohelp.com/threads/344136-How-to-encode-anime-to-10bit-x264?p=2146361&viewfull=1#post2146361

10Bit just does it better at lower bitrates. Which makes sense since it can store more. 8bit can handle banding fine. Just requires more bitrate. Also higher AQ and PSY-RD values help too.

[BeStBeSt]

8bit can handle banding fine. Just requires more bitrate. Also higher AQ and PSY-RD values help too.
Thanks for this advice: for instance, what values of aq-strength and psy-rd do you recommend me ? At least, what exact setting should I try at first ?

[Dion]

Thanks for this advice: for instance, what values of aq-strength and psy-rd do you recommend me ? At least, what exact setting should I try at first ?

AQ - 1.1 - 1.2. I wouldnt go over 1.2
PSY-RD - 1.10 - 1.30.

Also different decoders can have different results too. I use FFMS myself.

[nm]

BeStBeSt's sample clip needs an average bitrate of about 1000 kbps at --preset veryslow for good results (no clearly visible banding or compression artifacts when freeze-frame comparing against the source).

When using CRF encoding and --preset veryslow --tune film, I got ~1000 kbps average bitrate and about the same visible quality with these three combinations:

(a) --crf 14 --aq-strength 1.2
(b) --crf 16 --qcomp 0.8 --aq-strength 1.2
(c) --crf 18 --qcomp 0.8 --aq-strength 1.5

Increasing psy-rd to 1.3 didn't have much effect on this clip. Switching to aq-mode 2 required lowering CRF by 2 to get comparable quality.

If the task at hand is to balance the quality of this clip to the overall video while trying to minimize file size, playing with AQ strength and qcomp may be necessary. Or you could use zones to encode this particular scene with higher bitrate. It's hard to give specific advice without having access to the complete source video.

[BeStBeSt]

When using CRF encoding and --preset veryslow --tune film, I got ~1000 kbps average bitrate and about the same visible quality with these three combinations:

(a) --crf 14 --aq-strength 1.2
(b) --crf 16 --qcomp 0.8 --aq-strength 1.2
(c) --crf 18 --qcomp 0.8 --aq-strength 1.5

Increasing psy-rd to 1.3 didn't have much effect on this clip. Switching to aq-mode 2 required lowering CRF by 2 to get comparable quality.
Thanks a lot for these results.
I did a compression test with each one of these settings, but the ones that give the best results at given filesize (particularly for this clip, but also good on the whole video) are --qcomp 0.8 --aq-mode 2 and --qcomp 0.8 --aq-strength 1.5.

The reason must be that these settings allocate more bits to this particular clip and less to other "easier" parts (as said by Dark Shikari), rather than real better efficiency.

[nm]

The reason must be that these settings allocate more bits to this particular clip and less to other "easier" parts (as said by Dark Shikari), rather than real better efficiency.

Yep, that's the point of this tweaking.

[nudelbubu]

this seems to be the same problem i currently have. i recorded something from tv, that was rather grainy, so i used lremovedust on it. the encoding didn't look too well (which came from the compression, not lrd).

i have been using --preset slow --tune film --crf 20 --level 4 on this and don't really want to lower the crf. as i don't really have much experience, is this the same problem and should i try playing around with the same parameters? or should i create a new thread for this?

nvm, got a decent result with --fgo 10

[kolak]

I looked at the video samples and didn't have problems with decoding or color matrices and levels. The pictures posted here don't have colorspace problems either.

It's just that the video got encoded at too high quantizers (low bitrate) because of the content. When moving towards CQP encoding by raising qcomp, quality gets much better.

Than it's x264- I had same issues during my BD testes-even 35Mbit didn't help. This is area where x264 is not as good as pro encoders, but we went through this already. I'm not the only one who reported it. Low bitrate is yet another story- something has to be simplified- heh.

[nm]

Than it's x264- I had same issues during my testes-even 35Mbit didn't help.

Well, at least in this case it was just a matter of tweaking rate control and AQ parameters. Hardly an inherent problem of the encoder.

I'd suggest finding and posting a sample that exhibits the issue if there's a problem that you'd like to get fixed.

[kolak]

I went through this already- maybe something has changed since than- changing settings was not really helping.

We went through this- check mp3dom posts:

http://forum.doom9.org/showthread.php?t=161116&highlight=x264+gradient

and also had answer from DS:

"We will never optimize x264, ever, for the case of "not actually watching the video". It is beyond pointless."

And another one (from Sagittaire), which makes more sense:

"Final word: x264 never encode with your suggestions simply because your suggestions break HVS common rules. Personnaly I don't want grain in sources with low luma/contrast (filtering), I don't want that encoder have good grain retentions in these area (bit loss for other most important area)."

Both of them don't change fact that some pro encoders handle these kind of content (also fades) better than x264. My explanation- they were not designed to be good at 1Mbit, but at 30Mbits, so use different optimisations. It may be wrong, but I don't see reason to come back to it.
Criterion sponsored x264 BD compliancy testes, but never used it- even if it could work well on some of their titles.

[Biggiesized]

Perhaps the x264 team needs a really grainy 24p source for testing (assuming they would even try to optimize for this)?

I have a 10-bit 4:2:2 JPEG2000 encoded 30 sec clip from Spider-Man 3.

[nm]

Perhaps the x264 team needs a really grainy 24p source for testing (assuming they would even try to optimize for this)?

High amount of grain tends to hide banding issues, but that Spider-Man 3 clip might be interesting anyway. There are very few 10-bit 4:2:2 test clips publicly available.

[mp3dom]

Anyway, short clips tends to be useless. Rate control, with those clips, is already optimal. You need to use long clips (starting with at least 10-15 minutes) with a mix of different kind of footage (grain, color gradients with both complex and gentle parts).

[Biggiesized]

High amount of grain tends to hide banding issues, but that Spider-Man 3 clip might be interesting anyway. There are very few 10-bit 4:2:2 test clips publicly available.
My post was in reference to kolak's quote about mp3dom's grain retention thread.

[kolak]

High amount of grain tends to hide banding issues, but that Spider-Man 3 clip might be interesting anyway. There are very few 10-bit 4:2:2 test clips publicly available.

Yes- you don't really have banding issues with grainy sources.

Try footage from RED, EPIC, Alexa- so easy to end up with banding.

[BeStBeSt]

Yes- you don't really have banding issues with grainy sources.

Try footage from RED, EPIC, Alexa- so easy to end up with banding.

So if you want to avoid banding, apply 'hqdn3d=2:1:2' is a bad idea ?

[Atak_Snajpera]

So if you want to avoid banding, apply 'hqdn3d=2:1:2' is a bad idea ?

your settings are very weak. nevertheless i aways apply deband filter after denoising (gradfun2db or fk3db)

[kolak]

So if you want to avoid banding, apply 'hqdn3d=2:1:2' is a bad idea ?

To avoid banding you use good dithering (external or in encoder) to go from 10bit master to 8bit file for encode.

[nibus]

AQ - 1.1 - 1.2. I wouldnt go over 1.2
PSY-RD - 1.10 - 1.30.



Yup, Psy really is the key to preserving gradients. On especially problematic sources also raise trellis, something like 1.20:0.10.

Also keep in mind that CRF19 is too low of a bitrate for a transparent encode. If you want real transparency think 16-18. Whether the space is worth it is up to you.

[Asmodian]

Changing any of these settings (AQ, PSY, etc.) can have a large effect on the final size of an encode. Changing them but being unwilling to change the crf value is unreasonable.

[BeStBeSt]

Yup, Psy really is the key to preserving gradients. On especially problematic sources also raise trellis, something like 1.20:0.10.

Also keep in mind that CRF19 is too low of a bitrate for a transparent encode. If you want real transparency think 16-18. Whether the space is worth it is up to you.
I don't want a transparent encode at all, I just want a "look-able" result at the smaller size at possible.
Just remind also that --tune film implies --psy-rd 1.00:0.15, so your trellis value would be lower actually. I'll do some test on a large clip (>10 min) at similar bitrates to see if your suggestion helps.

PS: Solutions involving dithering or 10-bit encode is impossible for me because I use mencoder only (cause hard to make Avisynth work on Ubuntu).

[BeStBeSt]

So the possible solutions listed by you and the ones I found on other discussions are:

Tweaking qcomp (0.7 or 0.8)
Tweaking aq-strength (1.1 or 1.2)
Tweaking psy-rd (1.10 or 1.30)
Tweaking psy-trellis (>0.15)
Check aq-mode=2
Using x264 built-in nr instead of hqdn3d, and tweaking nr (60)

10-bit x264, or adding dithering would be the best solutions, but not in my case.

As there is a lot of settings to test, I would like to know which one(s) would be the more effective. But I think your answers will be "just do a visual check at identical bitrate with your movie".

[nm]

PS: Solutions involving dithering or 10-bit encode is impossible for me because I use mencoder only (cause hard to make Avisynth work on Ubuntu).

You could pipe the video to x264 from MEncoder or MPlayer. AviSynth is also not that hard to install and run with Wine.

[BeStBeSt]

You could pipe the video to x264 from MEncoder or MPlayer. AviSynth is also not that hard to install and run with Wine.

I don't like Wine (no specific reason), but I currently use mencoder (or mplayer): there is a way to add dithering through mencoder ? I didn't find it but I prefer experienced people like you to confirm what random guys say on Google :thanks:

Once I used "gradfun", but I've seen somewhere this is a bad idea when used at the compression step.

[nm]

Once I used "gradfun", but I've seen somewhere this is a bad idea when used at the compression step.

MPlayer's manual page includes that statement. Also read pengvado's response here: http://doom10.org/index.php?topic=471.msg3133#msg3133

Your source clip discussed in this thread is already dithered, so MPlayer's gradfun won't help. You could try and see if 10-bit encoding keeps the dither better than 8-bit.
For better efficiency, you'll need to deband to 16-bit depth with gradfun3 or flash3kyuu_deband (both are AviSynth filters).