x264: CRF18 & Annoying Rainbow-Effect

[moviefan]

Hi guys,

I try to encode a more or less clean video with x264 on CRF18. It sometimes shows pretty flat areas or very smooth color gradient areas and x264 produces a rainbow effect there. How can I prevent this disturbing effect? More detailed areas looked very good.

Regards

[Selur]

screenshot + settings you used with what version might help,...

[moviefan]

I used these settings with x264_667b_hadamard:

--crf 18 --ref 4 --no-fast-pskip --bframes 6 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -4,-3 --subme 6 --analyse all --8x8dct --vbv-maxrate 25000 --threads auto --thread-input --cqmfile "D:\megui\matrices\high_detail_matrix.cfg" --progress --no-dct-decimate --output "" "" --hadamard

[fields_g]

Did you resize your original video?

If so, it could be a moire pattern.

http://en.wikipedia.org/wiki/Moir%C3%A9_pattern

Like stated before screenshot + detailed info would help.

[Sharktooth]

first, update x264 to r680.
remove the silly high_detail_matrix.cfg, then use higher values for --filter. something like -3,-1 or -2,-1... or even better 0,0...
--hadamard does not exists. you have an experimental, outdated, patched (by who?!?) and probably uber-buggy build.
So your thread title should be changed since you're not using an original x264 encoder and you're even using questionable settings.

[moviefan]

Okay, I have updated to r680, set deblocking to 0:0 and changed from that custom matrix to the standard one. What settings of mine do you consider questionable?
I use meGUI and have only resized the movie from the anamorphic PAL to 720x404 (~16:9).

[Sharktooth]

--filter -4,-3, --hadamard (IIRC it was buggy in that version) and the CQM.

[nm]

I try to encode a more or less clean video with x264 on CRF18. It sometimes shows pretty flat areas or very smooth color gradient areas and x264 produces a rainbow effect there. How can I prevent this disturbing effect? More detailed areas looked very good.
Do you mean banding or blocking? If so, that is a frequently asked question and has been addressed in many threads already.

If you actually see something that resembles a rainbow with a color spectrum, and the source is progressive, that would be a new one :)

As others said, screenshots of the problem would really help here.

[moviefan]

Sorry, I mean banding of course.

[Sagekilla]

Sorry, I mean banding of course.

As others have said, screenshots please! Depending on what your problem is exactly, there's different things you can do to fix your problem, but we need to see what your encodes look like first. :)

[moviefan]

Here are the Screenshots:

Screenshot 1 (only pre-processed)
http://img219.imageshack.us/img219/2889/s1origex7.th.png (http://img219.imageshack.us/my.php?image=s1origex7.png)

Screenshot 1 (x264)
http://img219.imageshack.us/img219/1899/s1x264yy9.th.png (http://img219.imageshack.us/my.php?image=s1x264yy9.png)

Screenshot 2 (only pre-processed)
http://img229.imageshack.us/img229/9172/s2origvq2.th.png (http://img229.imageshack.us/my.php?image=s2origvq2.png)

Screenshot 2 (x264)
http://img219.imageshack.us/img219/1719/s2x264se2.th.png (http://img219.imageshack.us/my.php?image=s2x264se2.png)

The pre-processing scripts is:


ColorMatrix(mode="Rec.709->Rec.601")
crop(0,8,-4,0)
BicubicResize(720,404)
hq_filter()

hq_filter() comes from the RemoveNoiseMC script which can be found in the AviSynth section in this forum. How can I remove the banding and the terrible artifacts without decreasing the CRF?

[Sagekilla]

Try increasing deblock filter a bit. -4:-3 isn't a very good idea. Try increasing it to something more sane like -3:0 or -2:-1 and see what you get.


Edit: There may also be the possibility deblocking was disabled in the postprocessing. What media player do you use?

[moviefan]

-3:0 looks better, but then the banding gets more intense! Concerning the postprocessing deblocking - I would like to get along without that...
Can I increase the bitrate assigned to flat areas like in MPEG2? More bitrate for flat areas, less for detailed parts?

[Atak_Snajpera]

Try this
--aq-strength 0.5 --aq-sensitivity 5
with default deblocking

[Sagekilla]

-3:0 looks better, but then the banding gets more intense! Concerning the postprocessing deblocking - I would like to get along without that...
Can I increase the bitrate assigned to flat areas like in MPEG2? More bitrate for flat areas, less for detailed parts?

That's exactly where your problem would usually lie then. Turning off Postprocessing deblocking is what can kill an encode very easily and turn quality to mush. I've tried it on various encodes and I did not like the quality at all, even at a high quality level of crf 18. You can try the adaptive quant like moviefan suggested, but that usually comes with a slightly higher bitrate at the same time.


Also, may I ask why you are not choosing to use deblock on postprocess?

[Episodio1]

And what do you get if you don't use "filter/deblockin" while encoding ?

[moviefan]

Okay, I'll try the AQ setting and disable deblocking.

@Sagekilla: I think enabling post-processing is like cheating... you do some tricky thing to make the actually bad quality movie appear better on cost of additional performance needed. Okay, maybe I save some bitrate, but I somehow dislike that deblocking.

What settings do you suggest for "Number of B-Frames" and "Reference Frames"? Quality over speed...

[nm]

@Sagekilla: I think enabling post-processing is like cheating... you do some tricky thing to make the actually bad quality movie appear better on cost of additional performance needed. Okay, maybe I save some bitrate, but I somehow dislike that deblocking.
Do you mean inloop deblocking? It should never be disabled on playback; frames will be corrupted when you do that. If you don't like deblocking, turn it off when encoding (with option --no-deblock), but leave deblocking on in the decoder settings (it won't deblock unless the video stream has been marked to be deblocked).

[Atak_Snajpera]

@Sagekilla: I think enabling post-processing is like cheating...

As the matter of fact whole video compression is like cheating :)

[Sagekilla]

Okay, I'll try the AQ setting and disable deblocking.

@Sagekilla: I think enabling post-processing is like cheating... you do some tricky thing to make the actually bad quality movie appear better on cost of additional performance needed. Okay, maybe I save some bitrate, but I somehow dislike that deblocking.

What settings do you suggest for "Number of B-Frames" and "Reference Frames"? Quality over speed...


For frames, I would recommend 16 B-frames to let x264 choose what's best for B-frame placement. For # of ref's, I generally use 4 but in most cases you will see very little improvement over 3. Unless you're willing to take the speed hit for a slight quality increase (I mean really slight), stick with 3 references.


And as the previous person said, the inloop deblocking should NEVER be disabled. Yes it is post processing but it's an integral part of the H.264 standard and without it you suffer a massive quality hit. For x264, it's really difficult to get acceptable quality without having the inloop deblocker on. You really have to saturate the bitrate to excessive levels, relative to with it on, if you expect to receive any acceptable picture quality. Trust us, enable the inloop deblocker at playback and you'll see that the picture quality will be immensely better.

Oh and by the way, the bitrate savings is pretty big. Plus, performance lost from using it can be made up by a smaller bitstream to decode, and with decoders like CoreAVC available for a very reasonable fee ($10 for the basic edition, $15 for the pro I believe) there's no real argument to be made that you can't decode high resolution videos unless you're running a very old system.

[Episodio1]

"enable the inloop deblocker at playback and you'll see that the picture quality will be immensely better."

That's because most of the videos are encoded with inloop/deblock ON.

But if you dont use that filter in enconding, I guess it doesn't matter what you select while decoding. ^_^

[moviefan]

This is what I get with the following settings + post-process deblocking enabled:

--crf 18.0 --ref 3 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,0 --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --threads auto --thread-input --progress --no-dct-decimate --output "" "" --aq-strength 0.5 --aq-sensitivity 5

http://img218.imageshack.us/img218/1936/s2x2642ez6.th.png (http://img218.imageshack.us/my.php?image=s2x2642ez6.png)

It looks a lot better, but the flat areas still don't look good enough for my taste. They look blocky... Any ideas, how to make them appear more smooth?

[Sagekilla]

This is what I get with the following settings + post-process deblocking enabled:

--crf 18.0 --ref 3 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,0 --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --threads auto --thread-input --progress --no-dct-decimate --output "" "" --aq-strength 0.5 --aq-sensitivity 5

http://img218.imageshack.us/img218/1936/s2x2642ez6.th.png (http://img218.imageshack.us/my.php?image=s2x2642ez6.png)

It looks a lot better, but the flat areas still don't look good enough for my taste. They look blocky... Any ideas, how to make them appear more smooth?

You'll want to set trellis to 0, as Trellis and AQ interfere with eachother.

[Episodio1]

try with no filter

[moviefan]

trellis 0 is a little bit better, but it's not yet satisfying... Is there no further way for improvement? Is there no option to set the balance between bitrate for details or for flat areas?

[Dark Shikari]

trellis 0 is a little bit better, but it's not yet satisfying... Is there no further way for improvement? Is there no option to set the balance between bitrate for details or for flat areas?Raise AQ further.

[Sagekilla]

Can you send us a small sample of your video? I'm curious as to what it looks when it's played back on my computer. It seems really odd that you're getting that blocky look to the backgrounds.

[moviefan]

http://uploaded.to/?id=cs82iq

this sample has been created with the following settings:

--crf 18.0 --ref 3 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,0 --subme 6 --analyse all --8x8dct --vbv-maxrate 25000 --qcomp 0.5 --threads auto --thread-input --progress --no-dct-decimate --output "" "" --aq-strength 0.8 --aq-sensitivity 15

But to whatever I change the AQ settings, it doesn't really improve.

[Terranigma]

http://uploaded.to/?id=cs82iq
But to whatever I change the AQ settings, it doesn't really improve.
It shouldn't matter if you use trellis-1, just as long as you don't use trellis-2. Your problem seems to lie with direct-auto conflicting with b-rdo. Never ever use those 2 together. I haven't checked your sample; this is just from personal experience.
Also, try using a CQM that doesn't block, such as EQM AVC-HR.

[Sagekilla]

It shouldn't matter if you use trellis-1, just as long as you don't use trellis-2. Your problem seems to lie with direct-auto conflicting with b-rdo. Never ever use those 2 together. I haven't checked your sample; this is just from personal experience.
Also, try using a CQM that doesn't block, such as EQM AVC-HR.

Really? In my encode settings I use both b-rdo and direct auto, and I never seem to have any trouble with the blocking issues he encounters.


Edit: I'll have to go take an in depth look at my encodes to see how prevalent this may or may not be.

[moviefan]

Setting direct to temporal seems to improve quality but also increases file size by about 20-30% in my sample...

[Sagekilla]

Only setting it seems like you didn't set up there is your ME mode, but I can't see how that would be a problem. Your settings are nearly identical to mine so I'm at a loss as to why it's producing such odd blocking in your video. Is there any chance the source video already has some of those blocking issues already present?

[moviefan]

Hm, I have played around with the qcomp setting and read in a forum that it is basically the bitrate variability... When setting it to 0.0, the file size decreases a lot but picture quality remains more or less stable... So setting qcomp 0 and direct temporal and CRF 18 gives a pretty good picture compared to my first results!

@Sagekilla: The source is pretty good in my opinion as the screenshots show. Or do you disagree? I'm rather new in the video compression business...

[Terranigma]

Really? In my encode settings I use both b-rdo and direct auto
http://forum.doom9.org/showthread.php?p=1045651#post1045651

Also, some stuff are unavoidable such as banding. Perhaps this's what he's experiencing here? The only way you can deal with it, is by using deband on playback. :p

[Sagekilla]

Hm, I have played around with the qcomp setting and read in a forum that it is basically the bitrate variability... When setting it to 0.0, the file size decreases a lot but picture quality remains more or less stable... So setting qcomp 0 and direct temporal and CRF 18 gives a pretty good picture compared to my first results!

@Sagekilla: The source is pretty good in my opinion as the screenshots show. Or do you disagree? I'm rather new in the video compression business...

I'm a bit confused here, I was referring to the original source video fed to 264 to be encoded. I'm asking if that was of good quality, as a poor source will result in a likewise poor output.

Was just a thought.

[moviefan]

I'm a bit confused here, I was referring to the original source video fed to 264 to be encoded.

So did I ;). There is a screenshot of the source that is fed to x264 (source processed by avisynth which is the input). In my opinion the quality isn't too bad.

[Sharktooth]

@moviefan: try removing p4x4 partitions, lowering the subme to 1 or 2 (that means removing b-rdo too), removing trellis and AQ.