--chroma-qp-offset -4 fixes "noisy shimmering"

[tombstone]

Hi all,

Thank you for replying to my thread and helping me out. I will now tell you guys a problem I found and a setting that resolved the problem with the x264 encoder.

There is a certain Batman movie that "rises" and has a scene where batman googles catwoman's police profile. At 15-min:40-secs in the movie. When catwoman's profile displays on his monitors, there is a 'noisy' 'shimmering' effect on the letters and lines of his computer display with the x264 encoded result, but is NOT there in the original blu-ray. This noisey effect is NOT improved no matter what bitrate you use, even the highest bitrates don't help it. This is a detection problem with the default setting in the x264 encoder.

I spent the better part of a month adjusting various settings and re-encoding the movie over and over until I found a setting that gave a transparent result and stopped the "shimmering noise" on the letters/lines in that scene.

I added the following command line to my Staxrip encoder:
--chroma-qp-offset -4

As I adjusted the setting from -1, -2, -3, the shimmering effect was reducing and finally gets completely eliminated on -4.

I have applied the -4 setting on ALL my encodes since then and I've found that it doesn't hurt the quality of any other movie I've encoded.

Test this for yourselves if you want. I've verified it ad-nausem. I have other settings adjustments that I believe improve quality. But so that you guys will trust me, I will give you that setting and have you verify it.

[jpsdr]

It could be interesting to have feedback of x264 dev, but in that case, you should provide the full options you're using.
For exemple, according wiki settings, if psy is enabled, the value is automaticaly reduced of 2. So, knowing if you're disabling psy or not have a notable impact.

[tombstone]

It could be interesting to have feedback of x264 dev, but in that case, you should provide the full options you're using.
For exemple, according wiki settings, if psy is enabled, the value is automaticaly reduced of 2. So, knowing if you're disabling psy or not have a notable impact.

Like I said before, I'm was using recommended defaults at the time I discovered the problem.

Settings that don't make any difference on the problem and the problem appears regardless of the setting:

-Subme/Subpixel Refinement (changing Subme doesn't affect the problem, it's there regardless)
-Trellis (same thing)
-Deblocking (same thing, any setting, no difference)
-Bframes
-Ref frames
-RD

Go ahead and try just about any setting you wish. The problem is going to be there. If --chroma-qp-offset value isn't -4, then the problem will be visible.

If I'm not mistaken, various Subpixel Refinement settings are adjusting the --chroma-qp-offset value but it does nothing to remove the problem as those adjustments are not the correct value to overcome the problem. I gave the setting which I tested and verified will fix it.

If you want a basic preset to start with, then use Staxrip's x264, preset medium. The problem will be visible with that preset and any others you try. It simply won't matter. The problem is in the chroma-qp-offset parameters that are the default in the encoder. Whether this is a problem with some code and my adjustment is correcting errors in the encoder or if the originally chosen setting simply was not a good recommendation, I cannot say. All I can do is point out the setting that fixes the issue and hope that someone will find the cause and either implement a code-fix, or simply use the setting that corrects the problem.

[AMED]

Hi Tombstone,

Any chance you can upload a couple of samples that can be used to reproduce the problem?

[Sparktank]

I just did a test of The Dark Knight Rises (2012) with a personal BD rip.

I see some aliasing for the very thin lines on the computer screen and some chroma noise(?).
The lines appear to show more colors than the hues of blue it's meant to present.

-------
What was your encoding process/environment?
Did you use frameserve at all with Avisynth or let x264 index it?
What version of x264 are you using?

Although, since you're talking about Staxrip, I don't imagine you're using much else besides the provided software that's bundled with Staxrip.

------

My environment/process chain:
Windows 7 (x64)
ZOTAC NVidia GeForce GT 640 (2GB)
AVS 2.6 MT (SEt)(x86)
x264: x264_rev2431 [tMod](x64)
avs2pipemod: ver 0.4.2m (built on Jan 20 2013 15:09:37) (x86)

1) remux with MakeMKV
2) remux sample with video only (targeted frames in question: 22484-23354 (24000/1001 based_Region A))
3) index with DGnvDecode (crop; I know the movie switches AR, but for that isolated scene I cropped; going to do encode WITH bars to simulate full movie encode that maintains "IMAX-ish" AR)
4) avs2pipemod -> x264

x264 encode settings:
==============================================================================================
EDITING IN PROGRESS...
==============================================================================================

(Which translated to this in the encode log....)

==============================================================================================
EDITING IN PROGRESS...
==============================================================================================

Although, something there is changing the chroma-qp-offset to -7 automatically.
I could do the encodes again, removing --device and --tune. I'm not %100 sure of all the settings for those two options.

But that whole encode gave me near transparent results with the source, with the computer screen in a little clearer image than the source.

I'll get some screenshots and the samples uploaded in a few. Once I've done the other tests without the additional settings (tune/device).

EDIT: Without any samples or (lossless) screenshots, we have no distinction on these "shimmering" effects.
I also notice when I "step" through the sample with DGnvDecode, I can see reflections of Bruce Wayne and Alfred.
Adequate samples and evidence would surely substantiate your claim (word of mouth).


EDIT2: There's also additional information that --chroma-qp-offset will subtract 2 (and subtracting from a negative value goes further into the negative values).
MeWiki: "When using psy options are enabled (psy-rd, psy-trellis), x264 automatically subtracts 2 from this value to compensate for these optimisations overly favouring luma detail by default."
http://mewiki.project357.com/wiki/X264_Settings#chroma-qp-offset

So, when you state --chroma-qp-offset -4, it's taking into account what other settings you've placed and generates a total value to offset.
In case where "--tune film" is set with "--chroma-qp-offset -4", you get a total of -7 for the CQO in the final encode.

Still doing test samples (cropped, non-cropped (for IMAX-ish encode where AR differentiates), with tune/film, without tune/film, with device/dxva, without device/dxva, with tune+device/film+dxva, without tune+device/film+dxva)...
I've also trimmed the video sample to frames that involve the monitor only (now totalling 108 frames instead of 800+).

EDIT3: I will have to another set without "--no-fast-pskip" (as I currently include it in all my encodes) to ensure it has no direct effect on the resulting encode.
EDIT3b: "--no-fast-pskip" is only stated in the "placebo" preset. All 'faster' presets must invoke the option manually.

[tombstone]

I just did a test of The Dark Knight Rises (2012) with a personal BD rip.

I see some aliasing for the very thin lines on the computer screen and some chroma noise(?).
The lines appear to show more colors than the hues of blue it's meant to present.

-------
What was your encoding process/environment?
Did you use frameserve at all with Avisynth or let x264 index it? What version of x264 are you using?

I've done the same with all my videos. Here is an example of a notepad log from Ben Hur that I kept:

Index with ffmsindex
------------------------------------------------------------

E:\Video Temp\Ben Hur\Ben Hur Full.mkv
"C:\Program Files (x86)\Staxrip\Applications\AviSynth plugins\ffms2\ffmsindex.exe" "E:\Video Temp\Ben Hur\Ben Hur Full.mkv" "E:\Video Temp\Ben Hur\Ben Hur Full temp files\Ben Hur Full.ffindex"

AviSynth Filters
------------------------------------------------------------

FFVideoSource("E:\Video Temp\Ben Hur\Ben Hur Full.mkv", cachefile="E:\Video Temp\Ben Hur\Ben Hur Full temp files\Ben Hur Full.ffindex")
AssumeFPS(23.976)
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()

---------------------------

Writing library: x264 core 128 r2216 198a7ea

That is the version I used when the problem was discovered.

Windows 7 (64-bit)
Intel Core i5-2500k CPU @ 3.3GHz
8GB Ram
Sapphire Radeon HD 7970 OC with Boost 3GB DDR5

Writing library: x264 core 128 r2216 198a7ea
mkvmerge v5.8.0 ('No Sleep / Pillow') built on Sep 2 2012 15:37:04
libebml v1.2.3 + libmatroska v1.3.0

Make MKV is creating the original from which I encode.

These days I don't use the auto-crop with Staxrip (I uncheck it because I've found its cropping is not accurate and distorts the Aspect Ratio on playback in Media Player Classic), but I believe I did use it back then when doing the experiments. However, I remember defeating cropping just in case that was the issue. It didn't stop the shimmering effect. As I said, only the qp-comp setting adjustment resolved it. I don't have any fancy video editing software to slice up pieces of video to create examples for you. And the shimmering effect does not show up in still frames, you must 'watch' the video progress in real-time to see the 'metallic-like shimmering' on the letters and lines.

Look, it's pretty simple. Use any preset you want in Staxrip. Preset medium will suffice. Use any Quality number you want. Whether it's 22 or 18 crf, it won't matter. The shimmering will be there. Bitrate has no effect on the problem.

http://i197.photobucket.com/albums/aa278/tspence73/PresetMedium_zps773b1bdb.jpg (http://s197.photobucket.com/user/tspence73/media/PresetMedium_zps773b1bdb.jpg.html)

And you will see, pretty easily, a metallic-like shimmering on the letters/lines of the computer screen display. It's pretty obvious.

Then make an encode but add the qp chroma offset command with -4. The resulting encode will be identical to the original in this regard.

Although, something there is changing the chroma-qp-offset to -7 automatically.

Yes, when using subme-9 (or was it subme-7? I can't remember precisely, it could be 7 or 9) or greater, I observed that the qp-chroma is adjusted to -7 (when using the --chroma-qp-offset -4 command) due its normal behavior. But I can tell you, regardless of the subme setting, whether it's 1 or 10, doesn't matter, the default qp-chroma offset will cause that scene to have a shimmering effect on the letters and lines on the computer monitor. On EVERY setting, no matter the offset that the encoder applies to the qp chroma offset command, it will NOT fix the issue unless you use the -4 value. I've encoded the damn movie so many times to test around this to know that only the -4 value will fix the problem. No other setting you could use will do anything to cure the problem.

Oh, and I'm using Staxrip 1.1.8 apparently, but I doubt the version of x264 is going to make a difference if this problem was never previously recognized in another video and tuned to fix it. I have been hesitant to try an newer version after finding really good settings that work for me on r2216. I am considering running a test just for The Dark Knight Rises. But will take a couple of days as I'm also experimenting with The Ten Commandments and trying to nail down the film grain in medium to dark gray areas. Current problems take precedence over already resolved problems that I would be re-visiting. lol

The very PRECISE moment to pause at is 15:40. It's THAT screen, not the one with the newspaper copies, but the one with the Selina Kyle file. The letters and lines shimmer and you only get a second or two to see it until Alfred moves his head out of the shot, then it cuts to another view of Wayne. You mentioned Bruce and Alfred's reflection off the screen, so I think you could be on the right moment.

[Sparktank]

I would post a picture of this, but I don't want to violate any copyrights.

It wouldn't violate copyright laws unless:

A) you actually don't own the blu-ray and pirated it.
B) you live in a country that's completely Orwellian.

You're not posting the whole movie, just a screenshot.
Many others post screenshots of their problem and any future progress.

If you're truly paranoid, I'll be upping a trimmed video of that shot (remuxed in matroska format, video only), anyway. And multiple screenshots for evaluation so that others can have fun.

[Sparktank]

I have been hesitant to try an newer version after finding really good settings that work for me on r2216.


That's superstitious.
Updates for x264 are generally designed to improve perfromance/quality and to remove deprecated code that would otherwise compromise said performance/quality.
Bugfixes are a cause for concern, but enough to not update.
Bugfixes are bound to be fixed and new ones arise.

I'm still hectic on this end doing multiple things, but the samples will be up within 24 hours.

I'm quite curious about the relation of --chrome-qp-offset (and thereof of --no-fast-pskip).

[tombstone]

That's superstitious.
Updates for x264 are generally designed to improve perfromance/quality and to remove deprecated code that would otherwise compromise said performance/quality.
Bugfixes are a cause for concern, but enough to not update.
Bugfixes are bound to be fixed and new ones arise.

I'm still hectic on this end doing multiple things, but the samples will be up within 24 hours.

I'm quite curious about the relation of --chrome-qp-offset (and thereof of --no-fast-pskip).

There is another movie with a problem that I was not able to solve with any settings I tried and only brute force bitrate reduced the issue (but did not eliminate it). I'd like to bring that up, but maybe it's for another thread. I'm new here, so I don't want to be too presumptive by making lots of threads. I want to help out with one thing at a time (assuming I can even help people as experienced as you guys). I evaluate my encodes very obsessively but I'm sure there are things I'm fooled by that I don't see wrong. I'd like to show screen shots of some good results and also screenshots of some moments that are, IMO, encoder killers (where the original Blu-ray wasn't even using enough bits and should be done over completely).

I realize I'm interested in this stuff pretty late in the ballgame when x264 looks really good and is down to minor refinements, but there is still room to improve it and I think improve the recommended settings.

[Guest]

We only need the original and your encode parameters. But if you want to provide both that's OK too. Can you post the source snippet right away and the encode later so we don't have to wait? Since it happens with many sources according to you, can you at least give us your x264 encode line so we can do some experiments? You were already asked for that but demurred, unless I missed something.

[tombstone]

Okay. Here's a link and I hope it works, never used it before. Always have kept my files local.

This is the link mediafire gave me to share my folder.
https://www.mediafire.com/folder/37pbmuweev99x/Videos

[Guest]

Perfect. Thank you.

Can you give us your x264 encode line also please? Not everyone has Staxrip and to make things very concrete let's make sure we use the same encode parameters.

If your finding pans out, having these things will make it easier to approach the developers for advice/explanation.

[tombstone]

Perfect. Thank you.

Can you give us your x264 encode line also please? Not everyone has Staxrip and to make things very concrete let's make sure we use the same encode parameters.

Like I said, the Medium preset, but I'm giving a little more bitrate with a crf-19 quality rating just so people don't think it's an issue of low bitrate.

cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x133 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=0 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=19.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00

That is what the command line looks like for the preset medium.

Also, remember, I've been using r2216 core, not the most recent versions. So bear that in mind. I haven't been able to confirm whether the most recent releases do a better job on this issue. I have my doubts things would've improved if the developers were specifically tuning for this issue though.

[tombstone]

Aww man, I just noticed that I uploaded the wrong sample. The one I uploaded is a moment too early, but still shows the computer screen, just not as long as the clip I meant to send. I'll upload the sample with the longer look at the problem. Both clips should have the problem with the shimmering on the letters and lines (possibly aliasing noise?) after encoding. Sorry, new submitting this stuff.

There. The better sample should be there now. Labeled, "Problem Sample 2"

[Guest]

Thanks for the updated stream and letting us know the x264 version.

Sorry to be a pain but that isn't a command line. Can you give the command line? Or is it not possible to see it with Staxrip?

[tombstone]

Sorry to be a pain but that isn't a command line. Can you give the command line?

Copied straight off the Staxrip encode of the commandline it's using while it's running:

"C:\Program Files (x86)\Staxrip\Applications\x264\x264.exe" --crf 19 --output "E:\Video Temp\Dark Knight Rises\The_Dark_Knight_Rises_t01 temp files\The_Dark_Knight_Rises_t01_new_Output.h264" "E:\Video Temp\Dark Knight Rises\The_Dark_Knight_Rises_t01 temp files\The_Dark_Knight_Rises_t01_new.avs"

[Guest]

Ah, now we're cooking with gas. Thank you very much.

[tombstone]

Ah, now we're cooking with gas. Thank you very much.

Thanks. Hey, I just wanted to mention that I verified the problem on multiple monitors in my home. The living room theater TV, the PC monitor and my 720p bedroom TV all show the problem. Even when played back through Bluray players, using two different players when streaming from my PC.

[Guest]

Thanks. You've given our experts what they need to analyze this. I'm sure we will be hearing from them again shortly. :)

[tombstone]

Thanks. You've given our experts what they need to analyze this. I'm sure we will be hearing from them again shortly. :)

So, who are the experts?

[Guest]

We'll see. :)

[tombstone]

We'll see. :)

No, really, who are the people working on this? I'd like to know. I'm interested to know who works on x264.

[Guest]

Just review the forum's threads and posts. Or search the internet and commit logs for x264.exe. And please stay on topic for this thread. Thank you.

[SamNeal2001]

I just wanted to chime in and say I could not re-create the error on the commandline, but could with StaxRip. It seams if you keep the deinterlace ON (which is on by default) even though the source isn't interlace, the decoder assumes it could be. And those lines are small enough for it to think so, thereby, creating this bad shimmering.

Clicking the option off fixes it.

[jpsdr]

So, finaly, it wasn't realy an issue with x264, but just wrong settings, encoding in interlace mode a progressive video, resulting in some unexpected side effects.

[tombstone]

I just wanted to chime in and say I could not re-create the error on the commandline, but could with StaxRip. It seams if you keep the deinterlace ON (which is on by default) even though the source isn't interlace, the decoder assumes it could be. And those lines are small enough for it to think so, thereby, creating this bad shimmering.

Clicking the option off fixes it.

Sure enough, you are correct. Staxrip's interlace box is checked on by default and causes the shimmering. This is great news. I will be re-encoding a number of videos.

Also, I noticed something else about Staxrip. The Aspect Ratio is getting messed up (the top and the bottom of each frame of video is being squeezed down/distorted a couple of percent. (You can verify this by taking screen shots then comparing the frames in Window's photo viewer). This is occurring when "Crop" is checked, which is located right under the "Deinterlace" box. BOTH of those need to be recommended to be unchecked in order to preserve the original source accurately and the authors need to be informed so that the software can be fixed.