Trouble with an Encode

[Revenant88]

Hi there, I encoded an Anime using MeGui (latest dev) with 10bit x264 2085 patched build by Jeeb. (Have same problems with normal 8 bit revisions, too).

Please look at this Screenshot comparison.

http://screenshotcomparison.com/comparison/82937

If you hover over it, you will see the encode. Please look closely at all the edges and lines -> It looks like a lot of noise has been introduced. This effect is mostly only apparent on darker scenes, too.

What is causing this?
My settings where the following:

I took the preset placebo and changed the following things
for better speed:

CRF 15 (~1600 kbps in the end)
Deblocking 0:0
B-Frames: 6
Ref-Frames 4
MeAlgorithm is umh (multi hex)
Subme 9
MeRange 16
Psy-RD 1.0
AQ 1.0
RC Lookahead 40
No Fast P-Skip is checked

I tried giving a way way lower CRF, but it still produces this "noise". What could I do?

[LoRd_MuldeR]

For this kind of source you probably want to use "--tune anime", which means "--deblock 1:1", "--psy-rd 0.4" and "--aq-strength 0.6".

[Revenant88]

I understood that -tune animation is to be used with classic cartoons. It raises reference frames a lot,
but according to megui log I've never seen more than 4 be used (like 0,1% used 4).
A positive deblock should be used for TV captured video and not for BD sources, as it tends to
slightly blur the video.

Lowering AQ strength would take less bits from lines and fast moving things, mhm, maybe that could help.
I'm not sure about the effects Psy-Rd has here, maybe I should really lower it a lot.

[LoRd_MuldeR]

I understood that -tune animation is to be used with classic cartoons. It raises reference frames a lot,
but according to megui log I've never seen more than 4 be used (like 0,1% used 4).

Well, increasing the maximum number reference frames certainly doesn't hurt.

And Anime generally benefits more from an increased maximum number of reference frames than "real life" footage.

A positive deblock should be used for TV captured video and not for BD sources, as it tends to slightly blur the video.

It's the nature of the source that matters, not where it came from!

Anime may benefit from slightly more aggressive deblocking, while I'd use slightly reduced deblocking for "real life" footage.

Lowering AQ strength would take less bits from lines and fast moving things, mhm, maybe that could help.

Indeed.

I'm not sure about the effects Psy-Rd has here, maybe I should really lower it a lot.

The idea of Psy-RD is that the human eye doesn't just want the image to look similar to the original, it wants the image to have similar complexity. Therefore, we would rather see a somewhat distorted but still detailed block than a non-distorted but completely blurred block. Obviously Psy-RD is a trade-off between improved sharpness (better detail retention) and stronger distortions. It's all about finding the sweet spot! And for "real life" footage the sweet spot generally is higher (around 1.0) than for Anime (around 0.4). Also Psy-Trellis is not recommended for Anime. But try yourself...

[Revenant88]

I get it. Decrease AQ and Psy-rd, increase reference frames.
I haven't seen them making a lot of difference in terms of results in megui report file (as in lower bitrate or something).
Not sure if they do something visually.