Bug with high resolutions?

[Selur]

Can anyone tell me what's causing this (see pic1)?
I used the following commandline with x264 (Core44svn-408M):
L:\Programme\x264\x264.exe --pass 2 --bitrate 700 --stats ".stats" --ref 6 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 7 --trellis 1 --analyse all --8x8dct --qpmin 1 --qpstep 16 --me umh --progress --no-psnr --nr 1000 --output "D:\hdtest\bug.mp4" "D:\hdtest\slow.avs"
bitrate was set through 'AutoEncode' with targetbitrate 6000 using MeGUI.

you can get the source material my encode and the avs script I used here (http://rapidshare.de/files/11577116/x264problem.rar.html). (8MB)

pics1 (from the mp4 file)
http://www.selur.info/misc/bug.png

pic2 (from the avs file)
http://www.selur.info/misc/org.png

Cu Selur

Ps.: If I resize to 704x380 and encode, the problem doesn't appear.
Pss.: Sorry, if this has been reported and I overlooked it.

[nm]

I've got similar blocking with 64-bit r408 (on Linux) and almost the same options except for subme 6, but I haven't investigated it yet further. I encoded at 704x576.

[bond]

i guess thats the same as this:
http://forum.doom9.org/showthread.php?t=105489

[Selur]

ok, I'll test if switching to 406A helps, if someone could provide me with a download for 406A :)
(doing a test with --no-fast-pskip on now => en-/disabling --no-fast-pskip didn't help)

[Manao]

It was supposed to be fixed. Anyway, it's fixed on x264.nl's builds, from what I gathered on irc

[Selur]

ok, I'll get the 408 build from x264.nl then :)

[Selur]

=> yup, it's fixed with the 408 build from x264.nl

[nm]

Which patches do those builds include? I tried x264_check_subpel_predictors.2.diff (http://files.x264.nl/Sharktooth/force.php?file=./x264_patches/x264_check_subpel_predictors.2.diff), but it didn't help.

EDIT: Ok, seems that I have a problem with the noise reduction filter. Still need to try this on a 32-bit machine.

[nm]

My blocking problem is clearly tied to the dct-domain nr option introduced in r398. The more filtering I apply, the more blocking I get. Other options don't seem to matter and I get the same with just crf and nr set and others left to default values (in Mencoder). This happens at all resolutions though, so it might not be the same issue as Selur's. For example the Aeon Flux 480p trailer (http://movies.apple.com/movies/paramount/aeon_flux/aeon_flux_h480p.mov) from Apple. With nr > 500 it already has serious blocking right at the beginning (seconds 7-8).

As I said, I'm using self-built r408 on 64-bit Linux. I need to update my other 32-bit machine a bit before I can test if the same thing happens on it too.

[Manao]

nm : meanwhile, can you post a screenshot ? because noise reduction influences only a tiny part of the code, while Selur's code is due to incorrect mvs being encoded.

[nm]

Here's two shots (frame 175) from the Aeon Flux trailer, encoded with no nr and nr=1000 respectively:

mencoder -noskip -nosound -ovc x264 -x264encopts crf=20 aeon_flux_h480p.mov -o a1.avi
http://www.cs.helsinki.fi/u/mikkila/video/x264-r408-nonr.png

mencoder -noskip -nosound -ovc x264 -x264encopts crf=20:nr=1000 aeon_flux_h480p.mov -o a1.avi
http://www.cs.helsinki.fi/u/mikkila/video/x264-r408-nr.png

There is general blockiness all over the image, but also large blocks at the bottom left corner. More detailed and brighter shots don't suffer as much. Some sources also have lighter blocking at dark parts.

[Manao]

It's not a bug, I think it's simply the effect of the denoising. After all, adding noise is a way to fight blocks from appearing, so removing it might increase the blocks.

Edit : at least, imho, it's not a bug ;)

[nm]

I now tested the same versions of x264 and mencoder on another machine (a Pentium III) with same results. Perhaps a small clip (http://www.cs.helsinki.fi/u/mikkila/video/x264-r408-nr-1.avi) will give a better idea about the artifacts, if you don't mind the crappy AVI :)

I get a lot worse results with some other sources, but I haven't hunted down a publicly available one yet. If this were normal I wouldn't recommend noise reduction to anyone, but I really think there is a bug somewhere, or something is broken in my systems.

EDIT: By the way, x264 doesn't seem to write the nr setting into the bitstream unlike rest of the options. I used nr=500 for that clip.

[foxyshadis]

nr=1000 gives a sort of similar result to filter=6:6, probably for similar reasons. I'm not sure what blocksize it works on, but I'd guess 1000 and maybe even 500 are simply too high. I don't know if there's any guidance about what a good range is; have you experimented with 20, 50, 100?

Try running fft3dfilter with sigma=20 and watch it butcher details and create almost entirely solid blocks in smooth gradients.

If that is how nr is meant to be used, I guess it must be a bug. :p

[lexor]

It's not a bug, I think it's simply the effect of the denoising. After all, adding noise is a way to fight blocks from appearing, so removing it might increase the blocks.

Edit : at least, imho, it's not a bug ;)
such blocking at 6000kbps? not a bug? you kidding me, please tell me you kidding... mpeg2 looks better at that bitrate.

[akupenguin]

If you set an avisynth denoiser too strong, it will introduce artifacts no matter how much bitrate you give the codec afterward.
--nr is a filter, and behaves as such. It is not a compression option. The reason it's included in x264 as opposed to a separate avs plugin is so it can re-use motion estimation data, nothing more.

That said, Selur's looks like a bug. nm's just looks like he set nr too high.

[nm]

nr=1000 gives a sort of similar result to filter=6:6
Yes, maybe in lighter scenes. In dark scenes with certain changes it introduces a similar effect as can be seen in Selur's image and the clip I posted (seconds 7...8 (http://www.cs.helsinki.fi/u/mikkila/video/x264-r408-nr-500.png)). Large gray blocks with sharp edges appear and get left behind. I just tried the King Kong trailer, and it didn't have this problem as clearly, although there are some dark scenes with fades.

I'm not sure what blocksize it works on, but I'd guess 1000 and maybe even 500 are simply too high. I don't know if there's any guidance about what a good range is; have you experimented with 20, 50, 100?
Well, at first I clearly noticed this with nr=400. I get some sharp blocks even at nr=10 and at nr=50 they already disturb casual viewing if the monitor is set slightly brighter than normal. I certainly wouldn't recommend this filter at all if it is supposed to do this, but I still think there is something fishy going on, especially in fading gradients.

[Manao]

lexor : I was speaking of nm's issue. Selur's had already been adressed.

nm : I had a look at your clip. Much more expressive than your screenshots :) Indeed, there's something wrong going on in the bottom left corner.

[lexor]

It is not a compression option. The reason it's included in x264 as opposed to a separate avs plugin is so it can re-use motion estimation data, nothing more.
A denoiser that uses MVs? what kind is that, never heard of such a thing, the ones I worked with use complete frame as an array in memory to do their job. How do you use MVs to denoise?

[akupenguin]

Heard of PixieDust/GoldDust?

[lexor]

Heard of PixieDust/GoldDust?
I've heard of them, but never seen any algorithms for what they do, have you got links for papers on them? I'd appreciate it :)

[Manao]

It's basically using temporalsoften's algorithm, but using the compensated frames instead of the uncompensated one. That helps prevent ghosting when using higher thresholds, which in turn allows effectively to use higher thresholds, and hence to denoise more effectively.

PixieDust/GoldDust are closed source, so one can only guess what they are doing, but since they have the tendancy to create blocks when thresholds are really too high, I guess they use block based motion compensation ( really much like what a codec do, actually ).

[lexor]

so do you have papers for temporal soften algoritm? I never before considered block based denoise, for the very reason of blockiness they introduce when overused.

[foxyshadis]

Temporalsoften (aka TemporalCleaner) doesn't really have any special algorithm, it just combines several nearby frames with various weights, with maximum thresholds. Pixie probably does something similar to chaining them to mvtools.

If you want to see MC denoising in effect, try out Didee's MCNR_simple2. mvbob also uses it, but only for detection, it doesn't output the denoised clip. They use the newer "truemotion", which has its own subtle artifacts but very rarely blocky ones.

[Manao]

Foxyshadis : I like your signature :)

Lexor : i would have advise you to read the source code, but I just had a look at it, better stay away from it. Foxyshadis' description is as good as it get. I can only add the weights are ( or seems to be ) a reversed "hat" ( like 1 - abs(x) ).

[lexor]

ok what you guys are describing is far and wide away from the denoising I know and love (and learned to be the most effective) is based on solving large sparse system of equations, that's derived like in example below

Say your image is the following matrix:
[1 1 1 1]
[2 2 2 2]
[3 3 3 3]

you string it out into vector b:
b=[111122223333] length N=12

then you use model for noise, like a heat diffusion, laplacian, poisson equations to form a five point stancil matrix that is 12x12 (in general NxN) based on length of b, as such it is sparse band matrix (5 diagonals)

then you solve: Ax=b, where A is the matrix above, b is the vector above, and x is your estimate for new solution. You then set b=x, solve Ax=b for x again. Repeat until your measure of error is sufficiently small. The better the model you used to form A the better the denoiser.


So you guys telling me denoisers in AviSynth that I came to take for granted actually don't do it that way?

/P.S. Maybe a mod can split this? we kinda weared off topic here. Sorry.

[foxyshadis]

Most algorithms aren't nearly that complex, they're mostly 3x3, 3x3x3, or higher radius convolutions, with masks or max change thresholds or motion compensation. The other common method is fft->remove hotspots->ifft. Didee might have one that works that way - he certainly knows all about slooooooow filters - but most filters are more geared to be really fast and good enough. TBilateral is probably the most similar to that.

And then we have PeachSmoother, which gathers up the orange smoke and sprikles it onto bad pixels to make them go away. (Take it up with Lindsey.)