Strange problem with x264

[raziel666]

Hi
I've been experimenting with x264/MeGUI for a while, but I have come across this problem: I have an anime I want to encode (it's in WMV 1280x720) to x264, without changing the resolution. I have tried the AE-Good Quality profile (and almost everything else), but the result is like this:
http://img247.imageshack.us/img247/2641/pic2we8.th.jpg (http://img247.imageshack.us/my.php?image=pic2we8.jpg)
On the contrary, another encode of the same anime from the same source, has this result, which seems to be much better:
http://img214.imageshack.us/img214/5375/pic1ft3.th.jpg (http://img214.imageshack.us/my.php?image=pic1ft3.jpg)
Can anyone suggest a solution to this? (I don't even know how these colour differences are called). I don't think it's some x264/MeGUI option, but rather a filter that has been used. Anyone's help is appreciated. Thanks a lot.

[SeeMoreDigital]

What does your source frame look like?

[raziel666]

Unfortunately I don't have the source frame right now, because I'm not home. But tomorrow I'll make sure to post it here. But if I remember it looks more like the second screenshot rather than the first one.

[lexor]

On the contrary, another encode of the same anime from the same source
what does that mean? you tried different x264 settings or different codec?

also looking at the large images, I see no difference aside from the pattern inside the circle, is that what the problem is?

[raziel666]

Indeed the pattern in the circle is different because it's a couple of frames later than the first one. The problem is the different colours on the wall (all the left side of the picture). I'm posting the same frame from the raw file (WMV).
http://img219.imageshack.us/img219/5402/pic3eu2.th.jpg (http://img219.imageshack.us/my.php?image=pic3eu2.jpg)
I'm almost certain that some kind of filter has been used. Any ideas?

[Sharktooth]

banding (and some blocking) artifacts.
possible solution: add some noise and rise the bitrate in that "zone".

[raziel666]

Wouldn't the noise ruin the image? Forgive me if I understood wrong, but how can I put noise in the source? To do the test I would compress again with some noise filter, but I don't know if this will fix it...

[lexor]

well comparing with the original I can now see that the very first picture you posted preserve the patterns in the left side of the picture, so its not banding or blocking (at least x264 is not the one that put it there, it just preservers the ones from the source). Only the contrast and/or brightness seems higher. Anything you do in filter stage maybe? Do you use ColorMatrix? Try with and without it.

[raziel666]

Isn't colormatrix available only for mpeg2 streams? Mine is a wmv.

[lexor]

ah sorry, I thought ColorMatrix is universal for anything yv12, so what do you do for avisynth processing?

if you not sure your source is yv12 colourspace, you can put ConvertToYV12(), it won't do anything if it's already yv12 otherwise it'll do the converiosn. This one is safe to have if you ever unsure.

Also try ColorYUV("TV->PC") or "PC->TV", helped me once before with brightess issues.

[raziel666]

My source is RGB32 which I state in the AVISource() filter and then in the end I have the conversion to YV12. I'll try what you said about ColorYUV. Is there some pre-processing filter to use for banding? I searched in the forums but I came up with something for post-processing.

[SpAwN_gUy]

damn... i have the same problem.. and it seems(i've just read this thread) there are no good solutions to avoid theese blockness... even in AE-MaxQuality ...
Noise, IMHO, is not the solution... damn

[raziel666]

I've found a plugin for AFX (Tinderbox 3) which has a DeBand filter which is applied before encoding, but I haven't tried it yet. And yes, noise doesn't seem to solve anything. Has anyone tried this filter?

[check]

Try gradfun: http://project357.com/ul/upload/gradfun2db.rar. It's the debanding filter from ffdshow - and pretty useful. The important thing to realise is that this is a pre-processing problem, no x264 setting will properly fix it for you.

[lexor]

so ColorYUV didn't fix the brightness? or it did and now you want to get rid of the blocks in the source? I must admit I'm a bit at a loss as to where this thread went if the last couple of posts.