Strange blocking problem and x264

[Lil' Jer]

I'm trying to encode the Japanese version of City Hunter - Bay City Wars and I am coming across a weird problem in the opening scene of the movie. Basically it starts with a pan into the City Hunter logo and then the logo flashes and dims again. Now the problem is that with the x264 encode I did this scene gets bad blocks, but oddly enough an XviD encode I did doesn't have this problem and it looks nice and smooth.

Now I really want to do an encode of this with x264 because overall it has less noise around lines and generally less blocky, but this one scene just really stands out as being ugly in the encode. I've tried creating a zone to increase the bitrate to that scene alone but that usually results in even worse blocking than in this encode. I'm really at a loss of what to do and any help would be appreciated.

Here is a shot of the same frame that shows the problem:

x264:
http://img57.imageshack.us/img57/9347/x2642sc.png

XviD:
http://img57.imageshack.us/img57/4017/xvid8ko.png

Here are clips of the scene as it really shows up as looking ugly during playback : x264 clip (http://rapidshare.de/files/14814397/Bay_City_Wars_-_x264.mkv.html), XviD clip (http://rapidshare.de/files/14814423/Bay_City_Wars_-_XviD.avi.html)

And here is an unprocessed m2v file of the scene if anyone wants it: sample clip (http://rapidshare.de/files/14814692/Bay_City_Wars.m2v.html)

Both encodes are done at ~1367kbit/sec to get a 450 meg file with a single aac audio stream.

The command line from the megui-x264 encode is: E:\Program Files\x264\x264.exe --pass 2 --bitrate 1367 --stats ".stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --b-rdo --bime --weightb --filter 2,2 --subme 7 --trellis 2 --analyse all --8x8dct --me umh --merange 32 --sar 1:1 --cqm "jvt" --progress --no-psnr --nr 150 --output "I:\Encodes\City Hunter\Bay City Wars\thing.mkv" "I:\Encodes\City Hunter\Bay City Wars\thing.avs"

The settings for the XviD encode was:

B-VOPS: 2/1.0/1.0
No QPEL, No AQ, No GMC
H.263 Quantization
Motion Search Precision: 6
VHQ: 4 and VHQ for b-frames.
Chroma Motion
Trellis

And the avisynth script I used was:

mpeg2source("Bay City Wars.d2v",cpu=3)
RemoveGrain(mode=5)
FluxSmoothST(Spatial_Threshold=7)
Crop(2,4,-2,0)
LanczosResize(640,480)
ColorMatrix()

[Lil' Jer]

I forgot to mention that I am using Sharktooth's build r408.

[ChronoCross]

I forgot to mention that I am using Sharktooth's build r408.

try upgrading to the latest svn. that might solve your issues as there have been many revisions since 408

[foxyshadis]

Might also try AQ along with adding a custom zone for that scene, weight it 200% or so.

[Lil' Jer]

try upgrading to the latest svn. that might solve your issues as there have been many revisions since 408

Okay, but reading the changelog the vast, vast majority of the changes were either comestic or just optimizations so I don't see how that will necessary fix the problem I'm having. I really don't want to waste another 24 hours of my cpu to reencode something if the new builds aren't actually going to address the problem.

[Lil' Jer]

Might also try AQ along with adding a custom zone for that scene, weight it 200% or so.

Did both, didn't make a difference. As I said I already tried using a zone for that scene but it didn't help. I tried the zone at 200% and then 300%, because 200% didn't help, it just got progressively blockier which made no sense.

[Sharktooth]

use newer builds.

[bond]

sounds like the typical fast-pskip problem

try --no-fast-pskip

[Sharktooth]

he already has it in the commandline

[Daodan]

I don't think newer builds will help him much. My suggestion is to remove that FluxSmooth. I have the same kind of problem when using any kind of temporal smoothing.

[bond]

:stupid:

[Sharktooth]

maybe the unofficial patches may b0rk. so it's better try with SVN builds and see if they fix it.

[shon3i]

--cqm "jvt"try without any matrix or use sharktooth eqm_avc_hr or soulhunter v2.

[Lil' Jer]

I don't think newer builds will help him much. My suggestion is to remove that FluxSmooth. I have the same kind of problem when using any kind of temporal smoothing.

If fluxsmooth was the problem why doesn't it cause any problems when it comes to the XviD encode? What doesn't make any sense is the XviD encode is lower bitrate for that scene yet doesn't have any of the blocks. I'm not trying to dismiss what you are suggesting, it just doesn't make any sense.

[bond]

try a build without the experimental stuff as sharktooth said (and make sure to use --no-fast-pskip)

[Lil' Jer]

try a build without the experimental stuff as sharktooth said (and make sure to use --no-fast-pskip)

Yeah I downloaded the latest build that I can find from http://files.x264.nl/ and testing it.

[Sharktooth]

files.x264.nl?
try x264.nl...

[Lil' Jer]

files.x264.nl?
try x264.nl...

Yeah that's what I meant. Pasted the wrong link. The page with the chuck norris graphic is the one I meant.

[Lil' Jer]

Well I believe I isolated the problem. I downloaded the latest build and used the exact same options and it had the same problems. I even tested with a constant quant 2 encode to eliminate it being a bitrate problem. Commenting out fluxsmooth had no effect. The quant matrix didn't cause it because the blocking appeared when I selected none. The only thing that eliminated the blocking was turning the noise reduction down to 0. I've used it before on other encodes with no problem and with the exact same settings, so I don't know what it is that caused it this time with this source. Very odd.

[nm]

Yes, it is definitely the noise reduction filter. I had the same problem previously (http://forum.doom9.org/showthread.php?p=773196#post773196), but since there were no other reports, it didn't get fixed. My screenshot doesn't show the problem properly, but the clip (http://www.cs.helsinki.fi/u/mikkila/video/x264-r408-nr-1.avi) does.

[Lil' Jer]

Yes, it is definitely the noise reduction filter. I had the same problem previously (http://forum.doom9.org/showthread.php?p=773196#post773196), but since there were no other reports, it didn't get fixed. My screenshot doesn't show the problem properly, but the clip (http://www.cs.helsinki.fi/u/mikkila/video/x264-r408-nr-1.avi) does.

Yep and it's definitely not a problem of setting it too high because for the encode I did above it was set at 150, and I just tested it set to 1 and it had the same blocking. The thing is, it worked really good at eliminating grain from the source, but if its going to cause blocks even at a setting of 1 it's just not worth it.

[foxyshadis]

Oh yeah, I got quite a few weird artifacts with nr when I used it on a few hi-res pics, and forgot to report it, but I think I still have the clip. Another thing is it's very non-mod16 unfriendly, a lot of artifacts started at the edge and ran inward for zoom out scenes. Good catch!

[*.mp4 guy]

Actually I think it has more to do with how X264 handles droped frequencies, heavy denoising just causes a lot of frequencies to become dropable. Back a while ago I was testing a matrix that caused that error to happen (the light/dark blocks problem), the matrix wasn't usefull, I was just checking how X264 would deal with certain things, so I didn't report it.

[foxyshadis]

Well, nr has (had?) some actual bugs, like --nr 1, which should have imperceptible difference, caused huge blocks to appear all over one frame of several specific scenes compared to no nr, which were propogated through the scene. (Fortunately, one was the very first so I noticed.) I haven't tested the new fix to see if it does fix it. If not I'll send my clip to aku to play with.

[Lil' Jer]

I haven't tested the new fix to see if it does fix it. If not I'll send my clip to aku to play with.

Which build has that fix? I tested with 458 and it still has all the same problems of build 408 that I originally saw it in.

[Lil' Jer]

Well looks like build 459 fixes the problem.