bizarre blocking issue

[Blue_MiSfit]

Hey all,

I have been playing around with x264 a bunch recently and decided to do some compressibility tests between it and XviD.

I used Season 2 from R1 Family guy (which is actually all progressive, not that horrible hybrid crap from season 1), and did some "rough" encoding. The results are not that important, but what is important is what happened in my XviD tests. I saw all sorts of really wierd blocking artifacts - which looked to me like the problems x264 has when you try to use multiple consecutive b-frames.

These problems were NOT observed in my x264 encode or the yv12 output of my avs script.

I used DGIndex and GK to setup my script as usual. AVS is as follows:


LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\UnDot.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\FluxSmooth.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\SimpleResize.dll")

mpeg2source("D:\Family Guy Disc 3\212 - Fifteen Minutes of
Shame\Road.d2v")

simpleResize(720,528)

Undot()

Temporalsoften(2,3,3,mode=2,scenechange=6)
mergechroma(blur(1.3))
FluxSmoothST(5,7)


Pretty simple, a 4:3 correction and some light temporal and spatial denoising.

XviD settings were as follows:
Kopei's 1.1 CVS
Constant Quant 3
h263
adaptive quant
bvops @ 3/1.5/1
msp6
vhq4 w/ bvops
chroma motion
all quants capped at 2-31
cartoon mode
chroma optimizer
trellis

Here (http://people.ucsc.edu/~dpresteg/example.avi) is short clip of this happening.

Here is an example of the type of artifacts I am seeing.
http://people.ucsc.edu/~dpresteg/example0.png

[ChronoCross]

make sure your using the latest avs head from celtic druid. it corrected my blockiness issue that I discribed int he offical beta1 thread.

EDIT: I looked at your picture again and realized it is the same issue. so the latest head build should fix the problem.

[Blue_MiSfit]

Where can I get celtic_druid's avs from? so many sites mirror his compiles I'm not sure where to get it, and his http://celticdruid.no-ip.com/ doesn't seem to have it..

I didn't even know he did avisynth...

And on another note, why is the problem avisynth related if I don't see it in the yv12 output or in my x264 encode??

~misfit

[*.mp4 guy]

Its not avisynth related, its Xvid related. Celtic druids latest 1.1 beta compile of Xvid fixes the problem I beleive

make sure your using the latest avs head from celtic druid. it corrected my blockiness issue that I discribed int he offical beta1 thread.

I beleive he meant cvs head.

[Blue_MiSfit]

Thought that might be a typo :)

[Blue_MiSfit]

Confirmed... :)

[ChronoCross]

yeah sorry was typing that in a hurry. I meant "cvs".

[Heini011]

Hi,

i thought that issue was only related to gmc ? so, is xvid 1.1b1 even without using of gmc critical ?

greetings.

[ChronoCross]

you probably shouldn't use the borked version of beta1 to do your encoding.

EDIT: Oh yeah almost forgot we have not verified that it is a GMC only issues because some people have had the problem with trellis instead. it probably has something to do with mv's

[CruNcher]

that is the cartoon mode bug with b-vops i allways talked about
see also http://forum.doom9.org/showthread.php?s=&threadid=88367
try to disable cartoon mode or b-vops and encode it again it should be gone then, thats only a workaround, yet no solution for this problem has been found.

[Blue_MiSfit]

Problems seems to be fixed with an update to the latest Celtic Druid binary. This makes me want to set up MinGW to do my own nightly A64 Builds :)

For the record, I was using trellis, NO GMC, BVops and Cartoon Mode.

Didn't realize beta1 was b0rked :)

~misfit

[CruNcher]

Ok i tested alot of combinations now and i came to the conclusion that cartoon mode doesn't work with adaptive quantization AQ will couse these swiming macroblock problems. So to be on the safe side you have the decission either to deactivate adaptive quantization or cartoon mode, it's up to you. Blue_misfit you should test again i saw that this problem can move with different settings from one scene to another better watch the whole encode to be sure.

[Teegedeck]

Errrrr - hasn't it been known all along that adaptive quantization doesn't mix with cartoon and anime? Psychovisual assumptions about natural video not being applicable for drawn content etc. Nothing new AFAIK.

[CruNcher]

Then when enabling cartoon mode AQ should get disabled and vice versa, that is not the case right now.

[Leak]

Originally posted by Teegedeck
Errrrr - hasn't it been known all along that adaptive quantization doesn't mix with cartoon and anime? Psychovisual assumptions about natural video not being applicable for drawn content etc. Nothing new AFAIK.

Well, adaptive quantization or not - it still shouldn't skip (or otherwise b0rk) whole macroblocks that have more detail in them than others that aren't skipped...

np: Pole - Raum 1 Variation (Burnt Friedman) (R)

[Teegedeck]

Granted, I was just puzzled that AQ was used on such sources at all.

[Blue_MiSfit]

I had forgotten that bit about no AQ with anime... I have been messing with x264 and nero avc a lot recently and sort of forgot about changing the adaptive quant setting.

Anyway the problem is resolved, but now I have a wierd crashing bug with the late 2005.02.21 celtic druid. One of those evil ones when vdub just disapears (possibly at start of second pass) More details later, running confirmation right now.

~misfit


crashing bug seems to have disapeared... never mind :)

I got (with PP on playback) a full resolution 4:3 clip at 410 kbit looking great! Family guy compresses really well.

[lordadmira]

Greetings again.

I got Koepi's beta1 and I experienced the exact same problem. For me it was source related, i.e. no matter how many times I reran the encode the exact same corruption occured in the same spot. I think it is BVOP related because I finally had to set zones around the problem areas with Bsens -999 to alleviate the problem. AQ on, cartoon mode off, GMC on, QP off, Trellis on. Since VHQ is the last major thing to change it makes me think it has something to do with that code. Like certain sources causing some overflows or something. IE gif rendering bug anyone? ;)

On the positive side I've been greatly impressed with VHQ BVOPs. I was literally stunned at the quality increase. It even looks better than a straight quant 2 P encode.

http://w3.goodnews.net/~wagnerc/xvid%20error%201.avi

LA

[CruNcher]

@ lordadmira
the problem you faceing there is the GMC bug it looks the same get an updated CVS build, syskin fixed that bug allready.

[Selur]

Could it be that gmc is still broken?
With the "XviD.cvs.head.exe 21/02/2005 13.28.55" build from celticdruid.
(used AQ, Trellis, Qpel, GMC, VHQ4,...)
=> disabling gmc solved the problem

Cu Selur

[lordadmira]

I can't seem to find it either. Where does he host that?

LA

[BoNz1]

Blue_MiSfit, Isibaar committed a fix for the cartoon mode bug a couple hours ago so please retest with a new cvs head build. Probably someone will make a new xvid build shortly, I don't see a new one up *yet*.

[Selur]

03.03.05 XviD.cvs.head.exe build:
gmc+cartoon => crash, VD closes without a message
gmc, without cartoon => no Problem
cartoon, without gmc => still broken blocks

so seems like cartoon is still broken

Cu Selur

Ps.: using an Athlon64 with (32bit) WinXP

[Isibaar]

Selur, is that crash reproduceable? So is it crashing right at the beginning of an encode for you? Is it always crashing at the same position/frame? And is it crashing always regardless of the input?

I've tested with my own cvs head build, default settings + gmc and cartoon mode (with and without b-frames) and it's not crashing (for a sequence of ~3500 frames at PAL resolution). Also, I don't see any broken blocks (but that doesn't have to mean anything because I don't have any input material where the problem occurs at all).

Could you provide a _short_ sample input clip (plus settings) where you experience the crashing and bad blocks?

[squid_80]

I'm not getting crashes like Selur, but I'm not getting any improvement after patching.

Short source clip here. (http://home.iprimus.com.au/ajdunstan/FILE0015.m2v)

Cartoon mode + Adaptive Quant = A few misplaced blocks
Cartoon mode + GMC = Many misplaced blocks
All three options = Heaps of misplaced blocks

Other options (trellis, b-vops, vhq etc) don't seem to make a difference.

[Sharktooth]

Argh... it's interlaced...

[Selur]

it's reproduce able, it's at the beginging of a clip and I'll try to cut a small sample of the source if you still need/want it.

It's a RC1 DVD source and I used the following clip for ivtc, resizing and cropping:

LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\decomb.dll")
mpeg2source("D:\clips\bug.d2v")
Telecide(order=1)
Decimate()
crop(4,56,712,370)
BicubicResize(704,304,0,0.9)

Cu Selur

[CruNcher]

it happens allways in the middle of the spiderman2 trailer with cartoon mode active for the first pass

[Selur]

over here it's strange I demuxed the mpeg2 stream and created a d2v from the m2v and loaded it with the same script
=> crash after about 287 frames

So I thought fine cut out the first 350 frames. :)
To be sure I made a d2v from the 350frame m2v and tried to encode it.
=> no problem

The above is reproduce able.
Damn thing only crashed when I load the whole clip/m2v,...
So I'm confused.

=> going to investigate more :)

=> Okay, the crash is not Xvid related, if I load the m2v file directly via Virtual Dub Mod (or VD mpeg2) it works without a problem

Seems like Decomb, DGIndex or Avisynth is making trouble :D

Cu Selur

Ps.: did a RAM&Co check yesterday so I doubt that it's a hardware problem :)

=> Found the problem:
DGIndex uses 32bit SSE2 MMX as iDCT Algorithm.
If I change it to 32bit SSE MMX => no crash while encoding with Xvid
(though I got a Athlon64 which should support SSE2, shouldn't it?)
The strange thing is as long as I don't enable cartoon mode there's no problem with "32bit SSE2 MMX as iDCT Algorithm" and x264 and Nero AVC encoding also doesn't make a problem,....

So I guess there might still be a problem/bug xvid.

[squid_80]

Originally posted by Sharktooth
Argh... it's interlaced...
Tell me about it, these season one dvds are terrible. But even if the clip is deinterlaced before feeding to xvid, it still ends up with blocks everywhere.

[squid_80]

I've just applied Isibaar's most recent patch, output is now perfect. So hopefully we can say cartoon mode has been unb0rked and is safe for use. Hooray!

[Chainmax]

I assume celtic druid's next build will have this fix on it? In any case, I didn't fully get what th problem was. Was it Cartoon Mode+BVOPS, Cartoon Mode+AQ or GMC?

[celtic_druid]

Originally posted by Chainmax
I assume celtic druid's next build will have this fix on it?

Yep, although the current build (2005.03.05) has it to.

[squid_80]

Originally posted by Chainmax
I assume celtic druid's next build will have this fix on it? In any case, I didn't fully get what th problem was. Was it Cartoon Mode+BVOPS, Cartoon Mode+AQ or GMC?
There was a separate bug with GMC that syskin fixed a bit over a week ago. The latest bug that Isibaar fixed was causing blocking when cartoon mode+AQ or cartoon mode+GMC were used.