DGDecode 1.0.12 + IPP produces artifacts

[Janzki]

I'm using the latest version 1.0.12 of DGIndex and DGDecode.dll. Decomb version is 5.2.1. My AviSynth script is:

MPEG2Source("c:\flcl_disc1\flcl_disc1.d2v",ipp=true,cpu=4)
Telecide(order=1,guide=1)
Decimate(cycle=5,mode=2,quality=3)The post-processing in MPEG2Source produces strange little stain-like blocks that jump all over the video. Below is a short sample:

Uncompressed 1 second clip (http://janzki.bizhat.com/blox.ZIP)

Changing ipp=true to ipp=false doesn't remove the blocks, they just appear at a different time. If I remove post-processing altogether from the line, the blocks disappear. I tried an earlier version of DVD2AVI and mpeg2dec3dg.dll (1.1.0), but that didn't change anything. Neither did removing the IVTC lines.

I would appreciate any help with this, because I like the effect pp produces. I haven't had this problem before.

[Cyberia]

I'm sure Don will want to look at this when he returns. Keep checking for posts from Neuron2, and keep sample files handy.

[Janzki]

Sorry for shamelessly bumping this thread, but I noticed that Donald is back and reuploaded the sample.

Version with DGDecode 1.0.12 (http://janzki.bizhat.com/1012.zip)

I updated DGDecode to the latest beta but that didn't help:

Version with DGDecode 1.0.13 b6 (http://janzki.bizhat.com/1013b6.zip)

^Script for the above was

MPEG2Source("c:\flcl_disc1\flcl_disc1.d2v",cpu=4)
Telecide(order=1,guide=1)
Decimate(cycle=5,mode=2,quality=3)
Program versions used were Smartripper 2.41, DGIndex 1.0.12, Avisynth 2.5.5 and Decomb 5.2.1.

[Guest]

Janzki, I need the *unprocessed* MPEG2 input. Please provide a link to a short stream that demonstrates the problem when your filtering is applied to it.

EDIT: I looked at your 1.0.13b6 output. I don't see any "stains". Can you make a screen dump and circle them or something?

EDIT2: Never mind. I see them now! Please provide input clip.

[Janzki]

Here (http://janzki.bizhat.com/m2v.zip)

I hope I did it right, first time trying this. :rolleyes:

[Cyberia]

I can't see them.

[Janzki]

I had to make room for the MPEG2. But 1.0.13 b6 AVI is up again.

[Cyberia]

OK I see them now. It's definately a PP issue.

In fact, it's in the de-blocking routines. Selecting De-ringing only does NOT produce the effect. (cpu2="OOOOXX")

[tritical]

I'd bet that it is a problem with the horizontal deblocking routine. Looks like artifacts from the 9 tap low pass horizontal filter being used when it shouldn't. Which is probably caused by the fast decision code that is used. I bet if you change the decision code in:

int deblock_horiz_DC_on(uint8_t *v, int stride, int QP)

to loop through all 4 lines and test the differences instead of just testing the difference for the first line of the four it will probably eliminate the problem.

[Guest]

Thanks, tritical. I'll start from there in my investigation.

[Guest]

Originally posted by tritical
I'd bet that it is a problem with the horizontal deblocking routine. Looks like artifacts from the 9 tap low pass horizontal filter being used when it shouldn't. Which is probably caused by the fast decision code that is used. I bet if you change the decision code in:

int deblock_horiz_DC_on(uint8_t *v, int stride, int QP)

to loop through all 4 lines and test the differences instead of just testing the difference for the first line of the four it will probably eliminate the problem. Even if it erroneously decides to filter all the time, how could that corrupt the output? Seems to me it has to be more than a decision bug.

[tritical]

The output does not look corrupt to me. It just looks like the post processing decided that the areas around some of the lines were flat DC regions and decided to use the DC filter (the 9 tap low-pass horizontal filter) instead of the default horizontal filter. And I thought the reason for this was most likely due to the routine I mentioned above which decides whether the 9 tap filter should be used based on pixel differences and quant (it does not make the decision of whether an area is flat or not), but it could be something else. An easy way to see would be to replace the call to the 9 tap horizontal filter with a call to the default horizontal filter or simply comment out the 9 tap filter call.

EDIT: I went ahead and tested it real quick and changing:

INLINE int deblock_horiz_DC_on(uint8_t *v, int stride, int QP)
{
return (ABS(v[1]-v[8]) < 2*QP);
}
to:

INLINE int deblock_horiz_DC_on(uint8_t *v, int stride, int QP)
{
for (int i=0; i<4; ++i)
{
if (ABS(v[1]-v[8]) >= 2*QP) return false;
v += stride;
}
return true;
}
in postprocess.cpp does appear to fix the problem. By default if the above routine returns false then it does no deblocking at all. You could probably change it so that if this routine returns false, it uses the default horizontal filter instead... but I'm not sure if that is a good idea.

[Guest]

@tritical

No, I don't think that is right. With your change, go to frame 26 and look at the right side of the mouth.

I found that I could leave the horizontal decision as it is but I have to disable the vertical DC mode to get rid of that artifact. Maybe both DC modes are broke.

Please try again and see if you agree.

Here's another question. Use this:

a=mpeg2source("bug.d2v",upConv=true,cpu=4,info=false)
b=mpeg2source("bug.d2v",upConv=true,cpu=0,info=false)
subtract(a,b)

Then in Vdub, use levels to amplify the differences. I would have thought the differences would be limited to block boundaries. But you can find frames where there are differences everywhere. Why?

EDIT: After further testing, even with your change, I have to turn off both vertical and horizontal DC mode to get rid of ALL artifacts.

[tritical]

Yep, your right... stupidly I was testing with cpu2 and only had one x in there. The vertical problem is the same as the horizontal. The DC_on check simply doesn't consider enough pixel differences to catch things and uses the low pass filter when it shouldn't. For example on frame 26 around the mouth there are only a few black pixels within that block and they are exactly in the area where the differences are not checked when checking to use the vertical low pass filter. The only real way to leave the low pass filters enabled and eliminate these problems is to include more pixel difference checks in the decision routines. It will make things slower, and still isn't going to be perfect everytime.

For the other question... the default horizontal and vertical filters should only touch pixels directly on the boundaries, but the low pass filters can alter all pixels within the current blocksize I think.

EDIT: Alright, I tried using mplayer's pp decision code, but it still wasn't eliminating all the artifacts with this clip :devil:. So tried these, which are still based on mplayer's code but check more often. They are definitely overkill for most purposes, but do seem to eliminate all the artifacts with this clip and don't seem to be that much slower then the current code:

/* decide whether the DC filter should be turned on accoding to QP */
INLINE int deblock_vert_DC_on(uint8_t *v, int stride, int QP)
{
for (int i=0; i<8; ++i)
{
if (ABS(v[i+0*stride]-v[i+5*stride]) >= 2*QP) return false;
if (ABS(v[i+1*stride]-v[i+4*stride]) >= 2*QP) return false;
if (ABS(v[i+1*stride]-v[i+8*stride]) >= 2*QP) return false;
if (ABS(v[i+2*stride]-v[i+7*stride]) >= 2*QP) return false;
if (ABS(v[i+3*stride]-v[i+6*stride]) >= 2*QP) return false;
}
return true;
}

/* decide whether the DC filter should be turned on accoding to QP */
INLINE int deblock_horiz_DC_on(uint8_t *v, int stride, int QP)
{
for (int i=0; i<4; ++i)
{
if (ABS(v[0]-v[5]) >= 2*QP) return false;
if (ABS(v[1]-v[8]) >= 2*QP) return false;
if (ABS(v[1]-v[4]) >= 2*QP) return false;
if (ABS(v[2]-v[7]) >= 2*QP) return false;
if (ABS(v[3]-v[6]) >= 2*QP) return false;
v += stride;
}
return true;
}

[Zep]

Originally posted by neuron2
@tritical


I found that I could leave the horizontal decision as it is but I have to disable the vertical DC mode to get rid of that artifact. Maybe both DC modes are broke.

Please try again and see if you agree.

Here's another question. Use this:

a=mpeg2source("bug.d2v",upConv=true,cpu=4,info=false)
b=mpeg2source("bug.d2v",upConv=true,cpu=0,info=false)
subtract(a,b)




neuron2 and tritical i just have say it!

it is good to see you two teaming up :D

[Guest]

@tritical

I was thinking to just disable DC mode, but I'll try your new decision functions and see how they work. If they look good, I'll go that way. Thank you for your analysis.

[Guest]

@tritical

The changes look good. I'm going with them for 1.0.13b8.

[milan]

There is probably another bug in prostprocessing, this time in dering.

In dgdecode110src.zip in file PostProcess.cpp from line 2356 is this mov esi, b10x10
lea edi, b8x8filtered
mov eax, stride
mov bh, thr
mov bl, thr
psubusb mm6, mm6
movd mm7, eax <- bug
pshufw mm7, mm7, 0
which as I understand should move previously computed threshold to mm7. But the threshold is in ebx, not in eax (it contains the stride).

[Guest]

Thanks, Milan!

@tritical

Wanna fix this up for us? :) I can't get to it for a while.

[tritical]

Yep, that is definitely a bug and a very old one it seems :D. Think I'll make a c version of the dering code to check all that assembly code with and to make sure there isn't anything else wrong. Another thing about the deringing routine is that it processes inplace, which is technically incorrect since each 8x8 block depends on the surrounding 10x10 pixel block... so that means filtered pixel values could be used when processing the border pixels of the next block. Not sure that would really cause much difference, but will see.

EDIT: just changing the eax to ebx where milan specified seems to be the only thing. After doing so the results match the self_check c version I made. Gonna investigate how much difference there is from doing the filtering in place tommorrow sometime, but doubt it is much at all.

[tritical]

Alight, got to checking the differences and while doing the deringing in place does make some difference (usually very small +-2) it can be larger at times. Even on the largest differences, though, looking at the actual output frames the difference is invisible unless you zoom in so definitely don't think its worth changing.

I do have one question for anyone who knows the deblocking pp stuff well. Why are the useDC and DC_on checks separated and not just both in the same routine? I guess my question is: why is no deblocking at all done for blocks that appear to be DC, but then fail the DC_on test? Why not use the normal deblocking routine on such blocks? Just curious.

[Guest]

Thanks, tritical, I'll change the eax to ebx as you suggest.

Regarding your question, at this point you are probably as expert as anyone about this now. I suggest that you may advise us about the right answer. :)