H.264 deblock in DGDecode produces artifact

[redfordxx]

Hi, I tried the Deblock() function of dgdecode. It deblocks quite nicely but there remains such grid pattern after the deblocking. It is like 2pixel thick lines. Both horizontal and vertical.
So when the colour smewhere should be:

10 11 12 13 14 15 15 14 13 12 11 11 10 10

it is

10 10 12 12 14 14 15 15 13 13 11 11 11 11

In the other dimension it is the same.

I try to identify it with some convolution and remove it.
Here is my script for identification of the pattern artifact.
LoadPlugin("C:\Program Files\Multimedia\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\Multimedia\AviSynth 2.5\plugins\DGDecode.dll")

d=source.Deblock(quant=25, aOffset=30, bOffset=20)
pattern=d.GeneralConvolution(„-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 –1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 –1”,1.0)
maskh1= pattern.mt_Convolution("1 -1 -1 1 1 -1 -1 1 1","0 0 0 0 1 0 0 0 0", y=3,u=3,v=3)
maskh2= pattern.mt_Convolution("1 1 -1 -1 1 1 -1 -1 1","0 0 0 0 1 0 0 0 0", y=3,u=3,v=3)
maskv1= pattern.mt_Convolution("0 0 0 0 1 0 0 0 0", "1 -1 -1 1 1 -1 -1 1 1", y=3,u=3,v=3)
maskv2= pattern.mt_Convolution("0 0 0 0 1 0 0 0 0", "1 1 -1 -1 1 1 -1 -1 1", y=3,u=3,v=3)
maskv=mt_lutxy(maskv1,maskv2,"x y + 128 -",y=3,u=-128,v=-128)
maskh=mt_lutxy(maskh1,maskh2,"x y + 128 -",y=3,u=-128,v=-128)
mask=mt_lutxy(maskh,maskv,"x 128 - abs y 128 - abs +",y=3,u=-128,v=-128)
Explanation: maskv/maskh are supposed to detect lines in horiz and vert direction. As the lines are 2pixel thick, it is made from two masks shifted by one pixel.

Maybe you can help me with better kernel (both for GeneralConvolution and mt_Convolution) or approach.
After I have mask with the artifact occurence, I will have to rapair the areas of the pattern (that part of script not made yet).

[redfordxx]

Oh now I see... this is equivalent, isn't it?
maskh= pattern.mt_Convolution("1 0 -1 0 1 0 -1 0 1","0 0 0 0 1 0 0 0 0", y=3,u=3,v=3)
maskv= pattern.mt_Convolution("0 0 0 0 1 0 0 0 0", "1 0 -1 0 1 0 -1 0 1", y=3,u=3,v=3)

or better to normalize it somehow?
"1 0 -3 0 4 0 -3 0 1"

[redfordxx]

I made some screenshots, so you know what I mean:

What you see are six views on two different frames.
In first row are: original, deblocked, difference original-deblocked with contrast 25x enhanced.
In second row I made 5x5 convolution "-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1" for the frames in first row to enhace the artefacts. You see the blocking artefacts in first sub-picture, and the grid artefact I am speaking about in second sub-picture of second row.

I made it only in luma not to complicate it.
http://img355.imageshack.us/img355/5097/image16gm.th.png (http://img355.imageshack.us/img355/5097/image16gm.png)
http://img66.imageshack.us/img66/4835/image22pg.th.png (http://img66.imageshack.us/img66/4835/image22pg.png)

This is (simplified) script which I made the screenshots with:
LoadPlugin("C:\Program Files\Multimedia\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\Multimedia\AviSynth 2.5\plugins\DGDecode.dll")

function MyConvolution(clip source_, int type_, float enhance_)
{
matrix_="1"
matrix_=(type_==3) ? "-1 -1 -1 -1 8 -1 -1 -1 -1" : matrix_
matrix_=(type_==5) ? "-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1" : matrix_
divisor_ = 9.0/enhance_
divisor_= (type_==5) ? 25.0/enhance_ : divisor_
g5y=source_.mt_lut(y=3,u=-128,v=-128).ConvertToRGB32("PC.601").GeneralConvolution(bias=128,matrix=matrix_,divisor=divisor_,auto=false).ConvertBackToYUY2("PC.601").ConvertToYV12()
g5u=source_.UtoY.mt_lut(y=3,u=-128,v=-128).ConvertToRGB32("PC.601").GeneralConvolution(bias=128,matrix=matrix_,divisor=divisor_,auto=false).ConvertBackToYUY2("PC.601").ConvertToYV12()
g5v=source_.VtoY.mt_lut(y=3,u=-128,v=-128).ConvertToRGB32("PC.601").GeneralConvolution(bias=128,matrix=matrix_,divisor=divisor_,auto=false).ConvertBackToYUY2("PC.601").ConvertToYV12()
return YtoUV(g5u,g5v,g5y)
}

od=source.Deblock(quant=30, aOffset=30, bOffset=20)
s=Subtract(source,od).ColorYUV(off_y=2)

og5=source.MyConvolution(5,25)
odg5=od.MyConvolution(5,25)
sg5=s.MyConvolution(5,25)

StackVertical(StackHorizontal(source,od,s.ColorYUV(cont_y=24*256,cont_u=24*256,cont_v=24*256)),StackHorizontal(og5,odg5,sg5)).mt_lut(y=3,u=-128,v=-128)I understand from discussions, Deblock is quite used, and I hardly believe you dont have the problem when you take closed look...

[neuron2]

What exactly are you claiming is an artifact? Please tell us what it is without the super sharpening you are doing. In the first row...what are pointing us to as an artifact?

[redfordxx]

By artifact I mean horizontal and vertical lines in some areas.

First, for better description, let's say the attached pictures are made of six sub-pictures like
a b c
d e f
a is the original picture, b is the deblocked one, e is the (you say) supersharpened deblocked one
When you look on e, you see clearly significant lines --- vertical, horizontal or mixture --- on the areas where the deblocker did something.
On the same areas in picture b you can find light lines of same direction.

It is better to see on brighter areas.

On first frame:
-the brightest part of the cloud (vertical lines)
-the part in the middle of the picture, where the cloud meets the sky (both vertical or horizontal)
On second frame:
- the bright part of the outer wave (I don't mean the brightest part of the picture) (horizontal lines)

Just find the most visible lines in e and then you know where to find it on b.

[neuron2]

Can't you just circle the "artifact" with your graphics editor? I still have no idea what you are talking about.

[Didée]

Still without red circles:

http://img294.imageshack.us/img294/11/deblockstripes0jo.png

[redfordxx]

OK here it is:

http://img526.imageshack.us/img526/5550/image30fy.png
http://img526.imageshack.us/img526/4571/image45lj.png

Pictures by Didée:
same thing --- horizontal lines, except that the length is few pixels. So, the lines almost become dots. This happens on edges. Then, where you had nice horizontal edge you have edge with teeth. On 45 degree edges you have stairvay.

When you have the color change not edge-like (examle clouds) the lines are longer.


[EDIT]
FYI:
I have Dell 2405FPW 24 in. LCD Monitor (http://www.nextag.com/norob/PtitleSeller.jsp?nxtg=2c560a1c050a-5902A8EC9C652C58&tag=51696867&lgnode=300265&lgsearch=&chnl=main&ptitle=69657934) at home.
I have the opportunity to look at it with some CRT Dell 20". I had trouble to notice the artifacts which I noticed on 2405FPW easily...
(I didn't bother with monitor settings)

[onesoul]

@ Didée

What quantizer did you use, I guess the same effect happens with default value (25 which I normally use)? And how much is the picture zoomed?
But still I find the produced image very good considering such blocky image.

@ redfordxx

I wouldn't worry about the artifacts that you see, I honestly can't see anything wrong with naked eye, except a more pleasant image to the eyes.

And also I think everyone will agree is that the worse the source is, the harder will be to obtain "perfection", so don't expect miracles.

[redfordxx]

I wouldn't worry about the artifacts that you see, I honestly can't see anything wrong with naked eye, except a more pleasant image to the eyes.

And also I think everyone will agree is that the worse the source is, the harder will be to obtain "perfection", so don't expect miracles.
-I think this artifact hurts the compression
-IMO the source is relatively good
-especially on second frame, the artifact is on spots where no blocking was before, so the spots shouldn't be touched

[onesoul]

-I think this artifact hurts the compression
-IMO the source is relatively good
-especially on second frame, the artifact is on spots where no blocking was before, so the spots shouldn't be touched
I see that you used deblock with different settings from default.
Could you try use just deblock() and compare the outcome?

[redfordxx]

I see that you used deblock with different settings from default.
Could you try use just deblock() and compare the outcome?
You assume correctly, that the amount of artifact depends on the parameters. With defaults in some areas the artifacts are weaker. However, the blocking is not all solved then.

Anyway, the reason a filter has parameters is, that they could be changed, isn't it?

[onesoul]

You assume correctly, that the amount of artifact depends on the parameters. With defaults in some areas the artifacts are weaker. However, the blocking is not all solved then.

Anyway, the reason a filter has parameters is, that they could be changed, isn't it?
Sure you can change it to your wish. But since you were saying that you have relatively good source I see no need to apply such strong deblocking like (quant=30, aOffset=30, bOffset=20) .

From dgdecode manual:
To do default deblocking:
Deblock() #quant=25, aOffset=0, bOffset=0

To do strong deblocking with increased sensitivity:
Deblock(quant=32, aOffset=16, bOffset=24)
Anyway there will always be a trade off. But I am not saying that the artifacts you are seeing can't be solved as I am no programmer.

[redfordxx]

- concerning the deblocking parameters, I've been playing a little with the parameters, so I just posted the current ones. For my source is cca quant=27 enough, but increased sensitivity is necessary. And the artifact is still there.
- trade off... other deblockers have not "this kind of trade off", so I wondered.
- I don't expect that some programmer just sits down immediately and solves the problem. Originally I started this thread with expectations not to discuss whether the artifact is there or whether on default settings it's there (with all respect to you, definitely thanx for joining the discussion). For me it's just there. I started it because
---I want to make some script to clean it so I might use some helping ideas and share the script.
---IMO it happened not only for me but for every Deblock() user
---I wanted to point out that it's there.

[redfordxx]

BTW, look at edit @ post #8.
Maybe, thats why you do not consider this artifact significant...